Why we need to use jobs function to train models?

• Sometimes, We have to do multiple model training in parallel, because tasks take considerable time to complete, users are not able to engage with the whole of the progress.
• We can use Job Submission to create a job of sequential multiple tasks and submit the job for execution at background, meanwhile we can monitor the progress from the log.
• We sometimes want to submit certain jobs regularly according to schedules. We can create Schedules which are responsible for submitting jobs regularly according to the recurrence setting for us.

Why need to transfer code to .py file?

If you want to use jobs or schedules function to training model with different parameter in the same time. You can modify .ipynb code to .py code and help you run the code easily.

Tutorial

Prerequisite

1. Finish writing the .ipynb code and want to transfer the code to .py file.
2. Arrange the parameter we want to add into the code. Here, we use json format to input the parameter.

Step 1: Transfer .ipynb code to .py file.

• If you have the following ipynb file and want to transfer to python script file.

  • test.ipynb

• Go to File → Export Notebook as... → Export Notebook to Executable Script and download the .py file.

  • Snapshot

• Upload test.py into jupyterlab and you will see the python script content.

  • test.py

• Run the script to test the script is runnable.

  $ python test.py
  

Step 2: Use sys python package to setting parameter.

• Example:

  If you finish exporting Notebook to executable script have the following code:

  • test.py

  You want to get file_path parameter in command line arguments, then you need sys and json python package to get the arguments. After we modify the code, you will see the following code.

  • test.py

  Run the command line and the script will read your specific file:

  python test.py '{"file_path": "test.txt"}'
  

  If you want to run the same script on different parameter in parallel, then you can change the command arguments to quickly fulfill the demand.

  python test.py '{"file_path": "test.txt"}'
  python test.py '{"file_path": "test2.txt"}'
  python test.py '{"file_path": "test3.txt"}'
  

Step 3. Assign the job to run the script file.

• Come back to PrimeHub Platform and create a job. Please click jobs → + New Job.

• Enter relevant information.

  Job information

  • snapshot