Automating Jobs with Pipelines
What is a Pipeline
Pipelines allow you to run take multiple jobs and run them either in parallel and/or after a previous set of jobs finished.
This allows you to create many jobs to sync all your content types, and run all the jobs by starting a pipeline rather than running every single job. Effectively, "pipelines" are a collection of "steps" and "steps" are a collection of "jobs."
A pipeline will run all the steps in order. When a step is run all the jobs in a step will be run at one time. So, if you want to run job1 and then run job2 after job1 has finished. Then job1 and job2 will be in different steps. job1 will be in a step that runs before the step that job2 is in. Or if you want to run job1 and job2 at the time, then you will create a step with job1 and job2 in it.
Pipeline Screen
You can view all the pipeline and create a pipeline from the "Pipelines" screen. You can get to the screen by selecting "Pipelines" from the left.
Creating a pipeline
Walk through
While on the pipelines screen, you can create a new pipeline by pressing the "Create Pipeline" button in the top right.
This will take you to the pipeline creation screen. To create a pipeline you will need to add both "steps" and "jobs" to the pipeline.
Add a Step
The first action to take when creating a pipeline is to add a step. This can be done by pressing the "Add Step" button.
Once a step is added, you will see the newly created, empty step displayed and the jobs you can add to the step.
The next action to take is to add jobs to this step.
Add Jobs
Now that you added a step, you can now add jobs to that step. The motel window on the side will display the different jobs created that can be added to the pipeline step.
To add a job to a step, simply drag the name of the job you want to add to the step you want to add it to.
We can add two jobs to this step so when the pipeline runs, both of these jobs will run in parallel before the next step.
Adding additional steps
Now that the previous step has jobs you can add a new step to the pipeline. Like the first step created, you can add a new step by pressing the "Add Step" button again.
Now that a new step exists, you can add jobs to this step as well.
Saving a pipeline
Pipelines are saved as jobs are added or removed. So, there is no "Save Pipeline" button to press. Simply edit the pipeline and it will save as you work on it. You can select "Pipelines" from the left again to view the list of all pipelines and see your newly created pipeline.
Running a pipeline
Now that you've created a pipeline, you can find pipeline in the pipelines screen.
Once you find the pipeline you created you can run it by pressing the "Run" button in the "Transaction" column.
Than you can see the Run button will change into spinner, to indicate that the running process is starting
Then if the pipeline is able to run, the status column will change into the progress bar, with the progress details and number of steps done.
The line "Running with percent" indicates the progress of current step. The line "Finished steps" indicates how many steps are done. For example, in the image above, 0 step is done on total 1 step. That means currently the pipeline is running the first step.
Pipeline Deep Dive
Viewing the Pipeline Screen
From this page you can view all the pipelines created. Each pipeline has:
a name
creation date time
last modified date time
transaction actions
running status
more actions
Transaction Actions
Similar to jobs there are two actions a pipeline can do to start a transaction. However pipeline transactions are different than job transactions. A "pipeline transaction" is a collection of "job transactions". So while regular transactions will be started this will start pipeline transaction or multiple job transactions.
Immediately run a transaction.
More details on can be found in the "Starting a Transaction" section
Schedule when a transaction should run.
More details can be found in the "Scheduling Jobs and Pipelines" section
More Actions
There are a couple additional actions that a pipeline has. These can be viewed by pressing the downward arrow for the pipeline you are interested in.
Edit Pipeline
You can view current configuration for a pipeline and edit it by pressing the "Edit Pipeline" button. This will take you to the same screen as when you created the pipeline allowing you to add or remove steps and jobs to the pipeline.
Cancel Pipeline
You an cancel and stop a pipeline transaction (and thus all the associated job transactions) by pressing the "Cancel Pipeline" button. While a transaction is in progress for a pipeline you can cancel the transaction to stop all jobs currently running for the pipeline and cancel all future jobs from running.
Start Transaction Options
Start transaction options can be set for jobs run in a pipeline. When a pipeline runs any start transaction options set will be used.
To set these options, choose a pipeline to edit and press the pencil icon on a job.
The pencil icon will be shaded in when there are saved start transaction options for a job.
This will take you to a page similar to the edit job page. From here endpoint and manipulator config options can be set to overwrite the options for the base job when the pipeline's step runs.
Once you have configured any overwrite job options wanted, go to the job preview page. From here the options can be saved to the pipeline for this job at this step by pressing the "Save Job Overwrite Options".
After saving the job overwrite options, you will be returned to the edit pipeline page. And can see that the pencil icon is now shaded in. Signifying that job at that step has saved start transaction options.
Any start transaction options applied to a pipeline are saved and will be used when the pipeline runs.
To reset a job in a pipeline with start transaction options to it's default state (remove the options), you should remove the job from the pipeline step and read the job to that pipeline step.
Additional details for start transaction can be found here.
Debug Job
A pipeline can also generate debug reports for jobs.
When editing a pipeline, each job can be set to run in debug and generate a debug report for that job.
You can tell which jobs will run in debug mode if the bug icon is filled in.
In the above example, the article -> article job will be run in debug while the other two jobs will not.
To set which jobs in a pipeline should generate a debug report, simple press the bug icon and the debug option will be toggled. (If the debug option is set, pressing the bug icon will remove the debug option, leaving the job to run normally. And vice versa.)
When the pipeline runs, it will run as normal. With the exception that any job with the debug option set, will run in debug mode with the appropriate rules of a debug transaction.
When the start transaction options for a job are configured, the job will automatically remove the debug option. Be sure to check that the debug option is set appropriately after setting start transaction options for a job.
Full Sync Job
A pipeline can set each job to run as a full sync.
When editing a pipeline, each job can be set to run as full sync.
You can tell which jobs will run as full sync if the globe icon is filled in.
In the above example, the author and article jobs will be run as full sync while the assets job will not run with full sync.
To set which jobs in a pipeline should be run as full sync, simple press the globe icon and the full sync option will be toggled. (If the full sync option is set, pressing the globe icon will remove the full sync option, leaving the job to run normally. And vice versa.)
When the pipeline runs, it will run as normal. With the exception that any job with the full sync option set, will run in full sync mode with the appropriate rules of a full sync transaction.
When the start transaction options for a job are configured, the job will automatically remove the full sync option. Be sure to check that the full sync option is set appropriately after setting start transaction options for a job.
Last updated
