Automating Jobs with Pipelines

{
  "pipelineId": "0c08370b-bf1d-11ec-b314-0242ac150009",
  "name": "updated name",
  "steps": [
    {
      "pipelineId": "0c08370b-bf1d-11ec-b314-0242ac150009",
      "name": "Step 1",
      "order": 1,
      "jobs": [
        "6c454a83-a4ca-4c8f-acb5-ea10521458aa"
      ]
    },
    {
      "name": "Step 0",
      "order": 0,
      "jobs": [
        "87f0e362-82a4-11eb-8dcd-0242ac130003"
      ]
    }
  ]
}

Creating a Pipeline

To create a pipeline, you will want multiple jobs created. After you have created 2+ jobs you can get started with creating a pipeline. You can use the pipeliner API to do this.

Create an Empty Pipeline

The best way to create a pipeline is to do it in iterations. Start by creating an empty pipeline using the /pipelines endpoint.

curl --request POST '{{impulse-protocol}}://{{impulse-domain}}:{{impulse-port}}/private/pipeliner/pipelines'

This will create a pipeline without any steps. The response will include the pipelineId. You should save this value to be reused.

Add a Job to a Pipeline

To add jobs to a pipeline, you will need the:

• Pipeline ID

• Job ID

• Step order

Using all three of these values you can add a job to the pipeline by using the /pipelines/{{pipelineID}/job endpoint. This example adds three jobs to the pipeline. Two at step 0 and one at step 1.

curl --request POST '{{impulse-protocol}}://{{impulse-domain}}:{{impulse-port}}/private/pipeliner/pipelines/{{pipelineID}}/job' \
--data-raw '{
    "jobId": "87f0e362-82a4-11eb-8dcd-0242ac130003",
    "step": 0
}'

curl --request POST '{{impulse-protocol}}://{{impulse-domain}}:{{impulse-port}}/private/pipeliner/pipelines/{{pipelineID}}/job' \
--data-raw '{
    "jobId": "3a72ccda-c383-11ea-87d0-0242ac130003",
    "step": 0
}'

curl --request POST '{{impulse-protocol}}://{{impulse-domain}}:{{impulse-port}}/private/pipeliner/pipelines/{{pipelineID}}/job' \
--data-raw '{
    "jobId": "33791608-58fc-4aef-a3de-42ba71ce0e44",
    "step": 1
}'

Based on the step order provided, the job will be added to that step. When a pipeline runs, it will run all steps in their provided order. So in this example jobs 87f... and 3a7... will run in parallel before job 337...

The response to this request will show the updated pipeline, allowing you to verify that it looks as expected.

{
  "pipelineId": "0c08370b-bf1d-11ec-b314-0242ac150009",
  "name": "0c08370b-bf1d-11ec-b314-0242ac150009",
  "steps": [
    {
      "pipelineId": "0c08370b-bf1d-11ec-b314-0242ac150009",
      "name": "Step 1",
      "order": 1,
      "jobs": [
        "33791608-58fc-4aef-a3de-42ba71ce0e44"
      ]
    },
    {
      "name": "Step 0",
      "order": 0,
      "jobs": [
        "3a72ccda-c383-11ea-87d0-0242ac130003",
        "87f0e362-82a4-11eb-8dcd-0242ac130003"
      ]
    }
  ]
}

Removing a Job From a Pipeline

The same endpoint to add a job to a pipeline step can be used to remove a job from a pipeline step. The payload is the same for both the DELETE and POST requests.

curl --location -g --request DELETE '{{impulse-protocol}}://{{impulse-domain}}:{{impulse-port}}/private/pipeliner/pipelines/{{pipelineID}}/job' \
--data-raw '{
    "jobId": "87f0e362-82a4-11eb-8dcd-0242ac130111",
    "step": 0
}'

The response to this request will show the updated pipeline, allowing you to verify that it looks as expected.

{
  "pipelineId": "0c08370b-bf1d-11ec-b314-0242ac150009",
  "name": "0c08370b-bf1d-11ec-b314-0242ac150009",
  "steps": [
    {
      "pipelineId": "0c08370b-bf1d-11ec-b314-0242ac150009",
      "name": "Step 1",
      "order": 1,
      "jobs": [
        "33791608-58fc-4aef-a3de-42ba71ce0e44"
      ]
    },
    {
      "name": "Step 0",
      "order": 0,
      "jobs": [
        "3a72ccda-c383-11ea-87d0-0242ac130003"
      ]
    }
  ]
}

Updating a Pipeline

To update the pipeline with a new name or add multiple jobs/steps in a single request you can use the /pipelines/{{pipelineID}} endpoint.

First you will need to get the current pipeline.

curl --request GET '{{impulse-protocol}}://{{impulse-domain}}:{{impulse-port}}/private/pipeliner/pipelines/{{pipelineID}}'

The response will return the pipeline object. You can then manipulate this object to add more steps, jobs, or update the name of the pipeline.

For example, taking the above response you can change the name of the pipeline.

{
  "pipelineId": "0c08370b-bf1d-11ec-b314-0242ac150009",
  "name": "my-headless-pipeline",
  "steps": [
    {
      "pipelineId": "0c08370b-bf1d-11ec-b314-0242ac150009",
      "name": "Step 1",
      "order": 1,
      "jobs": [
        "33791608-58fc-4aef-a3de-42ba71ce0e44"
      ]
    },
    {
      "name": "Step 0",
      "order": 0,
      "jobs": [
        "3a72ccda-c383-11ea-87d0-0242ac130003"
      ]
    }
  ]
}

Once the object has be edited you can send it as a payload to the same endpoint in a PUT request.

curl --request PUT '{{impulse-protocol}}://{{impulse-domain}}:{{impulse-port}}/private/pipeliner/pipelines/{{pipelineID}}' \
--data-raw '{
  "pipelineId": "0c08370b-bf1d-11ec-b314-0242ac150009",
  "name": "my-headless-pipeline",
  "steps": [
    {
      "pipelineId": "0c08370b-bf1d-11ec-b314-0242ac150009",
      "name": "Step 1",
      "order": 1,
      "jobs": [
        "33791608-58fc-4aef-a3de-42ba71ce0e44"
      ]
    },
    {
      "name": "Step 0",
      "order": 0,
      "jobs": [
        "3a72ccda-c383-11ea-87d0-0242ac130003"
      ]
    }
  ]
}'

As stated in the API doc, you will need to list every value in the request. Otherwise the value will be defaulted and likely lost.

Running a Pipeline

Like a job pipelines can be run. This is done by using the pipeliner's /pipelines/{{pipelineID}}/start endpoint. This request will start a pipeline transaction which is combination of standard syncTransactions .

curl --location -g --request POST '{{impulse-protocol}}://{{impulse-domain}}:{{impulse-port}}/private/pipeliner/pipelines/{{pipelineID}}/start'

The response will include the id of the pipeline transaction . This can be used to track the status of the transaction.

Tracking Pipeline Transaction Status

To track the status of the pipeline transaction, you can use the /pipelines/{{pipelineID}}/transactions/{{transactionID}} endpoint.

curl --location -g --request GET '{{impulse-protocol}}://{{impulse-domain}}:{{impulse-port}}/private/pipeliner/pipelines/{{pipelineID}}/transactions/9bfe48f8-bf1f-11ec-bccf-0242ac150009'

The response will show the list of sync transaction started by the pipeline. You can then view the status of each individual sync transaction. The response will also show if the pipeline transaction is active and the last step that was completed by the pipeline.

Canceling a Pipeline Transaction

Similar to a sync transaction you can also cancel a pipeline transaction. You can use the /pipelines/{pipelineID}/transactions/{transactionID}/cancel endpoint.

curl --request POST '{{impulse-protocol}}://{{impulse-domain}}:{{impulse-port}}/private/pipeliner/pipelines/{{pipelineID}}/transactions/{{pipelineTransactionID}}/cancel'

This will cancel all sync transactions related to this pipeline transaction stopping the processes from continuing further.