What to do if I run into a Job Problem

As happens in software, you will run into a bug where a job doesn't work like it should (or you expect it to). When this happens, you can try steps listed in this section.

Some of the things to look for are:

• Is the job config correct?

  • Make sure the correct endpoints are selected and the job options are correct.

• Is the endpoint config correct?

  • Make sure the connection to the system is correct.

• How long has the transaction been running?

  • Depending on the amount of content being synced, a long transaction might be expected.

• Are there alerts or notifications?

  • Check if the job continually retrying (more than 3 times).

• Are there error log messages?

  • Check if there a specific error that is causing the job to stop.

• What is the last transaction detail run?

  • Check where in the process the transaction is.

• Are there currently running heartbeats?

  • Check if any heartbeats are currently failing.

Contacting Support

Before contacting support you should gather specific information. View the "Troubleshooting via UI" to see how to use the UI to troubleshoot and get the information.

Details to Send

1. A detailed description of the problem

2. Transaction ID for problematic transaction

3. Last 5 unique transaction report detail types for problematic transaction

   1. including detail data for some of these records will be helpful as well.

4. Messages for the date range as the transaction

   1. The last few messages may not be enough, especially if many jobs are running. It is best to send as many message as possible.

5. Running heartbeats

6. All heartbeats

Finding the Transaction ID

Transaction IDs can be found in the UI by pressing the "Show More Details" button for a report that is associated with a job that is not working.

The transaction ID can also be gotten from the sync-manager service by using the /report endpoint.

curl --request GET '{{impulse-protocol}}://{{impulse-domain}}:{{impulse-port}}/private/sync-manager/report?start={{startTime}}&end={{endTime}}&limit=100&detailLimit=100'

The response will show multiple transactions along with their status, related job, start time and end time, as well as their details. This can be used to find the problematic transaction.

Getting Transaction Details

Once you know the transaction ID, you can then find the details for the transaction.

these details can be retrieved from the sync-manager service by using the /transaction/{{transaction-id}} endpoint.

curl --request GET '{{impulse-protocol}}://{{impulse-domain}}:{{impulse-port}}/private/sync-manager/transactions/{{transaction-id}}?details_limit=-1

This request will get all the details for the transaction. From this you can select the last few transaction details to send to support or send the entire list of details.

Getting Transaction Logs

You can get the logs by from the logger service by using the /transactionLogs endpoint.

curl --request GET '{{impulse-protocol}}://{{impulse-domain}}:{{impulse-port}}/private/logger/transactionLogs?start={{startTime}}&end={{endTime}}&transaction={{transactionId}}'

This will return a list of all the logs for a transaction ID. You can then save this list and search for "error" to find all the logs with "error" in it.

Getting Logs for Time Range

With the transaction details, you can get the time range of the transaction to get all logs for that time range. You can get the start time from the transaction's start time.

If the transaction is inactive the end time should be the time the transaction completed. However, if the transaction is still active, the end time should be at least 5 minutes after the latest transaction detail's time.

You can get all the logs for a time range by using the logger service's /logs endpoint.

{{impulse-protocol}}://{{impulse-domain}}:{{impulse-port}}/private/logger/logs?start={{startTime}}&end={{endTime}}

This will return all the logs within the start and end time parameters used.

Getting messages

Messages will contain any details while running a transaction. This includes any errors, general messages, and warnings.

You can get messages from the transaction that is using sync-mangers call In order to do so you need the message transaction id which is located in the reports page for that job.

curl --request GET '{{impulse-protocol}}://{{impulse-domain}}:{{impulse-port}}/private/sync-manager/message/{{transactionId}}'

Get Running Heartbeats

You can get the running heartbeats by using the sync-manager's /transactions/{{transactionId}}/heartbeats/running endpoint.

Use time stamps to filter for all running heartbeats with a start time matching the transaction start time.

curl --request GET '{{impulse-protocol}}://{{impulse-domain}}:{{impulse-port}}/private/sync-manager/transactions/aee621f7-e68b-11ec-b2dd-0242ac190013/heartbeats/running?start={{transactionStartTime}}&limit=-1'

Get All Heartbeats

To view the heartbeats of a transaction use sync-manager's /transactions/{{transactionId}}/heartbeats endpoint.

Use time stamps to filter for all heartbeats with a start time matching the transaction start time.

curl --request GET '{{impulse-protocol}}://{{impulse-domain}}:{{impulse-port}}/private/sync-manager/transactions/{{transactionId}}/heartbeats?start={{transactionStartTime}}&limit=-1'

Get Content Diffuser Data

In Impulse the Content Diffuser Process Tables data is what tells us what exactly is happening at the heart of any sync. We have provided several REST endpoints to expose data from these tables:

1. delivery_status_by_endpoint: Status of each element that was sent to the Dropoff Taxi to Impulse Sync.

2. content_file: Information of each element that Diffuser receives when Impulse Sync starts.

3. content_status_by_endpoint: Processing Status of each Main Content by Endpoint.

4. content_by_endpoint_in_progress: Status of each Main Content by Endpoint that are being processing in a specific time.

5. content_by_processor: Processor that each element is in.

6. total_content_files_by_endpoint: The amount of contents that each endpoint that are expecting. Useful to see how many are missing/waiting and could be cross reference with the data from content_file.

7. motations_missing_by_endpoint: The Motations are each endpoint is expecting/waiting. In a successful Impulse Sync this records should be empty.

All the REST Endpoints should be accessed using the transactionId as filter.

Link to Content Diffuser Rest Endpoints