Watchdog flow

1. Each kafka message produced by the Sync Manager is associated with a Sub transaction id, an unique id for the produced message that allows us to track its state.

2. When a message is produced by the Sync Manager it is also created a Heartbeat record associated to a Sub transaction id.

3. When a message produced by the Sync Manager is consumed in any microservice a background operation to track the running process is created, this background operation is going to:

   1. Notify the message was Received.

   2. Notify the running process is In Progress (in case of taking too long).

   3. Notify when the running process is Done.

   4. Notify if the process fails for any reason. We are calling each of these notifications Heartbeat Pulses and each pulse or notification is sent back to the Sync Manager using kafka messages.

4. The Sync Manager listens for any incoming Heartbeat Pulse and stores its contents for tracking purposes and a later use, for example, this information allows us to know at any moment the current running operations and its statuses.

5. The Sync Manager will be constantly running a Cron Job (by default every two minutes) in order to check for idle and failed processes. If the Cron Job identifies there is an idle or failed process is going to retry the process sending again the original message.