Astro

Running Impulse

Astro

The easiest way to run impulse is through a bash script that allows for bringing up the necessary parts of Impulse. You can choose to bring up any available connector and repo.

The script makes use of the docker-tools/compose-template/config/astro.conf file for configuring the various docker images of Impulse. Use the ./docker-tools/compose-template/config/astro.conf file to control debug options and images for each service and repo.

If you want to run a service in debug mode, simply build the image of that service, giving it the appropriate name with the tag debug. i.e. sync-manager:debug The script assumes that all motiv labs github repos live in the same directory and will path a volume relatively to the appropriate service. i.e. /motiv-labs/impulse and /motiv-labs/sync-manager

The script assumes the user is running it from the root of Impulse. It makes use of relative pathing from the root of the Impulse project.

Commands

start

./astro start [flags] [name:repo]

example:

./astro start my-dcms:dotcmsSmokeSource

./astro start my-dcms:dotcmsSmokeSource my-strapi:strapiChoose

This command will start the list of repos with the given name prepended to their docker containers. It will also start impulse and any required connectors if they are not already running.

Names for each repo should be unique and attempt to avoid including the full name given to another running repo.

i.e. dcms-source and dcms-source-1 should be avoided. While dcms-source-1 and dcms-source-2 is good.

Pass in as many repos with unique names as you wish to start at a single time.

Flags

--jaeger

./astro start --jaeger

The --jaeger flag will start astro with a jaeger container for tracing. The default is for no jaeger to start.

--fleuntd

./astro start --fluentd

The --fluentd flag will start astro and set all containers to use fluentd logging driver. The default is for no fluentd to start.

--no-db

./astro start --no-db

The --no-db flag will start astro without a local DB container. The default is to start the local DB with astro.

reset

./astro reset [name given repo]

example:

./astro reset my-dcms

This command will reset the passed in list of repos to their default state.

This allows for an easier way to retest a sync job. You can reset a destination repo back to its default state and sync again.

setJanus

./astro setJanus

This command will create a Janus user "user:pass" and add API proxies to janus. (This command assumes janus admin api is reachable at localhost:8081)

clean

./astro clean [clean options]

example:

./astro clean impulse

This command will clean (truncate) tables in cassandra based on the input arguments. The input arguments select what group of tables to be cleaned allowing for a fresh run of Impulse.

This allows for easier retest of a sync job where motation does not exist. You can clean the motation tables and sync again.

down

./astro down

This command will run docker-compose down all docker containers and remove generated volumes. note: will ask for sudo access to remove volumes

build

./astro build [dir to build] [build type]

This command will build the service/connector/motator as the type selected.

Build types are:

• prod

  • Build a local image like production.

• dev

  • Build a local image that will compile local code.

• debug

  • Build a local image that will compile local code and allow remote debugger connectivity.

build help

./astro build help

List the help for the build command.

list repo

./astro list repo

This command will list all the repo options that can be passed in.

list ports

./astro list ports

This command will list all the containers ports currently running.

list clean

./astro list clean

This command will list all clean options that can be passed in.

list build

./astro list build

This command will list all services that can be built as well as all the types that each service can be built as.

logs

./astro logs

This command will follow the logs of all currently running docker containers generated by this script.

help

./astro help

This command will give quick help information on the program. help can be passed in after any command for help information on that specific command.

Finding a Repository's URL

Local Machine URL

Astro starts up and runs at the localhost domain. Meaning you will need to find the repository's port. You can use Astro's list ports command to find the port of each container running. You can pipe the output to grep and find the specific container you are looking for.

Example:

./astro.sh list ports | grep dotcms
dest-dotcms                       4000/tcp, 8000/tcp, 8081-8082/tcp, 0.0.0.0:9005->8080/tcp, :::9005->8080/tcp
dest-dotcms-db                    0.0.0.0:9004->5432/tcp, :::9004->5432/tcp
dest-dotcms-es                    9200/tcp, 9300/tcp
dest-adapter-read-dotcms-app      0.0.0.0:9006->8080/tcp, :::9006->8080/tcp, 0.0.0.0:40033->40000/tcp, :::40033->40000/tcp
dest-adapter-write-dotcms-app     0.0.0.0:9007->8080/tcp, :::9007->8080/tcp, 0.0.0.0:40034->40000/tcp, :::40034->40000/tcp
source-dotcms                     4000/tcp, 8000/tcp, 8081-8082/tcp, 0.0.0.0:9001->8080/tcp, :::9001->8080/tcp
source-dotcms-db                  0.0.0.0:9000->5432/tcp, :::9000->5432/tcp
source-adapter-read-dotcms-app    0.0.0.0:9002->8080/tcp, :::9002->8080/tcp, 0.0.0.0:40029->40000/tcp, :::40029->40000/tcp
source-adapter-write-dotcms-app   0.0.0.0:9003->8080/tcp, :::9003->8080/tcp, 0.0.0.0:40030->40000/tcp, :::40030->40000/tcp
source-dotcms-es                  9200/tcp, 9300/tcp
transformer-in-dotcms-app         8080/tcp, 0.0.0.0:40025->40000/tcp, :::40025->40000/tcp
transformer-out-dotcms-app        8080/tcp, 0.0.0.0:40026->40000/tcp, :::40026->40000/tcp

So, if you want to open the source dotCMS repository in your local browser, you can reach it at localhost:9001 .

Impulse Relative URL

Astro uses docker containers to run all of Impulse and the various repositories. This means that the relative URL for a repository is different for Impulse endpoints config when compared to your local machine. To find the proper relative URL, you need to know a few things.

1. The correct protocol to use. For instance, to reach dotCMS backend you can use http however to reach the postgres database you must use postgres

2. The container port. Note: that this is not the same port used for your local machine. Meaning, the Strapi container's backend port is 1337 while the dotCMS container's backend port 8080

With the knowledge of which protocol and port are meaningful to the repository, you can again use Astro's list ports command to find the name of each container running. You can pipe the output to grep and find the specific container you are looking for.

Example:

./astro.sh list ports | grep dotcms
dest-dotcms                       4000/tcp, 8000/tcp, 8081-8082/tcp, 0.0.0.0:9005->8080/tcp, :::9005->8080/tcp
dest-dotcms-db                    0.0.0.0:9004->5432/tcp, :::9004->5432/tcp
dest-dotcms-es                    9200/tcp, 9300/tcp
dest-adapter-read-dotcms-app      0.0.0.0:9006->8080/tcp, :::9006->8080/tcp, 0.0.0.0:40033->40000/tcp, :::40033->40000/tcp
dest-adapter-write-dotcms-app     0.0.0.0:9007->8080/tcp, :::9007->8080/tcp, 0.0.0.0:40034->40000/tcp, :::40034->40000/tcp
source-dotcms                     4000/tcp, 8000/tcp, 8081-8082/tcp, 0.0.0.0:9001->8080/tcp, :::9001->8080/tcp
source-dotcms-db                  0.0.0.0:9000->5432/tcp, :::9000->5432/tcp
source-adapter-read-dotcms-app    0.0.0.0:9002->8080/tcp, :::9002->8080/tcp, 0.0.0.0:40029->40000/tcp, :::40029->40000/tcp
source-adapter-write-dotcms-app   0.0.0.0:9003->8080/tcp, :::9003->8080/tcp, 0.0.0.0:40030->40000/tcp, :::40030->40000/tcp
source-dotcms-es                  9200/tcp, 9300/tcp
transformer-in-dotcms-app         8080/tcp, 0.0.0.0:40025->40000/tcp, :::40025->40000/tcp
transformer-out-dotcms-app        8080/tcp, 0.0.0.0:40026->40000/tcp, :::40026->40000/tcp

So, the relative URL for the source dotCMS would be a combination of the correct protocol, container name, and container port. For instance the destination dotCMS database connection path would be postgres://dest-dotcms-db:5432/dotcms .

Debugging

To debug a service you must locally build that service and give it debug in the tag. i.e. sync-manager:debug

You then have the option to put the service into dev or debug mode by setting, in the conf file, the ${SERVICE_DEBUG} variable to either 0 or 1 respectively.

dev (0) mode means that the service has a mapped volume and will rebuild code if you restart the container.

debug (1) mode means that in addition to dev mode, you can connect a remote debugger to the open debug port and step through the code with your ide.

Troubleshooting Astro

Astro uses GNU sed. If running on OSX, you may have to install GNU sed and set it as default. https://gist.github.com/icasimpan/b93297c17c3e81e33021e8f188b079df

If ports are not being released and a new repo is unable to start run 'service docker restart