Destination Connector

How to implement a Destination Connector?

To implement destination connectors, it is required to build two main sections, an Adapter write and a Transformer Out, to accomplish it, Impulse provides a set of interfaces the user needs to implement, implementing those interfaces is what allows the developer to build what we call a Destination connector.

Transformer Out

The Transformer Out primarily responsibility is to transform the different given Motation elements to Raw Data.

type TransformerOut interface {
	TransformContent(connectorCtx connectorelements.ConnectorCtx, contentMotation motationElements.Motation, definitionMotation definition.DefinitionMotation) (elements.RawDataWrapper, error)

	TransformCategory(connectorCtx connectorelements.ConnectorCtx, motation category.CategoryMotation) (elements.RawDataWrapper, error)

	TransformDefinition(connectorCtx connectorelements.ConnectorCtx, definitionMotation definition.DefinitionMotation, relationshipMotation relationship.RelationshipMotation) (elements.RawDataWrapper, error)

	TransformDomain(connectorCtx connectorelements.ConnectorCtx, motation domain.DomainMotation) (elements.RawDataWrapper, error)

	TransformFolder(connectorCtx connectorelements.ConnectorCtx, motation folder.FolderMotation) (elements.RawDataWrapper, error)

	TransformLanguage(connectorCtx connectorelements.ConnectorCtx, motation language.LanguageMotation) (elements.RawDataWrapper, error)

	TransformRelationship(connectorCtx connectorelements.ConnectorCtx, motation relationship.RelationshipMotation) (elements.RawDataWrapper, error)

	TransformTag(connectorCtx connectorelements.ConnectorCtx, motation tag.TagMotation) (elements.RawDataWrapper, error)

	GetConnectorDetails() connectorelements.ConnectorDetails
}

The Transformer Out interface is divided in 2 main categories, Motation transformation and Configuration.

• Motation methods:

  • TransformContent, TransformCategory, TransformDefinition, TransformDomain, TransformFolder, TransformLanguage, TransformRelationship, TransformTag: All the Motation methods the developer needs to implement in the Transformer Out interface are to transform given Motations of different types to also different types of Raw Data.

• Configuration methods:

  • GetConnectorDetails: The connector details are the details for the Connector, like unique ID, name, level, support versions, etc.

Transformation Rules

When transforming motation into raw data for the adapter, there are a few rules that must be followed.

Consistent Field Transformation

For deciding how fields should transform the transformer should use motation's field type and data type attributes. This will allow the transformer consistently transform fields from all motations into the same raw data.

When there are fields that are determined by their system field types and not motation field type and data type exclusively, if the motation did not come from the same system, there must be a default raw data field that the transformer uses. This default field should be documented as such in the motation mapping.

Adapter Write

The Adapter Write primarily responsibility is to communicate directly with the Content Repo and request and sync the data specified by the AdapterWrite interface.

type AdapterWrite interface {
	GetContentLocalIDs(
		connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, job syncmanstructs.Job,
		jobValues syncmanstructs.JobValues) ([]connectorelements.Content, error)

	RequestBinary(connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, rawData []byte,
		job syncmanstructs.Job, jobValues syncmanstructs.JobValues) ([]elements.BinaryData, error)

	SyncBinary(connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, binaryData elements.BinaryData,
		binaryFile io.ReadCloser, job syncmanstructs.Job, jobValues syncmanstructs.JobValues) error

	SyncCategory(connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, rawData []byte,
		job syncmanstructs.Job, jobValues syncmanstructs.JobValues) error

	SyncContent(connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, contentRawData,
		definitionRawData []byte, job syncmanstructs.Job, jobValues syncmanstructs.JobValues) (connectorelements.Content, error)

	SyncDefinition(connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, rawData []byte,
		job syncmanstructs.Job, jobValues syncmanstructs.JobValues) error

	SyncDomain(connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, rawData []byte,
		job syncmanstructs.Job, jobValues syncmanstructs.JobValues) error

	SyncFolder(connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, rawData []byte,
		job syncmanstructs.Job, jobValues syncmanstructs.JobValues) error

	SyncLanguage(connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, rawData []byte,
		job syncmanstructs.Job, jobValues syncmanstructs.JobValues) error

	SyncRelationship(connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, rawData []byte,
		job syncmanstructs.Job, jobValues syncmanstructs.JobValues) error

	SyncTag(connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, rawData []byte,
		job syncmanstructs.Job, jobValues syncmanstructs.JobValues) error

	SyncSystemObject(connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, rawData []byte,
		job syncmanstructs.Job, jobValues syncmanstructs.JobValues) error

	GetConfigOption() syncmanstructs.JobOptions

	GetMotationObjectSupport() map[syncmanstructs.MotationObjectTypeEnum]int

	GetMotationMapping() elements2.MotationTypeMapping

	GetConnectorDetails() connectorelements.ConnectorDetails
}

The Adapter Write interface is divided in 3 main categories, Local IDs, Sync and Configuration.

• Local IDs methods:

  • GetContentLocalIDs: This is where the sync process starts, requesting the source and destinations Local IDs, with the provided contents by this method, Impulse compares between source and destinations to decided what needs to be sync.

• Sync methods:

  • SyncBinary, SyncCategory, SyncContent, SyncDefinition, SyncDomain, SyncFolder, SyncLanguage, SyncRelationship, SyncTag, SyncSystemObject: The Sync methods will allow Impulse to delivery and sync the different types of elements Impulse decides has to be sync over each of the different Destination Content Repos, not all the types need to be implemented, this is a case by case scenario, for example, some systems have the concept of categories, some systems don't.

• Configuration methods:

  • GetConfigOption: Provides to Impulse the list of supported options the Connector supports, these options are filled when the Sync Job is created and is passed as parameter in all the interface methods.

  • GetConnectorDetails: The connector details are the details for the Connector, like unique ID, name, level, support versions, etc.

  • GetMotationObjectSupport: List of the objects the Connector supports, for example, Strapi supports Languages and Relationships but no Tags and Categories. This list is used by Impulse to decide what can be sent for syncing and what not.

  • GetMotationMapping: The Motation mapping helps Impulse and the developer to understand the mapping between the elements in Motation with the elements in the Raw Data.

• Other methods:

  • RequestBinary: Method that allows Impulse to request files to the Content Repo.

Diff Reporting

For diff reporting to work properly, the id map returned, must match the id sent to the content diffuser. Meaning, if a unique ID was generated by combining content type and content ID, that same ID must be returned as the ID by the adapter write.

For examples, view Strapi 3 or Strapi 4 connectors or Contentstack connector.

Writing a Remote Adapter

When writing a remote adapter, an adapter that does not live in the same network as Impulse, you will need to pass authentication in to Impulse. Many of the modules we use have a BasicAuth version of each function.

To use these functions, check the config for the remoteAdapter boolean. When this is true, use the impulseUsername and impulsePassword attributes from the config in the BasicAuth functions to call Impulse.

Motation Mapping

The destination motation mapping structure is returned in the GetMotationMapping() function. It's structure is akin to the following.

{
  relationships: [
    motation-cardinality-enum: {
      "systemCardinalityName": "system-cardinality-name",
      "systemCardinality": "system-data-cardinality",
      "testingCardinality": "testing-data-cardinality"
    }
  ],
  "fields": [
    content-motation-type-enum: [
      motation-field-type-enum: {
        "systemFieldName": "system-field-name",
        "systemFieldType": "raw-data-field-type",
        "testingFieldType": "system-data-field-type"
        }
    ]
  ],
  "motationTypes": [
    motation-type-enum: [
      "systemContentType": "system-content-type",
      "systemType": "raw-data-type",
      "testingType": "system-data-type"
    ]
  ],
  "supportedTypes": [
     content-motation-type-enum
  ],
  "SystemOnly": [
    "system-object1",
    "system-object2"
  ],
  "SupportedSystems": [
    {
      "name": "system-name",
      "versionRanges": [
        {
          "startVersion": range-start-version,
          "endVersion": range-end-version
        }
      ],
      "fieldMap": {
        "source-system-field-type.source-system-data-type": "destination-system-field-name"
      }
    }
  ]
}

Relationships

The relationships object is an array of system relationships to their motation cardinality.

motation-cardinality-enum is the Motation cardinality enum value that converts into the repository's cardinality as documented in the object.

motation-cardinality-enum.systemCardinalityName is the common name for selecting the cardinality in the system for a field. This is any string that closely matches the common name from the system.

motation-cardinality-enum.systemCardinality is the raw data cardinality returned in the raw data for testing the connector. This should match the values returned when raw data is retrieved during testing.

motation-cardinality-enum.testingCardinality is the system cardinality returned in the system data for testing the connector. This should match the values returned when system data is retrieved during testing.

Fields

The fields object is an array of system fields and their matching field and data types.

content-motation-type-enum is a supported content Motation type value. This is an array of all the Motation field types that are supported for transforming from Motation in the connector. Each supported type should fully document every field that the connector transformers from Motation.

content-motation-type-enum.motation-field-type-enum is a supported Motation field type value for that content Motation type. Each Motation field type for a content Motation type can transform into only one repository field.

Note: The same content-motation-type-enum.motation-field-type-enum cannot convert to multiple repository fields. Meaning it is a "one to one" relationship.

content-motation-type-enum.motation-field-type-enum.systemFieldName is the common name for selecting a field while using the system. This is any string that closely matches the common name from the system. This should be broken down to sub categories with a '-' as applicable.

content-motation-type-enum.motation-field-type-enum.systemFieldType is the value of the endpointSpecificFieldType attribute in Motation. This is required.

content-motation-type-enum.motation-field-type-enum.testingFieldType is the system field type returned in the system data for testing the connector. This should match the values returned when system data is retrieved during testing. This is required.

Motation Types

The motationTypes object is an array of system content types and their matching motation type enum.

motation-type-enum is the content Motation type enum and how it will convert to a specific content type in the repository.

motation-type-enum.systemContentType is the common name for the system content type. This is any string that closely matches the common name from the system.

motation-type-enum.systemType is the system content type returned in the raw data for testing the connector. This should match the values returned when raw data is retrieved during testing.

motation-type-enum.testingType is the system content type returned in the system data for testing the connector. This should match the values returned when system data is retrieved during testing.

Supported Types

The supported types object is an array of content Motation type enums that are supported for the connector. This array is used to quickly determine if a content Motation type is supported instead of iterating through the fields array. Meaning, if a content Motation type is in the fields array, it must also be listed as a supported type.

System Only

The system only object is an array of string values that list the system only objects that can be synced.

Each system object is a string value that matches the common name for the object in the system.

Supported Systems

The SupportedSystems object is an array of all the supported systems that have custom rules for syncing content. If a destination connector has custom rules for transforming fields beyond using only motation field type and motation data type, then those rules should be documented here.

SupportedSystems.name is the name of the supported system. This must match the value set in the motation object. This is required.

SupportedSystems.versionRanges is an array of all the version ranges that are supported for that connector.

SupportedSystems.versionRanges.startVersion is the first version that is supported. "latest" can be used when no specific version range is supported. i.e., GitHub or SCP connector

SupportedSystems.versionRanges.endVersion is the last version that is supported. "latest" can be used when no specific version range is supported. i.e., GitHub or SCP connector

SupportedSystems.fieldMap is a object with multiple key value pairs that match the source system field type and source system data type to the destination field name. This is done in the following format:

source-system-field-type : destination-system-field-type

While testing, the Motation can be checked for what system it came from. Then this field map can be used to match the field types to the custom destination field types as defined.

However, if a system is supported but the fieldMap is empty or null, it is assumed that the source field names directly match to destination field names.