Source Connector

How to implement a Source Connector?

To implement source connectors, it is required to build two main sections, an Adapter read and a Transformer In, to accomplish it, Impulse provides a set of interfaces the user needs to implement, implementing those interfaces is what allows the developer to build what we call a Source connector.

Adapter Read

The Adapter read primarily responsibility is to communicate directly with the Content Repo and request the data specified by the AdapterRead interface.

type AdapterRead interface {
	GetContentLocalIDs(
		connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, job syncmanstructs.Job,
		jobValues syncmanstructs.JobValues) ([]connectorelements.Content, error)

	GetContentDependenciesLocalIDs(
		connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, job syncmanstructs.Job, jobValues syncmanstructs.JobValues,
		forContent connectorelements.Content) ([]elements.Dependency, error)

	GetSystemObjectsLocalIDs(
		connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, job syncmanstructs.Job, jobValues syncmanstructs.JobValues,
		forContent connectorelements.Content) ([]elements.SystemObject, error)

	GetFile(connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, filePath string) (io.ReadCloser, error)

	GenerateContentRawData(
		connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, job syncmanstructs.Job, jobValues syncmanstructs.JobValues,
		forContent connectorelements.Content) (interface{}, error)

	GenerateSystemObjectRawData(
		connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, job syncmanstructs.Job, jobValues syncmanstructs.JobValues,
		forContent connectorelements.Content) (interface{}, error)

	GenerateDomainRawData(
		connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, job syncmanstructs.Job, jobValues syncmanstructs.JobValues,
		forContent connectorelements.Content) (interface{}, error)

	GenerateDefinitionRawData(
		connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, job syncmanstructs.Job, jobValues syncmanstructs.JobValues,
		forContent connectorelements.Content) (interface{}, error)

	GenerateCategoryRawData(
		connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, job syncmanstructs.Job, jobValues syncmanstructs.JobValues,
		forContent connectorelements.Content) (interface{}, error)

	GenerateFolderRawData(
		connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, job syncmanstructs.Job, jobValues syncmanstructs.JobValues,
		forContent connectorelements.Content) (interface{}, error)

	GenerateLanguageRawData(
		connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, job syncmanstructs.Job, jobValues syncmanstructs.JobValues,
		forContent connectorelements.Content) (interface{}, error)

	GenerateRelationshipRawData(
		connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, job syncmanstructs.Job, jobValues syncmanstructs.JobValues,
		forContent connectorelements.Content) (interface{}, error)

	GenerateTagRawData(
		connectorCtx connectorelements.ConnectorCtx, config connectorelements.Config, job syncmanstructs.Job, jobValues syncmanstructs.JobValues,
		forContent connectorelements.Content) (interface{}, error)

	GetConfigOption() syncmanstructs.JobOptions

	GetConnectorDetails() connectorelements.ConnectorDetails

	GetMotationObjectSupport() map[syncmanstructs.MotationObjectTypeEnum]int

	GetMotationMapping() elements2.MotationTypeMapping
}

The Adapter Read interface is divided in 3 main categories, Local IDs, Raw Data and Configuration.

• Local IDs methods:

  • GetContentLocalIDs: This is where the sync process starts, requesting the source and destinations Local IDs, with the provided contents by this method, Impulse compares between source and destinations to decided what needs to be sync.

  • GetContentDependenciesLocalIDs, GetSystemObjectsLocalIDs: It is used to request the list of dependencies for a given content.

• Raw Data methods:

  • GenerateContentRawData, GenerateSystemObjectRawData, GenerateDomainRawData, GenerateDefinitionRawData, GenerateCategoryRawData, GenerateFolderRawData, GenerateLanguageRawData, GenerateRelationshipRawData, GenerateTagRawData: The Raw Data methods will allow Impulse to request to the Content Repo the different types of data the System could have, not all the types need to be implemented, this is a case by case scenario, for example, some systems have the concept of categories, some systems don't. The Raw Data methods are always executed for a given content and usually the provided Raw Data will be passed down to the Transformer In to transform it into Motation.

• Configuration methods:

  • GetConfigOption: Provides to Impulse the list of supported options the Connector supports, these options are filled when the Sync Job is created and is passed as parameter in all the interface methods.

  • GetConnectorDetails: The connector details are the details for the Connector, like unique ID, name, level, support versions, etc.

  • GetMotationObjectSupport: List of the objects the Connector supports, for example, Strapi supports Languages and Relationships but no Tags and Categories. This list is used by Impulse to decide what needs to request or not to the Adapter Read binary.

  • GetMotationMapping: The Motation mapping helps Impulse and the developer to understand the mapping between the elements in the Raw Data with the elements in Motation.

• Other methods:

  • GetFile: Method that allows Impulse to request files to the Content Repo.

Diff Reporting

For diff reporting to work properly, the content id needs to match the ID passed to the content diffuser as part of the local immutable IDs response.

For examples, view Strapi 3 or Strapi 4 connectors or Contentstack connector.

Writing a Remote Adapter

When writing a remote adapter, an adapter that does not live in the same network as Impulse, you will need to pass authentication in to Impulse. Many of the modules we use have a BasicAuth version of each function.

To use these functions, check the config for the remoteAdapter boolean. When this is true, use the impulseUsername and impulsePassword attributes from the config in the BasicAuth functions to call Impulse.

Transformer In

The Transformer In primarily responsibility is to transform the different given Raw Data elements to Motation.

type TransformerIn interface {
	TransformContentMotation(connectorCtx connectorelements.ConnectorCtx, endpointId string, contentRawData, definitionRawData []byte) (elements.MotationWrapper, error)
	TransformCategoryMotation(connectorCtx connectorelements.ConnectorCtx, endpointId string, rawData []byte) (elements.CategoryMotationWrapper, error)
	TransformDefinitionMotation(connectorCtx connectorelements.ConnectorCtx, endpointId string, rawData []byte) (elements.DefinitionMotationWrapper, error)
	TransformDomainMotation(connectorCtx connectorelements.ConnectorCtx, endpointId string, rawData []byte) (elements.DomainMotationWrapper, error)
	TransformFolderMotation(connectorCtx connectorelements.ConnectorCtx, endpointId string, rawData []byte) (elements.FolderMotationWrapper, error)
	TransformLanguageMotation(connectorCtx connectorelements.ConnectorCtx, endpointId string, rawData []byte) (elements.LanguageMotationWrapper, error)
	TransformRelationshipMotation(connectorCtx connectorelements.ConnectorCtx, endpointId string, rawData []byte) (elements.RelationshipMotationWrapper, error)
	TransformTagMotation(connectorCtx connectorelements.ConnectorCtx, endpointId string, rawData []byte) (elements.TagMotationWrapper, error)
	GetConnectorDetails() connectorelements.ConnectorDetails
}

The Transformer In interface is divided in 2 main categories, Raw Data transformation and Configuration.

• Raw Data methods:

  • TransformContentMotation, TransformCategoryMotation, TransformDefinitionMotation, TransformDomainMotation, TransformFolderMotation, TransformLanguageMotation, TransformRelationshipMotation, TransformTagMotation: All the Raw Data methods the developer needs to implement in the Transformer In interface are to transform given Raw Data content of different types to also different types of Motations.

• Configuration methods:

  • GetConnectorDetails: The connector details are the details for the Connector, like unique ID, name, level, support versions, etc.

Transformation Rules

When transforming motation into motation, there are a few rules that must be followed.

Consistent Field Transformation

For deciding how fields should transform the transformer should use the systems field type and data type to determine the motation field type and data type. This will allow the transformer to consistently transform fields into motations.

It is expected that some systems will have system specific fields and generate non-unique motation fields by motation field type and data type combination. These fields should be documented appropriately in the motation mapping object.

Motation Mapping

The source Motation mapping structure is returned in the GetMotationMapping() function. It's structure is akin to the following.

{
  relationships: [
    "system-cardinality": {
      "motationCardinality": motation-cardinality-enum,
      "systemCardinality": "raw-data-cardinality",
      "testingCardinality": "system-data-cardinality"
    }
  ],
  "fields": [
    content-motation-type-enum: [
      "system-field-name": {
        "motationFieldType": motation-field-type-enum,
        "systemFieldType": "raw-data-field-type",
        "testingFieldType": "system-data-field-type",  
        "sameSystem": boolean
      }
    ]
  ],
  "motationTypes": [
    "system-content-type": {
      "motationType": motation-type-enum,
      "systemType": "raw-data-type"
      "testingType": "system-data-type"
    }
  ],
  "supportedTypes": [
     content-motation-type-enum
  ],
  "systemOnly": [
    "system-object1",
    "system-object2"
  ]
}

Relationships

The relationships object is an array of system relationships to their motation cardinality.

system-cardinality is the common name for selecting the cardinality in the system for a field. This is any string that closely matches the common name from the system.

system-cardinality.motationCardinality is the motation cardinality enum that the system's relationship transforms into.

system-cardinality.systemCardinality is the system type returned in the raw data for transforming into Motation in the connector. This should match the values returned when raw data is retrieved during testing.

system-cardinality.testingCardinality is the system type returned in the system data for testing the connector. This should match the values returned when system data is retrieved during testing.

Fields

The fields object is an array of system fields and their matching field and data types.

content-motation-type-enum is a supported content Motation type value. This is an array of all the field types from the repository that are supported for transforming into Motation in the connector. Each supported type should fully document every field that the connector transformers into Motation, even fields that transform into the "System" field type.

content-motation-type-enum.system-field-name is the common name for selecting a field while using the system. This is any string that closely matches the common name from the system. This should be broken down to sub categories with a '-' as applicable. Every supported system field, either same system or cross system, should be listed in the array.

Note: Multiple repository fields can map to the same Motation field type. And each repository field must transform into only one Motation field type. Meaning it is a "many to one" relationship.

content-motation-type-enum.system-field-name.motationFieldType is the Motation field type enum the field transforms into. This is a required field to be populated.

content-motation-type-enum.system-field-name.systemFieldType the value of the endpointSpecificFieldType attribute in Motation. This is required.

content-motation-type-enum.system-field-name.testingFieldType is the system field type returned in the system data for testing the connector. This should match the values returned when system data is retrieved during testing. This is required.

content-motation-type-enum.system-field-name.sameSystem is a boolean that is set to true the field requires system data to properly be transformed out. Meaning this field is unlikely to fully match the type of another system and only fully work when syncing between the same systems. Or the field may not work across systems at all. These are different than the "systemOnly" values, as these convert into Motation require the additional system information to properly transform the value out of Motation.

Motation Types

The motationTypes object is an array of system content types and their matching Motation type enum.

system-content-type is the common name for the system content type. This is any string that closely matches the common name from the system.

system-content-type.motationType is the Motation type enum that the system content type transforms into.

system-content-type.systemType is the system content type returned in the raw data for transforming into Motation. This should match the values returned when raw data is retrieved during testing.

system-content-type.testingType is the system content type returned in the system data for testing the connector. This should match the values returned when system data is retrieved during testing.

Supported Types

The supported types object is an array of content Motation type enums that are supported for the connector. This array is used to quickly determine if a content Motation type is supported instead of iterating through the fields array. Meaning, if a content Motation type is in the fields array, it must also be listed as a supported type.

System Only

The system only object is an array of string values that list the system only objects that can be synced.

Each system object is a string value that matches the common name for the object in the system.