Writing a custom plugin
Plugins have a simple contract with the host
ds-load command. They can be written in any language.
A set of go packages can be used as a "mini-SDK" for building custom plugins in go. This eliminates much of the "boilerplate" of writing a plugin, including commands like
export-transform, providing help, and so on.
The plugins that come with
ds-load can all be found here. They can be used as a guide for how to write a custom plugin for a new data source.
In addition, a standalone plugin for importing Hubspot data into the directory can be found here. This shows how to properly import the
ds-load SDK in an external plugin.
Most of the "heavy lifting" of building a plugin involves retrieving data from the source. The Hubspot plugin uses the Hubspot REST API, and can easily be adapted to any other service that supports a REST interaction pattern.
Transforming into objects and relations
The other main task is to author the default transform for the plugin. The transformation docs provide an annotated walkthrough of the Auth0 default transform, which can act as a guide.
The Hubspot sample plugin also has a default transform that shows how to create objects of different types, and relationships between them.
ds-load pick up your custom plugin, simply call it
ds-load-<plugin-name> and copy the binary to
ls -l ~/.ds-load/plugins
-rwxr-xr-x 1 ogazitt staff 16M Aug 3 10:50 ds-load-hubspot*```
To verify that
ds-load can load your plugin, use
ds-load exec --help. Note that
hubspot is in the list of available commands:
Usage: ds-load exec <command> ...
import data in directory by running fetch, transform and publish
<command> ... available commands are: hubspot|auth0|azuread|cognito|google|okta
Please drop us a line in our community slack if you need help writing a custom plugin! Also, we're happy to evaluate and accept new providers written by the community via a Pull Request in the ds-load repository.
Submitted plugins should follow the same design patterns as existing providers, and live in the plugins directory.