Upload data using Singer ETL

ETL technologies such as Singer is actively supported

Importing from a Singer Tap

The first message should be a SCHEMA message followed by a RECORD message
Schema message would be validated before it is passed onto the system The schema should be able to validate the following keys dataformat and data inside record

A SCHEMA message must be provided before every change in the RECORD message structure.

example:

{"type":"SCHEMA","stream":"users","key_properties":[],"schema":{"required":["dataformat","data"],"type":"object","properties":{"dataformat":{"type":"string"},"data":{"type":"string"},"markComplete":{"type":"boolean"},"markEval":{"type":"boolean"},"uploadName":{"type":"string"},"uploadSource":{"type":"string"},"recordList":{"type":"array","contains":{"type":"string"}}}}}
{"type":"RECORD","stream":"users","record":{"data":"DATAFROMSINGER","dataformat":"TXT","markComplete":true,"uploadName":"singerUpload"}}

Usage

To import data using Singer just add a -s or --singer flag in the import statement

example

<Output from singer tap> |./acharya task import -p Prj-1 -s

caution

The flag --dataformat will have no effect while importing data with singer
dataformat must be specified in the RECORD message