I did the below steps for the OpenAQ integration. Their data was stored in an S3 bucket.
- New data is placed in OpenAQ’s S3 bucket as it is added to their system: https://openaq-fetches.s3.amazonaws.com/index.html
- The arrival of new data into this bucket produces a notification using Amazon’s SNS: (https://aws.amazon.com/sns/).
- The above SNS notifications can be subscribed to. In this case, I made a queue using Amazon’s SQS: https://aws.amazon.com/sqs/. I set the queue to subscribe to the notifications. The queue saves the notifications and lets them be picked up one at a time by an external service which is processing the information.
- I used an EC2 box with a simple python script to pick up notifications from the queue, fetch the data (detailed in the notification) from the S3 bucket, process it and forward the data to Streamr.
Depending on the amount of data being forwarded, it may be possible to use one of the free EC2 instances. In the case of the OpenAQ data, there is a lot of data being generated so I needed to upgrade the EC2 box slightly.
This same mechanism should work with other systems (although I imagine other mechanisms would also work). Happy to provide more details about any step – just ask!