Create a Data Pipeline from GitHub to Discord
Create an end-to-end event pipeline that detects changes in github stars & forks and publishes the result to Discord. This guide uses two connectors:
- http-source: to read periodically from a github, parse the fields from the
json
output, and publish the result to a topic. - http-sink: to listen to the same topic, detect changes, and publish the result to Discord.
Let's get started.
Prerequisites
- Fluvio CLI running locally
- Account on InfinyOn Cloud
Step-by-Step
- Create http-source configuration file
- Create http-sink configuration file
- Download smartmodules
- Start Connectors
- Test Data Pipeline
Create http-source configuration file
Create an HTTP source connector configuration file called github.yaml
:
All versions are marked with x.y.z
. To find the latest version, run:
fluvio hub connector list
fluvio hub smartmodule list
apiVersion: 0.1.0
meta:
version: x.y.z
name: github-stars-in
type: http-source
topic: stars-forks
secrets:
- name: GITHUB_TOKEN
http:
endpoint: 'https://api.github.com/repos/infinyon/fluvio'
method: GET
headers:
- 'Authorization: token ${{ secrets.GITHUB_TOKEN }}'
interval: 30s
transforms:
- uses: infinyon/jolt@x.y.z
with:
spec:
- operation: shift
spec:
"stargazers_count": "stars"
"forks_count": "forks"
Github rate-limits API requests to 60 per hour, which you an extend to 5000 by creating an application token. Check out github documentation on how to create an Access Tokens.
Add the access token secret
in InfinyOn Cloud :
$ fluvio cloud secret set GITHUB_TOKEN <access-token>
Create http-sink configuration file
Create an HTTP source connector configuration file called discord.yaml
:
apiVersion: 0.1.0
meta:
version: x.y.z
name: discord-stars-out
type: http-sink
topic: stars-forks
secrets:
- name: DISCORD_TOKEN
http:
endpoint: "https://discord.com/api/webhooks/${{ secrets.DISCORD_TOKEN }}"
headers:
- "Content-Type: application/json"
transforms:
- uses: infinyon-labs/stars-forks-changes@x.y.z
lookback:
last: 1
- uses: infinyon/jolt@x.y.z
with:
spec:
- operation: shift
spec:
"result": "content"
Check out Discord Webhooks on how to create a channel webhook token.
Add the access token secret
in InfinyOn Cloud :
$ fluvio cloud secret set DISCORD_TOKEN <webhook-token>
Download Smartmodules
Download the smartmodules used by the connectors to your cluster:
$ fluvio hub sm download infinyon/jolt@x.y.z
$ fluvio hub sm download infinyon-labs/stars-forks-changes@x.y.z
Start Connectors
Start source & sink connectors:
$ fluvio cloud connector create -c github.yaml
$ fluvio cloud connector create -c discord.yaml
Check fluvio cloud connector log
to ensure they have been successfully provisioned.
Test Data Pipeline
Check the last values generated by the github connector:
$ fluvio consume -dT 1 stars-forks
{"stars":1770,"forks":138}
Produce a new value
$ fluvio produce stars-forks
> {"stars":1769,"forks":138}
OK
An alert with :star2: 1769
will show-up in your discord channel. See it live at Fluvio Community - #alerts channel.