Create a Hackernews Reader
Hackernews Reader helps you build an XML reader that ingests hackernews articles, converts them to json
, divides them into records, and publishes each record to a topic. This guide uses the following connector:
- http-source: to read periodically from a hackernews, parse the XML result into
json
records, and publish the result to a topic.
Prerequisites
- Fluvio CLI running locally
- Account on InfinyOn Cloud
Step-by-Step
Create http-source configuration file
Create an HTTP source connector configuration file called hackernews.yaml
:
info
All versions are marked with x.y.z
. To find the latest version, run:
fluvio hub connector list
fluvio hub smartmodule list
apiVersion: 0.1.0
meta:
version: x.y.z
name: hackernews
type: http-source
topic: hackernews
http:
method: GET
endpoint: 'https://hnrss.org/newest'
interval: 600s
transforms:
- uses: infinyon-labs/rss-json@x.y.z
- uses: infinyon/jolt@x.y.z
with:
spec:
- operation: shift
spec:
items: ""
- uses: infinyon-labs/array-map-json@x.y.z
Download smartmodules
Download the smartmodules used by the connectors to your cluster:
$ fluvio hub sm download infinyon/jolt@x.y.z
$ fluvio hub sm download infinyon-labs/rss-json@x.y.z
$ fluvio hub sm download infinyon-labs/array-map-json@x.y.z
Start Connector
$ fluvio cloud connector create -c hackernews.yaml
Check Results
Connector logs:
$ fluvio cloud connector log hackernews
Records produced:
$ fluvio consume hackernews -T 10