Skip to main content
Version: 0.11.11

Kafka Connector

This is a connector for taking data from a Fluvio topic and sending to a Kafka topic.

See docs here.

Configuration

Optdefaulttypedescription
url-StringThe url for the kafka connector
topic-StringThe kafka topic
partition0IntegerThe kafka partition
create-topicfalseBooleanCreate or not a topic before start
options-MappingThe kafka client options
security-MappingOptional. The kafka security config

Security configuration

Optiondefaulttypedescription
security_protocolsslStringThe kafka security protocol
ssl_key-MappingThe SSL key file to use
ssl_cert-MappingThe SSL cert file to use
ssl_ca-MappingThe SSL ca file to use

Parameters ssl_key, ssl_cert and ssl_ca can be defined via file - path to the file, or pem - content as string value.

Example without security:

apiVersion: 0.1.0
meta:
  version: x.y.z
  name: my-kafka-connector
  type: kafka-sink
  topic: kafka-topic
  
kafka:
  url: "localhost:9092"
  topic: fluvio-topic
  

Example with security enabled:

apiVersion: 0.1.0
meta:
  version: x.y.z
  name: my-kafka-connector
  type: kafka-sink
  topic: kafka-topic
  
  secrets:
    - name: KAFKA_BROKER_URL
    - name: SSL_CERT_PEM
kafka:
  url: ${{ secrets.KAFKA_BROKER_URL }}
  topic: fluvio-topic
  
  security:
    ssl_key:
      file: /path/to/file
    ssl_cert:
      pem: "${{ secrets.SSL_CERT_PEM }}"
    ssl_ca:
      file: /path/to/file
    security_protocol: ssl
info

All connector versions are marked with x.y.z. To find the latest version, run:

  • fluvio hub connector list

Usage

To try out Kafka Sink connector locally, you can use Fluvio CDK tool:

cdk deploy -p kafka-sink start --config crates/kafka-sink/sample-config.yaml

Testing with security

Instructions of how to deploy local kafka cluster with SSL using docker. After all steps done, in the secrets folder there will be fluvio.key.pem, fluvio.pem and fake-ca-1.crt files that can be used in the connector config as ssl_key, ssl_cert and ssl_ca correspondingly.

Transformations

Fluvio Kafka Connectors support Transformations.