Map Operator

The Map Operator is a versatile tool that allows you to transform an existing message and re-encode the input into a new message format. The operator takes the input and maps it into any type. Every message in the original topic will be mapped to a new topic. For this example, we implement a simple dataflow that outputs the size of the string input.

Prerequisites

This guide uses local Fluvio cluster. If you need to install it, please follow the instructions at here.

Transformation

Below is an example of a transform function. The example demonstrates how to map a sentence into a new string that outputs the size of the original string.

    transforms:
      - operator: map 
        run: |
          fn get_sentence_length(input: String) -> Result<String> {
            Ok(format!("Inserted string size {}",input.len()))
          }

Running the Example

Copy and paste following config and save it as dataflow.yaml.

# dataflow.yaml
apiVersion: 0.5.0
meta:
  name: map-example
  version: 0.1.0
  namespace: examples

config:
  converter: raw

topics:
  sentences:
    schema:
      value:
        type: string
  len:
    schema:
      value:
        type: string

services:
  map-service:
    sources:
      - type: topic
        id: sentences
    transforms:
      - operator: map 
        run: |
          fn get_sentence_length(input: String) -> Result<String> {
            Ok(format!("Inserted string size {}",input.len()))
          }
    sinks:
      - type: topic
        id: len

To run example:

$ sdf run

Produce sentences to in sentence topic:

$ echo "This string is size 22" | fluvio produce sentences
$ echo "This string is size 23." | fluvio produce sentences

Consume topic len to retrieve the result in another terminal:

$ fluvio consume len -Bd
Inserted string size 22
Inserted string size 23

Each string will be mapped via the function provided.

Cleanup

Exit sdf terminal and clean-up. The --force flag removes the topics:

$ sdf clean --force

Conclusion

We just covered another basic operator in SDF, the Map Operator.