Skip to main content
Version: latest

Map

SmartModule Maps are used to transform or edit each Record in a stream. We say that these SmartModules "map" each input record into a new output record by applying a function to the input data. This type of SmartModule may be used for many use-cases, such as:

  • Narrowing large records into a smaller subset of important fields
  • Scrubbing sensitive fields of data to be invisible to downstream consumers
  • Computing rich, derived fields from simple raw data

Let's create a brand-new SmartModule Map to see what a minimal working example looks like.

SmartModule Map

Create a SmartModule Project​

Run smdk generate with the name of the map and choose the "map" options:

$ smdk generate map-example
Generating new SmartModule project: map-example
project-group => 'john'
fluvio-smartmodule-cargo-dependency => '"0.2.5"'
🔧   Destination: ~/smdk/map-example ...
🔧   Generating template ...
✔ 🤷   Will your SmartModule use init parameters? · false
✔ 🤷   Which type of SmartModule would you like? · map
✔ 🤷   Will your SmartModule be public? · false
Ignoring: /var/folders/5q/jwc86771549058kmbkbqjcdc0000gn/T/.tmpNFObJj/cargo-generate.toml
[1/5]   Done: Cargo.toml
[2/5]   Done: README.md
[3/5]   Done: SmartModule.toml
[4/5]   Done: src/lib.rs
[5/5]   Done: src
🔧   Moving generated files into: `~/smdk/map-example`...
💡   Initializing a fresh Git repository
✨   Done! New project created ~/smdk/map-example

We should see a new folder has been created for our project map-example. Let's navigate inside and take a look at the sample Map generated for us by the template:

$ cd map-example && cat ./src/lib.rs

use fluvio_smartmodule::{smartmodule, Result, SmartModuleRecord, RecordData};

#[smartmodule(map)]
pub fn map(record: &SmartModuleRecord) -> Result<(Option<RecordData>, RecordData)> {
    let key = record.key.clone();

    let string = std::str::from_utf8(record.value.as_ref())?;
    let int = string.parse::<i32>()?;
    let value = (int * 2).to_string();

    Ok((key, value.into()))
}

Let's break down what's happening here:

  • Firstly, #[smartmodule(map)] marks the entry point for this SmartModule Map. There may only be one of these in the project, and it is called once for each record in the data stream.
  • The annotated function fn map may be named anything, but it must take a single &Record argument. This variable contains the contents of one record in the stream, and you may read the Key and Value of this record as bytes.
  • The fn map function must return a new Key and Value for the output record. The Key is the Option<RecordData> and the Value is the RecordData in the return type. RecordData is a helper type that may be constructed from any type that has impl Into<Vec<u8>> such as String, by using .into().
  • At any point in the SmartModule, errors may be returned using ? or via Err(e.into()). This works for any error type that has impl std::error::Error.

This template SmartModule will parse each record as an i32 integer, then multiply that value by 2.

Let's make sure our code compiles. If everything works as expected, there will be a .wasm file generated in the target directory.

$ smdk build
...
Compiling map-example v0.1.0 (~/smdk/map-example)
Finished release-lto [optimized] target(s) in1 12.83s

Your WASM binary is now ready for use.

Test with SMDK​

Now that we've written our map, let's test using the command line.

$ smdk test --text=6
loading module at: ~/smdk/map-example/target/wasm32-unknown-unknown/release-lto/map_example.wasm
1 records outputted
12

Good news! 🎉 it works as expected!

Test on Cluster​

Let's create a new topic to produce our source data:

$ fluvio topic create map-double
topic "map-double" created

In a new terminal, produce the following entries:

$ fluvio produce map-double
> 1
Ok!
> 2
Ok!
> 3
Ok!
> 4
Ok!
> 5
Ok!
> ^C

Let's double check it's all there.

$ fluvio consume map-double -B -d
Consuming records from the beginning of topic 'map-double'
1
2
3
4
5

Load SmartModule to Fluvio​

The SmartModule can be loaded to local Fluvio Cluster or [InfinyOn Cloud], as determined by the [current profile]. In this example, the profile points to InfinyOn Cloud.

$ smdk load
Loading package at: ~/smdk/map-example
Found SmartModule package: map-example
loading module at: ~/smdk/map-example/target/wasm32-unknown-unknown/release-lto/map_example.wasm
Trying connection to fluvio router.infinyon.cloud:9003
Creating SmartModule: map-example

Rust fluvio smartmodule list to ensure your SmartModule has been uploaded:

$ fluvio smartmodule list
 SMARTMODULE                    SIZE
 john/map-example@0.1.0         87.5 KB

SmartModules that have been uploaded on the cluster can be used by other areas of the system (consumers, producers, connectors, etc):

$ fluvio consume map-double -B --smartmodule=john/map-example@0.1.0
Consuming records from the beginning of topic 'map-double'
2
4
6
8
10

Congratulations! 🎉 Everything worked as expected!

Publish to SmartModule Hub​

As bonus, let's [publish] this SmartModule to [SmartModule Hub].

$ smdk publish
Creating package john/map-example@0.1.0
.. fill out info in hub/package-meta.yaml
Package hub/map-example-0.1.0.ipkg created
Package uploaded!

Let's double check that the SmartModule is available for download:

$ fluvio hub list
  SMARTMODULE                 Visibility
  john/map-exampler@0.1.0      private
  ...

Congratulations! 🎉 Your SmartModule is now available for download in the SmartModule Hub.

[publish]: [InfinyOn Cloud]: [current profile]: [SmartModule Hub]: