Swimlanes Transform

The Vector swimlanes transform accepts and outputs log events, allowing you to route events across parallel streams using logical filters.

Configuration

vector.toml
[transforms.my_transform_id]
# General
type = "swimlanes" # required
inputs = ["my-source-or-transform-id"] # required
# Lanes
[transforms.my_transform_id.lanes.`[swimlane-id]`]
type = "check_fields" # optional, default
"message.eq" = "this is the content to match against" # example
"message.eq" = ["match this", "or this"] # example
"message.contains" = "foo" # example
"message.contains" = ["foo", "bar"] # example
"environment.ends_with" = "-staging" # example
"environment.ends_with" = ["-staging", "-running"] # example
"message.regex" = " (any|of|these|five|words) " # example
"environment.starts_with" = "staging-" # example
"environment.starts_with" = ["staging-", "running-"] # example
  • tablecommonrequired

    lanes

    A table of swimlane identifiers to logical conditions representing the filter of the swimlane. Each swimlane can then be referenced as an input by other components with the name <transform_name>.<swimlane_id>.

    • tablecommonrequired

      [swimlane-id]

      The identifier of a swimlane.

      • stringenumcommonoptional
        type

        The type of the condition to execute.

        • Default: "check_fields"
        • Enum, must be one of: "check_fields" "is_log" "is_metric"
        • View examples
      • stringcommonoptional
        [field-name].eq

        Check whether a fields contents exactly matches the value specified.This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

        • Only relevant when: type = "check_fields"
        • No default
        • View examples
      • booloptional
        [field-name].exists

        Check whether a field exists or does not exist, depending on the provided value being true or false respectively.

        • Only relevant when: type = "check_fields"
        • No default
        • View examples
      • stringoptional
        [field-name].neq

        Check whether a fields contents does not match the value specified.This may be a single string or a list of strings, in which case this evaluates to false if any of the list matches.

        • Only relevant when: type = "check_fields"
        • No default
        • View examples
      • anyoptional
        [field-name].not_[condition]

        Check if the given [condition] does not match.

        • Only relevant when: type = "check_fields"
        • No default
        • View examples
      • stringcommonoptional
        [field_name].contains

        Checks whether a string field contains a string argument.This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

        • Only relevant when: type = "check_fields"
        • No default
        • View examples
      • stringcommonoptional
        [field_name].ends_with

        Checks whether a string field ends with a string argument.This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

        • Only relevant when: type = "check_fields"
        • No default
        • View examples
      • stringcommonoptional
        [field_name].regex

        Checks whether a string field matches a regular expression. Vector uses the documented Rust Regex syntax. Note that this condition is considerably more expensive than a regular string match (such as starts_with or contains) so the use of those conditions are preferred where possible.

        • Only relevant when: type = "check_fields"
        • No default
        • View examples
      • stringcommonoptional
        [field_name].starts_with

        Checks whether a string field starts with a string argument.This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

        • Only relevant when: type = "check_fields"
        • No default
        • View examples

Examples

The swimlanes transform, in it's simplest form, can act as a simple if/else stream splitter. For example, we can route events from the host gerry to a sink only_loves_gerry, and all other events to a sink hates_gerry:

vector.toml
[transforms.splitting_gerrys]
inputs = [ "somewhere" ]
type = "swimlanes"
[transforms.splitting_gerrys.lanes.only_gerrys]
type = "check_fields"
"host.eq" = "gerry"
[transforms.splitting_gerrys.lanes.no_gerrys]
type = "check_fields"
"host.neq" = "gerry"
[sinks.only_loves_gerry]
inputs = [ "splitting_gerrys.only_gerrys" ]
type = "something"
[sinks.hates_gerry]
inputs = [ "splitting_gerrys.no_gerrys" ]
type = "something_else"

This syntax makes it easy to create arbitrary numbers of swimlanes, each with their own logic, and easily route them to downstream components.

How It Works

Complex Processing

If you encounter limitations with the swimlanes transform then we recommend using a runtime transform. These transforms are designed for complex processing and give you the power of full programming runtime.

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.