Filter Transform

The Vector filter transform accepts and outputs log and metric events, allowing you to select events based on a set of logical conditions.

Configuration

vector.toml
[transforms.my_transform_id]
# General
type = "filter" # required
inputs = ["my-source-or-transform-id"] # required
# Condition
condition.type = "check_fields" # optional, default
condition."message.eq" = "this is the content to match against" # example
condition."message.eq" = ["match this", "or this"] # example
condition."message.contains" = "foo" # example
condition."message.contains" = ["foo", "bar"] # example
condition."environment.ends_with" = "-staging" # example
condition."environment.ends_with" = ["-staging", "-running"] # example
condition."message.regex" = " (any|of|these|five|words) " # example
condition."environment.starts_with" = "staging-" # example
condition."environment.starts_with" = ["staging-", "running-"] # example
  • tablecommonrequired

    condition

    The set of logical conditions to be matched against every input event. Only messages that pass all conditions will be forwarded.

    • stringenumcommonoptional

      type

      The type of the condition to execute.

      • Default: "check_fields"
      • Enum, must be one of: "check_fields" "is_log" "is_metric"
      • View examples
    • stringcommonoptional

      [field-name].eq

      Check whether a fields contents exactly matches the value specified.This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

      • Only relevant when: type = "check_fields"
      • No default
      • View examples
    • booloptional

      [field-name].exists

      Check whether a field exists or does not exist, depending on the provided value being true or false respectively.

      • Only relevant when: type = "check_fields"
      • No default
      • View examples
    • stringoptional

      [field-name].neq

      Check whether a fields contents does not match the value specified.This may be a single string or a list of strings, in which case this evaluates to false if any of the list matches.

      • Only relevant when: type = "check_fields"
      • No default
      • View examples
    • anyoptional

      [field-name].not_[condition]

      Check if the given [condition] does not match.

      • Only relevant when: type = "check_fields"
      • No default
      • View examples
    • stringcommonoptional

      [field_name].contains

      Checks whether a string field contains a string argument.This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

      • Only relevant when: type = "check_fields"
      • No default
      • View examples
    • stringcommonoptional

      [field_name].ends_with

      Checks whether a string field ends with a string argument.This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

      • Only relevant when: type = "check_fields"
      • No default
      • View examples
    • stringcommonoptional

      [field_name].regex

      Checks whether a string field matches a regular expression. Vector uses the documented Rust Regex syntax. Note that this condition is considerably more expensive than a regular string match (such as starts_with or contains) so the use of those conditions are preferred where possible.

      • Only relevant when: type = "check_fields"
      • No default
      • View examples
    • stringcommonoptional

      [field_name].starts_with

      Checks whether a string field starts with a string argument.This may be a single string or a list of strings, in which case this evaluates to true if any of the list matches.

      • Only relevant when: type = "check_fields"
      • No default
      • View examples

Examples

The filter transform is a simple conditional match, forwarding only those messages that pass all the conditions. In this example, we drop all events that do not come from the host gerry:

vector.toml
[transforms.from_gerry]
inputs = [ "somewhere" ]
type = "filter"
[transforms.from_gerry.condition]
"host.eq" = "gerry"
[sinks.only_gerry]
inputs = [ "from_gerry" ]
type = "something"

Any event that does not match all of the conditions in the filter will be dropped by the transform.

How It Works

Complex Processing

If you encounter limitations with the filter transform then we recommend using a runtime transform. These transforms are designed for complex processing and give you the power of full programming runtime.

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.