Graphite plugin

Overview

The Graphite plugin connects QALIPSIS to Graphite using the pickle or plaintext protocol.

Technology addressed

Graphite: https://graphiteapp.org/

Dependency

io.qalipsis:plugin-graphite

Namespace in scenario

graphite()

Client library

Supported steps

Poll step

The poll step within the Graphite plugin collects metrics from a Graphite server and broadcasts them to other components.

Ancestor

Scenario or Step

Functionality

The ‘poll’ step is typically used to collect and distribute metrics for monitoring and visualization purposes.

Example
graphite().poll {
  name = "my-poll-step"

  connect {
    server("http://localhost:8080")
  }

  query(
    GraphiteQuery("device.battery-state.*")
      .from(
        GraphiteMetricsTime(
          2,
          GraphiteMetricsTimeSignUnit.MINUS,
          GraphiteMetricsTimeUnit.DAYS
        )
      )
      .noNullPoints(true)
  )
  broadcast(123, Duration.ofSeconds(20))
  pollDelay(Duration.ofSeconds(1))
}
Reference documentation

Refer to the Graphite documentation for parameter and configuration information.

Save step

The save step within the Graphite plugin persists records into a Graphite database using the save query.

Ancestor

Scenario or Step

Functionality

The save step’s input and step context are used to generate documents to save into Graphite.

Example
.graphite()
.save {

  connect {
    server("localhost", 2003)
  }

  records { StepContext, input ->
    listOf(
      GraphiteRecord(
        metricPath = "device.battery-state.${input.deviceId.lowercase()}",
        value = input.batteryLevel,
        timestamp = input.timestamp
      )
    )
  }
  monitoring {
    events = false
    meters = true
  }
}
Reference documentation

Refer to the Graphite documentation for further parameter and configuration information.

Configuration

DSL parameters

Available parameters are described in the table below.

Parameter Description

connect

Configures the connection to the Graphite server.
Applicable Steps: Poll, Save
Optional/Required: Required
Data Type: Lambda within which connection parameters are defined.
Default Value: N/A

Example (Save)
connect {
  server("localhost", 2003)
  protocol(GraphiteProtocol.PICKLE)
  netty(channelClass) { workerGroup }
}
Example (Poll)
connect {
  server("http://localhost:8090/render")
  basicAuthentication("myUser", "password")
}

connect.server

Sets the base URL of the Graphite HTTP server used to execute queries.
Applicable Steps: Poll, Save
Optional/Required: Required
Data Type: String
Default Value: N/A

Example (Save)
server("localhost", 2003)
Example (Poll)
server("http://localhost:8090/render")

records

Factory that produces a collection of GraphiteRecord instances to be sent for each input.
Applicable Steps: Save
Optional/Required: Required
Data Type: Lambda that receives the step context and input to return Collection<GraphiteRecord>.
Default Value: N/A

Example
records { stepContext, input ->
  listOf(
    GraphiteRecord(
      metricPath = "device.battery-state.${input.deviceId.lowercase()}",
      value = input.batteryLevel,
      timestamp = input.timestamp
    )
  )
}

GraphiteRecord

Defines the data format of the saved records.
Applicable Steps: Save
Optional/Required: Required
Data Type: GraphiteRecord data class
Default Value: N/A

Example (arrayOf tags)
GraphiteRecord(
  metricPath = "device.battery-state.${input.deviceId.lowercase()}",
  value = input.batteryLevel,
  timestamp = input.timestamp,
  tags = arrayOf("env" to "prod", "zone" to "eu-west")
)
Example (mapOf tags)
GraphiteRecord(
  metricPath = "device.battery-state.${input.deviceId.lowercase()}",
  value = input.batteryLevel,
  timestamp = input.timestamp,
  tags = mapOf("env" to "prod", "zone" to "eu-west")
)

metricPath, path

Metric name or path used by Graphite.
Applicable Steps: Save
Optional/Required: Required
Data Type: String
Default Value: N/A

Example
metricPath = "device.battery-state.${input.deviceId.lowercase()}"

value

Numeric value to record.
Applicable Steps: Save
Optional/Required: Required
Data Type: Number
Default Value: N/A

Example
value = input.batteryLevel

timestamp

Captures the exact moment the value recorded.
Applicable Steps: Save
Optional/Required: Required
Data Type: java.time.Instant
Default Value: N/A

Example
timestamp = input.timestamp

tags

Optional labels attached to the record.
Applicable Steps: Save
Optional/Required: Required
Data Type: Map<String, String> | vararg tags: Pair<String, String>
Default Value: N/A

Examples
tags = mapOf("env" to "prod")
tags = arrayOf("env" to "prod", "zone" to "eu-west")

monitoring

Configures QALIPSIS monitoring (meters/events) for the step.
Refer to Monitoring parameters for configuration details and examples of usage.

query

Specifies a query to poll data from the Graphite server.
Applicable Steps: Poll
Optional/Required: Required
Data Type: Lambda within which query parameters are defined.
Default Value: N/A

Example
query {
  withTarget("device.battery-state.*")
    .from(GraphiteMetricsTime(-2, GraphiteMetricsTimeUnit.DAYS))
    .noNullPoints(true)
}

query.withTarget

Adds a new target to the request.
Applicable Steps: Poll
Optional/Required: Optional
Data Type: Function
Default Value: N/A

Example
query {
  withTarget("device.battery-state.*")
}

query.withTargetFromSeriesByTag

Adds a new target defined on a series by tag to the request.
Applicable Steps: Poll
Optional/Required: Optional
Data Type: Function
Default Value: N/A

Example
query {
  withTargetFromSeriesByTag("tag-name", "~qalipsis.test.key.*")
}

query.from

Sets the start time for the query. Accepts multiple types; Instant overload requires a date after epoch.
Applicable Steps: Poll
Optional/Required: Optional
Data Type: Function
Default Value: N/A

Examples
query {
  from("-2h")
}
query {
  from(1620000000L)
}
query {
  from(instant.now()) //Instant must be after epoch.
}
query {
  from(GraphiteMetricsTime(1, GraphiteMetricsTimeUnit.HOURS))
}

query.until

Sets the end time for the query.
Applicable Steps: Poll
Optional/Required: Optional
Data Type: Function
Default Value: N/A

Examples
query {
  until("-2h")
}
query {
  until(1620000000L)
}
query {
  until(instant.now()) //Instant must be after epoch
}
query {
  until(GraphiteMetricsTime(1, GraphiteMetricsTimeUnit.HOURS))
}

query.noNullPoints

When true, null points are excluded from the returned series.
Applicable Steps: Poll
Optional/Required: Optional
Data Type: Boolean Function
Default Value: True

Example
query {
  noNullPoints(true)
}

GraphiteMetricsTime

The window of time to retrieve data from Graphite.
Applicable Steps: Poll
Optional/Required: Optional
Data Type: Graphite time shorthand (amount + unit).
Default Value: True

Example
GraphiteMetricsTime(1, GraphiteMetricsTimeUnit.HOURS)

GraphiteMetricsTimeUnit

Units for the Graphic Metrics Time
Applicable Steps: Poll
Optional/Required: Optional
Data Type: Graphite time shorthand (amount + unit).
Default Value: True

Example
GraphiteMetricsTime(1, GraphiteMetricsTimeUnit.HOURS)

singletonConfiguration

Configures how polled results are delivered to scenario minions.
Refer to SingletonConfiguration parameters for details on strategies and examples of usage.

pollDelay

Frequency/delay between two poll executions.
Applicable Step: Poll
Optional/Required: Optional
Data Type: Duration
Default Value: 10s

Example
pollDelay(Duration.ofSeconds(1))

Shared defaults for Graphite steps

You can define defaults once in the scenario section or just after, and let all following Graphite steps inherit them.

scenario {
  graphite().defaults { (1)
    pollConnection {
      server("http://localhost:8090/render")
    }
    saveConnection {
      server("localhost", 2003)
      protocol(GraphiteProtocol.PICKLE)
    }
    monitoring {
      events = true
      meters = true
    }
  }
}
1 Defaults are applied to subsequent Graphite steps in the same scenario. Individual steps can still override values.

Analytics

QALIPSIS can publish the meters and events collected during a campaign to Graphite through dedicated publishers.

Configure the publishers in the factory configuration, separate from the scenario DSL. Refer to Provide configuration for information about specifying the configuration to QALIPSIS.

Meters

The meters publisher sends the meter snapshots collected during a campaign to Graphite using the configured transport protocol.

Example configuration file (YAML)
meters:
  export:
    graphite:
      enabled: true
      host: localhost
      port: 2004
      protocol: PICKLE
      batch-size: 100
      prefix: qalipsis.
      publishers: 1

The parameters used to configure the publication of meters to Graphite are described in the table below.

Parameter Description

meters.export.graphite.enabled

Activates the publication of meters to Graphite. Must be set to true to enable publishing.
Data Type: Boolean
Default Value: false

meters.export.graphite.host

Host of the Graphite instance receiving the meters.
Data Type: String
Default Value: localhost

meters.export.graphite.port

Port of the Graphite instance receiving the meters.
Data Type: Positive integer
Default Value: 2004

meters.export.graphite.protocol

Transport protocol used to send meters to Graphite. Acceptable values are PLAINTEXT and PICKLE.
Data Type: ENUM
Default Value: PICKLE

meters.export.graphite.batch-size

Maximum number of meter snapshots batched together before being sent to Graphite.
Data Type: Positive integer
Default Value: 100

meters.export.graphite.prefix

Prefix prepended to every metric name when publishing to Graphite.
Data Type: String
Default Value: qalipsis.

meters.export.graphite.publishers

Number of concurrent publishers that can send meter batches to Graphite.
Data Type: Positive integer
Default Value: 1 (no concurrency)

Events

The events publisher buffers the events emitted during a campaign and sends them to Graphite using the configured transport protocol.

Example configuration file (YAML)
events:
  export:
    graphite:
      enabled: true
      min-level: TRACE
      host: localhost
      port: 2004
      protocol: PICKLE
      batch-size: 100
      linger-period: 10s
      publishers: 2
      prefix: qalipsis.

The parameters used to configure the publication of events to Graphite are described in the table below.

Parameter Description

events.export.graphite.enabled

Activates the publication of events to Graphite. Must be set to true to enable publishing.
Data Type: Boolean
Default Value: false

events.export.graphite.min-level

Minimum event level published. Acceptable values are TRACE, DEBUG, INFO, WARN, ERROR, OFF.
Data Type: ENUM
Default Value: TRACE

events.export.graphite.host

Host of the Graphite instance receiving the events.
Data Type: String
Default Value: localhost

events.export.graphite.port

Port of the Graphite instance receiving the events.
Data Type: Positive integer
Default Value: 2004

events.export.graphite.protocol

Transport protocol used to send events to Graphite. Acceptable values are PLAINTEXT and PICKLE.
Data Type: ENUM
Default Value: PICKLE

events.export.graphite.batch-size

Maximum number of events batched together before being sent to Graphite.
Data Type: Positive integer
Default Value: 100

events.export.graphite.linger-period

Maximum time the events are buffered before forcing a flush to Graphite, even when the batch size is not reached.
Data Type: Positive duration
Default Value: 10s

events.export.graphite.publishers

Number of concurrent publishers that can send event batches to Graphite.
Data Type: Positive integer
Default Value: 2

events.export.graphite.prefix

Prefix prepended to every event name when publishing to Graphite.
Data Type: String
Default Value: qalipsis.
Reference Documentation

Refer to Monitoring test campaigns for a further explanation of the meter and event parameter values.