Monitoring the test campaigns

Overview

QALIPSIS aims at giving insights into the behavior of the tested system from a performance perspective.

The insights are provided to users by way of dashboards and detailed values describing the performed operations.

Definitions

In QALIPSIS, each step generates different kinds of analytics data that supply different levels of detail about the behavior and performances of the system.

Meters

The highest level of detail is built on meters, that represent the behavior or performance of a step or operation over a short period of time as defined in the meter’s configuration.

The shorter the time period that a meter represents, the more data will be saved into the time-series database.
The data can only be fetched and reported from the time-series database where they were saved. They are not injected in a QALIPSIS scenario workflow for assertion.
In QALIPSIS, the meters (counters, timers, gauges, rates, throughputs) are attached to a single campaign and are created and removed according to the campaign lifecycle.

Meters are tagged to specify the scenario, campaign, test step and zone that they are related to.

Events

The finest level of detail is built on events that are directly saved into a time-series database.

Events represent an operation to be executed at a given point in time.

Events can be compared to logs as managed by log4j and logback, and apply a similar concept for configuration.
Events are described by a timestamp, a severity, a log name, and values.

Events are also tagged to specify the scenario, campaign, test step and zone that they are related to.

Recording events and meters in campaigns

To evaluate and analyze the performance and failures of your system during a campaign, you need to monitor events and meters at different levels and export the monitored data.

Enabling monitoring in steps

Scenarios within campaigns are made up of steps. By default, monitoring is disabled for each step.

Not all steps are suitable for monitoring. For example, there would be nothing to monitor in a delay step. However, load-generating and assertion steps generate data that is important to monitor.

QALIPSIS allows you to enable monitoring and configure the different levels of data that you want to record for each step.

Use the monitoring operator to enable and configure monitoring for each relevant step.

To generate events only (no meters) for the step:
```
monitoring { events = true }
```
To generate meters only (no events) for the step:
```
monitoring { meters = true }
```
To generate both events and meters for the step:
```
monitoring { all() }
```
Be precise when specifying the information to monitor. Recording too much information may generate overhead on your infrastructure and overshadow the information most important and relevant to your system’s behaviors and performance.

Exporting monitored data

QALIPSIS supports different technologies to record events and meters, providing the flexibility of using different analytic solutions that best suit your preferences and requirements. You can globally enable the export of events and meters to databases or messaging platforms.

To enable the export of events:
```
events.export.enabled=true
```
To enable the export of meters:
```
meters.export.enabled=true
```

QALIPSIS plugins can be used to export the monitored data to different technologies, such as time-series databases, message brokers, or other storage solutions. The following plugins are available for exporting events and meters:

Graphite: Refer to the Graphite plugin for details.
Elasticsearch: Refer to the Elasticsearch plugin for details.
Apache Kafka: Refer to the Apache Kafka plugin for details.
TimescaleDB and PostgreSQL: Refer to the TimescaleDB plugin for details.
InfluxDB: Refer to the InfluxDB plugin for details.

You can configure 0 to any number of exporters for events and meters, depending on your needs.

It is important to balance the count of publishers (when configurable) with the amount of event data that steps generate. If the amount of data is too large for the publishers to keep up with, there may be unwanted side effects. For example, the events buffer size may increase enough to cause memory issues and QALIPSIS may need time to exit after the end of a campaign to empty the buffers.

Exporting specific event data

QALIPSIS events are very similar to, and can be configured as, logs. The major difference is that QALIPSIS events are filtered based on event name and specified level. You can select a minimum level of event that is to be filtered, based upon the name of the events.

Each publisher can be configured to filter events based on their levels.

Event levels defined on a publisher

TRACE (default): all events are kept.
DEBUG: only events with DEBUG, INFO, WARN, or ERROR are kept.
INFO: only events with INFO, WARN, or ERROR are kept.
WARN: only events with WARN or ERROR are kept.
ERROR: only events with ERROR are kept.
OFF: no event is kept.

See the documentation of each plugin for details on how to configure the event levels.

Customizing the exported data

Exported data are tagged with a set of metadata that includes information about the scenario, campaign, test step, and zone. This metadata can be used to filter and analyze the exported data more effectively.

However, it is also possible to add your own tags to the exported data. This can be useful for adding additional context or for integrating with other systems or simply for improving the analysis of your tests.

The configuration is individual to each factory or standalone instance.

You can configure a list of tags by adding the one or more values pairs to the configuration parameter factory.tags. They will then be included in the metadata of the exported data by the related factory.

Example of tags

factory:
  tags:
    environment: staging
    team: backend
    project: my-project