Elasticsearch plugin

Overview

The Elasticsearch plugin connects QALIPSIS to Elasticsearch.

Technology addressed

Elasticsearch: https://www.elastic.co/

Dependency

io.qalipsis.plugin:qalipsis-plugin-elasticsearch

Namespace in scenario

elasticsearch()

Client library

JAVA API: refer to Elasticsearch JAVA Client.

Supported steps

poll: polls the newest data periodically.
save: saves a batch records generated from the input and step context.
search: searches records and builds a query from the input and step context.

Example scenarios using a combination of Elasticsearch supported steps are provided on Github.

Poll step

The poll step within the Elasticsearch plugin periodically polls data from Elasticsearch. The output is a list of ElasticsearchDocument containing JSON data or deserialized to objects.

Ancestor: Scenario
Functionality: The poll step is created only once in a scenario; it is then used throughout the scenario as called. The poll step within the Elasticsearch plugin uses a strategy of “delivered exactly once”. The output of the poll step is an ElasticsearchDocument that contains maps of column to values.
Example

elasticsearch().poll {
    name = "poll"
    client { RestClient.builder(HttpHost.create("http://localhost:9200")).build() }
    query {
        """{
            "query":{
                "match_all":{}
            },
                "sort" : ["_score"]
            }""".trimIndent()
    }
    index("battery-state")
    pollDelay(Duration.ofSeconds(1))
}

Notable parameters

index: sets the elasticsearch indices to use for the queries; defaults to all.
client: configures the REST client to connect to Elasticsearch.
query: configures the builder for the JSON query to perform the first poll.
queryParameters(optional): adds parameters to the query.
pollDelay (required): specifies the delay between query executions.
broadcast (optional): specifies the broadcast parameters for the step.

Tips

The poll step is generally used in conjunction with a join to assert data or inject data into a workflow.
The query parameter requires at least one sorting field to filter the newest results and not fetch the same records repeatedly.
The output of the poll step is an ElasticsearchDocument that contains maps of column names to values. If flatten is called, the records are provided one by one to the next step; otherwise each poll batch remains complete.

Reference Documentation

Refer to Elasticsearch Query Parameters for more information on query parameters.
Refer to the Elasticsearch documentation for further parameter and configuration information.

Save step

The save step within the Elasticsearch plugin persists a batch of documents into Elasticsearch.

Ancestor: Step
Functionality: The save step’s input and step context are used to generate a list of documents that are forwarded to Elasticsearch.
Example

.elasticsearch()
.save {
    name = "save"
    client {
        RestClient.builder(HttpHost.create("http://localhost:9200"))
            .build()
    }
    documents { _, input ->
        listOf(
            Document(
                index = "battery-state",
                id = input.deviceId,
                source = objectMapper.writeValueAsString(input)
            )
        )
    }
}

Notable Parameters

client: configures the REST client to connect to Elasticsearch.
documents: defines the strategy to create documents to save from the input and step context.

Reference Documentation

Refer to the Elasticsearch documentation for further parameter information.

Search step

The search step within the Elasticsearch plugin searches documents in the Elasticsearch database using a search query.

Ancestor: Step
Functionality: The search step’s input and step context are used to generate a query; the batch of results is then forwarded to the next step(s).
Example

.elasticsearch()
.search {
    name = "search"
    client {
        RestClient.builder(HttpHost.create("http://localhost:9200")).build()
    }
    index { _, _ -> listOf("battery-state") }
    query { _, input ->
        """{
            "query": {
                "bool": {
                    "must": [
                        { "match": { "deviceId": "${input.input.deviceId}" }},
                        { "term": { "timestamp": ${input.input.timestamp.epochSecond} }}
                    ]
                }
            }
        }"""
    }
}

Notable parameters

client: configures the REST client to connect to Elasticsearch.
index: sets the elasticsearch indices to use for the queries; defaults to all.
query: configures the builder for the JSON query to perform a search.

Reference Documentation

Refer to the Elasticsearch documentation for further parameter and configuration information.
For specifics on queryParameters refer to Elasticsearch Query Parameters.
For specifics on search parameters, refer to Elasticsearch Search Parameters.

Configuration of publication of events and meters to Elasticsearch

Configuration namespace

Events: events.export.elasticsearch
Meters: meters.export.elasticsearch

Example configuration file

events:
    export:
        elasticsearch:
            enabled: true
            min-level: INFO
            urls: http://localhost:9200
            index-prefix: qalipsis-events
            index-date-pattern: yyyy-MM-dd
            duration-as-nano: false
            linger-period: 10s
            batch-size: 2000
            publishers: 1
            username:
            password:
            shards: 1
            replicas: 0

meters:
    export:
        elasticsearch:
            urls: http://localhost:9200
            host: ${meters.export.elasticsearch.urls}
            enabled: true
            step: PT10S
            index: qalipsis-meters
            index-date-format: yyyy-MM-dd
            auto-create-index: true

Event parameters

enabled (required): boolean flag that activates/deactivates event publishing to Elasticsearch; defaults to false; must be set to true.
min-level (required): minimal accepted level of events; allowable values are EventLevel`values: `TRACE, DEBUG, INFO, WARN, ERROR, OFF; defaults to INFO.
urls (required): list of URLs to the Elasticsearch instances; defaults to http://localhost:9200.
index-prefix (required): string prefix to use for the created indices containing the events; defaults to qalipsis-events.
path-prefix (required): string prefix used to demarcate url paths; defaults to http://.
index-date-pattern (required): string format of the date part of the index as supported by [java.time.format.DateTimeFormatter]; defaults to yyyy-MM-dd.
refresh-interval (required): string delay to refresh the indices in Elasticsearch; defaults to 10s.
store-source (required): boolean flag identifies whether the source of documents is stored with the values; set to true for yes and false for no; defaults to false.
linger-period (required): maximum period between publication of events to Elasticsearch; positive duration, defaults to 10s.
batch-size (required): maximum number of events buffered between two publications of events to Elasticsearch; positive integer; defaults to 2000.
publishers (required): number of concurrent publication of events that can run; positive integer; defaults to 1 (no concurrency).
shards (required): number of shards to apply on the created indices for events; positive integer; defaults to 1.
replicas (required): number of replicas to apply on the created indices for events; defaults to 0.
username (optional): name of the user for basic authentication when connecting to Elasticsearch.
password (optional): password of the user for basic authentication when connecting to Elasticsearch.

By default, Elasticsearch does not require a username or password for authentication. However, it is recommended that the information be provided to ensure secure access to the Elasticsearch data.

Meter parameters

urls (required): list of URLs to the Elasticsearch instances; defaults to http://localhost:9200.
host (required): defaults to the value of urls above.
enabled (required): boolean flag that activates/deactivates meter publishing to Elasticsearch; defaults to false; must be set to true.
step (required): The step size (reporting frequency) to use; defaults to PT10S.
index (required): string value to use for creation of index containing the meters; defaults to qalipsis-meters.
index-date-format (required): string format of the date part of the index as supported by [java.time.format.DateTimeFormatter]; defaults to yyyy-MM-dd.
auto-create-index (required): boolean flag to enable/disable auto creation of index; defaults to true.
proxy (optional): string URL of the HTTP proxy to use to access Elasticsearch; defaults to null.

It is recommended that the appropriate proxy configuration be identified (rather than letting it default to null) in order to enable support for various authentication mechanisms in Elasticsearch.
username (optional): name of the user for basic authentication when connecting to Elasticsearch.
password(optional): password of the user for basic authentication when connecting to Elasticsearch.

By default, Elasticsearch does not require a username or password for authentication. However, it is recommended that the information be provided to ensure secure access to the Elasticsearch data.