As the Elasticsearch ecosystem evolves, so do the tools and methodologies designed to streamline data management. One advancement that will significantly benefit our community is the ecs@mappings component template.
ECS (Elastic Common Schema) is a standardized data model for logs and metrics. It defines a set of common field names and data types that help ensure consistency and compatibility.
Elastic Common Schema and Open Telemetry
Elastic will preserve our user's investment in Elastic Common Schema by donating ECS to Open Telemetry. Elastic participates and collaborates with the OTel community to merge ECS and Open Telemetry's Semantic Conventions over time.
The Evolution of ECS Mappings
Historically, users and integration developers have defined ECS (Elastic Common Schema) mappings manually within individual index templates and packages, each meticulously listing its fields. Although straightforward, this approach proved time-consuming and challenging to maintain.
To tackle this challenge, integration developers moved towards two primary methodologies:
- Referencing ECS mappings
- Importing ECS mappings directly
These methods were steps in the right direction but introduced their challenges, such as the maintenance cost of keeping the ECS mappings up-to-date with Elasticsearch changes.
Enter ecs@mappings
The ecs@mappings component template supports all the field definitions in ECS, leveraging naming conventions and a set of dynamic templates.
Elastic started shipping the
With Elasticsearch v8.13.0, Elastic now includes
This move was a breakthrough because:
- Centralized and official: With ecs@mappings, we now have an official definition of ECS mappings.
- Out-of-the-box functionality: ECS mappings are readily available, reducing the need for additional imports or references.
- Simplified maintenance: The need to manually keep up with ECS changes has diminished since the template from Elasticsearch itself remains up-to-date.
Enhanced Consistency and Reliability
With
How Community Users Benefit
Community users stand to gain manifold from the adoption of
- Reduced configuration hassles: Whether you are an advanced user or just getting started, the simplified setup means fewer configuration steps and fewer opportunities for errors.
- Improved data integrity: Since ecs@mappings ensures that field definitions are accurate and up-to-date, data integrity is maintained effortlessly.
- Better performance: With less overhead in maintaining and referencing ECS fields, your Elasticsearch operations run more smoothly.
- Enhanced documentation and discoverability: As we standardize ECS mappings, the documentation can be centralized, making it easier for users to discover and understand ECS fields.
Let's explore how the
Reduced configuration hassles
Modern Elasticsearch versions come with out-of-the-box full ECS field support (see the “requirements” section later for specific versions).
For example, the Custom AWS Logs integration installed on a supported Elasticsearch cluster already includes the
GET _index_template/logs-aws_logs.generic
{
"index_templates": [
{
"name": "logs-aws_logs.generic",
...,
"composed_of": [
"logs@settings",
"logs-aws_logs.generic@package",
"logs-aws_logs.generic@custom",
"ecs@mappings",
".fleet_globals-1",
".fleet_agent_id_verification-1"
],
...
There is no need to import or define any ECS field.
Improved data integrity
The
To ensure that
Better Performance
Compact definitions
The ECS field definition is exceptionally compact; at the time of this writing, it is 228 lines long and supports all ECS fields. To learn more, see the
It relies on naming conventions and uses dynamic templates to achieve this compactness.
Lazy mapping
Elasticsearch only adds existing document fields to the mapping, thanks to dynamic templates. The lazy mapping keeps memory overhead at a minimum, improving cluster performance and making field suggestions more relevant.
Enhanced documentation and discoverability
All Elastic Agent integrations are migrating to the
Getting started
Requirements
To leverage the
- 8.9.0: if your data stream uses the logs index template or you define your index template.
- 8.13.0: if your data stream uses the index template of an Elastic Agent integration.
Example
We will use the Custom AWS Logs integration to show you how
Imagine you want to ingest the following log event using the Custom AWS Logs integration:
{
"@timestamp": "2024-06-11T13:16:00+02:00",
"command_line": "ls -ltr",
"custom_score": 42
}
Dev Tools
Kibana offers an excellent tool for experimenting with Elasticseatch API, the Dev Tools console. With the Dev Tools, users can run all API requests quickly and without much friction.
To open the Dev Tools:
- Open Kibana
- Select Management > Dev Tools > Console
Elasticsearch version < 8.13
On Elasticsearch versions before 8.13, the Custom AWS Logs integration has the following index template:
GET _index_template/logs-aws_logs.generic
{
"index_templates": [
{
"name": "logs-aws_logs.generic",
"index_template": {
"index_patterns": [
"logs-aws_logs.generic-*"
],
"template": {
"settings": {},
"mappings": {
"_meta": {
"package": {
"name": "aws_logs"
},
"managed_by": "fleet",
"managed": true
}
}
},
"composed_of": [
"logs-aws_logs.generic@package",
"logs-aws_logs.generic@custom",
".fleet_globals-1",
".fleet_agent_id_verification-1"
],
"priority": 200,
"_meta": {
"package": {
"name": "aws_logs"
},
"managed_by": "fleet",
"managed": true
},
"data_stream": {
"hidden": false,
"allow_custom_routing": false
}
}
}
]
}
As you can see, it does not include the ecs@mappings component template.
If we try to index the test document:
POST logs-aws_logs.generic-default/_doc
{
"@timestamp": "2024-06-11T13:16:00+02:00",
"command_line": "ls -ltr",
"custom_score": 42
}
The data stream will have the following mappings:
GET logs-aws_logs.generic-default/_mapping/field/command_line
{
".ds-logs-aws_logs.generic-default-2024.06.11-000001": {
"mappings": {
"command_line": {
"full_name": "command_line",
"mapping": {
"command_line": {
"type": "keyword",
"ignore_above": 1024
}
}
}
}
}
}
GET logs-aws_logs.generic-default/_mapping/field/custom_score
{
".ds-logs-aws_logs.generic-default-2024.06.11-000001": {
"mappings": {
"custom_score": {
"full_name": "custom_score",
"mapping": {
"custom_score": {
"type": "long"
}
}
}
}
}
}
These mappings do not align with ECS, so users and developers had to maintain them.
Elasticsearch version >= 8.13
On Elasticsearch versions equal to or newer to 8.13, the Custom AWS Logs integration has the following index template:
GET _index_template/logs-aws_logs.generic
{
"index_templates": [
{
"name": "logs-aws_logs.generic",
"index_template": {
"index_patterns": [
"logs-aws_logs.generic-*"
],
"template": {
"settings": {},
"mappings": {
"_meta": {
"package": {
"name": "aws_logs"
},
"managed_by": "fleet",
"managed": true
}
}
},
"composed_of": [
"logs@settings",
"logs-aws_logs.generic@package",
"logs-aws_logs.generic@custom",
"ecs@mappings",
".fleet_globals-1",
".fleet_agent_id_verification-1"
],
"priority": 200,
"_meta": {
"package": {
"name": "aws_logs"
},
"managed_by": "fleet",
"managed": true
},
"data_stream": {
"hidden": false,
"allow_custom_routing": false
},
"ignore_missing_component_templates": [
"logs-aws_logs.generic@custom"
]
}
}
]
}
The index template for
If we try to index the test document:
POST logs-aws_logs.generic-default/_doc
{
"@timestamp": "2024-06-11T13:16:00+02:00",
"command_line": "ls -ltr",
"custom_score": 42
}
The data stream will have the following mappings:
GET logs-aws_logs.generic-default/_mapping/field/command_line
{
".ds-logs-aws_logs.generic-default-2024.06.11-000001": {
"mappings": {
"command_line": {
"full_name": "command_line",
"mapping": {
"command_line": {
"type": "wildcard",
"fields": {
"text": {
"type": "match_only_text"
}
}
}
}
}
}
}
}
GET logs-aws_logs.generic-default/_mapping/field/custom_score
{
".ds-logs-aws_logs.generic-default-2024.06.11-000001": {
"mappings": {
"custom_score": {
"full_name": "custom_score",
"mapping": {
"custom_score": {
"type": "float"
}
}
}
}
}
}
In Elasticsearch 8.13, fields like
These mappings align with ECS, so users and developers do not have to maintain them. The same applies to all the hundreds of field definitions in the Elastic Common Schema. You can achieve this by including a 200-liner component template in your data stream.
Caveats
Some aspects of how the ecs@mappings component template deals with data types are worth mentioning.
ECS types are not enforced
The
For example, if you send the following document with a faas.coldstart field (defined as boolean in ECS):
{
"faas.coldstart": "true"
}
Elasticsearch will map
This is the tradeoff for having a compact and efficient ecs@mappings component template. It also allows for better compatibility when dealing with a mix of ECS and custom fields because documents won’t be rejected if the types are not consistent with the ones defined in ECS.
Conclusion
The introduction of
Whether you're an integration developer or a community user, moving to
Join the Conversation
Do you have questions or feedback about
Happy mapping!