Streams
Serverless Stack
Streams provides a single, centralized UI within Kibana that streamlines common tasks like extracting fields, setting data retention, and routing data, so you don't need to use multiple applications or manually configure underlying Elasticsearch components.
Streams can operate in two modes: wired and classic. Both manage data streams in Elasticsearch, but differ in configuration, inheritance, and field mapping.
Classic streams work with existing Elasticsearch data streams. Use classic streams when you want the ease of extracting fields and configuring data retention while working with data that's already being ingested into Elasticsearch.
Classic streams:
- Are based on existing data streams, index templates, and component templates.
- Can follow the data retention policy set in the existing index template.
- Do not support hierarchical inheritance or cascading configuration updates.
Serverless Stack
Wired streams data is sent directly to a single endpoint, from which you can route data into child streams based on partitioning set up manually or with the help of AI suggestions.
Wired streams:
- Allow you to organize streams in a parent-child hierarchy.
- Let child streams automatically inherit mappings, lifecycle settings, and processors.
- Send configuration changes through the hierarchy to keep streams consistent.
For more information, refer to sending data to wired streams.
When you configure classic or wired streams through the Streams UI or Streams API, Elasticsearch-level components like templates and pipelines are created for the stream. These components are considered managed and shouldn't be modified using Elasticsearch APIs. When managing a stream through the Streams UI or API, continue doing so whenever possible.
You can still edit non-managed ingest pipelines, templates, and other components, but avoid those marked as managed or any per-data-stream mappings and settings. This behavior is similar to how Elasticsearch handles components managed by integrations. Refer to the Advanced tab to review managed components.
Streams requires the following permissions:
Streams requires these Elastic Cloud Serverless roles:
- Admin: Ability to manage all Streams
- Editor/Viewer: Limited access, unable to perform all actions
To manage all streams, you need the following permissions:
- Cluster permissions:
manage_index_templates,manage_ingest_pipelines,manage_pipeline,read_pipeline - Data stream level permissions:
read,write,create,manage,monitor,manage_data_stream_lifecycle,read_failure_store,manage_failure_store,manage_ilm.
To view streams, you need the following permissions:
- Data stream level:
read,view_index_metadata,monitor
For more information, refer to Cluster privileges and Granting privileges for data streams and aliases
Open Streams from the following places in Kibana:
Select Streams from the navigation menu or use the global search field.
Open the data stream for a specific document from Discover. To do this, expand the details flyout for a document that's stored in a data stream, and select Stream or an action associated with the document's data stream. Streams will open filtered to the selected data stream.
Serverless Stack
You can also access Streams features using the Streams API. Refer to the Streams API documentation for more information.
Interact with and configure your streams in the following ways:
- Retention: Manage how your stream retains data and get insight into data ingestion and storage size.
- Partitioning: Stack Serverless Route data into child streams.
- Processing: Parse and extract information from documents into dedicated fields.
- Schema: Manage field mappings.
- Data quality: Get information about failed and degraded documents in your stream.
- Advanced: Review and manually modify underlying Elasticsearch components of your stream.