GraphQL offers an efficient and flexible way to query data. This blog will explain how Hasura DDN works with elasticsearch to make high performing and metadata-driven access to data.
The code and setup for this example can be found on this GitHub repo - elasticsearch-subgraph-example.
Hasura DDN is a metadata-driven data access layer built for the cloud. It automatically generates APIs that support both transactional and analytical workloads. By leveraging the metadata—such as models, relationships, permissions, and security rules—Hasura creates APIs that are optimized for performance, delivering low-latency responses and handling high-concurrency demands with ease.
Role of Metadata-driven APIs in Search AI World
Instead of manually coding each endpoint and its associated logic, metadata-driven APIs use a declarative approach. The structure of your data sources (like the elasticsearch indices) is described in a standardized format. Relationships between different entities are defined. Permissions and security rules are specified at a granular level, all using configuration.
Based on this metadata, the API layer is automatically provisioned and kept in sync with your data sources.
For elasticsearch, using Hasura DDN’s metadata-driven APIs provide unified and consistent data access. Changes in data are immediately reflected in the API, which is crucial for real-time search and AI applications.
Architecture
In the architecture above, Hasura is the Supergraph that connects to multiple subgraphs, elasticsearch being one of the data sources in a subgraph.
Setup a GraphQL API for elasticsearch
This setup walks you through connecting Hasura DDN to a locally running elasticsearch instance using Docker. However, you can easily switch to Elastic Cloud by updating the environment variables with the right credentials. Using Elastic Cloud is the recommended way to experience elasticsearch in production environments, offering a managed, scalable, and secure deployment.
Setup: Loading sample dataset
git clone https://github.com/hasura/elasticsearch-subgraph-exampleCopy .env.example to .env and set values for elasticsearch_PASSWORD
Start elasticsearch locally with sample indices
docker compose up -dVisit http://localhost:9200 to verify that elasticsearch is running with sample data.
GraphQL Subgraph for elasticsearch
In this section, we'll set up a GraphQL subgraph that connects Hasura DDN to your elasticsearch instance. A subgraph allows you to expose elasticsearch as a queryable API, providing a flexible and efficient way to perform complex searches, aggregations, and filtering through GraphQL.
Pre-requisites:
- Hasura CLI [Install from here]
- Login and authenticate using
ddn auth login
Initialize a Supergraph
ddn supergraph init .Initialize elasticsearch Connector
ddn connector init -iIn the quickstart wizard, enter the following values for environment variables:
elasticsearch_URL=http://local.hasura.dev:9200
elasticsearch_USERNAME=elastic
elasticsearch_PASSWORD=elasticpwdTo use Elastic Cloud instead of a local instance, simply modify the environment variables in your .env file. Replace the elasticsearch_URL, elasticsearch_USERNAME, and elasticsearch_PASSWORD values with the corresponding credentials from your Elastic Cloud deployment.
Hasura DDN connected to elasticsearch for introspecting and generating GraphQL APIs.
Introspect the elasticsearch instance and track all indices and collections
ddn connector introspect elasticsearch --add-all-resourcesStart Supergraph locally:
ddn run docker-startBuild Supergraph locally:
ddn supergraph build localVisit https://console.hasura.io/local/graphql?url=http://localhost:3000 to start exploring the local supergraph.
GraphQL queries for search
Now that we have set up our Hasura DDN and applied the metadata-driven APIs to elasticsearch, let's craft GraphQL queries to perform search operations.
The following queries highlight how Hasura translates even complex search and aggregation requirements into simple, declarative GraphQL operations. These examples showcase not only the flexibility of GraphQL, but also the standardization Hasura brings, enabling consistent API access across different data sources.
Fetch 5 products (simple query)
query searchProducts {
products(limit: 5) {
id
price
name
productId
}
}Fetch 5 products whose product name matches the term “shoes” (search query with phrase match)
query searchProducts {
products(limit: 5, where: {name: {match_phrase: "shoes"}}) {
id
price
name
productId
}
}Fetch an aggregate of products matching a filter criteria (aggregate query)
query aggregateOfProducts {
productsAggregate(filter_input: {where: {name: {match_phrase: "shoes"}}}) {
name {
_count
}
}
}Note: This integration is not limited to search APIs and can be extended to logging and observability data use cases in elasticsearch.
Hasura's support for composability and standard APIs makes it possible to connect multiple data sources (Postgres, MongoDB, REST, etc.) with elasticsearch, building a larger Supergraph that serves cross-team needs. This composability reduces technical debt by allowing different teams to access the same API endpoints and data sources in a consistent, standardized way.
Whether you're building search experiences or advanced analytics dashboards, Hasura enables your team to focus on application logic rather than API management, improving speed-to-market and reducing operational complexity.
Performance considerations at scale
One of the key benefits of combining Hasura and elasticsearch is the optimized performance through predicate pushdown. Hasura DDN intelligently compiles and pushes filters, limits, and sorts directly to elasticsearch, reducing the overhead of N+1 queries and minimizing data over-fetching.
For instance, the following GraphQL query:
query searchProducts {
products(limit: 5, where: { name: { match_phrase: "shoes" } }) {
id
price
name
productId
}
}Generates an elasticsearch query similar to:
{
"_source": [
"_id",
"price",
"name",
"product_id"
],
"query": {
"match_phrase": {
"name": "shoes"
}
},
"size": 5
}By requesting only the necessary fields (_source) and limiting the number of documents fetched (size), Hasura ensures that elasticsearch performs optimally. This is a significant improvement over traditional, manually-coded APIs, where each new requirement demands additional hand-written queries.
Summary
As explored in this post, the Hasura DDN connector for elasticsearch opens up new possibilities in accelerating the GraphQL APIs for elasticsearch and building a larger Supergraph in the organization, collaborating with multiple teams.
Hasura's metadata-driven approach simplifies API development, providing a fast, consistent, and secure layer for accessing elasticsearch data through GraphQL. By leveraging predicate pushdown, Hasura ensures optimal search performance. Learn more about Hasura’s capabilities for elasticsearch.
We're excited to see what you'll build!
Ready to try this out on your own? Start a free trial.
Want to get Elastic certified? Find out when the next elasticsearch Engineer training is running!
Related content
October 25, 2024
Using Kibana to ingest geospatial data into elasticsearch for use in ES|QL
How to use Kibana and the csv ingest processor to ingest geospatial data into elasticsearch for use with search in elasticsearch Query Language (ES|QL). elasticsearch has powerful geospatial search features, which are now coming to ES|QL for dramatically improved ease of use and OGC familiarity. But to use these features, we need Geospatial data.
October 24, 2024
How to use the ES|QL Helper in the elasticsearch Ruby Client
Learn how to use the elasticsearch Ruby client to craft ES|QL queries and handle their results.
October 21, 2024
Efficient bitwise matching in elasticsearch
Explore six approaches to encoding and matching binary data in elasticsearch, including terms encoding (my preferred approach), boolean encoding, sparse bit position encoding, integer encoding with exact matching, integer encoding with scripted bitwise matching, and integer encoding with ESQL for bitwise matching, with practical examples and use cases.
October 16, 2024
How to use elasticsearch with popular Ruby tools
Take a look at how to use elasticsearch with some popular Ruby libraries.
October 17, 2024
Unlock the Power of Your Data with RAG using Vertex AI and elasticsearch
Unlock your data's potential with RAG using Vertex AI and elasticsearch. This blog series covers data ingestion into elasticsearch for a robust knowledge base for creating advanced RAG based search applications.