Elastic Open Web Crawler
An intelligent, intuitive indexing tool
The fastest way to index web content into elasticsearch on serverless, in the cloud, or on-prem
Start crawling now!
Set up and and deploy a crawler for your web content with a terminal and elasticsearch.
elasticsearch — the most widely deployed vector database
Take control with open code
Customize Elastic Open Web Crawler (Open Crawler) to fit your needs. Inspect, modify, and contribute to your project while handling large documents, running transformations, and retrieving data in your desired format.
Flexible and fast: The Open Crawler advantage
Benefit from index naming without limitations and the ability to use custom mappings before crawling. Boost performance by bulk indexing crawl results into elasticsearch instead of one web page at a time.
Manage deployments with ease
Manage your open web crawler programmatically with simple CLI commands. Scale deployments easily with Terraform or Puppet — and spin up or down as needed. Eliminate unnecessary dependencies for simplified management. Deploy it anywhere, including serverless environments, and connect easily with small, simple tools.
