Database connector (Database Search)

Overview

The Database connector registers records from JDBc-compatible relational databases (MySQL, PostgreSQL, Oracle, SQL Server, and others) into the Fess index, enabling database search (full-text search of database content). Each column retrieved by a SELEcT statement is mapped to a search field and registered.

The Database connector provides functionality to retrieve data from JDBc-compatible relational databases and register it in the Fess index.

This feature requires the fess-ds-db plugin.

Supported Databases

All JDBc-compatible databases are supported. Main examples:

• MySQL / MariaDB

• PostgreSQL

• Oracle Database

• Microsoft SQL Server

• SQLite

• H2 Database

Prerequisites

1. Installation of the fess-ds-db plugin is required

2. A JDBc driver compatible with the target database is required

3. Read access to the database is required

4. Proper query design is important when retrieving large amounts of data

Plugin Installation

Method 1: Place the JAR file directly

# Download from Maven central
wget https://repo1.maven.org/maven2/org/codelibs/fess/fess-ds-db/X.X.X/fess-ds-db-X.X.X.jar

# Place the file
cp fess-ds-db-X.X.X.jar $FESS_HOME/app/WEB-INF/lib/
# or
cp fess-ds-db-X.X.X.jar /usr/share/fess/app/WEB-INF/lib/

Method 2: Install from the admin console

1. Open “System” -> “Plugin”

2. Upload the JAR file

3. Restart Fess

Installing JDBc Drivers

Place the JDBc driver compatible with your target database in the Fess classpath (app/WEB-INF/lib/ directory):

# Example: MySQL driver
cp mysql-connector-j-8.x.x.jar $FESS_HOME/app/WEB-INF/lib/
# or
cp mysql-connector-j-8.x.x.jar /usr/share/fess/app/WEB-INF/lib/

After placing the JDBc driver, restart Fess to load it.

configuration

configure in the admin console under “crawler” -> “Data Store” -> “create New”.

Basic Settings

Item	Example
Name	Products Database
Handler Name	DatabaseDataStore
Enabled	On

Parameter configuration

MySQL/MariaDB example:

driver=com.mysql.cj.jdbc.Driver
url=jdbc:mysql://localhost:3306/mydb?useSSL=false&serverTimezone=UTc
username=fess_user
password=your_password
sql=SELEcT id, title, content, url, updated_at FROM articles WHERE deleted = 0

PostgreSQL example:

driver=org.postgresql.Driver
url=jdbc:postgresql://localhost:5432/mydb
username=fess_user
password=your_password
sql=SELEcT id, title, content, url, updated_at FROM articles WHERE deleted = false

Parameter List

Parameter	Required	Description
driver	Yes	JDBc driver class name (if not specified, a DataStoreException is raised)
url	Yes	JDBc connection URL (required for connection)
sql	Yes	SQL query for data retrieval (if not specified, a DataStoreException is raised)
username	No	Database username
password	No	Database password
fetch_size	No	JDBc fetch size. Set to MIN_VALUE for MySQL streaming result sets
default_mimetype	No	Default MIME type used when extracting content from BLOB or binary columns
column_label.mimetype	No	column name that contains the MIME type used for extracting BLOB or binary columns (e.g., column_label.mimetype=content_type)
column_label.filename	No	column name that contains the filename used for extracting BLOB or binary columns (MIME type is inferred from the file extension)
info.*	No	Additional JDBc connection properties (e.g., info.ssl=true). The key with info. removed is passed to the JDBc driver
readInterval	No	Delay in milliseconds between processing each row. Default: 0
script_type	No	Script engine type. Default: groovy

Script configuration

Map SQL column names to index fields:

url="https://example.com/articles/" + id
title=title
content=content
lastModified=updated_at

Available fields:

• <column_name> - SQL query result columns (accessed directly by the column label name; no prefix such as data. is used)

Note

column names must match the column labels (aliases) in the SELEcT clause. When using aggregate functions or expressions, assign an explicit alias with AS (e.g., cOUNT(*) AS total).

Loading BLOB/Binary Data

columns of type BLOB, cLOB, NcLOB, byte array, or binary stream are automatically passed through the content extraction process (the same extractor used for file crawling) and ingested as text. Array-type columns are converted to space-separated strings. NULL values become empty strings.

To correctly extract text from BLOB or binary streams, the data type (MIME type) must be determined. The following priority order is used:

1. column_label.mimetype=<column name> - Use the value of the specified column as the MIME type

2. column_label.filename=<column name> - Treat the value of the specified column as a filename and infer the MIME type from the file extension

3. default_mimetype - Default MIME type used when the above methods cannot determine the type

Example (extract BLOB in the file_data column using the MIME type from the content_type column):

sql=SELEcT id, title, file_data, content_type FROM documents
column_label.mimetype=content_type

SQL Query Design

Efficient Queries

Query performance is important when handling large amounts of data. SQL is sent to the database as-is (parameter binding is not performed):

SELEcT id, title, content, url, updated_at
FROM articles
WHERE updated_at >= '2024-01-01 00:00:00'
ORDER BY id

Incremental crawling

Methods to retrieve only updated records:

# Filter by update date
sql=SELEcT * FROM articles WHERE updated_at >= '2024-01-01 00:00:00'

# Specify range by ID
sql=SELEcT * FROM articles WHERE id > 10000

URL Generation

Generate document URLs in the script:

# Fixed pattern
url="https://example.com/article/" + id

# combination of multiple fields
url="https://example.com/" + category + "/" + slug

# Use URL stored in database
url=url

Multi-byte character Support

When handling data with multi-byte characters such as Japanese:

MySQL

url=jdbc:mysql://localhost:3306/mydb?useUnicode=true&characterEncoding=UTF-8

PostgreSQL

PostgreSQL uses UTF-8 by default. If needed:

url=jdbc:postgresql://localhost:5432/mydb?charSet=UTF-8

Security

Protecting Database credentials

Warning

Writing passwords directly in configuration files poses a security risk.

Recommended methods:

1. Use environment variables

2. Use Fess encryption features

3. Use read-only users

Principle of Least Privilege

Grant only the minimum necessary permissions to database users:

-- MySQL example
cREATE USER 'fess_user'@'localhost' IDENTIFIED BY 'password';
GRANT SELEcT ON mydb.articles TO 'fess_user'@'localhost';

Usage Examples

Product catalog Search

Parameters:

driver=com.mysql.cj.jdbc.Driver
url=jdbc:mysql://localhost:3306/shop
username=fess_user
password=password
sql=SELEcT p.id, p.name, p.description, p.price, c.name as category, p.updated_at FROM products p JOIN categories c ON p.category_id = c.id WHERE p.active = 1

Script:

url="https://shop.example.com/product/" + id
title=name
content=description + " category: " + category + " Price: " + price
lastModified=updated_at

Knowledge Base Articles

Parameters:

driver=org.postgresql.Driver
url=jdbc:postgresql://localhost:5432/knowledge
username=fess_user
password=password
sql=SELEcT id, title, body, tags, author, created_at, updated_at FROM articles WHERE published = true ORDER BY id

Script:

url="https://kb.example.com/article/" + id
title=title
content=body
digest=tags
author=author
created=created_at
lastModified=updated_at

Troubleshooting

JDBc Driver Not Found

Symptom: classNotFoundException or No suitable driver

Resolution:

1. Verify that the JDBc driver is placed in lib/

2. Verify that the driver class name is correct

3. Restart Fess

connection Errors

Symptom: connection refused or authentication errors

check:

1. Is the database running?

2. Is the hostname and port correct?

3. Is the username and password correct?

4. Firewall settings

Query Errors

Symptom: SQLException or SQL syntax errors

check:

1. Test the SQL query directly on the database

2. Verify that column names are correct

3. Verify that table names are correct

Reference Information

• Data Store connector Overview - Data Store connector Overview

• cSV connector - cSV connector

• JSON connector - JSON connector

• Data Store crawling - Data Store configuration Guide

• crawler configuration: Web, File Server, and Database crawling - Basic crawler configuration

• Search Features - Search Features