Elasticsearch Primer: From Raw Text to Ranked Search Results
In a relational database, reaching for WHERE name LIKE '%headphones%' is often the most intuitive first step when searching for products containing “headphones”. For systems with modest datasets and straightforward queries, simple substring matching gets the job done.
Real-world search, however, quickly outgrows basic substring matching. When users type wireless headphones, they typically expect items matching both terms to appear at the very top, followed by items matching only one. Nor do they expect search results to fail completely just because of differing letter cases.
Elasticsearch is a search and analytics engine built specifically to solve these kinds of problems. It runs on top of Apache Lucene—the open-source full-text search library that serves as its underlying engine—and exposes its capabilities through an HTTP JSON API. Using four minimal product records, this guide walks through the core mental model of full-text search: from how text is tokenized and indexed to how search results are filtered and ranked.
A Minimal Example: Four Sample Products
To make every query result and ranking order verifiable by inspection, we use four sample products throughout this guide:
| Document ID | Product Name name | SKU sku | Category category | Price price |
|---|---|---|---|---|
| 1 | 無線耳機 Wireless Headphones | WH-1000XM5 | audio | 8990 |
| 2 | 無線滑鼠 Wireless Mouse | MS-200 | computer | 990 |
| 3 | 有線耳機 Wired Headphones | HP-100 | audio | 1290 |
| 4 | 耳機收納盒 Headphones Case | CASE-01 | accessory | 390 |
In Elasticsearch, a named collection of documents is called an Index (such as products-primer), roughly equivalent to a table in a relational database. Each individual record is stored as a JSON Document.
From SQL LIKE to the Inverted Index: Retrieval Mechanisms and Semantics
Traditional relational database tables are organized by rows. When executing WHERE name LIKE '%headphones%', the database engine must inspect each row sequentially to check whether the target column contains that substring. Because the query starts with a wildcard %, standard B-tree indexes cannot be used to accelerate the scan.
While PostgreSQL supports GIN indexes, the pg_trgm extension, and native full-text search capabilities, the fundamental difference lies in search semantics and relevance scoring. Substring inclusion merely answers is it present?, whereas full-text search answers how relevant is it? and how should results be ranked?.
At the core of Elasticsearch is the inverted index. At write time, it analyzes and normalizes text into tokens, then builds a mapping from terms to the documents containing them. Using the English names of our four sample products as an example:
Term --> Postings List (Documents)
+------------+------------------------------------+
| wireless | Document 1, Document 2 |
| headphones | Document 1, Document 3, Document 4 |
| mouse | Document 2 |
| case | Document 4 |
+------------+------------------------------------+
When querying wireless headphones, the system no longer scans every document. Instead, it looks up the postings lists for wireless and headphones, performs a set union or intersection based on query criteria, and computes relevance scores. Elasticsearch trades extra processing and disk storage at write time for rich search semantics and low-latency retrieval at query time.
Defining Mappings: Determining Field Search Semantics
Before indexing documents, you define each field’s data type through a mapping. A mapping is also a blueprint for search behavior:
PUT /products-primer
{
"mappings": {
"properties": {
"name": { "type": "text", "analyzer": "standard" },
"sku": { "type": "keyword" },
"category": { "type": "keyword" },
"price": { "type": "integer" }
}
}
}
This mapping highlights two fundamentally different text types:
textfields (such asname): Processed by an analyzer into individual tokens and stored in an inverted index, designed for full-text search and relevance scoring.keywordfields (such asskuandcategory): Stored intact as a single token without tokenization, designed for exact matching, filtering, sorting, and aggregations (similar to SQLGROUP BYoperations).
After creating the index, we ingest the four sample products using the Bulk API.
Text Analysis and Tokenization: Inside the Analyzer
The engine powering full-text search is the analyzer. An analyzer is a pipeline composed of three sequential components: character filters, a tokenizer, and token filters.
The standard text analysis pipeline looks like this:
- Raw Text
- Character FiltersHTML strip, etc.
- TokenizerStandard, etc.
- Token FiltersLowercase, etc.
- Tokens
Testing the text in the name field against the built-in standard analyzer:
POST /products-primer/_analyze
{
"field": "name",
"text": "無線耳機 Wireless Headphones WH-1000XM5"
}
The analyzer output reveals three key behaviors:
- Lowercasing:
WirelessandHeadphonesare normalized towirelessandheadphones. - Punctuation splitting: Hyphens are dropped;
WH-1000XM5is split intowhand1000xm5. - Unigram tokenization for Chinese: By default, Unicode segmentation rules split Chinese characters into single-character tokens (
無,線,耳,機), rather than recognizing compound words like無線(wireless) and耳機(headphones).
By contrast, testing the same string WH-1000XM5 against the keyword-typed sku field produces a single unchanged token: WH-1000XM5.
(Note: Subsequent query examples focus on English terms to illustrate the core mechanics of full-text search. In production, semantic Chinese tokenization requires installing dedicated analyzer plugins like IK or Smart Chinese.)
Common Query Pitfalls: The Essential Difference Between Term and Match
Understanding tokens in the inverted index helps you avoid a classic pitfall that trips up many newcomers.
Pitfall 1: Using a Term Query on a Text Field
POST /products-primer/_search
{
"query": {
"term": { "name": "Wireless" }
}
}
This query should return 0 hits. A term query does not analyze the input string. It looks up the exact string "Wireless" directly in the inverted index, whereas the index only contains the lowercased token "wireless".
The fix: Use a match query for full-text search. The match query processes the input string with the target field’s analyzer, converting "Wireless" to "wireless" before lookup, matching Documents 1 and 2 as expected.
POST /products-primer/_search
{
"query": {
"match": { "name": "Wireless" }
}
}
Likewise, querying match on name with "wire" should return 0 hits. The inverted index contains the token wireless, not wire. Full-text search operates on token matching, not arbitrary substring search.
Exact Matching on Identifiers: Keyword Fields with Term Queries
For keyword fields such as SKUs, statuses, or categories, a term query is the right tool:
POST /products-primer/_search
{
"query": {
"term": { "sku": "WH-1000XM5" }
}
}
This query should return an exact hit on Document 1. Searching for lowercased wh-1000xm5 returns nothing because keyword fields are case-sensitive by default.
Key Differences: Term vs. Match
| Query | Analyzed Query Input? | Matching Target | Target Field Type | Typical Use Case |
|---|---|---|---|---|
term | No (exact literal lookup) | Exact token in the inverted index | keyword, numeric, boolean | SKUs, status codes, category IDs |
match | Yes (via field’s analyzer) | Individual analyzed tokens | text | Product titles, article bodies, full-text search |
Compound Queries: Relevance Scoring and Structured Filtering
Production search interfaces usually blend full-text keywords with structured criteria, such as categories or price ceilings.
Default Boolean Logic in Match Queries
When given multiple terms, the match query defaults to OR logic:
POST /products-primer/_search
{
"query": {
"match": { "name": "wireless headphones" }
}
}
This query should match all four products: Document 1 contains both, Document 2 contains wireless, and Documents 3 and 4 contain headphones. If your business logic requires both terms to be present, specify operator: "and" explicitly:
POST /products-primer/_search
{
"query": {
"match": {
"name": {
"query": "wireless headphones",
"operator": "and"
}
}
}
}
With and configured, the search results should narrow down to Document 1 only.
The Bool Query: Separating Scoring from Filtering
Suppose the requirement is: find products whose name contains headphones, restricted to the audio category, and priced at or below 2,000:
POST /products-primer/_search
{
"query": {
"bool": {
"must": [
{ "match": { "name": "headphones" } }
],
"filter": [
{ "term": { "category": "audio" } },
{ "range": { "price": { "lte": 2000 } } }
]
}
}
}
This bool query cleanly separates two execution contexts:
mustclause: Each clause must match and contribute to relevance scoring (_score), determining the ranking order of the results.filterclause: Evaluates each document on a simple yes/no basis and does not affect scoring. Elasticsearch automatically caches frequently used filters to accelerate subsequent queries.
Under these conditions, Document 2 fails the must clause because its name lacks headphones; Document 1 is excluded because its price exceeds 2,000; Document 4 is filtered out because its category is accessory. In the end, only Document 3 should match.
Search Ranking Mechanics: Three Intuitions Behind BM25 Scoring
Unless explicit sort criteria are specified, Elasticsearch ranks results by _score (relevance score). The default scoring algorithm is BM25, and understanding it comes down to three key intuitions:
- Inverse Document Frequency (IDF): The fewer documents a term appears in across the entire index, the rarer and more discriminative it is, earning a higher score boost for documents containing it. In our four-document example,
wirelessappears in only 2 documents, making it more discriminative thanheadphones, which appears in 3. - Term Frequency (TF): The more frequently a term appears in a single document, the more relevant that document is considered. BM25 imposes a saturation ceiling, however, so 10 occurrences won’t score 10 times higher than a single occurrence.
- Field Length Normalization: When a short title and a long article contain the same keyword, the short title generally represents a tighter topical focus. Hits in shorter fields are weighted more heavily.
Now query wireless headphones in default OR mode and apply these three intuitions. Document 1 should score the highest by matching both terms; Document 2, while matching only one term, may receive a strong IDF boost from the rarer term wireless; Documents 3 and 4 then have their relative order decided by field length normalization and term weights.
NOTE
The _score is strictly meant for relative ranking within a single query execution. It has no absolute numerical significance, cannot be compared across different queries, and does not represent a percentage score.
Practical System Architecture and Technology Boundaries
When integrating Elasticsearch into an existing system, three operational concepts are worth clarifying up front:
1. Near Real-Time Latency After Writes
Elasticsearch provides near real-time (NRT) search. After a document is written, a refresh operation must occur to flush in-memory buffers into searchable Lucene segments (immutable internal index segments, generated automatically once per second by default on indices that received a search request in the last 30 seconds).
A successful write API response only guarantees that data is safely recorded in the transaction log (translog); it does not mean the document is immediately queryable via _search. Production environments should plan around this second-level delay. Avoid forcing refresh=true on every write request, which creates many tiny segments and degrades indexing throughput.
2. Separating the Source of Truth from the Search Model
In a typical backend architecture, a relational database like PostgreSQL serves as the single source of truth (SSOT), handling transactional consistency, inventory deductions, and order states. Elasticsearch serves strictly as a read-only search model (read model).
A standard data-flow architecture looks like this:
[ Client ] --Write--> [ PostgreSQL (SSOT) ]
| |
| v Sync (CDC/Worker)
[ Elasticsearch ]
| ^
| |
+--------- Search Query ---+
Introducing a dedicated search engine means taking on the operational overhead of synchronization pipelines (such as Change Data Capture or async task queues), network latency, retry logic, and data reconciliation.
3. Architectural Decision: Stay with an RDBMS or Adopt Elasticsearch?
| Decision Factor | Stay with Relational DB (PostgreSQL) | Adopt Dedicated Search Engine (Elasticsearch) |
|---|---|---|
| Query Requirements | Exact lookups, prefix matching, simple substring scans | Multi-term semantic relevance, flexible tokenization, custom boosting |
| Ranking Mechanism | Structured column sorting (timestamp, price, ID) | BM25 relevance scoring (_score) and hybrid ranking |
| Architectural Overhead | Single database; zero sync latency or reconciliation overhead | Must operate sync pipelines (CDC/queues) and handle second-level delay |