Elasticsearch Primer: From Raw Text to Ranked Search Results

In a relational database, reaching for WHERE name LIKE '%headphones%' is often the most intuitive first step when searching for products containing “headphones”. For systems with modest datasets and straightforward queries, simple substring matching gets the job done.

Real-world search, however, quickly outgrows basic substring matching. When users type wireless headphones, they typically expect items matching both terms to appear at the very top, followed by items matching only one. Nor do they expect search results to fail completely just because of differing letter cases.

Elasticsearch is a search and analytics engine built specifically to solve these kinds of problems. It runs on top of Apache Lucene—the open-source full-text search library that serves as its underlying engine—and exposes its capabilities through an HTTP JSON API. Using four minimal product records, this guide walks through the core mental model of full-text search: from how text is tokenized and indexed to how search results are filtered and ranked.


A Minimal Example: Four Sample Products

To make every query result and ranking order verifiable by inspection, we use four sample products throughout this guide:

Document IDProduct Name nameSKU skuCategory categoryPrice price
1無線耳機 Wireless HeadphonesWH-1000XM5audio8990
2無線滑鼠 Wireless MouseMS-200computer990
3有線耳機 Wired HeadphonesHP-100audio1290
4耳機收納盒 Headphones CaseCASE-01accessory390

In Elasticsearch, a named collection of documents is called an Index (such as products-primer), roughly equivalent to a table in a relational database. Each individual record is stored as a JSON Document.


From SQL LIKE to the Inverted Index: Retrieval Mechanisms and Semantics

Traditional relational database tables are organized by rows. When executing WHERE name LIKE '%headphones%', the database engine must inspect each row sequentially to check whether the target column contains that substring. Because the query starts with a wildcard %, standard B-tree indexes cannot be used to accelerate the scan.

While PostgreSQL supports GIN indexes, the pg_trgm extension, and native full-text search capabilities, the fundamental difference lies in search semantics and relevance scoring. Substring inclusion merely answers is it present?, whereas full-text search answers how relevant is it? and how should results be ranked?.

At the core of Elasticsearch is the inverted index. At write time, it analyzes and normalizes text into tokens, then builds a mapping from terms to the documents containing them. Using the English names of our four sample products as an example:

 Term --> Postings List (Documents)

 +------------+------------------------------------+
 | wireless   | Document 1, Document 2             |
 | headphones | Document 1, Document 3, Document 4 |
 | mouse      | Document 2                         |
 | case       | Document 4                         |
 +------------+------------------------------------+

When querying wireless headphones, the system no longer scans every document. Instead, it looks up the postings lists for wireless and headphones, performs a set union or intersection based on query criteria, and computes relevance scores. Elasticsearch trades extra processing and disk storage at write time for rich search semantics and low-latency retrieval at query time.


Defining Mappings: Determining Field Search Semantics

Before indexing documents, you define each field’s data type through a mapping. A mapping is also a blueprint for search behavior:

PUT /products-primer
{
  "mappings": {
    "properties": {
      "name": { "type": "text", "analyzer": "standard" },
      "sku": { "type": "keyword" },
      "category": { "type": "keyword" },
      "price": { "type": "integer" }
    }
  }
}

This mapping highlights two fundamentally different text types:

After creating the index, we ingest the four sample products using the Bulk API.


Text Analysis and Tokenization: Inside the Analyzer

The engine powering full-text search is the analyzer. An analyzer is a pipeline composed of three sequential components: character filters, a tokenizer, and token filters.

The standard text analysis pipeline looks like this:

  1. Raw Text
  2. Character FiltersHTML strip, etc.
  3. TokenizerStandard, etc.
  4. Token FiltersLowercase, etc.
  5. Tokens

Testing the text in the name field against the built-in standard analyzer:

POST /products-primer/_analyze
{
  "field": "name",
  "text": "無線耳機 Wireless Headphones WH-1000XM5"
}

The analyzer output reveals three key behaviors:

  1. Lowercasing: Wireless and Headphones are normalized to wireless and headphones.
  2. Punctuation splitting: Hyphens are dropped; WH-1000XM5 is split into wh and 1000xm5.
  3. Unigram tokenization for Chinese: By default, Unicode segmentation rules split Chinese characters into single-character tokens (無, 線, 耳, 機), rather than recognizing compound words like 無線 (wireless) and 耳機 (headphones).

By contrast, testing the same string WH-1000XM5 against the keyword-typed sku field produces a single unchanged token: WH-1000XM5.

(Note: Subsequent query examples focus on English terms to illustrate the core mechanics of full-text search. In production, semantic Chinese tokenization requires installing dedicated analyzer plugins like IK or Smart Chinese.)


Common Query Pitfalls: The Essential Difference Between Term and Match

Understanding tokens in the inverted index helps you avoid a classic pitfall that trips up many newcomers.

Pitfall 1: Using a Term Query on a Text Field

POST /products-primer/_search
{
  "query": {
    "term": { "name": "Wireless" }
  }
}

This query should return 0 hits. A term query does not analyze the input string. It looks up the exact string "Wireless" directly in the inverted index, whereas the index only contains the lowercased token "wireless".

The fix: Use a match query for full-text search. The match query processes the input string with the target field’s analyzer, converting "Wireless" to "wireless" before lookup, matching Documents 1 and 2 as expected.

POST /products-primer/_search
{
  "query": {
    "match": { "name": "Wireless" }
  }
}

Likewise, querying match on name with "wire" should return 0 hits. The inverted index contains the token wireless, not wire. Full-text search operates on token matching, not arbitrary substring search.

Exact Matching on Identifiers: Keyword Fields with Term Queries

For keyword fields such as SKUs, statuses, or categories, a term query is the right tool:

POST /products-primer/_search
{
  "query": {
    "term": { "sku": "WH-1000XM5" }
  }
}

This query should return an exact hit on Document 1. Searching for lowercased wh-1000xm5 returns nothing because keyword fields are case-sensitive by default.

Key Differences: Term vs. Match

QueryAnalyzed Query Input?Matching TargetTarget Field TypeTypical Use Case
termNo (exact literal lookup)Exact token in the inverted indexkeyword, numeric, booleanSKUs, status codes, category IDs
matchYes (via field’s analyzer)Individual analyzed tokenstextProduct titles, article bodies, full-text search

Compound Queries: Relevance Scoring and Structured Filtering

Production search interfaces usually blend full-text keywords with structured criteria, such as categories or price ceilings.

Default Boolean Logic in Match Queries

When given multiple terms, the match query defaults to OR logic:

POST /products-primer/_search
{
  "query": {
    "match": { "name": "wireless headphones" }
  }
}

This query should match all four products: Document 1 contains both, Document 2 contains wireless, and Documents 3 and 4 contain headphones. If your business logic requires both terms to be present, specify operator: "and" explicitly:

POST /products-primer/_search
{
  "query": {
    "match": {
      "name": {
        "query": "wireless headphones",
        "operator": "and"
      }
    }
  }
}

With and configured, the search results should narrow down to Document 1 only.

The Bool Query: Separating Scoring from Filtering

Suppose the requirement is: find products whose name contains headphones, restricted to the audio category, and priced at or below 2,000:

POST /products-primer/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "name": "headphones" } }
      ],
      "filter": [
        { "term": { "category": "audio" } },
        { "range": { "price": { "lte": 2000 } } }
      ]
    }
  }
}

This bool query cleanly separates two execution contexts:

Under these conditions, Document 2 fails the must clause because its name lacks headphones; Document 1 is excluded because its price exceeds 2,000; Document 4 is filtered out because its category is accessory. In the end, only Document 3 should match.


Search Ranking Mechanics: Three Intuitions Behind BM25 Scoring

Unless explicit sort criteria are specified, Elasticsearch ranks results by _score (relevance score). The default scoring algorithm is BM25, and understanding it comes down to three key intuitions:

  1. Inverse Document Frequency (IDF): The fewer documents a term appears in across the entire index, the rarer and more discriminative it is, earning a higher score boost for documents containing it. In our four-document example, wireless appears in only 2 documents, making it more discriminative than headphones, which appears in 3.
  2. Term Frequency (TF): The more frequently a term appears in a single document, the more relevant that document is considered. BM25 imposes a saturation ceiling, however, so 10 occurrences won’t score 10 times higher than a single occurrence.
  3. Field Length Normalization: When a short title and a long article contain the same keyword, the short title generally represents a tighter topical focus. Hits in shorter fields are weighted more heavily.

Now query wireless headphones in default OR mode and apply these three intuitions. Document 1 should score the highest by matching both terms; Document 2, while matching only one term, may receive a strong IDF boost from the rarer term wireless; Documents 3 and 4 then have their relative order decided by field length normalization and term weights.

NOTE

The _score is strictly meant for relative ranking within a single query execution. It has no absolute numerical significance, cannot be compared across different queries, and does not represent a percentage score.


Practical System Architecture and Technology Boundaries

When integrating Elasticsearch into an existing system, three operational concepts are worth clarifying up front:

1. Near Real-Time Latency After Writes

Elasticsearch provides near real-time (NRT) search. After a document is written, a refresh operation must occur to flush in-memory buffers into searchable Lucene segments (immutable internal index segments, generated automatically once per second by default on indices that received a search request in the last 30 seconds).

A successful write API response only guarantees that data is safely recorded in the transaction log (translog); it does not mean the document is immediately queryable via _search. Production environments should plan around this second-level delay. Avoid forcing refresh=true on every write request, which creates many tiny segments and degrades indexing throughput.

2. Separating the Source of Truth from the Search Model

In a typical backend architecture, a relational database like PostgreSQL serves as the single source of truth (SSOT), handling transactional consistency, inventory deductions, and order states. Elasticsearch serves strictly as a read-only search model (read model).

A standard data-flow architecture looks like this:

 [ Client ] --Write--> [ PostgreSQL (SSOT) ]
     |                          |
     |                          v  Sync (CDC/Worker)
                        [ Elasticsearch ]
     |                          ^
     |                          |
     +--------- Search Query ---+

Introducing a dedicated search engine means taking on the operational overhead of synchronization pipelines (such as Change Data Capture or async task queues), network latency, retry logic, and data reconciliation.

3. Architectural Decision: Stay with an RDBMS or Adopt Elasticsearch?

Decision FactorStay with Relational DB (PostgreSQL)Adopt Dedicated Search Engine (Elasticsearch)
Query RequirementsExact lookups, prefix matching, simple substring scansMulti-term semantic relevance, flexible tokenization, custom boosting
Ranking MechanismStructured column sorting (timestamp, price, ID)BM25 relevance scoring (_score) and hybrid ranking
Architectural OverheadSingle database; zero sync latency or reconciliation overheadMust operate sync pipelines (CDC/queues) and handle second-level delay