System Design: Search and Indexing

Beyond text score

Production ranking mixes the text score with field boosts (a title match counts more than a body match), freshness, popularity, personalisation and business rules such as promoted or out-of-stock items. A common design is two stages: a cheap first stage retrieves a few hundred candidates, and a heavier re-ranker orders only those.

How a search cluster is built

Elasticsearch and OpenSearch are widely used examples; the architecture below is common to Lucene-based systems (Solr is another), while engines on other foundations differ in the details.

• A document is a JSON-like record. An index (in the search-engine sense) is a collection of documents with a shared mapping, a bit like a table.
• An index is split into shards. Each shard is a self-contained search index holding a subset of the documents, placed by hashing the document id. This is the same horizontal split described in sharding.
• Each shard has one or more replicas: copies on other nodes. They provide high availability and extra read capacity, following the ideas in replication.
• A query goes to a coordinating node, which fans it out to one copy of every shard, each shard returns its local top results, and the coordinator merges them into a global top list. This is called scatter-gather.

Inside a shard, the index is made of immutable segments. New documents go into a small in-memory buffer that is periodically written as a new segment; deletes mark documents rather than removing them; background merging combines segments and drops deleted documents. This is the same append-and-merge idea you will meet in storage engines. It has a visible consequence: search is near real time. A newly indexed document usually becomes searchable after a short refresh interval (often about a second by default), not instantly.

Design points to remember:

• Shard count is hard to change later. Too few limits parallelism; too many wastes overhead. Plan from expected data size.
• Scatter-gather latency follows the slowest shard.
• Deep pagination is costly. Asking for results 100,000 to 100,010 forces every shard to produce and sort 100,010 candidates. Use cursor-style approaches such as search-after instead of large offsets.
• Search is not a system of record. Treat it as a derived, rebuildable view of data held elsewhere.

Keeping the index in sync

Your source of truth is usually a relational or document database. The search index is a copy shaped for queries, so it must be updated whenever the source changes. How you do that matters a lot.

Dual writes and why they bite

The obvious approach: in the application, write to the database, then write to the search index. If the first write succeeds and the second fails (network blip, cluster down, crash), the two stores disagree and nothing records that fact. Reversing the order swaps which side is wrong, and transactions across both are rarely available. Even without failures, two concurrent updates can reach the stores in opposite orders and leave the index permanently different.

Safer options

This decision flowchart picks a sync approach.

How to read it: every path ends at a rebuild path. Dual writes do not appear because no branch of the decision justifies them. The target shape of the recommended CDC route looks like this.

Whichever you choose, make index updates idempotent: indexing the same document version twice should be harmless. Use the document id as the key and a version (or the database's change sequence) so that an old event arriving late cannot overwrite a newer one. That gives you at-least-once delivery with correct results; see message queues.

Typeahead and autocomplete

Typeahead suggests completions as the user types, so suggestions must usually arrive within tens of milliseconds, as often as every keystroke.

A classic structure is the trie (prefix tree). Each node represents a character; a path from the root spells a prefix. Typing "ap" walks two nodes, and the subtree below holds all completions such as "apple" and "apricot".

To return only the best suggestions, each node can store its top few completions by popularity, precomputed, so a lookup is a walk of prefix-length steps plus a short list read.

Practical notes:

• Source and freshness. Suggestions come from popular past queries, product names or titles, ranked by frequency and recency, filtered for bad entries, and rebuilt hourly or daily rather than on every search.
• Serving. Prefix results are highly cacheable, and short prefixes ("a") are extremely hot, so cache them aggressively. On the client, debounce keystrokes and cancel stale requests.
• Alternatives. Search engines offer built-in prefix or "edge n-gram" analysis that indexes prefixes of words as terms. It is simpler to operate, at the cost of a larger index.

Vector search in one section

Keyword search matches words. If a user searches "cheap place to sleep" and your document says "budget hotel", keyword matching fails unless you maintain synonyms. Vector search matches meaning.

The idea: a machine learned embedding model converts text (or images, audio) into a list of numbers, a vector, positioned so that similar meanings land close together. You embed every document at index time and store the vectors. At query time you embed the query with the same model and find the nearest neighbours by a distance measure such as cosine similarity.

Exact nearest neighbour search compares the query to every vector, which is too slow at scale. Systems use approximate nearest neighbour (ANN) indexes, such as graph-based HNSW or clustering-based approaches, which trade a little recall (sometimes missing a true neighbour) for large speed gains.

Many production systems use hybrid search: run keyword and vector retrieval, then merge the two ranked lists (rank fusion is a common technique) or re-rank. This keeps exact matches reliable while adding semantic recall, and it is also the retrieval step in retrieval-augmented generation. Vector support exists in dedicated vector databases and in general search engines and databases; choose by data volume, filtering needs and the stack you already run.

In practice: marketplace search with filters and typeahead

The situation

A second-hand goods marketplace has about 2 million active listings from 15,000 sellers (assumed). Buyers search by words and filter by category, price range, condition and distance. A team of ten runs a Spring Boot application on Kubernetes with PostgreSQL as the system of record, Redis for caching, Kafka already in use for order events, and Terraform for infrastructure.

Search is currently ILIKE queries plus a handful of B-tree indexes on the primary database. Search p95 is 2.4 seconds at the evening peak of about 300 searches per second, and those scans compete with checkout writes (all numbers assumed). Facet counts ("Used, 214 results") are computed with extra aggregate queries. There is no typeahead. The business wants suggestions within 100 ms and a sold item gone from results within seconds.

Starting architecture

How to read it: the same database serves checkout, listing edits and every search scan, so one popular search trend slows the money path.

Options considered

The decision

Rollout and verification

Build the first index from a database snapshot while the connector buffers changes in Kafka, then catch up. Run shadow traffic: copy real queries to OpenSearch, discard the answers to users, and compare latency and result overlap with the PostgreSQL path. Build a set of judged queries (the top few hundred by volume, each with expected good results) and track how rank changes affect them. Then enable the new path for 5 percent of searches behind a flag, keeping the old path as the rollback.

Dashboards that prove it worked: end-to-end index lag (database commit time to searchable), Kafka consumer lag for the indexer, search p95 and p99, zero-result rate, indexing error and dead-letter counts, primary-database CPU during the evening peak, and node disk and heap pressure. Schedule a nightly reconciliation job that compares counts and the highest version per seller between PostgreSQL and the index.

The 2 a.m. failure

The page: "Index lag above 5 minutes for 10 minutes". Sold items are still appearing in results. The cause is a poison message: a seller-defined attribute that the index maps as a number arrives holding text, the index rejects the document with a mapping error, and the indexer retries the same record forever without committing its offset, blocking its partition.

First three checks:

1. kafka-consumer-groups.sh --bootstrap-server <broker> --describe --group listing-indexer to see which partition is stuck and how far behind it is.
2. kubectl logs deploy/listing-indexer --since=15m | grep -i error to find the failing document id and reason.
3. curl -s "<opensearch-host>/_cluster/health?pretty" to rule out a cluster problem.

Mitigation: move the poison record to a dead-letter topic so the partition drains, then fix the data and replay it. Lag clears as the backlog is indexed. Follow-ups: validate and coerce fields in the indexer (CDC events come straight from the database log, so there is no producer to fix), send records that the index rejects to a dead-letter topic after a capped number of retries, alert on dead-letter growth, and let the reconciliation job repair any divergence. See message queues for retries and dead-letter queues.

Architect's notes

• Measure first. Before choosing a cluster size, record real query shapes, filter usage, result-set sizes and peak queries per second. Many "search" problems are one missing index.
• Ask what staleness the business can accept per field. Price and availability tolerate far less lag than a description edit; that decides whether you need a read-through check at checkout.
• The mapping is a contract. Treat index mappings like a database schema: reviewed, versioned and tested, with a rehearsed alias-swap reindex.
• Ownership. Decide who is paged for index lag: the search team, or the team that owns the PostgreSQL schema. Lag caused by schema drift needs both.
• Cost trap. Replicas and long retention of old index versions double storage quickly; delete the previous index only after the rollback window ends.
• Defer vector search. Ship keyword relevance and typeahead first, measure zero-result queries, and add semantic retrieval only when you can show queries that keywords miss.

Common mistakes

• Using a search index as the primary database. Indexes are near real time, can lose or lag updates, and are shaped for queries. Keep a system of record and treat search as rebuildable.
• Dual-writing application code to database and index. One failure leaves them permanently inconsistent. Use CDC or an outbox with idempotent, versioned updates.
• Different analysers at index and query time. Terms fail to match. Test analysis with real examples before launch.
• Choosing shard counts blindly. Hundreds of tiny shards waste memory; too few cannot scale. Size from data volume and plan reindexing.
• Deep pagination with large offsets. Cost grows with the offset on every shard. Use cursor-style pagination and limit depth.
• Ignoring relevance testing. Tuning boosts by gut feeling regresses quality. Keep a set of judged queries and measure changes.
• Adding vector search without a rebuild plan. Changing the embedding model invalidates all stored vectors, so budget for re-embedding.

Key takeaways

• Inverted indexes turn search into lookup. Terms map to sorted lists of document ids, which are intersected for multi-term queries.
• Analysis decides what matches. Tokenising, lowercasing, stemming and synonyms must be consistent between indexing and querying.
• BM25 ranks by rarity, saturating frequency and length. Combine it with boosts and business signals, often in a two-stage retrieve-then-rerank design.
• Shards scale data, replicas scale reads and availability. Queries scatter to shards and gather results; the slowest shard sets latency.
• Search is a derived view. Keep the system of record elsewhere and make the index rebuildable.
• Use CDC or an outbox, not dual writes. Make updates idempotent and versioned.
• Typeahead is a prefix problem. Tries with precomputed top suggestions, aggressive caching and client debouncing keep it fast.
• Vector search finds meaning, keyword search finds exactness. Hybrid retrieval often gives the best results.

Practice exercises

Exercise 1 — Starter (30 min): Build the inverted index by hand for these four documents: "fast red car", "slow red bus", "fast blue car", "red car wash". Write the postings lists, then answer the queries "red car" (AND) and "fast OR slow", showing each intersection or union.

Exercise 2 — Intermediate (2 hours): Design the search for an online shop with 2 million products (assumed). Choose fields and which are analysed versus exact, decide boosts, choose a shard and replica count with reasoning from an assumed index size, and draw the sync path from the product database using CDC. List what happens to the index if the consumer is down for an hour and how it catches up.

Exercise 3 — Advanced (half day): Design a typeahead service for 20,000 suggestion requests per second (assumed) at peak. Specify the data structure, how top suggestions are computed and refreshed from query logs, caching layers with expected hit behaviour for short versus long prefixes, how you filter bad suggestions, and a degradation plan if the suggestion service is slow. Then extend the design with a hybrid keyword and vector search path for the full results page and describe how you would evaluate whether it improved relevance.

Related reading

• Sharding — the data-splitting ideas behind search shards.
• Replication — why search shards have replicas and what lag means.
• Storage engines — immutable segments and merging, the same idea as LSM trees.
• Message queues — the transport between a CDC stream and the index writer.
• Caching basics — caching hot prefixes and popular queries.