Vector databases and vector search are the data systems behind many forms of semantic search, similarity search, recommendations and retrieval-augmented generation (RAG). They store or index high-dimensional embeddings—numeric representations produced by machine-learning models—and retrieve records whose vectors are close under a chosen distance or similarity measure such as cosine distance, dot product or Euclidean distance. When collections become large, systems commonly use approximate nearest-neighbour (ANN) indexes such as HNSW or IVF-style structures to trade some retrieval recall for lower latency and higher throughput.

A production vector database is more than an embedding column plus a nearest-neighbour function. It must coordinate metadata filtering, namespaces or tenant boundaries, updates and deletes, index construction, memory and storage, exact-versus-approximate search, hybrid search that combines vectors with keyword methods such as BM25, second-stage reranking, and evaluation for recall, relevance, latency and cost. In RAG systems, those operational choices determine which evidence is even available to the language model before generation begins.

This guide explains vector search as one complete data-management problem: document or object → embedding model → vector record → similarity metric → exact baseline → ANN index → metadata filter → hybrid candidate retrieval → reranker → RAG or application consumer → update/delete/version lifecycle → quality evaluation → recovery. The goal is not to crown one vector database product. It is to understand which information each layer preserves, which trade-offs it introduces, how HNSW and filtering interact, why hybrid retrieval often outperforms pure semantic search for identifiers and exact terminology, and how to prove that a fast vector search system is still returning the right evidence.

ARTICLE ID: DATA.MANAGEMENT.066
Canonical function: design, operation and evaluation of vector-search data systems that store/index embeddings and combine similarity retrieval with structured metadata, lexical search and downstream reranking
Owner boundary: Retrieval Representation owns how queries and documents become sparse, dense and hybrid search signals; Data Indexing and Query Optimisation owns general database access-path planning; AI Data Management owns the broader AI-data lifecycle. This article owns the vector-search storage/index/retrieval system itself.

Start here: read One question, many retrieval routes for the system model, Exact search before approximate search for the baseline discipline, HNSW from first principles for the main ANN mechanism, Filtering changes the graph problem for metadata constraints, or RAG is downstream of retrieval for the application consequence.

Concrete engine examples are source-bounded. Current pgvector documentation distinguishes exact nearest-neighbour search from approximate HNSW and IVFFlat indexes and recommends monitoring ANN recall against an exact baseline. Current Weaviate and Qdrant documentation describe filtered HNSW strategies; Databricks AI Search documents ANN, full-text, hybrid retrieval and reranking. These examples illustrate mechanisms rather than universal performance guarantees or a vendor ranking.

1. One question can travel through several retrieval systems

Imagine a support engineer searching a technical knowledge base for “error 4817 after certificate rotation”. The system contains 20 million chunks from product manuals, incident reports, release notes and internal runbooks. Some documents use the exact code 4817. Others describe the underlying certificate problem without that number. Still others mention certificate rotation but concern a different failure.

A traditional keyword engine can excel at the exact identifier. A dense embedding search can surface semantically related passages even when the wording differs. A metadata filter may need to exclude obsolete product versions, restrict the tenant, keep only English documents, or enforce an access-control boundary. A reranker can examine the top candidates more deeply and reorder them. A RAG application can then give the resulting evidence to a language model.

Those stages are easy to collapse into one phrase—“vector search”—but they solve different problems. The embedding model defines representation. The vector index narrows candidates. The filter defines eligibility. Lexical retrieval captures exact terms. Fusion combines result sets. Reranking spends more computation on a smaller candidate pool. Generation consumes whatever survived.

If the final answer is wrong, diagnosis should ask which layer failed. Did the relevant document lack an embedding? Did its embedding version differ from the query vector? Did ANN miss it? Did a restrictive filter remove it? Did keyword search find it but fusion push it down? Did the reranker misorder it? Did the generator ignore correct evidence?

A vector database becomes useful when it makes these distinctions operationally visible. It should know which records exist, which vector fields and dimensions they carry, which metadata fields are filterable, which index/version served the query, and how updates, deletes and rebuilds propagate.

The reader job is therefore broader than “how do I store embeddings?” A durable vector-search system must answer four questions at once:

• Representation: what does this vector encode and under which model/version?
• Eligibility: which records are allowed to compete for this query?
• Approximation: how much search work can be skipped without losing too much recall?
• Evidence: how do we know the chosen retrieval configuration works for the real workload?

This article builds those answers in that order. The result is a data-management view of vector search rather than a list of product features.

2. An embedding is a coordinate system learned for a task, not meaning itself

An embedding maps an object into a numeric vector. Text, images, audio, products, users, code and graph entities can all be represented this way. The vector is useful because geometry in the learned space often corresponds to relationships the model was trained to preserve.

Two passages can have similar vectors because their language or concepts are related. Two product images can be close because their visual properties resemble each other. A query vector can be compared with document vectors to produce candidate evidence.

But an embedding is not a universal semantic truth. Change the model and the coordinate system changes. Change the pooling method, prompt template, input language, normalisation or model revision and the vectors may no longer be directly comparable.

This creates the first vector-database invariant:

Vectors that are compared must belong to a compatible representation contract.

That contract should identify at least:

• embedding model and version;
• vector dimensionality;
• input preprocessing or prompt instructions where material;
• normalisation policy;
• distance/similarity metric;
• source object/version;
• creation time or embedding edition;
• language/modality scope where relevant.

Suppose document vectors come from model A with 1,536 dimensions and a new query service switches to model B with 3,072 dimensions. The incompatibility is obvious because dimensions differ. A more dangerous migration keeps the same dimension but changes the embedding geometry. The database can still compute cosine similarity. The numerical operation succeeds. The ranking may be meaningless.

Version vectors with the same seriousness as schemas. A vector field named embedding is insufficient if its model changes silently. Use versioned columns, named vectors, collection versions or explicit embedding metadata so old and new representations cannot mix accidentally.

Embeddings can also preserve biases, source errors and sensitive information. A vector is derived data, not an exemption from governance. Treat storage, access, deletion and provenance accordingly.

Similarity is also query-dependent. A general-purpose embedding can perform well on ordinary semantic paraphrases and poorly on exact identifiers, rare names, numerical constraints or specialised jargon. This is one reason hybrid retrieval remains valuable.

The correct operational question is therefore not “Are embeddings semantic?” It is “Which relationships did this embedding model learn strongly enough for this retrieval job, and what evidence do we have on our own queries?”

3. A vector database and a vector index are related but not identical

A vector index is an access structure used to search vectors efficiently. An HNSW graph or an IVF partitioning structure is an index. A vector database or vector-search data system surrounds that index with record identity, persistence, metadata, updates, filters, tenancy, APIs, durability, replication, monitoring and lifecycle behaviour.

This distinction matters because teams sometimes compare an HNSW library with a managed vector service as if both objects solve the same operational job. The library can be excellent at nearest-neighbour search while leaving persistence, filtering, replication and recovery to the application. A managed system can provide those controls but impose different cost, latency or consistency trade-offs.

Vector capability can also live inside a general database. pgvector adds vector types and exact/ANN search to PostgreSQL. Databricks exposes vector/AI search integrated with governed tables. Other systems are purpose-built around vector-first storage. The label “vector database” does not determine architecture by itself.

A useful decomposition is:

• record store: IDs, vectors, payload/metadata and source references;
• vector index: HNSW, IVF-style, flat/exact or another ANN structure;
• scalar/filter index: structures supporting metadata constraints;
• query coordinator: vectorisation, filter planning, candidate retrieval and pagination;
• fusion/reranking layer: hybrid lexical/vector combination and second-stage ranking;
• lifecycle layer: upserts, deletes, compaction, snapshots, replication and rebuilds;
• governance layer: tenancy, permissions, lineage and audit.

Different products collapse or separate these layers differently. That is acceptable if the user-facing contract remains clear.

For small datasets, exact vector search inside an existing database may be enough. For a few thousand records, adding an ANN system can increase complexity without a meaningful latency benefit. For hundreds of millions of vectors with strict latency and filtering requirements, index design and distributed operation become central.

The threshold is workload-specific. Do not choose a vector database because the application uses embeddings. Choose it because the retrieval, filtering, scale, latency, lifecycle and governance job exceeds what the simpler existing system can satisfy economically.

4. Similarity metrics define what “nearest” means

Nearest-neighbour search requires a way to compare vectors. Common choices include Euclidean/L2 distance, cosine similarity or distance, and inner/dot product.

For vectors x and y, Euclidean distance measures straight-line distance:

d(x,y) = √Σ(xᵢ – yᵢ)²

Cosine similarity compares direction:

cos(x,y) = (x · y) / (||x|| ||y||)

Dot product is:

x · y = Σ xᵢyᵢ

If vectors are normalised to unit length, cosine similarity and dot product induce the same ordering. For unit vectors, squared Euclidean distance is also monotonically related to cosine similarity:

||x-y||² = 2 – 2cos(x,y)

Therefore several metrics can produce identical rankings under normalisation, but not necessarily identical raw scores.

This matters operationally. A model may have been trained for cosine similarity, dot product or another geometry. Selecting a distance because it is the database default can degrade retrieval.

Do not compare similarity-score thresholds across metrics casually. A cosine similarity of 0.8 and an L2 distance of 0.8 do not represent the same relationship. Scores can also shift after an embedding-model migration even when relevance remains similar.

The vector store should bind the index to the intended metric. pgvector, for example, creates different operator classes for L2, inner product and cosine indexes. That concrete implementation illustrates the broader rule: index geometry and query geometry must agree.

Distance is also not probability. A nearest result can still be irrelevant if the query lies far from every indexed item. Always consider score distributions, nearest-neighbour quality and minimum-quality logic where applications require it.

Thresholding alone can be unstable across queries. Some queries naturally have a strong nearest match; others have several moderate matches. A reranker or downstream relevance model can often make better use of the top candidate set than a universal distance cutoff.

5. Start with exact nearest-neighbour search before trusting ANN

Exact k-nearest-neighbour (kNN) search compares the query vector with every eligible vector, computes distance/similarity, and returns the best k. It is conceptually simple and, for the indexed population, provides the correct nearest neighbours under the chosen metric.

pgvector documents exact nearest-neighbour search as its default and explicitly notes that approximate indexes trade recall for speed. That distinction makes exact search a powerful evaluation baseline. pgvector documentation.

Suppose a collection has one million vectors of 768 dimensions. A brute-force query can require roughly 768 million component-level comparisons before overhead and optimisation, although vectorised CPU/GPU implementations can process this efficiently. At ten thousand vectors, exact search may be entirely practical. At a billion vectors and high query throughput, scanning all candidates for every request becomes expensive.

Before adding approximation, measure the exact system on representative workloads. It gives three valuable things:

1. a relevance baseline under the embedding model and metric;
2. a recall oracle for evaluating ANN indexes;
3. a way to distinguish representation failure from index failure.

If exact search cannot retrieve relevant results for the query set, changing HNSW parameters will not repair the embedding model or chunking strategy.

Exact search is also useful for selective filters. If a metadata predicate reduces ten million vectors to 2,000 eligible records, scanning those 2,000 exactly can be cheaper and more reliable than traversing a global ANN graph.

This is why filtered vector search needs a planner-like mentality. The system should choose between exact scan, filtered ANN, graph traversal or another path based on candidate population and operational cost.

Use exact search in evaluation even when production uses ANN. A sampled offline job can compare approximate top-k with exact top-k and monitor recall through index changes, deletions and growth.

For each query q, one simple recall-at-k definition is:

Recall@k = |ANNₖ(q) ∩ Exactₖ(q)| / k

If exact top-10 contains ten reference neighbours and ANN returns eight of them, Recall@10 is 0.8. That metric measures approximation relative to exact vector geometry, not human relevance. Both layers should be evaluated separately.

The discipline is important: first prove the representation and metric can retrieve useful candidates exactly; then measure how much quality is lost when approximation is introduced to meet latency or scale requirements.

6. Approximate nearest-neighbour search is a controlled loss of search work

Exact search asks every eligible vector how close it is to the query. Approximate nearest-neighbour search tries to avoid most of those comparisons while still returning most of the neighbours that exact search would have returned.

The word approximate describes retrieval completeness under the vector metric. It does not mean the stored vectors are approximate, and it does not mean the final application result is necessarily poor. A well-tuned ANN system can deliver very high recall with dramatically lower latency than brute force on large collections.

The trade-off has several dimensions:

• recall: how many true nearest neighbours are recovered;
• latency: how long one query takes;
• throughput: how many queries can be served per unit time;
• memory: how much index structure must remain in RAM or fast storage;
• build cost: time and compute required to construct or update the index;
• update behaviour: how quickly new vectors become searchable;
• filter behaviour: how metadata constraints affect candidate traversal.

There is no single “best ANN algorithm” independent of workload. A large mostly-static image corpus, a frequently updated RAG document store, and a multitenant recommendation system with strict filters stress different parts of the design.

Think of ANN configuration as a search budget. The system spends distance calculations, graph visits or cluster probes to find candidates. Spending more generally improves recall and costs more latency/CPU. Spending less does the reverse.

A useful evaluation curve plots recall against latency rather than choosing one point in isolation. If increasing a search parameter raises Recall@10 from 0.94 to 0.97 while doubling p99 latency beyond the product SLO, the “better” recall setting may be operationally worse. If a medical research retrieval system can tolerate extra latency for higher recall, the decision may reverse.

Quality must also be evaluated after filtering. An ANN index can show excellent unfiltered recall and much lower recall once only 1% of nodes satisfy a metadata condition. Filter interaction is not a secondary implementation detail; it can change which ANN strategy is appropriate.

Approximation should therefore be treated like compression: useful when the loss is quantified relative to the reader’s job. The system should know what it is giving up and why the saved work matters.

7. HNSW from first principles: turn the vector cloud into a navigable graph

Hierarchical Navigable Small World (HNSW) is one of the most common ANN index families in modern vector search. Its central idea is to connect vectors in a graph so search can move from one promising region to another instead of comparing the query with every point.

Imagine every stored vector as a city. Exact search measures distance from the query to every city. HNSW builds roads between nearby cities plus a hierarchy of sparser “express-road” layers. Search begins on an upper layer with relatively few nodes, makes large movements toward the query region, descends into denser layers, and finishes by exploring local neighbours near the best candidates found.

The “small-world” idea is that mostly local connections plus some longer navigational structure can keep graph paths short. The “hierarchical” layer helps avoid starting every search in the dense base graph.

A simplified search process is:

1. Choose an entry point on the highest available layer.
2. Compare the query with the current node and its neighbours.
3. Move toward a neighbour that improves proximity.
4. Continue until no immediate move improves the candidate on that layer.
5. Descend to the next denser layer using the current best position.
6. At the bottom layer, maintain a broader candidate set and explore enough neighbours to return the top-k approximation.

Real implementations contain more detail, but this mental model explains the major tuning parameters and failure modes.

HNSW is attractive because query time can remain low at large collection sizes while recall is high. It also supports incremental insertion more naturally than cluster-based algorithms that require a training phase. pgvector’s current documentation notes that its HNSW index has a stronger speed–recall trade-off than its IVFFlat implementation but takes longer to build and uses more memory. pgvector documentation.

The cost is memory and graph maintenance. Each node stores links. Higher connectivity improves navigability but increases index size, build work and update overhead. Deletions can leave graph structure that requires cleanup or rebuild behaviour depending on implementation.

HNSW also changes the semantics of “LIMIT”. An approximate graph search returning 20 candidates can explore a different region than one asked for 100. Some implementations therefore do not guarantee that the first 20 results from a limit-100 query exactly match a separate limit-20 query. Qdrant’s current FAQ explicitly notes that HNSW is approximate and a larger requested limit can increase search scope enough to find better matches. Qdrant fundamentals.

The graph should not be treated as an immutable mathematical object. Data growth, deletions, payload indexes, quantisation and compaction can change effective traversal. Monitoring recall over time is essential.

8. HNSW parameters: M, construction effort and query effort

HNSW implementations expose parameters with names that vary slightly, but three concepts recur.

M roughly controls graph connectivity—the number of neighbours retained per node or an equivalent connectivity parameter. Higher M generally improves graph navigability and recall, especially in difficult/high-dimensional datasets, but increases memory usage and index-build work.

efConstruction controls how broadly the algorithm searches while inserting a point and selecting its neighbours. Higher construction effort can build a better graph at the cost of slower indexing.

efSearch or equivalent controls how many candidate nodes are explored at query time. Increasing it generally improves recall while increasing latency and CPU work.

These parameters interact. Raising query effort cannot always rescue a poorly built graph. Raising M can make every query and index update more expensive. Optimising one benchmark on one dataset does not produce a universal configuration.

Build a tuning table for your workload. For example, on a synthetic corpus of ten million vectors, a team might measure:

Those numbers are illustrative only. The useful pattern is the curve and the receiver’s SLO.

Test multiple query classes. Short general queries may have many near neighbours; rare technical queries may have sparse meaningful matches. Dense product catalogues and multilingual text can behave differently. Average recall can hide a category where the graph systematically misses relevant items.

Memory calculations should include vector storage, graph links, metadata/filter indexes, object payloads and allocator overhead—not just the raw vector bytes. A 768-dimensional float32 vector alone is 3,072 bytes before row metadata. Ten million such vectors contain roughly 30.72 GB of raw component data before index structure or compression.

This arithmetic shows why compression and disk-backed strategies matter. It also explains why vector dimensions and model choice have infrastructure cost.

Query effort can sometimes be adjusted dynamically. A low-latency autocomplete feature might use lower efSearch, while a user-triggered deep research query can spend more search budget. The system should expose such policy explicitly instead of letting every caller set arbitrary resource-intensive parameters.

Production tuning should monitor tail latency, CPU saturation and recall together. An efSearch value that looks good at low concurrency can collapse throughput under peak load.

9. IVF-style search: divide the space, then probe promising regions

Inverted file (IVF) approaches partition vectors into clusters or lists. At query time, the system compares the query with cluster representatives, chooses a subset of promising lists, and searches vectors inside those lists rather than scanning the entire collection.

The idea is spatial pruning. If vectors are grouped so that nearby items tend to share a list, a query only needs to open the most relevant regions.

IVFFlat in pgvector, for example, divides vectors into lists and probes only a configured number of lists. Its documentation states that increasing probes improves recall and costs speed; probing every list approaches exact search and can make the ANN index unattractive to the planner. pgvector documentation.

IVF requires decisions about:

• number of lists/clusters;
• training sample and when clustering is performed;
• number of lists probed per query;
• how updates are assigned;
• how imbalance between lists is handled;
• whether vectors inside lists are stored exactly or compressed.

Too few lists make each candidate region large. Too many lists can create small or unstable partitions and increase routing overhead. A poor training sample can place centroids badly for later data.

IVF therefore fits mostly-static or bulk-built corpora naturally, though modern systems can support updates. HNSW often handles incremental insertions more simply because it does not need a prior clustering training phase.

Data drift matters. If an embedding corpus changes distribution substantially—new language, new product family, new embedding model—the old cluster structure can become less efficient. Rebuild or retrain decisions should be based on measured recall/latency changes, not a fixed calendar alone.

IVF and HNSW are not mutually exclusive in the wider ANN literature; some systems combine partitioning, graph search and quantisation. The important architectural distinction is the pruning principle: graph navigation versus partition selection.

Use exact evaluation to compare candidate structures. A team can discover that HNSW achieves 0.97 recall under the latency target but consumes too much memory, while IVF with compression achieves 0.94 recall at lower cost. The product decision then depends on whether that recall difference affects relevance downstream.

10. Quantisation trades numerical precision for a smaller, faster working set

Large vector collections are expensive because dense vectors consume memory and bandwidth. Quantisation compresses vector values or index representations so more of the search working set fits in memory or cache.

Common families include reduced-precision floating point, scalar quantisation, product quantisation, binary quantisation and newer rotational approaches. Each preserves different aspects of geometry at different cost.

The broad pattern is:

1. store or derive a compact representation;
2. use it for fast candidate generation;
3. optionally rerank the candidate set using higher-precision/original vectors.

This two-stage approach mirrors hybrid retrieval more generally: cheap approximation first, expensive accuracy later on a smaller pool.

Suppose a 768-dimensional vector uses float32: 768 × 4 = 3,072 bytes. Storing the same components at 16 bits reduces raw component storage to about 1,536 bytes, before metadata and index overhead. An 8-bit scalar representation can reduce it to about 768 bytes. These savings can materially change whether an index fits in RAM.

Compression introduces error. A near-tie between two candidates can reverse after quantisation. That may be harmless when a reranker examines the top 100, or harmful if the relevant item falls outside the candidate pool entirely.

Evaluate recall using the same query distribution and downstream k. A quantised index that preserves Recall@100 well can work beautifully with a reranker even if its top-5 ordering differs from exact search.

Compression can affect distance computation differently by metric. Binary approximations and product quantisation introduce their own search procedures. Do not compare “bits per vector” without measuring the end-to-end retrieval pipeline.

Recent vector systems continue to add compression methods. Weaviate’s 1.39 release, for example, announced a preview of 4-bit rotational quantisation while also changing HNSW snapshot behaviour. Such features evolve quickly; production designs should bind to the exact product/version and validate recall under their corpus rather than treating a compression label as a stable universal capability.

Quantisation also influences recovery and migration. If the compressed index is fully derivable from authoritative original embeddings, it can be rebuilt. If only quantised vectors are retained, future reranking, migration or reindexing options may be constrained.

Keep authoritative representations according to the application’s needs. An index is usually a derived acceleration structure; losing it should not necessarily mean losing the underlying semantic record.

11. Metadata is not decoration: it defines who is allowed to compete

Vector similarity answers one narrow question: which vectors are close under this geometry? Real retrieval systems usually ask a constrained version: which eligible items are close?

Eligibility can depend on:

• tenant or organisation;
• document access-control list;
• language;
• product version;
• date range;
• region;
• content type;
• publication state;
• price or stock status;
• legal hold, expiry or deletion state.

Those attributes belong in structured metadata or payload fields that the vector-search system can filter reliably.

A common anti-pattern is to put every useful attribute into the embedding text and hope similarity learns the constraint. Semantic representations are probabilistic and task-shaped. A price filter, tenant boundary or legal eligibility condition should not depend on whether a model happened to encode the attribute strongly enough.

Another anti-pattern is to retrieve globally and filter only after the application receives results. If top-20 global neighbours contain only two eligible items, the caller gets two results even when thousands of highly relevant eligible candidates exist just outside the top 20.

Metadata therefore has two roles:

• hard eligibility: records outside the filter must not appear;
• ranking context: attributes such as recency or popularity may influence ranking but are not absolute exclusion rules.

Keep those roles separate. An access-control filter is not a soft boost. A freshness preference is not necessarily a hard delete.

Metadata fields need data-quality discipline. If one document’s tenant field is missing, should it become globally visible, globally hidden or quarantined? Security-sensitive defaults should be explicit. “Null means public” is usually a dangerous accidental policy.

Index the fields used for frequent filtering. Qdrant’s current guidance recommends payload indexes on filter fields and explains that filter-aware index structures can materially improve filtered search. Weaviate likewise combines structured filter indexes with vector traversal. These implementations differ, but both illustrate that metadata indexing is a first-class performance structure rather than a JSON blob attached after search.

Version metadata semantics too. A field called status that changes from “published/draft” to “active/inactive” can alter eligibility without any vector change. Search correctness depends on both the embedding contract and the filter contract.

12. Filtering changes the graph problem

Unfiltered HNSW assumes that moving through nearby nodes helps reach good neighbours. A restrictive metadata filter can break that intuition.

Imagine the graph contains ten million products, but the query must return only items from one small supplier representing 0.1% of the corpus. The nearest graph neighbourhoods to the query may contain excellent semantic matches from other suppliers that are all ineligible.

Three broad strategies exist:

1. pre-filter: identify eligible IDs first, then search within or constrained by that set;
2. post-filter: run vector search globally and discard ineligible results afterward;
3. integrated/filter-aware traversal: combine the filter with ANN traversal so search can navigate while respecting eligibility more intelligently.

Each has trade-offs.

Pre-filtering can be excellent when the eligible subset is small and a fast scalar index can produce it. Exact vector search over a few thousand candidates can beat ANN over millions. But if the eligible subset is large, materialising an allow-list can be expensive.

Post-filtering is simple but can destroy recall. If only 1% of global candidates are eligible, retrieving 20 global neighbours may yield zero or one valid item even though many relevant eligible items exist deeper in the graph.

Integrated filtering changes traversal behaviour. Weaviate’s current documentation describes pre-filtered vector search using an inverted index to construct an allow-list that constrains HNSW search, and it documents ACORN-inspired traversal for difficult low-correlation filters. Qdrant documents filterable HNSW that extends graph connectivity using payload indexes and also exposes ACORN-style search in stricter multi-filter cases.

pgvector illustrates another design: for approximate indexes, ordinary filtering can occur after ANN candidates are scanned; its iterative index-scan feature can continue exploring more of the index until enough filtered results are found. This demonstrates why filtered ANN can require more search work than unfiltered ANN.

The correct strategy depends on filter selectivity and correlation with vector neighbourhoods.

If the filter selects 0.01% of records, exact search over that small subset can be ideal. If it selects 80%, normal ANN plus filtering may work well. The difficult middle arises when the filter excludes enough of the graph to damage navigation but still leaves too many vectors for brute force.

Measure filtered recall separately. A system that reports Recall@10 = 0.98 on unfiltered benchmarks can produce Recall@10 = 0.60 for one important filtered query class.

Evaluation datasets should therefore include:

• no filter;
• broad filters;
• moderate-selectivity filters;
• very restrictive filters;
• multi-field conjunctions;
• filters poorly correlated with semantic neighbourhoods;
• tenant filters with skewed tenant sizes.

Filtered search is an access-path problem with a semantic index inside it. Treat it with the same seriousness as relational selectivity and query planning.

13. Design metadata schemas for retrieval, governance and migration

A vector record commonly contains an ID, one or more vectors, and metadata/payload. The metadata schema should support the retrieval contract rather than mirror the source object indiscriminately.

For a RAG document chunk, useful fields might include:

• chunk_id;
• document_id;
• document_version;
• chunk_index or span coordinates;
• tenant_id;
• language;
• content_type;
• effective_from / effective_to;
• published_state;
• source_uri or canonical reference;
• embedding_model_version;
• rights/access_class;
• deleted/tombstone state where relevant.

Do not place large source documents inside payload fields merely because the vector service permits it. Payload size affects network, memory and update costs. Store enough text/content for reranking and application display, or retain a stable pointer to authoritative storage.

Stable identity is crucial. A chunk ID derived from transient array position can change every time chunking is adjusted. If downstream systems cite chunk IDs, deletion and update operations become difficult.

Use an identity hierarchy:

logical document → document version → chunk identity → embedding edition

This lets the system answer whether a search result refers to the same logical content under a new embedding or to a materially changed source edition.

Metadata should also support repair. If one source document is withdrawn, the system should find all chunks and vector fields derived from that document. If one embedding model is deprecated, the system should identify every record using it.

Filtering fields need index strategy. A payload field used only for display may not need a scalar index. A tenant ID or access class used in every query does.

High-cardinality filters can have different cost from low-cardinality filters. Ten million unique user IDs behave differently from five languages. Systems can offer specialised tenant partitioning or sharding because the physical problem differs.

Qdrant, for example, documents a tenant-aware payload index option and custom sharding alternatives for multitenancy. The exact feature is product-specific; the general principle is to distinguish logical filtering from physical isolation.

Metadata types should be explicit. Store dates as dates, numbers as numbers and booleans as booleans where the system supports them. Serialising everything as strings makes range filtering and validation harder.

Schema evolution needs backward compatibility. Adding language is not useful if historical records remain null and the new search layer interprets null as “English”. Backfill deliberately or define the unknown state.

Metadata is part of vector-search correctness because it controls the population the ANN algorithm is allowed to search. It deserves the same provenance, quality and versioning discipline as the vectors.

14. Hybrid search combines semantic neighbourhoods with exact lexical evidence

Dense vector search is powerful at paraphrase and conceptual similarity. Traditional lexical search is powerful at exact strings, rare terms, identifiers, product codes, names and syntax. Hybrid search combines the two candidate/ranking signals.

Consider the query:

“CVE-2026-4817 cert rollover auth failure”

A vector model may understand certificate rotation and authentication failure but underweight the exact CVE identifier. BM25 or full-text search will give strong weight to the exact rare token. Hybrid retrieval can keep both strengths.

Current Databricks AI Search documentation recommends hybrid search as a general-purpose starting point and describes it as parallel ANN plus full-text retrieval merged with Reciprocal Rank Fusion. It specifically highlights identifiers, product names and technical jargon as cases where lexical search matters. Weaviate similarly documents hybrid search combining vector search with BM25.

Hybrid search is not merely “add the scores”. Vector similarity and BM25 scores live on different scales. A cosine score of 0.82 cannot be compared directly with a BM25 score of 14.3.

Common fusion strategies include:

• rank fusion such as Reciprocal Rank Fusion (RRF);
• normalise scores then combine with a weight;
• learn a fusion model from relevance judgments;
• union candidate sets and defer final ordering to a reranker.

RRF uses rank positions rather than raw score scales. A common form is:

RRF(d) = Σ 1 / (k + rankᵢ(d))

where each retrieval system contributes a reciprocal-rank term and k dampens differences among lower ranks.

RRF is robust because it does not require calibrated score distributions. The trade-off is that it discards information in score magnitude.

Weighted score fusion preserves magnitude but requires careful normalisation. Min-max normalisation can become unstable when candidate score ranges differ query to query. Z-score or learned calibration introduces its own assumptions.

Hybrid weights should be evaluated by query class. Natural-language conceptual queries may benefit from stronger vector weighting. SKU lookups and technical codes may need lexical dominance.

Some systems support sparse learned vectors such as SPLADE or late-interaction models in addition to dense vectors and BM25. “Hybrid” is therefore a family of multi-signal retrieval designs, not one fixed algorithm.

Evaluation should label query intent. A global metric can hide that hybrid improves identifier queries while slightly hurting broad semantic questions. Product teams need to know whether the affected query class matters to users.

Hybrid search also helps graceful degradation. If embedding generation fails temporarily, lexical retrieval may still return evidence. If the full-text index lags, semantic retrieval may continue. The application should know which retrieval modes actually contributed to the result.

15. Candidate generation and reranking should be designed as separate stages

ANN is designed to retrieve a useful candidate set cheaply. It does not need to perform the final ranking perfectly if a second-stage model can inspect a smaller set more accurately.

This creates a common retrieval architecture:

Fast candidate retrieval → merge/filter → rerank top N → return top K

The reranker can use:

• a cross-encoder reading query and document together;
• a late-interaction model;
• a learned-to-rank model with structured features;
• a domain-specific relevance classifier;
• business constraints such as recency or authority.

Cross-encoders often produce stronger relevance judgments because they evaluate query-document token interactions directly rather than comparing two independently generated vectors. The cost is much higher compute per candidate.

That makes candidate recall critical. A perfect reranker cannot recover a relevant document that ANN/hybrid retrieval never placed in its candidate pool.

Evaluate at the candidate depth used by reranking. If the reranker receives top 100, measure Recall@100 for relevant documents, not only ANN Recall@10.

Suppose an exact relevance set contains one highly relevant document. ANN retrieves it 92% of the time in top 20 and 99% in top 100. If reranking top 100 keeps latency within SLO, the deeper candidate pool can materially improve final quality.

Reranking also allows cheaper compressed indexes to act as candidate generators. Quantised ANN can retrieve 200 candidates, then exact vectors or a cross-encoder reorder them.

Current Databricks AI Search supports optional reranking on ANN, full-text or hybrid candidate sets and notes the latency trade-off. The exact performance estimate in vendor documentation should not be generalised; the mechanism is what matters.

Reranking inputs need trusted text. If vector payload contains an outdated chunk while the source document changed, the reranker can confidently rank stale evidence. Sync and versioning remain essential.

Business rules should be explicit after relevance. A product search can remove out-of-stock items as a hard filter, rerank relevance, then apply diversity or commercial constraints. Mixing every requirement into one opaque relevance score makes failures difficult to diagnose.

Diversity can be useful when top semantic neighbours are near-duplicates. Maximal Marginal Relevance (MMR) and crowding/group limits are examples of methods that trade some individual similarity for broader result coverage. Google Vector Search documents crowding controls; Weaviate exposes MMR. The user job determines whether diversity helps.

Final retrieval evaluation should therefore decompose:

• candidate recall;
• reranker ordering quality;
• filter correctness;
• diversity where intended;
• end-to-end latency and cost.

This decomposition prevents the team from blaming the embedding index for a reranker failure—or tuning a reranker when the relevant evidence was never retrieved.

16. Chunking determines what the vector index is allowed to retrieve

Text RAG systems often embed chunks rather than whole documents. The chunk boundary defines the unit that can become a nearest neighbour.

A paragraph-sized chunk can preserve local specificity but lose necessary context. A very large chunk can preserve context but dilute the semantic signal and consume more model context after retrieval.

Chunking choices include:

• fixed token or character windows;
• sentence or paragraph boundaries;
• section-aware chunking;
• recursive splitting;
• semantic segmentation;
• document-type-specific rules;
• overlap between neighbouring chunks;
• parent-child retrieval where small chunks retrieve larger context.

The vector database does not know whether a chunk boundary is pedagogically, legally or semantically meaningful unless the pipeline records that structure.

Stable chunk identity matters. If a document is re-chunked after a small edit, every chunk ID can change if IDs are based only on position. Downstream citations, deletion receipts and cached RAG answers then lose continuity.

A useful chunk record can preserve:

• logical document ID;
• document version;
• chunk ID;
• source span coordinates or heading path;
• chunking algorithm/version;
• embedding model/version;
• canonical source URL;
• hash/checksum of embedded text where appropriate.

Overlap creates deliberate duplication. A sentence can appear in two neighbouring chunks, causing near-duplicate top results. Reranking or MMR can reduce redundancy, but the source should still identify that the chunks overlap.

Chunk quality should be evaluated through retrieval questions. If users ask “What happens after the warranty expires?” and the answer depends on a clause split across two chunks, the issue is upstream of HNSW.

Use a retrieval test set containing questions whose relevant evidence lies near headings, tables, bullet lists, cross-references and chunk boundaries. Average semantic similarity cannot substitute for this structural evaluation.

See Token Budgets and Chunking for the wider representation problem. The vector-database job is to keep chunk identity and lifecycle coherent once those units become indexed records.

17. Synchronisation decides whether search reflects the current source

A vector index is usually derived from another source: document store, warehouse table, application database, content repository or event stream.

Changes can arrive through:

• batch rebuilds;
• change-data capture;
• webhooks/events;
• scheduled crawls;
• streaming updates;
• manual publication workflows.

The important question is not merely whether an embedding job ran. It is whether the vector index converged to the intended source state.

Track source-to-index lag:

index_lag = searchable_at – source_effective_or_published_at

The correct source timestamp depends on the application. A CMS edit timestamp can precede editorial publication. An object-upload time can precede validation. A database commit can be the right boundary for an operational table.

Use versioned upserts. An older delayed event should not overwrite a newer source version. If source version 12 is already indexed, a retry for version 11 should be rejected or ignored.

Partial failure creates divergence. The source update may succeed while embedding generation fails. The vector record can remain stale with a healthy index service.

A useful pipeline state distinguishes:

• source version observed;
• text extracted;
• embedding generated;
• vector record written;
• scalar/filter index updated;
• record searchable;
• old version removed/tombstoned;
• reconciled.

Do not collapse this into one “indexed” boolean for high-value retrieval.

Databricks AI Search, for example, supports indexes that sync from underlying tables. That integrated route can reduce custom plumbing, but users still need to understand which source columns are synced, which embedding edition is active and how quickly updates become searchable.

Reconciliation should compare source and vector-store populations. Count mismatches, missing IDs, unexpected extra IDs, version mismatches and tombstones. Sample records should compare source text hashes and metadata.

Freshness can be more important than ANN recall for fast-changing knowledge. A 99.9%-recall index returning yesterday’s policy document is worse than a 95%-recall index containing today’s authoritative edition.

This is why vector-search observability needs both index metrics and data-synchronisation metrics.

18. Embedding-model migration is a coordinate-system migration

Changing the embedding model can improve retrieval quality, support more languages, reduce dimensions or lower inference cost. It also changes every vector in the corpus.

Do not overwrite old vectors in place before validating the new representation.

A controlled migration can use dual fields or parallel collections:

• embedding_v1 with index V1;
• embedding_v2 with index V2;
• query service capable of generating both during evaluation;
• side-by-side retrieval on a labelled query set;
• consumer migration after evidence.

Compare:

• human relevance metrics;
• exact vector retrieval quality;
• ANN recall;
• filter performance;
• latency;
• embedding generation cost;
• memory/index size;
• multilingual/domain-specific performance.

A model with better average semantic retrieval can regress exact terminology or one important language. Hybrid search may change the impact.

Queries and corpus vectors must use compatible embedding editions. During dual running, route query embedding V1 to index V1 and V2 to index V2. Never compare V2 query vectors with V1 document vectors simply because dimensions match.

Re-embedding is a large backfill. If 500 million chunks require new embeddings, generation can take substantial compute and time. Plan capacity and throttle to protect ordinary production traffic.

Source changes during re-embedding create races. A document embedded under V2 at source version 7 should not later be overwritten by a backfill job embedding source version 6.

Use a version guard with both source and embedding editions.

A migration can complete when:

1. all in-scope source versions have V2 embeddings;
2. V2 vector/filter indexes are built and reconciled;
3. query generation uses V2;
4. retrieval/reranker evaluation passes;
5. production canary/shadow results are acceptable;
6. all consumers leave V1;
7. rollback window ends;
8. V1 retention or deletion policy is executed.

Keep the old representation long enough for rollback when feasible. A vector-model migration resembles a database schema and index migration more than a simple model configuration change.

19. RAG is downstream of retrieval: the language model cannot cite evidence it never received

Retrieval-augmented generation uses retrieved material to provide a language model with external context. Vector search is often one candidate-retrieval mechanism inside RAG.

A common pipeline is:

User question → query rewriting/vectorisation → filters → vector/lexical candidates → fusion → reranking → context assembly → language-model answer → citations

Every upstream decision constrains the answer.

If the authoritative document is absent from the vector store, generation cannot recover it. If metadata marks it inaccessible incorrectly, the model cannot see it. If ANN misses it, reranking cannot promote it. If context assembly truncates the relevant paragraph, a correct retrieval can still become a weak answer.

This gives a useful diagnostic hierarchy:

1. Was the authoritative evidence indexed?
2. Was it eligible under the query’s filters?
3. Did candidate retrieval include it?
4. Did fusion/reranking keep it?
5. Did context assembly include the required passage?
6. Did the model use and cite it correctly?

RAG evaluation should measure these stages separately.

Vector relevance is not authority. A semantically close community forum can rank above the official policy. Metadata or reranking should represent source authority when the application requires it.

Similarly, recency is not authority. A newly published blog can be newer than a current legal standard while being less authoritative. Ranking policies should distinguish freshness, authority and semantic relevance.

Chunk retrieval can also lose document-level context. A clause might refer to “the exception above”. The chunk alone is ambiguous. Parent-document expansion or neighbouring-chunk retrieval can restore context after the candidate is found.

Context windows are finite. Returning 100 chunks to a generator can dilute important evidence and increase cost. Candidate retrieval and reranking should aim to deliver a small, diverse, high-quality evidence set.

RAG systems need citation identity. The vector record should map back to a stable document/version and source location. A citation pointing only to an internal chunk UUID is not useful to a reader.

Deleted or superseded source material should stop being eligible for ordinary current-state answers while remaining retrievable for historical questions where policy allows. Metadata can express current/historical state; query intent determines which state is appropriate.

Security is especially important. A RAG index can become a path around source permissions if all embeddings/chunks are globally searchable. The retrieval query must enforce the same access boundary as the source or a deliberately governed derivative.

See AI Data Management for the broader provenance, rights, evaluation and feedback lifecycle.

20. Update, delete and tombstone semantics are part of search correctness

Vector stores are often demonstrated as append-only collections. Production data changes.

An update can mean:

• metadata changed but content/embedding did not;
• source text changed and needs a new embedding;
• embedding model changed;
• record moved to another tenant/category;
• access policy changed;
• record was superseded by a new source version.

A delete can mean:

• hard removal;
• soft deletion/tombstone;
• expiry;
• withdrawal from current retrieval but preservation for audit/history;
• tenant/account deletion;
• legal or rights revocation.

These states should not share one ambiguous boolean if consumers need different behaviour.

Search should exclude tombstoned records immediately under the application’s consistency contract. Physical graph/index cleanup can occur later if the system guarantees the deleted record is no longer returned.

Approximate graph indexes make deletion operationally interesting. Removing many nodes can degrade connectivity or leave dead references until compaction/rebuild. pgvector notes that HNSW vacuuming can be expensive; Qdrant and Weaviate maintain their own segment/graph cleanup strategies. Implementation details differ, but the general principle is that logical deletion and physical index optimisation are separate phases.

Stale caches can resurrect deleted results. If application caches search responses, invalidate or version them. A vector-store delete receipt does not prove every downstream cached answer disappeared.

Deletion propagation should include:

• source;
• chunk store;
• dense vectors;
• sparse vectors/full-text index;
• metadata indexes;
• reranker cache;
• application caches;
• RAG response/evidence logs according to policy;
• backups or retained historical copies.

See Data Deletion and Destruction Verification for the full distributed obligation.

Update order needs version guards. If document version 9 is indexed, a delayed worker processing version 8 must not replace it. Store source version/effective time and reject stale writes.

Two-phase migrations can help large changes:

1. write new vector record/version;
2. verify searchable and eligible;
3. switch active alias/filter state;
4. tombstone old version;
5. clean old index physically later.

This prevents a period where the document disappears entirely because the old record was deleted before the new one became searchable.

Search correctness includes absence. A result that should have been withdrawn but still appears is a retrieval defect even if its similarity score is perfect.

21. Multitenancy is a search-boundary problem before it is a billing feature

A multitenant vector system stores records for several customers, organisations or domains. The first requirement is not performance. It is preventing one tenant’s data from entering another tenant’s search result.

Logical multitenancy can use a tenant field and mandatory filter on every query. Physical multitenancy can use separate collections, namespaces, shards or indexes. Many systems support combinations.

Logical filtering is operationally efficient when many small tenants share infrastructure. Physical separation can improve isolation, noisy-neighbour control, migration and deletion at the cost of more objects to manage.

Choose the boundary using:

• security consequence of cross-tenant leakage;
• tenant count;
• tenant size distribution;
• query volume skew;
• custom model/index needs;
• data residency;
• backup/restore requirements;
• per-tenant deletion/export;
• cost accounting.

Tenant IDs should never come only from untrusted request payloads when the authenticated identity already implies tenant scope. The application should derive or validate the filter from the authorised session.

A missing tenant filter should fail closed. Do not interpret null tenant metadata as shared/public unless a deliberate shared-content model exists.

Test collisions explicitly. Create identical document IDs in tenant A and tenant B with obviously different vectors/content. Search as each tenant and verify both retrieval and metadata isolation.

Large tenants can become hotspots. If one tenant owns 40% of vectors and 70% of queries, a shared shard strategy can produce noisy-neighbour latency. Dedicated shards or indexes may be justified for heavy tenants while smaller tenants remain shared.

Qdrant’s documentation distinguishes logical payload filtering, tenant-aware indexing and custom sharding. Other systems use namespaces or separate collections. These are different physical implementations of the same governance question: which data is allowed to coexist operationally, and under which query contract?

Backups must preserve tenant boundaries. Restoring one tenant should not require exposing or duplicating all tenants unnecessarily where the service contract expects isolated recovery.

Embedding migrations can be per-tenant if customers use different models or languages. That flexibility increases version complexity; the query path must know which vector field/index belongs to the authenticated tenant.

Multitenancy therefore belongs in schema, index, security, observability and recovery design—not as a final query parameter added after the vector service is built.

22. Access control must be enforced before or during retrieval, not after the answer is generated

Enterprise RAG systems often index documents with different permissions. A user may be allowed to search public manuals and their own project documents but not another department’s confidential records.

If retrieval occurs globally and unauthorised chunks are filtered only after a language model receives them, the access boundary has already failed.

Permissions can be represented through:

• tenant/project identifiers;
• document ACLs;
• groups/roles;
• classification levels;
• purpose tags;
• time-bound access;
• source-system permissions resolved at query time.

Large ACL lists can be difficult to index directly. One document may be visible to thousands of users. Rather than store every user ID, model groups or policy attributes where appropriate.

Permission freshness matters. A user’s access can be revoked while the vector record remains. Query-time authorisation should use current effective identity/roles or synchronise ACL metadata fast enough for the risk.

Do not rely on embedding similarity to “hide” confidential information. A secret can be semantically close to a harmless query.

Search logs can also leak protected content through query text, result titles or snippets. Protect observability data according to the same classification model.

When a source permission changes, propagate the policy update independently of re-embedding if the content is unchanged. Metadata-only ACL updates should not require expensive vector regeneration.

Test negative cases:

• revoked user;
• user moved groups;
• missing ACL metadata;
• stale cache;
• shared document visible to two tenants;
• historical document under changed classification;
• service account with narrower scope than a human administrator.

Databricks AI Search documents access-control requirements around search indexes and catalog objects. The exact permission model is platform-specific; the principle is broader: retrieval is a data-access operation and belongs under normal entitlement governance.

See Data Access Governance and Entitlements for the canonical access-control architecture.

23. Retrieval evaluation needs both geometric recall and human relevance

ANN Recall@k answers whether approximate search recovered the same neighbours as exact vector search. It does not answer whether those neighbours are useful to a person.

A complete retrieval evaluation usually has at least three layers:

1. index recall: ANN versus exact vector ranking;
2. retrieval relevance: returned records versus human/task relevance judgments;
3. application success: search click, recommendation outcome, RAG answer quality or another product metric.

Common ranking metrics include:

• Precision@k: proportion of top-k results judged relevant;
• Recall@k: proportion of known relevant items found in top-k;
• MRR: reciprocal rank of the first relevant item, averaged over queries;
• nDCG: ranking quality when relevance has graded levels and rank position matters;
• Hit Rate: whether at least one relevant item appears in top-k.

For one query with the first relevant item at rank 4, reciprocal rank is 1/4 = 0.25. MRR averages that value across the query set.

nDCG is useful when a perfect answer at rank one is better than a merely related answer at rank one. It discounts relevance by position and normalises against an ideal ranking.

Build a labelled query set from actual user intent, not only synthetic paraphrases. Include:

• conceptual questions;
• exact names/IDs;
• misspellings;
• multi-lingual queries;
• short vague queries;
• long detailed queries;
• filtered queries;
• no-answer queries;
• queries with several valid answers;
• adversarial or ambiguous wording.

For RAG, label supporting evidence rather than just final answer text. A language model can sometimes produce the right answer from prior knowledge even when retrieval is wrong; that should not be scored as retrieval success.

Relevance judgments can disagree. Use multiple reviewers for high-consequence benchmarks and preserve adjudication rules. The benchmark itself is a dataset requiring versioning.

Measure by query class. Suppose hybrid search improves overall nDCG from 0.71 to 0.75 but degrades legal citation queries from 0.90 to 0.72. The global gain can hide an unacceptable specialist regression.

Do not overfit the retrieval system to one static benchmark. Hold out queries, refresh the evaluation set as user behaviour changes, and preserve a stable regression core.

Online click metrics need caution. Users click what the system shows; position bias can make already-high results appear more relevant. Controlled experiments or interleaving can help when appropriate.

A vector system is trustworthy when its evaluation distinguishes representation quality, ANN approximation, filtering, fusion/reranking and downstream application behaviour.

24. Observability should explain why search quality changed

Vector-search monitoring needs infrastructure metrics and semantic/retrieval metrics.

Infrastructure signals:

• query latency p50/p95/p99;
• QPS/throughput;
• CPU;
• RAM/index residency;
• disk I/O;
• cache hit rate;
• network;
• index-build/rebuild duration;
• replication lag;
• error/timeout rate.

Data/index signals:

• vector count;
• missing vectors;
• dimension/model-version distribution;
• metadata-null rates;
• tombstone count;
• segment count;
• source-to-index lag;
• filter-index coverage;
• unindexed filter usage;
• quantisation/index version.

Retrieval signals:

• sampled ANN Recall@k versus exact;
• candidate depth;
• filtered recall;
• zero-result rate;
• average eligible-candidate count;
• hybrid contribution rates;
• reranker score/rank movement;
• duplicate result rate;
• stale/superseded result rate.

These metrics help localise change. If p99 latency spikes after adding a metadata filter while unfiltered latency is stable, investigate filter indexing/traversal. If recall drops after a rebuild but relevance under exact search is unchanged, investigate ANN configuration. If exact relevance drops after an embedding migration, the representation changed.

Version deployments on dashboards. Mark embedding model, HNSW parameters, filter-index changes, shard rebalances and reranker releases so operators can align quality changes with system changes.

Monitor the query mix. A shift from broad semantic queries toward exact product codes can make a vector-only system appear worse even with unchanged index quality. The user population changed.

Zero-result rate should distinguish truly no matching records from filtered ANN under-retrieval. If exact filtered search finds candidates but ANN returns none, the issue is the search path, not the corpus.

Audit slow queries with their filter shape and candidate statistics. One poorly selective or unindexed payload field can dominate tail latency.

Privacy applies to query logs. Search queries can reveal sensitive intentions. Hash or redact where possible, restrict access and retain only what evaluation/operations need.

See Data Observability and Monitoring for the broader discipline.

25. Latency, throughput and concurrency are different performance questions

A vector search can be fast for one request and fail under production concurrency.

Latency measures time per query. Throughput measures queries handled per unit time. Concurrency is the number of overlapping requests/work. The same ANN settings can behave differently as concurrency rises.

Tail latency matters. A median of 20 ms with p99 of 900 ms can create poor interactive experience.

Break down end-to-end search latency:

• query embedding generation;
• network to vector service;
• filter planning;
• ANN search;
• lexical search;
• fusion;
• payload fetch;
• reranking;
• parent-document expansion;
• network/application serialisation.

Tuning only ANN can miss the dominant cost. A cross-encoder reranker can add more latency than the vector search itself. Remote embedding generation can dominate small indexes.

Query batching can improve throughput for offline workloads. GPU exact search may become attractive in batches even if CPU ANN wins per interactive query.

Concurrency can reduce cache locality and increase memory pressure. HNSW traversal under many simultaneous requests can saturate CPU or memory bandwidth before raw QPS appears high.

Filtered queries can have heavy-tail behaviour depending on selectivity. A search for one large tenant may be cheap; a complex combination of rare filters may force deeper graph exploration.

Measure realistic payload-return sizes. Fetching 100 large text chunks can dominate network time even when ANN completes in 5 ms.

Use workload classes:

• interactive search;
• RAG retrieval;
• recommendation fan-out;
• offline batch retrieval;
• reindex/backfill;
• evaluation exact-search jobs.

Protect interactive traffic from backfill/index-build workloads through resource separation, throttling or scheduling.

A useful SLO can combine quality and performance:

99% of authorised hybrid queries return at least K results within 250 ms from the retrieval service, with sampled candidate Recall@100 above the approved threshold.

The exact numbers are application-specific. The important feature is that speed and retrieval quality appear in the same operational conversation.

26. Cost begins with bytes, but it ends with the entire retrieval path

Vector systems can become expensive because they combine high-dimensional data, memory-hungry indexes, duplicate representations and model inference.

A simple raw-vector estimate is:

vector bytes = number of vectors × dimensions × bytes per component

For 50 million vectors × 768 dimensions × 4 bytes:

50,000,000 × 768 × 4 = 153,600,000,000 bytes, about 153.6 GB decimal before graph edges, row IDs, allocator overhead, metadata, replicas or full-text indexes.

HNSW adds graph links. Metadata indexes add more structures. Hybrid search may keep BM25/full-text indexes. Reranking may require chunk text. Replication can multiply storage. Backups/snapshots add another copy path.

Embedding generation can dominate one-time ingestion cost. Re-embedding the full corpus during model migration repeats that cost.

Query cost includes:

• query embedding inference;
• vector search CPU/GPU;
• storage I/O;
• network;
• lexical search;
• reranker inference;
• context assembly;
• downstream language-model tokens for RAG.

A vector database with cheap ANN but an expensive reranker can have high end-to-end cost. A high-quality retriever that returns too much context can increase generator cost.

Cost modelling should therefore use one completed user request as the unit.

For a hypothetical RAG query:

The percentages are illustrative, not benchmarks. The lesson is to optimise the dominant stage measured in your system.

Compression can reduce memory at possible recall cost. On-disk vectors can reduce RAM while increasing I/O latency. Fewer replicas reduce cost while reducing failure tolerance/read capacity. Lower-dimensional embeddings can reduce index size and perhaps retrieval quality.

Keep cost per useful retrieval, not cost per million vectors, as a decision metric. A smaller index with poor relevance can increase user retries and generator calls.

Capacity planning should include growth rate. A corpus adding one million chunks per week can double in a year. HNSW build/update cost and snapshot time can cross operational thresholds before raw storage fills.

Measure data lifecycle. Expired documents should leave the live vector index rather than accumulate forever unless historical retrieval requires them. Retention is performance management as well as governance.

27. Sharding and replication turn one vector index into a distributed search problem

When one machine cannot hold or serve the collection economically, the system can distribute vectors across shards.

A simple sharded query becomes:

1. route query to relevant shards;
2. each shard retrieves local top candidates;
3. coordinator merges global candidates;
4. apply final reranking/fusion.

If every query fans out to every shard, adding shards can increase network/merge overhead. Partitioning should ideally help route some queries to fewer shards.

Shard keys can follow:

• tenant;
• region;
• hash of record ID;
• content domain;
• time;
• custom business boundaries.

Hash distribution balances storage but often requires all-shard fan-out for semantic queries. Tenant sharding reduces fan-out for tenant-scoped search but can produce uneven shard sizes.

Replication serves two roles: availability and read throughput. Multiple copies of a shard can survive node failure and serve parallel queries.

Replication creates consistency questions. How soon does an upsert or deletion reach every replica? Can a query hit a stale replica and return a revoked document?

High-risk access/deletion changes may require stronger read-after-write consistency than ordinary content updates.

Rebalancing is itself a workload. Moving large vector shards can consume network and disk, invalidate caches and temporarily change latency. Schedule/observe it like a deployment.

Index builds can be local per shard. New shards may need bulk index construction before serving. A coordinator should not route production traffic to a partially indexed shard unless the serving contract permits degraded exact search or another explicit state.

Global top-k merging needs enough candidates from each shard. If each shard returns only local top-5 and global K=20 across ten shards, relevant global results can be lost depending on score distributions. Retrieve deeper local candidate lists before merge when needed.

Scores should be comparable across shards when using the same embedding model/metric. If shards use different embedding versions, global ranking is invalid.

Regional deployments can reduce latency and support sovereignty, but a global user search may require cross-region fan-out. Decide whether data may move, whether queries stay regional, and how shared/public content is replicated.

See Data Partitioning and Sharding for the broader distribution principles.

28. Recovery should rebuild the search system from authoritative records, not from hope

Vector indexes are usually derived state. A recovery plan should identify which parts are authoritative and which can be regenerated.

Possible authoritative components include:

• source documents/records;
• document versions;
• metadata/ACLs;
• embedding outputs if regeneration is expensive or model unavailable;
• embedding model/version artifact;
• chunking configuration;
• evaluation benchmark;
• index configuration.

The HNSW or IVF structure itself can often be rebuilt from vectors. But rebuilding millions or billions of vectors can take substantial time. Snapshots can reduce recovery time even when the index is conceptually derived.

Define RPO/RTO separately:

• source data RPO;
• vector/embedding RPO;
• search-service RTO;
• permission/metadata RPO;
• evaluation/configuration recovery.

A system can tolerate rebuilding ANN for hours if lexical search remains available. Another critical recommendation service may require rapid vector restoration.

Recovery tests should verify more than record count:

1. restore source/vector data;
2. restore metadata and permissions;
3. restore correct embedding/index versions;
4. rebuild or load ANN structure;
5. reconcile tombstones/deletions;
6. run exact-versus-ANN recall tests;
7. run labelled relevance benchmark;
8. verify tenant/access isolation;
9. verify current documents supersede obsolete versions;
10. resume traffic gradually.

A restored index can be structurally healthy and semantically wrong if it uses an old source snapshot or embedding version.

Backups can resurrect deleted documents. Apply deletion/tombstone logs newer than the snapshot before serving, or rebuild from a source that already reflects the current policy state.

Snapshot compatibility matters across software versions. Test restoration using the actual target version, not only same-version backups.

Keep infrastructure/configuration as code where possible: distance metric, HNSW parameters, scalar indexes, sharding, quantisation and aliases. Manual configuration becomes hard to reproduce during disaster.

A vector service should support a degraded mode if appropriate. Keyword-only search can provide reduced functionality while ANN rebuilds. A RAG system can state that semantic search is temporarily unavailable rather than silently serve a stale index.

Recovery is complete when search quality and policy are restored, not merely when the process accepts queries.

29. Choose the simplest vector architecture that meets the receiver’s job

Several architectures can satisfy vector retrieval:

Option A — Existing relational database with vector extension

Best when vector data belongs naturally with transactional/relational metadata, collection size and QPS are manageable, strong SQL filtering/transactions matter, and operating one database is simpler.

Advantages:

• single source for vectors and structured data;
• mature transactions/backups/tooling;
• easy joins and filters;
• fewer moving parts.

Trade-offs:

• ANN scale/performance may differ from specialised systems;
• vector workloads can compete with transactional workloads;
• horizontal scale may require extra architecture.

Option B — Dedicated vector database/service

Best when vector search is a major workload with large collections, high QPS, specialised filtered ANN, multitenancy or operational features.

Advantages:

• purpose-built ANN/index lifecycle;
• vector-specific scaling/compression;
• integrated metadata filtering;
• search-oriented APIs.

Trade-offs:

• another system to operate;
• source synchronisation;
• possible transactional limitations;
• cost/lock-in/migration complexity.

Option C — Search engine with vector + lexical indexes

Best when full-text search is already central and hybrid lexical/vector ranking is the dominant job.

Advantages:

• mature keyword search;
• filters/facets;
• hybrid ranking;
• one retrieval system.

Trade-offs depend on vector scale, update pattern and operational complexity.

Option D — Warehouse/lakehouse vector search

Best when governed analytical data already lives there and RAG/semantic retrieval benefits from integrated lineage, batch sync and access control.

Advantages include governance integration and reduced data movement; trade-offs include service latency/cost and platform coupling.

Option E — In-process ANN library

Best for embedded applications, research, offline jobs or controlled single-service deployments where persistence/distribution are handled elsewhere.

The selection criteria should include:

• corpus size;
• vector dimension;
• QPS and latency;
• write/update rate;
• filter complexity;
• hybrid lexical need;
• reranking;
• multitenancy;
• transactions;
• availability;
• residency;
• backup/recovery;
• team expertise;
• cost.

A dedicated vector database is not automatically more advanced. The most mature design can be the simplest system whose failure modes the team understands.

30. Worked laboratory: build a 20-million-chunk technical RAG index

We now design a fictional system called Northstar Manuals. It supports engineers searching product manuals, release notes and incident runbooks. The corpus has 20 million chunks across 500,000 document versions.

The workload:

• 1,500 search requests per second at peak;
• 80% broad natural-language questions;
• 15% exact error codes/product IDs;
• 5% mixed queries;
• tenant + product-version filters on every query;
• RAG reranks top 100 candidates and sends top 8 chunks to the generator;
• new/updated documents should become searchable within five minutes;
• p95 retrieval-service latency target: 180 ms before generation.

These numbers are synthetic.

Step 1 — Establish exact relevance before ANN

The team creates 2,000 labelled queries with relevance judgments. It embeds the corpus and runs exact vector search on a representative subset where exact evaluation is computationally practical.

Dense exact search achieves nDCG@10 of 0.70. Pure BM25 achieves 0.68. The failure sets differ: vector search misses exact error identifiers; BM25 misses paraphrased conceptual descriptions.

Step 2 — Hybrid baseline

They run dense and BM25 candidate retrieval, union top 100 from each and apply RRF. Hybrid nDCG@10 rises to 0.77 on the fictional benchmark. Identifier-query nDCG rises from 0.52 dense-only to 0.81 hybrid.

This does not prove hybrid is universally better; it shows why the query mix matters.

Step 3 — Add a cross-encoder reranker

Reranking top 100 hybrid candidates raises nDCG@10 to 0.83 but adds 65 ms p95 in the synthetic test.

The retrieval path budget becomes:

The path fits the 180 ms p95 target with little margin.

Step 4 — Choose ANN parameters by Recall@100

Because the reranker sees 100 candidates, the ANN goal is high Recall@100, not perfect top-8 ordering.

Three HNSW query-effort settings on a representative exact-evaluation sample produce:

Setting B is selected because reranked final relevance is statistically indistinguishable from C on the current benchmark while preserving latency headroom. This is a hypothetical decision, not a recommended universal threshold.

Step 5 — Test filters, not just the global graph

Every search includes tenant_id and product-version constraints. The team stratifies filtered tests by tenant size.

Large tenants have excellent filtered recall. Several tiny tenants suffer because global HNSW neighbourhoods contain few eligible nodes.

The team adds tenant-aware filtering/indexing and an exact-search fallback when the eligible candidate set is under 5,000 vectors. Small-tenant Recall@100 improves from 0.86 to 0.98 in the fictional benchmark.

Step 6 — Estimate memory

20 million × 768 × 4 bytes = 61.44 GB raw float32 vector components.

Graph links, metadata, text payload, lexical index and replicas multiply that. Two serving replicas mean raw vectors alone represent 122.88 GB across replica copies before compression.

The team tests half-precision/quantised candidate indexes while retaining authoritative float vectors for selected reranking/evaluation. Quantisation reduces the ANN memory footprint enough to keep hot graph structure in RAM while Recall@100 remains above the approved threshold.

Step 7 — Synchronisation

Source documents carry document_version. The ingestion pipeline rejects stale writes. An update creates new chunks/embeddings, verifies searchability, switches the active version, then tombstones the old version.

A reconciliation job compares active source versions with vector records every hour and alerts on missing or extra active chunks.

Step 8 — Access control

Tenant scope is derived from authentication, not request text. Product permissions become metadata filters. Records with missing tenant/access fields are quarantined and never public by default.

Step 9 — Embedding migration

A new embedding model is evaluated as V2 in a parallel collection. Query traffic is shadowed to V2. V1 remains authoritative until relevance, filtered recall, latency and multilingual subsets pass review.

Step 10 — Recovery

The authoritative source documents and embedding manifests are backed up. Vector index snapshots reduce RTO. Quarterly recovery tests restore one isolated environment, replay deletion logs, run the benchmark and verify cross-tenant negative tests before declaring recovery complete.

Step 11 — Product result

The final Northstar system is not defined by “HNSW enabled”. Its trust comes from the full chain:

versioned source → compatible embedding → governed metadata → exact relevance baseline → measured ANN → filter-aware search → hybrid candidates → reranking → access-safe RAG context → monitored lifecycle → tested recovery

That is the vector-database reader job in one worked example.

31. Failure library: fast vector search can fail in ways that look healthy

Vector-search incidents are difficult because the service often continues returning plausible results. A database outage is obvious. A retrieval system using the wrong embedding edition can return fluent, related-looking evidence for days.

Named failure modes make diagnosis faster.

Embedding-coordinate mismatch

The query service upgrades to embedding model V2 while indexed documents remain V1. Dimensions happen to match, so the database accepts the vector and computes distances. Search quality collapses without a schema error.

Repair: bind query and corpus embedding editions explicitly; reject incompatible pairs; dual-run migrations.

Metric mismatch

The embedding model is evaluated with cosine similarity, but the production index uses unnormalised dot product. Rankings shift, especially for vectors with varying norms.

Repair: record the metric in the representation contract and test exact ranking before index construction.

ANN recall collapse

HNSW parameters are reduced to lower latency. Average query time improves, but hard query classes lose critical neighbours.

Repair: monitor sampled Recall@k against exact search by query class, not only aggregate latency.

Filter starvation

Global ANN returns excellent neighbours, but a restrictive tenant/product filter removes most candidates after traversal. Results become sparse or empty even though relevant eligible records exist.

Repair: use pre-filtering, filter-aware traversal, iterative scans or exact filtered fallback according to selectivity.

Null-metadata exposure

A document arrives without tenant_id. The application interprets null as unscoped/public and exposes it across tenants.

Repair: fail closed for security-sensitive metadata; quarantine incomplete records; test missing-field behaviour.

Stale source, fresh index

The vector service was rebuilt five minutes ago from a source snapshot that is two days old. Infrastructure freshness looks good while knowledge freshness is bad.

Repair: track source effective/published version and source-to-index lag, not only index-build time.

Old version survives update

A new document edition is indexed, but the earlier chunks remain active. Search returns contradictory current and obsolete passages.

Repair: version source records, switch active editions atomically where possible, tombstone old chunks after new searchability is verified.

Delete resurrection

A document is deleted from the live vector store, but a cache or restored snapshot later returns it again.

Repair: propagate deletion through caches, lexical indexes, replicas and recovery replay; test deletion after restore.

Chunk-boundary amputation

The relevant sentence is retrieved, but its condition appears in the preceding chunk. RAG answers the rule without the exception.

Repair: use structure-aware chunking, parent/adjacent context expansion and boundary-focused evaluation queries.

Overlap echo

Heavy chunk overlap produces five nearly identical top results. The generator receives redundant context and misses a complementary document.

Repair: deduplicate, group by parent document, use MMR/crowding or diversify after reranking.

Identifier blindness

Dense embeddings understand the concept but miss an exact error code, SKU or regulation number.

Repair: hybrid vector + lexical retrieval, exact-field boosts or query classification.

Lexical dominance

A hybrid weight over-favours exact terms and suppresses semantically equivalent passages using different language.

Repair: evaluate fusion by query class; tune or learn combination rules; inspect candidate contributions.

Reranker bottleneck

ANN finishes in 15 ms, but reranking 500 candidates takes 400 ms and exceeds p99 latency.

Repair: reduce candidate depth after recall analysis, use smaller reranker, batch inference or route only ambiguous queries to deep reranking.

Reranker overconfidence on stale payload

The vector record points to an old text payload. The cross-encoder strongly ranks it because it directly matches the query.

Repair: version payload with source; reconcile hashes; make source freshness a retrieval eligibility condition.

Benchmark overfitting

The team repeatedly tunes HNSW, fusion and reranker settings on the same 500 queries. Offline metrics rise while unseen production questions do not improve.

Repair: hold out evaluation queries, maintain a regression core, rotate fresh samples and inspect online outcomes.

Popularity feedback loop

Clicked results become training positives, so already-visible items receive stronger representation/ranking. New but relevant material struggles to surface.

Repair: preserve exposure information, diversify evaluation labels and separate observed preference from position effects.

Cross-tenant graph leakage

Physical index traversal is global and a bug in filtering allows another tenant’s candidate through.

Repair: treat tenant filter as a hard invariant, test negative cases continuously, consider physical separation for high-consequence tenants.

Reindex storm

An embedding migration, metadata backfill and source crawl all run together. CPU/memory pressure degrades interactive search and replication lag grows.

Repair: separate workload classes, throttle backfills, stage migrations and reserve capacity for serving.

Exact-baseline drift

Teams compare ANN to an “exact” table that is itself missing newly ingested vectors or uses another filter version.

Repair: version the evaluation corpus and eligibility rules; prove baseline population equality before computing recall.

Score-threshold mythology

A universal cosine threshold of 0.8 is used across languages, models and query types. Good results disappear in one domain while weak results pass in another.

Repair: calibrate thresholds per representation/task or use ranking/reranking and no-answer evaluation rather than raw score folklore.

Healthy service, empty authority

The system retrieves semantically relevant pages but all are unofficial or outdated. RAG answers confidently from weak sources.

Repair: model authority and edition state in metadata/reranking; test authoritative-evidence recall separately from generic relevance.

The pattern across these failures is consistent: vector similarity is only one layer. Trust requires identity, version, eligibility, approximation, lifecycle and evidence to remain aligned.

32. A vector-search design checklist

1. Reader job: What decision or retrieval task is the system serving?
2. Object: What exactly is embedded—document, chunk, image, product, user, code function, event?
3. Identity: What stable ID connects the vector to its authoritative source?
4. Version: How are source edition, chunking edition and embedding edition represented?
5. Embedding contract: Which model, dimensions, preprocessing and normalisation apply?
6. Metric: Cosine, dot product, L2 or another metric—and why?
7. Exact baseline: Can exact search be run on a representative population for evaluation?
8. ANN choice: Why HNSW, IVF, flat/exact or another structure?
9. Quality target: Which Recall@k or candidate-recall threshold matters downstream?
10. Filters: Which conditions are hard eligibility versus ranking preferences?
11. Filter indexes: Which metadata fields need scalar/payload indexing?
12. Filtered recall: Has ANN been tested under restrictive filters?
13. Hybrid: Which query classes require lexical evidence?
14. Fusion: RRF, weighted scores, learned fusion or reranker-only combination?
15. Reranking: How many candidates and what latency budget?
16. Diversity: Should near-duplicates be suppressed or grouped?
17. Freshness: How quickly must source changes become searchable?
18. Upsert ordering: How are stale source versions rejected?
19. Deletion: What makes a record immediately ineligible and when is physical cleanup done?
20. Tenancy: Logical filter, namespace, collection or shard?
21. Access: How are ACLs derived from authenticated identity?
22. Sharding: Which key minimises unnecessary fan-out without creating hotspots?
23. Replication: What consistency is required for updates/deletes?
24. Compression: Which quantisation is acceptable under measured recall?
25. Capacity: What are current vector bytes, index bytes, growth and rebuild duration?
26. Observability: Are latency, source lag, vector versions, recall and zero-result rates visible?
27. Evaluation: Is there a labelled, versioned query/relevance set?
28. Migration: Can a new embedding model run in parallel without mixing coordinate systems?
29. Recovery: Can the index be rebuilt/restored with deletion and permission state intact?
30. Exit: Can the organisation migrate away or simplify if a dedicated vector platform stops earning its cost?

This checklist is intentionally broader than an ANN benchmark. Vector search is a production data system; the hard failures frequently live outside the nearest-neighbour algorithm.

33. Evaluation and benchmark checklist: prove each layer separately

A benchmark should answer a decision, not create a decorative score.

Corpus freeze

• Record corpus/source snapshot or version range.
• Record embedding model/version and vector count.
• Record metadata/filter schema.
• Record deleted/superseded treatment.

Query set

• Use real or realistically sampled user intents.
• Separate semantic, exact-ID, multilingual, filtered, ambiguous and no-answer classes.
• Hold out a portion from tuning.
• Version the query set.

Relevance judgments

• Define relevance levels.
• Identify authoritative source expectations where authority matters.
• Use more than one reviewer for consequential domains.
• Record disagreements/adjudication.

Exact vector baseline

• Run exact search on the same eligible population.
• Verify metric and embedding edition match production.
• Compute ANN Recall@k at candidate depth used downstream.

Filtered tests

• Broad filters.
• Selective filters.
• Low-correlation filters.
• Multi-field filters.
• Small/large tenants.

Hybrid tests

• Dense-only.
• Lexical-only.
• Hybrid fusion.
• Hybrid + reranker.
• Per query-class comparison.

Ranking metrics

• Precision@k.
• Recall@k.
• MRR.
• nDCG.
• Hit rate.
• Authoritative-evidence recall where relevant.

System metrics

• p50/p95/p99 latency.
• QPS at realistic concurrency.
• CPU/RAM/I/O.
• index build/rebuild time.
• ingestion/update throughput.
• source-to-index lag.
• cost per completed retrieval.

Failure tests

• missing metadata;
• stale embedding model;
• index node/shard failure;
• source update during backfill;
• delete during query/cache lifetime;
• ACL revocation;
• embedding service outage;
• reranker outage;
• lexical index lag;
• restore from snapshot.

Benchmark reports should name what they do not establish. High ANN recall does not establish human relevance. High nDCG on a static dataset does not establish freshness. Low latency under one client does not establish concurrency capacity. A comprehensive report keeps those claims separate.

34. Migration and rebuild checklist: change the search system without losing the corpus

Major vector-search changes often involve a full-corpus backfill. Treat them as controlled migrations.

Before migration

1. Freeze/identify source population and active deletion rules.
2. Define old and new embedding/index configurations.
3. Estimate compute, storage and completion time.
4. Reserve serving capacity.
5. Prepare a parallel index/collection/field.
6. Define benchmark and release gates.
7. Define rollback window.

During backfill

1. Embed/index records with source-version guards.
2. Throttle background work.
3. Track progress by authoritative population, not only job count.
4. Handle source updates arriving during migration.
5. Reconcile missing/failed records.
6. Preserve deletion/tombstone events.

Before cutover

1. Population reconciliation passes.
2. All critical metadata indexes exist.
3. Exact retrieval relevance benchmark passes.
4. ANN recall passes.
5. Filtered recall passes.
6. Hybrid/reranker evaluation passes.
7. Latency/capacity tests pass.
8. ACL/tenant negative tests pass.
9. Recovery snapshot/rollback path is ready.

Cutover

1. Shadow traffic first where possible.
2. Canary a bounded user/query population.
3. Compare old/new result sets and application outcomes.
4. Switch alias/router only when gates pass.
5. Keep old index read-only through rollback window.

After cutover

1. Monitor query-class relevance.
2. Monitor zero-result and filtered-recall regressions.
3. Monitor source-to-index lag.
4. Confirm all writers target new edition.
5. Retire old vectors/index under lifecycle policy.
6. Update documentation and consumer dependencies.

A migration is complete only when the old representation is no longer a hidden dependency and the new one can be recovered independently.

35. Teaching workshop: diagnose ten vector-search systems

Each case changes one part of the pipeline. The task is to identify the first weak link rather than prescribe a fashionable product.

Case 1 — The small catalogue

A school resource library has 8,000 documents. It receives 20 searches per minute. Filters reduce most queries to fewer than 2,000 documents. The team proposes a distributed HNSW service.

Suggested reasoning: benchmark exact vector search in the existing governed database first. At this scale and selectivity, exact search may meet latency with far less operational complexity. A dedicated ANN service earns its place only if measurements show a real need.

Case 2 — The perfect ANN, bad embeddings

ANN Recall@10 versus exact is 0.995, but users say search results are irrelevant.

Suggested reasoning: the ANN is faithfully approximating the vector geometry. Investigate embedding model, input representation, chunking and human relevance. Increasing efSearch will not repair a poor semantic representation.

Case 3 — The selective filter

Global search has Recall@10 of 0.98. Queries filtered to one rare product version return only 2 results although exact filtered search finds 10 strong candidates.

Suggested reasoning: filtered ANN traversal is starving. Test filter-aware search, iterative scanning or exact fallback for small candidate sets. Measure filtered recall as its own metric.

Case 4 — The model upgrade

Documents were embedded with model V1. A new API version generates V2 query embeddings with the same dimension. Results suddenly become strange.

Suggested reasoning: dimension compatibility is insufficient. Query and corpus are in different coordinate systems. Restore V1 queries or build/evaluate a V2 corpus index before cutover.

Case 5 — The stale policy

RAG cites a superseded school policy although the current document is present in the CMS.

Suggested reasoning: check source-to-index synchronisation and active-edition metadata before HNSW. The vector index may be healthy but stale. Verify whether old chunks remain active alongside the current edition.

Case 6 — The cross-tenant leak

A user from tenant A receives one title belonging to tenant B after a metadata backfill.

Suggested reasoning: treat as an access-control incident. Stop/contain affected search path; verify missing/corrupt tenant metadata and query authorisation. Do not categorise it as a minor ranking error. Add fail-closed defaults and negative regression tests.

Case 7 — The identifier query

Users search “XR-5521 rev C”. Dense vector results return conceptually similar products but miss the exact item.

Suggested reasoning: use lexical/exact-field retrieval or hybrid search. Embeddings are not required to encode arbitrary rare identifiers strongly. Evaluate identifier queries separately.

Case 8 — The duplicate context

A RAG request retrieves eight chunks; seven are overlapping paragraphs from the same document.

Suggested reasoning: candidate relevance can be high while context diversity is poor. Group by parent document, deduplicate, apply MMR/crowding or retrieve parent sections after one chunk is selected.

Case 9 — The deleted document after restore

A document deleted on Monday reappears after restoring a Sunday vector snapshot.

Suggested reasoning: recovery did not replay post-snapshot deletion state. Apply tombstone/change logs or rebuild from authoritative current source before serving. Add deletion-after-restore test.

Case 10 — The expensive “fast” search

ANN p95 is 20 ms, but complete retrieval p95 is 480 ms.

Suggested reasoning: break down the path. Query embedding, hybrid lexical retrieval, network payload or reranker can dominate. Vector-index optimisation may be irrelevant to the user’s latency.

Transfer questions

For any vector-search problem, ask:

1. Is the relevant item represented/indexed at all?
2. Is it eligible under filters/access?
3. Can exact search find it?
4. Can ANN approximate exact search?
5. Can lexical retrieval find evidence vectors miss?
6. Does fusion/reranking preserve the right candidates?
7. Is the source current and authoritative?
8. Can the result be traced to a stable source edition?

This sequence prevents premature tuning of the wrong layer.

36. The deeper principle: similarity is not authority, and speed is not trust

Vector search became important because learned representations make a powerful operation possible: ask for things that are like this even when they do not share the same words. That changes search, recommendation and AI retrieval profoundly.

But similarity is only one relationship among many that a production system must preserve.

A record can be semantically close and belong to the wrong tenant. It can be relevant and superseded. It can be current and unauthoritative. It can be authoritative and filtered out by a bad index. It can be the exact nearest neighbour and still not answer the user’s question.

That is why a mature vector database is not defined by the existence of an HNSW graph. It is defined by the alignment of six layers:

1. Representation: compatible embedding geometry.
2. Population: correct, current source records.
3. Eligibility: metadata and permissions define who may compete.
4. Search: exact or approximate access path meets recall/latency needs.
5. Ranking: hybrid and reranking reflect the user job.
6. Lifecycle: update, delete, migration, observation and recovery preserve the first five layers.

The first principle of vector-search engineering is therefore not “use embeddings”. It is:

Know exactly what population, representation and authority your nearest-neighbour operation is searching.

The second principle is:

Use exact search to understand the geometry before approximation hides work from you.

The third is:

Measure filtered candidate recall at the depth your downstream reranker actually needs.

The fourth is:

Keep lexical evidence for the things embeddings are not designed to remember perfectly: identifiers, rare names, exact phrases and syntax.

The fifth is:

Treat embedding migrations like coordinate-system migrations, not ordinary model-version toggles.

And the sixth is:

Declare recovery complete only when the restored search system retrieves the right, current, authorised evidence—not when the endpoint merely responds.

Return to Northstar Manuals. The useful system is not the one that can search 20 million chunks. It is the one that can explain why one chunk was eligible, which embedding edition placed it near the query, whether HNSW found it relative to the exact baseline, whether BM25 contributed the identifier match, why the reranker preferred it, which source version it cites, and whether that whole chain remains valid after tomorrow’s update.

That is the data-management job of a vector database.

Primary-source reading guide

The following current primary documentation supports the product-specific mechanisms used as examples. It is not a product ranking or endorsement. Service behaviour changes; check the exact version and deployment before relying on a capability.

• pgvector documentation — vector types, exact nearest-neighbour search, HNSW, IVFFlat, filtering/iterative scans and recall monitoring.
• Weaviate — Filtering — inverted-index allow lists, filtered HNSW and ACORN-style filtering concepts.
• Weaviate — Hybrid search — vector + BM25 retrieval and fusion controls.
• Weaviate 1.39 release — current 2026 examples of MMR, quantisation and HNSW operational evolution.
• Qdrant — Indexing — payload indexes, filtered HNSW and tenant-aware indexing concepts.
• Qdrant — FAQ / fundamentals — ANN approximation behaviour, result-limit effects and multitenancy/sharding guidance.
• Databricks — Mosaic AI Vector Search — vector search service and governed-data integration.
• Databricks — Query a vector search index — ANN/full-text/hybrid query modes and reranking.
• Google Cloud Vertex AI Vector Search — Filtering — numeric/token restrictions and crowding/diversity controls.
• Milvus — Hybrid Search API — multiple ANN requests, scalar filters and reranking in current API documentation.

Continue through the Data Management library

• What Is Data Management?
• Retrieval Representation
• Data Indexing and Query Optimisation
• Token Budgets and Chunking
• AI Data Management
• Data Access Governance and Entitlements
• Data Deletion and Destruction Verification
• Data Observability and Monitoring
• Data Partitioning and Sharding
• Data Synchronisation and Reconciliation
• Data Versioning and Change Management

Data Management Series · DATA.MANAGEMENT.066 · Educational technical edition.