TF-IDF algorithm for document retrieval

Define the retrieval task

Start with the collection, query source, result unit, and user action. A support-article search, product search, legal-document lookup, and retrieval-augmented generation system have different requirements. Decide whether a result is a complete document, a section, a paragraph, or a fixed-size passage. Decide the unit of indexing before the unit of scoring. A system that indexes whole documents but serves passages will return the right document with the wrong excerpt. Splitting strategy, overlap, and heading context belong in the retrieval configuration, versioned with the tokenizer settings. Record the fields available for retrieval. Titles, headings, identifiers, tags, body text, and metadata often need different weights. Product codes and policy numbers can matter more than repeated body terms. Keep access-control filters outside the relevance score so a high score can never expose an ineligible document. Collect real queries when possible. Include frequent queries, rare terminology, abbreviations, spelling variation, long questions, and queries with no relevant answer. A retrieval system needs a defined response for the last group.

Calculate term frequency and inverse document frequency

Let tf(t,d) represent the frequency of term t in document d. Let N be the number of documents and df(t) the number of documents that contain the term. A common smoothed inverse document frequency is: idf(t) = log((1 + N) / (1 + df(t))) + 1 The TF-IDF weight is: tfidf(t,d) = tf(t,d) × idf(t) The smoothing avoids a division by zero and keeps terms that appear in every document from receiving a zero value in implementations that add one after the logarithm. Libraries vary in their term-frequency scaling, smoothing, vocabulary pruning, and normalization. Record the exact implementation before comparing results. Raw term frequency can let a long document dominate because it repeats a term many times. Alternatives include binary occurrence, logarithmic scaling, or sublinear term frequency. Evaluate the choice against the real collection rather than selecting it from a generic rule.

Work through a small example

Consider three documents: retail demand forecast hourly demand forecast for restaurants retail inventory replenishment The word demand appears in two documents, so it receives less inverse-document weight than hourly, which appears in one. The word retail also appears twice. A query for retail forecast will favor a document that contains both terms after vectorization and normalization. With the smoothed formula above and natural logarithms, the three-document corpus gives: retail: 1 and 3; 2; 1.29. demand: 1 and 2; 2; 1.29. forecast: 1 and 2; 2; 1.29. hourly: 2; 1; 1.69. The query retail forecast scores document 1 above documents 2 and 3, because document 1 contains both query terms and each term carries the same inverse document frequency. If the query were hourly forecast, document 2 would win by a wider margin, because hourly is the rarer term. The worked example stays useful only while the table matches the deployed library's smoothing; recompute it whenever the implementation changes. The exact score depends on tokenization, case handling, stop-word choices, smoothing, and vector normalization. A worked example should therefore use the same library configuration as the deployed service. A spreadsheet calculation and the library output should agree for a small test corpus.

Normalize text with care

Lowercasing can help ordinary prose but can damage case-sensitive identifiers. Removing punctuation can split product codes, versions, and paths. Generic stop-word lists can remove words that matter in a specific domain. Stemming and lemmatization can improve recall while reducing precision for technical terms. Keep a normalization test set: a short list of queries and documents that must survive the pipeline unchanged. Product codes, version strings, email addresses, and error codes should round-trip through tokenization exactly. Run this set on every configuration change. Normalization regressions are silent and common. Choose word n-grams when phrases carry meaning. A bigram such as demand forecast is more specific than either word alone. Character n-grams can improve tolerance for spelling variation and joined words. They also increase the vocabulary and memory requirement. Set minimum and maximum document frequency only after inspecting the corpus. A minimum threshold can remove typing mistakes and one-off noise, but it can also remove rare identifiers. A maximum threshold can remove collection-wide boilerplate, but it can hide a common term that is central to the domain.

Rank with cosine similarity

Transform the query with the same vocabulary and weighting rules as the documents. Cosine similarity compares the angle between the query and document vectors. When vectors are L2-normalized, the dot product gives the cosine similarity. Cosine similarity reduces the effect of document length, but it does not correct every length bias. Chunk size still matters. A short passage with one matching term and a long section with several useful matches may trade places when chunking changes. Treat chunking as part of the retrieval configuration and evaluation. For fielded documents, build separate vectors or apply field weights. A title match may deserve more weight than a body match. Keep those weights in configuration and include them in the evaluation record.

Build relevance judgments

Create a query set and label candidate results. Use graded judgments when some results are fully relevant and others are only useful. Write a rubric that tells reviewers how to handle partial answers, outdated documents, duplicated content, unsafe results, and missing context. Use more than one reviewer for a sample and resolve disagreements. A low agreement rate means the task or rubric is ambiguous. That problem cannot be fixed by tuning a model. Protect a test set from routine tuning. Use a development set to change tokenization, n-grams, thresholds, and weights. Use the held-out set for a final comparison. Record the collection snapshot because document additions and removals change inverse-document frequency.

Measure retrieval quality

Recall at k measures whether the relevant material appears in the first k results. Precision at k measures how much of that result set is relevant. Mean reciprocal rank rewards putting the first relevant result early. Normalized discounted cumulative gain handles graded relevance and position. Choose k from the surface, not from convention. A search page that shows ten results cares about recall at 10 and precision at 10. A retrieval-augmented generation path that feeds three passages to a model cares about recall at 3. Report one primary k and show the curve around it. Measure zero-result and low-score rates. Review queries where no document is relevant. A system should be able to say that it has no supported answer. In a retrieval-augmented generation path, this is a safety control because weak retrieval can produce an unsupported generated response. Report latency, index size, build time, and query cost with relevance. A more accurate configuration that cannot meet the response budget is not a deployable improvement.

Inspect expected failure modes

TF-IDF can miss synonyms, paraphrases, and concepts that share few words. It can overvalue rare misspellings and copied identifiers. Boilerplate can dominate if repeated text is not removed. New vocabulary is invisible until the index is rebuilt or updated. The method can perform very well for exact terminology, model numbers, names, codes, error messages, and technical corpora. Do not assume semantic retrieval is better for every query. Compare results by query class.

Compare with BM25

Okapi BM25 is the other lexical baseline worth testing before any neural retriever. It keeps the inverse-document-frequency idea and adds two controls: term-frequency saturation, so the tenth occurrence of a term adds little over the third, and document-length normalization, so long documents do not dominate by repetition. Two parameters tune these controls. k1 sets the saturation rate, and b sets how strongly length matters. Published defaults, k1 around 1.2 to 2.0 and b around 0.75, are a reasonable starting point. BM25 usually matches or beats plain TF-IDF on natural-language collections, and it powers the lexical side of widely used search engines. That does not end the evaluation. Run both on the same judgments, latency budget, and index constraints, because the margin varies by query class. Exact-code and error-message queries often show little difference. Natural-language questions usually favor BM25.

Operate the index

TF-IDF weights depend on the whole collection, so the index is a versioned artifact. Adding or removing documents changes inverse document frequencies, and therefore changes scores for queries that never touched those documents. Record the collection snapshot with every evaluation, and rebuild or update on a schedule that matches content churn. Decide between full rebuilds and incremental updates. A full rebuild is simple and exact but expensive on a large corpus. Incremental updates are fast but let statistics drift until the next rebuild. Many systems update incrementally through the day and rebuild nightly. Track index size and memory alongside relevance. Character n-grams and large vocabularies grow the index quickly. Minimum document-frequency pruning and vocabulary caps control the growth at some recall cost. A configuration that doubles index memory for a one-point recall gain is a cost decision, and the cost owner should make it.

Use TF-IDF in a modern retrieval system

TF-IDF can remain the primary lexical retriever, a deterministic fallback, a candidate generator, or one part of a hybrid system. Compare it with BM25, dense retrieval, and reranking on the same judgments. A practical hybrid path retrieves lexical and semantic candidates, merges and deduplicates them, applies access and freshness filters, and reranks the remaining passages. Log the source and score of each candidate. That evidence shows whether the semantic path adds useful recall or only more cost. When two retrievers both run, merge them with a documented rule. Reciprocal rank fusion scores each candidate by the sum of 1/(k + rank) across the lists it appears in, with k commonly set to 60. It needs no score calibration between systems, which is its main practical advantage. Keep the raw scores too. Fusion hides them, and debugging a bad result needs the original evidence. For retrieval-augmented generation, evaluate retrieval before evaluating the answer. Track whether the supporting passage is present, whether the response cites it correctly, and whether the system declines when evidence is missing. A strong language model cannot recover a document that the retriever never supplied.

Use a reproducible baseline checklist

A complex retriever should beat the lexical baseline on the queries that matter. If it does not, the team has a data, task-definition, or evaluation problem to solve before adding infrastructure.

The document unit, fields, filters, and query source are defined.
Tokenization, normalization, n-grams, thresholds, and field weights are versioned.
The index and query use the same fitted vocabulary and transforms.
Development and held-out query sets have written relevance judgments.
Results include relevance, latency, index size, and query cost.
Failure review covers synonyms, identifiers, spelling, boilerplate, and no-answer queries.
BM25 is compared on the same judgments, latency, and index constraints.
The rebuild and update schedule matches content churn.
Fusion rules are documented when multiple retrievers run.
More complex retrievers use the same collection, judgments, and operating boundary.

Related BluePi work

Where to read next on the same subject.