Full-Text Search

Now that we have covered the simple case of searching for structured data, it is time to explore full-text search: how to search within full-text fields in order to find the most relevant documents.

The two most important aspects of full-text search are as follows:

Relevance

The ability to rank results by how relevant they are to the given query, whether relevance is calculated using TF/IDF (see [relevance-intro]), proximity to a geolocation, fuzzy similarity, or some other algorithm.

Analysis

The process of converting a block of text into distinct, normalized tokens (see [analysis-intro]) in order to (a) create an inverted index and (b) query the inverted index.

As soon as we talk about either relevance or analysis, we are in the territory of queries, rather than filters.

Term-Based Versus Full-Text

While all queries perform some sort of relevance calculation, not all queries have an analysis phase. Besides specialized queries like the bool or function_score queries, which don’t operate on text at all, textual queries can be broken down into two families:

Term-based queries

Queries like the term or fuzzy queries are low-level queries that have no analysis phase. They operate on a single term. A term query for the term Foo looks for that exact term in the inverted index and calculates the TF/IDF relevance _score for each document that contains the term.

It is important to remember that the term query looks in the inverted index for the exact term only; it won’t match any variants like foo or FOO. It doesn’t matter how the term came to be in the index, just that it is. If you were to index ["Foo","Bar"] into an exact value not_analyzed field, or Foo Bar into an analyzed field with the whitespace analyzer, both would result in having the two terms Foo and Bar in the inverted index.

Full-text queries

Queries like the match or query_string queries are high-level queries that understand the mapping of a field:

  • If you use them to query a date or integer field, they will treat the query string as a date or integer, respectively.

  • If you query an exact value (not_analyzed) string field, they will treat the whole query string as a single term.

  • But if you query a full-text (analyzed) field, they will first pass the query string through the appropriate analyzer to produce the list of terms to be queried.

Once the query has assembled a list of terms, it executes the appropriate low-level query for each of these terms, and then combines their results to produce the final relevance score for each document.

We will discuss this process in more detail in the following chapters.

You seldom need to use the term-based queries directly. Usually you want to query full text, not individual terms, and this is easier to do with the high-level full-text queries (which end up using term-based queries internally).

Note

If you do find yourself wanting to use a query on an exact value not_analyzed field, think about whether you really want a scoring query, or if a non-scoring query might be better.

Single-term queries usually represent binary yes/no questions and are almost always better expressed as a filter, so that they can benefit from caching:

GET /_search
{
    "query": {
        "constant_score": {
            "filter": {
                "term": { "gender": "female" }
            }
        }
    }
}

results matching ""

    No results matching ""