That is a useful starting point. It is not necessarily a complete search system.
While developing an AI-powered search application with Azure AI Search, we found that the quality of the experience depended on much more than embedding similarity. The underlying data included descriptive text, structured attributes, geographic information, and useful information contained in images. User searches could be precise or ambiguous, and sometimes the most relevant result did not use the same terminology as the query.
The solution evolved into a hybrid retrieval pipeline that combined traditional search, vector retrieval, image-derived information, domain-aware query expansion, filtering, and application-level result ranking. The important lesson was that good AI search comes from the retrieval architecture around the model as much as from the model itself.
Why Vector Search Alone Wasn't Enough
Semantic similarity is valuable because users do not always search using the exact words contained in the source data. Embeddings provide a way to retrieve conceptually related information even when there is little direct keyword overlap.
But keyword matching still matters.
Specific names, attributes, terminology, and other literal details can be strong relevance signals. A traditional lexical search engine is particularly good at recognizing those matches. Replacing it entirely with vector retrieval can discard useful information.
Azure AI Search gave us both pieces: conventional text search and vector search. Rather than choosing one, we treated them as complementary retrieval methods.
The basic pattern became:
- Break longer text into manageable chunks of roughly 500 tokens.
- Generate embeddings for those chunks.
- Store the vectors alongside searchable text and metadata in Azure AI Search.
- Generate an embedding for the incoming query.
- Retrieve semantically similar candidates using vector search.
- Retrieve strong literal matches using lexical search.
- Combine those signals before producing the final ranked results.
This provided a much stronger foundation than treating the vector database as the entire search engine.
Making Search Understand the Domain
A second challenge was vocabulary.
Users do not necessarily know how information is described in the underlying dataset. Two people looking for essentially the same thing may phrase their searches very differently.
We addressed that at the application layer with domain-specific query expansion and an ontology of related terminology. Instead of blindly sending one search string to the index, the system could recognize useful concepts and broaden retrieval with closely related terms.
That distinction is important.
An embedding model can identify semantic similarity, but domain knowledge can tell the retrieval system which relationships are specifically useful for the application. Those are not always the same thing.
We also placed limits on query expansion. More search variants do not automatically produce better results. Expanding too aggressively can introduce unrelated candidates and dilute the strongest matches. The system therefore treated expansion as a controlled retrieval strategy rather than simply generating as many related terms as possible.
Fuzzy matching provided another layer for cases where spelling or terminology did not line up perfectly with the indexed data.
Together, these techniques let the search experience tolerate more variation in how users describe what they want without abandoning the precision of traditional search.
Treating Images as Searchable Information
Text was only part of the available information.
Images can contain details that are valuable to a user but absent from a written description. If those details never become part of the searchable representation, a text-only search system has no way to use them.
Our indexing pipeline therefore incorporated image-derived text alongside the primary textual data. That information could be embedded and searched independently rather than simply being appended indiscriminately to the main description.
Keeping text-derived and image-derived retrieval signals distinct gave us more control over how each contributed to the final result.
This is an important architectural principle for multimodal search: converting an image into text does not mean every extracted observation should be treated as equally authoritative as the source's written metadata.
Instead, the different sources can remain separate during retrieval and be combined intentionally during ranking.
Combining Multiple Retrieval Paths
Once a search system has lexical results, text-vector results, expanded-query results, and image-derived results, another problem appears: how should they be combined?
The scores generated by different retrieval methods are not necessarily directly comparable. A lexical relevance score and a vector similarity score represent different things.
Rather than pretending those values were interchangeable, we combined ranked result sets using weighted Reciprocal Rank Fusion.
RRF works from the position of a result within each ranked list rather than requiring every retrieval method to produce scores on the same scale. A document that performs well across multiple retrieval strategies can therefore rise in the combined ranking.
Weighting also allowed the application to control the relative importance of different retrieval paths.
This gave us a practical way to combine:
- lexical relevance,
- text-vector similarity,
- expanded semantic searches, and
- image-derived vector matches.
Duplicate suppression was applied as those result sets came together so that the same underlying result did not consume multiple positions simply because several retrieval paths found it.
The result was not one search algorithm. It was a coordinated set of retrieval strategies feeding a final ranking process.
Search Quality Also Depends on Understanding the Query
Retrieval improvements were only one side of the problem.
Some user input contains information that should not simply be embedded as an undifferentiated sentence. Geographic intent is one example. When a query contains a location, recognizing that information explicitly gives the application an opportunity to treat geography as structured search context rather than hoping semantic similarity captures it correctly.
The search layer therefore included geographic parsing and filtering alongside its textual retrieval logic.
We also added mechanisms for refinement so that a search could build on previous user intent instead of forcing every interaction to start from zero.
These application-level capabilities illustrate an important division of responsibility. Azure AI Search provides powerful retrieval primitives, including lexical and vector search. The application still needs to decide how those primitives should be used for its particular domain.
Designing for Imperfect Searches
A useful search experience also needs to handle the absence of an obvious perfect match.
Rigid retrieval thresholds can make a system appear confident when results are strong but strangely empty when a user's wording falls just outside the expected range. At the other extreme, removing quality thresholds altogether can return weak results that happen to be the closest available matches.
We implemented result-floor relaxation so retrieval behavior could adapt when the initial search did not produce enough useful candidates.
The objective was not to manufacture relevance. It was to give the search process room to consider reasonable secondary candidates when a strict first pass was too restrictive.
Oversampling served a related purpose. Retrieving a broader candidate pool before final ranking allowed the application-level fusion process to make a better decision about which results ultimately deserved the highest positions.
This is one of the advantages of separating candidate retrieval from final ranking. The first stage can prioritize recall while the later stages restore precision.
Building Reliability Into the Data Pipeline
Search quality depends on the index being correct and current.
The system therefore needed more than a successful initial indexing run. The data pipeline was designed so that source information could be processed, embeddings generated, and updated records uploaded to the Azure AI Search indexes as the underlying dataset changed.
We later automated that refresh process using a scheduled Azure containerized job. This separated recurring index maintenance from a developer workstation or manually initiated process and provided a repeatable path for keeping the search data current.
We also added validation around embedding dimensions. Vector fields in a search index have a defined dimensionality, so mismatches between the embedding model and index configuration need to be detected rather than allowed to become subtle retrieval problems.
These pieces are less visible than the search interface, but they are part of the same engineering problem. A sophisticated ranking strategy does not help if its index is stale or its vector data is inconsistent.
What We Learned
The biggest lesson from this work was that an AI search application should not be designed around a single retrieval technique.
Vector search is powerful, but it works best as part of a broader system.
Traditional lexical search remains valuable for exact terminology. Embeddings help bridge differences in language. Domain-aware expansion adds knowledge that a general-purpose model may not have. Structured parsing can recognize intent that should become a filter instead of a similarity calculation. Image processing can expose information that would otherwise be invisible to a text search engine. Fusion provides a way to bring those independent signals together.
Just as importantly, the surrounding engineering matters: chunking strategy, candidate counts, deduplication, thresholds, index validation, and repeatable data refreshes all influence what users ultimately see.
The architecture that emerged can be summarized as a pipeline:
Understand the query → retrieve through multiple paths → combine the evidence → remove duplication → rank the candidates → return useful context.
That is a more useful way to think about production AI search than simply asking which embedding model to use.
Next Steps
Organizations evaluating AI search should start by looking at the information their users actually need to find and the ways they naturally ask for it.
If exact terminology matters, preserve lexical search. If users describe concepts rather than known keywords, add vector retrieval. If important information exists in images or other unstructured sources, determine how that information can become searchable. If the domain has its own vocabulary, consider whether the application needs an explicit layer of domain knowledge.
Azure AI Search provides a strong set of building blocks for this approach, but the quality of the finished experience comes from how those pieces are assembled.
For CBITS, this project reinforced a development principle we continue to apply to AI and automation work: use AI where it improves the system, preserve deterministic techniques where they are stronger, and design the architecture around the problem rather than around a single technology.