AI-Powered Hybrid Search for Smart Search

AI-Powered Hybrid Search for Smart Search is a new natural language search feature that unlocks a GPT-like search experience for its users.

About

AI-Powered Hybrid Search is a test feature that improves Smart Search by combining traditional keyword-based search with a natural language understanding search, or, “Semantic Search”. This approach uses Machine Learning to produce more relevant search results. In Hybrid Search, the basic keyword search results are combined with the results from our Semantic Search, offering a more comprehensive search experience.

• Lexical Search – WordPress’ standard search looks to match the exact word entered into the search bar, with that same exact word in the content it returns in the search results. This of course ensures high precision, exact-match results, but also is highly inflexible (remember: exact-match!) and tends to return far fewer results.
• Full Text Search – WP Engine Smart Search allows for more flexibility with what can be entered into the search box, in order to get more results. Smart Search can handle misspellings and simplify words to their base forms, unlike default WordPress (“exact-match”) search. This method finds a balance between finding exact matches and getting more related results, though it might include some unrelated ones.
• Semantic Search – AI-Powered Hybrid Search enhances search by focusing on the meaning of words or phrases within context, not just their spelling. It finds more relevant results by analyzing words used in similar contexts, and inferring user intent. This is also called “Natural Language” search, and is accomplished by using recent advances in Machine Learning technology.
• Hybrid Search – Combines Full Text and Semantic Search to offer a balance of getting more relevant results (higher recall) and maintaining accuracy.

Full Text, Semantic, and Hybrid Search

Full Text Search and Semantic Search have their advantages and disadvantages. Whether or not your website can benefit from one or the other depends on the precision of your content itself and the style of search phrasing your users may favor.

In general; Full Text Search should be used when queries are specific (“Nike 504s”), Semantic Search should be used when queries are phrase- or natural-language based (“I tend to overpronate when I run, what kind of shoes should I buy?”), and Hybrid Search should be used to achieve the best of both worlds (“Nike shoes for shin splints”).

For example, imagine an eCommerce store selling sporting goods:

• The search query “soccer ball size 4” will achieve better results as a Full Text Search than a Semantic Search. This is because the search query itself is designed to yield a high precision result, by specifying type and size.
• The search query “soccer ball suitable for young children” will achieve better as a Semantic Search than a Full Text Search. This is because the search query is designed for a less precise and higher recall type search, by specifying type and age group.
• The search query “durable soccer ball for kids under 10” will achieve better results as a Hybrid Search. Specifying “durable soccer ball” identifies the type of product and a desired quality, while “kids under 10” provides additional context by specifying the target age group. 

Enable

AI-Powered Hybrid Search is currently in closed beta. After an account has been added to the closed beta this feature will be assigned to your account while working with our team. To access AI-Powered Hybrid Search for Smart Search:

1. Open the wp-admin dashboard
2. Expand the WP Engine Smart Search menu
3. Click AI-Powered Hybrid Search
   • If you are part of the Beta but this menu option is unavailable, release the Smart Search license, then add it back to the environment.
4. Toggle AI-Powered Hybrid Search to Enabled
5. After adjusting Search Bias and Fields, click Save and Configure

Search Bias

Use the Search Bias slider to adjust between Full Text Search, Hybrid, and Semantic Search. Adjusting the search bias impacts the order that relevant results are returned.

• Full text search only (Slider fully to the left) – Smart Search default. Uses a mix of precise terms, keyword stemming, and misspelled word logic. Results are ranked based on how closely they match your exact terms.
• Mix of both (Slider in the middle) – Full Hybrid Search: a combination of Full Text Search and Semantic Search. Uses a mix of precise terms, keyword stemming, and misspelled word logic, while also assessing words/phrases in context. Results are then ranked by a combination of term precision and contextual relevance, offering a mix of exact and semantically related results.
• Semantic search only (Slider fully to the right) – AI-Powered Hybrid Search only. Assess words/phrases in context. Results are ranked based on their contextual relevance to your query, which may bring more varied but relevant results to the top.

Fields

Select one field from the website’s database to leverage when using Semantic and Hybrid Search. 

Full Text Search analyzes all fields. AI-Powered Search, however, can be resource prohibitive. Therefore specifying a field from the database will help “define” where more context can be found when using Semantic Search and Hybrid Search functionality without overusing resources.

For example, if the field post_content contains a store’s product descriptions, this field could be a good choice to provide more contextual information about the product for Semantic and Hybrid searches.

Bear in mind that only the first ~500 words of the selected field will be referenced for each instance. If our example store regularly has 2000+ words in the post_content field, another field may be preferable.

---

NEXT STEP: Learn more about Smart Search here.