Guidelines for search index creation to support auto complete / suggestions by maneuvertomars · Pull Request #2217 · blevesearch/bleve

maneuvertomars · 2025-08-20T08:20:01Z

Did a write up for search-autocomplete.
Comprehensive documentation for implementing search autocomplete functionality using edge n-gram tokenization in Bleve search engine. It provides detailed analysis of different tokenization methods and demonstrates why edge n-grams are optimal for autocomplete features.

Key changes:

Adds detailed documentation on edge n-gram autocomplete implementation
Compares various tokenization methods (single token, whitespace, regex, n-gram, edge n-gram)
Provides practical code examples and configuration samples

docs/search_autocomplete.md -> Dsicussion about edge n-gram autocomplete theory, implementation examples, and json mappings

docs/create_and_search_your_first_index.md -> Basic Bleve index creation and search operations with incomplete ending section

Copilot

Pull Request Overview

The PR introduces comprehensive documentation for implementing search autocomplete functionality using edge n-gram tokenization in Bleve search engine. It provides detailed analysis of different tokenization methods and demonstrates why edge n-grams are optimal for autocomplete features.

Key changes:

Adds detailed documentation on edge n-gram autocomplete implementation
Compares various tokenization methods (single token, whitespace, regex, n-gram, edge n-gram)
Provides practical code examples and configuration samples

Reviewed Changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 3 comments.

File	Description
docs/search_autocomplete.md	New comprehensive guide covering edge n-gram autocomplete theory, implementation examples, and best practices
docs/create_and_search_your_first_index.md	New tutorial on basic Bleve index creation and search operations with incomplete ending section

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-08-20T15:33:57Z

+3. **Better caching**: Exact term queries cache better than prefix queries
+4. **Consistent performance**: Query time doesn't increase with index size
+
+## 5. On low level implementaion sample:


Typo in 'implementaion' should be 'implementation'.

Suggested change

## 5. On low level implementaion sample:

## 5. On low level implementation sample:

Copilot · 2025-08-20T15:33:58Z

+    for _, token := range input {
+        runeCount := utf8.RuneCount(token.Term)
+        runes := bytes.Runes(token.Term)
+        // ..builds tokens based form either end, specified in the input


Typo in 'form' should be 'from'.

Suggested change

// ..builds tokens based form either end, specified in the input

// ..builds tokens based from either end, specified in the input

Copilot · 2025-08-20T15:33:58Z

+n gram token filter
+
+
+


The ending section appears to be incomplete notes rather than proper documentation. This section should either be completed with proper documentation or removed.

Suggested change

capemox · 2025-08-20T17:07:11Z

+```
+
+### search autocomplete
+return the response for 


@maneuvertomars you can add a link here to your search_autocomplete.md file, so its convenient for readers

abhinavdangeti · 2025-08-20T21:38:11Z

@maneuvertomars It's our practice to be as descriptive as possible in the commit message section about what the commit covers. This'll benefit us in the future when we look at the commit log.

abhinavdangeti · 2025-08-21T14:00:36Z

+      "autocomplete": {
+        "analyzer": "edge_ngram_analyzer",
+        "min_gram": 3,
+        "max_gram": 20


abhinavdangeti · 2025-08-21T14:00:56Z

+      "autocomplete": {
+        "analyzer": "edge_ngram_analyzer",
+        "min_gram": 2,
+        "max_gram": 12


Syntax is off - i think it is just min and max in the config.

abhinavdangeti · 2025-08-21T14:01:12Z

+```json
+{
+  "min_gram": 2,
+  "max_gram": 15


Syntax is off - i think it is just min and max in the config.

abhinavdangeti · 2025-08-21T14:01:17Z

+```json
+{
+  "min_gram": 1,
+  "max_gram": 20  


…plete including stepwise flow

CascadingRadium · 2025-08-28T10:30:56Z

@@ -0,0 +1,221 @@
+# Create and Search Index


Creating a Bleve Index

CascadingRadium · 2025-08-28T10:33:46Z

@@ -0,0 +1,221 @@
+# Create and Search Index
+
+Demonstration of creating an index on Documents and making it searchable.


A simple how-to example using Bleve in Go to create an index, add documents, and run search queries with results.

CascadingRadium · 2025-08-28T10:35:17Z

+import (
+	"fmt"
+	"log"
+	"os"


"os" package is not used, must be removed

CascadingRadium · 2025-08-28T11:04:55Z

+	// Search the created index
+	query := bleve.NewQueryStringQuery("bleve")
+	searchRequest := bleve.NewSearchRequest(query)
+	searchRequest.Size = 10
+	searchResult, err := index.Search(searchRequest)
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	for i, hit := range searchResult.Hits {
+		fmt.Printf("%d. Document: %s (Score: %.2f)\n", i+1, hit.ID, hit.Score)
+		if len(hit.Fragments) > 0 {
+			for field, fragments := range hit.Fragments {
+				fmt.Printf("   %s: %s\n", field, fragments[0])
+			}
+		}
+		fmt.Println()
+	}
+}


simplify query

// Search the created index query := bleve.NewMatchQuery("bleve") searchRequest := bleve.NewSearchRequest(query) searchRequest.Explain = true searchRequest.Fields = []string{"title", "content"} searchResult, err := index.Search(searchRequest) if err != nil { log.Fatal(err) } fmt.Println(searchResult)

CascadingRadium · 2025-08-28T11:12:39Z

+	}
+}
+```
+### Output:


add a space between the code and the Output - also ## Output:

CascadingRadium · 2025-08-28T12:00:28Z

+**Field configuration explained:**
+- `"analyzer": "search_autocomplete_feature"` - Use our custom analyzer
+- `"store": true` - Keep original text for display
+- `"index": true` - Make it searchable
+- `"include_in_all": true` - Include in default search field


Not required, as the Bleve API will allow users to modify these options

CascadingRadium · 2025-08-28T12:01:18Z

+- `"index": true` - Make it searchable
+- `"include_in_all": true` - Include in default search field
+
+![Index Mapping Configuration](/docs/name_filed_searchable_search_autocomplete_analyzer.png "Index mapping showing how the name field is configured with the custom analyzer")


remove Couchbase UI

CascadingRadium · 2025-08-28T12:03:34Z

+**User types "sc":**
+1. Query: `name:sc`
+2. Bleve looks up exact term "sc" in the index
+3. Finds document with "Schaumbergfest" 


and "Script" right?

Finds documents with "Schaumbergfest" and "Script"

CascadingRadium · 2025-08-28T12:04:51Z

+	ID      string `json:"id"`
+	Title   string `json:"title"`
+	Content string `json:"content"`
+	Author  string `json:"author"`


Author is unused

CascadingRadium · 2025-08-28T12:07:04Z

+3. Finds document with "Schaumbergfest" 
+4. Returns suggestion instantly
+
+![Search Results](/docs/index_search_using_prefix.png "Search results showing 'Schaumbergfest' highlighted when searching for 'sc'")


Replace Couchbase UI with code

type Document struct { ID string `json:"id"` Title string `json:"title"` } // 4. Index Documents documents := []Document{ { ID: "doc1", Title: "Schaumbergfest", }, { ID: "doc2", Title: "Script", }, } batch := index.NewBatch() for _, doc := range documents { batch.Index(doc.ID, doc) } if err := index.Batch(batch); err != nil { log.Fatal(err) } // 5. Search the created index query := bleve.NewMatchQuery("sc") query.SetField("title") searchRequest := bleve.NewSearchRequest(query) searchRequest.Explain = true searchRequest.Fields = []string{"title"} searchResult, err := index.Search(searchRequest) if err != nil { log.Fatal(err) } fmt.Println(searchResult)

Output

$ go run main.go 2 matches, showing 1 through 2, took 311.125µs 1. doc2 (0.343255) title Script 2. doc1 (0.343255) title Schaumbergfest

@CascadingRadium

all previous reviews fixed and got it re-reviewed from @CascadingRadium

Search autpcomplete using suggestions

a897942

abhinavdangeti requested review from CascadingRadium, Likith101, Thejas-bhat, abhinavdangeti, capemox and Copilot August 20, 2025 15:33

Copilot AI reviewed Aug 20, 2025

View reviewed changes

abhinavdangeti changed the title ~~Search autocomplete using suggestions~~ Guidelines for search index creation to support auto complete / suggestions Aug 20, 2025

abhinavdangeti added this to the v2.5.4 milestone Aug 20, 2025

capemox reviewed Aug 20, 2025

View reviewed changes

Fixed typos

bdd9635

abhinavdangeti previously requested changes Aug 21, 2025

View reviewed changes

Added mappings and different token filters involved in search autocom…

9f8ef7e

…plete including stepwise flow

maneuvertomars requested a review from abhinavdangeti August 22, 2025 15:27

CascadingRadium requested changes Aug 28, 2025

View reviewed changes

maneuvertomars added 3 commits August 29, 2025 15:23

Removed CB UI png and added runnable code

456c032

Added identation in code

391f9d6

Fixed identation

ed81df1

CascadingRadium approved these changes Aug 29, 2025

View reviewed changes

capemox approved these changes Sep 3, 2025

View reviewed changes

maneuvertomars merged commit 674e516 into master Sep 3, 2025
9 checks passed

CascadingRadium deleted the search-autocomplete-writeup branch May 5, 2026 18:22

	## 5. On low level implementaion sample:
	## 5. On low level implementation sample:

	// ..builds tokens based form either end, specified in the input
	// ..builds tokens based from either end, specified in the input

		@@ -0,0 +1,221 @@
		# Create and Search Index

		Demonstration of creating an index on Documents and making it searchable.

Conversation

maneuvertomars commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abhinavdangeti commented Aug 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

maneuvertomars commented Aug 20, 2025 •

edited

Loading