Repository navigation
Term vectors for nested fields #91902
Description
Activity
Pinging @elastic/es-search (Team:Search)
Pinging @elastic/es-search-relevance (Team:Search Relevance)
Looks like this limitation where term vectors are not directly accessible for nested fields using the standard _termvectors endpoint. This limitation exists because nested fields are stored as separate Lucene documents, and the term vector API doesn't traverse these nested structures.
Try this:
GET /my-index-000001/_termvectors/1
{
"fields" : ["user.first", "user.last", "user.description"],
"offsets" : true,
"positions" : true,
"term_statistics" : true,
"field_statistics" : true,
"payloads" : false,
"per_field_analyzer" : {
"user.first": "standard",
"user.last": "standard",
"user.description": "standard"
}
}After reproducing the issue:
Response:
{
"_index": "my-index-000001",
"_id": "1",
"_version": 1,
"found": true,
"took": 0,
"term_vectors": {
"user.description": {
"field_statistics": {
"sum_doc_freq": 4,
"doc_count": 2,
"sum_ttf": 4
},
"terms": {
"description": {
"doc_freq": 2,
"ttf": 2,
"term_freq": 2,
"tokens": [
{
"position": 1,
"start_offset": 6,
"end_offset": 17
},
{
"position": 103,
"start_offset": 25,
"end_offset": 36
}
]
},
"first": {
"doc_freq": 1,
"ttf": 1,
"term_freq": 1,
"tokens": [
{
"position": 0,
"start_offset": 0,
"end_offset": 5
}
]
},
"second": {
"doc_freq": 1,
"ttf": 1,
"term_freq": 1,
"tokens": [
{
"position": 102,
"start_offset": 18,
"end_offset": 24
}
]
}
}
},
"user.first": {
"field_statistics": {
"sum_doc_freq": 2,
"doc_count": 2,
"sum_ttf": 2
},
"terms": {
"alice": {
"doc_freq": 1,
"ttf": 1,
"term_freq": 1,
"tokens": [
{
"position": 101,
"start_offset": 5,
"end_offset": 10
}
]
},
"john": {
"doc_freq": 1,
"ttf": 1,
"term_freq": 1,
"tokens": [
{
"position": 0,
"start_offset": 0,
"end_offset": 4
}
]
}
}
},
"user.last": {
"field_statistics": {
"sum_doc_freq": 2,
"doc_count": 2,
"sum_ttf": 2
},
"terms": {
"smith": {
"doc_freq": 1,
"ttf": 1,
"term_freq": 1,
"tokens": [
{
"position": 0,
"start_offset": 0,
"end_offset": 5
}
]
},
"white": {
"doc_freq": 1,
"ttf": 1,
"term_freq": 1,
"tokens": [
{
"position": 101,
"start_offset": 6,
"end_offset": 11
}
]
}
}
}
}
}So, the linked PR: #92568 handles one edge case, but not fully.
Term vectors are still not displayed for more than one nested doc in an artificial doc.
Example:
POST nested_term_vectors_test/_termvectors { "fields": ["nested_field.text"], "doc":{ "nested_field": [ { "text": "Nested text 1" }, { "text": "some other nest" } ] } }This will only show the terms
nestedtestand1.This is likely due to the
seenFieldstracking. We simply take the first seen field with that name and ignore the rest.I am not sure how to actually address this part as when scoring, the nested query will treat the term frequencies as individual for each nested doc.
Term vectors for stored documents still doesn't work with nested
PUT nested_term_vectors_test { "mappings": { "properties": { "text": { "type": "text", "term_vector": "with_positions_offsets" }, "nested_field": { "type": "nested", "properties": { "text": { "type": "text", "term_vector": "with_positions_offsets" } } } } } } POST nested_term_vectors_test/_doc/1 { "text": "This is a test", "nested_field": [ { "text": "Nested text 1" }, { "text": "Nested text text text" } ] } POST nested_term_vectors_test/_termvectors/1 { }Returns only information for the
textfield, notnested_field.text.Reacted by Carlos Delgado
Elasticsearch Version
7.14.2
Installed Plugins
No response
Java Version
openjdk version "1.8.0-262" OpenJDK Runtime Environment (build 1.8.0-262-b10) OpenJDK 64-Bit Server VM (build 25.71-b10, mixed mode)
OS Version
windows 10
Problem Description
Using the mtermvectors api to get the term vectors (with term_statistics) for an artificial document Elasticsearch will ignore the nested fields.
Steps to Reproduce
To reproduce:
Create the index and upload a doc
Request the term vector for an artificial document
Response
The response does not include the term vector for the nested field.
Logs (if relevant)
No response