Skip to content

_bulk truncates an action-line _id at an escaped quote: distinct ids collapse and the second document silently overwrites the first #954

Description

@xerj-org

What happens

POST /_bulk truncates an action-line _id (and _index, op_type) at the first backslash-escaped quote. Two distinct ids that share a prefix up to an escaped quote collapse to the same stored id, and the second document silently overwrites the first. The single-document API handles the same id correctly.

Reproduce (any node; this was a throwaway --insecure node built from 02aa319d, the code in question is unchanged from main)

printf '%s\n' '{"index":{"_index":"t-bulkid","_id":"a\"b"}}' '{"n":1}' \
              '{"index":{"_index":"t-bulkid","_id":"a\"c"}}' '{"n":2}' > bulkid.ndjson
curl -s -XPOST -H 'content-type: application/x-ndjson' \
  'http://127.0.0.1:9854/_bulk?refresh=true' --data-binary @bulkid.ndjson
curl -s 'http://127.0.0.1:9854/t-bulkid/_search'

Observed: the bulk response reports _id: "a\\" for both items (created, then updated), and the index holds ONE document, _id = a\ with {"n":2}. {"n":1} is gone, and the response says nothing went wrong.

Expected: two documents, ids a"b and a"c — which is what PUT /t-bulkid/_doc/x%22y does ("_id":"x\"y", created).

Root cause

engine/crates/xerj-engine/src/bulk.rs, the fast-path find_field closure (around line 567-616): after the opening quote it advances with while j < inner.len() && inner[j] != b'"' and returns the raw bytes. It does not skip \", and it does not JSON-unescape the value, so \\, \/ and \uXXXX in an id are also stored as their escape sequences rather than the characters they name.

Suggested fix

Either fall back to the serde_json slow path when the value contains a backslash, or skip escaped bytes in the scan and unescape the slice. A test with two ids differing only after an escaped quote pins the overwrite.

Found while loading hostile-id fixtures for the security review of #945; it is not caused by that PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions