[TSDB] not all documents with invalid timestamps are redirected to the failure store.

Elasticsearch Version

9.3.5, 9.4.1, 9.5.0, main

Installed Plugins

No response

Java Version

bundled

OS Version

n/a

Problem Description

When failure store is enabled and a document with an invalid timestamp, or a timestamp outside of the eligible write window of the time series data stream is indexed and failure store is enabled, the document should be redirected to the failure store. This works correctly, when a document is not matched to any backing index, but it does fail if there is a read-only backing index because it's detected as backpressure. The reproduction path shows how two different documents behave, ideally, they should behave the same way.

Steps to Reproduce

# Add index template
PUT /_index_template/my-template
{
  "index_patterns": [
    "ds*"
  ],
  "data_stream": {},
  "template": {
    "mappings": {
      "properties": {
        "@timestamp" : {
          "type" : "date"
        },
        "metrics.cpu_load": {
          "type": "double",
          "time_series_metric": "gauge"
        },
        "host.name": {
          "type": "keyword",
          "time_series_dimension": true
        }
      }
    },
    "settings": {
      "index.mode": "time_series",
      "number_of_replicas": 0
    },
    "data_stream_options": {
      "failure_store": {
        "enabled": true
      }
    }
  }
}

# Create data stream
PUT /_data_stream/ds

# Rollover data stream
POST /ds/_rollover

# Make first index readonly, you will need to adjust the index name
PUT /.ds-ds-2026.10.05-000001/_settings
{
  "index.blocks.write":true
}

# Get the time series covered by this data stream
GET /_data_stream/ds
{
....,
  "time_series": {
        "temporal_ranges": [
          {
            "start": "2026-10-05T08:09:09.000Z",
            "end": "2026-10-05T10:31:09.000Z"
          }
        ]
      }
}

### Index old document not covered by a backing index
POST http://localhost:9200/ds/_doc
{"host.name": "my-host-1", "metrics.cpu_load":  5, "@timestamp": "2026-10-04T08:09:09.000Z"}

# Response
{
  "_index": ".fs-ds-2026.10.05-000002",
  "_id": "lfeXC6EBVB09Fq0K-8XB",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 4,
  "_primary_term": 1,
  "failure_store": "used"
}

### Index old document covered by a backing index
POST http://localhost:9200/ds/_doc
{"host.name": "my-host-1", "metrics.cpu_load":  5, "@timestamp": "2026-10-05T08:09:09.000Z"}

# Response
{
  "error": {
    "root_cause": [
      {
        "type": "cluster_block_exception",
        "reason": "index [.ds-ds-2026.10.05-000001] blocked by: [FORBIDDEN/8/index write (api)];"
      }
    ],
    "type": "cluster_block_exception",
    "reason": "index [.ds-ds-2026.10.05-000001] blocked by: [FORBIDDEN/8/index write (api)];"
  },
  "status": 403
}

Logs (if relevant)

No response

Pinging @elastic/es-storage-engine (Team:StorageEngine)

Bug analysis

Verdict: Confirmed on main (6b22cff517be).

Root cause: For TSDB data streams, BulkOperation.groupRequestsByShards routes a CREATE by @timestamp through DataStream#selectTimeSeriesWriteIndex. That method picks the covering backing index by time range only and never checks index blocks. The write block on that older index only fails at the shard (TransportReplicationAction) as a ClusterBlockException. BulkOperation#shouldRedirectRequestToFailureStore deliberately never redirects that exception (case ClusterBlockException err -> false, treated like backpressure), so the client gets a 403. Timestamps with no covering index instead throw DataStream.TimestampError during routing, which is redirected. That explains the asymmetry.

Reproduced: Yes. I ran it as a Buildkite fallback because the local JDK was unusable:

TSDBIndexingIT.testDocForWriteBlockedBackingIndexRedirectedToFailureStore fails with index [.ds-k8s-…-000001] blocked by: [FORBIDDEN/8/index write (api)]. The out-of-range doc in the same test gets failure_store: used.

Evidence: repro job · scan · candidate fix: TSDBIndexingIT green · scan

Recommended fix:

In groupRequestsByShards, after getConcreteWriteIndex: if the target is a data stream, the resolved index is not its write index, and it has an index-level WRITE block, throw DataStream.TimestampError (with the block exception as cause). The existing catch then redirects to the failure store. Blocks on the write index and on failure-store requests keep today's no-redirect behaviour. This is in candidate-fix.patch.
Alternative: make DataStream#getWriteIndex(IndexRequest, …) treat write-blocked backing indices as non-matching. This touches more callers.
Add a BulkOperationTests unit test next to testTsdbTimestampErrorDuringRoutingRedirectsToFailureStore.

Risk: Low. This has been the behaviour since 8.19, not a regression. Clients get an explicit 403, so no data is silently lost; it's just a gap in failure-store coverage.

Additional notes

Reporter: gmarouli, verified active member of elastic/elasticsearch-team via the GitHub team membership API.
Test layer: ESSingleNodeTestCase (TSDBIndexingIT). It needs real index-block enforcement, rollover, TSDB time ranges and failure-store creation. A BulkOperationTests unit test with a mocked shard action could only inject the ClusterBlockException by hand, and that case is already asserted as intentional in testFailingDocumentIgnoredByFailureStoreWhenInvalidException.
Repro command: ./gradlew ":modules:data-streams:internalClusterTest" --tests "org.elasticsearch.datastreams.TSDBIndexingIT.testDocForWriteBlockedBackingIndexRedirectedToFailureStore" -Dtests.seed=D844C5615730989E. The test needs MapperExtrasPlugin added to getPlugins() because the failure-store mapping uses match_only_text.
Open policy question: should read_only_allow_delete (disk flood-stage) on an old backing index also redirect? It is closer to backpressure. If not, match on INDEX_WRITE_BLOCK / INDEX_READ_ONLY_BLOCK explicitly.
Patches reproduction.patch and candidate-fix.patch are uploaded as artifacts on the build above.

Generated by bug-analysis · archimedes v0.52.4 · openrouter/anthropic/claude-opus-5.5

@lukewhiting

After looking at the code and the intention of #148154, I am thinking that the goal was to reject back pressure exceptions only, not any kind of read-only cases. Maybe we should follow here the same advice we gave to logstash and reject only failures that are retryable:

case ClusterBlockException err -> err.retryable() == false;

What do you think @lukewhiting ?

@gmarouli

We had an offline chat with @gmarouli, @martijnvg and @lukewhiting to discuss this. We agreed that not writing to the failure store on index level blocks (such as read only due to frozen index) was a bug introduced by #148154 being overly broad in the kinds of cluster exceptions it pushed back on the client.

We agreed to move forward with Mary's proposed fix and @lukewhiting will implement and test that.

[TSDB] not all documents with invalid timestamps are redirected to th…

Navigation Menu

Repository navigation

[TSDB] not all documents with invalid timestamps are redirected to the failure store. #161001

Elasticsearch Version

Installed Plugins

Java Version

OS Version

Problem Description

Steps to Reproduce

Logs (if relevant)

Bug analysis

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[TSDB] not all documents with invalid timestamps are redirected to the failure store. #161001

Description

Elasticsearch Version

Installed Plugins

Java Version

OS Version

Problem Description

Steps to Reproduce

Logs (if relevant)

Activity

elasticsearchmachine commented on Oct 5, 2026

elasticsearchmachine commented on Oct 5, 2026

Bug analysis

gmarouli commented on Oct 6, 2026

lukewhiting commented on Oct 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions