Skip to content

[Final][Cases] Cases analytics v2 - Attachments Index (.cases-attachments) - #276117

Merged
michaelolo24 merged 21 commits into
elastic:mainfrom
michaelolo24:cases/analytics-v2-attachments-surface-pr3-clean
Jul 8, 2026
Merged

michaelolo24 merged 21 commits into
elastic:mainfrom
michaelolo24:cases/analytics-v2-attachments-surface-pr3-clean

Conversation

@michaelolo24

@michaelolo24 michaelolo24 commented Jul 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Feature Flag: xpack.cases.analyticsV2.enabled

This is a follow up to #269581 and #275686, which introduced the cases as data V2 work and its first two indices, .cases and .cases-activity. This PR adds the third and final one, .cases-attachments, which holds the case comments and attachments data (one document per comment / attachment). The Cases Analytics data view added in the first PR already points at all three indices, so this data can be joined back to its case via cases.id using an ES|QL LOOKUP JOIN.

Like the other two indices, .cases-attachments follows the same approach set up in the previous PRs:

  • Whenever a comment or attachment is created, updated, or deleted, we asynchronously write the same change to the .cases-attachments index. As with the other indices, this will not block the primary write from happening if the analytics write fails.
  • A reconciliation task runs periodically to check all comments and attachments since the last run and make sure they've been written to .cases-attachments. Each index is reconciled independently, so an issue with one won't hold up the others.
  • The /reset and /reconcile/run_soon admin routes and the initial backfill now cover all three indices. The backfill and /reset rebuild the three indices in parallel, and read saved objects in larger batches, so a full rebuild finishes noticeably faster than doing them one after another.

Of note:

  • This index holds both the older case comments (cases-comments) and the newer unified attachments (cases-attachments), normalized into a single document shape, so it works whether or not a deployment has moved to the new attachments saved object. Both source types are always mirrored and reconciled, since the unified attachments saved object is always registered. xpack.cases.attachments.enabled only affects which saved object new attachments are written to — not what the analytics index reads.

  • Deleting a case also deletes its comments and attachments, so when a case is deleted we make sure to remove its .cases-attachments docs as well.

With all three indices now in place, a final PR will follow to remove the existing V1 implementation. The current cases as data v1 work is unaffected.

Technical Implementation

For a complete set of details, please see: x-pack/platform/plugins/shared/cases/server/cases_analytics_v2/README.md

Testing

Setup

Add to config/kibana.dev.yml:

xpack.cases.analyticsV2.enabled: true
xpack.cases.analyticsV2.enableAdminRoutes: true

# Optional — makes NEW attachments write to the unified `cases-attachments`
# saved object (default is the legacy `cases-comments`). The analytics index
# mirrors/reconciles both source types either way.
xpack.cases.attachments.enabled: true

Restart Kibana, then open Dev Tools → Console.

1. Real-time writes

Create a case and add a couple of things to it (a comment, and an attachment if you enabled attachments.enabled). Then confirm the data landed:

GET .cases-attachments/_search
{
  "query": { "term": { "cases.id": "<case id>" } }
}

Each comment / attachment should have a matching doc, with the attachment type and a few of the important fields pulled out for easier analysis. Editing or deleting a comment should be reflected on the next search (upsert / delete path).

2. Deleting a case

Delete the case, then re-run the search above → 0 hits (its attachment docs are removed along with the case).

3. Reconciliation + reset

  • GET kbn:/internal/cases/_analyticsV2/state now reports all three indices (.cases, .cases-activity, .cases-attachments) and includes attachments_last_run_at alongside the existing reconciliation info.
  • POST kbn:/internal/cases/_analyticsV2/reset rebuilds all three indices. Poll /state and confirm they all come back. While a reset is running, /state reports a single running phase with the per-index processed counts (cases_processed, activity_processed, attachments_processed) climbing together, since the three rebuilds run in parallel.

Automated coverage

node scripts/functional_tests \
  --config x-pack/platform/test/cases_api_integration/spaces_only/config_analytics_v2.ts

yarn jest --config x-pack/platform/plugins/shared/cases/jest.config.js cases_analytics_v2

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

  • All changes are net new and behind xpack.cases.analyticsV2.enabled (off by default); the primary case / comment / attachment writes are never blocked by an analytics write, and the reconciliation task backstops any missed writes.

@michaelolo24
michaelolo24 requested a review from a team as a code owner July 2, 2026 21:02
@michaelolo24 michaelolo24 added backport:skip This PR does not require backporting Feature:Cases Cases feature release_note:feature Makes this part of the condensed release notes Team:Cases Security Solution Cases team v9.5.0 labels Jul 2, 2026
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown
Contributor

Pinging @elastic/kibana-cases (Feature:Cases)

@infra-vault-gh-plugin-prod

Copy link
Copy Markdown
Contributor

Pinging @elastic/kibana-cases (Team:Cases)

@michaelolo24 michaelolo24 added ci:cloud-deploy Create or update a Cloud deployment ci:cloud-persist-deployment Persist cloud deployment indefinitely ci:project-deploy-security Create a Security Serverless Project ci:project-persist-deployment Persist project deployment indefinitely labels Jul 2, 2026
@michaelolo24
michaelolo24 force-pushed the cases/analytics-v2-attachments-surface-pr3-clean branch from 04434da to 15a71aa Compare July 3, 2026 02:19
@michaelolo24
michaelolo24 force-pushed the cases/analytics-v2-attachments-surface-pr3-clean branch from 597d60e to 88e56c4 Compare July 3, 2026 04:40
@michaelolo24
michaelolo24 force-pushed the cases/analytics-v2-attachments-surface-pr3-clean branch from 23a63de to fd8d596 Compare July 3, 2026 11:32
@michaelolo24
michaelolo24 force-pushed the cases/analytics-v2-attachments-surface-pr3-clean branch from 8176b33 to dbe2a25 Compare July 3, 2026 12:36
kibanamachine and others added 2 commits July 3, 2026 12:50
…ertion

The managed cases-analytics data view now spans three indices
(.cases,.cases-activity,.cases-attachments); the per-space data views FTR
test still expected only the first two, so it failed on the new attachments
index. Update the expected title to match.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@michaelolo24 michaelolo24 removed the ci:cloud-persist-deployment Persist cloud deployment indefinitely label Jul 5, 2026
michaelolo24 and others added 2 commits July 5, 2026 19:36
The .cases-attachments analytics doc records the UNIFIED attachment type,
so a legacy `user` comment lands with `attachment.type: 'comment'`, not
'user'. The test matched the doc by the legacy `AttachmentType.user` enum,
so `docs.find(...)` returned undefined and `expect(userDoc).to.be.an('object')`
failed. `--bail` stopped the run at the first test, hiding the same latent
mismatch in the updateComment test. Match on the unified `'comment'` type
at both doc-lookup sites; the updateComment request payload keeps the
legacy `AttachmentType.user` (correct for the SO/API).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@christineweng

Copy link
Copy Markdown
Contributor

When xpack.cases.attachments.enabled is false, attachments in cases-attachments are excluded, I think it's okay to include now that always register attachment SO is merged.

// Mirroring the partial `update` response directly would drop the
// immutable creation fields (`created_at` → `@timestamp`) and flicker
// the edited attachment off time-filtered views until reconciliation.
this.mirrorUpdatedAttachments([{ type: CASE_ATTACHMENT_SAVED_OBJECT, id: savedObjectId }]);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mirrorUpdatedAttachments is called unconditionally from update and bulkUpdate, and it always does a bulkGet re-read before touching the writer

@michaelolo24
michaelolo24 enabled auto-merge (squash) July 8, 2026 00:32
@kibanamachine

kibanamachine commented Jul 8, 2026 •

Copy link
Copy Markdown
Contributor

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] Scout Lane #25 - serverless-observability_complete / default / local-serverless-observability_complete - Discover app - saved search embeddable - should allow removing the dashboard panel after the underlying saved search has been deleted
  • [job] [logs] Scout Lane #28 - serverless-security_complete / default / local-serverless-security_complete - Discover app - saved search embeddable - should allow removing the dashboard panel after the underlying saved search has been deleted
  • [job] [logs] Scout Lane #11 - stateful-classic / default / local-stateful-classic - Discover app - saved search embeddable - should allow removing the dashboard panel after the underlying saved search has been deleted

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id before after diff
cases 100 105 +5

Total ESLint disabled count

id before after diff
cases 112 117 +5

History

@michaelolo24
michaelolo24 merged commit 848a267 into elastic:main Jul 8, 2026
38 checks passed
@michaelolo24 michaelolo24 mentioned this pull request Jul 28, 2026
28 of 29 tasks
patrykkopycinski pushed a commit to patrykkopycinski/kibana that referenced this pull request Aug 5, 2026
…ents) (elastic#276117)

## Summary

Feature Flag: `xpack.cases.analyticsV2.enabled`

This is a follow up to elastic#269581 and elastic#275686, which introduced the cases
as data V2 work and its first two indices, `.cases` and
`.cases-activity`. This PR adds the third and final one,
`.cases-attachments`, which holds the case comments and attachments data
(one document per comment / attachment). The `Cases Analytics` data view
added in the first PR already points at all three indices, so this data
can be joined back to its case via `cases.id` using an ES|QL `LOOKUP
JOIN`.

Like the other two indices, `.cases-attachments` follows the same
approach set up in the previous PRs:

- Whenever a comment or attachment is created, updated, or deleted, we
asynchronously write the same change to the `.cases-attachments` index.
As with the other indices, this *will not* block the primary write from
happening if the analytics write fails.
- A reconciliation task runs periodically to check all comments and
attachments since the last run and make sure they've been written to
`.cases-attachments`. Each index is reconciled independently, so an
issue with one won't hold up the others.
- The `/reset` and `/reconcile/run_soon` admin routes and the initial
backfill now cover all three indices. The backfill and `/reset` rebuild
the three indices in parallel, and read saved objects in larger batches,
so a full rebuild finishes noticeably faster than doing them one after
another.

Of note:

- This index holds both the older case comments (cases-comments) and the
newer unified attachments (cases-attachments), normalized into a single
document shape, so it works whether or not a deployment has moved to the
new attachments saved object. Both source types are always mirrored and
reconciled, since the unified attachments saved object is always
registered. `xpack.cases.attachments.enabled` only affects which saved
object new attachments are written to — not what the analytics index
reads.

- Deleting a case also deletes its comments and attachments, so when a
case is deleted we make sure to remove its `.cases-attachments` docs as
well.

With all three indices now in place, a final PR will follow to remove
the existing V1 implementation. The current cases as data v1 work is
unaffected.

## Technical Implementation

For a complete set of details, please see:
`x-pack/platform/plugins/shared/cases/server/cases_analytics_v2/README.md`

## Testing

### Setup

Add to `config/kibana.dev.yml`:

```yaml
xpack.cases.analyticsV2.enabled: true
xpack.cases.analyticsV2.enableAdminRoutes: true

# Optional — makes NEW attachments write to the unified `cases-attachments`
# saved object (default is the legacy `cases-comments`). The analytics index
# mirrors/reconciles both source types either way.
xpack.cases.attachments.enabled: true
```

Restart Kibana, then open **Dev Tools → Console**.

### 1. Real-time writes

Create a case and add a couple of things to it (a comment, and an
attachment if you enabled `attachments.enabled`). Then confirm the data
landed:

```
GET .cases-attachments/_search
{
  "query": { "term": { "cases.id": "<case id>" } }
}
```

Each comment / attachment should have a matching doc, with the
attachment `type` and a few of the important fields pulled out for
easier analysis. Editing or deleting a comment should be reflected on
the next search (upsert / delete path).

### 2. Deleting a case

Delete the case, then re-run the search above → 0 hits (its attachment
docs are removed along with the case).

### 3. Reconciliation + reset

- `GET kbn:/internal/cases/_analyticsV2/state` now reports all three
indices (`.cases`, `.cases-activity`, `.cases-attachments`) and includes
`attachments_last_run_at` alongside the existing reconciliation info.
- `POST kbn:/internal/cases/_analyticsV2/reset` rebuilds all three
indices. Poll `/state` and confirm they all come back. While a reset is
running, `/state` reports a single `running` phase with the per-index
processed counts (`cases_processed`, `activity_processed`,
`attachments_processed`) climbing together, since the three rebuilds run
in parallel.

#### Automated coverage

```bash
node scripts/functional_tests \
  --config x-pack/platform/test/cases_api_integration/spaces_only/config_analytics_v2.ts

yarn jest --config x-pack/platform/plugins/shared/cases/jest.config.js cases_analytics_v2
```

### Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed

### Identify risks

Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.

- [ ] All changes are net new and behind
`xpack.cases.analyticsV2.enabled` (off by default); the primary case /
comment / attachment writes are never blocked by an analytics write, and
the reconciliation task backstops any missed writes.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting ci:cloud-deploy Create or update a Cloud deployment ci:project-deploy-security Create a Security Serverless Project ci:project-persist-deployment Persist project deployment indefinitely Feature:Cases Cases feature release_note:feature Makes this part of the condensed release notes Team:Cases Security Solution Cases team v9.5.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants