[Azure Logs / Custom Azure Logs] Switch default processor to v2 #16618

zmoog · 2025-12-18T09:59:23Z

Proposed commit message

Makes the processor v2 the default processor for all integrations in the Azure Logs and Custom Azure Logs packages. The processor v1 is still available as option, but it's listed as "legacy".

The processor v1 uses deprecated libraries and it will be removed in the next release (targeting 9.4).

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
~~I have verified that Kibana version constraints are current according to guidelines.~~
~~I have verified that any added dashboard complies with Kibana's Dashboard good practices~~

Related issues

Closes [azure logs] make azure-eventhub processor v2 the default option in the Azure Logs integration package #15703

elasticmachine · 2025-12-18T23:27:43Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

elastic-vault-github-plugin-prod · 2025-12-18T23:56:58Z

🚀 Benchmarks report

Package `azure` 👍(7) 💚(3) 💔(2)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`events`	29411.76	17857.14	-11554.62 (-39.29%)	💔
`graphactivitylogs`	2695.42	1848.43	-846.99 (-31.42%)	💔

To see the full report comment with /test benchmark fullreport

tommyers-elastic · 2025-12-22T10:40:03Z

packages/azure/_dev/build/docs/events.md

 ### Event Hub Processor v2 ✨

-The integration v2 preview offers a new processor v2 starting with integration version 1.23.0.
+As of version 1.23.0, the integration preview includes the new Processor v2.


what does "the integration preview" mean?

We introduced the new events data stream as the entry point and router for the Azure Logs integration package.

I plan to make the events data stream (aka Azure Logs v2) as the only data stream running an input.

Unfortunately, years ago we updated the Azure Logs packages to allow multiple inputs to read from the same event hub, but this is a fatal flaw. We need to get rid of this.

The other data streams (activitylogs, firewall_logs, etc) are going to stay without an input.

We marked this events data stream named Azure Logs v2 as "preview" because we were not 100% sure this approach would work. But it seems working pretty well.

tommyers-elastic · 2025-12-22T10:40:55Z

packages/azure/_dev/build/docs/events.md

+* Features a more efficient checkpoint store leveraging Azure Blob Storage metadata.

-The processor v2 is in preview. Processor v1 is still the default and is recommended for typical use cases.
+Processor v2 is now the default and is recommended for most use cases. Processor v1 remains available as a legacy option for backward compatibility until the next major release.


most use cases

is there somewhere we can link that details when it's not recommended?

Um, no, there aren't known use cases. Maybe change this to only inform users v1 is still available for some time?

I clarified the description. Let me know if it makes sense.

tommyers-elastic · 2025-12-22T10:42:58Z

packages/azure/_dev/build/docs/events.md

 (processor v2 only) Flag to control whether the processor should perform the checkpoint information migration from v1 to v2 at startup. The checkpoint migration converts the checkpoint information from the v1 format to the v2 format.

-Default is `false`, which means the processor will not perform the checkpoint migration.
+Default is `true`, which means the processor will attempt to migrate the checkpoint information from v1 to v2 the first time you v1 to v2.


what happens if you have processor_version: v2 and processor_update_interval: false?

and processor_update_interval: false?

I assume you mean migrate_checkpoint: false.

In that case, the input will start over from the beginning of the stream. If the retention is set to 1 hour, the input will re-process the last hour's worth of events.

I think we should make this explicit in the description rather than leaving it implied, right?

@tommyers-elastic, I’ve fleshed out the migrate_checkpoint description. I think this version is much more comprehensive.

tommyers-elastic · 2025-12-22T10:44:27Z

packages/azure/data_stream/activitylogs/agent/stream/azure-eventhub.yml.hbs

 {{/if}}
 {{#if storage_account_key}}
 storage_account_key: {{storage_account_key}}
+storage_account_connection_string: DefaultEndpointsProtocol=https;AccountName={{storage_account}};AccountKey={{storage_account_key}};EndpointSuffix={{endpoint_suffix}}


nit: i'm guessing storage_account shouldn't be empty if storage_account_key is set, but since it's not in the {{#if storage_account}} there is a chance it's unset.

same with endpoint_suffix

The storage_account contains the storage account name; this option must be set for the input to work.

The storage_account_key contains the auth for processor v1.

The storage_account_connection_string contains the auth for processor v2

We'll have the opportunity to clean things up a little when we'll drop v1 in a couple of iterations.

tommyers-elastic · 2025-12-22T10:47:47Z

packages/azure/data_stream/activitylogs/manifest.yml

+      #
+      # Processor v2 only settings
+      #
+      - name: processor_version


are we ok that integration upgrades are going to automatically enable v2 without the user explicitly choosing?

We need to push users to the processor v2 while v1 is still available as fallback option. So if v2 falls short for any reason, users can still fallback to v1 while we address the problem.

We need to drop v1¹ as soon as possible, since the v1's libraries are so old that sticking with them is becoming a risk.

Footnotes

drop v1 as an option and remove it from the Beats codebase. ↩

zmoog · 2025-12-23T10:56:14Z

@tommyers-elastic, let me know if there's anything else you'd like me to tweak!

So v1 checkpoint information will be migrated to v2 on the fist v2 execution.

# Conflicts: # packages/azure/changelog.yml

We retain v1 as legacy option for backwards compatibility only, just in case we run into an unknown issues.

elasticmachine · 2025-12-23T23:23:37Z

💚 Build Succeeded

Buildkite Build
Commit: 7edce00

History

💚 Build #35811 succeeded 62991aa
💚 Build #35712 succeeded 9220bbb
💚 Build #35681 succeeded e784951

cc @zmoog

elastic-vault-github-plugin-prod · 2025-12-23T23:41:52Z

Package azure - 1.34.0 containing this change is available at https://epr.elastic.co/package/azure/1.34.0/

elastic-vault-github-plugin-prod · 2025-12-23T23:41:53Z

Package azure_logs - 0.5.0 containing this change is available at https://epr.elastic.co/package/azure_logs/0.5.0/

zmoog self-assigned this Dec 18, 2025

zmoog added Integration:azure Azure Logs Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] labels Dec 18, 2025

zmoog force-pushed the zmoog/azure/v2-default branch from e784951 to 721ea15 Compare December 18, 2025 23:17

zmoog changed the title ~~[azure logs] Make processor v2 the default option~~ [Azure Logs] Switch default processor to v2 Dec 18, 2025

zmoog added Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Dec 18, 2025

zmoog marked this pull request as ready for review December 18, 2025 23:27

zmoog requested review from a team as code owners December 18, 2025 23:27

zmoog changed the title ~~[Azure Logs] Switch default processor to v2~~ [Azure Logs / Custom Azure Logs] Switch default processor to v2 Dec 18, 2025

zmoog force-pushed the zmoog/azure/v2-default branch from 84c2dbe to 9220bbb Compare December 18, 2025 23:36

kaiyan-sheng approved these changes Dec 19, 2025

View reviewed changes

chrisberkhout approved these changes Dec 22, 2025

View reviewed changes

jakubgalecki0 approved these changes Dec 22, 2025

View reviewed changes

tommyers-elastic reviewed Dec 22, 2025

View reviewed changes

zmoog requested a review from tommyers-elastic December 23, 2025 10:19

tommyers-elastic approved these changes Dec 23, 2025

View reviewed changes

zmoog added 7 commits December 24, 2025 00:01

Make processor v2 the default option

1b6d77c

Update docs

ab9c392

Set migrate_checkpoint: true

c39ea5f

So v1 checkpoint information will be migrated to v2 on the fist v2 execution.

Update changelogs

a8661cb

# Conflicts: # packages/azure/changelog.yml

Clarify v2 as recommended for all use cases

ca9e60d

We retain v1 as legacy option for backwards compatibility only, just in case we run into an unknown issues.

Clarify migrate_checkpoint description

aa0a8b0

Update docs

7edce00

zmoog force-pushed the zmoog/azure/v2-default branch from 62991aa to 7edce00 Compare December 23, 2025 23:04

zmoog enabled auto-merge (squash) December 23, 2025 23:05

zmoog merged commit bd70287 into main Dec 23, 2025
8 checks passed

zmoog deleted the zmoog/azure/v2-default branch December 23, 2025 23:23

[Azure Logs / Custom Azure Logs] Switch default processor to v2 #16618

[Azure Logs / Custom Azure Logs] Switch default processor to v2 #16618

Uh oh!

Conversation

zmoog commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Related issues

Uh oh!

elasticmachine commented Dec 18, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Package azure 👍(7) 💚(3) 💔(2)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Footnotes

Uh oh!

zmoog commented Dec 23, 2025

Uh oh!

elasticmachine commented Dec 23, 2025

💚 Build Succeeded

History

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Dec 23, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Dec 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

zmoog commented Dec 18, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Dec 18, 2025 •

edited

Loading

Package `azure` 👍(7) 💚(3) 💔(2)