Skip to content

Offset translator consistency validation #24607

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mmaslankaprv merged 3 commits into from
Dec 20, 2024
Merged

Offset translator consistency validation #24607

mmaslankaprv merged 3 commits into from
Dec 20, 2024

Conversation

@mmaslankaprv
Copy link
Member

@mmaslankaprv mmaslankaprv commented Dec 18, 2024 •
edited
Loading

Added basic validation of offset translator consistency

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v24.3.x
  • v24.2.x
  • v24.1.x

Release Notes

  • none

@mmaslankaprv mmaslankaprv force-pushed the offset-translator-consistency-validation branch from a303925 to 6ca53e1 Compare December 18, 2024 17:47
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Dec 18, 2024 •
edited
Loading

Retry command for Build#59925

please wait until all jobs are finished before running the slash command


/ci-repeat 1
tests/rptest/tests/tiered_storage_model_test.py::TieredStorageTest.test_tiered_storage@{"cloud_storage_type_and_url_style":[1,"path"],"test_case":{"name":"(TS_Read == True, SegmentRolledByTimeout == True)"}}
tests/rptest/tests/e2e_shadow_indexing_test.py::ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy@{"cloud_storage_type":2,"short_retention":true}

@mmaslankaprv mmaslankaprv removed the request for review from bashtanov December 18, 2024 20:58
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Dec 18, 2024 •
edited
Loading

CI test results

test results on build#59925
test_id test_kind job_url test_status passed
rptest.tests.e2e_shadow_indexing_test.ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy.short_retention=True.cloud_storage_type=CloudStorageType.ABS ducktape https://buildkite.com/redpanda/redpanda/builds/59925#0193db12-acaa-4526-8465-07b982abf5f8 FAIL 0/6
rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type_and_url_style=.CloudStorageType.S3.1.path.test_case=.TS_Read==True.SegmentRolledByTimeout==True ducktape https://buildkite.com/redpanda/redpanda/builds/59925#0193db12-acab-4f4b-ab92-ca19fd4ffe51 FAIL 0/1
test results on build#59985
test_id test_kind job_url test_status passed
kafka_server_rpfixture.kafka_server_rpfixture unit https://buildkite.com/redpanda/redpanda/builds/59985#0193e302-4d59-4b8f-8480-791a3186d62c FLAKY 1/2
rptest.tests.datalake.partition_movement_test.PartitionMovementTest.test_cross_core_movements.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/59985#0193e35f-54f9-4bf3-bea9-4998e2663483 FLAKY 2/6
rptest.tests.e2e_shadow_indexing_test.ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy.short_retention=True.cloud_storage_type=CloudStorageType.ABS ducktape https://buildkite.com/redpanda/redpanda/builds/59985#0193e35f-54f5-4343-a274-12c654e1cc36 FAIL 0/6

bashtanov
bashtanov reviewed Dec 19, 2024
View reviewed changes
@bashtanov
Copy link
Contributor

bashtanov commented Dec 19, 2024

Why does segment size matter?

@mmaslankaprv
Copy link
Member Author

mmaslankaprv commented Dec 19, 2024

/cdt

WillemKauf
WillemKauf reviewed Dec 20, 2024
View reviewed changes
tests/rptest/tests/random_node_operations_test.py
# not to emit spurious errors
"raft_io_timeout_ms": 20000,
"compacted_log_segment_size": 1024 * 1024,
"log_segment_size": 2 * 1024 * 1024,
Copy link
Contributor

@WillemKauf WillemKauf Dec 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just fyi, we can go smaller than this with segment size using #24544

environment = {"__REDPANDA_TEST_DISABLE_BOUNDED_PROPERTY_CHECKS": "ON"}

@mmaslankaprv
tests: added validation of offset translator consistency in ducktape
026fa9b 
Added code that validates if deltas are consistent across different
replicas.

Signed-off-by: Michał Maślanka <michal@redpanda.com>
@mmaslankaprv
tests: validate offset translator state in random_node_operations_test
59eba45 
Signed-off-by: Michał Maślanka <michal@redpanda.com>
@mmaslankaprv
tests: made segment size smaller in random_node_operations_test
9d885f5 
Signed-off-by: Michał Maślanka <michal@redpanda.com>
@mmaslankaprv mmaslankaprv force-pushed the offset-translator-consistency-validation branch from 017dfee to 9d885f5 Compare December 20, 2024 07:36
bashtanov
bashtanov approved these changes Dec 20, 2024
View reviewed changes
Copy link
Contributor

@bashtanov bashtanov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A nit only. Segment is small for non-data batches to happen more often, right?

tests/rptest/utils/node_operations.py
all_partitions = admin.get_partitions(node=n)

def _state_consistent(ns, topic, partition):

Copy link
Contributor

@bashtanov bashtanov Dec 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: excessive empty line

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Dec 20, 2024

Retry command for Build#59985

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/e2e_shadow_indexing_test.py::ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy@{"cloud_storage_type":2,"short_retention":true}

@mmaslankaprv
Copy link
Member Author

mmaslankaprv commented Dec 20, 2024

A nit only. Segment is small for non-data batches to happen more often, right?

Exactly, also rolling segments is not trivial operation so it may trigger anomalies and race conditions.

@mmaslankaprv
Copy link
Member Author

mmaslankaprv commented Dec 20, 2024

the failure is unrelated

@mmaslankaprv mmaslankaprv merged commit be58f47 into redpanda-data:dev Dec 20, 2024
13 of 16 checks passed
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Dec 20, 2024

/backport v24.3.x

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Dec 20, 2024

/backport v24.2.x

@vbotbuildovich vbotbuildovich mentioned this pull request Dec 20, 2024
Merged
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Dec 20, 2024

Failed to create a backport PR to v24.2.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-24607-v24.2.x-121 remotes/upstream/v24.2.x
git cherry-pick -x 026fa9b14e 59eba45a56 9d885f5a4e

Workflow run logs.

@vbotbuildovich vbotbuildovich mentioned this pull request Dec 20, 2024
Closed
@mmaslankaprv mmaslankaprv mentioned this pull request Jan 7, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bashtanov bashtanov bashtanov approved these changes

@bharathv bharathv Awaiting requested review from bharathv

@dotnwat dotnwat Awaiting requested review from dotnwat

@ztlpn ztlpn Awaiting requested review from ztlpn

@WillemKauf WillemKauf Awaiting requested review from WillemKauf

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@mmaslankaprv @vbotbuildovich @bashtanov @WillemKauf