Skip to content

[aggregator] Followers CanLead before the first flush is persisted - #4169

Merged
justinjc merged 8 commits into
masterfrom
juchan/agg-follower-no-data
Dec 3, 2022
Merged

justinjc merged 8 commits into
masterfrom
juchan/agg-follower-no-data

Conversation

@justinjc

@justinjc justinjc commented Nov 9, 2022

Copy link
Copy Markdown
Collaborator

Fixes #4168

@SokolAndrey SokolAndrey left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

zap.Error(err),
)
} else if flushTimes == nil || len(flushTimes.ByShard) == 0 {
// If there are no flush times, the cluster is just coming up (the leader has not

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sanity: I think this probably makes sense for the new topology case. What about the "nodes just added to the topology case"? Those two nodes will have a new shardset; do we need to ensure that the follower is up at least as long as the leader?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm probably cutover/cutoff times handle that, actually; worth reasoning through imo (though it probably just works).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. The function used to set the cutover for a shard is here, which happens when we try to generate a placement for instance adds. The function essentially sets the cutover as the next (max) window in the future. When nodes are added, even if the follower comes up later than the leader, the cutover time will prevent it from flushing until that time is passed.

I think this makes this diff still work. Concerns?

@andrewmains12
andrewmains12 requested a review from abliqo November 22, 2022 15:20
"go.uber.org/zap"
)

//nolint:dupl

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anything we can do to dedupe some of this? Perhaps a common setup method? (setup here is pretty long).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Donezo.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, much cleaner. Get rid of the nolint:dupl directive? (assuming linter passes now).

@@ -241,6 +241,20 @@ func (mgr *followerFlushManager) CanLead() bool {
mgr.metrics.notCampaigning.Inc(1)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@justinjc for my understanding: we're not campaigning if we're not cutover right?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep that's right (here's the check).

@justinjc
justinjc force-pushed the juchan/agg-follower-no-data branch from b57e193 to a7eefec Compare December 2, 2022 20:34
Comment thread src/aggregator/integration/setup.go Outdated
func newTestServerSetups(
t *testing.T,
customServerOpts ...func(testServerOptions) testServerOptions,
) (testServerSetups, *testClock, topic.Service) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer structs for this--multi returns get real unwieldy quickly.

@justinjc
justinjc merged commit 297c37e into master Dec 3, 2022
@justinjc
justinjc deleted the juchan/agg-follower-no-data branch December 3, 2022 02:01
safa-topal pushed a commit to aiven/m3 that referenced this pull request Jan 12, 2023
…3db#4169)

* Have follower report CanLead if there are no flush times
* Refactor out common integration test multiServerSetup
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[aggregator] All nodes should have CanLead == true before the first ever flush

3 participants