Skip to content

Conversation

@bdarnell
Copy link
Contributor

I suspect that #26257 is caused by the unquiescedReplicas map
introduced in #24956 getting out of sync with the per-replica
quiescent flag. Add debug pages to help us see if that's happening.

Release note: None

@bdarnell bdarnell requested review from a team and andreimatei May 31, 2018 02:32
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@tbg
Copy link
Member

tbg commented May 31, 2018

LGTM, but you need to update replica_test.go.

@bdarnell bdarnell force-pushed the quiesce-debugging branch from 3838684 to d807a4e Compare May 31, 2018 14:31
c.liveness, &c.desc, c.raftStatus, LeaseStatus{},
c.storeID, c.expected.Quiescent, CommandQueueMetrics{}, CommandQueueMetrics{})
c.storeID, c.expected.Quiescent, !c.expected.Quiescent, CommandQueueMetrics{}, CommandQueueMetrics{})
if c.expected != metrics {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused, shouldn't c.expected need Ticking set on it as well?

problems: this.contentProblems(info.problems, awaitingGC),
raftState: raftState,
quiescent: info.quiescent ? rangeTableQuiescent : rangeTableEmptyContent,
ticking: this.createContent(info.ticking.toString()),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be nice to roll this in with the quiescent row. I'm imagining we'd show quiescent and empty as normal states and "quiescent but ticking" and "not ticking but not quiescent" as error states.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I just wanted to get something in quickly for cyan without having to fiddle with and test all the combinations.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with that, we can just have @BramGruneir come through and add some niceties here.

I suspect that cockroachdb#26257 is caused by the unquiescedReplicas map
introduced in cockroachdb#24956 getting out of sync with the per-replica
quiescent flag. Add debug pages to help us see if that's happening.

Release note: None
@bdarnell bdarnell force-pushed the quiesce-debugging branch from d807a4e to 0a84d51 Compare May 31, 2018 15:10
@BramGruneir
Copy link
Member

:lgtm: Just with @couchand's nits.


Review status: all files reviewed at latest revision, 2 unresolved discussions, some commit checks pending.


Comments from Reviewable

@bdarnell
Copy link
Contributor Author

bors r+

@andreimatei
Copy link
Contributor

:lgtm:


Review status: all files reviewed at latest revision, 2 unresolved discussions, some commit checks pending.


pkg/server/serverpb/status.proto, line 142 at r2 (raw file):

  // If the replica's quiescent flag doesn't agree with the store's
  // list of replicas that are ticking, warn about it.
  bool quiescent_equals_ticking = 6;

Even with the comment I was confused about whether the true value is the good one or not. How about calling it ticking_disagreement and also adding a quiescent field to determine which way the disagreement goes.


Comments from Reviewable

craig bot pushed a commit that referenced this pull request May 31, 2018
26269: server,ui: Add debugging for quiesced ranges r=bdarnell a=bdarnell

I suspect that #26257 is caused by the unquiescedReplicas map
introduced in #24956 getting out of sync with the per-replica
quiescent flag. Add debug pages to help us see if that's happening.

Release note: None

Co-authored-by: Ben Darnell <ben@cockroachlabs.com>
@craig
Copy link
Contributor

craig bot commented May 31, 2018

Build succeeded

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants