NotifyCenter may cause naming data inconsistent #5918
Description
Describe the bug
When I do performance test with nacos1.x client and nacos2.0 server, I found that there some instances can't be removed after I stop client. The behavior really like #5769 .
After researched, I found there are client disconnection logs in naming-server.log but no client delete distro logs in protocol-distro.log.
2021-06-01 11:40:28,379 INFO Client remove for service Service{namespace='abcd', group='DEFAULT_GROUP', name='YIXInWfg9.f6Usp.net', ephemeral=true, revision=124}, 127.0.0.1:440#true
2021-06-01 11:40:28,379 INFO Client remove for service Service{namespace='abcd', group='DEFAULT_GROUP', name='YIXIsmbwJ.Zl7wU.net', ephemeral=true, revision=131}, 127.0.0.1:440#true
2021-06-01 11:40:28,384 INFO Client connection 127.0.0.1:440#true disconnect, remove instances and subscribers
2021-06-01 11:40:16,797 INFO [DISTRO-END] DistroKey{resourceKey='127.0.0.1:440#true', resourceType='Nacos:Naming:v2:ClientData', targetServer='xxx:8848'} result: true
2021-06-01 11:41:01,938 INFO [DISTRO-START] DistroSyncChangeTask for DistroKey{resourceKey='127.0.0.1:440#true', resourceType='Nacos:Naming:v2:ClientData', targetServer='xxx:8848'}
2021-06-01 11:41:01,938 WARN [DISTRO] DistroSyncChangeTask for DistroKey{resourceKey='127.0.0.1:440#true', resourceType='Nacos:Naming:v2:ClientData', targetServer='xxx:8848'} with null data to sync, skip
2021-06-01 11:41:03,965 INFO [DISTRO-START] DistroSyncChangeTask for DistroKey{resourceKey='127.0.0.1:440#true', resourceType='Nacos:Naming:v2:ClientData', targetServer='xxx:8848'}
2021-06-01 11:41:03,965 WARN [DISTRO] DistroSyncChangeTask for DistroKey{resourceKey='127.0.0.1:440#true', resourceType='Nacos:Naming:v2:ClientData', targetServer='xxx:8848'} with null data to sync, skip
2021-06-01 11:41:03,966 INFO [DISTRO-START] DistroSyncChangeTask for DistroKey{resourceKey='127.0.0.1:440#true', resourceType='Nacos:Naming:v2:ClientData', targetServer='xxx:8848'}
2021-06-01 11:41:03,966 WARN [DISTRO] DistroSyncChangeTask for DistroKey{resourceKey='127.0.0.1:440#true', resourceType='Nacos:Naming:v2:ClientData', targetServer='xxx:8848'} with null data to sync, skip
2021-06-01 11:41:03,971 INFO [DISTRO-START] DistroSyncChangeTask for DistroKey{resourceKey='127.0.0.1:440#true', resourceType='Nacos:Naming:v2:ClientData', targetServer='xxx:8848'}
2021-06-01 11:41:03,971 WARN [DISTRO] DistroSyncChangeTask for DistroKey{resourceKey='127.0.0.1:440#true', resourceType='Nacos:Naming:v2:ClientData', targetServer='xxx:8848'} with null data to sync, skip
From the logs we can see that the last DistroSyncChangeTask has no found the data and the time later then disconnection, so it should be DeleteChangeTask not DistroSyncChangeTask
I doubt that there are some problem in NotifyCenter and cause this problem. Each event has a separate event thread to call back event subscribers. Although the change event and the delete event are published in order, they are processed in different threads and the number of change events is usually more. Therefore, it may cause the delete event released later to be processed earlier than the change event released earlier, and resulting in the delete task being eventually overwritten by the change event.
Expected behavior
Clean all instances in all nodes
Acutally behavior
Only clean instances in responsible node for some of service.
How to Reproduce
Steps to reproduce the behavior:
- Register many of service in one client, and start many of clients.
- stop all clients and wait a few minutes
- see service data in different nodes.
- If no happen, try again.
Desktop (please complete the following information):
- OS: Centos
- Version nacos-server 1.4.2, nacos-server 2.0.2-SNAPSHOT
- Module naming/config
- SDK original
Additional context
This issue is a doubt issue. try to fix this problem.