Page MenuHomePhabricator

Bursts of occasional severe contention on s4 (commonswiki) primary mariadb causing recurrent user-facing outages on all wikis
Closed, ResolvedPublicSecurity

Assigned To
Authored By
RoySmith
Jul 17 2024, 3:57 PM
Referenced Files
F57281403: image.png
Aug 19 2024, 2:21 PM
F57281295: grafik.png
Aug 19 2024, 12:36 PM
F57279891: quarry-85603-untitled-run894445.png
Aug 19 2024, 12:57 AM
F57272257: image.png
Aug 13 2024, 10:22 PM
F56755206: grafik.png
Jul 29 2024, 12:05 AM
F56755204: grafik.png
Jul 29 2024, 12:05 AM
F56755201: grafik.png
Jul 29 2024, 12:05 AM
F56622426: grafik.png
Jul 23 2024, 7:00 PM
Tokens
"Pterodactyl" token, awarded by FriedrickMILBarbarossa."The World Burns" token, awarded by Hyphen.

Description

Steps to replicate the issue (include links if applicable):

  • Do stuff a lot
  • Maybe it'll happen

What happens?:

On July 13, I got this:

MediaWiki internal error.
Original exception: [c8466b8a-0c54-4456-8259-2e3fd3f4be95] 2024-07-13 19:12:17: Fatal exception of type "Wikimedia\Rdbms\DBUnexpectedError"
Exception caught inside exception handler.
Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information.

I just got another one

MediaWiki internal error.
Original exception: [38973e56-81e9-4748-81c2-6735336e3a90] 2024-07-17 15:51:37: Fatal exception of type "Wikimedia\Rdbms\DBUnexpectedError"
Exception caught inside exception handler.
Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information.

URL for today's was: https://en.wikipedia.org/w/index.php?title=Wikipedia:Arbitration/Requests/Case&curid=22747298&diff=1235079535&oldid=1235079140

I figure if I got two of them in a few days, it might be something significant, so logging this report.


Error
normalized_message
[{reqId}] {exception_url}   Wikimedia\Rdbms\DBUnexpectedError: Database servers in extension1 are overloaded. In order to protect application servers, the circuit breaking to databases of this section have been activated. Please try again a few seconds.
exception.trace
from /srv/mediawiki/php-1.43.0-wmf.13/includes/libs/rdbms/loadmonitor/LoadMonitor.php(124)
#0 /srv/mediawiki/php-1.43.0-wmf.13/includes/libs/rdbms/loadbalancer/LoadBalancer.php(455): Wikimedia\Rdbms\LoadMonitor->scaleLoads(array)
#1 /srv/mediawiki/php-1.43.0-wmf.13/includes/libs/rdbms/loadbalancer/LoadBalancer.php(781): Wikimedia\Rdbms\LoadBalancer->getReaderIndex(string)
#2 /srv/mediawiki/php-1.43.0-wmf.13/includes/libs/rdbms/database/DBConnRef.php(111): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(integer, array, string, integer)
#3 /srv/mediawiki/php-1.43.0-wmf.13/includes/libs/rdbms/database/DBConnRef.php(125): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#4 /srv/mediawiki/php-1.43.0-wmf.13/includes/libs/rdbms/database/DBConnRef.php(351): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#5 /srv/mediawiki/php-1.43.0-wmf.13/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(746): Wikimedia\Rdbms\DBConnRef->select(array, array, array, string, array, array)
#6 /srv/mediawiki/php-1.43.0-wmf.13/extensions/Echo/includes/Mapper/EventMapper.php(177): Wikimedia\Rdbms\SelectQueryBuilder->fetchResultSet()
#7 /srv/mediawiki/php-1.43.0-wmf.13/extensions/Echo/includes/Hooks.php(879): MediaWiki\Extension\Notifications\Mapper\EventMapper->fetchUnreadByUserAndPage(MediaWiki\User\User, integer)
#8 /srv/mediawiki/php-1.43.0-wmf.13/extensions/Echo/includes/Hooks.php(979): MediaWiki\Extension\Notifications\Hooks->processMarkAsRead(MediaWiki\User\User, MediaWiki\Request\WebRequest, MediaWiki\Title\Title)
#9 /srv/mediawiki/php-1.43.0-wmf.13/includes/HookContainer/HookContainer.php(159): MediaWiki\Extension\Notifications\Hooks->onSkinTemplateNavigation__Universal(MediaWiki\Skins\Vector\SkinVectorLegacy, array)
#10 /srv/mediawiki/php-1.43.0-wmf.13/includes/HookContainer/HookRunner.php(3591): MediaWiki\HookContainer\HookContainer->run(string, array, array)
#11 /srv/mediawiki/php-1.43.0-wmf.13/includes/skins/SkinTemplate.php(969): MediaWiki\HookContainer\HookRunner->onSkinTemplateNavigation__Universal(MediaWiki\Skins\Vector\SkinVectorLegacy, array)
#12 /srv/mediawiki/php-1.43.0-wmf.13/skins/Vector/includes/SkinVectorLegacy.php(39): SkinTemplate->runOnSkinTemplateNavigationHooks(MediaWiki\Skins\Vector\SkinVectorLegacy, array)
#13 /srv/mediawiki/php-1.43.0-wmf.13/includes/skins/SkinTemplate.php(1361): MediaWiki\Skins\Vector\SkinVectorLegacy->runOnSkinTemplateNavigationHooks(MediaWiki\Skins\Vector\SkinVectorLegacy, array)
#14 /srv/mediawiki/php-1.43.0-wmf.13/includes/skins/SkinTemplate.php(609): SkinTemplate->buildContentNavigationUrlsInternal()
#15 /srv/mediawiki/php-1.43.0-wmf.13/includes/skins/SkinTemplate.php(185): SkinTemplate->getPortletsTemplateData()
#16 /srv/mediawiki/php-1.43.0-wmf.13/includes/skins/SkinMustache.php(139): SkinTemplate->getTemplateData()
#17 /srv/mediawiki/php-1.43.0-wmf.13/skins/Vector/includes/SkinVectorLegacy.php(173): SkinMustache->getTemplateData()
#18 /srv/mediawiki/php-1.43.0-wmf.13/includes/skins/SkinMustache.php(92): MediaWiki\Skins\Vector\SkinVectorLegacy->getTemplateData()
#19 /srv/mediawiki/php-1.43.0-wmf.13/includes/skins/SkinTemplate.php(178): SkinMustache->generateHTML()
#20 /srv/mediawiki/php-1.43.0-wmf.13/includes/Output/OutputPage.php(3027): SkinTemplate->outputPage()
#21 /srv/mediawiki/php-1.43.0-wmf.13/includes/actions/ActionEntryPoint.php(162): MediaWiki\Output\OutputPage->output(boolean)
#22 /srv/mediawiki/php-1.43.0-wmf.13/includes/MediaWikiEntryPoint.php(200): MediaWiki\Actions\ActionEntryPoint->execute()
#23 /srv/mediawiki/php-1.43.0-wmf.13/index.php(58): MediaWiki\MediaWikiEntryPoint->run()
#24 /srv/mediawiki/w/index.php(3): require(string)
#25 {main}
Impact
Notes

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

I am making this private for now, as it is a DDoS vector - we still don't know what it is, but this brings down many wikis. Just being careful here.

It seems that, the initial linter changes were likely just one part of the overall effort to stabilize the system. Once this is resolved, we should consider re-enabling the linting rules to ensure they are unrelated, unless you see a risk on doing it. Thoughts?

This just happened again.
I am not posting the list of queries as it can be just a red herring, but I am going to be posting the list of IPs that were making the write requests that got stuck at around 14:22

110.67.177.154:33858
210.67.188.105:60518
310.67.137.180:49490
410.67.154.85:42378
510.67.160.43:39320
610.67.178.150:58012
710.67.186.61:38074
810.67.144.24:41676
910.67.183.152:59672
1010.67.150.252:36458
1110.67.152.124:48928
1210.67.138.80:55714
1310.67.168.171:51018
1410.67.160.160:53440
1510.67.164.46:56220
1610.67.145.165:48656
1710.67.187.26:39474
1810.67.137.138:58128
1910.67.134.128:32964
2010.67.144.175:50296
2110.67.136.183:46752
2210.67.162.170:37452
2310.67.152.212:45720
2410.67.144.49:54582
2510.67.131.216:48590
2610.67.189.228:38632
2710.67.162.170:37460
2810.67.175.62:58774
2910.67.151.209:36316
3010.67.170.139:45322
3110.67.174.124:46170
3210.67.188.73:53434
3310.67.139.73:41586
3410.67.184.55:42402
3510.67.144.215:38262
3610.67.128.226:41434
3710.67.152.88:53708
3810.67.175.62:58790
3910.67.144.205:57258
4010.67.158.44:57956
4110.67.176.115:37680
4210.67.138.217:50578
4310.67.171.204:53446
4410.67.163.124:38264
4510.67.159.90:51012
4610.67.163.124:38280
4710.67.151.209:36326
4810.67.160.151:60196
4910.67.187.139:39678
5010.67.183.26:40690
5110.67.144.205:57268
5210.67.145.35:47290
5310.67.135.96:55340
5410.67.161.243:45694
5510.67.145.165:48666
5610.67.188.228:39820
5710.67.133.94:53062
5810.67.183.152:59698
5910.67.175.62:58802
6010.67.175.62:58812
6110.67.190.169:53470
6210.67.174.60:38298
6310.67.169.73:41694
6410.67.163.124:38306
6510.67.154.85:33610
6610.67.135.12:56654
6710.67.140.87:40096
6810.67.174.206:42570
6910.67.132.4:52032
7010.67.186.233:60396
7110.67.186.184:45258
7210.67.152.88:51480
7310.67.152.20:34872
7410.67.157.35:42414
7510.67.185.93:40214
7610.67.140.139:47546
7710.67.147.135:52784
7810.67.181.221:38686
7910.67.133.94:37738
8010.67.154.85:33630
8110.67.184.149:42828
8210.67.145.255:50734
8310.67.134.60:34744
8410.67.158.230:46494
8510.67.130.8:33012
8610.67.180.161:42548
8710.67.141.54:34236
8810.67.141.226:48748
8910.67.153.68:46278
9010.67.157.86:56312
9110.67.164.46:56230
9210.67.187.26:39486
9310.67.156.251:49698
9410.67.170.24:58002
9510.67.180.134:57156
9610.67.157.86:56314
9710.67.150.227:57714
9810.67.134.128:32994
9910.67.188.228:39836
10010.67.181.106:33750
10110.67.139.73:41620
10210.67.144.231:59940
10310.67.190.233:44340
10410.67.135.12:56668
10510.67.152.212:45760
10610.67.183.152:53302
10710.67.153.223:39790
10810.67.158.225:41218
10910.67.135.96:55362
11010.67.176.36:51434
11110.67.132.22:42118
11210.67.187.216:52396
11310.67.142.12:44418
11410.67.188.249:51312
11510.67.164.81:58546
11610.67.160.206:37982
11710.67.180.40:43678
11810.67.134.60:34756
11910.67.170.139:48122
12010.67.186.61:38128
12110.67.179.223:32990
12210.67.180.134:57160
12310.67.164.81:58562
12410.67.170.139:48130
12510.67.138.217:43858
12610.67.185.230:34802
12710.67.178.102:54706
12810.67.145.165:39626
12910.67.168.79:50678
13010.67.142.64:42612
13110.67.186.233:60408
13210.67.149.140:55800
13310.67.134.60:34758
13410.67.145.44:50654
13510.67.145.71:58668
13610.67.130.8:33028
13710.67.158.47:39940
13810.67.146.71:35634
13910.67.151.209:37066
14010.67.174.124:36576
14110.67.179.223:33002
14210.67.160.206:37996
14310.67.130.149:58920
14410.67.163.138:51926
14510.67.146.71:35650
14610.67.165.218:57954
14710.67.160.2:43464
14810.67.174.119:54768
14910.67.130.241:55386
15010.67.186.61:38138
15110.67.140.87:40102
15210.67.136.182:45342
15310.67.160.160:56430
15410.67.136.182:45344
15510.67.150.50:59202
15610.67.180.134:57170
15710.67.157.35:42422
15810.67.137.180:59162
15910.67.152.88:51488
16010.67.169.216:41968
16110.67.131.216:48302
16210.67.152.88:51498
16310.67.132.121:53176
16410.67.138.251:55082
16510.67.140.139:58436
16610.67.142.158:49020
16710.67.161.243:45716
16810.67.134.128:32996
16910.67.144.231:59946
17010.67.128.226:52226
17110.67.150.50:59212
17210.67.140.139:58444
17310.67.141.54:34244
17410.67.145.165:39634
17510.67.167.12:36370
17610.67.163.79:59184
17710.67.148.118:53090
17810.67.186.233:60418
17910.67.161.243:45718
18010.67.145.66:34300
18110.67.135.12:56678
18210.67.150.27:34494
18310.67.183.152:53304
18410.67.131.29:40518
18510.67.183.192:45408
18610.67.163.249:53556
18710.67.183.26:49270
18810.67.130.8:52346
18910.67.186.184:58798
19010.67.165.248:43536
19110.67.129.137:45724
19210.67.156.251:57192
19310.67.183.26:49272
19410.67.158.225:41230
19510.67.141.235:51486
19610.67.135.12:56686
19710.67.169.216:41982
19810.67.181.106:33764
19910.67.164.46:56240
20010.67.136.98:49288
20110.67.153.18:49864
20210.67.143.217:52162
20310.67.188.11:44772
20410.67.141.119:55206
20510.67.136.183:59040
20610.67.133.94:37744
20710.67.174.124:36586
20810.67.145.35:55828
20910.67.178.102:54718
21010.67.187.26:39506
21110.67.146.207:34470
21210.67.141.119:55208
21310.67.141.193:41780
21410.67.164.46:56244
21510.67.174.60:38304
21610.67.163.124:38314
21710.67.152.148:42700
21810.67.132.121:53188
21910.67.171.204:53456
22010.67.163.138:35460
22110.67.133.158:40982
22210.67.131.216:48304
22310.67.129.103:60896
22410.67.165.205:39850
22510.67.138.48:58740
22610.67.181.106:42000
22710.67.161.243:50846
22810.67.190.233:45610
22910.67.186.184:58804
23010.67.132.88:36040
23110.67.148.100:43774
23210.67.137.143:35716
23310.67.152.219:35926
23410.67.142.143:55376
23510.67.134.23:43270
23610.67.178.102:54722
23710.67.147.132:42194
23810.67.190.232:45964
23910.67.158.47:41344
24010.67.168.40:43950
24110.67.180.40:43692
24210.67.151.53:60708
24310.67.178.150:57052
24410.67.148.100:43786
24510.67.191.68:41928
24610.67.145.66:33082
24710.67.183.26:49280
24810.67.181.221:49196
24910.67.188.25:57910
25010.67.138.217:43872
25110.67.150.15:52360
25210.67.179.223:33014
25310.67.153.223:39794
25410.67.145.66:33088
25510.67.129.103:60912
25610.67.188.105:57598
25710.67.128.226:52236
25810.67.135.181:50854
25910.67.161.243:50852
26010.67.170.24:58010
26110.67.150.227:57720
26210.67.170.139:48142
26310.67.191.68:41942
26410.67.145.174:53940
26510.67.161.42:57488
26610.67.174.124:36602
26710.67.138.217:43886
26810.67.160.160:56438
26910.67.163.79:56444
27010.67.181.138:57174
27110.67.128.219:56398
27210.67.183.26:49288
27310.67.173.239:55852
27410.67.131.216:48306
27510.67.163.138:35488
27610.67.136.183:59042
27710.67.128.226:52244
27810.67.145.66:33096
27910.67.184.55:34588
28010.67.139.227:47314
28110.67.188.105:57604
28210.67.189.251:44490
28310.67.150.27:34506
28410.67.163.236:35280
28510.67.138.80:55766
28610.67.140.139:58450
28710.67.163.124:49684
28810.67.157.35:42434
28910.67.136.182:45358
29010.67.186.233:39718
29110.67.189.251:44494
29210.67.152.219:35938
29310.67.150.227:57730
29410.67.138.251:55084
29510.67.152.212:50924
29610.67.145.71:49780
29710.67.146.144:56840
29810.67.147.21:33110
29910.67.138.110:49526
30010.67.189.208:55356
30110.67.132.121:49554
30210.67.190.233:45614
30310.67.185.230:34812
30410.67.135.12:43380
30510.67.131.124:46704
30610.67.140.215:53468
30710.67.141.226:37966
30810.67.166.189:49562
30910.67.163.79:56476
31010.67.140.215:53472
31110.67.186.61:39304
31210.67.167.72:45090
31310.67.132.33:59846
31410.67.140.215:53488
31510.67.180.153:39458
31610.67.134.74:47552
31710.67.163.168:38716
31810.67.145.172:45428
31910.67.162.3:33550
32010.67.181.115:45002
32110.67.164.144:50428
32210.67.151.53:37836
32310.67.133.215:59836
32410.67.154.25:50786
32510.67.134.159:46270
32610.67.133.109:45876
32710.67.143.182:40202
32810.67.157.110:41112
32910.67.146.144:56856
33010.67.133.158:42526
33110.67.139.225:47912
33210.67.163.249:36750
33310.67.136.112:48324
33410.67.168.40:54506
33510.67.183.70:58178
33610.67.174.173:46092
33710.67.190.8:39486
33810.67.145.35:40110
33910.67.160.43:56208
34010.67.133.158:42542
34110.67.174.167:58880
34210.67.133.243:45816
34310.67.174.39:48564
34410.67.184.190:36416
34510.67.146.198:56692
34610.67.157.20:56198
34710.67.190.204:50570
34810.67.131.120:32964
34910.67.164.144:45810
35010.67.152.133:38296
35110.67.144.135:44654
35210.67.162.145:47206
35310.67.150.27:44636
35410.67.163.254:38854
35510.67.187.182:36390
35610.67.184.14:57040
35710.67.131.124:45772
35810.67.131.250:56976
35910.67.188.65:60806
36010.67.172.222:59796
36110.67.188.25:37940
36210.67.168.118:58570
36310.67.172.29:37596
36410.67.163.179:57346
36510.67.143.240:51480
36610.67.166.153:41740
36710.67.137.180:52066
36810.67.146.198:56694
36910.67.138.115:58038
37010.67.148.118:41202
37110.67.149.98:53568
37210.67.137.233:53912
37310.67.178.147:54352
37410.67.145.174:36806
37510.67.167.78:46864
37610.67.170.172:37406
37710.67.171.204:41324
37810.67.145.44:56820
37910.67.168.14:60770
38010.67.136.73:32980
38110.67.163.79:38694
38210.67.168.40:48856
38310.67.159.127:33180
38410.67.190.232:35498
38510.67.151.50:51720
38610.67.144.208:57882
38710.67.148.26:58038
38810.67.188.82:41268
38910.67.148.118:59578
39010.67.138.61:38090
39110.67.174.77:54870
39210.67.130.151:47846
39310.67.165.248:51896
39410.67.181.201:34864
39510.67.139.70:52604
39610.67.188.105:36708
39710.67.131.11:57276
39810.67.184.176:45200
39910.67.187.242:45948
40010.67.162.145:55396
40110.67.152.204:53646
40210.67.142.143:41386
40310.67.168.130:57092
40410.67.142.143:41400
40510.67.146.89:59324
40610.67.134.12:58962
40710.67.152.41:41294
40810.67.164.144:36478
40910.67.142.218:50152
41010.67.158.38:58656
41110.67.152.171:42736
41210.67.153.196:38402
41310.67.188.25:37954
41410.67.182.175:54562
41510.67.158.38:56102
41610.67.154.25:36120
41710.67.176.9:51434
41810.67.164.127:33108
41910.67.153.115:48544
42010.67.181.138:57138
42110.67.164.127:33120
42210.67.162.3:43092
42310.67.145.222:60590
42410.67.145.172:60008
42510.67.143.99:49034
42610.67.153.243:45700
42710.67.154.48:37698
42810.67.164.18:51036
42910.67.135.96:48728
43010.67.147.188:57196
43110.67.151.182:35634
43210.67.146.71:41548
43310.67.188.232:60202
43410.67.185.93:55736
43510.67.129.15:60506
43610.67.130.225:56272
43710.67.140.215:54274
43810.67.142.64:39708
43910.67.143.217:47518
44010.67.136.135:48228
44110.67.160.43:55858
44210.67.176.164:48702
44310.67.152.148:46512
44410.64.48.191:46450
44510.67.143.197:34854
44610.67.154.28:60948
44710.67.167.12:38636
44810.67.134.159:50926
44910.67.146.71:50684
45010.67.147.150:55108
45110.67.184.143:45316
45210.67.158.195:47708
45310.67.158.195:47710
45410.67.138.217:44052
45510.67.179.245:42322
45610.67.156.202:49036
45710.67.160.206:53110
45810.67.132.107:48058
45910.67.138.251:44502
46010.67.179.251:38946
46110.67.157.112:57274
46210.67.133.194:50222
46310.67.172.30:45584
46410.67.181.241:35520
46510.67.180.40:51006
46610.67.154.90:32918
46710.67.139.35:55798
46810.67.162.240:58714
46910.67.152.219:49846
47010.67.140.215:55044
47110.67.133.247:37758
47210.67.137.138:41646
47310.67.177.130:48982
47410.67.133.93:51766
47510.67.162.191:46942
47610.67.174.154:54776
47710.67.162.170:39040
47810.67.190.232:54118
47910.67.136.135:50950
48010.67.174.60:55392
48110.67.132.22:49308
48210.67.142.69:60028
48310.67.181.106:35076
48410.67.139.58:44016
48510.194.182.41:45658
48610.67.183.9:34526
48710.67.182.132:57146
48810.67.148.50:54750
48910.67.131.29:56224
49010.67.132.107:43370
49110.67.183.9:39670
49210.67.158.207:35464
49310.67.148.216:33134
49410.67.187.65:56480
49510.67.170.161:34030
49610.67.188.229:45858
49710.67.133.243:45214
49810.67.148.15:58384
49910.67.135.181:36870
50010.67.158.38:52686
50110.67.133.158:43302
50210.67.159.147:48990
50310.67.183.26:59334
50410.67.152.171:44720
50510.67.188.105:47904
50610.67.143.197:41728
50710.67.180.193:33216
50810.67.130.190:49500
50910.67.131.124:55112
51010.67.159.213:49644
51110.67.182.175:45620
51210.67.140.77:43572
51310.67.163.254:38878
51410.67.187.182:49094
51510.67.166.189:45350
51610.67.138.61:51074
51710.67.162.206:53214
51810.67.190.232:60340
51910.67.183.153:50632
52010.67.186.131:38476
52110.67.172.168:55094
52210.67.176.85:58434
52310.67.180.193:39482
52410.67.138.110:58076
52510.67.138.110:58082
52610.67.142.233:50172
52710.67.134.12:54202
52810.67.148.118:49842
52910.67.191.119:47894
53010.67.184.9:38320
53110.67.173.239:38304
53210.67.130.190:44210
53310.67.159.40:46864
53410.67.147.188:60456
53510.67.180.153:34400
53610.67.180.43:55852
53710.67.138.231:57866
53810.67.142.12:36966
53910.67.137.31:59310
54010.67.139.227:46276
54110.67.151.182:43852
54210.67.190.232:32976
54310.67.131.88:42398
54410.67.148.213:58268
54510.67.172.239:43538
54610.67.130.59:41304
54710.67.168.14:43666
54810.67.168.14:43674
54910.67.132.33:43516
55010.67.142.64:39290
55110.67.182.175:47560
55210.67.157.227:52836
55310.67.188.82:37228
55410.67.151.50:50004
55510.67.170.24:60178
55610.67.187.4:58316
55710.67.136.68:41054
55810.67.139.119:42248
55910.67.181.184:34668
56010.67.180.22:51320
56110.67.145.172:54042
56210.67.157.205:57002
56310.67.154.25:33810
56410.67.142.76:50954
56510.67.144.41:51880
56610.67.139.119:42254
56710.67.150.2:43026
56810.67.187.4:58322
56910.67.131.88:44654
57010.67.160.145:41340
57110.67.187.4:58336
57210.67.139.3:48316
57310.67.137.254:47924
57410.67.176.160:53854
57510.67.153.18:40564
57610.67.169.217:43470
57710.67.189.251:53052
57810.67.180.193:55200
57910.67.129.164:39476
58010.67.181.115:58804
58110.67.162.3:56874
58210.67.144.208:41784
58310.67.157.24:49084
58410.67.134.139:47116
58510.67.139.47:51808
58610.67.162.211:37312
58710.67.171.202:37078
58810.67.149.140:41182
58910.67.160.43:37998
59010.67.148.26:46990
59110.67.180.40:42062
59210.67.174.124:33602
59310.67.168.14:32832
59410.67.132.245:60188
59510.67.132.4:56984
59610.67.185.98:49532
59710.67.142.17:37406
59810.67.174.123:56474
59910.67.151.50:54560
60010.67.177.154:55704
60110.67.135.125:40856
60210.67.178.172:56350
60310.67.174.189:45606
60410.67.131.124:42612
60510.67.188.63:41162
60610.67.167.63:53278
60710.67.163.70:44094
60810.67.170.150:58844
60910.67.140.139:59366
61010.67.137.219:52626
61110.67.181.138:40412
61210.67.169.73:56428
61310.67.157.48:44464
61410.67.158.44:50990
61510.67.188.105:33260
61610.67.150.2:57566
61710.67.175.154:46104
61810.67.133.194:45890
61910.67.144.41:54496
62010.67.141.54:43810
62110.67.181.175:48260
62210.67.163.138:52748
62310.67.150.27:60658
62410.194.155.138:49708
62510.67.137.233:35842
62610.67.136.53:57690
62710.67.183.155:47624
62810.67.181.221:59224
62910.67.138.48:40014
63010.67.187.249:57210
63110.67.137.9:60894
63210.67.146.144:49096
63310.67.162.3:40014
63410.67.145.21:37242
63510.67.185.162:43744
63610.67.131.73:43126
63710.67.149.130:52152
63810.67.174.60:46080
63910.67.142.179:58784
64010.67.159.209:60366
64110.67.132.4:43366
64210.67.135.32:56382
64310.67.159.90:41784
64410.67.172.239:35610
64510.67.134.162:49954
64610.67.174.60:46082
64710.67.134.162:49956
64810.67.136.212:51926
64910.67.135.183:50208
65010.67.174.167:49526
65110.67.153.18:36256
65210.67.161.35:47020
65310.67.149.218:47116
65410.67.142.207:46182
65510.67.144.231:49262
65610.67.186.250:49326
65710.67.136.71:48516
65810.67.150.136:34248
65910.67.148.229:60380
66010.67.149.108:49868
66110.67.146.71:34244
66210.67.190.169:54276
66310.67.150.67:46296
66410.67.138.48:37668
66510.67.134.159:59974
66610.67.133.243:56620
66710.67.183.198:40760
66810.67.145.236:60744
66910.67.174.77:41500
67010.67.188.11:47680
67110.67.132.33:55476
67210.67.149.130:38692
67310.67.130.8:56590
67410.67.132.15:52064
67510.67.130.21:37278
67610.67.157.205:51050
67710.67.190.123:41160
67810.67.158.195:52318
67910.67.180.134:60032
68010.67.162.162:37478
68110.67.149.108:49882
68210.67.158.44:52426
68310.67.137.254:56172
68410.67.152.124:42100
68510.67.177.130:49744
68610.67.180.193:37404
68710.67.177.141:58760
68810.67.180.193:37410
68910.67.184.0:43002
69010.67.142.76:49280
69110.67.132.121:39070
69210.67.151.209:39586
69310.67.129.105:48442
69410.67.174.242:37196
69510.67.187.138:42604
69610.67.131.88:41716
69710.67.135.182:57210
69810.67.190.156:46696
69910.67.147.188:52008
70010.67.151.50:50674
70110.67.152.148:35984
70210.67.152.41:56724
70310.67.149.150:34470
70410.67.185.223:46784
70510.67.164.18:53258
70610.67.132.33:39288
70710.67.134.139:51864
70810.67.182.175:36662
70910.67.174.167:58868
71010.67.130.151:42620
71110.67.153.223:46114
71210.67.180.198:53282
71310.67.176.85:54142
71410.67.185.122:57892
71510.67.181.115:49052
71610.67.165.218:37772
71710.67.191.249:47462
71810.67.175.40:39004
71910.67.138.42:52884
72010.67.191.119:51930
72110.67.133.194:54230
72210.67.154.90:60438
72310.67.174.176:42000
72410.67.152.41:56428
72510.67.175.177:56356
72610.67.166.189:35446
72710.67.131.120:57776
72810.67.160.43:35044
72910.67.174.242:52514
73010.67.174.242:52512
73110.67.130.149:59762
73210.67.164.46:45452
73310.67.168.118:55272
73410.67.180.203:55538
73510.67.180.198:52372
73610.67.142.230:47700
73710.67.162.211:40722
73810.67.141.193:54888
73910.67.146.244:38540
74010.67.145.35:46312
74110.67.157.86:47156
74210.67.156.202:35414
74310.67.178.147:36824
74410.67.186.205:49646
74510.67.180.40:60096
74610.67.129.164:43936
74710.67.145.255:48656
74810.67.129.128:50964
74910.67.139.70:56006
75010.67.172.30:43024
75110.67.163.172:50854
75210.67.150.150:47408
75310.67.175.177:46960
75410.67.138.66:55046
75510.67.154.48:53384
75610.67.138.48:34828
75710.67.176.164:49546
75810.67.188.25:50222
75910.67.175.154:43982
76010.67.146.52:55266
76110.67.153.193:34306
76210.67.153.223:35898
76310.67.141.235:40690
76410.67.172.167:38014
76510.67.169.216:59970
76610.67.151.50:54294
76710.67.183.192:45948
76810.67.148.45:43906
76910.67.132.116:46252
77010.67.160.158:41736
77110.67.141.193:50466
77210.67.132.4:53774
77310.67.166.97:40332
77410.67.187.249:59218
77510.67.188.73:53066
77610.67.139.34:60596
77710.67.190.37:43422
77810.67.132.245:48898
77910.67.135.125:40516
78010.67.152.124:35324
78110.67.130.21:33734
78210.67.183.82:54208
78310.67.137.143:35988
78410.67.151.53:48752
78510.67.140.185:33552
78610.67.165.218:50464
78710.67.152.148:33474
78810.67.181.115:60798
78910.67.190.204:37316
79010.67.183.70:36960
79110.67.128.252:40358
79210.67.135.157:51954
79310.67.145.255:35438
79410.67.171.202:55636
79510.67.183.97:33480
79610.67.151.166:45846
79710.67.187.26:53988
79810.67.137.253:52818
79910.67.183.147:42096
80010.67.181.0:57126
80110.67.134.74:50222
80210.67.156.8:40192
80310.67.136.112:49396
80410.67.170.60:42116
80510.67.133.158:52354
80610.67.164.127:42812
80710.67.183.70:36962
80810.67.184.176:33450
80910.67.166.189:47232
81010.67.152.219:50202
81110.67.133.194:50818
81210.67.175.40:46780
81310.67.157.112:34652
81410.67.160.145:34404
81510.67.160.158:38720
81610.67.134.74:50232
81710.67.143.225:33222
81810.67.166.189:47236
81910.67.141.226:49902
82010.67.167.78:57972
82110.67.144.135:32956
82210.67.190.123:47164
82310.67.133.158:52356
82410.67.153.115:60342
82510.67.146.159:33148
82610.67.183.218:33728
82710.67.163.168:58012
82810.67.154.85:50220
82910.67.160.53:55394
83010.67.157.86:39802
83110.67.139.61:36598
83210.67.138.251:45232
83310.67.176.115:42540
83410.67.174.233:47538
83510.67.172.239:54844
83610.67.147.135:52068
83710.67.184.9:60022
83810.67.172.30:52794
83910.67.187.4:33188
84010.67.184.176:40852
84110.67.150.67:37634
84210.67.128.207:50788
84310.67.161.35:41258
84410.67.181.195:34182
84510.67.161.221:43920
84610.67.152.241:39618
84710.67.174.233:47554
84810.67.133.109:44716
84910.67.169.73:60124
85010.67.169.216:51938
85110.67.161.221:44256
85210.67.150.252:54002
85310.67.177.141:47814
85410.67.162.185:35766
85510.67.172.167:55694
85610.67.172.214:45342
85710.67.142.236:54966
85810.67.139.61:46830
85910.67.170.188:33170
86010.67.168.171:52084
86110.67.184.143:58942
86210.67.130.8:50252
86310.67.154.220:35456
86410.67.143.147:56028
86510.67.146.254:42150
86610.67.144.147:56546
86710.67.146.207:52940
86810.67.176.36:57298
86910.67.165.205:43018
87010.67.134.12:37576
87110.194.131.107:35290
87210.67.146.159:36566
87310.67.138.110:54360
87410.67.170.15:52326
87510.67.150.34:34432
87610.67.157.227:42344
87710.67.181.133:40798
87810.67.136.112:59250
87910.67.154.60:45732
88010.67.161.53:52248
88110.67.150.15:59818
88210.67.161.35:57334
88310.67.132.33:38056
88410.67.184.143:58958
88510.67.134.12:37584
88610.67.174.173:51382
88710.67.144.135:60248
88810.67.145.172:35510
88910.67.152.133:35604
89010.67.166.153:39696
89110.67.156.8:47982
89210.67.143.217:54870
89310.67.142.69:57896
89410.67.148.152:37868
89510.67.136.71:44786
89610.67.187.12:51264
89710.67.185.98:56128
89810.67.186.205:43918
89910.67.171.223:38716
90010.67.148.220:54556
90110.67.139.47:34116
90210.67.146.89:52886
90310.67.167.63:43424
90410.67.138.109:51020
90510.67.184.231:60516
90610.67.168.79:57988
90710.67.190.232:40498
90810.67.152.204:60616
90910.67.181.241:39638
91010.67.185.230:53486
91110.67.178.183:60120
91210.67.135.183:55162
91310.67.188.3:53256
91410.67.177.154:59546
91510.67.154.150:54414
91610.67.182.175:58574
91710.67.157.86:37598
91810.67.141.43:33820
91910.67.172.214:34428
92010.67.167.115:57292
92110.67.147.132:50152
92210.67.186.233:59844
92310.67.187.150:41646
92410.67.154.28:58076
92510.67.152.124:46812
92610.67.151.156:60228
92710.67.142.236:49126
92810.67.138.217:53316
92910.67.186.131:51846
93010.67.148.249:37618
93110.67.134.64:55778
93210.67.174.119:51700
93310.67.185.93:51204
93410.67.154.220:34968
93510.67.145.213:45966
93610.67.129.137:42864
93710.67.141.50:57906
93810.67.141.93:46108
93910.67.169.105:58600
94010.67.150.150:60840
94110.67.135.183:33120
94210.67.178.172:35632
94310.67.162.211:51522
94410.67.164.18:42214
94510.67.176.36:38116
94610.67.153.223:37032
94710.67.146.75:59152
94810.67.172.214:47230
94910.67.177.224:55174
95010.67.153.118:46414
95110.67.181.0:53548
95210.67.131.250:57922
95310.67.149.218:44842
95410.67.139.175:51840
95510.67.177.224:55182
95610.67.160.151:38688
95710.67.154.60:59506
95810.67.149.130:33726
95910.67.188.249:56198
96010.67.189.208:49050
96110.67.134.23:35896
96210.67.136.135:44150
96310.67.151.53:34344
96410.67.188.11:54508
96510.67.189.232:47972
96610.67.148.229:38526
96710.67.140.215:51532
96810.67.143.217:34278
96910.67.142.124:55184
97010.67.158.23:54908
97110.67.142.69:51054
97210.67.191.249:43390
97310.194.182.1:58148
97410.67.183.155:34594
97510.67.143.197:54294
97610.67.130.204:58212
97710.67.146.144:60036
97810.67.156.8:41874
97910.67.132.33:41004
98010.67.172.168:50968
98110.67.167.78:50352
98210.67.172.168:50954
98310.67.164.144:38412
98410.67.136.190:53888
98510.67.152.171:54406
98610.67.136.73:39030
98710.67.130.21:56668
98810.67.138.231:37350
98910.67.144.215:48914
99010.67.188.105:46468
99110.67.134.139:57852
99210.67.158.225:54038
99310.67.130.151:54184
99410.67.136.183:56920
99510.67.186.184:37804
99610.67.174.0:56550
99710.67.174.60:49720
99810.67.146.207:34896
99910.67.156.52:54250
100010.67.150.34:51688
100110.67.168.40:55920
100210.67.152.171:54408
100310.67.188.43:60154
100410.67.168.14:55846
100510.67.145.222:41488
100610.67.189.208:46794
100710.67.153.115:50846
100810.67.154.220:53984
100910.67.183.218:40886
101010.67.176.16:56252
101110.67.152.124:40136
101210.67.148.15:34090
101310.67.178.113:40722
101410.67.177.229:48910
101510.67.142.162:42582
101610.67.159.44:45776
101710.67.138.195:36158
101810.67.145.21:45514
101910.67.142.64:55972
102010.67.183.198:53584
102110.67.142.124:45626
102210.67.140.87:47742
102310.67.176.115:41330
102410.67.131.120:56316
102510.67.180.161:58086
102610.67.148.220:46054
102710.67.176.115:41336
102810.67.180.224:44356
102910.67.138.48:57062
103010.67.132.214:41478
103110.67.162.185:44220
103210.67.149.218:47572
103310.67.183.97:48244
103410.67.152.148:49124
103510.67.137.94:37448
103610.67.190.204:55450
103710.67.190.232:42998
103810.67.136.182:50506
103910.67.186.131:42060
104010.67.140.139:44912
104110.67.150.136:43130
104210.67.158.47:55548
104310.67.190.98:51614
104410.67.175.40:51722
104510.194.162.150:56456
104610.67.133.191:53646
104710.67.150.136:56792
104810.67.133.93:46972
104910.67.181.241:36702
105010.67.134.64:51202
105110.67.137.253:47946
105210.67.129.103:39980
105310.67.164.83:49638
105410.67.153.207:58956
105510.67.135.125:51436
105610.67.188.112:53332
105710.67.174.173:50646
105810.67.176.115:39206
105910.67.133.109:50946
106010.67.145.71:35958
106110.67.172.167:57448
106210.67.174.119:35528
106310.67.132.214:51926
106410.67.181.92:45856
106510.67.143.217:41834
106610.67.150.27:46226
106710.67.140.112:59484
106810.67.145.44:40112
106910.67.148.216:39086
107010.67.184.55:47702
107110.67.129.137:49268
107210.67.190.38:48766
107310.67.148.53:55806
107410.67.190.96:34564
107510.67.144.49:42484
107610.67.180.224:34736
107710.67.139.13:46794
107810.67.184.55:57454
107910.67.135.122:43548
108010.67.133.243:34288
108110.67.145.21:60312
108210.67.165.218:35528
108310.67.156.8:33258
108410.67.162.191:52854
108510.67.172.168:55064
108610.67.181.96:53948
108710.67.143.217:41840
108810.67.148.50:38836
108910.67.157.35:35116
109010.67.181.187:39718
109110.67.154.17:47744
109210.67.168.40:53904
109310.67.132.238:54718
109410.67.183.147:41186
109510.67.143.217:46330
109610.194.175.16:56048
109710.67.148.233:51144
109810.67.159.75:34358
109910.67.140.112:48040
110010.67.183.192:35924
110110.67.145.255:57086
110210.67.167.12:57772
110310.67.188.105:43778
110410.67.136.190:45430
110510.67.134.85:57112
110610.67.164.18:52750
110710.67.136.53:32998
110810.67.133.93:48118
110910.67.159.75:52904
111010.67.191.126:43912
111110.67.148.216:35754
111210.67.151.17:45290
111310.67.145.255:56756
111410.67.128.207:47746
111510.67.167.12:46524
111610.67.129.105:48940
111710.67.158.21:33250
111810.67.181.105:44860
111910.67.190.175:39242
112010.67.188.239:39986
112110.67.149.150:41760
112210.67.187.12:47686
112310.67.145.174:53174
112410.67.181.92:34704
112510.67.154.28:37132
112610.67.162.206:38794
112710.67.162.223:35604
112810.67.176.164:59336
112910.67.154.30:47128
113010.67.163.236:57932
113110.67.130.241:33006
113210.67.185.214:48718
113310.67.174.119:56384
113410.67.137.233:57882
113510.67.143.197:40598
113610.67.161.214:50032
113710.67.152.143:49296
113810.67.149.98:46336
113910.67.184.231:56010
114010.67.188.3:41590
114110.67.170.60:34096
114210.67.188.11:57252
114310.67.174.189:59634
114410.67.174.167:55392
114510.67.172.214:51790
114610.67.136.135:33350
114710.67.168.72:46438
114810.67.163.249:37906
114910.67.190.221:49428
115010.67.163.79:44900
115110.67.134.139:59818
115210.67.158.38:34924
115310.67.157.205:38834
115410.67.154.25:41118
115510.67.132.121:50814
115610.67.136.112:56956
115710.67.131.11:46028
115810.67.187.139:48772
115910.67.160.145:44522
116010.67.139.127:34920
116110.67.141.100:41076
116210.67.136.217:55908
116310.67.163.223:45398
116410.67.159.75:45788
116510.67.150.136:38692
116610.67.144.49:47170
116710.67.146.71:49610
116810.67.131.120:42100
116910.67.159.25:40778
117010.67.148.45:44132
117110.67.169.93:43116
117210.67.142.143:43946
117310.67.136.73:48986
117410.67.132.116:55420
117510.67.135.181:35850
117610.67.185.78:45086
117710.67.169.93:59916
117810.67.174.154:55936
117910.67.161.30:37366
118010.67.136.73:39494
118110.67.181.96:55696
118210.67.161.30:54166
118310.67.149.98:50814
118410.67.185.219:52022
118510.67.165.205:55512
118610.67.173.239:36882
118710.67.170.172:35542
118810.67.181.201:45652
118910.67.176.160:41272
119010.67.159.40:60206
119110.67.144.175:45494
119210.67.176.99:50370
119310.67.191.119:45192
119410.67.166.101:42928
119510.67.135.125:58194
119610.67.128.155:54144
119710.67.150.2:59454
119810.67.190.34:32930
119910.67.145.236:49356
120010.67.140.77:53154
120110.67.158.76:59448
120210.67.167.12:47174
120310.67.132.214:53320
120410.67.139.119:44558
120510.67.131.76:39106
120610.67.170.172:49710
120710.67.160.249:46812
120810.67.180.43:44914
120910.67.174.77:34590
121010.67.145.198:35226
121110.67.137.94:57958
121210.67.182.154:47988
121310.67.179.110:55580
121410.67.149.150:40780
121510.67.149.98:37098
121610.67.145.236:48760
121710.67.153.196:48228
121810.67.137.180:58992
121910.67.161.137:39564
122010.67.147.163:35158
122110.67.149.71:48388
122210.67.153.58:39506
122310.67.161.127:43600
122410.67.146.71:43412
122510.67.174.154:48480
122610.67.156.49:52612
122710.67.151.3:52860
122810.67.163.72:40880
122910.67.170.24:58412
123010.67.161.116:56204
123110.67.188.136:50886
123210.67.149.108:53422
123310.67.167.72:42846
123410.67.181.175:33228
123510.67.132.116:60522
123610.67.134.46:57076
123710.67.144.226:47758
123810.67.151.3:57528

While investigating primary DB queries from RevisionStore to find the source of the ::fetchRevisionRowFromConds queries, we found that DiscussionsTools attempts to fetch the previous revision of the revision that has been edited from the primary DB on every edit.
https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DiscussionTools/+/1062733 optimizes this a bit.

Quick update from another occurrence starting at ~ 20:45 UTC today:

The processlist polling that @CDanis set up worked quite nicely here. Specifically, if you look at the sample files in /home/cdanis/processlist and do something simple like count threads not in Command: Sleep, you see something like:

2024-08-14T20:45:30+00:00 20
2024-08-14T20:45:45+00:00 237
2024-08-14T20:46:00+00:00 489
2024-08-14T20:46:15+00:00 692
2024-08-14T20:46:30+00:00 764
2024-08-14T20:46:45+00:00 833
2024-08-14T20:47:00+00:00 882
2024-08-14T20:47:15+00:00 965
2024-08-14T20:47:31+00:00 1042
2024-08-14T20:47:46+00:00 1084
2024-08-14T20:48:01+00:00 1156
2024-08-14T20:48:16+00:00 1255
2024-08-14T20:48:31+00:00 1482
2024-08-14T20:48:46+00:00 1543
2024-08-14T20:49:01+00:00 1583
2024-08-14T20:49:17+00:00 1623
2024-08-14T20:49:32+00:00 1663
2024-08-14T20:49:47+00:00 1706
2024-08-14T20:50:02+00:00 1744
2024-08-14T20:50:18+00:00 1789
2024-08-14T20:50:33+00:00 1938
2024-08-14T20:55:39+00:00 1939
2024-08-14T21:00:00+00:00 548
2024-08-14T21:04:36+00:00 66
2024-08-14T21:04:51+00:00 26

Setting aside the possibility of missing a short-lived precursor event that cleared before we could see it, the 20:45:45 sample looks promising.

Looking at that processlist, of the 219 commonswiki queries present, 167 are UserEditCountUpdate::doUpdate for the same exact user_id. Of the remaining 52 queries, 40 are ChangeTagsStore::updateTags predominantly related to Cat-a-lot, which we believe this user was using at the time.

These eventually clear out by the 20:55:39 sample, which seems to be a bit before db1160 appears to fully recover some time around / after 21:00. These queries also seem to be rolling back quite slowly (e.g., a given query might be in rollback for 15-30s, with only 1 or 2 in Rollback per sample over the majority of the bad period).

Of the 12 remaining queries present in the 20:45:45 sample, I've spot-checked what happens to them in later samples (e.g., by tracking the same client ID), and they all appear to clear out quickly - either with no subsequent queries on that ID, or replaced with a UserEditCountUpdate::doUpdate shortly after.

Anyway, while far from a definitive culprit, these queries do seem to be the very first to get "stuck" after which further queries quickly pile up. It's unclear to me why these in particular would be special, as they seem to be rather simple in practice.

Paste containing the 20:45:45 sample: https://phabricator.wikimedia.org/P67322

This is most likely unrelated, Thumbor doesn't really touch the databases

This happened again, but it was very brief and didn't generate any pages.
Things I saw:

  • Hundreds of DELETEs to linter like this one:
| 3985847190 | wikiuser2023    | 10.67.176.184:39952  | commonswiki | Query       |     166 | Updating                                                      | DELETE /* MediaWiki\Linter\Database::setForPage  */ FROM `linter` WHERE linter_page = 145347895      |    0.000 |
  • Hundreds of SELECTs again on the master, to the page table similar to this
| 3985845073 | wikiuser2023    | 10.67.137.19:45296   | commonswiki | Query       |     261 | Statistics                                                    | SELECT /* WikiPage::lockAndGetLatest  */  page_latest  FROM `page`    WHERE page_id = 13828077 AND p |    0.000 |

List of PODS generating writes:

110.64.16.77:48882
210.67.132.180:56876
310.67.147.29:60142
410.67.152.212:51008
510.67.149.144:41394
610.67.133.174:52418
710.67.135.118:43748
810.67.191.124:49726
910.67.129.10:55766
1010.67.176.22:43972
1110.67.136.14:40968
1210.67.181.179:57398
1310.67.151.1:58222
1410.67.184.170:60616
1510.67.139.85:50278
1610.67.137.236:35996
1710.67.188.79:55782
1810.67.130.142:52586
1910.67.163.97:41066
2010.67.161.118:46818
2110.67.148.220:48084
2210.67.158.244:35176
2310.67.149.12:54384
2410.67.163.188:51048
2510.67.135.118:43758
2610.67.144.201:57818
2710.67.163.233:43212
2810.67.163.113:40090
2910.67.186.167:43710
3010.67.166.24:45796
3110.67.179.93:53598
3210.67.149.207:44920
3310.67.185.142:37822
3410.67.168.146:37928
3510.67.168.146:37942
3610.67.144.103:36588
3710.67.129.72:52132
3810.67.164.5:57680
3910.67.137.236:36012
4010.67.145.104:43610
4110.67.152.212:51016
4210.67.152.212:51018
4310.67.148.76:49600
4410.67.184.33:38754
4510.67.133.56:49014
4610.67.185.142:37830
4710.67.158.29:42236
4810.67.144.189:55212
4910.67.154.9:39064
5010.67.157.76:39022
5110.67.140.244:57876
5210.67.163.113:40100
5310.67.145.104:43614
5410.67.148.147:52974
5510.67.181.118:60000
5610.67.184.170:60632
5710.67.161.118:46832
5810.67.133.174:52434
5910.67.153.156:55052
6010.67.158.244:35182
6110.67.179.208:46028
6210.67.159.48:47150
6310.67.176.22:43986
6410.67.188.79:55798
6510.67.180.222:41476
6610.67.152.141:40170
6710.67.158.244:35190
6810.67.133.244:58790
6910.67.162.240:49492
7010.67.177.227:48620
7110.67.129.5:42024
7210.67.159.183:45718
7310.67.135.118:43768
7410.67.129.162:40490
7510.67.136.14:40970
7610.67.188.79:55812
7710.67.184.170:60640
7810.67.156.167:43570
7910.67.163.97:41070
8010.67.180.3:60894
8110.67.185.96:33902
8210.67.137.236:51770
8310.67.132.48:36146
8410.67.147.150:37666
8510.67.179.208:46044
8610.67.161.118:46838
8710.67.145.62:39986
8810.67.172.250:50986
8910.67.180.222:41484
9010.67.138.131:40270
9110.67.139.85:50288
9210.67.162.47:45926
9310.67.162.47:45930
9410.67.158.29:42244
9510.67.161.77:41652
9610.67.149.12:37252
9710.67.130.220:60998
9810.67.138.40:33150
9910.67.143.20:53424
10010.67.185.206:57392
10110.67.129.5:36686
10210.67.181.118:60962
10310.67.148.86:47400
10410.67.160.148:48290
10510.67.175.43:39476
10610.67.171.202:49878
10710.67.143.20:53430
10810.67.144.147:40976
10910.67.172.6:33358
11010.67.180.228:47954
11110.67.131.235:35914
11210.67.162.46:45586
11310.67.152.191:37408
11410.67.148.180:46940
11510.67.137.30:47826
11610.67.150.242:50634
11710.67.183.100:32820
11810.67.157.35:41422
11910.67.174.25:36248
12010.67.166.226:34660
12110.67.161.232:56098
12210.67.136.54:44954
12310.67.132.176:60474
12410.67.186.167:41184
12510.67.160.38:36668
12610.67.158.66:57850
12710.67.170.178:56538
12810.67.188.111:38328
12910.67.190.78:40906
13010.67.190.78:40910
13110.67.152.191:37412
13210.67.191.235:44586
13310.67.140.147:58450
13410.67.157.20:53666
13510.67.138.1:40532
13610.67.151.6:45174
13710.67.147.150:40152
13810.67.146.93:40224
13910.67.183.100:33594
14010.67.162.170:45852
14110.67.142.110:38678
14210.67.131.222:52732
14310.67.140.194:57812
14410.67.180.61:42208
14510.67.152.191:37414
14610.67.157.251:49758
14710.67.133.223:50790
14810.67.142.15:55850
14910.67.139.143:60578
15010.67.182.172:42924
15110.67.132.176:60476
15210.67.138.167:58346
15310.67.157.96:58652
15410.67.129.137:38536
15510.67.162.235:41634
15610.67.189.223:41054
15710.67.175.43:39478
15810.67.159.23:55374
15910.67.133.180:49364
16010.67.162.26:47544
16110.67.168.3:46902
16210.67.185.155:37462
16310.67.140.77:43750
16410.67.191.121:54984
16510.67.172.185:40396
16610.67.170.178:56546
16710.67.148.147:38720
16810.67.180.228:47982
16910.67.170.142:52526
17010.67.160.85:42914
17110.67.159.183:56962
17210.67.129.149:58148
17310.67.135.236:50560
17410.67.162.26:43600
17510.67.170.178:56554
17610.67.184.0:56096
17710.67.164.73:52728
17810.67.161.83:47610
17910.67.171.30:43034
18010.67.185.142:40846
18110.67.190.140:50242
18210.67.164.36:35078
18310.67.174.128:40270
18410.67.160.180:36684
18510.67.169.254:33722
18610.67.179.223:56762
18710.67.177.134:50134
18810.67.131.46:38996
18910.67.171.223:44744
19010.67.132.245:44460
19110.67.149.180:40122
19210.67.143.20:47118
19310.67.161.232:56104
19410.67.138.9:38966
19510.67.138.131:54620
19610.67.162.46:36012
19710.67.176.15:49062
19810.67.153.156:43352
19910.67.140.194:57064
20010.67.148.192:60238
20110.67.180.61:44316
20210.67.170.142:44836
20310.67.167.58:52700
20410.67.148.50:54316
20510.67.143.87:36550
20610.67.162.170:41038
20710.67.159.48:39812
20810.67.191.68:52904
20910.67.177.175:48388
21010.67.135.8:32882
21110.67.186.126:37322
21210.67.162.235:44606
21310.67.158.231:39114
21410.67.168.105:55078
21510.67.188.91:37462
21610.67.162.46:55400
21710.67.178.85:37268
21810.67.160.118:41206
21910.67.129.5:32952
22010.67.140.187:42256
22110.67.183.100:44394
22210.67.138.211:54474
22310.67.181.200:42936
22410.67.140.194:42454
22510.67.147.29:43020
22610.67.164.179:44026
22710.67.132.32:50282
22810.67.181.104:42130
22910.67.140.194:42470
23010.67.146.189:49414
23110.67.134.123:47494
23210.67.139.50:50260
23310.67.140.188:40338
23410.64.0.155:35256
23510.67.183.187:55352
23610.67.164.131:38086
23710.67.185.206:56936
23810.67.160.85:43484
23910.67.179.254:58142
24010.67.157.35:53520
24110.67.161.231:34390
24210.67.188.111:52348
24310.67.130.129:55952
24410.67.181.131:43928
24510.67.157.45:57962
24610.67.176.86:46378
24710.67.161.34:57404
24810.67.143.193:46124
24910.67.140.7:40408
25010.67.143.193:46134
25110.67.162.26:33556
25210.67.129.149:41934
25310.67.138.1:46356
25410.67.164.39:41554
25510.67.175.57:40848
25610.67.133.180:53636
25710.67.129.149:41946
25810.67.140.194:38972
25910.67.164.36:37836
26010.67.162.183:39790
26110.67.159.82:47522
26210.67.187.244:41836
26310.67.148.194:59056
26410.67.181.200:54938
26510.67.129.5:58368
26610.67.160.56:57714
26710.67.160.15:47300
26810.67.149.64:37090
26910.67.157.98:60604
27010.67.172.243:46050
27110.67.161.77:54420
27210.67.167.15:37800
27310.67.130.81:50502
27410.67.176.165:47790
27510.67.180.179:46566
27610.67.143.222:42358
27710.67.133.157:46112
27810.67.149.98:45706
27910.67.136.14:57866
28010.67.163.188:58830
28110.67.187.244:41842
28210.67.185.142:56434
28310.67.162.144:58458
28410.67.168.146:40370
28510.67.169.126:37360
28610.67.140.194:56780
28710.67.181.200:53826
28810.67.188.43:40740
28910.67.160.118:55074
29010.67.146.89:45772
29110.67.152.191:33268
29210.67.144.137:33968
29310.67.153.28:50932
29410.67.164.73:51910
29510.67.175.57:36804
29610.67.145.208:44500
29710.67.170.178:57782
29810.67.179.254:38452
29910.67.137.178:44504
30010.67.170.178:57788
30110.67.164.73:51914
30210.67.180.254:49676
30310.67.137.242:47674
30410.67.160.38:37466
30510.67.151.239:43738
30610.67.160.56:56030
30710.67.191.124:50380
30810.67.129.149:40222
30910.67.149.144:55360
31010.67.139.255:34674
31110.67.152.212:59064
31210.67.168.3:44476
31310.67.143.84:32802
31410.67.144.124:52002
31510.67.128.221:51424
31610.67.161.118:60260
31710.67.142.232:58602
31810.67.164.36:60150
31910.67.139.50:49276
32010.67.138.1:38722
32110.67.188.55:54714
32210.67.143.193:47914
32310.67.130.12:36160
32410.67.149.144:59242
32510.67.145.208:35170
32610.67.142.15:53248
32710.67.175.16:43954
32810.67.186.203:33970
32910.67.147.30:34866
33010.67.135.96:38668
33110.67.129.79:46612
33210.67.189.201:55908
33310.67.136.202:36630
33410.67.138.209:44512
33510.67.180.228:47840
33610.67.171.46:45518
33710.67.175.128:43152
33810.67.188.189:41174
33910.67.144.230:43680
34010.67.184.25:55208
34110.67.180.217:53682
34210.67.190.23:33412
34310.67.132.180:55148
34410.67.142.232:56704
34510.67.190.180:36690
34610.67.180.221:50332
34710.67.150.144:56272
34810.67.134.106:53764
34910.67.190.180:36706
35010.67.156.137:43126
35110.67.158.66:56876
35210.67.143.20:33418
35310.67.172.232:47750
35410.67.135.75:58040
35510.67.177.214:42970
35610.67.140.147:40608
35710.67.130.18:38624
35810.67.149.9:41098
35910.67.145.221:36108
36010.67.183.47:36610
36110.67.164.36:35906
36210.67.175.86:54344
36310.67.156.167:37468
36410.67.191.68:45974
36510.67.139.85:51332
36610.67.162.144:44158
36710.67.175.54:57676
36810.67.190.143:47514
36910.67.136.184:50292
37010.67.188.12:37480
37110.67.129.137:41412
37210.67.183.216:60678
37310.67.164.73:44320
37410.67.164.172:51160
37510.67.148.86:49710
37610.67.131.235:43754
37710.67.156.54:43250
37810.67.148.125:53326
37910.67.139.85:51354
38010.67.139.161:41342
38110.67.183.34:36416
38210.67.131.23:33300
38310.67.148.233:38564
38410.67.164.179:56352
38510.67.165.87:45116
38610.67.160.180:33970
38710.67.161.184:34416
38810.67.191.24:51912
38910.67.133.223:56572
39010.67.149.255:55558
39110.67.152.44:44212
39210.67.145.231:53332
39310.67.148.125:38802
39410.67.154.18:38360
39510.67.190.117:58064
39610.67.141.53:45128
39710.67.172.6:48108
39810.67.145.227:40704
39910.67.164.73:38002
40010.67.129.57:47990
40110.67.140.235:51178
40210.67.154.147:37154
40310.67.161.40:36472
40410.67.142.15:43946
40510.67.174.132:50068
40610.67.164.36:36788
40710.67.181.142:56662
40810.67.148.110:42156
40910.67.158.231:35280
41010.67.168.146:47728
41110.67.132.238:51102
41210.67.163.97:51108
41310.67.172.185:35460
41410.67.176.184:44572
41510.67.138.209:49362
41610.67.188.91:42348
41710.67.186.161:33174
41810.67.148.15:45018
41910.67.188.185:59436
42010.67.189.158:55168
42110.67.157.35:60836
42210.67.176.86:47946
42310.67.141.42:53662
42410.67.191.252:52368
42510.67.152.45:33262
42610.67.173.201:49996
42710.67.163.188:45986
42810.67.153.118:41218
42910.67.176.64:46486
43010.67.190.84:52530
43110.67.158.29:41256
43210.67.128.221:48008
43310.64.0.155:33506
43410.67.167.15:52588
43510.67.130.94:51852
43610.67.162.46:47648
43710.67.149.9:44892
43810.67.184.170:33730
43910.67.154.18:60438
44010.67.175.16:44152
44110.67.154.50:36254
44210.67.146.73:38286
44310.67.151.237:43044
44410.67.173.244:54278
44510.67.128.253:49486
44610.67.167.15:41516
44710.64.48.14:33618
44810.67.142.234:55598
44910.67.190.88:33740
45010.67.162.46:47662
45110.67.156.60:50384
45210.67.185.122:56432
45310.67.136.54:53014
45410.67.133.237:43058
45510.67.144.103:51180
45610.67.176.64:45916
45710.67.147.150:55200
45810.67.144.124:52184
45910.67.152.36:35090
46010.67.130.107:57712
46110.67.140.7:46878
46210.67.140.87:60470
46310.67.176.165:58460
46410.67.140.194:46818
46510.67.130.12:49404
46610.67.160.92:59852
46710.67.175.172:39594
46810.67.183.187:58670
46910.67.183.228:42578
47010.67.170.169:39274
47110.67.185.142:53138
47210.67.188.43:55482
47310.67.180.53:34048
47410.67.144.189:38950
47510.67.180.61:44994
47610.67.171.46:44000
47710.67.144.124:52190
47810.67.162.235:60614
47910.67.171.30:43128
48010.67.185.206:51238
48110.67.157.205:55176
48210.67.137.242:38312
48310.64.48.192:34480
48410.67.180.61:45004
48510.67.152.44:32856
48610.67.139.27:46614
48710.67.135.87:39152
48810.67.137.162:42866
48910.67.189.158:39000
49010.67.148.15:45418
49110.64.48.193:33642
49210.67.133.157:43872
49310.67.173.250:41600
49410.67.149.255:50568
49510.67.186.213:42476
49610.67.143.222:57784
49710.67.181.212:37938
49810.67.158.66:45396
49910.67.173.244:59512
50010.67.160.15:60224
50110.67.186.203:45532
50210.67.172.6:50218
50310.67.140.207:60496
50410.67.133.174:47344
50510.67.129.10:37308
50610.67.176.165:42034
50710.67.153.118:33838
50810.67.185.202:43916
50910.67.138.42:60836
51010.67.152.197:34174
51110.67.147.138:59350
51210.67.151.1:56934
51310.67.172.139:39998
51410.67.140.12:49362
51510.64.0.155:38586
51610.67.179.93:55136
51710.67.148.26:47980
51810.67.165.115:58824
51910.67.134.50:59498
52010.67.190.78:35066
52110.67.146.18:56920
52210.67.152.45:45180
52310.67.151.239:52046
52410.64.48.54:41100
52510.67.189.202:43356
52610.67.151.237:36046
52710.67.153.115:49058
52810.67.143.99:45226
52910.67.181.200:46138
53010.67.185.96:49262
53110.67.168.146:34992
53210.67.139.82:58158
53310.64.48.54:41118
53410.67.163.97:60844
53510.67.150.234:47226
53610.67.150.242:58332
53710.67.189.158:51304
53810.67.154.51:33708
53910.67.137.242:35266
54010.67.189.158:51314
54110.67.158.59:34156
54210.67.181.115:52702
54310.67.148.86:35300
54410.67.159.206:45132
54510.67.162.47:49610
54610.67.148.76:47172
54710.67.153.29:57150
54810.194.179.164:54636
54910.67.139.41:38390
55010.67.134.106:35854
55110.67.145.75:50614
55210.67.181.179:57612
55310.67.138.9:51008
55410.67.168.111:38606
55510.67.185.211:42936
55610.67.175.86:49336
55710.67.132.176:49754
55810.67.185.96:59634
55910.67.180.129:51140
56010.64.48.192:37526
56110.67.137.19:45302
56210.67.147.59:59708
56310.67.156.167:50336
56410.67.168.37:40572
56510.67.185.206:60542
56610.67.134.123:37484
56710.67.132.174:58430
56810.67.137.253:42370
56910.67.180.61:50870
57010.67.140.207:55674
57110.67.133.180:46216
57210.67.156.60:52502
57310.67.138.167:42770
57410.67.152.36:45116
57510.67.161.34:39694
57610.67.176.184:44904
57710.67.184.155:42456
57810.64.48.14:46772
57910.67.181.142:46410
58010.67.170.169:35216
58110.67.153.73:41204
58210.67.168.160:38488
58310.67.134.106:41554
58410.67.185.87:50236
58510.67.146.30:39464
58610.67.148.192:36862
58710.67.144.201:45010
58810.67.177.160:41700
58910.67.139.154:37312
59010.67.136.87:34126
59110.67.134.106:41568
59210.67.184.247:48244
59310.67.148.194:45570
59410.67.159.243:39640
59510.67.146.30:35260
59610.67.186.161:33646
59710.67.151.185:42666
59810.67.140.188:49186
59910.67.163.233:40626
60010.67.178.85:39894
60110.67.162.191:48794
60210.67.149.207:51064
60310.67.161.156:50332
60410.67.183.251:35018
60510.67.176.15:35886
60610.67.156.191:51846
60710.64.48.54:51092
60810.67.140.7:33048
60910.67.181.179:34636
61010.67.191.90:54200
61110.67.166.218:57420
61210.67.144.230:46730
61310.67.183.112:60186
61410.64.48.192:50094
61510.67.139.41:60672
61610.67.136.110:57162
61710.67.140.40:40152
61810.67.161.143:50000
61910.67.185.155:39880
62010.67.136.202:44010
62110.67.143.20:41930
62210.67.167.15:55128
62310.67.167.15:55134
62410.67.150.87:52934
62510.67.183.131:50580
62610.67.172.185:54772
62710.67.166.67:37538
62810.67.170.169:36326
62910.67.168.105:33844
63010.67.184.25:44818
63110.67.183.112:60202
63210.67.190.143:46656
63310.67.182.172:36830
63410.67.148.86:57408
63510.67.148.76:49346
63610.67.190.47:57516
63710.67.128.240:41188
63810.67.187.244:36628
63910.67.135.96:35680
64010.67.169.209:38468
64110.67.182.172:36832
64210.67.140.147:58504
64310.67.163.97:39414
64410.67.131.229:38232
64510.67.171.30:51878
64610.67.157.76:55364
64710.67.141.226:58882
64810.67.181.104:34154
64910.67.153.73:40292
65010.67.158.5:50026
65110.67.151.221:58420
65210.67.162.235:39562
65310.67.150.68:60110
65410.67.158.233:39802
65510.67.162.191:47830
65610.67.162.235:39578
65710.67.147.133:58570
65810.67.191.24:51994
65910.67.161.156:55726
66010.67.130.69:54118
66110.67.182.167:38240
66210.64.48.54:33594
66310.67.170.169:43804
66410.67.166.194:50192
66510.67.190.84:48300
66610.67.181.142:33534
66710.67.128.221:34006
66810.64.48.192:58618
66910.67.133.174:60388
67010.67.186.37:59474
67110.67.138.134:54484
67210.67.148.86:43192
67310.67.164.168:37048
67410.67.153.158:39584
67510.67.157.76:44138
67610.67.149.194:40528
67710.67.145.231:50572
67810.67.163.188:37568
67910.67.156.54:38846
68010.67.130.213:41600
68110.67.150.240:44830
68210.67.135.21:46688
68310.67.160.91:38650
68410.67.149.194:47544
68510.67.131.46:41842
68610.67.179.104:56264
68710.67.148.220:34446
68810.67.143.49:51250
68910.67.139.161:52790
69010.67.138.131:55966
69110.67.145.75:57678
69210.67.171.223:57106
69310.67.165.116:47760
69410.67.183.47:56878
69510.67.176.91:39972
69610.67.139.5:49052
69710.67.180.24:33538
69810.67.145.75:57690
69910.67.130.254:33778
70010.67.186.213:58730
70110.67.139.249:42592
70210.67.147.29:43446
70310.67.133.249:45084
70410.67.163.209:57722
70510.67.130.30:58196
70610.67.130.220:34028
70710.67.139.249:42596
70810.67.133.180:39226
70910.194.132.54:32890
71010.67.180.50:38092
71110.67.160.189:58204
71210.67.161.232:48360
71310.67.165.116:35590
71410.67.129.5:44330
71510.67.154.112:55324
71610.67.163.183:52010
71710.67.140.87:37678
71810.67.150.92:38418
71910.194.169.155:54058
72010.67.150.72:37274
72110.67.138.200:43878
72210.67.146.181:50612
72310.67.136.229:56738
72410.67.172.139:51762
72510.67.148.192:43704
72610.67.134.181:40998
72710.67.179.208:43582
72810.67.150.75:34542
72910.67.157.40:60962
73010.67.132.245:34946
73110.67.140.40:53552
73210.67.130.254:46832
73310.67.132.245:34954
73410.67.134.106:45574
73510.67.141.42:41818
73610.67.154.18:59264
73710.67.150.51:39926
73810.67.149.255:58004
73910.67.163.75:45360
74010.67.166.150:50060
74110.67.149.207:36352
74210.67.133.157:44258
74310.67.188.185:45056
74410.67.137.222:58554
74510.67.136.14:51288
74610.67.143.99:52674
74710.67.186.161:59464
74810.67.180.3:42026
74910.67.149.207:36356
75010.67.177.214:47278
75110.67.186.105:44052
75210.67.148.26:60800
75310.67.190.14:39428
75410.67.149.130:46340
75510.67.150.242:33704
75610.67.152.44:53384
75710.67.148.26:60806
75810.67.163.188:54424
75910.67.172.30:44684
76010.67.187.201:40234
76110.67.175.109:51902
76210.67.159.23:58302
76310.67.162.240:40180
76410.67.181.115:41728
76510.67.153.139:60552
76610.67.154.9:58852
76710.67.152.212:38490
76810.67.178.142:55222
76910.67.145.104:59070
77010.67.162.235:43424
77110.67.140.77:52668
77210.67.157.96:46476
77310.67.183.187:40322
77410.67.133.223:40840
77510.67.133.223:40848
77610.67.177.214:54468
77710.67.154.18:45224
77810.67.181.142:36596
77910.67.138.205:45204
78010.67.138.241:48126
78110.67.176.184:39920
78210.67.185.122:38906
78310.67.143.2:51202
78410.67.132.176:54904
78510.67.164.73:32796
78610.67.188.111:46876
78710.67.156.223:41368
78810.67.140.147:37198
78910.67.161.232:44104
79010.67.161.156:45698
79110.67.191.235:59722
79210.67.161.34:55494
79310.67.161.156:45708
79410.67.152.36:52386
79510.67.191.126:37256
79610.67.185.187:33902
79710.67.136.96:35810
79810.67.153.210:59046
79910.67.161.156:45712
80010.67.134.64:36514
80110.67.154.123:32808
80210.67.160.38:35748
80310.67.131.235:33528
80410.67.165.115:48922
80510.67.148.180:37040
80610.67.177.134:37954
80710.67.153.73:38390
80810.67.168.105:51118
80910.67.189.197:45434
81010.67.140.77:52678
81110.67.151.237:36660
81210.67.143.20:34324
81310.67.188.63:56456
81410.67.159.70:56004
81510.67.134.181:36474
81610.67.159.70:56012
81710.67.191.121:49150
81810.67.190.140:53398
81910.67.143.2:51218
82010.67.150.129:35380
82110.67.151.174:56900
82210.67.175.16:35160
82310.67.134.166:41662
82410.67.137.178:35136
82510.67.185.206:57468
82610.67.156.60:47136
82710.67.176.165:48196
82810.67.181.142:36600
82910.67.161.34:55496
83010.67.168.105:51134
83110.67.148.180:37052
83210.67.142.110:42346
83310.67.172.185:52426
83410.67.159.70:56022
83510.67.148.50:54282
83610.67.156.60:47146
83710.67.172.185:52442
83810.67.128.207:43864
83910.67.187.244:54976
84010.67.156.60:47156
84110.67.179.223:42128
84210.67.157.98:58778
84310.67.136.54:40754
84410.67.164.36:38072
84510.67.170.142:44552
84610.67.157.96:46492
84710.67.181.226:49264
84810.67.153.73:38404
84910.67.157.20:42946
85010.67.149.83:52180
85110.67.145.75:50456
85210.67.132.32:60152
85310.67.183.187:40324
85410.67.191.126:37262
85510.67.191.126:37266
85610.67.150.129:35386
85710.67.144.124:37858
85810.67.138.1:36346
85910.67.151.237:36674
86010.67.188.111:46886
86110.67.175.172:49036
86210.67.128.207:43876
86310.67.191.121:49166
86410.67.162.26:50634
86510.67.160.91:40804
86610.67.184.25:54920
86710.67.156.223:41376
86810.67.175.16:35170
86910.67.174.25:47844
87010.67.132.174:37092
87110.67.169.126:54034
87210.67.151.174:56904
87310.67.176.184:39934
87410.67.129.149:45774
87510.67.166.226:38530
87610.67.133.180:35598
87710.67.189.197:45440
87810.67.176.165:48200
87910.67.152.219:39702
88010.67.148.233:52072
88110.67.169.126:54048
88210.67.189.161:36786
88310.67.171.30:36756
88410.67.138.205:45214
88510.67.133.237:41360
88610.67.148.50:54288
88710.67.157.96:44294
88810.67.179.223:42144
88910.67.162.235:43436
89010.67.163.180:58462
89110.67.132.176:54912
89210.67.175.43:37462
89310.67.128.253:42060
89410.67.154.18:45228
89510.67.132.32:60154
89610.67.190.78:36602
89710.67.165.115:48934
89810.67.170.178:38570
89910.67.160.38:35750
90010.67.133.180:35604
90110.67.172.6:36058
90210.67.173.201:53340
90310.67.157.40:49902
90410.67.168.105:51136
90510.67.134.64:45224
90610.67.175.43:37470
90710.67.166.226:38546
90810.67.148.110:38364
90910.67.131.46:42436
91010.67.190.140:53408
91110.67.132.32:60160
91210.67.137.102:33896
91310.67.175.172:49050
91410.67.174.132:41538
91510.67.175.43:37472
91610.67.156.189:52990
91710.67.180.179:51712
91810.67.132.32:60162
91910.67.139.143:48898
92010.67.148.110:38372
92110.67.162.26:50644
92210.67.178.149:51042
92310.67.188.63:56470
92410.67.140.207:51108
92510.67.144.253:56904
92610.67.179.223:42160
92710.67.183.100:33152
92810.67.190.88:41310
92910.67.189.197:45456
93010.67.154.123:35790
93110.67.136.96:35820
93210.67.145.208:33570
93310.67.136.96:35828
93410.67.168.3:52524
93510.67.154.123:35796
93610.67.137.175:54870
93710.67.143.87:41048
93810.67.166.226:38556
93910.67.156.189:53002
94010.67.145.61:59106
94110.67.190.140:53424
94210.67.149.255:33908
94310.67.158.66:47280
94410.67.176.165:48212
94510.67.175.57:44922
94610.67.148.233:52078
94710.67.187.68:42846
94810.67.146.89:52138
94910.67.128.253:42068
95010.67.143.87:41064
95110.67.152.191:34528
95210.67.175.16:35182
95310.67.148.180:37058
95410.67.138.205:45226
95510.67.133.180:35618
95610.67.135.96:34054
95710.67.169.254:36860
95810.67.134.166:41676
95910.67.176.184:39950
96010.67.153.158:51888
96110.67.130.129:55498
96210.67.161.34:55508
96310.67.154.18:45240
96410.67.153.118:50030
96510.67.142.110:42350
96610.67.129.137:45336
96710.67.138.1:36354
96810.67.188.63:56472
96910.67.162.170:58574
97010.67.178.136:35732
97110.67.176.165:48228
97210.67.172.6:36072
97310.67.135.96:34066
97410.67.133.180:35620
97510.67.146.89:52142
97610.67.162.223:34916
97710.67.145.43:54712
97810.67.149.180:43566
97910.67.145.208:33572
98010.67.134.64:45240
98110.67.138.205:45232
98210.67.154.18:45246
98310.67.145.61:59120
98410.67.156.60:40344
98510.67.137.102:33898
98610.67.152.136:55910
98710.67.145.75:50462
98810.67.176.42:43116
98910.67.181.131:44784
99010.67.183.131:33724
99110.67.148.50:54302
99210.67.188.63:56484
99310.67.139.206:55150
99410.67.160.91:58592
99510.67.180.61:37858
99610.67.130.18:47120
99710.67.154.123:35802
99810.67.146.89:52146
99910.67.143.2:39682
100010.67.176.184:39952
100110.67.130.12:42656
100210.67.183.131:33738
100310.67.190.140:53440
100410.67.183.187:40328
100510.67.147.59:55908
100610.67.183.187:40338
100710.67.147.150:50854
100810.67.190.23:33326
100910.67.147.1:33962
101010.67.165.75:60218
101110.67.134.2:47898
101210.67.142.234:53772
101310.67.130.69:55680
101410.67.154.39:58266
101510.67.170.146:45952
101610.67.161.77:46716
101710.67.143.84:49764
101810.67.181.118:39808
101910.67.146.174:32994
102010.67.137.19:48652
102110.67.178.149:56352
102210.67.146.189:42308
102310.67.134.64:45252
102410.67.131.226:57020
102510.67.133.56:34780
102610.67.180.221:60020
102710.67.163.52:53910
102810.64.48.192:41322
102910.67.138.42:44958
103010.67.132.176:55602
103110.67.130.142:44186
103210.67.177.143:49852
103310.67.139.154:43984
103410.67.190.251:33724
103510.67.171.219:35098
103610.67.157.117:41428
103710.67.188.111:46196
103810.67.156.22:40528
103910.67.138.200:51142
104010.67.191.24:50060
104110.67.153.66:34016
104210.67.157.76:51858
104310.67.141.217:38246
104410.67.156.223:60050
104510.67.128.240:38612
104610.67.150.75:58854
104710.67.161.143:60362
104810.67.145.221:59476
104910.67.150.92:40072
105010.67.132.180:42438
105110.67.166.151:44546
105210.67.129.15:47456
105310.67.130.81:39326
105410.64.48.192:41870
105510.67.181.212:40716
105610.67.180.254:58716
105710.67.137.253:37430
105810.67.178.180:45972
105910.67.135.189:47076
106010.67.136.14:60056
106110.67.142.45:49164
106210.67.152.10:55694
106310.67.159.24:40218
106410.67.146.189:46582
106510.67.188.43:59658
106610.67.166.194:35660
106710.67.145.208:47972
106810.67.156.191:42606
106910.67.151.19:39380
107010.67.158.206:33966
107110.194.162.215:37406
107210.67.168.105:53446
107310.67.150.51:45040
107410.67.141.217:53926
107510.67.150.234:45572
107610.67.160.214:50978
107710.67.145.227:38896
107810.67.190.23:59942
107910.67.185.87:55950
108010.67.143.193:45776
108110.67.163.188:39870
108210.67.149.65:51840
108310.67.156.175:45168
108410.67.145.104:41200
108510.67.136.14:52008
108610.67.129.79:54172
108710.67.160.180:56140
108810.67.164.5:60794
108910.67.185.202:57926
109010.67.151.45:59140
109110.67.166.194:35072
109210.67.162.183:44804
109310.67.130.254:38010
109410.67.190.209:33058
109510.67.183.34:59802
109610.67.157.35:59598
109710.67.176.64:45368
109810.67.176.22:55462
109910.67.159.243:34898
110010.67.132.238:54688
110110.67.129.57:43798
110210.67.190.143:46144
110310.67.133.244:37624
110410.67.160.118:38790
110510.67.136.87:37640
110610.67.153.135:34040
110710.67.163.209:45806
110810.67.139.255:36132
110910.67.184.170:33440
111010.67.158.212:53128
111110.67.135.170:50160
111210.67.140.244:37350
111310.67.148.147:35882
111410.67.174.11:58288
111510.67.175.113:39268
111610.67.145.62:45580
111710.67.175.172:55918
111810.67.185.206:58498
111910.67.189.202:44748
112010.67.188.43:39616
112110.67.181.179:45260
112210.67.148.125:34644
112310.67.159.94:33658
112410.67.190.114:32790
112510.67.151.221:33142
112610.67.182.135:50740
112710.67.185.206:45984
112810.67.135.96:46984
112910.67.180.228:52974
113010.67.174.25:45432
113110.67.181.142:39282
113210.67.162.46:47098
113310.67.152.191:44562
113410.67.190.78:35320
113510.67.170.178:45828
113610.67.142.110:53626
113710.67.183.100:41516
113810.67.162.170:59500
113910.67.148.50:47566
114010.67.188.63:47576
114110.67.129.137:52432
114210.67.188.111:45252
114310.67.157.96:53610
114410.67.162.26:55432
114510.67.181.226:59324
114610.67.172.185:46694
114710.67.132.176:37050
114810.67.183.187:50908
114910.67.145.45:56326
115010.67.177.134:37080
115110.67.189.161:39994
115210.67.172.243:53864
115310.67.139.154:52244
115410.67.153.118:59898
115510.67.170.178:45842
115610.67.166.194:33060
115710.67.139.154:52258
115810.67.136.149:46528
115910.67.143.193:49128
116010.67.149.180:34278
116110.67.162.170:51300
116210.67.162.26:33822
116310.67.129.162:56582
116410.67.147.150:40380
116510.67.161.232:55352
116610.67.179.223:56856
116710.67.181.104:49496
116810.67.178.142:36432
116910.67.163.200:42198
117010.67.143.20:53362
117110.67.142.15:40898
117210.67.153.29:48044
117310.67.164.5:60084
117410.67.131.152:53752
117510.67.188.111:53022
117610.67.164.73:51702
117710.67.148.50:47742
117810.67.166.151:51622
117910.67.138.167:59290
118010.67.131.46:38196

Here's a dump of the pod details of each of those IP addresses, counted and sorted by number of appearances: https://phabricator.wikimedia.org/P67349

As this happened again tonight, @Mahir256 reached out to me in private and pointed out that most recent edits on Commons before the outage seemed to come from Cat-a-lot (edit: also previously mentioned at T370304#10065664). Looking at its source code, I notice that it makes edits without the maxlag parameter; it’s allowed to do that according to the API etiquette because it’s an interactive tool, but maybe it should do that use maxlag anyway, so it’ll automatically back off if replication lag increases too much? It can apparently produce substantial numbers of edits (it’s the #2 tag on Commons, only slightly behind SDC bots.)

(Note that users will definitely notice this, and it will result in some lost work because I don’t think Cat-a-lot has strong support for retrying failed edits later – as far as I can tell, it just retries up to 4 times after waiting for 300 ms each time, which is probably not enough for replication lag to significantly go down. As it happens, volunteer me has also written QuickCategories, a tool that can edit categories in batches which are saved more permanently and which automatically retries failed commands after a longer delay, and which also uses maxlag of course; maybe asking some of the Cat-a-lot power users to use that tool instead would also help. But all this assumes that Cat-a-lot and its lack of maxlag are related to the problem at all, which is pure speculation so far.)

Closer inspection of Cat-a-lot also reveals that it makes all of its API requests in parallel, rather than waiting for one to complete before starting the next one as the API etiquette suggests. This should be fixed, but it’s not easy given the code structure :/

I noticed that as well. Cat-a-lot can absolutely produce massive edit counts at massive edit rates, especially considering the high rate limits for most users and effective lack of rate limiting for autopatrolled users on Commons. This ratelimit was increased in T194864 specifically to accommodate the use of tools like Cat-a-lot and VFC, so it's absolutely something people are sensitive to. As @Lucas_Werkmeister_WMDE noticed while I'm typing this, Cat-a-lot is old and isn't especially easy to work on, especially since it hasn't had an active maintainer since 2019. It also doesn't handle failures well at all, but I'd say having to batch another 200 files is less annoying than bringing down the cluster, so adding maxlag at some reasonable value might be a good idea while other causes of database load are investigated.

In a hacky way I made it wait a 1 second before every edit and it seems to work: https://commons.wikimedia.org/w/index.php?title=MediaWiki:Gadget-Cat-a-lot.js&diff=prev&oldid=912122056 It just needs get client caches purged and propagated.

Since cat-a-lot doesn't make edit sequentially and just unleashes all of them at once (e.g. 200 edits at once), it is a violation of API etiquette and very likely the reason behind all of these issues we have had in the past couple of months. The emergency action to make edits serial should stay for now and we continue monitoring to see if it reduces the issue or something else is also a contributing factor here.

very likely the reason behind all of these issues we have had in the past couple of months.

With respect, Cat-a-lot has been functioning the same way for a very long time, and usage patterns have not recently changed. It's not the root cause here if there have only been problems for the past couple of months.

quarry-85603-untitled-run894445.png (1×2 px, 154 KB)
(data from https://quarry.wmcloud.org/query/85603 )

I guess there's no way to correlate the queries we've captured to Cat-a-lot right? It just makes generic writes?

Random note: The gadget currently waits 1 second before next edit, I think it's fine to lower it to 500ms.

very likely the reason behind all of these issues we have had in the past couple of months.

With respect, Cat-a-lot has been functioning the same way for a very long time, and usage patterns have not recently changed. It's not the root cause here if there have only been problems for the past couple of months.

These are not mutually exclusive. A bad pattern can feed into another change and cause outages. None can do harm on their own but it doesn't mean they shouldn't be fixed. An example that comes to mind is that due to some bug, update.php was being run on production every night for years, that didn't cause harm until due to other issues a run dropped one of the most important tables we have in production. It doesn't mean running update.php in production was fine because time-wise it doesn't correlate (not to mention cat-a-lot is violating API etiquette and should be fixed regardless). Other reasons that can trigger this to happen is that every edit might get more expensive (new features, T372718: Translate extension should not automatically switch to querying master in all non-GET requests, etc.), it can be that baseline of edits has increased, it can be that people use much larger batches instead (which your graph can't show) or usage pattern has changed, it can be that newer mariadb is less tolerant of large spikes (as it's not designed to), it might be that due to other changes, these slow downs cascade further, and so on and so on. In an environment as complex as ours, we can never be sure and that's why there is an etiquette.

Noting that the rate limit for autopatrolled or other trusted users is way too high, 10500 edits in three minutes translates to roughly 60 edits a second (at the most gentle version that the user doesn't do spikes of 10K edits in second, wait three minutes and then next batch). Our recommendation for bots in Wikidata is one edit per second. The ratelimit should be cut to half at least.

I guess there's no way to correlate the queries we've captured to Cat-a-lot right? It just makes generic writes?

The pressure it puts is the general writes, that's true but we can measure edits done by the gadget through change_tags.

very likely the reason behind all of these issues we have had in the past couple of months.

With respect, Cat-a-lot has been functioning the same way for a very long time, and usage patterns have not recently changed. It's not the root cause here if there have only been problems for the past couple of months.

These are not mutually exclusive.

Yeah in order to scale with the continuing complexity, MediaWiki has made a lot of optimisations in the past years. I'm not at all surprised that something like this could happen.

And let's not forget that up to a few weeks ago, Cat-a-Lot for search results was broken for many many months (ever since the switch to <figure> ????), that is a very long timeframe for 'MediaWiki' to change in a way that nobody was noticing.

Additionally, it only takes one user becoming 'very active' and adopting a process where they use the '500 per page' option, where previous active users were using the 50 items per page view, and you might suddenly tip the bucket.

Browsers do limit the amount of concurrent connections per domain to about 6, so it's not as if all these 500 will come in at once (I think, we might want to double check http2 multiplexing on this detail?). There will simply be a continuous 6 parallel streams of connections at full speed, all leading to deferred async load that needs to be handled at some time. When there is no backing off, and especially when DB rows are 'close' together (not unlikely with categories and a single user doing the actions), I can easily see how that would lead to lock contention. It used to be that every connection would not return until almost all the work was done, so there used to be a bit of a natural brake on streams like this, but nowadays there is a lot more work that happens after the request completes (for very good reasons btw).

So many elements come together in scenarios like this.

Random note: The gadget currently waits 1 second before next edit, I think it's fine to lower it to 500ms.

Somewhat tangential, but enWS often needs to delete up to 1k+ pages in one go. To deal with this we use a gadget that chains $.Deferred() ($.Deferred() instead of Promise so one failure doesn't break the chain) to sequentially delete via the API. This approach should be applicable to Cat-A-Lot too.

PS. The MassDelete Gadget would be a lot less hacky if the Action API supported deleting more than one page at a time, and supported supplying the pages via a generator. Less wacky code to manage client side and MediaWiki could be in complete control of how fast to execute the deletes and with how much parallelism. That probably can't easily be applied to Cat-A-Lot, but for other use cases like deletes it would be a lot cleaner all around.

With a lot of encouragement I was able to get a long-term graph of that row locks metric in s4:

image.png (1×1 px, 228 KB)

Note that short bursts are likely missing from this graph, or overly-smoothed, ofc.

DB rows are 'close' together (not unlikely with categories and a single user doing the actions),

It's actually the same rows being updated and causing contention: it's updating user_editcount for the same user and it's updating category membership numbers for the same categories so it's quite likely they would deadlock or cause contention (as most locks in databases are row-level). These two were the ones we saw the most contentions.

PS. The MassDelete Gadget would be a lot less hacky if the Action API supported deleting more than one page at a time, and supported supplying the pages via a generator. Less wacky code to manage client side and MediaWiki could be in complete control of how fast to execute the deletes and with how much parallelism. That probably can't easily be applied to Cat-A-Lot, but for other use cases like deletes it would be a lot cleaner all around.

Delete is much much more expensive than edits (at least currently due to T20493) it has to remove all the rows from revision table and insert them in archive table, we will eventually fix this and it would be much easier to allow batching and so on but for now we even had to move in the opposite direction (Nuke extension switched to actually queue jobs that would delete each page).

Delete is much much more expensive than edits (at least currently due to T20493) it has to remove all the rows from revision table and insert them in archive table, we will eventually fix this and it would be much easier to allow batching and so on but for now we even had to move in the opposite direction (Nuke extension switched to actually queue jobs that would delete each page).

Have we ever considered a mechanism for gadgets/userscripts/other JS to queue some types of jobs?

For the record, we reduced the rate limit for trusted users to 1500 edits every three minutes which is still quite high but better than 10500 every three minutes: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1063837

Delete is much much more expensive than edits (at least currently due to T20493) it has to remove all the rows from revision table and insert them in archive table, we will eventually fix this and it would be much easier to allow batching and so on but for now we even had to move in the opposite direction (Nuke extension switched to actually queue jobs that would delete each page).

Have we ever considered a mechanism for gadgets/userscripts/other JS to queue some types of jobs?

That's a great idea, at least we should be able to accommodate this for admins for example. Someone needs to just build it :)

It's actually the same rows being updated and causing contention: it's updating user_editcount for the same user and it's updating category membership numbers for the same categories

I wonder if there should be an edit API flag that disables updating aggregate stats like those, and instead queues a job to recalculate them. Recalculating something like edit counts isn't cheap, but probably still cheaper than making a thousand +1 writes.

The more efficient but much more complex approach would be to handle them like site stats: instead of writing the field immediately, storing deltas in a dedicated table and batch processing them.

Have we ever considered a mechanism for gadgets/userscripts/other JS to queue some types of jobs?

That would mean a significant amount of time passing between the user or client-side logic taking action and the action actually happening, which would make it very hard to determine if the action is still valid (what if someone else edited the page in the meantime?), and it would be hard to tell the user when the edit is discarded.

What would probably work better is queueing the editcount etc. updates specifically. These are already deferreds, and there is already a mechanism for converting deferreds to jobs, so that should be relatively straightforward, other than when to trigger it.

For the record, we reduced the rate limit for trusted users to 1500 edits every three minutes which is still quite high but better than 10500 every three minutes: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1063837

That doesn't affect admins, right? How much is it going to help in practice, given that frequency of editing tends to correlated with privileges?

That doesn't affect admins, right? How much is it going to help in practice, given that frequency of editing tends to correlated with privileges?

I think just search for noratelimit at https://commons.wikimedia.org/wiki/Special:ListGroupRights
it has 6 groups.

@Marostegui Given that the issue has been identified (and only trusted users have/had the higher rate limits), can this be made public again?

@Marostegui Given that the issue has been identified (and only trusted users have/had the higher rate limits), can this be made public again?

I was planning to do so by the end of the week if we don't have more incidents.

CDanis triaged this task as High priority.Aug 21 2024, 6:16 PM

I am going to close this as fixed for now. We've not had any issues since Sunday. Thanks a lot to everyone who's helped to get fixed, it's been a difficult one. I've asked @Aklapper to make this task public as I don't have rights for it.

Aklapper changed the visibility from "Custom Policy" to "Public (No Login Required)".Aug 22 2024, 8:32 AM

In a hacky way I made it wait a 1 second before every edit and it seems to work: https://commons.wikimedia.org/w/index.php?title=MediaWiki:Gadget-Cat-a-lot.js&diff=prev&oldid=912122056 It just needs get client caches purged and propagated.

This change would need to generate a warning that the action will take hours if there are thousends of files in the category. Or am I missing something?

With respect, Cat-a-lot has been functioning the same way for a very long time, and usage patterns have not recently changed. It's not the root cause here if there have only been problems for the past couple of months.

I recall years ago that cat-a-lot, while not taking down the site, still had enough of an impact that you could see when someone was using it on site performance graphs. I'm honestly not that surprised it eventually caused a real problem.

Noting that the rate limit for autopatrolled or other trusted users is way too high, 10500 edits in three minutes translates to roughly 60 edits a second (at the most gentle version that the user doesn't do spikes of 10K edits in second, wait three minutes and then next batch). Our recommendation for bots in Wikidata is one edit per second. The ratelimit should be cut to half at least.

For historical context, the rate limit is high essentially because the commons community demaned a high rate limit for this specific tool.

In T370304#10073278, @TheDJ wrote:

It used to be that every connection would not return until almost all the work was done, so there used to be a bit of a natural brake on streams like this, but nowadays there is a lot more work that happens after the request completes (for very good reasons btw).

Perhaps we should disable deferred updates in API requests? At least per default, we could enable them for certain modules or for users with specific rights.

Api:Etiquette states: Making your requests in series rather than in parallel, by waiting for one request to finish before sending a new request, should result in a safe request rate. But as TheDJ points out, deferred updates lead to a situation where this is no longer true.

In T370304#10073278, @TheDJ wrote:

It used to be that every connection would not return until almost all the work was done, so there used to be a bit of a natural brake on streams like this, but nowadays there is a lot more work that happens after the request completes (for very good reasons btw).

Perhaps we should disable deferred updates in API requests? At least per default, we could enable them for certain modules or for users with specific rights.

Api:Etiquette states: Making your requests in series rather than in parallel, by waiting for one request to finish before sending a new request, should result in a safe request rate. But as TheDJ points out, deferred updates lead to a situation where this is no longer true.

That wasn't really the problem here. Yes. deferred updates will be handled on the server-side while the client has sent the next request but given that server side first needs to be done with the main request, there is enough delay to reduce lock contention time and even if there is a lock contention, it's minimal since the deferred updates form a nice queue. The way Cat-a-Lot was operating was not at all like that, it was unleashing all 500 requests at the same time and was bound only by browsers having a limit of concurrent connections but with HTTP/2 it might have been way higher. We had simultaneous flood of 500 (or more) conflicting requests in an extremely short period of time and that brought down wikis. immediate serial requests might not be very nice but there is no way that they can bring down wikis. That being said, I recommend that between each request, the tool should wait as long as max lag which is between 0.1s to 0.5s in normal times. That would automatically makes it back off during stress.

I recommend that between each request, the tool should wait as long as max lag which is between 0.1s to 0.5s in normal times. That would automatically makes it back off during stress.

Is there an easy way for Gadgets and user scripts to get at maxlag? Does mw.api client-side handle maxlag, or help the coder handle maxlag in any way?

And my point up above somewhere about moving some logic server-side (in my case deletes of multiple pages in a single request, for Cat-a-lot that'd be edits) is that on that side you can more easily serialize operations, rate limit them, have configurable max concurrency, etc.

The facilities available to Gadget and user script developers are extremely limited (we can't even use async/await yet) so brute force approaches are to be expected.

I recommend that between each request, the tool should wait as long as max lag which is between 0.1s to 0.5s in normal times. That would automatically makes it back off during stress.

Is there an easy way for Gadgets and user scripts to get at maxlag? Does mw.api client-side handle maxlag, or help the coder handle maxlag in any way?

You can get maxlag via queries like this: https://www.wikidata.org/w/api.php?action=query&format=json&titles=Main%20Page&maxlag=-1

And my point up above somewhere about moving some logic server-side (in my case deletes of multiple pages in a single request, for Cat-a-lot that'd be edits) is that on that side you can more easily serialize operations, rate limit them, have configurable max concurrency, etc.

Having an API gateway would help here (a bit) but this really the responsibility of the author of API consumer. See https://meta.wikimedia.org/wiki/API_Policy_Update_2024

The facilities available to Gadget and user script developers are extremely limited (we can't even use async/await yet) so brute force approaches are to be expected.

I honestly think we should start allowing async/await, From my understanding and https://caniuse.com/?search=await there shouldn't be any blocker for it now. Considering https://www.mediawiki.org/wiki/Compatibility. Some other frontend dev might chime in and correct me if I'm wrong.

I recommend that between each request, the tool should wait as long as max lag which is between 0.1s to 0.5s in normal times. That would automatically makes it back off during stress.

Even better, vary the wait time based on how long the previous API call took to complete. As long as calls are completing quickly, you keep firing off the next call just as quickly. When the system gets bogged down, you back off. There's a ton of variations on this, but the basic idea is widely used in congestion control algorithms in many applications.

So you have to parse it out of the returned error block? And only if it errors due to exceeding the maxlag value you provided, so you'll need an extra Action API call to get at it? I think maybe there's some scope for the mw.api client-side module to make this a little more accessible to on-wiki developers…

As a reference, I believe Pywikibot just transparently honors maxlag so most bot writers or users of the canned scripts don't even have to think about it. If mw.API did that for Gadget writers too it could help.

And my point up above somewhere about moving some logic server-side (in my case deletes of multiple pages in a single request, for Cat-a-lot that'd be edits) is that on that side you can more easily serialize operations, rate limit them, have configurable max concurrency, etc.

Having an API gateway would help here (a bit) but this really the responsibility of the author of API consumer. See https://meta.wikimedia.org/wiki/API_Policy_Update_2024

Don't get me wrong, I'm not trying to absolve Gadget authors of responsibility for using the Action API endpoints in a sane way, but cf. the massdelete Gadget I linked previously, the hoops we have to jump through to do so are rather excessive. That Gadget could have been a single Action API call (with a generator) or a small number of calls (with a list of pageids) sent in serial inside a simple foreach loop. Instead it involves deeply recursive call chains, factory functions, and jQuery $.Deferred to replace native Promises (most people can't even tell you the difference between those two). You're not going to take the Deadpool approach ("Maximum effort!") unless forced to, so maxing apilimits with 500-1000+ concurrent requests is the most natural and obvious way for Gadget writers to go.

Make it easy to do the right thing, either in the API itself or in the access library (mw.api), and people's lazyness will work in your favour. The alternative is expending the effort on running around and policing everyone that does it "wrong".

The facilities available to Gadget and user script developers are extremely limited (we can't even use async/await yet) so brute force approaches are to be expected.

I honestly think we should start allowing async/await, From my understanding and https://caniuse.com/?search=await there shouldn't be any blocker for it now. Considering https://www.mediawiki.org/wiki/Compatibility. Some other frontend dev might chime in and correct me if I'm wrong.

This would be T357197 and, more broadly, T277675. And, yes, giving the community the option to, and trusting them to, make good tradeoffs with opt-in async/await would be good. I haven't looked at Cat-a-lot's code so I don't know if it'd help any there, but I have an inherited default Gadget that does 3 Action API calls for every single page load on the project that it would simply be too painful and time-consuming to rewrite
without access to async/await (well, and T358660 would also really help that use case). Saying that the functionality the Gadget provides is only available to contributors with browsers that support async/await would be entirely reasonable when held up against reducing needless API calls by up to one order of magnitude.

Whether raising minima for core makes sense is somewhat above my pay grade, but from my perspective it would certainly seem so. It makes no sense for MediaWiki to go to such extreme lengths to preserve better backward-compatibility than standards organizations (W3C, ECMA, etc.) and browser vendors actually enable you to do. It's a naïve approach that ends up pretty much just wasting resources despite its good intentions.

But I'm a ways out in the weeds relative to the core issues in this task so I'll shut up now.

For gadgets using mw.Api() (which is hopefully most of them), the nice solution would be to add a new constructor option to pass a scheduler. Then within mw.Api, we'd just have to change mw.Api.ajax() so that it gets a promise from the scheduler and chains the API call to that, and in gadgets you only need to make some trivial change like new mw.Api() -> new mw.Api( { scheduler: mw.Api.defaultScheduler } ). The scheduler could then decide whether to sequentialize requests or limit them to a fixed number of connections or inject delay or whatever.

So you have to parse it out of the returned error block?

Yes, that's what pywikibot does too.

And only if it errors due to exceeding the maxlag value you provided, so you'll need an extra Action API call to get at it?

Yes, but you can keep the same value during the gadget run. So you don't need to request it before every API request, specially for gadgets. It's also quite cheap to call.

I think maybe there's some scope for the mw.api client-side module to make this a little more accessible to on-wiki developers…

Indeed but e.g. cat-a-lot doesn't wait for the response so any action we do here won't make an impact there. The problem is how javascript handles asynchronous calls vs python which is much more synchronous when it comes to API calls. await/async helps a lot here but first we need to get it out of the door.

https://www.mediawiki.org/wiki/Manual:Maxlag_parameter says that you can add e.g. &maxlag=5 to all of your requests, and retry if they fail with the maxlag error code. You don't need a separate request to fetch the current maxlag value. And you can read the suggested delay to wait before retrying from the Retry-After HTTP response header.

We could add something making use of that to mw.Api. It could be similar to how postWithToken() already works (that method implements retrying on specific failures).

https://www.mediawiki.org/wiki/Manual:Maxlag_parameter says that you can add e.g. &maxlag=5 to all of your requests, and retry if they fail with the maxlag error code. You don't need a separate request to fetch the current maxlag value. And you can read the suggested delay to wait before retrying from the Retry-After HTTP response header.

We could add something making use of that to mw.Api. It could be similar to how postWithToken() already works (that method implements retrying on specific failures).

That would be nice. I'd need to be pretty hard up before I implemented retrying and pulling intervals out of HTTP headers in my Gadget code (the amount of added complexity relative to what's inherent for any given little Gadget does not compute). Now I've been reminded of it I'll probably go add the magic value to my API requests, but anything that errors out is just going to get thrown to the user. Retrying, backoff, etc. belongs in the access library code; not spread out in various half-baked independent implementations cut&paste-propagated across hundreds of Gadgets and user scripts.

Or put another way, when I use mw.Api, make it easy for me to be a good API citizen and make me jump through hoops to not be a good citizen. The status quo is somewhat the opposite.

https://www.mediawiki.org/wiki/Manual:Maxlag_parameter says that you can add e.g. &maxlag=5 to all of your requests, and retry if they fail with the maxlag error code. You don't need a separate request to fetch the current maxlag value. And you can read the suggested delay to wait before retrying from the Retry-After HTTP response header.

That's actually IMHO outdated and needs to be revisited. For example, in Wikidata we ended up in a situation that bots edited so fast that maxlag would go up to above 5s. Then all bots stopped allowing for slow recovery, then once it got to below five seconds, the flood gates opened again and on and on. This built a visible cyclical edit pattern. The idea put forward back then was to build a metric using PID controllers, advertise that value as Retry-After value in all requests (the specs allow it for 200 responses too, I can't find it in RFC 7231 though, I will check but I'm sure we checked the specs) and force bots to respect it in every request. We never got to implement it though.

… the specs allow [Retry-After] for 200 responses too, I can't find it in RFC 7231 though, I will check but I'm sure we checked the specs …

HTTP doesn't generally limit what header fields you can send with a given status code. Both status codes and header fields are extensible, and implementations are required to ignore and pass on any field they do not understand. So at worst the semantics are undefined outside the context of the MediaWiki Action API and its clients. And if you absolutely need to, inventing your own custom header field for this would be entirely ok from a standards perspective (implementation and maintenance perspectives might differ).

PS. RFC 9110 is current, but its description of Retry-After is identical to the one in RFC 7231.

About cat-a-lot. I made a version which does editing in series instead of parallel and one thing what I noticed that it takes more time to write them in series than using 1s sleep between new edits. Even without any templates in page the edit took almost 0.5s.

Do you have any ideas what is causing that? (ie. is it just that system is rendering the page and could it be done so that it would just flag it dirty in cache without rendering it?)

For example

Removing category from file page without any other wikicode in wikitext than categories took 450ms

Removing categories from file pages with image information in wikitext took 1000 -1660ms:

Duration: 1040ms

Duration: 1661 ms

Duration: 1088 ms

Duration: 1341 ms

HTTP doesn't generally limit what header fields you can send with a given status code. Both status codes and header fields are extensible, and implementations are required to ignore and pass on any field they do not understand. So at worst the semantics are undefined outside the context of the MediaWiki Action API and its clients. And if you absolutely need to, inventing your own custom header field for this would be entirely ok from a standards perspective (implementation and maintenance perspectives might differ).

HTTP doesn't, but browsers do (with CORS). There are ways to fix that again, i'm just pointing out that adding headers requires more thought, then just slapping them in the response.

I made a new ticket for fixing cat-a-lot.js (T375355) and there is now parallel edit version which limits the editing to 5 parallel edits and 1 if the maxlag is high.