Page MenuHomePhabricator

Emergency database primary master failover on s3
Closed, ResolvedPublic

Description

Due to hardware malfunction on the primary s3 database master T219115 we have to do a database master failover.

We would need a 30 minutes read-only window for all those wikis:

1root@cumin1001:/home/marostegui/git/mediawiki-config/dblists# cat s3.dblist
2aawiki
3aawikibooks
4aawiktionary
5abwiki
6abwiktionary
7acewiki
8advisorswiki
9advisorywiki
10adywiki
11afwiki
12afwikibooks
13afwikiquote
14afwiktionary
15akwiki
16akwikibooks
17akwiktionary
18alswiki
19amwiki
20amwikimedia
21amwikiquote
22amwiktionary
23angwiki
24angwikibooks
25angwikiquote
26angwikisource
27angwiktionary
28anwiki
29anwiktionary
30arbcom_cswiki
31arbcom_dewiki
32arbcom_enwiki
33arbcom_fiwiki
34arbcom_nlwiki
35arcwiki
36arwikibooks
37arwikimedia
38arwikinews
39arwikiquote
40arwikisource
41arwikiversity
42arwiktionary
43arzwiki
44astwiki
45astwikibooks
46astwikiquote
47astwiktionary
48aswiki
49aswikibooks
50aswikisource
51aswiktionary
52atjwiki
53auditcomwiki
54avwiki
55avwiktionary
56aywiki
57aywikibooks
58aywiktionary
59azbwiki
60azwiki
61azwikibooks
62azwikiquote
63azwikisource
64azwiktionary
65barwiki
66bat_smgwiki
67bawiki
68bawikibooks
69bclwiki
70bdwikimedia
71be_x_oldwiki
72betawikiversity
73bewiki
74bewikibooks
75bewikimedia
76bewikiquote
77bewikisource
78bewiktionary
79bgwikibooks
80bgwikinews
81bgwikiquote
82bgwikisource
83bhwiki
84bhwiktionary
85biwiki
86biwikibooks
87biwiktionary
88bjnwiki
89bmwiki
90bmwikibooks
91bmwikiquote
92bmwiktionary
93bnwiki
94bnwikibooks
95bnwikisource
96bnwiktionary
97bnwikivoyage
98boardgovcomwiki
99boardwiki
100bowiki
101bowikibooks
102bowiktionary
103bpywiki
104brwiki
105brwikimedia
106brwikiquote
107brwikisource
108brwiktionary
109bswiki
110bswikibooks
111bswikinews
112bswikiquote
113bswikisource
114bswiktionary
115bugwiki
116bxrwiki
117cawikibooks
118cawikimedia
119cawikinews
120cawikiquote
121cawikisource
122cawiktionary
123cbk_zamwiki
124cdowiki
125cewiki
126chairwiki
127chapcomwiki
128checkuserwiki
129chowiki
130chrwiki
131chrwiktionary
132chwiki
133chwikibooks
134chwiktionary
135chywiki
136ckbwiki
137cnwikimedia
138collabwiki
139cowiki
140cowikibooks
141cowikimedia
142cowikiquote
143cowiktionary
144crhwiki
145crwiki
146crwikiquote
147crwiktionary
148csbwiki
149csbwiktionary
150cswikibooks
151cswikinews
152cswikiquote
153cswikisource
154cswikiversity
155cswiktionary
156cuwiki
157cvwiki
158cvwikibooks
159cywiki
160cywikibooks
161cywikiquote
162cywikisource
163cywiktionary
164dawiki
165dawikibooks
166dawikiquote
167dawikisource
168dawiktionary
169dewikibooks
170dewikinews
171dewikiquote
172dewikisource
173dewikiversity
174dewikivoyage
175dewiktionary
176dinwiki
177diqwiki
178dkwikimedia
179donatewiki
180dsbwiki
181dtywiki
182dvwiki
183dvwiktionary
184dzwiki
185dzwiktionary
186ecwikimedia
187eewiki
188electcomwiki
189elwiki
190elwikibooks
191elwikinews
192elwikiquote
193elwikisource
194elwikiversity
195elwikivoyage
196elwiktionary
197emlwiki
198enwikibooks
199enwikinews
200enwikisource
201enwikiversity
202eowikibooks
203eowikinews
204eowikiquote
205eowikisource
206eowiktionary
207eswikibooks
208eswikinews
209eswikiquote
210eswikisource
211eswikiversity
212eswikivoyage
213eswiktionary
214etwiki
215etwikibooks
216etwikimedia
217etwikiquote
218etwikisource
219etwiktionary
220euwiki
221euwikibooks
222euwikiquote
223euwikisource
224euwiktionary
225execwiki
226extwiki
227fawikibooks
228fawikinews
229fawikiquote
230fawikisource
231fawikivoyage
232fawiktionary
233fdcwiki
234ffwiki
235fiu_vrowiki
236fiwikibooks
237fiwikimedia
238fiwikinews
239fiwikiquote
240fiwikisource
241fiwikiversity
242fiwikivoyage
243fiwiktionary
244fixcopyrightwiki
245fjwiki
246fjwiktionary
247foundationwiki
248fowiki
249fowikisource
250fowiktionary
251frpwiki
252frrwiki
253frwikibooks
254frwikinews
255frwikiquote
256frwikisource
257frwikiversity
258frwikivoyage
259furwiki
260fywiki
261fywikibooks
262fywiktionary
263gagwiki
264ganwiki
265gawiki
266gawikibooks
267gawikiquote
268gawiktionary
269gdwiki
270gdwiktionary
271glkwiki
272glwiki
273glwikibooks
274glwikiquote
275glwikisource
276glwiktionary
277gnwiki
278gnwikibooks
279gnwiktionary
280gomwiki
281gorwiki
282gotwiki
283gotwikibooks
284grantswiki
285guwiki
286guwikibooks
287guwikiquote
288guwikisource
289guwiktionary
290gvwiki
291gvwiktionary
292hakwiki
293hawiki
294hawiktionary
295hawwiki
296hewikibooks
297hewikinews
298hewikiquote
299hewikisource
300hewikivoyage
301hewiktionary
302hifwiki
303hifwiktionary
304hiwiki
305hiwikibooks
306hiwikimedia
307hiwikiquote
308hiwikiversity
309hiwikivoyage
310hiwiktionary
311howiki
312hrwiki
313hrwikibooks
314hrwikiquote
315hrwikisource
316hrwiktionary
317hsbwiki
318hsbwiktionary
319htwiki
320htwikisource
321huwikibooks
322huwikinews
323huwikiquote
324huwikisource
325huwiktionary
326hywiki
327hywikibooks
328hywikiquote
329hywikisource
330hywiktionary
331hzwiki
332iawiki
333iawikibooks
334iawiktionary
335idwikibooks
336idwikimedia
337id_internalwikimedia
338idwikiquote
339idwikisource
340idwiktionary
341iegcomwiki
342iewiki
343iewikibooks
344iewiktionary
345igwiki
346iiwiki
347ikwiki
348ikwiktionary
349ilowiki
350ilwikimedia
351incubatorwiki
352inhwiki
353internalwiki
354iowiki
355iowiktionary
356iswiki
357iswikibooks
358iswikiquote
359iswikisource
360iswiktionary
361itwikibooks
362itwikinews
363itwikiquote
364itwikisource
365itwikiversity
366itwikivoyage
367itwiktionary
368iuwiki
369iuwiktionary
370jamwiki
371jawikibooks
372jawikinews
373jawikiquote
374jawikisource
375jawikiversity
376jawiktionary
377jbowiki
378jbowiktionary
379jvwiki
380jvwiktionary
381kaawiki
382kabwiki
383kawiki
384kawikibooks
385kawikiquote
386kawiktionary
387kbdwiki
388kbpwiki
389kgwiki
390kiwiki
391kjwiki
392kkwiki
393kkwikibooks
394kkwikiquote
395kkwiktionary
396klwiki
397klwiktionary
398kmwiki
399kmwikibooks
400kmwiktionary
401knwiki
402knwikibooks
403knwikiquote
404knwikisource
405knwiktionary
406koiwiki
407kowikibooks
408kowikinews
409kowikiquote
410kowikisource
411kowikiversity
412kowiktionary
413krcwiki
414krwiki
415krwikiquote
416kshwiki
417kswiki
418kswikibooks
419kswikiquote
420kswiktionary
421kuwiki
422kuwikibooks
423kuwikiquote
424kuwiktionary
425kvwiki
426kwwiki
427kwwikiquote
428kwwiktionary
429kywiki
430kywikibooks
431kywikiquote
432kywiktionary
433ladwiki
434lawiki
435lawikibooks
436lawikiquote
437lawikisource
438lawiktionary
439lbewiki
440lbwiki
441lbwikibooks
442lbwikiquote
443lbwiktionary
444legalteamwiki
445lezwiki
446lfnwiki
447lgwiki
448lijwiki
449liwiki
450liwikibooks
451liwikinews
452liwikiquote
453liwikisource
454liwiktionary
455lmowiki
456lnwiki
457lnwikibooks
458lnwiktionary
459loginwiki
460lowiki
461lowiktionary
462lrcwiki
463ltgwiki
464ltwiki
465ltwikibooks
466ltwikiquote
467ltwikisource
468ltwiktionary
469lvwiki
470lvwikibooks
471lvwiktionary
472maiwiki
473maiwikimedia
474map_bmswiki
475mdfwiki
476mediawikiwiki
477mgwiki
478mgwikibooks
479mhrwiki
480mhwiki
481mhwiktionary
482minwiki
483miwiki
484miwikibooks
485miwiktionary
486mkwiki
487mkwikibooks
488mkwikimedia
489mkwikisource
490mkwiktionary
491mlwiki
492mlwikibooks
493mlwikiquote
494mlwikisource
495mlwiktionary
496mnwiki
497mnwikibooks
498mnwiktionary
499movementroleswiki
500mrjwiki
501mrwiki
502mrwikibooks
503mrwikiquote
504mrwikisource
505mrwiktionary
506mswiki
507mswikibooks
508mswiktionary
509mtwiki
510mtwiktionary
511muswiki
512mwlwiki
513mxwikimedia
514myvwiki
515mywiki
516mywikibooks
517mywiktionary
518mznwiki
519nahwiki
520nahwikibooks
521nahwiktionary
522napwiki
523nawiki
524nawikibooks
525nawikiquote
526nawiktionary
527nds_nlwiki
528ndswiki
529ndswikibooks
530ndswikiquote
531ndswiktionary
532newiki
533newikibooks
534newiktionary
535newwiki
536ngwiki
537nlwikibooks
538nlwikimedia
539nlwikinews
540nlwikiquote
541nlwikisource
542nlwikivoyage
543nlwiktionary
544nnwiki
545nnwikiquote
546nnwiktionary
547noboard_chapterswikimedia
548nostalgiawiki
549novwiki
550nowikibooks
551nowikimedia
552nowikinews
553nowikiquote
554nowikisource
555nowiktionary
556nrmwiki
557nsowiki
558nvwiki
559nycwikimedia
560nywiki
561nzwikimedia
562ocwiki
563ocwikibooks
564ocwiktionary
565officewiki
566olowiki
567ombudsmenwiki
568omwiki
569omwiktionary
570orwiki
571orwikisource
572orwiktionary
573oswiki
574otrs_wikiwiki
575outreachwiki
576pa_uswikimedia
577pagwiki
578pamwiki
579papwiki
580pawiki
581pawikibooks
582pawikisource
583pawiktionary
584pcdwiki
585pdcwiki
586pflwiki
587pihwiki
588piwiki
589piwiktionary
590plwikibooks
591plwikimedia
592plwikinews
593plwikiquote
594plwikisource
595plwikivoyage
596plwiktionary
597pmswiki
598pmswikisource
599pnbwiki
600pnbwiktionary
601pntwiki
602projectcomwiki
603pswiki
604pswikibooks
605pswikivoyage
606pswiktionary
607ptwikibooks
608ptwikimedia
609ptwikinews
610ptwikiquote
611ptwikisource
612ptwikiversity
613ptwikivoyage
614ptwiktionary
615punjabiwikimedia
616qualitywiki
617quwiki
618quwikibooks
619quwikiquote
620quwiktionary
621rmwiki
622rmwikibooks
623rmwiktionary
624rmywiki
625rnwiki
626rnwiktionary
627roa_rupwiki
628roa_rupwiktionary
629roa_tarawiki
630romdwikimedia
631rowikibooks
632rowikinews
633rowikiquote
634rowikisource
635rowikivoyage
636rowiktionary
637rswikimedia
638ruewiki
639ruwikibooks
640ruwikimedia
641ruwikinews
642ruwikiquote
643ruwikisource
644ruwikiversity
645ruwikivoyage
646ruwiktionary
647rwwiki
648rwwiktionary
649sahwiki
650sahwikiquote
651sahwikisource
652satwiki
653sawiki
654sawikibooks
655sawikiquote
656sawikisource
657sawiktionary
658scnwiki
659scnwiktionary
660scowiki
661scwiki
662scwiktionary
663sdwiki
664sdwikinews
665sdwiktionary
666searchcomwiki
667sewiki
668sewikibooks
669sewikimedia
670sgwiki
671sgwiktionary
672shwiktionary
673shnwiki
674simplewiki
675simplewikibooks
676simplewikiquote
677simplewiktionary
678siwiki
679siwikibooks
680siwiktionary
681skwiki
682skwikibooks
683skwikiquote
684skwikisource
685skwiktionary
686slwiki
687slwikibooks
688slwikiquote
689slwikisource
690slwikiversity
691slwiktionary
692smwiki
693smwiktionary
694snwiki
695snwiktionary
696sourceswiki
697sowiki
698sowiktionary
699spcomwiki
700specieswiki
701sqwiki
702sqwikibooks
703sqwikinews
704sqwikiquote
705sqwiktionary
706srnwiki
707srwikibooks
708srwikinews
709srwikiquote
710srwikisource
711srwiktionary
712sswiki
713sswiktionary
714stewardwiki
715stqwiki
716strategywiki
717stwiki
718stwiktionary
719suwiki
720suwikibooks
721suwikiquote
722suwiktionary
723svwikibooks
724svwikinews
725svwikiquote
726svwikisource
727svwikiversity
728svwikivoyage
729svwiktionary
730swwiki
731swwikibooks
732swwiktionary
733szlwiki
734tawiki
735tawikibooks
736tawikinews
737tawikiquote
738tawikisource
739tawiktionary
740tcywiki
741techconductwiki
742tenwiki
743test2wiki
744testwiki
745testwikidatawiki
746tetwiki
747tewiki
748tewikibooks
749tewikiquote
750tewikisource
751tewiktionary
752tgwiki
753tgwikibooks
754tgwiktionary
755thwikibooks
756thwikinews
757thwikiquote
758thwikisource
759thwiktionary
760tiwiki
761tiwiktionary
762tkwiki
763tkwikibooks
764tkwikiquote
765tkwiktionary
766tlwiki
767tlwikibooks
768tlwiktionary
769tnwiki
770tnwiktionary
771towiki
772towiktionary
773tpiwiki
774tpiwiktionary
775transitionteamwiki
776trwikibooks
777trwikimedia
778trwikinews
779trwikiquote
780trwikisource
781trwiktionary
782tswiki
783tswiktionary
784ttwiki
785ttwikibooks
786ttwikiquote
787ttwiktionary
788tumwiki
789twwiki
790twwiktionary
791tyvwiki
792tywiki
793uawikimedia
794udmwiki
795ugwiki
796ugwikibooks
797ugwikiquote
798ugwiktionary
799ukwikibooks
800ukwikinews
801ukwikiquote
802ukwikisource
803ukwikivoyage
804ukwiktionary
805urwiki
806urwikibooks
807urwikiquote
808urwiktionary
809usabilitywiki
810uzwiki
811uzwikibooks
812uzwikiquote
813uzwiktionary
814vecwiki
815vecwikisource
816vecwiktionary
817vepwiki
818vewiki
819viwikibooks
820viwikiquote
821viwikisource
822viwikivoyage
823viwiktionary
824vlswiki
825votewiki
826vowiki
827vowikibooks
828vowikiquote
829vowiktionary
830warwiki
831wawiki
832wawikibooks
833wawiktionary
834wbwikimedia
835wg_enwiki
836wikimania2005wiki
837wikimania2006wiki
838wikimania2007wiki
839wikimania2008wiki
840wikimania2009wiki
841wikimania2010wiki
842wikimania2011wiki
843wikimania2012wiki
844wikimania2013wiki
845wikimania2014wiki
846wikimania2015wiki
847wikimania2016wiki
848wikimania2017wiki
849wikimania2018wiki
850wikimaniawiki
851wikimaniateamwiki
852wowiki
853wowikiquote
854wowiktionary
855wuuwiki
856xalwiki
857xhwiki
858xhwikibooks
859xhwiktionary
860xmfwiki
861yiwiki
862yiwikisource
863yiwiktionary
864yowiki
865yowikibooks
866yowiktionary
867yuewiktionary
868zawiki
869zawikibooks
870zawikiquote
871zawiktionary
872zeawiki
873zerowiki
874zh_classicalwiki
875zh_min_nanwiki
876zh_min_nanwikibooks
877zh_min_nanwikiquote
878zh_min_nanwikisource
879zh_min_nanwiktionary
880zh_yuewiki
881zhwikibooks
882zhwikinews
883zhwikiquote
884zhwikisource
885zhwikiversity
886zhwikivoyage
887zhwiktionary
888zuwiki
889zuwikibooks
890zuwiktionary

Sorry for the short notice, but this is an emergency.

Date: Thu 11th April
Time: 05:00 AM UTC - 05:30 AM UTC (we expect not to use the full 30 minutes window)

Impact: All those wikis will go read-only. No edits will be allowed. Reads will not be impacted.

If it helps crafting the announcement, this is exactly the same operation, situation, impact and affected wikis as T213864: Emergency database primary master failover on s3 primary master

Event Timeline

Marostegui renamed this task from Emergency database primary master failover on s3 primary master to Emergency database primary master failover on s3.Apr 4 2019, 8:46 AM

Thanks very much!

@Marostegui - Is it the 10th or 11th? You've said 10th in the email but 11th on here. Linked task says Wednesday 11th as well.

It is the 11th, sorry!
I will amend my email, thanks for noticing

I've created a message and asked for translations, and will send it out on Monday, which is also when the Tech News issue including this will go out. I've also pinged @Trizek-WMF to see if we could add another banner warning people ahead of time.

(I almost missed I had been subscribed to this – too many Phab notifications sometimes. We'll normally pay attention to wikitech-l and notice if there's an upcoming read-only period, of course, but I'd recommend sending an email to us pinging us about a support request be on the safe side if it's time-sensitive.)

Thanks @Johan! Any preferred email address for next time?

Our team support email address is probably the safest way to get our attention if something is important and urgent, so it doesn't risk getting lost because one person is out of office or too swamped with other work.

Thank you, we'll do that next time too!

A MassMessage has been delivered to the wikis.

Just to confirm, Wikidata is not impacted, right?

Wikidata itself, and the top most visited wikipedias, meta, commons, centralauth, etc. will not be impacted. However, if wikidata changes something that would result on changes to the s3 local databases, those won't go through during the maintenance. What that means, if they will be retried or aborted, will depend on how Wikibase responds to the read only time on client wikis. I think Amir did monitor the effects last time (T213858) and there was not huge issues, but please ask him if he has any input. Maybe you may want to do something to prevent stale info? I don't know.

The following wikis can't be targeted by a CentralNotice banner. They have to be messaged to setup a banner if needed.

  • wikimedia
  • noboard_chapters
  • wikimania
  • arbcom
  • otrs
  • advisors
  • advisory
  • auditcom
  • board
  • boardgovcom
  • chair
  • chapcom
  • checkuser
  • collab
  • donate
  • electcom
  • exec
  • fdc
  • fixcopyright
  • foundation
  • grants
  • iegcom
  • incubator
  • internal
  • legalteam
  • login
  • movementroles
  • nostalgia
  • office
  • ombudsmen
  • outreach
  • projectcom
  • quality
  • searchcom
  • steward
  • strategy
  • techconduct
  • ten
  • test
  • test2
  • transitionteam
  • usability
  • vote

Just to confirm, Wikidata is not impacted, right?

test.wikidata is impacted.

I don't think the listed wikis need a banner.

Banner set for Wikipedias.
Other wikis will be set tomorrow.

Wikidata itself, and the top most visited wikipedias, meta, commons, centralauth, etc. will not be impacted. However, if wikidata changes something that would result on changes to the s3 local databases, those won't go through during the maintenance. What that means, if they will be retried or aborted, will depend on how Wikibase responds to the read only time on client wikis. I think Amir did monitor the effects last time (T213858) and there was not huge issues, but please ask him if he has any input. Maybe you may want to do something to prevent stale info? I don't know.

Yes, this will affect change dispatching of Wikidata. Last time it was for a few minutes but this one is for half an hour which means probably failed jobs will go to retry and pile up and bring down the whole jobqeue (probably bringing down everything with themselves including all wikis). I suggest disabling change dispatching totally for the period of downtime.

@Ladsgroup, should I display the banner on Wikidata as well, then?

@Ladsgroup, should I display the banner on Wikidata as well, then?

Editing won't be disabled on wikidata and disabling it would cause issues in other wikis (for example, in case of a page move in enwiki, that wouldn't reach wikidata). We can have a different banner asking user to not edit to reduce the load when everything is back to normal

The window is for 30 minutes, but if all goes fine we are expecting it to take 4-5 minutes tops

The window is for 30 minutes, but if all goes fine we are expecting it to take 4-5 minutes tops

Then disabling change dispatching won't make much sense. I can stay around to monitor.

More concrete numbers from the last failover of this same section: T213858#4887175
03:20 minutes of read only, we hope to be around that number if all goes as planned

I'll be at a conference tomorrow; if anything specific would require our attention, @Whatamidoing-WMF will be around late afternoon/evening UTC.

This was done successfully.

Times in UTC:

Read only starts: 05:01:05
Read only stops: 05:03:55

Total read only time: 02:50 minutes

Thanks everyone for your help!