Everything was left on the table for the moment. I sat opposite the human and tried to wake him up with a gentle voice: « Good morning little guy, I prepared some breakfast for us. » He shuffled around, groaned a bit like a cranky teenager getting out of bed.

« I feel like a brick hit my head, he complained sitting and holding his head in his hands.
– A brick. You have a headache ? That's expected from an intoxication to dragonsbane.
– Dragons-what now ?
– Dragonsbane.
– Never heard of it.
– It's the common name of a toxic, putrid-smelling substance a human wouldn't survive given enough prolonged contact, I explained while serving breakfast on the table, I was worried you wouldn't make it, but I'm very glad you did. I assume they talked about it during your training, right ?
– Never heard that name during training.
– Huh, I answer stupefied, odd they haven't told you about this.
– Do you have something for that headache though ? It's hurting badly…
– Yep ! I prepared a little concoction during your sleep. I think it should be effective for your headache. It just tastes terrible. »

The young man grabbed the bottle and drank it in one breath. I was taken aback he asked some medication from me, a wolf-dragon he barely knew he once framed as an enemy of his kingdom to be slain, and drank it. I then told him breakfast was served on the table. Since he struggled with balance, I offered him my forearm so he could lean on it, walk across the room, and we sat opposite to each other. I resumed the discussion:

« I'm surprised you just drank that potion or accepted what I prepared without asking questions. Usually, dragon slayers would either not ask me anything or tell me their have enough in their stash fearing I'll poison them or make them have a bad time.
– Logical, the human replied, that's what they always tell us. Quote, don't accept anything from a dragon, they can only do bad things to you given the torment they went through dealing with you, if they survive. They can only seek revenge after being injured, end quote.
– I suppose most, if not all of them, would react like that. Yet here we are having an open discussion around a table and some food. Considering you declared me being the terror of your kingdom and I had to be dealt with a few hours ago, it's unexpected. By the way, the medication I gave you should be kicking in now. Feeling a little better ?
– Yeah, that massive headache is slowly going away, thank you very much, he replied munching on some bread.
– You're welcome, that's good to hear, I reply nearly finishing my plate and drinking a cup of water. By the way, I'm sorry to spoil the mood. Given your current state and your wobbliness, I don't think it's a good idea to head into the wilderness now. We'll see how you fare tomorrow.
– I don't intend to leave this place anytime soon…
– Really ? I slightly tilted my head in surprise. The usual dragon slayers I've seen wanted to speedrun their way out of here so bad, I thought you would do the same. At the same time, as you said, fear of the revenge a dragon has in store. You really aren't like the others that came before you ! I'm curious to know why. »

As we finished our breakfast, I collected the plates and stacked them aside in the kitchen. Only breadcrumbs remained, a quick wipe with a wet towel and they were ready for reuse; the water bucket and cup stayed on the table. I came back and listened carefully to the pensive young adult:

« It's hard to say, it's a conflict of situations between the training given and what I'm living. They tell us many things on dragons, how they supposedly behave, how mean they are to us and the kingdom, the whole lot you've probably heard a thousand times. Here I am in front of a dragon displaying the opposite behavior of what they drilled into our heads. You could be taking advantage of my vulnerability, hurt me, what have you, even straight up kill me if you wished. Yet you're kind, you worry about my well-being, you stayed awake the whole night to make sure I'm fine, you're treating me well and you go out of your way to make sure I'm comfortable. Of course I don't want to leave ! » A few seconds later, the man continued « I feel terrible, I'm dizzy but at least my head is not hurting as much thanks to your concoction. What a misery to be thrown into against my will…
– Hey hum… I know this whole ordeal is a bit much to process for you. I'll leave you you for a few minutes and take care of some chores and write a few letters. Don't worry, I'm still around in case you need me. We can talk again later just at the entrance of the cave if you wish. The weather is lovely. »

I got off the table, grabbed the bucket and emptied it on the burning embers of the camp. The sun was providing the much needed warmth and light. I returned the bucket filled with fresh water to the table.

A kobold arrived and used the "doorbell", a forged metal ring hung on some stone and that made a surprising amount of noise for its simplicity. He handed me a few letters: one was from a kingdom at the foot of the mountain, another from a kingdom further away and a few others from other dragons. I chatted with him for a while, discussing about random things. I thanked him for his precious service and handed him a pouch of golden coins. I sat on the same log I was on during the night and read the letters.

The first one from the nearby kingdom was a weekly digest of the news. They knew I would pay them a cordial visit if something caught my eye and I usually spent a lot of money while I was there. I was on good terms with its royal family, despite keeping my distance with others. The second letter from a kingdom further away just pretty much listed what was going on in their territories and the upcoming events, nothing extraordinary. The good news is they finally have sorted their internal affairs. Hopefully, it was for the long run.

I heard the human's voice from the back: « Hey hum… dragon ? I kinda need someone to talk to… if I'm not disturbing you…
– Yeah sure, let me just put those letters away in my office and I'll be back to you ! » I replied. After a quick hop in my office, I offered the human I still don't know the name of to discuss just outside the cave. Once on the terrasse, we sat together on some wooden chairs. He was very dizzy from his intoxication and sore from the physical effort. I wondered how many days he has been huffing that putrid substance while walking up the mountain, without questioning much what's happening to him. At least, he knew he was safe with me, and that's all that mattered at the moment.

He explained me why he « hated it down there », how miserable he felt and how the king wouldn't deliver fully on his promise of higher living standards for those choosing and training to chase dragons for « the well-being of the kingdom ». Some things were definitely better, yes, but not distinctively better than your average knight. He felt understandably cheated and was seething at the whole situation. Those who failed their missions repeatedly were met with various punishments, from humiliation to getting impossible "training". The human also explained me how he received the order to walk to my mountain because « that dragon is causing us misery » and he had to « teach them a good lesson with that blade » without telling him the nature of the coating. He felt as if the king was looking for a way to get rid of him despite doing okay, the feeling was reinforced by the non-disclosure of the nature of the coating. It has been a lot to unpack for him.

The morning turned into dinner time. I made something light for my little guest. He was surprised a wolf-dragon was able to cook something that tasty, as dragons cooking human food were usually unheard of. The human ate slowly though, nothing to be alarmed of; the world was probably spinning in his head and he was just taking his time. Once he was done, the usual dance of washing dishes took place.

During the afternoon, the little guy was feeling a bit better despite having puked a few times. When the headache came back in full force like an anvil was dropped on him, I gave him another bottle. I advised him to rest for a bit and that the next day his condition would certainly improve. While he took a nap, I went back to my letters and replied to some.

This article won't go in the details of what is Mastodon, or the global network, the Fediverse, also called "the federation". There are a lot of articles and videos on the web that explain these two concepts.

This article is written around one year after the facts. Some fine details can be omitted to simplify the article, or I don't recall exactly how things went down.

Mastodon components

Let's begin with giving a bit of context on what there is in the virtual machine managing that service. There's a PostgreSQL database that stores local data from the service and remote data from other servers in the federation. Then, there is all the software stack of Mastodon, Ruby or Javascript:

• A web server to answer HTTP queries from users and servers
• A Websocket server to send posts in real time to the connected users
• A Redis server (external service) for caching and the queues of the next service
• A piece of software managing queues to run background tasks

The last component is crucial to the good working condition of the service. In fact, it sends posts to remote servers, finishes the reception of remote publications, finalizes image uploads, translates posts on demand and generates preview cards for links. In other words, this component does a lot.

The overload of the component can be felt more or less directly for the user, with, for example, file uploads taking an eternity to finalize.

The calm before the storm

Tooting was a service at a small scale (a hundred of users, I think ?), when one compares it to its current size. Each software brick was running the "default" configuration, which was sufficient back then. The storage was spacious enough to keep the remote media and the data in PostgreSQL.

Since sign-ups go through a manual validation by administrators and moderators, to get rid of the spam and the problem at the source, they have a vague idea of what could happen and if more resources should be planned. The amount of sign-ups had already made a few jumps with what was happening in an other realm of the Internet. However, nothing to write home about.

The weather is getting dark, windy and cloudy

When the purchase of Twitter was concluded, many people rushed on the servers of the global network, creating a more or less brutal load increase for everybody. Some servers arriving at capacity for technical or sizing reasons had to close open sign-ups or limit them with invitations.

To give an idea of the load increase on Tooting, there was around 20 new users per month; during the wave, about 100 users per day were creating accounts.

The new owner begins to make modifications to his newly acquired platform that eventually result in the site we know. These changes contributed to amplify the migration wave already sweeping across the Fediverse. For tooting, the interesting problems are just starting to appear.

The storm; the river wanders out of its riverbed

Initially, the consequences were an increase of volume of emails to send, volume that blew up the quota of the mail hosting service. The SMTP¹ server rejected the email and the task went back in the queue of things to do. To accommodate the volume that exploded, one request to increase the quota was sent to Infomaniak, eventually granted. While waiting for the answer, not much was to do with regards to emails: disable superfluous emails for the moderation and administration, and wait. To the pace of 100 users per day, mail inboxes were able to breathe.

In a nutshell, you register on Tooting, you could take a coffee or do your things; the email address confirmation will arrive by physical mail !

Other than emails that were having trouble to be sent, consequences of an overloaded queue. Do you remember this piece of sentence ?

The last component [of Mastodon's software stack] is crucial to the good working condition of the service

It so happens that the counter of pending tasks was beating records and was keeping on increasing. Sidekiq's default configuration (I'll explain that later !) was no longer enough. I'll enumerate a few issues encountered with a queue that wouldn't stop growing:

• Federated posts would arrive "late"
• Relaying of local publications is delayed
• Sending images takes considerably longer. Technically, the image or attachment is on the server, the task representing the processing was just at the bottom of the todo-list.
• The generation of link integration cards was delayed
• Emails that need no further explaining !

The list is not exhaustive, of course.

I could waffle on the diverse issues that were cropping up on Tooting, but we understood that the river got out of its bed and causes flooding a bit everywhere.

How does Sidekiq, the queue manager, work in a nutshell

Before we continue, it is interesting to know how the queue manager, Sidekiq is the name, works in a nutshell. When the administrator starts a Sidekiq process, the main process creates as many secondary tasks (also called threads) as configured. The main process is in charge of receiving the tasks to realize from an external service and distribute them to the threads. The threads do the work.

Once the requested action has been executed without errors, a counter is incremented and the thread does the next job or waits. In case of problems, the task is put in another queue, the "Tentative" one, and will be reattempted later. When there are too many errors for a task, it will end up in the "Dead" queue and won't be attempted again unless an administrator manually puts it back in the more active queues.

Therefore, it is in the interest of the administrator to keep a short waiting queue so the platform is reactive. Otherwise, the diverse symptoms I described above will appear.

Sidekiq is also capable of executing scheduled tasks but it doesn't matter much in this context.

Managing the load increase

The priority was to shorten the queue in Sidekiq. It could take as long as needed, but the tasks counter must go down.

Only one thing to do: increase the amount of worker threads spawned to a good handful and keep only one main process. In theory, it's a good idea, in practice, the conductor finds itself fully occupied and unable to occupy all threads. The other solution was to start multiple main Sidekiq processes with a limited amount of threads.

The other components didn't need a particular attention; they seemed to manage what's happening without external intervention. The default configuration has been mostly kept as is.

During the "purge" of Sidekiq's todo-list, Tooting was… usable ? What I mean, a page loaded properly, we could consult and send posts. Of course, we'll talk about sending posts to remote servers later. Same goes for remote publications. Sending an attachment to a post was impossible mission, due to reasons I described above. At one moment, I did force a bit to shorten Sidekiq's todolist by releasing the kraken !

A new crusing pace

While the mass migration was getting over, the resource allocation to Tooting was adjusted by increasing what was at its disposal: CPU, RAM, storage space. The Sidekiq's runners have been configured as such: 4 processes with 15 theads each, for the magical total of 60. It will suffice to process new tasks and absorb the small extra work quickly when they happen.

The systemd service configuration was slighty tweaked so one can start a group of 15 Sidekiq threads "on demand"; it is a templated systemd service.

PostgreSQL — the database server — didn't see much change, so did the HTTP² server. The current condfiguration seems enough to resopond to inbound HTTP queries, as far as I'm aware.

In the end, Tooting "was hit" but was able to whistand, considering it could been hit a bit more heavily. This migration wave is behind us, we're ready to have manage one more, be it a bit larger or smaller.

Cet article ne va pas rentrer dans les détails de qu'est-ce que Mastodon ou le réseau global, le Fedivers, aussi appelé la fédération. Il existe suffisamment d'articles et de vidéos sur le web qui expliquent ces deux concepts.

Cet article est écrit un an environ après les faits. Certains fins détails peuvent être omis pour simplifier la lecture de l'article, ou bien je ne me rappelle plus exactement comment ça s'est déroulé.

Les composants de Mastodon

Commençons par donner un peu de contexte avec ce qu'il y a sur la machine virtuelle qui gère ce service. Il y a une base de données PostgreSQL qui s'occupe de stocker les données du serveur local et des autres serveurs de la fédération. Ensuite, il y a toute la brique logicielle de Mastodon, que ce soit du Ruby ou du Javascript:

• Un serveur web pour répondre aux demandes des utilisateurs ou autres serveurs (requêtes HTTP)
• Un serveur Websocket pour envoyer les publications en direct aux utilisateurs connectés
• Un serveur Redis (service externe) pour le cache et les files d'attente du service suivant
• Un logiciel de gestion de files d'attentes pour exécuter des tâches en fond

Le dernier composant est important au bon fonctionnement du service. En effet, il envoie les publications aux serveurs distants, achève la réception des publications distantes, finalise les envois d'images ou de fichiers, traduit les publications sur demande et génère la prévisualisation des liens, entre autres. Autrement dit, ce composant fait beaucoup de tâches qui contribuent au bon fonctionnement de la plateforme.

En d'autres termes, si ce composant est surchargé, cela se fera sentir directement sur la plateforme sous la forme, par exemple, d'envois de fichiers qui prennent une éternité à être finalisés.

Le calme avant la tempête

Avant, Tooting était un service de petite taille (une centaine d'utilisateurs, je crois ?), quand on le compare à ce qu'il est actuellement. Chaque brique logicielle de la plateforme était configurée "par défaut", ce qui était largement suffisant à l'époque. L'espace disque est resté suffisamment grand pour conserver une copie des médias distants et locaux et les données sauvegardées dans PostgreSQL.

Comme les inscriptions sont derrière une validation manuelle par les administrateurs et modérateurs, histoire de chasser le spam et de se débarrasser du problème en amont, ils ont une idée plus ou moins directe de ce qui allait pouvoir arriver et s'il faut prévoir davantage de ressources. Le nombre d'inscriptions sur une période a déjà fait des sursauts avec ce qui se passait sur un autre bout d'Internet. Cependant, rien d'extraordinaire qui a nécessité une grosse intervention.

Le temps se gâte

Lorsque le rachat de Twitter a été conclu, de nombreuses personnes se sont jetées sur les divers serveurs du réseau global, créant une montée en charge plus ou moins brutale pour tout le monde. Certains serveurs qui arrivaient à capacité pour des raisons techniques ou de taille ont dû se résoudre à fermer les inscriptions libres et les limiter avec des invitations.

Avant le rachat et pour donner un ordre d'idée de la montée en charge soudaine sur Tooting, il y avait environ 20 inscriptions par mois; durant la vague, environ 100 utilisateurs par jour créaient un compte.

Le nouveau propriétaire commence à faire ses modifications qui finiront par résulter au site que l'on connaît. Ces changements contribuent à l'ampleur de la vague de migration qui s'abat déjà sur le Fedivers. Pour Tooting, les problèmes intéressants ne font que commencer.

La tempête, la rivière sort de son lit

Initialement, les conséquences se sont traduites en une augmentation du volume d'e-mails à envoyer, volume qui a explosé le quota de notre prestataire. Le serveur SMTP¹ rejetait l'envoi et la tâche revenait au fond de la file d'attente. Pour accommoder le volume qui a explosé, une demande d'augmentation du quota a été faite à Infomaniak, éventuellement accordée. En attendant la réponse, il ne restait plus grand-chose à faire du côté des mails: désactiver l'envoi de mails superflus pour l'administration et modération, et attendre. Au rythme de 100 utilisateurs par jour, des boîtes mail ont pu respirer.

En clair, vous vous inscriviez sur Tooting, vous pouviez aller vous faire un café ou vaquer à vos occupations; la confirmation de l'adresse e-mail utilisée viendra par la poste !

Outre les mails qui avaient de la peine à partir, il y a d'autres conséquences directe d'une file d'attente surchargée. Vous vous souvenez de ce petit bout de phrase ?

Le dernier composant [de la brique logicielle de Mastodon] est important au bon fonctionnement du service

Il s'avère que le compteur de tâches en attente a atteint des sommets et continuait de grimper. La configuration "par défaut" de Sidekiq (j'explique plus bas ce que c'est !) ne suffisait plus. Je liste quelques pépins rencontrés avec une file d'attente qui ne cesse de grandir:

• Les publications fédérées arrivent "en retard".
• L'envoi des publications locales est retardée.
• Le temps d'envoi des images est considérablement rallongé. Techniquement l'image ou la pièce jointe est envoyée, la tâche représentant le traitement était juste au fond de la todo-list.
• La génération des cartes d'intégration des liens était retardée.
• Les e-mails qu'on ne présente plus !

La liste n'est pas exhaustive, bien sûr.

Je pourrais continuer à écrire sur les divers problèmes qu'il y a eu sur Tooting, mais on a compris que la rivière est sortie de son lit et cause des inondations un peu partout.

Fonctionnement global de Sidekiq, le gestionnaire de files d'attente

Avant de continuer, il est intéressant de connaître le fonctionnement global du service de file d'attente, Sidekiq de son vrai nom. Quand l'administrateur lance un processus Sidekiq, la tâche principale crée autant de tâches secondaires (aussi appelés thread) que configuré. Le processus principal se charge de récupérer ce qu'il y a à faire depuis un service externe et de les distribuer pour réalisation. Les threads s'occupent de réaliser l'action demandée.

Une fois l'action exécutée correctement et sans erreur, un compteur est augmenté de un et le thread passe à la suite ou attend. En cas de problème, la tâche en erreur est remise dans une file "Tentatives" et sera ré-exécutée ultérieurement. S'il y a trop d'erreurs pour une tâche donnée, elle finit éventuellement dans une file "Mortes" et ne sera plus touchée à moins que l'administrateur les remette manuellement dans le circuit.

Par conséquent, il est dans l'intérêt de l'administrateur de garder une longueur de file d'attente réduite pour que la plateforme reste réactive. Sinon, on se retrouve avec les divers symptômes que j'ai décrit plus haut.

Sidekiq est aussi capable d'exécuter des actions programmées, mais cela ne nous importe pas dans ce contexte.

Gérer la montée en charge

La priorité a été de rétrécir la file d'attente de Sidekiq. Cela pouvait prendre le temps qu'il faudra, mais le compteur de tâches en attente devait descendre.

Une seule chose à faire: augmenter le nombre de threads à une bonne poignée en ne gardant qu'un seul processus principal. En théorie, c'est une bonne idée, en pratique, le chef d'orchestre se retrouve pleinement occupé, sans jamais occuper tous les threads. Une autre solution est de lancer plusieurs processus Sidekiq avec un nombre de threads limité.

Les autres composants n'ont pas eu besoin d'une attention particulière; ils semblaient capables de gérer ce qui leur arrive sans intervention externe. La configuration a été gardée plus ou moins "par défaut".

Durant la "purge" de la liste d'attente de Sidekiq, Tooting était… utilisable ? Ce que je veux dire, une page chargeait correctement, on pouvait consulter les publications et en publier. Naturellement, on reparlera plus tard de l'envoi des publications locales aux autres serveurs. Idem pour les publications distantes. Par contre envoyer une pièce jointe était mission impossible, pour la raison que j'ai citée plus haut. À un moment, je suis passé en force pour vider cette fichue file d'attente en ouvrant les vannes !

Le nouveau rythme de croisière

Pendant que la migration de masse se tassait, l'allocation des ressources à Tooting a été ajustée en augmentant ce qu'il y a à disposition: CPU, RAM, disque. Les exécuteurs de tâches de Sidekiq ont été paramétrés à leur nouveau rythme de croisière: 4 processus avec 15 threads chacun pour le total magique de 60. Cela suffit à traiter les nouvelles tâches et absorber rapidement des petits surplus de travaux qui surgissent.

Le fichier de configuration systemd pour le service a été légèrement modifié pour laisser la possibilité de démarrer des processus Sidekiq de 15 threads "au besoin"; il s'agit maintenant d'un service avec un chablon.

PostgreSQL — le serveur de bases de données — n'a pas vu de gros changements, ni le serveur HTTP² d'ailleurs. La configuration presque par défaut de ces derniers suffit encore à satisfaire les requêtes entrantes, à ce que je sache.

Finalement, Tooting a "pris cher", mais s'en est quand même "bien sorti" considérant qu'on aurait pu avoir pire. Cette vague de migration est maintenant derrière, nous sommes prêts à gérer une nouvelle, qu'elle soit un peu plus grande ou plus petite.