Hey Guys
Just letting you know we're aware of the current issues the network has been unstable for the past 2 days. We're aware of the issue and are working on fixing it as soon as possible. We apologize for any inconvenience this is causing - we understand Hypixel is a happy distraction away from current world events, and we'll do our best to ensure it remains as such.
Update #7 from Donpireso (March 23 - 4:36 PM EST): Last night we installed another similar powerful server to our core services and we're currently monitoring how things are going for the next 48 hours. So far so good, it appears that the hardware is stable on this one! Our plan is to add more redundancy by having more big servers to that cluster and avoid any future interruptions.
Update #6 from Simon (March 22 - 8:30 AM EST): Datacenter changed every single piece of hardware on the core system machine but it still reboots itself, for no reason, even with nothing on it. That being said, we are currently using the old machine and we will be limiting the player count to 65,000 today while we order a completely different machine - this is very frustrating and exhausting, we finally had a solution and now we have to deal with a problem that makes absolutely no sense. We will be ordering a few powerful machines with different specs in the hope to find one that just works.
Update #5 from Donpireso (March 21 - 6.06 AM EST): We had a good day, but for some reason this morning our big beefy machine has started rebooting on its own - we're thinking hardware issues. We are working as quickly as we can to fix it but in the meantime, some players may experience random kicking and instability.
Update #4 from Simon (March 20 - 9:24 AM EST): The hardware upgrade for MasterControl (one of our core systems) finally arrived and it's a very powerful machine that has 72 threads. We are seeing massive improvements, in fact, at 64k players it barely uses 8% of its total power. Let's see how it goes today!
Update #3 from Simon (March 19 - 9:42 AM EST): Deployed over 20 optimizations updates on core systems, we are seeing major performance results and will monitor for the day. We also just installed over 70 new dedicated machines to help the performances of minigames. We are also looking into a major hardware system upgrade for one of our core machine, we are talking about twice the performance but also 10x the price, so this would be a temporary upgrade but give us a lot of stability to reach well over 100k players online if needs be. Rest assured, our goal is performance and stability first.
Update #2 from Simon: Found another core issue past 65k players, need to test a few fixes live at peak - this will break things! Apologies for interruptions but it's truly needed for us to be able to have a proper fix once and for all.
Update #1 from Simon: We just rolled a lot of optimizations and we are now monitoring. We are already seeing great results. In the mix of all these updates, we also disabled a few features such as the Friend List and Hypixel Experience for some part of the server - rest assured they will be back later today, we just need to gather more data on the updates we just rolled. Fortunately, these features were not impacting the server, but would rather not take chances to mess our data that we are working with.
Continue reading...