we're doing this eight reasons why WhatsApp was able to support 50

billion messages a day with only 32 Engineers now this is a lot of messages I don't
care what anyone has to say people are like oh you're just going from A to B how's
this so difficult whenever there's 50 billion anythings you have to do you got to
remember that what is it 84 600 seconds in a day that means every single second
you're doing 591 000 messages that means every millisecond you're doing 591
messages every millisecond that's

really really really really really really really fast right that's a lot
think about it JavaScript can't even wake up on a timer when you tell it to wake up
on a timer okay it will be off by a millisecond you just pushed back 591 messages
because you were off by a half millisecond okay that is crazy in this post we'll be
outlining the incredible story of WhatsApp co-founder uh Jan comb or and the
engineering techniques employed to scale WhatsApp quickly if you know somebody who
wants to study scalability

patterns considering sharing this post with them we're gonna share right
now actually in January 2008 California United States classic 2008. everyone was
there uh Jan Coleman engineer at Yahoo why is that I swear to good to gosh he
double hockey sticks what the he double hockey sticks is going on here the best
Engineers I know were early day Yahoo Engineers I think there is something I think
there's something to it I think there's something in there do you know what I mean
anyone that's been a part of the

valley knows for a fact that when you were early Yahoo this was like a
sign of the great Engineers like these are the great ones now everybody had this
idea that you know Google was the great place I'm telling you Google was not the
place in the valley everybody talked about the Yahoo Engineers my time there
especially in the early 2013s I went from 2013 to 2020 was my time in the valley
and during like 2013 to 15 that's all you heard about was Yahoo Engineers it's wild
yui I worked with some of the

creators of yui it's crazy it is crazy uh let's see applies to work at

Facebook rejected loser this was not the end he moved on with his life he buys an
iPhone in a subsequent year and immediately recognizes the huge potential of the
new app store he decides to build an instant these pop-ups are just the worst he
buys an iPhone in the subsequent year and immediately recognized the huge potential
of the new app store he decides to build an instant messenger with some of his
former former co-workers from Yahoo called it Yahoo's

skill issue they name it WhatsApp the vision behind WhatsApp was to
replace the expensive SMS brilliant really SMS was expensive during those days
remember those days do you remember that you used to have to buy like a thousand
messages and then you had to like pay for more than a thousand messages isn't that
wild like that's wild you had to do that crazy with one million people signing up
each day the growth rate of WhatsApp was mind-boggling agreed WhatsApp was able to
support 50 billion messages a day

from 450 million daily active users with only 32 Engineers so I do want
to say something right here that makes this slightly less impressive in some sense
we Netflix supported billion hours of streaming with like um with less than 2 000
Engineers for quite some time and part of that is because when you have a very
narrow product you can also make a really big things happen on a very narrow scale
so I don't think it's like so WhatsApp is a very narrow product you're not trying
to build the soup kitchen this ain't Amazon you're

building one thing and one thing well right so is this good is this bad I
still think it's impressive although explosive product growth is a good problem to
have yawn I'm just gonna call them yon and the team behind WhatsApp had to adopt
the best engineering practices to overcome the challenges in my opinion Jan if you
were on here be like oh yeah yeah it is yawn all that problem's so hard oh that's
me doing a German accent which I'm very bad at yeah the German accent so bad oh
yeah canceled all right all right now that we

got done a speed speed running canceling today uh WhatsApp engineering

the WhatsApp engineering practices to attain extreme scalability can be summarized
as follows single responsibility principle the product Focus was always on the core
feature messaging yes I just I literally said this single likes single focused
single focused yeah you got to put all your focus on it anyways [Music] they didn't
bother to build an advertising Network or a social media platform yeah users needs
stakeholders ideas this

is great this is really great this is a great beautiful picture right

here this is great I'm not blushing you're blushing uh they eliminate feature creep
at all costs feature creep is when you add excessive features to a product that
make it too difficult to use uh the minimalistic product requirements the team was
able to prioritize the reliability of WhatsApp over everything else absolutely we
did this a lot in my early days at Netflix I built out the social network feature
in Netflix way back in the day

uh 2013 2014 2014 it came out Darwin social mountain is what we called it
and we saw a lift in streaming more people used our product and started streaming
but because we were doing a rewrite coming up next of the website and the streaming
lift was there but it wasn't huge we literally just scraped the feature because why
support a feature long term right wasn't worth it get it out get it out technology
stack erlang programming language was used to implement the core functionality of
WhatsApp servers for

the following reasons of course here it comes functional Bros shy Rye
come on shy Rye you want to talk about Elixir now shy right shy right it's not oh
Campbell kind of sounds pretty stupid to me sounds kind of lame just saying it
sounds kind of lame not oh camel not my problem that's what I always say provides
extremely high scalability with tiny footprint monads okay we're gonna do it we're
gonna go to imager uh we're gonna go to imager it just feels like the right time to
do this right uh Ancient Aliens guy right

this just feels like the right one we jump onto here this feels like a
good time to do this and just toss in monads it just feels like the right thing to
say in the moment you know sometimes you just gotta do it and you just generate the
meme and you just save it and you moan ads it just feels right you know and then
you go to you go to what appears to be twitter.com but it says X but we're still at
Twitter nobody knows what's actually happening with this whole twitter.com business
take this post it and there you go that's how you

become a successful social marketer okay that's when I was looking up

stuff about that so there we go we're back in we're back in blazingly fast memes
monads a less problem that's true uh uh Joe Armstrong did not uh regard erlang as a
functional programming language he regarded it as a concurrent programming language
above all else oh that's actually extremely interesting take huh okay that's cool
that's kind of like a cool way to think about it uh extremely high scalability with
a tiny uh footprint supports hot reloading

that's cool hot reloading is a big thing the faster your stuff happens
obviously the nicer it is really like that threads our native feature of erlang
unlike Java or C plus where threads belong to the operating system the native
threads in erlang make context switching cheaper because there's no need to save
the entire CPU state in other words green threads but green threads have been
implemented everywhere at this point how reloading makes it easier to deploy code
changes without the server restarting or traffic reduction a

redirection oh interesting you can just hot reload that damn thing live
and so in simple words hot reloading offers extreme High availability that's cool
just literally go that's cool that's cool why reinvent the wheel do not reinvent
the wheel either use open source or purchase a commercial solution sorry I'm too
busy yeah okay fair fair uh Egypt Baird is an open source real-time messaging
server written in erlang WhatsApp was built on top of eduard um really that stands
for edgebeard which is actually Edge Lord

with the beard I think is what it means the team extended Edge beard by
writing uh some of the core components to fit their requirements what's app
leveraged third-party services such as Google push to provide push notifications
yeah that I mean that seems reasonable cross-cutting concerns cross-cutting
concerns are the things that affect many parts of the product and are hard to
separate for example monitoring and alerting the health of services a huge emphasis
was given to cross-cutting concerns to improve product quality okay

yeah yeah I mean these all look good I love your circle thank you for
that continuous integration is a practice where Engineers regularly merge their
code changes into a central repository continuous delivery is a practice of
automating or automatically deploying your code okay WhatsApp employed continuous
Integrations and continuous delivery to improve their software development process
with hot reloading Okay so I mean this is just called up good prep just like
everybody everybody does this okay that's not

what are you trying to impress me at this point you trying to impress me
what are you trying to like make me feel happy about something come on come on come
on uh scalability horizontal scaling is the process of increasing the number of
machines in the resource pool vertical scaling is the process of increasing the
capacity of an existing machine such as CPU memory diagonal scaling is the hybrid
of horizontal and vertical scaling where Computing and resources add both
vertically and horizontally okay WhatsApp utilized

diagonal scaling uh to keep costs and operational complexity low this is

actually really good because I do think I do like the idea of diagonal scaling if
you have the ability to use all your processes like unlike a node application in
which you have to treat you have to like come up with this highly theoretical idea
of how you're supposed to handle how many node instances are on a machine and you
have to go all confusing as opposed to a single language managing at all I like the
idea that you want to do both right because a

more expensive machine might be able to handle three times more of the
load but cost only twice as much as the smaller ones so there is something really
good here right there's something very very good here free BSD operating system was
used to run a WhatsApp neckbeards for sure uh because I mean I guess they are using
erlang I mean they're already there they're already there not surprised at this
point because they had previous experience with FreeBSD while working at Yahoo in
addition FreeBSD offers a nicely tuned

and reliable Network stack yeah I've heard really great things about the
network stack FreeBSD was fine-tuned to accommodate two plus million connections
per server the kernel parameters such as files and sockets were modified class
classic gotta bump that you limit dog uh the servers were over provisioned to
handle sudden traffic spikes and keep head room for failure such as Network
partitions or Hardware faults okay the metrics such as CPU contact switches and
system calls were measured the bottlenecks were identified and
 eliminated this was performed at regular intervals okay The Continuous
feedback cycle tremendously improved the performance of WhatsApp load testing is
the process of measuring I mean by the way this all sounds really good right you
use the right they I mean it really sounds like they used the right tools for the
job right you have a language that can hot reload itself in production you used the
operating system in which is like known for being Best in Class networking you
offload everything you don't want to build

yourself and you just Crush I mean this it seems this seems and then you
can continuously try to improve performance this seems like the right model for a
very simple app right I don't understand anything we'll start understanding then
load testing is the process of measuring the performance of a system under
anticipated load uh load testing was performed to identify single points of failure
yep that's good uh load testing was performed by either generating artificial
production traffic or configuring DNS to redirect more traffic

to a particular server classic DNS is always the issue I I always worry

about anytime you touch DNS terrifying it's just terrifying Okay small team size
the communication path between Engineers increased quadratically as the wow
quadratically um pound mini daddy redirection that's what it stands for uh as the
team uh grows in size uh this is a recipe for degraded productivity I actually
agree with this heavily the more I have to communicate the harder it is like just
like that thing I was talking to you

earlier about about uh having a schema and a data output and I just
wanted the schema decoding function that was used clearly on a website and this
website was was handled by like six different teams and I must have communicated
with at least 20 to 30 different Engineers none of whom could actually tell me I
went to several slack rooms and at the end it was more efficient for me just to
rewrite the decoding process than to figure out what the hell happened to that
library or whatever they used right

like there just does come a point where duplication is a requirement as

Engineers grow and it's very very difficult the size of WhatsApp engineering team
was a was a small kept 32 Engineers right WhatsApp uh so I do think that this is a
huge W when you have a narrow product plus few Engineers you can accomplish an
incredible amount of stuff I don't know if you can say it's a fully connected graph
like this but it is something that really does scale disproportional uh WhatsApp is
considered one of the most successful

instant Messengers on the market in 2014 WhatsApp was acquired by for a

whopping 19 billion USD by the same Facebook that rejected Yan according to Forbes
Jana has a net worth of 14 billions in 2023. so many commas um moral of the story
Facebook loves spying on your ass thanks for the moral of your story I appreciate
that a good reason by the way this is a great reason to use signal okay signal is
designed in a way that it's unacquirable right signal use signal because guess what
it's actual encryption that takes place

not in some place in which is saving all your data love it love signal
hashtag ad this is not an ad actually I just use signal all the time because it's
way way way way way good right signal is good but it could be better of course it
could but that's what happens when you make zero dollars signal do you know that
signal is ran off of Charity do re you realize that right signals ran by the the
NSA dude greatest anime betrayal of all time the name is I've actually never used
WhatsApp I mean I I was a part of it one time and

then I just never got it again because it just made me feel uncomfortable
and I didn't really like WhatsApp this was like back in 2018 when I was text
messaging some of my Yahoo former Yahoo uh co-workers at Netflix it was very
fantastic again