DEV Community: Juma Evans

Cron Jobs in Linux: Automating Tasks Without Lifting a Finger

Juma Evans — Sat, 10 Oct 2026 16:43:11 +0000

Imagine having a Bash script that backs up your files every day. The script works perfectly, but there is one problem: you have to remember to run it yourself.

What if Linux could run it automatically at a specific time, even when you are busy doing something else?

That is exactly what cron jobs are designed to do.

Cron is a time-based job scheduler available on many Linux and Unix-like systems. It allows you to schedule commands and scripts to run automatically at specified times or intervals.

In this article, we will explore how cron works, how to schedule tasks, and how to troubleshoot common problems.

1. What is a cron job?

A cron job is a scheduled command or task that runs automatically according to a defined schedule.

For example, you might want to:

Back up important files every night.
Run a maintenance script every Sunday.
Generate a report every morning.
Clean up temporary files regularly.

Instead of running these tasks manually, you can configure cron to execute them for you.

Cron uses a scheduling configuration called a crontab, short for cron table.

To open your personal crontab, run:

crontab -e

The first time you run this command, your system may ask you to choose a text editor. Once the editor opens, you can add your scheduled commands.

To view your existing scheduled jobs, use:

crontab -l

To remove your personal crontab entirely, use:

crontab -r

Be careful with the last command: it removes all jobs in your personal crontab, not just one entry.

2. Understanding cron syntax

A typical cron entry contains five time fields followed by the command to execute.

* * * * * command

Each asterisk represents a scheduling field.

Field	Allowed values	Meaning
Minute	0–59	Minute of the hour
Hour	0–23	Hour of the day
Day of month	1–31	Calendar day
Month	1–12	Month of the year
Day of week	0–7	Day of the week; 0 and 7 usually mean Sunday

Let's look at a real example:

0 8 * * * /home/user/scripts/backup.sh

This schedules backup.sh to run every day at 8:00 AM.

Why?

0 means minute zero.
8 means 8 AM.
The remaining three asterisks mean every day of the month, every month, and every day of the week.

The command after the five fields is what cron executes.

Important: Cron generally uses the machine's configured local time zone unless configured otherwise. Check your system's time zone when scheduling important tasks.

3. Common cron expressions you should know

Once you understand the five fields, reading cron expressions becomes much easier.

* * * * * command

Runs every minute.

*/5 * * * * command

Runs every five minutes.

0 9 * * 1-5 command

Runs at 9:00 AM, Monday through Friday.

0 0 * * 0 command

Runs at midnight every Sunday.

30 18 * * * command

Runs every day at 6:30 PM.

The */5 expression means every five units within that field, while 1-5 represents a range of values.

One detail worth remembering: when both the day-of-month and day-of-week fields are restricted, traditional cron implementations generally run the job when either field matches. If you need a complicated schedule, check the documentation for your system's cron implementation.

4. Scheduling your first Bash script

Let's create a simple script and schedule it.

First, create a directory for your scripts:

mkdir -p "$HOME/scripts"

Create a file named daily-task.sh:

nano "$HOME/scripts/daily-task.sh"

Add the following code:

#!/bin/bash

echo "Cron ran at $(date)" >> "$HOME/cron-log.txt"

This script appends the current date and time to a log file whenever it runs.

Make the script executable:

chmod +x "$HOME/scripts/daily-task.sh"

Test it manually:

"$HOME/scripts/daily-task.sh"

Then inspect the log:

cat "$HOME/cron-log.txt"

If everything works, you are ready to schedule it.

Open your crontab:

crontab -e

Add this entry:

*/5 * * * * /home/yourusername/scripts/daily-task.sh

Replace /home/yourusername with your actual home directory path.

This runs the script every five minutes. After several minutes, inspect the log again:

cat "$HOME/cron-log.txt"

You should see additional timestamps.

And viola! You have created a Bash script and configured Linux to execute it automatically.

5. Why do cron jobs sometimes fail?

A script can work perfectly in your terminal and still fail when executed by cron.

Here are some common reasons.

Using relative paths

Your terminal might be in the directory containing your script, but cron may run it with a different working directory.

Instead of:

*/5 * * * * ./backup.sh

Prefer an absolute path:

*/5 * * * * /home/yourusername/scripts/backup.sh

Assuming your terminal environment is available

Cron usually provides a smaller environment than an interactive shell. Variables defined in your .bashrc may not be available.

If your script depends on environment variables, define the necessary ones explicitly or load the appropriate configuration deliberately.

Forgetting permissions

Check that the script exists and has the required permissions:

ls -l "$HOME/scripts/daily-task.sh"

Ignoring output and errors

Redirecting output to a log can make failures easier to investigate.

For example:

*/5 * * * * /home/yourusername/scripts/daily-task.sh >> /home/yourusername/cron-output.log 2>&1

The >> operator appends standard output to the log file, while 2>&1 sends standard error to the same destination.

For system-level troubleshooting, the available logs and service commands depend on your Linux distribution and cron implementation.

6. Cron vs. running a script manually

The difference is simple:

Manual execution: You decide when to run the command.
Cron: You define a schedule, and the system runs the command automatically when the schedule matches.

Cron does not make a script more powerful. It makes recurring execution more convenient.

You still need to write a correct script, handle errors, and ensure that its dependencies are available.

Final thoughts

Cron is one of the simplest ways to introduce automation into your Linux workflow. A few lines in a crontab can turn a manually repeated task into a scheduled process.

Start with something small, such as writing timestamps to a log. Then experiment with daily reports, backups, and other tasks that benefit from regular execution.

As you become more comfortable with Bash, cron will help you move from simply running commands to building repeatable workflows.

The real benefit of automation is not just saving time. It is making sure important tasks happen consistently, without depending on your memory.

Bash Scripting: From Your First Script to Linux Automation

Juma Evans — Sat, 10 Oct 2026 16:09:27 +0000

When I first started working with Linux, the terminal felt like a place where I typed mysterious commands and hoped they worked.

I ran commands such as ls, cd, mkdir, and chmod. I literally copied commands from documentation, fixed errors when something broke, and gradually began to understand what each command does.

But eventually, you encounter a more interesting question:

What if you could combine these commands into a program that makes decisions and performs tasks automatically?

That is where Bash scripting comes in.

In this article, we will move from the fundamentals of Bash to practical concepts such as variables, conditions, command-line arguments, file permissions, environment variables, executable commands, and scheduled tasks.

Whether you are learning Linux for the first time or preparing for practical systems programming exercises, understanding these concepts will help you work more confidently in the terminal.

1. What is Bash?

Bash stands for Bourne Again SHell. It is a Unix shell that allows you to interact with your operating system by entering commands.

A shell can execute programs, manage variables, combine commands, and control how tasks run.

For example, consider these commands:

pwd
ls
mkdir projects

They display your current directory, list its contents, and create a directory called projects.

Instead of entering every command manually, you can place several commands inside a script and execute them together.

A Bash script is simply a text file containing shell commands and, optionally, programming logic.

2. Writing your first Bash script

Let's create a simple script.

First, create a file:

nano hello.sh

Add the following:

#!/bin/bash

echo "Hello, Linux!"
echo "I am learning Bash scripting."

Save the file and exit the editor.

Let's understand the two important parts.

#!/bin/bash is called the shebang. It tells the system which interpreter should execute the script when it is run as an executable.
echo prints text to the terminal.

You can execute the script using:

bash hello.sh

Alternatively, make it executable:

chmod +x hello.sh
./hello.sh

The chmod +x command adds execute permission, while ./hello.sh runs the file from the current directory.

Key lesson: Writing a script and making a script executable are two different things.

3. Variables: Giving information a name

Variables allow you to store values that can be reused throughout a script.

#!/bin/bash

name="Evans"
language="Bash"

echo "My name is $name."
echo "I am learning $language."

Notice that there are no spaces around the = sign when assigning a variable.

This works:

name="Evans"

This does not work as an ordinary assignment:

name = "Evans"

Bash interprets the second example as a command named name with additional arguments.

To access a variable, prefix its name with $, or use ${name} when clearer boundaries are needed.

project="backend"

echo "Working on ${project}-development"

Reading input from a user

Variables can also store information entered at runtime.

#!/bin/bash

echo "What is your name?"
read -r name

echo "Welcome, $name!"

The read command collects input, and -r prevents backslashes from being interpreted as escape characters.

This makes your script interactive instead of relying entirely on values written into the file.

4. Command-line arguments

Sometimes you want a script to accept information when you execute it rather than ask questions interactively.

Consider this script:

#!/bin/bash

echo "First argument: $1"
echo "Second argument: $2"
echo "Total arguments: $#"

Save it as arguments.sh, then run:

bash arguments.sh apple mango

The output will be:

First argument: apple
Second argument: mango
Total arguments: 2

Here is what the special variables mean:

Variable	Meaning
`$0`	Name or invocation path of the script
`$1`	First positional argument
`$2`	Second positional argument
`$#`	Number of positional arguments
`$@`	All positional arguments
`$?`	Exit status of the previous command

These variables become especially useful when building scripts that accept filenames, directories, or configuration values.

Validating arguments

What if a script requires exactly one filename?

#!/bin/bash

if [ "$#" -ne 1 ]; then
    echo "Usage: $0 <filename>"
    exit 1
fi

echo "Provided file: $1"

The script checks the number of arguments before proceeding.

-ne means not equal.
exit 1 terminates the script with a nonzero status, conventionally indicating failure.
"$1" preserves the argument as a single value, even if it contains spaces.

This is an important programming principle: validate your inputs before using them.

5. Conditions: Modifying scripts to make decisions

So far, our scripts have executed commands in sequence. Conditions allow them to behave differently depending on the situation.

Consider a numerical comparison:

#!/bin/bash

X=10
Y=5

if [ "$X" -gt "$Y" ]; then
    echo "X is greater than Y"
else
    echo "X is not greater than Y"
fi

The condition checks whether X is greater than Y.

Common integer comparison operators include:

Operator	Meaning
`-eq`	Equal to
`-ne`	Not equal to
`-gt`	Greater than
`-lt`	Less than
`-ge`	Greater than or equal to
`-le`	Less than or equal to

Why do the spaces matter?

This is valid:

if [ "$X" -gt "$Y" ]; then

This is incorrect:

if ["$X" -gt "$Y"]; then

In the first example, [ is a command that receives the condition's components as separate arguments. The spaces ensure those arguments are separated correctly.

The closing ] must also be a separate argument.

The then keyword begins the block of commands that runs when the condition succeeds, and fi closes the conditional statement.

You can add more branches using elif:

if [ "$X" -gt "$Y" ]; then
    echo "X is greater"
elif [ "$X" -eq "$Y" ]; then
    echo "They are equal"
else
    echo "Y is greater"
fi

6. Working with files and directories

File checks are among the most practical features of Bash scripting.

Suppose you want a script to verify that a file exists before attempting to read it.

#!/bin/bash

if [ -f "$1" ]; then
    echo "File exists"
else
    echo "File does not exist"
fi

The -f test checks whether the path refers to a regular file.

But existence is not the only thing that matters. You might also need to know whether a file is readable, writable, or executable.

Test	Meaning
`-e`	Path exists
`-f`	Regular file
`-d`	Directory
`-r`	Read permission is available
`-w`	Write permission is available
`-x`	Execute permission is available

You can combine these checks:

#!/bin/bash

if [ "$#" -ne 1 ]; then
    echo "Error: Provide one file"
    exit 1
fi#!/bin/bash

echo "What is your name?"
read -r name

echo "Welcome, $name!"

file="$1"

if [ -f "$file" ]; then
    echo "File exists"

    if [ -r "$file" ]; then
        echo "File is readable"
    fi

    if [ -w "$file" ]; then
        echo "File is writable"
    fi

    if [ -x "$file" ]; then
        echo "File is executable"
    fi
else
    echo "Not a regular file"
    exit 1
fi

This combines argument validation, variables, conditions, and file tests in one practical example.

One important detail: -r, -w, and -x test access available to the current process. They do not simply report whether a permission bit appears in the output of ls -l.

7. File permissions and timestamps

Linux file permissions determine who can read, modify, or execute a file.

For example:

chmod 600 file1.txt

This gives the owner read and write permissions while removing those permissions from group members and others.

The numeric values represent combinations of permissions:

4 means read.
2 means write.
1 means execute.

Therefore, 6 represents read plus write, because (4 + 2 = 6).

You can inspect file details using:

stat file1.txt

And change a file's modification timestamp using:

touch -d "2022-01-01 10:30:00" file1.txt

The -d option lets you specify a date and time.

These commands are useful for understanding permissions, filesystem metadata, and how Linux represents files beyond their names and contents.

Be careful when experimenting with timestamps: touch changes metadata, not the contents of the file.

8. Environment variables and PATH

Some variables are used by a single script, while others are available to child processes.

An environment variable is a variable exported into the environment inherited by child processes.

For example:

export APP_ENV="development"

You can inspect environment variables with:

printenv

Or search for a particular variable:

printenv PATH

The PATH variable contains a list of directories where the shell searches for executable commands.

When you type:

ls

your shell searches for an executable named ls in the directories listed in PATH.

You can inspect those directories with:

echo "$PATH"

Directory entries are separated by colons.

Adding your own executable directory

Suppose you create a directory called myBins in your home directory:

mkdir -p "$HOME/myBins"

Add it to PATH for the current shell session:

export PATH="$HOME/myBins:$PATH"

Now you can place executable scripts in that directory and run them by name.

For example, if an executable script is saved as:

~/myBins/01exec

You can run:

01exec

provided the directory is on PATH and the script has execute permission.

To keep this change across new interactive Bash sessions, add the export command to your ~/.bashrc file and reload it:

source ~/.bashrc

The source command executes the file in the current shell, allowing its variable assignments and other shell changes to affect that session.

Important distinction: source runs commands in your current shell; executing a script as a separate process generally does not change the parent shell's variables.

9. Aliases: Creating shortcuts

An alias lets you define a shorter name for a command.

For example:

alias ll='ls -lah'

Now typing ll runs ls -lah in the interactive shell.

You can define a custom alias for a command you use frequently:

alias myfiles='ls -l'

Aliases are useful for interactive convenience, but they are generally not the best way to build reusable scripts. For automation and scripts, prefer functions or standalone executable programs where appropriate.

To make aliases available in future interactive Bash sessions, add them to ~/.bashrc.

10. Redirection and pipes

A command normally receives input from standard input and writes output to standard output. Error messages are commonly written to standard error.

Bash allows you to redirect these streams.

echo "Hello" > output.txt

The > operator writes output to a file, replacing its previous contents.

To append instead:

echo "Another line" >> output.txt

You can redirect errors:

ls missing-file 2> errors.txt

Here, 2> redirects standard error to errors.txt.

A pipe sends one command's standard output into another command's standard input:

cat names.txt | grep "Evans"

For this simple example, you can also write:

grep "Evans" names.txt

Understanding redirection and pipes helps you combine small tools into useful workflows without writing a separate program for every task.

11. Automating tasks with cron

Running a script manually is useful, but sometimes you want it to run automatically.

Cron is a scheduling service on many Unix-like systems that can run commands at specified times.

For example, this cron expression:

0 8 * * * /home/user/scripts/backup.sh

requests execution every day at 8:00 a.m., according to the cron service's local time configuration.

The five fields represent:

minute hour day-of-month month day-of-week

You can edit your personal cron schedule with:

crontab -e

Cron is useful for recurring jobs such as log cleanup, backups, or periodic reports.

However, scripts run by cron often have a more limited environment than scripts run from an interactive terminal. Use absolute paths where practical, configure required environment variables explicitly, and redirect output to a log when debugging.

For example:

0 8 * * * /home/user/scripts/backup.sh >> /home/user/backup.log 2>&1

This appends both standard output and standard error to the log file.

Scheduling automation introduces an important new responsibility: a script must not only work when you run it manually; it must also work reliably in the environment where it is scheduled.

12. Putting the concepts together

Let's combine several of the ideas into one useful script.

This script checks whether a regular file exists, verifies that it is readable, and then searches it for a phrase.

#!/bin/bash

if [ "$#" -ne 2 ]; then
    echo "Usage: $0 <filename> <search-term>"
    exit 1
fi

file="$1"
search_term="$2"

if [ ! -f "$file" ]; then
    echo "Error: File does not exist or is not a regular file"
    exit 1
fi

if [ ! -r "$file" ]; then
    echo "Error: File is not readable"
    exit 1
fi

grep -n -- "$search_term" "$file"
status=$?

if [ "$status" -eq 0 ]; then
    echo "Search completed: matches found"
elif [ "$status" -eq 1 ]; then
    echo "Search completed: no matches found"
else
    echo "Error: Search could not be completed"
    exit "$status"
fi

Save it as search-file.sh, then run:

chmod +x search-file.sh
./search-file.sh notes.txt "Bash"

The script combines several concepts:

Command-line argument validation.
Variables and quoting.
File existence and readability tests.
Negation using !.
Running an external command.
Capturing and interpreting its exit status.
Returning meaningful exit codes.

The distinction between grep exit statuses matters here: 0 means matches were found, 1 means no matches were found, and a status greater than 1 indicates an error.

This is no longer just a sequence of terminal commands. It is a small program that validates input, checks its environment, handles different outcomes, and communicates results.

13. How to keep progressing

The best way to learn Bash is to build progressively more capable scripts rather than using single commands in isolation.

A sensible learning path looks like this:

Fundamentals: commands, shebangs, variables, input, and output.
Control flow: conditions, comparisons, logical operators, and loops.
Arguments and validation: positional parameters, exit codes, and error handling.
Filesystem operations: files, directories, permissions, and timestamps.
Shell environment: environment variables, PATH, aliases, and source.
Command composition: pipes, redirection, grep, and text processing.
Automation: cron jobs, logging, and scheduled execution.
Reliable scripts: quoting, defensive checks, debugging, and testing.

As you progress, also learn the difference between shell syntax and external commands. For example, if is a Bash keyword, [ is a command used to evaluate conditions, and grep is an external utility.

That distinction makes errors easier to understand and helps you reason about what your scripts are actually doing.

Final thoughts

Bash scripting is more than learning how to automate a few commands. It teaches you to think in terms of inputs, conditions, execution environments, permissions, and failure handling.

Those ideas are useful well beyond Bash. They also appear in backend development, deployment workflows, CI/CD pipelines, containers, and production troubleshooting.

Start with a script that prints a message. Then write one that accepts arguments, checks a file, and handles errors. Eventually, you will be able to automate tasks that would otherwise require repeated manual work.

HTTP Methods: GET, POST, PUT, PATCH, and DELETE

Juma Evans — Sat, 10 Oct 2026 15:13:28 +0000

Every time the frontend communicates with a backend, it sends a request.

Maybe it wants to retrieve a user's profile. Maybe it wants to register a new account, update an email address, or delete a record.

But how does the server know what the client wants to do?

That's where HTTP methods come in.

HTTP methods describe the intended action of a request. They help clients and servers communicate consistently, making APIs easier to understand, build, and maintain.

If you've ever worked with a REST API, you've probably encountered GET, POST, PUT, PATCH, and DELETE.

Let's understand what each one does, when to use it, and the mistakes worth avoiding.

1. GET — Retrieve Data

Imagine you're building a user management API. A user wants to view their profile.

The frontend sends:

GET /users/42 HTTP/1.1
Host: api.example.com

The server retrieves the user with ID 42 and returns the information.

HTTP/1.1 200 OK
Content-Type: application/json

{
  "id": 42,
  "name": "Evans",
  "email": "evans@example.com"
}

That's the purpose of GET: retrieve a resource without requesting a change to the server's state.

Common examples include:

Fetching a list of products.
Viewing a user's profile.
Retrieving an order's details.
Searching for available courses.

For example:

GET /products
GET /users/42
GET /orders/1001

One important detail: GET is intended to be safe, meaning the request itself shouldn't change the resource's state. Reading a profile shouldn't unexpectedly delete a user or charge their account.

Think of GET as asking, "Can you show me this?"

2. POST — Create or Submit Data

Now imagine someone registers a new account.

The client needs to send information to the server:

POST /users HTTP/1.1
Content-Type: application/json

{
  "name": "Evans",
  "email": "evans@example.com"
}

The backend validates the input, checks the relevant business rules, and creates the account if everything is valid.

A successful response might look like this:

HTTP/1.1 201 Created
Content-Type: application/json

{
  "id": 43,
  "name": "Evans",
  "email": "evans@example.com"
}

Notice the 201 Created status code. It communicates that a new resource was successfully created.

POST is commonly used for:

Registering users.
Creating orders.
Submitting forms.
Sending messages.
Triggering operations such as a payment request.

For example:

POST /users
POST /orders
POST /messages

However, POST isn't exclusively for creating resources. It can also submit data for processing or trigger an operation.

Think of POST as saying, "Here is some data; process it."

3. PUT — Replace a Resource

Suppose a user updates their profile. You might want to replace the current representation of that resource with a new one.

A PUT request could look like this:

PUT /users/42 HTTP/1.1
Content-Type: application/json

{
  "name": "Evans Juma",
  "email": "evans.juma@example.com"
}

The server processes the request and replaces the resource's representation with the supplied one, according to the API's design.

The response might be:

HTTP/1.1 200 OK

Or, if the operation succeeds without returning a response body:

HTTP/1.1 204 No Content

Here's the important distinction: PUT generally represents replacement, not a partial update.

If your API treats the request body as the complete editable representation of a user, omitting a field may cause that field to be reset or removed, depending on the API's rules.

That doesn't mean every field in a database must be overwritten. Server-managed values, such as IDs and creation timestamps, can remain unchanged.

A PUT request can also create a resource at a known URI when the API supports that behavior.

Think of PUT as saying, "Make this resource match the representation I'm sending."

4. PATCH — Update Part of a Resource

What if the user wants to change only their email address?

Sending every profile field again may be unnecessary.

That's where PATCH is useful.

PATCH /users/42 HTTP/1.1
Content-Type: application/json

{
  "email": "new-email@example.com"
}

The backend updates the email address while leaving unrelated profile fields unchanged.

For example, the user's name remains the same.

PATCH is commonly used for:

Updating a profile field.
Changing an order's status.
Modifying notification preferences.
Updating a product's price.

One technical detail worth knowing: the meaning of a PATCH request depends on its patch format and the API's implementation. A JSON body containing one field is a common design, but it isn't the only supported format.

Unlike PUT, PATCH isn't inherently idempotent. Its behavior depends on the operation being performed.

Think of PATCH as saying, "Change this part, but leave the rest alone."

PUT vs PATCH: What's the Difference?

This is one of the most common points of confusion.

Suppose the current user resource is:

{
  "name": "Evans",
  "email": "evans@example.com",
  "city": "Kisumu"
}

You want to change only the city.

With PUT, your API might expect the complete editable representation:

{
  "name": "Evans",
  "email": "evans@example.com",
  "city": "Nairobi"
}

With PATCH, you might send only:

{
  "city": "Nairobi"
}

The distinction is about the intended operation, not simply the number of fields in the request body.

5. DELETE — Remove a Resource

Finally, suppose a user wants to delete an account.

The client sends:

DELETE /users/42 HTTP/1.1
Host: api.example.com

If the operation succeeds, the server might respond:

HTTP/1.1 204 No Content

There is no response body because the server has nothing else to return.

Alternatively, the server could return 200 OK with a confirmation message or resource representation.

DELETE is commonly used for:

Removing a user account.
Deleting a product.
Removing a saved address.
Deleting a comment.

One important distinction: deleting a resource doesn't necessarily mean immediately erasing every related database record. An application might use soft deletion, retention policies, or other business rules.

Think of DELETE as saying, "Remove this resource."

A Quick Comparison

Here's a simple reference you can return to when building an API.

Method	Main purpose	Example	Typical success response
`GET`	Retrieve data	`GET /users/42`	`200 OK`
`POST`	Create or process data	`POST /users`	`201 Created`
`PUT`	Replace a resource representation	`PUT /users/42`	`200 OK` or `204`
`PATCH`	Partially modify a resource	`PATCH /users/42`	`200 OK` or `204`
`DELETE`	Remove a resource	`DELETE /users/42`	`204 No Content`

These are common choices, not rigid rules. The appropriate response depends on what the server actually did.

Two Concepts Every Backend Developer Should take into consideration

Knowing the methods is only the beginning. Two additional concepts help explain why HTTP methods matter.

1. Safe Methods

A method is considered safe when the client isn't requesting a change to the server's state.

GET is safe. It should retrieve information rather than perform an action such as placing an order or deleting an account.

This is why you shouldn't design an API that deletes a user when someone visits:

GET /delete-user/42

Automated systems, browsers, and caches may make GET requests in situations where the user isn't explicitly asking to change data.

2. Idempotency

A method is idempotent when repeating the same request has the same intended effect on the server as making it once.

For example, sending this request once or five times should leave the resource in the same intended state:

PUT /users/42

{
  "name": "Evans",
  "email": "evans@example.com"
}

GET, PUT, and DELETE are defined as idempotent methods. POST is not inherently idempotent, and PATCH is not guaranteed to be idempotent.

For DELETE, this doesn't mean every repeated request must return the same status code. The first request might return 204, while a later request returns 404. The key is that repeating the request shouldn't cause additional unintended changes.

This concept is especially important when dealing with retries, network failures, and payment APIs. A client may need an idempotency key or another application-level mechanism to prevent a repeated payment submission from charging someone twice.

Bringing It All Together

Imagine you're building a simple e-commerce API.

Your frontend needs to:

View products.
Create an order.
Replace a delivery address.
Change an order's status.
Delete an item from a saved list.

You could represent those operations with HTTP methods:

GET    /products
POST   /orders
PUT    /users/42/address
PATCH  /orders/1001
DELETE /saved-items/7

Each method communicates the intended action, while the response status code communicates the outcome.

For example:

POST /orders
      |
      v
201 Created

Or:

GET /users/999
      |
      v
404 Not Found

HTTP methods describe what the client wants to do. HTTP status codes describe how the server handled the request.

Together, they make APIs more predictable and easier for developers to work with.

Final Thoughts

These methods are easier to learn through a practical project as you build the APIs.

Start by understanding the five methods:

GET retrieves.
POST submits or creates.
PUT replaces.
PATCH partially updates.
DELETE removes.

Then learn the principles behind them, especially safe methods and idempotency.

As you build APIs, you'll discover that choosing the right HTTP method isn't just about making a request work. It's about making your API's behavior clear to other developers and predictable for the systems that depend on it.

And that's one of the small decisions that separates an API that merely works from one that's well designed.

HTTP Status Codes

Juma Evans — Mon, 14 Sep 2026 09:47:43 +0000

You send a request to an API.

Sometimes everything works:

200 OK

Sometimes you made a mistake:

404 Not Found

And sometimes the problem isn't yours at all:

500 Internal Server Error

These three-digit numbers are HTTP status codes.

They are one of the simplest ways for a server to tell a client what happened to its request.

But with dozens of possible status codes, do you really need to memorize all of them?

Probably not.

As a backend developer, you should understand the important ones and, more importantly, know when to use them.

Let's break them down.

The Five HTTP Status Code Categories

HTTP status codes are grouped into five categories:

The first digit tells you the general story.

For example:

2xx → Good
3xx → Go somewhere else
4xx → Check your request
5xx → Check the server

Now let's look at the ones we actually encounter more often.

200 OK

The classic.

200 OK

It means:

«The request was successfully processed.»

For example:

GET /users/42

might return:

HTTP/1.1 200 OK

{
"id": 42,
"name": "Evans"
}

If you're requesting data and everything went well, "200 OK" is usually what you expect.

201 Created

This one is especially important when building APIs.

201 Created

It means:

«The request successfully created a new resource.»

For example:

POST /users

creates a new user.

Instead of returning:

200 OK

you can return:

201 Created

to tell the client:

«"Your request succeeded, and something new was created."»

For REST APIs, this is commonly used after successful "POST" requests.

204 No Content

Sometimes the request succeeds, but there's nothing to send back.

That's where:

204 No Content

comes in.

A common example is deleting something:

DELETE /users/42

The server successfully deletes the user, but doesn't need to return a response body.

So:

204 No Content

is perfectly appropriate.

Think:

«Success, but there's nothing to return.»

301 and 302: Redirects

Now we enter the "3xx" category.

These codes tell the client:

«"The resource you're looking for is somewhere else."»

301 Moved Permanently

The resource has permanently moved to another location.

For example:

http://example.com
↓
https://example.com

A browser can follow the redirect automatically.

302 Found
This also redirects the client, but traditionally represents a temporary redirect.

You don't need to memorize every nuance immediately.

The important idea is:

3xx → The client needs to follow a different path.

400 Bad Request

Now we get to the codes backend developers see constantly.

400 Bad Request

It generally means:

«The server couldn't process the request because the request itself was invalid.»

Imagine your API expects:

{
"age": 26
}

But the client sends:

{
"age": "twenty-six"
}

The server may respond with:

400 Bad Request

Other examples include:

malformed JSON
invalid request parameters
missing required information
invalid request syntax

The important distinction is:

«The client sent something the server couldn't reasonably process.»

401 Unauthorized

This one is commonly misunderstood.

401 Unauthorized

usually means:

«Authentication is required or the provided authentication credentials are invalid.»

For example:

GET /profile
Authorization: Bearer invalid-token

The server might respond:

401 Unauthorized

Think:

«"Who are you?"»

This is about authentication.

403 Forbidden

Now compare that with:

403 Forbidden

This generally means:

«The server understands who you are, but you aren't allowed to access this resource.»

For example:

User → authenticated ✓
User → administrator ✗

The user might be logged in but still unable to access:

/admin/users

So a useful mental model is:

401 → You haven't successfully authenticated.
403 → You're authenticated, but you're not allowed.

That distinction is extremely useful when designing APIs.

404 Not Found

Probably the most famous HTTP status code.

404 Not Found

It means:

«The requested resource could not be found.»

For example:

GET /users/999999

If that user doesn't exist:

404 Not Found

It can also happen when a route doesn't exist:

GET /something-that-does-not-exist

The server is essentially saying:

«"I don't have what you're asking for."»

405 Method Not Allowed

Here's another useful one.

Suppose an endpoint supports:

GET /users

but doesn't support:

DELETE /users

The server may respond:

405 Method Not Allowed

The resource exists.

The problem is the HTTP method being used.

Think:

«"You're knocking on the right door, but you're using the wrong way to ask."»

429 Too Many Requests

This one becomes particularly important when dealing with APIs.

429 Too Many Requests

means the client has sent too many requests in a given period.

For example, an API might allow:

100 requests per minute

If a client sends 500 requests in a minute, the server might respond:

429 Too Many Requests

This is commonly associated with rate limiting.

It helps protect services from:

accidental request floods
abusive clients
poorly designed applications
certain automated attacks

This is also one reason rate limiting is an important backend concept.

500 Internal Server Error

Now we've reached the "5xx" category.

500 Internal Server Error

This generally means:

«Something went wrong while the server was processing the request.»

For example, your application might encounter an unexpected error:

Request
↓
Go API
↓
Database query
↓
Unexpected failure
↓
500

The important thing to understand is that "500" isn't supposed to mean:

«"The user did something wrong."»

It's a server-side failure.

As a developer, seeing lots of "500" responses should make you start checking:

application logs
database connections
dependencies
configuration
unexpected exceptions
recent deployments

502 Bad Gateway

This one becomes particularly interesting when you have reverse proxies.

Having this architecture in mind:

Client
↓
Nginx(reverse proxy)
↓
Backend

What happens if Nginx tries to communicate with your backend and receives an invalid response?

You might see:

502 Bad Gateway

Think:

«"The server acting as a gateway/proxy received a bad response from another server."»

This is one reason you'll sometimes see:

502 Bad Gateway

when an application behind Nginx is down or misconfigured.

503 Service Unavailable

means the server is currently unable to handle the request.

Possible reasons include:

server overload
maintenance
temporary unavailability
unavailable dependencies

For example:

Load Balancer
│
├──► Server A ✓
├──► Server B ✗
└──► Server C ✗

If there aren't enough healthy servers to handle requests, a service might return:

503 Service Unavailable

It's often a signal that the problem may be temporary.

504 Gateway Timeout

And finally:

504 Gateway Timeout

This usually means a gateway or proxy waited too long for another server to respond.

For example:

Client
↓
Nginx
↓
Backend
↓
Database
↓
... taking too long ...

Eventually:

504 Gateway Timeout

This can be a useful clue when debugging slow services.

A Simple Cheat Sheet

You don't need to memorize everything.

Start with these:

One More Important Thing

A status code is not just something your framework generates.

As a backend developer, you choose appropriate status codes.

For example, imagine you're writing a Go API:

if user == nil {
http.Error(w, "User not found", http.StatusNotFound)
return
}

You're communicating something important to the client:

The server is working.
The request was understood.
But the requested resource doesn't exist.

That's much more useful than returning:

500 Internal Server Error

for every possible problem.

Good APIs communicate clearly.

Final Takeaway

HTTP status codes are essentially a language between clients and servers.

You don't need to memorize every status code ever created.

Start by understanding the categories:

1xx → Information
2xx → Success
3xx → Redirection
4xx → Client problem
5xx → Server problem

Then remember the most useful ones:

200 → OK
201 → Created
204 → No Content

400 → Bad Request
401 → Authentication required
403 → Forbidden
404 → Not Found
405 → Method Not Allowed
429 → Too Many Requests

500 → Server Error
502 → Bad Gateway
503 → Service Unavailable
504 → Gateway Timeout

The next time you see:

HTTP/1.1 404 Not Found

don't just think "something went wrong."

Ask:

«Who is responsible for this response, what does it communicate, and what should happen next?»

That's when HTTP status codes stop being numbers and start becoming useful tools for building and debugging backend systems.

Load Balancing: How Backend Systems Handle Millions of Requests

Juma Evans — Wed, 26 Aug 2026 16:16:48 +0000

Imagine you have built a backend API that works perfectly.

You deploy it to a server, connect your database, and everything is running smoothly.

Then your application becomes popular.

Instead of 100 requests per minute, you're suddenly handling 10,000. Then 100,000.

Your single server now has too much work to handle.

It becomes slow.

Eventually, it crashes.

So what do you do?

One solution is to make the server more powerful. But there is a limit to how much you can scale a single machine.

A more practical approach is to run multiple backend servers and distribute incoming traffic between them.

This is where load balancing comes in.

In this article, we'll explore what load balancing is, why it matters, how different algorithms work, the difference between Layer 4 and Layer 7 load balancing, health checks, sticky sessions, and how load balancing fits into a real-world backend architecture.

What Is Load Balancing?

Load balancing is the process of distributing incoming network traffic across multiple servers.

The system responsible for doing this is called a load balancer.

Instead of clients communicating directly with one backend server, they communicate with the load balancer.

The load balancer then decides which backend server should handle each request.

A simplified architecture looks like this:

                Clients
                   │
                   ▼
            ┌──────────────┐
            │ Load Balancer│
            └──────┬───────┘
                   │
          ┌────────┼────────┐
          │        │        │
          ▼        ▼        ▼
      ┌──────┐ ┌──────┐ ┌──────┐
      │ API 1│ │ API 2│ │ API 3│
      └──────┘ └──────┘ └──────┘

The clients don't necessarily need to know that three backend servers exist.

From their perspective, they're communicating with one application.

The load balancer handles the distribution behind the scenes.

Why Do We Need Load Balancing?

The biggest reason is scalability.

Suppose your application initially has one server:

        Users
          │
          ▼
      ┌─────────┐
      │ Server 1│
      └─────────┘

This might work perfectly when you have a small number of users.

But as traffic increases:

        Thousands of Users
                │
                ▼
          ┌─────────┐
          │ Server 1│
          └─────────┘
                │
              💥

The server eventually becomes a bottleneck.

You could upgrade the server:

2 CPU → 8 CPU
8 GB RAM → 32 GB RAM

This is known as vertical scaling.

But vertical scaling has physical and financial limits.

Eventually, you may need a different approach:

                Users
                  │
                  ▼
           Load Balancer
                  │
       ┌──────────┼──────────┐
       ▼          ▼          ▼
    Server 1   Server 2   Server 3

This is horizontal scaling.

Instead of making one machine significantly more powerful, you add more machines.

Vertical Scaling vs Horizontal Scaling

There are two common ways to scale a system.

Vertical Scaling

Vertical scaling means making an existing server more powerful.

For example:

Before:

4 CPU
8 GB RAM

        ↓

After:

16 CPU
64 GB RAM

The advantage is simplicity.

You don't necessarily need to change your application architecture.

However, the machine has a physical limit.

There is also another problem.

If that server goes down, your entire application goes down.

Horizontal Scaling

Horizontal scaling means adding more servers.

For example:

Before:

          Server
            │
            ▼
         API App


After:

             Load Balancer
                  │
        ┌─────────┼─────────┐
        ▼         ▼         ▼
     API 1      API 2      API 3

Now traffic can be distributed across multiple machines.

If one server fails, the others can continue serving requests.

This gives us both scalability and availability.

How Does a Load Balancer Work?

At a high level, the process looks like this:

1. Client sends request
          │
          ▼
2. Load balancer receives request
          │
          ▼
3. Load balancer selects backend
          │
          ▼
4. Backend processes request
          │
          ▼
5. Response goes back to client

Suppose a user requests:

GET /api/users

Instead of going directly to:

api-server-1

the request first reaches the load balancer.

The load balancer might decide:

Server 1 already has many active connections. I'll send this request to Server 2.

So the request becomes:

Client
  │
  │ GET /api/users
  ▼
Load Balancer
  │
  │ forwards request
  ▼
API Server 2

The load balancer is essentially acting as a traffic manager.

Load Balancing Algorithms

The load balancer needs a way to decide:

Which server should receive this request?

There are several strategies for answering that question.

1. Round Robin

Round Robin is one of the simplest approaches.

Requests are distributed sequentially.

Suppose we have three servers:

Server A
Server B
Server C

Requests might be distributed like this:

Request 1 → Server A
Request 2 → Server B
Request 3 → Server C
Request 4 → Server A
Request 5 → Server B
Request 6 → Server C

It basically goes around in a circle.

Advantages

Simple
Easy to implement
Works well when servers have similar capacity

Disadvantages

It doesn't consider how busy a server currently is.

For example:

Server A → 10 active requests
Server B → 2 active requests
Server C → 1 active request

Round Robin might still send the next request to Server A.

2. Weighted Round Robin

Sometimes servers don't have equal capacity.

For example:

Server A → 16 CPU
Server B → 8 CPU
Server C → 4 CPU

Giving each server the same amount of traffic wouldn't necessarily be ideal.

Weighted Round Robin allows us to assign different weights.

For example:

Server A → weight 3
Server B → weight 2
Server C → weight 1

Traffic could approximately look like:

A → A → A
B → B
C

The more powerful server receives more traffic.

This is useful when backend instances have different capacities.

3. Least Connections

Instead of simply counting requests sequentially, the load balancer looks at the number of active connections.

Suppose:

Server A → 20 connections
Server B → 8 connections
Server C → 3 connections

The next request would probably go to Server C.

New Request
     │
     ▼
Least Connections
     │
     ▼
Server C

This can be useful when requests take different amounts of time to complete.

4. IP Hash

With IP Hash, the load balancer uses the client's IP address to determine which backend server receives the request.

Conceptually:

Client IP
   │
   ▼
Hash Function
   │
   ▼
Backend Server

For example:

192.168.1.10 → Server A
192.168.1.20 → Server B
192.168.1.30 → Server C

The goal is often to make requests from the same client consistently reach the same server.

This can be useful in some session-based architectures.

However, it also has limitations.

If the distribution of client IPs is uneven, traffic may become unevenly distributed.

5. Random

Another simple strategy is to randomly select a backend server.

For example:

Request 1 → Server B
Request 2 → Server A
Request 3 → Server A
Request 4 → Server C

Random selection can work surprisingly well with a large number of requests, although more sophisticated algorithms are often preferable when the system needs better control.

Layer 4 vs Layer 7 Load Balancing

This is one of the most important concepts when learning load balancing.

Load balancers can operate at different layers of the network stack.

Two common approaches are:

Layer 4
Layer 7

Layer 4 Load Balancing

Layer 4 operates at the transport layer.

It primarily works with protocols such as:

The load balancer doesn't necessarily need to understand the contents of an HTTP request.

It can make decisions based on information such as:

Source IP
Destination IP
Source Port
Destination Port
Protocol

For example:

Client
  │
  │ TCP connection
  ▼
Layer 4 Load Balancer
  │
  ├──────► Server A
  │
  └──────► Server B

Because it operates at a lower level, Layer 4 load balancing can be fast and efficient.

Layer 7 Load Balancing

Layer 7 operates at the application layer.

For web applications, this usually means understanding HTTP or HTTPS.

Now the load balancer can inspect things such as:

HTTP Method
URL
Headers
Cookies
Host

For example:

GET /api/users

could be routed to one group of servers:

/api/* → API Servers

while:

GET /images/logo.png

could be routed somewhere else:

/images/* → Static Content Servers

This gives Layer 7 load balancers much more control over routing.

Layer 4 vs Layer 7: Simple Comparison

Feature	Layer 4	Layer 7
Network layer	Transport	Application
Common protocols	TCP, UDP	HTTP, HTTPS
Understands HTTP	No	Yes
Can inspect URL	No	Yes
Can inspect headers	No	Yes
Routing flexibility	Lower	Higher
Typical use	Network-level traffic	Application-aware routing

Neither is universally better.

The right choice depends on the architecture and requirements of the system.

Health Checks

Imagine you have three servers:

Server A → Healthy
Server B → Healthy
Server C → Crashed

What happens if the load balancer continues sending requests to Server C?

Users will receive errors.

This is why load balancers commonly use health checks.

The load balancer periodically checks whether backend servers are healthy.

For example:

GET /health

The server might respond:

HTTP/1.1 200 OK

The load balancer interprets that as:

This server is healthy.

But if the server repeatedly fails:

Connection refused
Timeout
HTTP 500

the load balancer can temporarily remove it from the pool.

Now traffic becomes:

             Load Balancer
                  │
          ┌───────┴───────┐
          ▼               ▼
       Server A         Server B

       Server C
       ❌ Unhealthy

This is one of the mechanisms that makes load balancing useful for fault tolerance.

Active vs Passive Health Checks

There are different approaches to detecting failures.

Active Health Checks

The load balancer actively sends requests to the server.

For example:

GET /health

If the server responds correctly, it remains available.

If it repeatedly fails, the load balancer removes it.

Passive Health Checks

The load balancer observes actual traffic.

If a backend repeatedly produces failures or connection errors, the load balancer can mark it as unhealthy.

In practice, systems can use a combination of health-check mechanisms.

Sticky Sessions

Here's an interesting problem.

Suppose your application stores user session information directly in server memory.

For example:

Server A
└── Session for User 123

The user's next request might go to Server B.

Request 1 → Server A
Request 2 → Server B

Server B doesn't know about the session stored inside Server A.

The user might suddenly appear logged out.

One solution is sticky sessions.

The load balancer attempts to keep the user connected to the same backend server.

User 123
   │
   ├── Request 1 → Server A
   ├── Request 2 → Server A
   ├── Request 3 → Server A
   └── Request 4 → Server A

This can solve some problems, but it introduces another dependency.

If Server A fails, the user's session may disappear.

Stateless Applications

A more scalable approach is often to make backend servers stateless.

Instead of storing important session state inside one server:

Server A
└── User Session

we can store shared state somewhere accessible to all servers.

For example:

               Load Balancer
                    │
          ┌─────────┼─────────┐
          ▼         ▼         ▼
       Server A  Server B  Server C
          │         │         │
          └─────────┼─────────┘
                    ▼
              Shared Storage

This could be a database, cache, or another shared state-management system.

Now any backend server can handle the request.

This makes horizontal scaling much easier.

Load Balancer vs Reverse Proxy

These concepts are closely related but aren't exactly the same.

A reverse proxy sits between clients and backend servers.

For example:

Client
  │
  ▼
Reverse Proxy
  │
  ▼
Backend

A reverse proxy can perform tasks such as:

TLS termination
Request routing
Compression
Caching
Security filtering
Header manipulation

A load balancer focuses specifically on distributing traffic across multiple backend instances.

However, one piece of software can perform both roles.

For example, Nginx can act as a reverse proxy and load balancer.

A Real-World Backend Architecture

Let's put everything together.

A production application might look something like this:

                         Internet
                            │
                            ▼
                     ┌─────────────┐
                     │    DNS      │
                     └──────┬──────┘
                            │
                            ▼
                     ┌─────────────┐
                     │Load Balancer│
                     └──────┬──────┘
                            │
             ┌──────────────┼──────────────┐
             │              │              │
             ▼              ▼              ▼
         ┌────────┐     ┌────────┐     ┌────────┐
         │ API 1  │     │ API 2  │     │ API 3  │
         └───┬────┘     └───┬────┘     └───┬────┘
             │              │              │
             └──────────────┼──────────────┘
                            │
                  ┌─────────┴─────────┐
                  ▼                   ▼
             ┌─────────┐         ┌─────────┐
             │ Database│         │  Redis  │
             └─────────┘         └─────────┘

A request might travel through the system like this:

User
 │
 │ HTTPS request
 ▼
DNS
 │
 ▼
Load Balancer
 │
 ▼
API Server
 │
 ├──► Redis
 │
 └──► Database
 │
 ▼
Response
 │
 ▼
User

Each component has a specific responsibility.

DNS helps the client find the service.

The load balancer distributes traffic.

The backend handles business logic.

Redis can provide fast access to cached or shared data.

The database provides persistent storage.

This separation allows each part of the system to scale independently.

What Happens When a Server Fails?

Let's say we have:

Server A → Healthy
Server B → Healthy
Server C → Healthy

Then Server B crashes.

Without load balancing:

Users
  │
  ▼
Server B
  ❌

Requests fail.

With a load balancer:

                 Load Balancer
                /             \
               ▼               ▼
           Server A         Server C
            Healthy          Healthy

           Server B
             ❌

The load balancer detects that Server B is unhealthy and stops sending new traffic to it.

The application can continue operating using the remaining servers.

This is the difference between:

"One server is down."

and:

"The entire application is down."

That distinction is extremely important in highly available systems.

Does a Load Balancer Eliminate Downtime?

No.

A load balancer improves availability, but it doesn't magically eliminate every failure.

For example, the load balancer itself can become a single point of failure.

If your architecture looks like:

Users
  │
  ▼
One Load Balancer
  │
  ├── Server A
  ├── Server B
  └── Server C

what happens if the load balancer crashes?

Everything behind it becomes unreachable.

This is why production systems often use redundant load balancers or managed load-balancing services.

The architecture might look more like:

                 Users
                   │
          ┌────────┴────────┐
          ▼                 ▼
     Load Balancer 1   Load Balancer 2
          │                 │
          └────────┬────────┘
                   │
          ┌────────┼────────┐
          ▼        ▼        ▼
       Server A Server B Server C

Now the load-balancing layer itself has redundancy.

Common Load Balancing Mistakes

Understanding load balancing also means understanding what it doesn't solve.

1. Adding More Servers Doesn't Fix Everything

If your database is the bottleneck:

Load Balancer
     │
 ┌───┼───┐
 ▼   ▼   ▼
API API API
 \   |   /
  \  |  /
 Database
    💥

Adding more API servers won't necessarily solve the problem.

The database might still be overloaded.

2. Ignoring Health Checks

If unhealthy servers continue receiving traffic, the load balancer becomes part of the problem rather than the solution.

3. Poor Session Management

Sticky sessions can hide architectural problems.

If your application depends heavily on one specific backend server, scaling becomes more difficult.

4. Ignoring Connection Limits

Every backend server has limits.

CPU, memory, network connections, database connections, file descriptors, and other resources can become bottlenecks.

A load balancer doesn't remove those limits.

It simply distributes traffic.

Load Balancing and Backend Engineering

As a backend developer, you don't necessarily need to build a load balancer from scratch.

But you should understand what happens around your API.

For example, suppose you build a Go API:

GET /api/users
POST /api/orders
GET /api/products

Initially, you might deploy one instance:

Client
  │
  ▼
Go API

Later, your application grows:

Client
  │
  ▼
Load Balancer
  │
  ├── Go API #1
  ├── Go API #2
  └── Go API #3

Now your Go application needs to behave correctly when multiple instances are running.

This means thinking about:

Shared state
Database connections
Authentication
Sessions
Caching
Idempotency
Concurrency
Timeouts
Retries
Graceful shutdown
Health endpoints

This is where load balancing becomes more than just a networking concept.

It starts influencing how you design backend applications.

A Simple Mental Model

When learning load balancing, remember this:

              MANY CLIENTS
                   │
                   ▼
             LOAD BALANCER
                   │
        ┌──────────┼──────────┐
        ▼          ▼          ▼
     SERVER A   SERVER B   SERVER C
        │          │          │
        └──────────┼──────────┘
                   │
                   ▼
              SHARED DATA

The load balancer answers one fundamental question:

"Which backend should handle this request?"

The answer can depend on:

Round Robin
Server weight
Active connections
Client identity
Request information
Server health

Once you understand that, the rest of the topic becomes much easier.

Final Thoughts

Load balancing is one of the fundamental building blocks of scalable backend systems.

When an application is small, one server may be enough.

But as traffic grows, relying on a single server creates bottlenecks and increases the impact of failures.

Load balancing allows us to distribute traffic across multiple backend instances while improving scalability, availability, and fault tolerance.

The important thing isn't just memorizing algorithms like Round Robin or Least Connections.

It's understanding why load balancing exists in the first place.

A scalable backend is rarely just:

Client → Server

Instead, it increasingly becomes:

Client
   │
   ▼
DNS
   │
   ▼
Load Balancer
   │
   ├──────► Backend 1
   ├──────► Backend 2
   └──────► Backend 3
              │
              ▼
        Database / Cache

And this is an important shift in thinking for backend engineers:

You're no longer just writing code that works on one machine. You're designing systems that continue working when traffic, users, and failures increase.

That is where concepts like load balancing start to matter.

Caching: How Backend Systems Get Faster with Redis

Juma Evans — Sat, 08 Aug 2026 21:41:39 +0000

Analogy

Imagine you have built a backend API that works perfectly.

You deploy it. Users start coming in. At first, everything feels fast.

Then the traffic grows.

Suddenly, your API is receiving thousands of requests every minute. Many of those requests are asking for the same data:

The same user profile
The same product information
The same popular posts
The same configuration
The same exchange rates
The same dashboard statistics

Your application keeps sending the same queries to PostgreSQL over and over again.

The database starts working harder.

Response times increase.

Eventually, your application may become slow or even unavailable.

So, how do large-scale backend systems avoid doing the same expensive work repeatedly?

Caching.

Caching is one of the most important concepts a backend developer should understand because it sits at the intersection of performance, databases, APIs, scalability, and system design.

In this article, we'll understand how caching works, where Redis fits into the picture, how to implement a basic caching strategy in Go, and some of the problems that caching introduces.

What Exactly Is Caching?

A cache is a temporary storage location for data that is expensive or time-consuming to retrieve.

The basic idea is simple:

If we already calculated or retrieved something recently, why do the same work again?

Consider a simple API:

Client
   |
   v
API Server
   |
   v
PostgreSQL

Suppose a client requests:

GET /users/42

The backend might execute:

SELECT * FROM users WHERE id = 42;

PostgreSQL finds the user and returns the data.

Nothing is wrong with this.

But imagine that 10,000 users request the same resource within a short period.

Your backend could potentially execute the same database query thousands of times.

Instead, we can introduce a cache:

Client
   |
   v
API Server
   |
   v
Cache
   |
   |-- Cache Hit --> Return data
   |
   |-- Cache Miss
          |
          v
      PostgreSQL
          |
          v
        Cache
          |
          v
      API Server

The first request retrieves the data from PostgreSQL.

The backend then stores the result in the cache.

The next request can retrieve the data directly from the cache instead of querying PostgreSQL again.

That is the fundamental idea behind caching.

Cache Hits and Cache Misses

Two terms appear constantly when working with caches:

Cache hit and cache miss.

Cache hit

A cache hit occurs when the requested data already exists in the cache.

Request
   |
   v
Cache
   |
   |--- Found!
   |
   v
Return data

This is the ideal scenario.

The application avoids an expensive database operation.

Cache miss

A cache miss occurs when the requested data is not in the cache.

Request
   |
   v
Cache
   |
   |--- Not found
   |
   v
Database

The application retrieves the data from the database and can then store it in the cache for future requests.

A simplified flow looks like this:

Request
   |
   v
Check Cache
   |
   +------ Found ------> Return cached data
   |
   +------ Not Found --> Query Database
                           |
                           v
                       Store in Cache
                           |
                           v
                       Return data

This pattern is extremely common in backend systems.

Why Not Just Use the Database?

A reasonable question is:

"If PostgreSQL already stores the data, why do we need another system?"

Because databases are designed primarily for durable storage and querying, while caches are designed for fast access.

A database might need to:

parse a query
find the appropriate table
use indexes
read data
perform filtering
manage transactions
handle concurrency

A cache can often retrieve a value directly using a key.

For example:

user:42

can map directly to:

{
  "id": 42,
  "name": "Evans",
  "role": "developer"
}

The cache doesn't need to perform a complex SQL query.

This is why caching can dramatically reduce database load and improve response times.

But there is an important trade-off:

A cache is usually not the source of truth.

The database normally remains the authoritative source of the data.

The cache is there to make frequently accessed data faster to retrieve.

Where Does Redis Come In?

This is where Redis becomes useful.

Redis is an in-memory data store that is commonly used for:

caching
session storage
counters
rate limiting
queues
temporary data
distributed locks
pub/sub

For caching, Redis is particularly useful because data is kept in memory, making access extremely fast.

Instead of asking PostgreSQL:

SELECT * FROM products WHERE id = 100;

your application might ask Redis:

GET product:100

If the value exists, Redis returns it.

The application can then send the response without contacting PostgreSQL.

The Cache-Aside Pattern

One of the most common caching strategies is called cache-aside.

The application is responsible for checking the cache and loading data into it when necessary.

The process looks like this:

1. Client requests data
          |
          v
2. Application checks Redis
          |
       +--+--+
       |     |
      Hit   Miss
       |     |
       |     v
       |   Query DB
       |     |
       |     v
       |   Store in Redis
       |     |
       +-----+
          |
          v
     Return response

Let's walk through an example.

The client requests:

GET /products/100

The application generates a Redis key:

product:100

It checks Redis.

If Redis contains the value:

product:100 -> {"id":100,"name":"Laptop"}

the application immediately returns it.

If Redis doesn't contain it, the application queries PostgreSQL.

After retrieving the product, it stores the result:

product:100 -> {"id":100,"name":"Laptop"}

Now future requests can use Redis.

This pattern is simple, powerful, and widely applicable.

What Is TTL?

One of the biggest problems with caching is that cached data can become outdated.

Imagine a user changes their profile name.

PostgreSQL contains:

Evans Juma

but Redis still contains:

Evans

If the application keeps returning the cached value forever, users may receive stale information.

This is where TTL, or Time To Live, comes in.

TTL defines how long a cached value should remain available.

For example:

product:100
TTL = 60 seconds

After 60 seconds, Redis can automatically remove the key.

The next request becomes a cache miss:

Redis
  |
  |-- expired
  |
  v
PostgreSQL

The application retrieves the latest value and puts it back into Redis.

TTL is useful because it provides a balance between:

Performance

and

Freshness

For example:

Data	Possible TTL
User profile	5–15 minutes
Product information	5–30 minutes
News feed	Seconds–minutes
Configuration	Minutes–hours
Exchange rates	Minutes
Static reference data	Hours

These are examples, not universal rules.

The correct TTL depends on how frequently the underlying data changes and how stale the application can tolerate the data becoming.

A Simple Go + Redis Example

Let's make this more concrete.

Suppose we're building a Go API that retrieves users.

Without caching, the flow might look like:

func GetUser(id int64) (*User, error) {
    return database.GetUser(id)
}

Every request goes directly to the database.

With Redis, the logic becomes:

func GetUser(id int64) (*User, error) {
    key := fmt.Sprintf("user:%d", id)

    cached, err := redisClient.Get(ctx, key).Result()

    if err == nil {
        var user User

        if err := json.Unmarshal([]byte(cached), &user); err == nil {
            return &user, nil
        }
    }

    user, err := database.GetUser(id)
    if err != nil {
        return nil, err
    }

    data, err := json.Marshal(user)
    if err != nil {
        return nil, err
    }

    err = redisClient.Set(
        ctx,
        key,
        data,
        10*time.Minute,
    ).Err()

    if err != nil {
        // Log the cache error, but don't fail
        // the request if the database succeeded.
    }

    return user, nil
}

The important part is understanding the flow:

Check Redis
     |
     +---- Found ----> Return cached user
     |
     +---- Not found
             |
             v
        Query database
             |
             v
        Store in Redis
             |
             v
        Return user

This is the kind of pattern you'll encounter when building production APIs.

The Hardest Problem: Cache Invalidation

There's a famous saying in software engineering:

"There are only two hard things in Computer Science: cache invalidation and naming things."

The joke exists for a reason.

Suppose we have:

Database:
username = "Evans"

Redis:
username = "Evans"

A user changes their username to:

"Evans Juma"

The database is updated.

But what about Redis?

If we don't update or remove the cached value, Redis might continue returning:

"Evans"

even though the database contains:

"Evans Juma"

One solution is to delete the cached value when the database changes.

UPDATE DATABASE
      |
      v
DELETE CACHE

The next request produces a cache miss, retrieves the latest value from the database, and repopulates the cache.

Another strategy is to update both the database and cache.

The correct approach depends on the application's consistency requirements.

Caching Isn't Always Good

Caching sounds like a magic solution.

It isn't.

Caching introduces another layer of complexity.

You now have:

Application
     |
   Redis
     |
 PostgreSQL

instead of:

Application
     |
 PostgreSQL

That means you now need to think about:

stale data
cache invalidation
cache expiration
memory usage
cache failures
serialization
cache consistency
monitoring

And there is another important problem.

What happens when the cache goes down?

Your application shouldn't necessarily become completely unavailable just because Redis is unavailable.

A well-designed system can treat Redis as an optimization layer.

For example:

Request
   |
   v
Redis
   |
   X Redis unavailable
   |
   v
Database
   |
   v
Response

The request might be slower, but the system can still function.

This is an important backend design principle:

A cache should not automatically become a single point of failure for your application.

The Cache Stampede Problem

There's another interesting problem called a cache stampede.

Imagine a popular cache entry expires:

product:100
TTL expired

Now thousands of requests arrive at almost exactly the same time.

Every request checks Redis:

MISS
MISS
MISS
MISS
MISS
...

All of them then hit PostgreSQL.

Instead of reducing database traffic, your cache expiration has suddenly created a huge spike in database traffic.

This can be dangerous.

Solutions include techniques such as:

locking
request coalescing
staggered expiration
background refresh
adding jitter to TTLs

The important thing is to recognize that caching creates new system-design problems that you have to solve.

When Should You Use Caching?

Caching is particularly useful when:

1. Data is read frequently

If thousands of requests repeatedly access the same data, caching can be extremely valuable.

2. Data is expensive to calculate

For example, an API might perform an expensive aggregation:

SELECT ...
GROUP BY ...

Caching the result can avoid repeating that computation.

3. Data doesn't change frequently

If data changes every millisecond, caching becomes more complicated.

If it changes every few hours, caching becomes much easier.

4. Low latency matters

Applications such as:

social networks
e-commerce platforms
financial dashboards
gaming systems
recommendation systems

often benefit heavily from caching.

When Should You Avoid Caching?

Not everything needs to be cached.

Avoid blindly caching data simply because Redis is available.

Caching might be unnecessary when:

the data is rarely requested
the database query is already extremely cheap
the data changes constantly
stale data would cause serious problems
the added complexity isn't worth the performance gain

The goal isn't:

"Cache everything."

The goal is:

Cache the right things.

A Production-Style Architecture

A more realistic backend might look like this:

                    ┌──────────────┐
                    │    Client    │
                    └──────┬───────┘
                           │
                           ▼
                    ┌──────────────┐
                    │ API Server   │
                    └──────┬───────┘
                           │
                           ▼
                    ┌──────────────┐
                    │    Redis     │
                    │    Cache     │
                    └──────┬───────┘
                           │
                    Cache Miss
                           │
                           ▼
                    ┌──────────────┐
                    │  PostgreSQL  │
                    │   Database   │
                    └──────────────┘

With more traffic, the architecture can evolve further:

                 ┌─────────────┐
                 │   Clients   │
                 └──────┬──────┘
                        │
                        ▼
                ┌───────────────┐
                │ Load Balancer │
                └───────┬───────┘
                        │
             ┌──────────┼──────────┐
             ▼          ▼          ▼
          API #1      API #2      API #3
             │          │          │
             └──────────┼──────────┘
                        │
                        ▼
                  ┌───────────┐
                  │   Redis   │
                  └─────┬─────┘
                        │
                        ▼
                  ┌───────────┐
                  │ PostgreSQL│
                  └───────────┘

Now we have a system where multiple API servers can share the same cache.

This is particularly useful when your backend is horizontally scaled.

The Bigger Lesson

Caching isn't just about making an API faster.

It teaches an important lesson about backend engineering:

Every system has a cost.

A database query costs CPU, memory, disk I/O, connections, and time.

If you repeatedly perform the same expensive operation, you're wasting resources.

Caching allows us to trade some memory and complexity for:

lower latency
fewer database queries
higher throughput
better scalability

But that trade-off comes with responsibilities.

You have to think about:

What should we cache?
        ↓
How long should we cache it?
        ↓
When should it expire?
        ↓
What happens when the data changes?
        ↓
What happens when Redis fails?
        ↓
What happens when thousands of requests miss simultaneously?

These are no longer just programming questions.

They're system design questions.

Final Thoughts

Caching is one of those backend concepts that looks incredibly simple at first.

Store data somewhere faster.

Retrieve it later.

Done.

But once you start building real systems, you discover that caching is much deeper than that.

You have to understand cache hits, cache misses, TTLs, invalidation, consistency, failure handling, cache stampedes, and memory management.

Redis makes implementing many caching strategies relatively straightforward, but Redis itself isn't the solution to every performance problem.

Before introducing a cache, understand what is actually slow.

Measure your application.

Look at database queries.

Check response times.

Find bottlenecks.

Then decide whether caching is the right tool.

Because good backend engineering isn't about adding more technology.

It's about understanding the problem well enough to know when a technology is actually necessary.

And once your application starts receiving thousands or millions of requests, that difference between:

"Query the database every time."

and:

"Query the database only when necessary."

can become the difference between a backend that struggles under load and one that scales.

Key Takeaways

Caching stores frequently accessed data closer to the application.
Cache hits avoid expensive database operations.
Cache misses retrieve data from the original source.
Redis is a popular in-memory data store used for caching.
Cache-aside is one of the most common caching patterns.
TTL prevents cached data from living forever.
Cache invalidation is one of the hardest parts of caching.
Cache stampedes can cause sudden database overload.
Redis failures should ideally not bring down the entire application.
Not every piece of data needs to be cached.
Caching is a performance optimization, not a replacement for your database.

The best cache is not the one that stores the most data. It's the one that eliminates the right work.

JWT Authentication: A Backend Engineer's Mental Model

Juma Evans — Mon, 03 Aug 2026 12:56:51 +0000

Introduction

Imagine you arrive at a hotel.

At the reception, you show your ID and prove who you are. The receptionist then gives you a room key card.

You don't need to show your ID every time you enter your room. Instead, you simply present the key card.

The hotel doesn't need to ask your name again because the card itself proves that you already authenticated.

JWT (JSON Web Token) works exactly like that.

Username and password = Your ID
JWT = Hotel key card
Server = Receptionist

What is JWT?

JWT stands for JSON Web Token.

It is a compact string that proves a user has already logged in successfully.

Instead of storing login sessions on the server, the server gives the client a signed token.

The client sends this token with every request.

Example:

Authorization: Bearer eyJhbGciOiJIUzI1NiIs...

The server verifies the token and allows access.

Why Do We Need JWT?

Without JWT, every request would require sending the username and password repeatedly.

Browser
   |
Username
Password
   |
Server

That would be inefficient and insecure.

Instead:

Login once

↓

Receive JWT

↓

Reuse JWT for every request

Stateless Authentication

JWT enables stateless authentication.

Stateful Authentication

Server
|
|-- Session #12345
|-- Session #91821
|-- Session #44211

The server stores every user's session.

Stateless Authentication (JWT)

Server

(No session storage)

↓

Only verifies token signature

The server doesn't remember users.

The token remembers.

JWT Structure

A JWT consists of three parts separated by periods.

Header.Payload.Signature

Example

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9
.
eyJzdWIiOiIxMjMiLCJuYW1lIjoiRXZhbnMiLCJyb2xlIjoiYWRtaW4ifQ
.
K6L6GQX....

Think of it like

Envelope
Letter
Wax Seal

Part 1 — Header

Example

{
  "alg": "HS256",
  "typ": "JWT"
}

The header tells us:

Which algorithm signed the token.
What type of token it is.

Fields:

alg → Signing algorithm
typ → JWT

Common algorithms:

HS256
RS256
ES256

Part 2 — Payload

The payload contains claims.

Example:

{
    "user_id": 42,
    "name": "Evans",
    "role": "admin"
}

Think of it as your digital identity card.

Typical information:

User ID
Username
Email
Role
Permissions

Standard Claims

Claim	Meaning
sub	Subject (User ID)
exp	Expiration Time
iat	Issued At
iss	Issuer
aud	Audience

Example

{
    "sub":"42",
    "role":"admin",
    "exp":1754440000
}

Important

The payload is NOT encrypted.

Anyone can decode it.

JWT

↓

Base64URL Decode

↓

Payload

Never store:

Passwords
PINs
Secret Keys
API Keys
Credit Card Numbers

inside a JWT.

Part 3 — Signature

The signature protects the token from tampering.

The server computes something similar to:

HMACSHA256(

Base64(Header)

+

Base64(Payload),

SecretKey

)

This produces the signature.

The secret key never leaves the server.

Why the Signature Matters

Suppose someone changes:

{
    "role":"user"
}

{
    "role":"admin"
}

The payload changes.

Therefore the signature changes.

Since the attacker doesn't know the server's secret key, they cannot generate a valid signature.

Result:

Server

↓

Verify Signature

↓

Invalid

↓

401 Unauthorized

This is why JWTs are tamper-evident.

Login Flow

User

↓

POST /login

↓

Username

Password

↓

Server

Server:

Checks the database.
Verifies the password.
Generates a JWT.
Returns it.

Example response:

{
    "token":"eyJhbGc..."
}

The client stores the token.

Authenticated Request

GET /profile
Authorization: Bearer eyJhbGc...

Server:

Reads the Authorization header.
Verifies the signature.
Checks expiration.
Extracts the user ID.
Returns the requested resource.

Complete Request Lifecycle

Client

↓

Login

↓

Receive JWT

↓

Store JWT

↓

Send JWT

↓

Server verifies

↓

Access granted

Where Should Tokens Be Stored?

Browser

Preferred:

Secure HttpOnly Cookies
Memory (for SPAs)

Avoid storing long-lived access tokens in localStorage because XSS attacks can expose them.

Mobile

Use secure platform storage:

iOS Keychain
Android Keystore

Token Expiration

Example

{
    "exp":1754440000
}

Server checks:

Current Time

↓

Expired?

↓

Yes

↓

401 Unauthorized

Expired tokens cannot be used.

Refresh Tokens

Access tokens should have short lifetimes.

Example:

Access Token

15 Minutes

When expired:

Client

↓

Refresh Token

↓

Server

↓

New Access Token

Refresh tokens allow users to remain logged in without entering credentials repeatedly.

JWT vs Sessions

JWT	Sessions
Stateless	Stateful
No server-side session storage	Server stores sessions
Easy to scale	More difficult to scale
Great for APIs	Great for traditional web applications
Client sends token	Client sends session cookie

JWT in a Go (Gin) Backend

Login Endpoint

POST /login

↓

Validate Request

↓

Find User

↓

Compare Password Hash

↓

Generate JWT

↓

Return Token

Protected Route

GET /users

↓

JWT Middleware

↓

Read Authorization Header

↓

Verify Signature

↓

Check Expiration

↓

Extract Claims

↓

Next()

↓

Handler Executes

Middleware Flow

Incoming Request
       │
       ▼
Read Authorization Header
       │
       ▼
Token Present?
   │        │
  No       Yes
   │        ▼
401      Verify Signature
             │
      Valid? │
        │    │
       No   Yes
        │    ▼
      401  Check Expiration
               │
        Expired? │
          │      │
         Yes    No
          │      ▼
        401   Extract Claims
                   │
                   ▼
      Store User in Context
                   │
                   ▼
          Execute Next Handler

Authentication vs Authorization

Authentication answers:

Who are you?

Authorization answers:

What are you allowed to do?

Example:

Login

↓

Authentication

↓

JWT

↓

Authorization

↓

Access Granted

Common Interview Questions

What is JWT?

A signed JSON token used for stateless authentication.

Is JWT encrypted?

No.

It is Base64URL encoded and digitally signed, but not encrypted.

Why can't users modify the payload?

Because changing the payload invalidates the signature.

What is the purpose of the signature?

To guarantee integrity and authenticity.

Why does JWT expire?

To reduce the damage if a token is stolen.

What is a Refresh Token?

A long-lived credential used to request a new access token after the current one expires.

What happens if the signature is invalid?

The server rejects the request with 401 Unauthorized.

Why is JWT called stateless?

Because the server does not store user sessions.

Every request contains all the information needed to authenticate the user.

Best Practices

Always use HTTPS.
Never store passwords inside JWTs.
Keep access tokens short-lived (10–30 minutes).
Use refresh tokens for long-lived sessions.
Store refresh tokens securely.
Validate the signature on every request.
Validate the expiration (exp) claim.
Use strong signing algorithms.
Rotate signing keys when appropriate.
Implement token revocation if your application requires immediate logout or compromised-token handling.

Key Takeaways

JWT stands for JSON Web Token.
A JWT has three parts:
- Header
- Payload
- Signature
The payload is readable by anyone who has the token.
The signature protects against tampering.
JWT enables stateless authentication.
Access tokens should expire.
Refresh tokens provide a secure way to obtain new access tokens.
JWT authentication is commonly implemented using middleware in Go (Gin), Express.js, Spring Boot, ASP.NET, and many other backend frameworks.

Final Mental Model

Think of JWT like a hotel key card.

User
 │
 │ Login (Username + Password)
 ▼
Server verifies credentials
 │
 ▼
Issues a signed JWT
 │
 ▼
Client stores the token
 │
 ▼
Client sends the token with every request
 │
 ▼
Server verifies:
    ✔ Signature
    ✔ Expiration
    ✔ Claims
 │
 ▼
Access Granted

The server does not need to remember the user.

The signed token carries the identity, while the server only needs its secret (or public key, depending on the algorithm) to verify that the token is authentic and has not been altered.

Your Code Doesn't Run. A Translation of Your Code Does.

Juma Evans — Sat, 01 Aug 2026 20:24:28 +0000

Every time you write a program and hit run, something extraordinary happens before a single instruction executes. Your source code,the plain text(human-readable), gets transformed through several distinct stages into something a CPU can actually understand.

Most developers never look at those stages. They trust the compiler the way they trust electricity: they know it works, they just don't know how.

This article is about how it works.

By the end, you'll understand every major stage a compiler goes through, what it's doing and why, and you'll never look at a compiler error the same way again.

What Is a Compiler, Really?

A compiler is a program that reads a program written in one language and outputs an equivalent program in another language.

Usually that means: read source code written by a human, output machine code understood by a CPU. But that's the end result of a pipeline, not a single step.

Think of it like translating a novel from Swahili to English. You don't just look at the whole book and magically produce a translation. You read sentence by sentence, understand grammar, resolve meaning, restructure where necessary, then write the output. The compiler does something structurally similar, just with ruthless precision, because machines tolerate zero ambiguity.

The pipeline has roughly five stages:

Lexing :— breaking source code into tokens
Parsing:— building a tree structure from those tokens
Semantic Analysis :— checking that the tree actually makes sense
Intermediate Representation :— translating to a language-neutral form
Code Generation :— producing the final machine code

Let's walk through each one.

Stage 1: Lexing (Also Called Tokenization)

The very first thing a compiler does is read your source file as raw text and break it into tokens, the smallest meaningful units of the language.

Take this line of Go:

go
x := 42 + y

The lexer reads this character by character and groups them into:

Token Type
x Identifier
:= Operator
42 Integer Literal

Operator y Identifier

Whitespace gets discarded. Comments get discarded. What's left is a flat stream of tokens, each labelled with its type.

This is why your compiler doesn't care whether you write x:=42+y or x := 42 + y. Both produce the exact same token stream because the spaces were never meaningful to begin with.

When Lexing Fails

If you type a character the language doesn't recognize say, a @ symbol in Go where it isn't valid the lexer is the one that catches it. That "unexpected character" error you've seen this before? That's the lexer talking.

Stage 2: Parsing — Building the AST

A flat list of tokens isn't enough. 42 + y * 3 means something different from (42 + y) * 3. The parser's job is to take the token stream and build a tree that encodes structure and precedence.

That tree is called an Abstract Syntax Tree, or AST.

For the expression x := 42 + y, the AST looks roughly like:

AssignStatement
├── Left: Identifier("x")
└── Right: BinaryExpression(+)
├── Left: IntLiteral(42)
└── Right: Identifier("y")

Every node in the tree represents a construct in the language. Statements contain expressions. Expressions contain sub-expressions. Functions contain blocks. Blocks contain statements. The whole program becomes one giant tree, with the root at the top and leaves at the bottom.

Why a Tree?

Because code is inherently hierarchical. A function contains statements. Statements contain expressions. Expressions contain sub-expressions. A flat list can't represent that nesting but a tree can.

The parser uses the grammar rules of the language to this tree. Grammar rules are what define what valid code even looks like. If your tokens don't fit the grammar say, you write x := := 42 the parser rejects it. That's a "syntax error." Not a character problem (the lexer passed), but a structure problem.

Stage 3: Semantic Analysis — Does This Actually Make Sense?

A program can be grammatically valid and still be nonsense.

go
var x int = "hello"

That's syntactically fine, it's a valid assignment statement. But it's semantically wrong. You're trying to put a string into an integer variable.

Semantic analysis is where the compiler checks meaning, not just structure. This stage does several things:

Type checking; are the types compatible? Is a function being called with the right argument types? Is a return value the right type?

Scope resolution; when you write y, does y exist? Was it declared before it was used? Is it in scope here?

Name binding; connecting every reference to a variable or function back to its declaration.

Constant folding; if you write 2 + 3, the compiler can compute this at compile time and just treat it as 5. No need to calculate it every time the program runs.

This is the stage that produces errors like "undefined variable", "cannot use string as int", and "function takes 2 arguments, got 3." These aren't syntax errors the structure was fine. The meaning was wrong.

Stage 4: Intermediate Representation (IR)

After semantic analysis, the compiler has a fully validated AST. Now it needs to translate that into something closer to machine code, but not quite machine code yet.

This middle step is called Intermediate Representation, or IR.

Why not go straight to machine code? Because different CPUs speak different machine languages. An x86 chip and an ARM chip have completely different instruction sets. If you went straight from AST to machine code, you'd need a separate compiler backend for every target architecture.

IR solves this by being a neutral, low-level language that isn't tied to any specific CPU. The Go compiler uses its own internal IR. LLVM (used by Clang, Rust, and others) uses a well-known IR called LLVM IR.

IR also enables optimization. Before generating final output, the compiler can analyze the IR and improve it:

Remove code that can never be reached
Eliminate variables that are assigned but never used
Inline small functions to avoid call overhead
Reorder instructions to keep the CPU pipeline busy

These optimizations happen on the IR, not the source code, which is why they work the same regardless of which language you wrote the program in.

Stage 5: Code Generation

The final stage. The compiler takes the (now optimized) IR and translates it into actual machine code for a specific target architecture.

Machine code is just numbers, binary instructions the CPU reads directly. For example, the instruction to move a value into a register on x86-64 might be encoded as 48 89 C3. The code generator knows the instruction set of the target CPU and emits the right bytes.

The output is typically an object file, a binary file containing machine code, but not yet a complete program. Object files get passed to a linker, which combines them with other object files (and libraries) to produce the final executable.

That final executable is what you actually run.

Putting It All Together

Let's trace a simple Go program through the full pipeline:

go
package main

import "fmt"

func main() {
x := 10
y := 20
fmt.Println(x + y)
}

Lexer: Reads the file, produces tokens: package, main, import, "fmt", func, main, (, ), {, x, :=, 10, ...

Parser: Builds an AST. The root is a PackageDeclaration. It has an ImportDeclaration and a FunctionDeclaration. The function body contains two AssignStatements and a CallExpression.

Semantic Analysis: Checks that x and y are declared before use. Checks that x + y produces an int. Checks that fmt.Println accepts an int. Resolves fmt to the imported package.

IR Generation: Translates the AST into a lower-level representation. The addition x + y becomes an ADD instruction. The function call becomes a CALL instruction with arguments set up correctly.
**
Optimization*: In this case, the compiler might even compute 10 + 20 = 30 at compile time and replace the addition with a constant.
*
Code Generation*: Emits x86-64 (or ARM, depending on your machine) instructions. Produces an object file.
*
Linker**: Combines the object file with the fmt package's compiled code. Produces the final binary.

You type go run main.go. In milliseconds, all of that happens. Then your program prints 30.

What This Changes About Reading Errors

Once you understand the pipeline, compiler errors stop being mysterious.

"unexpected token" → the lexer or parser failed. Your code's structure is wrong. Check syntax.

"undefined: x" → semantic analysis failed. You used something that wasn't declared, or declared it in the wrong scope.

"cannot use string as type int" → type checking failed. The types don't match. Semantic analysis caught it.

"x declared and not used" → Go's compiler enforces this at the semantic analysis stage as a hard rule, not a warning. Unused variables indicate bugs or dead code, so Go refuses to compile.

Each error is the compiler telling you exactly which stage it got to before it couldn't continue and what specifically went wrong.

Something worth noting on Interpreted Languages

Not every language uses this full pipeline. Python, JavaScript (in some contexts), and Ruby are traditionally interpreted, instead of compiling to machine code ahead of time, an interpreter reads the source and executes it directly, often line by line.

Interpreters still lex and parse. They build ASTs. But instead of generating machine code, they walk the AST and execute each node directly.

The tradeoff: interpreted languages start faster (no compilation step) but run slower (no ahead-of-time optimization, no direct machine code). That's why Python is convenient for scripting but Go or C are used when performance matters.

Modern JavaScript engines (V8 in Chrome, SpiderMonkey in Firefox) blur this line — they use JIT (Just-In-Time) compilation, which compiles hot code paths to machine code at runtime. It's a hybrid: interpret first, compile the parts that matter.

Why This Matters For Backend Engineers

You might be thinking: I write application code. I'm not building a compiler. Why does this matter?

A few reasons.

You debug faster. When you know what a compiler error is actually telling you, you spend less time guessing and more time fixing.

You write better code. Understanding what the compiler can and can't optimize helps you write code that performs well not through premature optimization, but through avoiding patterns that defeat the optimizer.

You understand your tools. Every build system, linter, code formatter, and static analysis tool is, at its core, doing some version of what a compiler does. Understanding the pipeline makes all of those tools less magical and more predictable.

You grow as an engineer. The engineers who built the systems you use every day databases, runtimes, operating systems understood this. Closing that gap between "I write code" and "I understand what runs my code" is part of what separates junior engineers from senior ones.

The One-Line Summary

Your source code is text. CPUs understand binary. A compiler is the translator that bridges that gap in five careful, well-defined stages, each building on the last.

Lexing breaks text into tokens. Parsing builds structure. Semantic analysis checks meaning. IR enables optimization. Code generation produces the final output.

Every time you compile, that entire process runs. Now you know what it's doing.

This is what building a Two-Node Lightning Network From Scratch looked like in one week.

Juma Evans — Sat, 01 Aug 2026 19:29:09 +0000

There's a version of learning Bitcoin where you read the whitepaper, nod along, and move on. Then there's the version where you're staring at a terminal at 11pm wondering why two nodes refuse to talk to each other then you actually understand the answer when you figure it out.

We did the second version. This article is what came out of it.

By the end of one week, I understood what Bitcoin actually is under the hood, why the Lightning Network exists, what a payment channel really means, and what it looks like to build all of this from zero, in a local test environment.

No hype. No "Bitcoin is the future" takes. Just the engineering.

First: What Is Bitcoin, Actually?

Before Lightning makes any sense, Bitcoin has to make sense. And I mean the technical version, not the investment version.

Bitcoin is a distributed ledger :— a database that no single person owns or controls, replicated across thousands of computers around the world. Every 10 minutes or so, a new "block" of transactions gets added to this ledger. That chain of blocks is the blockchain.

Here's the key property that makes it interesting: once a transaction is written into the chain, it is practically irreversible. There's no customer service desk. There's no refund button. The database is append-only, and changing historical records would require redoing an astronomically expensive amount of computational work.

This is great for finality. It's terrible for speed.

The Problem: Bitcoin Is Slow By Design

Bitcoin's base layer processes roughly 7 transactions per second. Globally. For everyone.

Visa handles tens of thousands per second. M-Pesa handles millions of transactions per day in Kenya alone. Bitcoin, as a base layer, cannot compete on throughput, and that's not a bug, it's a consequence of the tradeoffs that make it trust-less and decentralized.

So what do you do if you want fast, cheap Bitcoin payments? You build on top of it. That's where Lightning comes in.

What Is the Lightning Network?

The Lightning Network is Bitcoin's Layer 2 — a payment network that sits on top of Bitcoin and inherits its security without inheriting its slowness.

The core idea is elegant: what if two people could transact with each other thousands of times, but only touch the blockchain twice?

Once to open the channel. Once to close it.

Everything in between happens off-chain, instantly, with near-zero fees.

The Analogy That Made It Click For Me

Imagine you and a colleague work in the same office and you frequently owe each other small amounts like lunch money, airtime, split bills. Every time you settle, you don't go to the bank. You keep a running tab. "I owe you 200, you owe me 350 net, you owe me 150." At the end of the month, one person pays the other. One transaction.

A Lightning channel is that tab, except it's crypto-graphically enforced, so neither party can cheat. You put real Bitcoin into it when you open it. The channel tracks who owns what. When you're done, you close it and the final balances get written to the blockchain.

Scale this to millions of people with overlapping channels, and you have a network where you can pay anyone, even without a direct channel to them just by routing through intermediate nodes.

What we Actually Built

This was a structured bootcamp environment with a real set of goals:

Compile Bitcoin Core from source (no sudo apt install shortcuts)
Build a Bitcoin Explorer CLI in Go using only the standard library
Set up a two-node Lightning Network with LND in regtest
Fund channels, make payments, and observe what happens

Let's take a tour of each stage.

Part 1: Compiling Bitcoin Core From Source

Most guides tell you to download a binary. We didn't do that. We compiled Bitcoin Core v31.99.0 directly from source code, without sudo privileges.

Why does this matter? Because in production fintech systems, you often can't just trust a binary you downloaded. Compiling from source means you're running exactly what the code says — nothing added, nothing modified.

The process taught me something I hadn't thought about before: software has dependencies on dependencies on dependencies. Getting Bitcoin Core to compile meant first ensuring the right versions of Boost, lib-event, and other libraries were present. The compiler errors are honest and they tell you exactly what's missing.

Once compiled, I ran it in regtest mode.

What Is Regtest?

Regtest (short for regression test) is a local, private Bitcoin network that only exists on your machine. You control everything: the mining, the blocks, the funds. You can mine 1000 blocks in a second. It's basically a sandbox.

This is how you develop and test anything Bitcoin-related without spending real money or waiting for real confirmations.

Part 2: Building a Bitcoin Explorer CLI in Go

With Bitcoin Core running, I built a command-line tool in Go that could query it.

Bitcoin Core exposes a JSON-RPC interface — you send it HTTP POST requests with JSON bodies, and it responds with blockchain data. Think of it as the node's API.

Here's a minimal example of what that looks like in Go:

go
package main

import (
"bytes"
"encoding/json"
"fmt"
"net/http"
)

type RPCRequest struct {
Method string json:"method"
Params []interface{} json:"params"
ID int json:"id"
JSONRPC string json:"jsonrpc"
}

func callRPC(method string, params []interface{}) (map[string]interface{}, error) {
req := RPCRequest{
Method: method,
Params: params,
ID: 1,
JSONRPC: "1.0",
}

body, _ := json.Marshal(req)

resp, err := http.Post(
    "http://localhost:18443/",
    "application/json",
    bytes.NewBuffer(body),
)
if err != nil {
    return nil, err
}
defer resp.Body.Close()

var result map[string]interface{}
json.NewDecoder(resp.Body).Decode(&result)
return result, nil

}

func main() {
info, _ := callRPC("getblockchaininfo", nil)
fmt.Println(info)
}

The constraint was standard library only, no external HTTP clients, no JSON helper packages beyond encoding/json. This was intentional. It forced me to understand exactly what was happening at the network level, not abstract it away.

The explorer could query block height, fetch transaction details by ID, decode raw transactions, and display UTXO (Unspent Transaction Output) information. UTXOs are how Bitcoin actually tracks balances — not accounts, but unspent outputs you have the right to spend.

Part 3: Setting Up Two LND Nodes

LND (Lightning Network Daemon) is the most widely used Lightning implementation, written in Go by Lightning Labs.

I set up two nodes: Alice and Bob. Each one was a separate LND instance, each connected to the same local Bitcoin Core regtest node.

The configuration looks roughly like this for Alice:

ini
[Application Options]
datadir=/home/user/bootcamp-lnd/alice/data
logdir=/home/user/bootcamp-lnd/alice/logs
listen=0.0.0.0:9735
rpclisten=0.0.0.0:10009
restlisten=0.0.0.0:8080
noseedbackup=true

[Bitcoin]
bitcoin.active=1
bitcoin.regtest=1
bitcoin.node=bitcoind

[Bitcoind]
bitcoind.rpchost=localhost
bitcoind.rpcuser=your_rpc_user
bitcoind.rpcpass=your_rpc_password
bitcoind.zmqpubrawblock=tcp://127.0.0.1:28332
bitcoind.zmqpubrawtx=tcp://127.0.0.1:28333

Bob gets his own instance on different ports. Same structure, different directories, different ports.

The ZMQ Issue (and Why It Matters)

One thing that tripped me up: LND communicates with Bitcoin Core not just through RPC, but also through ZMQ (ZeroMQ) — a messaging protocol that lets Bitcoin Core push new block and transaction notifications to LND in real time.

Without ZMQ configured correctly, LND would start but wouldn't know about new blocks. Payments would appear to hang. The fix was ensuring zmqpubrawblock and zmqpubrawtx were properly set on both the Bitcoin Core side and the LND config side, and that the ports matched.

This is the kind of thing a tutorial glosses over. In practice, it's the difference between a working node and a node that silently does nothing.

Part 4: Funding a Channel

Once both nodes were running, I connected them:

bash

Get Alice's node info

lncli --rpcserver=localhost:10009 getinfo

Connect Alice to Bob (using Bob's pubkey@host:port)

lncli --rpcserver=localhost:10009 connect @localhost:9736

Open a channel from Alice to Bob, funding it with 1,000,000 satoshis

lncli --rpcserver=localhost:10009 openchannel --node_key= --local_amt=1000000

Opening a channel requires an on-chain transaction. So I mined a few blocks to confirm it:

bash
bitcoin-cli -regtest generatetoaddress 6

After 3 confirmations, the channel was active. Alice had 1,000,000 satoshis of outbound liquidity :- meaning she could send up to that amount to Bob. Bob had zero outbound liquidity toward Alice unless he funded his side too.

This asymmetry is one of the most important (and often confusing) things about Lightning. Liquidity is directional. Having a channel doesn't mean you can pay in both directions, it depends on where the funds sit within that channel.

Part 5: Making a Payment

With the channel open, Bob creates an invoice, essentially a payment request:

bash
lncli --rpcserver=localhost:10036 addinvoice --amt=50000 --memo="Coffee"

This spits out a BOLT11 payment string :- a long encoded string that starts with lnbcrt... in regtest. It contains the amount, a payment hash, Bob's public key, and an expiry time.

Alice pays it:

bash
lncli --rpcserver=localhost:10009 payinvoice

This happens in under a second. No block confirmation required. The channel's internal balance shifts: Alice now has 950,000 satoshis, Bob has 50,000. The blockchain hasn't changed at all. That shift only gets recorded on-chain when the channel closes.

The LND REST API

At the end of the bootcamp, I was also given access to a remote regtest LND node via its REST API, a taste of what integrating Lightning into a real backend looks like.

LND exposes every operation through HTTP endpoints. To check node info:

bash
curl -k \
-H "Grpc-Metadata-macaroon: " \
https://your-lnd-node/v1/getinfo

Macaroons are LND's authentication mechanism bearer tokens with baked-in permissions. The admin macaroon can do everything. The invoice macaroon can only create and read invoices. You'd use the invoice macaroon in a web server that generates payment requests, and never expose the admin macaroon to anything internet-facing.

In Go, you'd integrate this with:

go
tr := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, // only for dev/regtest
}
client := &http.Client{Transport: tr}

req, _ := http.NewRequest("GET", "https://your-lnd-node/v1/getinfo", nil)
req.Header.Set("Grpc-Metadata-macaroon", adminMacaroonHex)

resp, err := client.Do(req)

This is the pattern you'd use when building a payment backend on top of Lightning, creating invoices on demand, polling for payment confirmation, and triggering downstream actions (like unlocking content or marking an order as paid) when the payment settles.

What This Changes About How I Think

Before this bootcamp, I understood Lightning conceptually. After it, I understand it structurally. There's a difference.

A few things that specifically shifted:

Settlement finality is not binary. On-chain Bitcoin is "final" after enough confirmations. Lightning is final the moment the HTLC (Hash Time Locked Contract) resolves. Different guarantees, different use cases. Knowing which one you need matters when you're designing a payment flow.

How HTTPS Actually Works: TLS, Certificates, and Encryption

Juma Evans — Fri, 24 Jul 2026 23:41:00 +0000

Analogy

Imagine you're sitting in a coffee shop.

You connect to the free Wi-Fi.

You decide to log into your bank account.

Your browser sends your username and password over the network.

Now imagine every person connected to that same Wi-Fi can read everything you're sending.

Your password.

Your balance.

Your account number.

Terrifying.

Yet that's exactly how the early Internet worked.

There was no encryption.

No certificates.

No secure connections.

Just plain text traveling across the network.

HTTPS was invented to solve this problem.

But here's the fascinating part:

Your browser has never met your bank's server before.

So how do two complete strangers agree on a secret encryption key while an attacker is listening to every message?

Let's find out.

Before HTTPS

HTTP sends everything as plain text.

Imagine requesting a webpage.

GET /login HTTP/1.1
Host: bank.com

username=alice
password=myPassword123

Anyone who intercepts this packet can read it.

Wireshark.

Hackers.

Malicious Wi-Fi hotspots.

ISPs.

Nothing is hidden.

HTTP provides functionality.

It provides zero confidentiality.

Enter HTTPS

HTTPS is simply:

HTTP

TLS

Notice something important.

HTTP didn't change.

TLS wraps HTTP in encryption.

Think of TLS as an armored truck carrying ordinary letters.

The letters remain the same.

The transport becomes secure.

What Does HTTPS Actually Protect?

HTTPS provides three major guarantees.

1. Confidentiality

Nobody can read your data.

Even if someone captures every packet...

They see encrypted gibberish.

Instead of:

password=OpenSesame
They see:
8FA91B4D928AC17C3D...

2. Integrity

Imagine a hacker intercepts:
Transfer $100 and changes it to:

Transfer $10,000

TLS detects that the message was modified.

The connection immediately fails.

3. Authentication

How do you know you're actually talking to your bank?

Not a fake server pretending to be it?

This is where certificates enter the picture.

The Biggest Problem

Imagine I want to send you a locked box.

I lock it.

Now...

How do I safely send you the key?

If I send the key with the box...

Anyone can steal both.

This is exactly the problem HTTPS had to solve.

Symmetric Encryption

Symmetric encryption uses one secret key.

Key
↓
Encrypt
↓
Ciphertext
↓
Decrypt
↓
Same Key

Advantages:

Extremely fast
Perfect for large files
Efficient

Problem:

Both sides need the same secret key.
How do they agree on that key?
Asymmetric Encryption

Instead of one key...

There are two.
Public Key
Private Key

Anyone may know the public key.
Nobody should ever know the private key.

Think of it like a mailbox.
Anyone can drop letters inside.
Only the owner can open it.

Why Not Encrypt Everything with RSA?

RSA (or modern alternatives like Elliptic Curve Cryptography) is computationally expensive.

Encrypting an entire Netflix movie with asymmetric encryption would be painfully slow.

Instead HTTPS combines both worlds.
Fast symmetric encryption.
Safe asymmetric key exchange.

The best of both.

Meet TLS

TLS performs a handshake before any HTTP data is exchanged.

This handshake establishes trust.

Generates shared secrets.
Chooses encryption algorithms.
Verifies certificates.
Only after all of this...
Does HTTP begin.

Step 1 — Client Hello

Your browser starts.
Hello!

I support:
AES
ChaCha20
TLS 1.3

Random Number

Think of this as introducing yourself.

Step 2 — Server Hello

The server replies.

Great.

Let's use:

TLS 1.3

AES-256

Here's my certificate.

Now comes the interesting part.

Step 3 — Certificate

The certificate contains information like:

Domain

Public Key

Expiration Date

Certificate Authority

Digital Signature

Notice...

The server isn't saying:

"Trust me."

Instead it's saying:

"Someone you already trust verified me."

Certificate Authorities

Browsers already trust organizations like:

Let's Encrypt
DigiCert
GlobalSign
Sectigo

When a certificate is signed by one of these trusted authorities, the browser can verify that signature.

It's similar to a passport.
You don't personally know the passport holder.
You trust the government that issued it.
What If Someone Creates a Fake Certificate?
Suppose an attacker generates a fake certificate for:

bank.com

The browser checks the digital signature.
It doesn't match any trusted Certificate Authority.

Immediately:
Your connection is not private.
The browser refuses the connection
Key Exchange

Now the browser knows it's talking to the real server.

Both sides perform a key exchange (commonly using Elliptic Curve Diffie–Hellman Ephemeral (ECDHE) in TLS 1.3) to derive the same shared secret.

Here's the remarkable part:

Neither side ever sends the secret encryption key across the network.

They independently calculate the same shared key using exchanged public information and their own private values.

Even if someone captures every packet, they cannot derive the session key.

This is one of the cleverest ideas in modern cryptography.

Session Keys

Once both sides derive the shared secret...

RSA (or more accurately in modern TLS, the key exchange mechanism) has done its job.

From this point onward...

Everything uses symmetric encryption.

Usually:

AES-GCM
ChaCha20-Poly1305

Because they're much faster.

Why HTTPS Is Fast

Many developers assume HTTPS encrypts every message with RSA.

It doesn't.

RSA or ECDHE helps establish the secure session.

After that...

Symmetric encryption handles almost all application data.

This is why HTTPS is nearly as fast as HTTP on modern hardware.

What Happens Every Time You Visit a Website?

The flow looks like this:

Browser
↓
TCP Connection
↓
TLS Handshake
↓
Certificate Verification
↓
Key Exchange
↓
Shared Secret
↓
Encrypted HTTP
↓
Website Loads

Common Myths
"HTTPS encrypts the entire Internet."

No.

It encrypts the communication between your client and the server.

"HTTPS hides which website I'm visiting."

Not entirely.

Your ISP can usually see the destination IP address and some metadata, although modern technologies reduce what is exposed.

"The padlock means the website is safe."

No.

It only means the connection is encrypted and the certificate is valid.

A phishing website can also have HTTPS.

"HTTP is obsolete."

Not completely.

HTTP still exists.

HTTPS is simply HTTP running over TLS.

Real-World Examples

Uses HTTPS:

Online banking
GitHub
Gmail
Amazon
APIs
Payment systems

Without HTTPS:

Passwords
Cookies
Tokens
Credit card numbers

would all travel in plain text.

Interview Questions
a.Why does HTTPS need certificates?

To verify the server's identity and prevent attackers from impersonating legitimate websites.

b.Why use both symmetric and asymmetric encryption?

Asymmetric encryption (or key exchange) solves the problem of establishing trust and securely deriving a shared secret. Symmetric encryption is then used because it's much faster for encrypting the actual data.

c.Why is HTTPS built on TCP?

TLS requires reliable, ordered delivery during the handshake and while exchanging encrypted records. TCP provides those guarantees.

d.Does HTTPS stop hackers?

It protects data in transit between you and the server.

It does not protect against:

Weak passwords
SQL Injection
XSS
Malware on your computer
Social engineering

HTTPS solves one specific problem:

Protecting communication while it's traveling across the network.

Key Takeaways

Every time you visit an HTTPS website, your browser and the server perform a carefully orchestrated dance before a single web page is loaded. They agree on encryption algorithms, verify the server's identity through a trusted certificate, securely derive a shared secret without ever transmitting it, and then switch to fast symmetric encryption for the rest of the session.

The result is that even if someone intercepts every packet traveling across the network, they cannot read or modify the protected data without the session keys.

That small padlock in your browser represents decades of cryptographic research working together to make everyday activities—like online banking, shopping, email, and APIs—secure enough to use over an untrusted Internet.

TCP or UDP? Choosing the Right Protocol

Juma Evans — Fri, 24 Jul 2026 23:00:46 +0000

1. Analogy

Imagine you're sending your friend a 500-page book.

You have two delivery companies.

The first guarantees every page arrives in order.
If page 187 gets lost, they'll resend only page 187.

The second company is incredibly fast.
They throw pages onto trucks immediately.
If page 187 disappears...

They simply keep driving.

Which company would you choose?

The answer depends on what you're sending.

That's exactly why the Internet has TCP and UDP.

Why One Protocol Isn't Enough

Different applications have different priorities.

Imagine these scenarios:

Downloading Ubuntu ISO

↓

Watching Netflix

↓

Video Calling

↓

Playing Valorant

↓

Sending Bank Transactions

Should they all behave the same?

Absolutely not.

A.Meet TCP

TCP says:

I refuse to lose data.

Its priorities:

✔ Reliability

✔ Correct order

✔ Error recovery

✔ Flow control

✔ Congestion control

Not speed.

How TCP Works

Before sending anything:

TCP says:

Let's introduce ourselves.

This is the famous Three-Way Handshake.

Client Server

SYN ------------------>

  <----------------  SYN-ACK

ACK ------------------>

Now both sides know:

"I'm ready."

Only then does data start flowing.

Explanation:

SYN
ACK
Sequence Numbers
Sequence Numbers

Imagine sending:

Hello World

TCP splits it.

Packet 1

Packet 2

Packet 3

Each receives a number.

If packet 2 disappears:

❌

The receiver says:

"I got 1."

"I got 3."

"I'm still missing 2."

TCP resends only packet 2.

This is reliability.

Acknowledgments

Every successful delivery receives an ACK.

Packet

↓

ACK

No ACK?

Resend.

Simple.

Flow Control

Imagine:

Sender:

1000 Mbps

Receiver:

20 Mbps

Without control:

The receiver drowns.

TCP asks:

"How much can you handle?"

Receiver answers:

512 KB

TCP obeys.

Congestion Control

What if the Internet itself is busy?

Highways become congested.

TCP slows down.

Not because the receiver is slow.

Because the network is crowded.

Explain:

Traffic jam analogy.

B.Meet UDP

UDP has one philosophy.

Send it.

That's it.

No handshake.

No acknowledgments.

No retries.

No ordering.

No waiting.

UDP in Action
Packet 1

↓

Packet 2

↓

Packet 3

Packet 2 disappears.

UDP simply continues.

❌

No resend.

Why Would Anyone Want That?

Because sometimes waiting is worse than losing data.

Example:

a.Video Call

Imagine hearing:

Hello...

(wait 3 seconds)

How...

(wait)

are...

(wait)

you?

Terrible.

Instead,

if one audio packet disappears...

Your brain barely notices.

Speed matters more than perfection.

b.Gaming

Suppose you're playing FIFA.

Every 16 milliseconds your position changes.

If one packet disappears:

Do you want the old position?

No.

You want the newest one.

Old data is useless.

UDP wins.

c.Live Streaming

Watching football.

Frame 246 disappears.

Should Netflix stop?

No.

Show frame 247.

Keep going.

d.DNS Uses UDP Too

DNS requests are tiny.

Where is github.com?

The answer is tiny too.

If one packet gets lost:

Just ask again.

Using TCP would waste time establishing a connection for every lookup.

That's why most DNS queries use UDP.

When TCP Wins

Downloading files.

Example:

Linux ISO

PDF

ZIP

Database backup

One missing byte corrupts the entire file.

TCP ensures:

Nothing is lost.

Common Applications
TCP UDP
HTTP DNS
HTTPS VoIP
SSH Online Gaming
FTP Live Streaming
Email DHCP
Database Connections NTP

TCP vs UDP
Feature TCP UDP
Connection Yes No
Reliable Yes No
Ordered Delivery Yes No
Error Recovery Yes No
Speed Slower Faster
Header Size 20–60 bytes 8 bytes
Best For Files, APIs, Banking Games, Calls
How the OSI Model Fits

TCP and UDP both live in:

Layer 4

Transport Layer

Above them:

HTTP

HTTPS

DNS

SMTP

Below them:

Ethernet

Wi-Fi

Interview Questions

a.Why doesn't TCP always replace UDP?

Because reliability has a cost. Handshakes, acknowledgments, retransmissions, and congestion control all add latency.

b.Why doesn't UDP replace TCP?

Because some data must arrive intact and in order. Losing a byte in a bank transaction or software download is unacceptable.

c.Can UDP be made reliable?

Yes. Applications can implement their own reliability mechanisms on top of UDP. A good example is QUIC, which runs over UDP and powers HTTP/3 by handling reliability and security in user space.

d.Why does HTTP/3 use UDP?

Because it builds its own transport features on top of UDP, avoiding some of TCP's limitations—particularly connection setup delays and head-of-line blocking.

Final Takeaways

TCP and UDP aren't competitors trying to replace one another. They solve different problems. TCP prioritizes reliability, ensuring data arrives completely, in order, and without corruption. UDP prioritizes speed and low latency, accepting occasional packet loss when timely delivery is more important than perfect delivery.

The next time you're downloading a file, making a video call, joining an online game, or performing a DNS lookup, you'll know why the Internet chooses one protocol over the other.

Understanding the OSI Model Through One Network Request

Juma Evans — Fri, 24 Jul 2026 22:30:10 +0000

The Hook

Every networking course begins with the same seven mysterious layers:

Physical
Data Link
Network
Transport
Session
Presentation
Application

Students memorize them.

Developers ignore them.

Network engineers live by them.

But here's the problem:

Most people never learn why these layers exist.

In this article, we'll follow a single message;"Hello, Server!" as it travels through every OSI layer, crosses routers and switches, reaches another computer, and climbs back up the stack.

By the end, you'll understand not just the names of the layers, but why the Internet couldn't exist without them.

The Story

Imagine you open your browser.

https://github.com

You press Enter.

Now follow that request.

Layer 7 : Application

This is where humans interact with software.

Examples:

Browser
WhatsApp
Discord
Gmail
Spotify

The browser creates:

GET / HTTP/1.1
Host: github.com

Notice:

No IP.

No MAC address.

No Ethernet.

Just application data.

Layer 6 : Presentation

This layer asks:

How should the data look?

Examples include:

Encryption (TLS/SSL)
Compression (gzip)
Character encoding (UTF-8)
Serialization (JSON, XML, Protocol Buffers)

Here your HTTP request is encrypted into ciphertext before leaving your machine.

Without this layer:

Everyone on the network could read your passwords.

Layer 5 : Session

This layer manages conversations.

Think of it as the meeting organizer.

Responsibilities include:

Opening communication
Keeping it alive
Reconnecting if interrupted
Closing the session cleanly

Modern TCP/IP doesn't expose this as a separate layer, but the concept still exists in many protocols.

Layer 4 : Transport

Now things become interesting.

Imagine sending a 100 MB video.

Should it be one enormous packet?

No.

Transport breaks it into manageable pieces called segments.

It also adds:

Source port
Destination port
Sequence number
Acknowledgments
Error recovery

Protocols:

TCP
UDP

Analogy:

A courier numbers every box before shipping.

If Box #7 disappears...

Only Box #7 is resent.

Layer 3 : Network

Now we need directions.

The Network layer adds:

Source IP

Destination IP

Example:

192.168.1.10

↓

140.82.121.3

Routers read this information.

Their only job is:

Which road gets this packet closer to its destination?

Think Google Maps.

Layer 2 : Data Link

Now the packet reaches your home Wi-Fi.

The router doesn't care about IP first.

It wants:

MAC Address

Why?

Because devices communicate locally using hardware addresses.

The frame now contains:

Destination MAC

Source MAC

Payload

Switches live here.

They forward frames only to the correct device.

This is why switches are much smarter than hubs.

Layer 1 : Physical

Finally...

Everything becomes electricity.

Or light.

Or radio waves.

Bits become:

101001011010011001

Those bits travel through:

Copper cables
Fiber optics
Wi-Fi radio
Satellite signals

Layer 1 doesn't know HTTP.

It doesn't know IP.

It doesn't even know bytes.

It only knows:

1
Encapsulation

Here's the beautiful part.

Each layer wraps the previous layer with its own information.

Think of Russian nesting dolls.

Application Data

↓

Segment

↓

Packet

↓

Frame

↓

Bits

Or like mailing a package:

Letter

↓

Envelope

↓

Shipping Box

↓

Truck

↓

Road

Each layer adds just enough information for the next part of the journey.

At the Destination

The server receives:

Bits

↓

Frame

↓

Packet

↓

Segment

↓

Application Data

Each layer removes the information added by its counterpart on the sender's side.

This process is called decapsulation.

Eventually GitHub receives:

GET /

and responds with HTML.

Which Devices Work at Each Layer?
Layer Device
7 Browser, Web Server
6 TLS, SSL
5 Session managers
4 TCP, UDP
3 Router
2 Switch
1 Cable, Fiber, Wi-Fi

This table alone helps readers connect abstract layers to real-world hardware and software.

Why Was the OSI Model Invented though?

Before the OSI model, networking vendors often built proprietary systems that worked only with their own hardware and software. There wasn't a common language for how devices should communicate.

The OSI model introduced a layered architecture where each layer has a single responsibility and communicates only with the layers directly above and below it.

This separation provides several advantages:

Modularity: You can improve one layer without redesigning the entire stack.
Interoperability: Devices from different vendors can communicate because they follow the same layer responsibilities.
Troubleshooting: Network problems become easier to isolate. If you can't even establish a physical connection, there's no point debugging HTTP.
Scalability: New technologies can be introduced within a layer without affecting the rest of the system.

Although the Internet actually uses the simpler TCP/IP model, the OSI model remains one of the best conceptual tools for understanding how data moves across networks.

A Developer's View of the OSI Model

When you're writing backend services or APIs, you interact with multiple layers—even if you don't think about them:

You build REST or GraphQL APIs at the Application layer.
HTTPS relies on Presentation layer concepts like encryption.
Socket connections and ports depend on the Transport layer.
IP addresses and routing involve the Network layer.
Switches, Ethernet, and Wi-Fi operate at the Data Link and Physical layers.

Understanding where a problem occurs makes debugging much faster.

Key Takeaways

The OSI model isn't just a list to memorize for an exam—it's a way of thinking about networking. Every request you send travels down the stack, where each layer adds the information needed for its specific job. It crosses the network as electrical signals, light, or radio waves, then climbs back up the layers on the receiving machine until the original application data is reconstructed.

Once you see networking as a journey through these seven layers rather than seven isolated definitions, concepts like routers, switches, TCP, IP, TLS, and HTTP start fitting together naturally.