In the Linux world, syscall is typically the primary mechanism for userspace programs to interact with the kernel and leverage the full capabilities of the Linux kernel.

In this blog post, we will explore Linux system calls in-depth, not just theoretically but also through practical examples. We will start from the fundamentals and gradually build a comprehensive understanding of Linux system calls.

Intro

It’s straightforward to make a system call in C, as demonstrated by this simple example of the write system call:

// write.c
#include <unistd.h>
int main() {
    write(1, "Hello World!\n", 13);
    return 0;
}

/*
$ gcc -o write write.c && ./write
Hello, World!
*/

This simple example does only one thing: to print "Hello World!" to stdout(file descriptor 1). But what if I want to do the same thing in other programming language?

Most languages simply reuse the same standard library that C uses, which is called the C standard library or libc. While there are several implementations of libc, the most popular one is the GNU C Library, commonly known as glibc. Glibc is the default libc implementation for most Linux distributions. But there are also other libc implementations, such as musl, which provide different features and capabilities. musl is commonly used for creating portable applications that doesn’t rely on the version of glibc that comes with the distribution.

You may have also heard that Go can perform system calls without reply on libc¹, that is because it has its own runtime that can directly interact with the system call interface. This allows Go programs to be built as fully static binaries, which can run without any external dependencies. This feature makes Go programs highly portable, especially suitable for running in containerized environments.

Application Binary Interface

It seems that system calls can be language-agnostic, and there must be some common protocols that all languages can follow. This common interface is known as the Application Binary Interface (ABI).

In the context of system calls, the most significant part of ABI is the Calling Convention, which essentially defines the low-level rules for how a procedure call is performed in the binary code.

Since Linux is primarily written in C, the system call interface is designed to be compatible with the C calling convention, which is also the most common calling convention used globally. By adhering to the C calling convention, developers can not only perform system calls in different programming languages, they can also call procedures written in another language, because most languages use C calling convention for their FFI feature. The latter case is even more common, as most languages won’t interact with the system call interface directly in binary level, but by calling a C-based system call wrapper defined in the libc. We can say that the C calling convention is the lingua franca of the programming world.

Before delving into the details of system calls, let’s first take a look at some normal procedure calls and examine what the calling convention entails.

Example: sum3

// sum3.c
int sum3(int a, int b, int c) {
    return a + b + c;
}
int main(void) {
    int sum = sum3(1, 2, 3);
    return 0;
}

The main function in this simple C program calls a function sum3 to calculate the sum of three integers, and saves the result to the variable sum. Let’s compile and disassemble the program to examine the generated assembly code:

# gcc -O0 -o sum3 sum3.c && objdump -d --no-show-raw-insn sum3
# ...
0000000000001119 <sum3>:
    1119:	push   %rbp
    111a:	mov    %rsp,%rbp
    111d:	mov    %edi,-0x4(%rbp)
    1120:	mov    %esi,-0x8(%rbp)
    1123:	mov    %edx,-0xc(%rbp)
    1126:	mov    -0x4(%rbp),%edx
    1129:	mov    -0x8(%rbp),%eax
    112c:	add    %eax,%edx
    112e:	mov    -0xc(%rbp),%eax
    1131:	add    %edx,%eax
    1133:	pop    %rbp
    1134:	ret

0000000000001135 <main>:
    1135:	push   %rbp
    1136:	mov    %rsp,%rbp
    1139:	sub    $0x10,%rsp
    113d:	mov    $0x3,%edx
    1142:	mov    $0x2,%esi
    1147:	mov    $0x1,%edi
    114c:	call   1119 <sum3>
    1151:	mov    %eax,-0x4(%rbp) # sum = %eax
    1154:	mov    $0x0,%eax
    1159:	leave
    115a:	ret
# ...

As we can see, in the main function, the parameters (1, 2, 3) are first placed in the registers %edi, %esi, and %edx respectively. Then, the call instruction is executed, with the address 1119 (the address of the sum3 function) as the operand. Finally, the return value is placed in the %eax register.

To summarize: First, the parameters are placed in the appropriate registers in a specific order, then the call instruction is executed with the address of the function to be called as the operand. After the function returns, the return value can be retrieved from the %eax register.

Example: bigret

But here arises an important question: since all parameters and the return value are placed in registers in this example, and registers are typically only 64-bit in size, what if we want to pass a parameter or return a value that is larger than 64-bit, like a struct? Let’s conduct some experiments to find out.

// bigret.c
typedef unsigned long long u64;

struct big {
    u64 a, b, c;
};

struct big ret_big(u64 a, u64 b, u64 c) {
    return (struct big){a, b, c};
}

int main(void) {
    struct big big = ret_big(1, 2, 3);
    return 0;
}

Again, compile and disassemble:

# gcc -O0 -o bigret bigret.c && objdump -d --no-show-raw-insn bigret
# ...
0000000000001139 <ret_big>:
    1139:	push   %rbp
    113a:	mov    %rsp,%rbp
    113d:	mov    %rdi,-0x28(%rbp)
    1141:	mov    %rsi,-0x30(%rbp)
    1145:	mov    %rdx,-0x38(%rbp)
    1149:	mov    %rcx,-0x40(%rbp)
    114d:	mov    -0x28(%rbp),%rax
    1151:	mov    -0x30(%rbp),%rdx
    1155:	mov    %rdx,(%rax)
    1158:	mov    -0x28(%rbp),%rax
    115c:	mov    -0x38(%rbp),%rdx
    1160:	mov    %rdx,0x8(%rax)
    1164:	mov    -0x28(%rbp),%rax
    1168:	mov    -0x40(%rbp),%rdx
    116c:	mov    %rdx,0x10(%rax)
    1170:	mov    -0x28(%rbp),%rax
    1174:	pop    %rbp
    1175:	ret

0000000000001176 <main>:
    1176:	push   %rbp
    1177:	mov    %rsp,%rbp
    117a:	sub    $0x20,%rsp
    117e:	mov    %fs:0x28,%rax
    1187:	mov    %rax,-0x8(%rbp)
    118b:	xor    %eax,%eax
    118d:	lea    -0x20(%rbp),%rax
    1191:	mov    $0x3,%ecx
    1196:	mov    $0x2,%edx
    119b:	mov    $0x1,%esi
    11a0:	mov    %rax,%rdi
    11a3:	call   1139 <ret_big>
    11a8:	mov    $0x0,%eax
    11ad:	mov    -0x8(%rbp),%rdx
    11b1:	sub    %fs:0x28,%rdx
    11ba:	je     11c1 <main+0x4b>
    11bc:	call   1030 <__stack_chk_fail@plt>
    11c1:	leave
    11c2:	ret
# ...

Hmm, that’s quite interesting. It appears that an address is implicitly passed as the first parameter to the ret_big function, and the return value is also at the same address. Let’s examine this step-by-step:

In the main function:

1. 117a: sub $0x20,%rsp: allocate 32 bytes on the stack by subtracting 0x20(32) from the stack pointer register %rsp. This can be viewed as declare an uint64_t[4] array on stack.
2. 117e: mov %fs:0x28,%rax: read a value from address %fs+0x28 and store it in the register %rax.
3. 1187: mov %rax,-0x8(%rbp): save the read value on the stack at the address (%rbp-0x8), which corresponds to arr[3].
4. 118b: xor %eax,%eax: clear register %eax to 0.
5. 118d: lea -0x20(%rbp),%rax: load the address of %rbp-0x20(&arr[0]) to %rax.
6. 1191 - 11a0: place the parameters in the registers in the correct order, with a shift by one position, since $eax is now the first parameter.
7. 113a call 1139 <ret_big>: call ret_big.

The code in the ret_big function appears a bit verbose, but if we recompile it with the -O1 optimization flag, it becomes much simpler:

0000000000001119 <ret_big>:
    1119:	mov    %rdi,%rax
    111c:	mov    %rsi,(%rdi)
    111f:	mov    %rdx,0x8(%rdi)
    1123:	mov    %rcx,0x10(%rdi)
    1127:	ret

Since the first parameter $rdi is the address of the array arr that we allocated earlier, and the three parameters we passed to ret_big are placed in the registers $rsi, $rdx, and $rcx respectively, this optimized code simply copies the three parameters to the array arr in order:

1. 111c: mov %rsi,(%rdi): arr[0] = %rdi.
2. 111f: mov %rdi,0x8(%rdi): arr[1] = %rsi.
3. 1123: mov %rcx,0x10(%rdi): arr[2] = %rdx.
4. 1127: ret: return arr

Now it’s clear. If we want to return a large data structure that cannot fit into a single register, we can pass the address of a buffer as the return value parameter and let the function save the data at that address.

However, this approach introduces a potential risk of buffer overflow, as the function might write more data than the allocated buffer can accommodate. This could result in a segment fault error, or even worse, an Arbitrary Code Execution vulnerability. So, how can we prevent this from happening?

Canary: The Guardian of the stack

You might notice that in the main function(step 3), the fourth instruction places a secret value on the stack at the location arr[3], which is the last element of the array arr we allocated.

Most buffer overflow attacks exploit the fact that the attacker can provide an unbounded string as input, and if the program uses a null character (\0) as the input delimiter, it will write the user input along the buffer and overwrite the memory that does not belong to the buffer, which may contain the return address.

After ret_big returns, the main function immediately checks if the secret value has been changed (instructions from 11ad to 11ba). If the secret value remains unchanged, it means the memory contents following it have also not been altered, so the program considers it safe and jumps to the instruction at <main+0x4b>(11c1), and returns normally. However, if the secret value has been changed, the program assumes the memory after that value has been altered too, and it will call the __stack_chk_fail function to terminate the program and prevent further damage.

This secret value is a random number known as a Canary, and it is a common technique used to defend against buffer overflow attacks. The term “Canary” is a reference to the historic practice of using canaries in coal mines to warn miners of toxic gases, which is another somber story².

x86_64 Calling Convention

There are many details defined by the calling convention, but for the purpose of our discussion on system calls, the following specifications are the most important:

• Where parameters are placed.
• The order in which parameters are passed.
• How the stack changes during the call
• How return values are delivered back to the caller.
• Which registers are guaranteed to have the same value before and after the call.

From the two examples above, we already know the first three, and remain of them is defined in the following specification, you can also find it at here:

x86 function call convention, 64-bit:
-------------------------------------
 arguments           |  callee-saved      | extra caller-saved | return
[callee-clobbered]   |                    | [callee-clobbered] |
---------------------------------------------------------------------------
rdi rsi rdx rcx r8-9 | rbx rbp [*] r12-15 | r10-11             | rax, rdx [**]

( rsp is obviously invariant across normal function calls. (gcc can 'merge'
  functions when it sees tail-call optimization possibilities) rflags is
  clobbered. Leftover arguments are passed over the stack frame.)

[*]  In the frame-pointers case rbp is fixed to the stack frame.

[**] for struct return values wider than 64 bits the return convention is a
     bit more complex: up to 128 bits width we return small structures
     straight in rax, rdx. For structures larger than that (3 words or
     larger) the caller puts a pointer to an on-stack return struct
     [allocated in the caller's stack frame] into the first argument - i.e.
     into rdi. All other arguments shift up by one in this case.
     Fortunately this case is rare in the kernel.

Normally, the callee-save" means the caller can assume that the value of the register is not changed after the function call, so the callee function itself is responsible for saving the original value in these registers before using it for other purposes. On the other hand, the callee-clobbered means the caller cannot assume the register value is preserved, so the caller needs to manually them before the function call if it wants to use it afterwards.

Now that we have a necessary understanding of the calling convention, let’s go deeper and examine how system calls are implemented.

Userspace Stub

So how does syscall happens, actually? If you look up the definition of the write, you will see the following function signature:

/* Write N bytes of BUF to FD.  Return the number written, or -1.

   This function is a cancellation point and therefore not marked with
   __THROW.  */
extern ssize_t write (int __fd, const void *__buf, size_t __n) __wur
    __attr_access ((__read_only__, 2, 3));

But we still don’t know exactly what happens when we call it. All we know is that it’s like a normal C function: we call it, and everything gets done.

write in userspace

Since the source code of libc is open to everyone, we can look up how write is implemented. But before doing that, let’s use gdb to find out what actually happens.

// write.c
#include <unistd.h>
int main() {
    write(1, "Hello, World!\n", 14);
    return 0;
}

After compiling this simple code with gcc -O1 -g -o write write.c³, we get an executable named write. All it does is print the string “Hello, World!”.

$ ./write
Hello, World!

Now we’ll use gdb to run the executable again.

$ gdb write
...(Messages printed at start)
(gdb)

First, set a breakpoint at the main symbol to stop at the entry point. Then enable the disassemble-next-line option and run the program.

(gdb) b main
Breakpoint 1 at 0x1139: file write.c, line 3.
(gdb) set disassemble-next-line on
(gdb) show disassemble-next-line
Debugger's willingness to use disassemble-next-line is on.
(gdb) r
Starting program: ./write

Breakpoint 1, main () at write.c:3
3       int main() {
=> 0x0000555555555139 <main+0>: 48 83 ec 08             sub    $0x8,%rsp

We can see from the assembly code that the stack grows by 0x8 bytes, giving main a stack size of 0x8 bytes. However, that’s not our focus. Let’s execute this program instruction-by-instruction using si (step instruction).

(gdb) si
4         write(1, "Hello, World!\n", 14);
=> 0x000055555555513d <main+4>: ba 0e 00 00 00          mov    $0xe,%edx
   0x0000555555555142 <main+9>: 48 8d 35 bb 0e 00 00    lea    0xebb(%rip),%rsi        # 0x555555556004
   0x0000555555555149 <main+16>:        bf 01 00 00 00          mov    $0x1,%edi
   0x000055555555514e <main+21>:        e8 dd fe ff ff          call   0x555555555030 <write@plt>

From the last blog, we know that arguments are passed by registers in the order of rdi, rsi, rdx, rcx, r8, and r9. We can see that rdi, rsi, edx are assigned to 0x1(1), %rip + 0xebb (which is annotated as 0x555555556004), and 0xe(14) respectively. That’s exactly what we passed to the write function. The second argument("Hello, World!\n") is passed by pointer though, we can verify this by checking what resides at this pointer:

(gdb) x/s 0x555555556004
0x555555556004: "Hello, World!\n"

OK, now that we understand argument passing, let’s dive into the write function.

(gdb) s
__GI___libc_write (fd=fd@entry=1, buf=buf@entry=0x555555556004, nbytes=nbytes@entry=14) at ../sysdeps/unix/sysv/linux/write.c:25

We can even see what is passed to which parameter thanks to the GDB debuginfod feature, but we already know that, so let’s print out the assembly code of the __GI___libc_write⁴:

(gdb) x/16i $pc
=> 0x7ffff7e9b4f0 <__GI___libc_write>:  endbr64
   0x7ffff7e9b4f4 <__GI___libc_write+4>:        cmpb   $0x0,0xe0b45(%rip)        # 0x7ffff7f7c040 <__libc_single_threaded>
   0x7ffff7e9b4fb <__GI___libc_write+11>:       je     0x7ffff7e9b510 <__GI___libc_write+32>
   0x7ffff7e9b4fd <__GI___libc_write+13>:       mov    $0x1,%eax
   0x7ffff7e9b502 <__GI___libc_write+18>:       syscall
   0x7ffff7e9b504 <__GI___libc_write+20>:       cmp    $0xfffffffffffff000,%rax
   0x7ffff7e9b50a <__GI___libc_write+26>:       ja     0x7ffff7e9b560 <__GI___libc_write+112>
   0x7ffff7e9b50c <__GI___libc_write+28>:       ret
   0x7ffff7e9b50d <__GI___libc_write+29>:       nopl   (%rax)
   0x7ffff7e9b510 <__GI___libc_write+32>:       push   %rbp
   0x7ffff7e9b511 <__GI___libc_write+33>:       mov    %rsp,%rbp
   0x7ffff7e9b514 <__GI___libc_write+36>:       sub    $0x20,%rsp
   0x7ffff7e9b518 <__GI___libc_write+40>:       mov    %rdx,-0x18(%rbp)
   0x7ffff7e9b51c <__GI___libc_write+44>:       mov    %rsi,-0x10(%rbp)
   0x7ffff7e9b520 <__GI___libc_write+48>:       mov    %edi,-0x8(%rbp)
   0x7ffff7e9b523 <__GI___libc_write+51>:       call   0x7ffff7e20d90 <__GI___pthread_enable_asynccancel>

Well, it’s quite verbose, but we only need to focus on two lines:

   0x7ffff7e9b4fd <__GI___libc_write+13>:       mov    $0x1,%eax
   0x7ffff7e9b502 <__GI___libc_write+18>:       syscall

These codes simply set the eax register to 0x1 and then execute the syscall instruction.

(gdb) x/i $pc
=> 0x7ffff7e9b502 <__GI___libc_write+18>:       syscall
(gdb) si
Hello, World!
(gdb)

After the syscall instruction returns, we can see “Hello, World!” from the GDB console. The entire process can be summarized in three steps:

1. Pass the arguments following the C Calling Convention.
2. Set the rax register (or eax in 32-bit mode) to a number specified as the syscall number in Linux documentation.
3. Execute the syscall instruction⁵.

This is surprisingly simple, we can just set an extra register rax to the required syscall number, replace the call instruction with syscall, and an normal C function call becomes a syscall. We can even create our own write function with just the two lines of assembly code above.

However, there is actually another small difference in the calling convention: for syscalls, the fourth argument is passed via register r10 instead of rcx. This is because syscall need to store the address of the next userspace instruction in register rcx, so after returning from kernel, execution can continue at that saved address.

Glibc syscall stub

Previously, we use GDB to understand what happens with write under the hood. Now, let’s examine the code to see how write is implemented. You can skip this chapter if you’re not familiar with or interested in C macro magic.

The following code is based on glibc-2.38, which you can download from the GNU FTP server. The definition of write isn’t straightforward; it involves expanding complex macros dynamically. After manually expanding these magic-like macros, the simplified write.c looks like this:

define __glibc_unlikely

/* NB: This also works when X is an array.  For an array X,  type of
   (X) - (X) is ptrdiff_t, which is signed, since size of ptrdiff_t
   == size of pointer, cast is a NOP.   */
#define TYPEFY1(X) __typeof__ ((X) - (X))
/* Explicit cast the argument.  */
#define ARGIFY(X) ((TYPEFY1 (X)) (X))
/* Create a variable 'name' based on type of variable 'X' to avoid
   explicit types.  */
#define TYPEFY(X, name) __typeof__ (ARGIFY (X)) name

typedef int ssize_t;
typedef unsigned int size_t;

extern int __libc_single_threaded;
extern int __libc_errno;

int __pthread_enable_asynccancel();
void __pthread_disable_asynccancel(int);

#define __NR_write 1

#define internal_syscall3(number, arg1, arg2, arg3)			\
({									\
    unsigned long int resultvar;					\
    TYPEFY (arg3, __arg3) = ARGIFY (arg3);			 	\
    TYPEFY (arg2, __arg2) = ARGIFY (arg2);			 	\
    TYPEFY (arg1, __arg1) = ARGIFY (arg1);			 	\
    register TYPEFY (arg3, _a3) asm ("rdx") = __arg3;			\
    register TYPEFY (arg2, _a2) asm ("rsi") = __arg2;			\
    register TYPEFY (arg1, _a1) asm ("rdi") = __arg1;			\
    asm volatile (							\
    "syscall\n\t"							\
    : "=a" (resultvar)							\
    : "0" (number), "r" (_a1), "r" (_a2), "r" (_a3)			\
    : "memory", REGISTERS_CLOBBERED_BY_SYSCALL);			\
    (long int) resultvar;						\
})

/* Write NBYTES of BUF to FD.  Return the number written, or -1.  */
ssize_t
__libc_write (int fd, const void *buf, size_t nbytes)
{
    long int ret;
    if (__libc_single_threaded != 0)
    {
        long int sc_ret = internal_syscall3 (__NR_write, fd, buf, nbytes);
        if (__glibc_unlikely ((unsigned long int) (sc_ret) > -4096UL)) {
            __libc_errno = -sc_ret;
            ret = -1L;
        } else {
            ret = sc_ret;
        }
    }
    else
    {
        int sc_cancel_oldtype = __pthread_enable_asynccancel();
        long int sc_ret = internal_syscall3 (__NR_write, fd, buf, nbytes);
        if (__glibc_unlikely ((unsigned long int) (sc_ret) > -4096UL)) {
            __libc_errno = -sc_ret;
            ret = -1L;
        } else {
            ret = sc_ret;
        }
        __pthread_disable_asynccancel(sc_cancel_oldtype);
    }
    return ret;
}
libc_hidden_def (__libc_write)

// weak_alias (__libc_write, __write)
// libc_hidden_weak (__write)
// weak_alias (__libc_write, write)
// libc_hidden_weak (write)

int main(void) {
    write(1, "Hello, world!\n", 14);
    return 0;
}

The main part of write is in __libc_write, where we can see the pthread synchronization guard that we saw earlier in the GDB disassembly. The syscall number for write is defined in the macro __NR_write, which is 1. The actual arguments are already passed outside of the write, so the internal syscall stub only cares about the number of arguments. Here, we have internal_syscall3, which use inline assembly to call the instruction syscall. However, if we look at the internal_syscall4, we will notice some differences:

#define internal_syscall4(number, arg1, arg2, arg3, arg4)		\
({									\
    unsigned long int resultvar;					\
    TYPEFY (arg4, __arg4) = ARGIFY (arg4);			 	\
    TYPEFY (arg3, __arg3) = ARGIFY (arg3);			 	\
    TYPEFY (arg2, __arg2) = ARGIFY (arg2);			 	\
    TYPEFY (arg1, __arg1) = ARGIFY (arg1);			 	\
    register TYPEFY (arg4, _a4) asm ("r10") = __arg4;			\
    register TYPEFY (arg3, _a3) asm ("rdx") = __arg3;			\
    register TYPEFY (arg2, _a2) asm ("rsi") = __arg2;			\
    register TYPEFY (arg1, _a1) asm ("rdi") = __arg1;			\
    asm volatile (							\
    "syscall\n\t"							\
    : "=a" (resultvar)							\
    : "0" (number), "r" (_a1), "r" (_a2), "r" (_a3), "r" (_a4)		\
    : "memory", REGISTERS_CLOBBERED_BY_SYSCALL);			\
    (long int) resultvar;						\
})

As we mentioned, the fourth argument is assigned to register r10 instead of rcx; besides that, nothing is different.

Prepare for entering kernel space

From the userspace perspective, we only see an instruction being executed, and the job is done. But what happens behind the scenes? The online x86 reference gives an detailed operation specification of the syscall instruction:

IF (CS.L ≠ 1 ) or (IA32_EFER.LMA ≠ 1) or (IA32_EFER.SCE ≠ 1)
(* Not in 64-Bit Mode or SYSCALL/SYSRET not enabled in IA32_EFER *)
    THEN #UD;
FI;
RCX := RIP; (* Will contain address of next instruction *)
RIP := IA32_LSTAR;
R11 := RFLAGS;
RFLAGS := RFLAGS AND NOT(IA32_FMASK);
CS.Selector := IA32_STAR[47:32] AND FFFCH (* Operating system provides CS; RPL forced to 0 *)
(* Set rest of CS to a fixed value *)
CS.Base := 0;
                (* Flat segment *)
CS.Limit := FFFFFH;
                (* With 4-KByte granularity, implies a 4-GByte limit *)
CS.Type := 11;
                (* Execute/read code, accessed *)
CS.S := 1;
CS.DPL := 0;
CS.P := 1;
CS.L := 1;
                (* Entry is to 64-bit mode *)
CS.D := 0;
                (* Required if CS.L = 1 *)
CS.G := 1;
                (* 4-KByte granularity *)
IF ShadowStackEnabled(CPL)
    THEN (* adjust so bits 63:N get the value of bit N–1, where N is the CPU’s maximum linear-address width *)
        IA32_PL3_SSP := LA_adjust(SSP);
            (* With shadow stacks enabled the system call is supported from Ring 3 to Ring 0 *)
            (* OS supporting Ring 0 to Ring 0 system calls or Ring 1/2 to ring 0 system call *)
            (* Must preserve the contents of IA32_PL3_SSP to avoid losing ring 3 state *)
FI;
CPL := 0;
IF ShadowStackEnabled(CPL)
    SSP := 0;
FI;
IF EndbranchEnabled(CPL)
    IA32_S_CET.TRACKER = WAIT_FOR_ENDBRANCH
    IA32_S_CET.SUPPRESS = 0
FI;
SS.Selector := IA32_STAR[47:32] + 8;
                (* SS just above CS *)
(* Set rest of SS to a fixed value *)
SS.Base := 0;
                (* Flat segment *)
SS.Limit := FFFFFH;
                (* With 4-KByte granularity, implies a 4-GByte limit *)
SS.Type := 3;
                (* Read/write data, accessed *)
SS.S := 1;
SS.DPL := 0;
SS.P := 1;
SS.B := 1;
                (* 32-bit stack segment *)
SS.G := 1;
                (* 4-KByte granularity *)

The pseudo code above contains many variable-like identifiers that are actually registers, including General Purpose Registers (RCX, R11), Pointer Registers (RIP), Segment Registers (CS, SS), the RFLAGS Register (RFLAGS), and Model-Specific Registers (MSRs) (IA32_EFER, IA32_LSTAR, IA32_FMASK, IA32_PL3_SSP). The operator . accesses specific bit fields within these registers. Despite the numerous operations occurring, we only need to focus on a few:

RCX := RIP; (* Save address of next instruction(userspace) to RCX *)
RIP := IA32_LSTAR; (* Set address of next instruction(kernel) to IA32-LSTAR *)
R11 := RFLAGS; (* Save old RFLAGS to R11 *)
RFLAGS := RFLAGS AND NOT(IA32_FMASK); (* Clear some flags *)
CPL := 0; (* Set current privilege level to 0, also known as Protection Ring *)

Basically, the syscall instruction backs up some userspace context information (such as the return address), sets the next instruction pointer (RIP) to somewhere in kernel space (IA32-LSTAR), and switches the protection ring to 0, which is kernel mode. For security reasons, only in this mode can the CPU execute kernel code.

It’s important to note that all of this occurs within a single syscall instruction, so set RIP to point to kernel space does make sense. After executing this instruction, we are finally in kernel space.

Prepare bootable kernel

To examine what happened in the kernel space, we will use QEMU⁶ and GDB to debugging a running kernel. But first, we need to create a bootable Linux kernel image, which means we need to compile the kernel.

Compiling the Linux kernel

For a general C project, the basic build process typically follows three steps, and the Linux kernel is no exception:

Get the source

You can obtain the source code for every released Linux kernel version from the official kernel.org website. In this post, I will use version 6.6.6, but you can use any version you prefer. Additionally, make sure to download the GPG signature file for the kernel version you choose.

# Download source
curl -OL https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.6.6.tar.xz
# Download the corresponding signature
curl -OL https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.6.6.tar.sign
# Uncompressing source
unxz linux-6.6.6.tar.xz
# Import keys belonging to Linus Torvalds and Greg Kroah-Hartman
gpg2 --locate-keys torvalds@kernel.org gregkh@kernel.org
# Verify the .tar archive with the signature
gpg2 --verify linux-6.6.6.tar.sign

If everything is set up correctly, you should see the following output after verifying the GPG signature(make sure you see gpg: Good signature in the output):

gpg: assuming signed data in 'linux-6.6.6.tar'
gpg: Signature made Mon 11 Dec 2023 05:40:57 PM CST
gpg:                using RSA key 647F28654894E3BD457199BE38DBBDC86092693E
gpg: Good signature from "Greg Kroah-Hartman <gregkh@kernel.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 647F 2865 4894 E3BD 4571  99BE 38DB BDC8 6092 693E

After verifying the signature, extract the .tar file to get the kernel source tree linux-6.6.6/.

Configuration

Since our focus is on examining the system call implementation, we can use the default kernel configuration.

cd linux-6.6.6/
make defconfig

you can also use the kernel configuration that your Linux distribution employs if you like⁷, as many distributions save the configuration under /boot/config-$(uname -r). If you are using Arch Linux like me, you will find it at /proc/config.gz.

If you decide to use a custom kernel configuration, you will need to rename the configuration file to .config (and potentially uncompress it first) and place it in the root of the kernel source tree (linux-6.6.6/ in our case). Then, you need to run make olddefconfig to apply the custom configuration.

There are also some useful make targets for managing the kernel configuration:

# ncurse based TUI
make nconfig

# X based GUI
make menuconfig

# Print help
make help

We will also add a custom build tag to the kernel configuration, but this step is optional, feel free to skip it.

# (optional)Add a build tag syscall_irl
./scripts/config --file .config --set-str LOCALVERSION "syscall_irl"

Compiling

Now that everything is ready, there is just one big thing left. Be careful your CPU might catch fire!

# Compile the kernel with all available cores, you can change the -j flag to the number of cores you want to use
make -j$(nproc) 2>&1 | tee build.log

It takes about 64 seconds to compile the Linux kernel on my i7-13700K platform when using all available CPU cores. As long as no errors are encountered during the build process, you should see the following output at the end:

...
	LD      arch/x86/boot/compressed/vmlinux
	ZOFFSET arch/x86/boot/zoffset.h
	OBJCOPY arch/x86/boot/vmlinux.bin
	AS      arch/x86/boot/header.o
	LD      arch/x86/boot/setup.elf
	OBJCOPY arch/x86/boot/setup.bin
	BUILD   arch/x86/boot/bzImage
Kernel: arch/x86/boot/bzImage is ready  (#1)

The final steps in the compilation process build the uncompressed Linux kernel vmlinux, and then convert it into a bootable compressed kernel image bzImage. You can find the resulting bzImage file at arch/x86/boot/bzImage, as indicated in the output.

Make a root filesystem

Now you can actually boot the kernel in QEMU.

qemu-system-x86_64 -kernel arch/x86/boot/bzImage

But you will find that the kernel panics immediately. This is because we have only provided the kernel image itself, but a significant part is still missing: the root filesystem that makes the system functional. We need to attach an initramfs⁸ that contains a basic root filesystem to provide a working environment.

Filesystem structure

Since we are using the initramfs solely to run the kernel, and not for switching to a real disk-based root filesystem, we only need to create three directories in this root filesystem: sys/ for mounting sysfs, proc/ for mounting procfs, and bin/ for all necessary binaries.

# Make a directory that contains the whole content of the initramfs
mkdir -p initramfs/{bin,proc,sys}

Copy necessary binaries

There are quite a few binaries we need, instead of manually including all these small components, we will use busybox, which staticly compiled and included many utils into one single file. You can download the busybox binary here, or compile from source if you like.⁹

# Download busybox-1.35.0 static binary
cd initramfs/bin/
curl -OL https://www.busybox.net/downloads/binaries/1.35.0-x86_64-linux-musl/busybox
chmod +x busybox

The busybox binary is a special executable that takes its first argument into account, including the filename. So, if you rename it or create a symbolic link to it using a Unix utility name, such as ls, it will behave exactly like that utility. Running busybox --list will print all the binary functions it supports.

With busybox, we can simply place it in the initramfs/bin/ directory and create symbolic links to it with different utility names.

for cmd in $(./busybox --list); do ln -s busybox $cmd; done

Create init script

busybox provides all the necessary binaries, but we still need to mount the sysfs and procfs, and most importantly, specify the first userspace program to execute. To handle these tasks, we will create an init script.

cat <<EOF > initramfs/init
#!/bin/sh
mount -t proc none /proc
mount -t sysfs none /sys
exec /bin/sh
EOF

The Linux kernel will run /init in the initramfs automatically after booting. So we need to make sure the script is executable.

chmod +x initramfs/init

Make initramfs image

We will use cpio to create the image file initramfs.cpio.gz.

cd initramfs
find . | cpio -H newc -o | gzip > ../initramfs.cpio.gz
cd ..

Run kernel in QEMU

This time, let’s try to boot the kernel with the initramfs.

qemu-system-x86_64 -kernel arch/x86/boot/bzImage -initrd initramfs.cpio.gz

If everything works as expected, you should see the boot sequence in a new window, and eventually a sh prompt. There may be some additional kernel messages that appear after the sh prompt is presented, so you can hit Enter a few times to ensure the sh shell is running properly.

As you can see, the uname output shows the custom build tag syscall_irl that we added during the configuration step. This confirms that we have successfully booted the kernel we compiled and have a functional environment to explore. It’s time to dive in and start doing something interesting!

Debug running kernel in QEMU

Attach GDB to QEMU

QEMU provides the capability to attach a GDB debugger to the program running within the QEMU environment. This allows us to use GDB to debug the Linux kernel and examine everything at runtime.

# Boot kernel with supporting gdb remote attach
qemu-system-x86_64 -kernel arch/x86/boot/bzImage -initrd initramfs.cpio.gz -append "nokaslr" -s -S

Explaination of the newly added flags:

• -append "nokaslr": add kernel commandline parameter nokaslr to disable KASLR (Kernel Address Space Layout Randomization), otherwise the GDB breakpoint may not work as expected.
• -s: a shorthand for -gdb tcp::1234, which listen on tcp port 1234 that GDB can attach remotely.
• -S: freeze the CPU at startup, so we can control the execution from GDB at the very beginning.

You will see the QEMU window prompt a message about “display not initialized”. This is because the execution is currently paused. Now, open a new terminal and attach GDB to the running QEMU instance.

$ gdb -q vmlinux
Reading symbols from vmlinux...
(gdb) target remote :1234  # Attach to QEMU gdbserver at port 1234
0x000000000000fff0 in exception_stacks ()
(gdb)

The execution has stopped at the address 0x000000000000fff0, which is the first instruction the CPU will execute.

The syscall entry

In the previous chapter, we learned that the CPU jumps to the address stored in the IA32_LSTAR register after the syscall instruction is executed. In Linux, this register is addressed as MSR_LSTAR, and the definition is located in the arch/x86/asm/msr-index.h header file.

// arch/x86/asm/msr-index.h
/* CPU model specific register (MSR) numbers. */

/* x86-64 specific MSRs */
#define MSR_EFER 0xc0000080 /* extended feature register */
#define MSR_STAR 0xc0000081 /* legacy mode SYSCALL target */
#define MSR_LSTAR 0xc0000082 /* long mode SYSCALL target */
#define MSR_CSTAR 0xc0000083 /* compat mode SYSCALL target */
#define MSR_SYSCALL_MASK 0xc0000084 /* EFLAGS mask for syscall */
#define MSR_FS_BASE 0xc0000100 /* 64bit FS base */
#define MSR_GS_BASE 0xc0000101 /* 64bit GS base */
#define MSR_KERNEL_GS_BASE 0xc0000102 /* SwapGS GS shadow */
#define MSR_TSC_AUX 0xc0000103 /* Auxiliary TSC */

From the comment, we can see that the LSTAR name stands for “Long mode Syscall Target”. There is also a STAR register used for legacy systems.

After a simple search through the kernel source, we can find that the MSR_LSTAR register is initialized in the syscall_init() function during the boot sequence.

void syscall_init(void)
{
	wrmsr(MSR_STAR, 0, (__USER32_CS << 16) | __KERNEL_CS);
	wrmsrl(MSR_LSTAR, (unsigned long)entry_SYSCALL_64); // The MSR_LSTAR was set to entry_SYSCALL_64

#ifdef CONFIG_IA32_EMULATION
	wrmsrl_cstar((unsigned long)entry_SYSCALL_compat);
	/*
	 * This only works on Intel CPUs.
	 * On AMD CPUs these MSRs are 32-bit, CPU truncates MSR_IA32_SYSENTER_EIP.
	 * This does not cause SYSENTER to jump to the wrong location, because
	 * AMD doesn't allow SYSENTER in long mode (either 32- or 64-bit).
	 */
	wrmsrl_safe(MSR_IA32_SYSENTER_CS, (u64)__KERNEL_CS);
	wrmsrl_safe(MSR_IA32_SYSENTER_ESP,
		    (unsigned long)(cpu_entry_stack(smp_processor_id()) + 1));
	wrmsrl_safe(MSR_IA32_SYSENTER_EIP, (u64)entry_SYSENTER_compat);
#else
	wrmsrl_cstar((unsigned long)ignore_sysret);
	wrmsrl_safe(MSR_IA32_SYSENTER_CS, (u64)GDT_ENTRY_INVALID_SEG);
	wrmsrl_safe(MSR_IA32_SYSENTER_ESP, 0ULL);
	wrmsrl_safe(MSR_IA32_SYSENTER_EIP, 0ULL);
#endif

	/*
	 * Flags to clear on syscall; clear as much as possible
	 * to minimize user space-kernel interference.
	 */
	wrmsrl(MSR_SYSCALL_MASK,
	       X86_EFLAGS_CF|X86_EFLAGS_PF|X86_EFLAGS_AF|
	       X86_EFLAGS_ZF|X86_EFLAGS_SF|X86_EFLAGS_TF|
	       X86_EFLAGS_IF|X86_EFLAGS_DF|X86_EFLAGS_OF|
	       X86_EFLAGS_IOPL|X86_EFLAGS_NT|X86_EFLAGS_RF|
	       X86_EFLAGS_AC|X86_EFLAGS_ID);
}

At line 4, the MSR_LSTAR register is set to the symbol entry_SYSCALL_64. Now, let’s set a breakpoint in GDB at the syscall_init() and continue the boot sequence to see what this entry_SYSCALL_64 symbol represents.

(gdb) b syscall_init
Breakpoint 1 at 0xffffffff81047020: file arch/x86/kernel/cpu/common.c, line 2073.
(gdb) c
Continuing.

Breakpoint 1, syscall_init () at arch/x86/kernel/cpu/common.c:2073
2073    {
(gdb) n
2074            wrmsr(MSR_STAR, 0, (__USER32_CS << 16) | __KERNEL_CS);
(gdb) n
2075            wrmsrl(MSR_LSTAR, (unsigned long)entry_SYSCALL_64);
(gdb) info symbol entry_SYSCALL_64
entry_SYSCALL_64 in section .text

From the symbol information, we can see that entry_SYSCALL_64 is a symbol located in the .text section, which means it is the entry point for the 64-bit system call at the address 0xffffffff82000040. We can use GDB to find the corresponding source code line information for this symbol.

(gdb) info line entry_SYSCALL_64
Line 89 of "arch/x86/entry/entry_64.S" starts at address 0xffffffff82000040 <entry_SYSCALL_64>
and ends at 0xffffffff82000044 <entry_SYSCALL_64+4>.

So GDB has informed us that the entry_SYSCALL_64 symbol is defined at line 89 of the file arch/x86/entry/entry_64.S, which contains some assembly code. Let’s walk through it line by line, but to keep the content focused, I will skip over some unrelated parts.

SYM_CODE_START(entry_SYSCALL_64)
	UNWIND_HINT_ENTRY
	ENDBR

The SYM_CODE_START is a assembler annotation for interrupt handlers and similar where the calling convention is not the C one¹⁰. It defines entry_SYSCALL_64 as a function-like symbol(i.e. the address of a procedure) that the CPU can jump to. In our case, this is the first code executed in kernel space when a system call occurs. The following lines are some compiler related details, which we will not discuss here.

	swapgs

The gs register is a limited form of segmentation register, where only the base address(GSBase) is needed to calculate the effective address¹¹, similar to a pointer to an array. So, when reading from gs[:0x10], the actual address is GSBase + 0x10. In the arch/x86/asm/msr-index.h file we saw earlier, there are two GSBase registers: MSR_GS_BASE and MSR_KERNEL_GS_BASE. The gs register uses the value in MSR_GS_BASE as the GSBase, and the swapgs instruction swaps the values in MSR_GS_BASE and MSR_KERNEL_GS_BASE, effectively changing the GSBase value. The MSR_KERNEL_GS_BASE stores the address of the kernel’s per-CPU structure, so after the swapgs instruction, we can suddenly access some kernel data structures using the gs register.

	/* tss.sp2 is scratch space. */
	movq	%rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2)

Save the value in the rsp register to a per-CPU scratch space tss.sp2¹², so that we can use the rsp register for other purposes.

	SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp

This is another crucial part of the system call entry process: switching the pagetable from the userspace pagetable to the kernel’s pagetable. CR3¹³ is a control register that holds the physical address of the top-level pagetable (PML4 in x86_64). The SWITCH_TO_KERNEL_CR3 macro will clear the PTI_USER_PCID_BIT and PTI_USER_PGTABLE_BIT to switch the CR3 register to the kernel’s pagetables.

You may wonder how could this code even be executed before switching the address space, won’t the address of these instructions be somewhere in user address space? Well, this behavior is exactly by design in how the kernel and userspace interaction is managed. In x86_64 Linux, the virtual address space is typically split between userspace and kernel space¹⁴. The upper half (usually from 0xffff800000000000 and above) is always reserved for the kernel and mapped to a portion of the kernel area. This means some kernel code, including the syscall entry, is located in the user address space and can be executed before and after switching to the kernel pagetable, as they have the same virtual address in both user and kernel pagetable.

	movq	PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp

The PER_CPU_VAR(pcpu_hot + X86_top_of_stack) expression gives the address of the per-CPU kernel stack. By setting the stack pointer register rsp to this address, we can now use stack-related instructions like push, pop, and call within the kernel space.

	/* Construct struct pt_regs on stack */
	pushq	$__USER_DS				/* pt_regs->ss */
	pushq	PER_CPU_VAR(cpu_tss_rw + TSS_sp2)	/* pt_regs->sp */
	pushq	%r11					/* pt_regs->flags */
	pushq	$__USER_CS				/* pt_regs->cs */
	pushq	%rcx					/* pt_regs->ip */
SYM_INNER_LABEL(entry_SYSCALL_64_after_hwframe, SYM_L_GLOBAL)
	pushq	%rax					/* pt_regs->orig_ax */

	PUSH_AND_CLEAR_REGS rax=$-ENOSYS

This is what a “context switch” literaly means: saving the current register state onto the stack. The order in which the registers are pushed onto the stack is crucial, as it will construct a struct pt_regs on the stack, as the comment indicates.

struct pt_regs {
/*
 * C ABI says these regs are callee-preserved. They aren't saved on kernel entry
 * unless syscall needs a complete, fully filled "struct pt_regs".
 */
	unsigned long r15;
	unsigned long r14;
	unsigned long r13;
	unsigned long r12;
	unsigned long bp;
	unsigned long bx;
/* These regs are callee-clobbered. Always saved on kernel entry. */
	unsigned long r11;
	unsigned long r10;
	unsigned long r9;
	unsigned long r8;
	unsigned long ax;
	unsigned long cx;
	unsigned long dx;
	unsigned long si;
	unsigned long di;
/*
 * On syscall entry, this is syscall#. On CPU exception, this is error code.
 * On hw interrupt, it's IRQ number:
 */
	unsigned long orig_ax;
/* Return frame for iretq */
	unsigned long ip;
	unsigned long cs;
	unsigned long flags;
	unsigned long sp;
	unsigned long ss;
/* top of stack page */
};

Since the stack grows from higher to lower addresses, we need to push the registers onto the stack in the reverse order of the fields defined in struct pt_regs. The PUSH_AND_CLEAR_REGS macro will push the registers from di to r15 onto the stack and then clear them. It also sets the rax register (used for the return value) to -ENOSYS as a default return value.

	/* IRQs are off. */
	movq	%rsp, %rdi
	/* Sign extend the lower 32bit as syscall numbers are treated as int */
	movslq	%eax, %rsi

	/* clobbers %rax, make sure it is after saving the syscall nr */
	IBRS_ENTER
	UNTRAIN_RET

	call	do_syscall_64		/* returns with IRQs disabled */

Syscall handler

Now, a complete struct pt_regs has been constructed on the stack, which means the rsp register is now a pointer to the saved registers struct pt_regs*. Then, following the C calling convention, the first two function arguments rdi and rsi are set to rsp and eax, respectively, which are the struct pt_regs* and the system call number. The do_syscall_64 function is the system call handler, and we can use GDB to find the source code line information for this function.

(gdb) info line do_syscall_64
Line 74 of "arch/x86/entry/common.c" starts at address 0xffffffff81ec2550 <do_syscall_64>
and ends at 0xffffffff81ec2554 <do_syscall_64+4>.
(gdb)

The defination of do_syscall_64 is below:

__visible noinstr void do_syscall_64(struct pt_regs *regs, int nr)
{
	add_random_kstack_offset();
	nr = syscall_enter_from_user_mode(regs, nr);

	instrumentation_begin();

	if (!do_syscall_x64(regs, nr) && !do_syscall_x32(regs, nr) && nr != -1) {
		/* Invalid system call, but still a system call. */
		regs->ax = __x64_sys_ni_syscall(regs);
	}

	instrumentation_end();
	syscall_exit_to_user_mode(regs);
}

The parameter list of do_syscall_64 is exactly what we see in the assembly code. Skipping the trace points, the syscall handler logic is actually defined in do_syscall_x64:

static __always_inline bool do_syscall_x64(struct pt_regs *regs, int nr)
{
	/*
	 * Convert negative numbers to very high and thus out of range
	 * numbers for comparisons.
	 */
	unsigned int unr = nr;

	if (likely(unr < NR_syscalls)) {
		unr = array_index_nospec(unr, NR_syscalls);
		regs->ax = sys_call_table[unr](regs);
		return true;
	}
	return false;
}

This handler uses the unsigned system call number to index into an array called sys_call_table, which contains the actual system call functions. The sys_call_table is defined as follows:

// arch/x86/entry/syscall_64.c
#define __SYSCALL(nr, sym) extern long __x64_##sym(const struct pt_regs *);
#include <asm/syscalls_64.h>
#undef __SYSCALL

#define __SYSCALL(nr, sym) __x64_##sym,

asmlinkage const sys_call_ptr_t sys_call_table[] = {
#include <asm/syscalls_64.h>
};

The sys_call_table lists all the syscalls and construct an array for indexing them. The full list is defined in the file arch/x86/include/generated/asm/syscalls_64.h, but it’s a compile-time generated header file, so we need to compile the kernel first to see its content, which we have done previously. In my kernel build, there are 454 syscalls in this header file. The first four syscalls are listed below:

__SYSCALL(0, sys_read)
__SYSCALL(1, sys_write)
__SYSCALL(2, sys_open)
__SYSCALL(3, sys_close)

We can find the actual symbol for the write system call by expanding the __SYSCALL macro, which reveals that the symbol is __x64_sys_write. We can then use GDB to find the source code line information for this function.

(gdb) info line __x64_sys_write
Line 646 of "fs/read_write.c" starts at address 0xffffffff8128b0d0 <__x64_sys_write> and ends at 0xffffffff8128b0d4 <__x64_sys_write+4>.
(gdb)

And the code in fs/read_write.c:

SYSCALL_DEFINE3(write, unsigned int, fd, const char __user *, buf,
		size_t, count)
{
	return ksys_write(fd, buf, count);
}

The SYSCALL_DEFINE3 macro reads the system call arguments from the registers, following the C calling convention, and then calls the ksys_write function, which is the final system call handler.

// fs/read_write.c
ssize_t ksys_write(unsigned int fd, const char __user *buf, size_t count)
{
	struct fd f = fdget_pos(fd);
	ssize_t ret = -EBADF;

	if (f.file) {
		loff_t pos, *ppos = file_ppos(f.file);
		if (ppos) {
			pos = *ppos;
			ppos = &pos;
		}
		ret = vfs_write(f.file, buf, count, ppos);
		if (ret >= 0 && ppos)
			f.file->f_pos = pos;
		fdput_pos(f);
	}

	return ret;
}

Now we have reached the actual logic of the write system call. We can set a breakpoint on the ksys_write function to see when it is called and what arguments are passed.

Breakpoint on ksys_write()

Since the write system call is used for almost all output to the screen, setting a breakpoint on it will likely result in many noise hits from various processes. To get a clearer understanding, we can continue execution until the sh shell is running, then set the breakpoint on ksys_write and run the echo syscall_irl command to observe what happens.

(gdb) c
Continuing.  // Wait sh to be executed
^C           // Press Ctrl+C to interrupt the program and set the breakpoint at ksys_write
Program received signal SIGINT, Interrupt.
default_idle () at arch/x86/kernel/process.c:743
743             raw_local_irq_disable();
1: $rip = (void (*)()) 0xffffffff81ec790f <default_idle+15>
(gdb) b ksys_write
Breakpoint 2 at 0xffffffff8128afd0: file fs/read_write.c, line 627.
(gdb) c
Continuing.  // Run command echo syscall_irl

Breakpoint 2, ksys_write (fd=3, buf=0x7f9abf5b8040 "echo syscall_irl\n", count=17) at fs/read_write.c:627
627     {
1: $rip = (void (*)()) 0xffffffff8128afd0 <ksys_write>
(gdb) c
Continuing.

Breakpoint 2, ksys_write (fd=1, buf=0x7f9abf5b9950 "syscall_irl\n", count=12) at fs/read_write.c:627
627     {
1: $rip = (void (*)()) 0xffffffff8128afd0 <ksys_write>

The breakpoint was hit as expected! However, there seems to be something unusual: two write system calls were made. In the first one, the fd argument is 3, and the buf argument is “echo syscall_irl\n”. This is weird, because the typical file descriptors for printing are stdin(0) or stderr(2). The second write call appears to be the one we expected. So 3 must be a file opened by sh itself, we can use the strace utility to trace the system calls made by the sh shell and find out what the file descriptor 3 is.

strace busybox sh
# Some verbose output...
$ echo syscall_irl
open("$HISTFILE", O_WRONLY|O_CREAT|O_APPEND, 0600) = 3 # I replace the actually path with $HISTFILE
lseek(3, 0, SEEK_END)                   = 9
write(3, "echo syscall_irl\n", 17)      = 17
close(3)                                = 0

So the first write call with file descriptor 3 is related to the command history feature of the sh shell. We can ignore this by adding a condition to the breakpoint in GDB:

(gdb) del 2 // Delete the #2 breakpoint for ksys_write we set previously

(gdb) b ksys_write if fd == 1 // Break only when fd is 1(stdout)

Breakpoint 3 at 0xffffffff8128afd0: file fs/read_write.c, line 627.
(gdb) c

Continuing. // Run command echo syscall_irl again

Breakpoint 3, ksys_write (fd=1, buf=0x7f82e39b7970 "syscall_irl\n", count=12) at fs/read_write.c:627
627     {
1: $rip = (void (*)()) 0xffffffff8128afd0 <ksys_write>
(gdb) c

Continuing. // "syscall_irl" is printed.

Now the GDB breakpoint will only trigger when the fd argument of ksys_write is 1. If we continue the execution, we should see the syscall_irl is printed to the console.

You may wondering why the prompt of sh does not hit the breakpoint, but if you look at the previous output of strace carefully, you will notice that the prompt is actually printed through the syscall writev, not write, that’s why it won’t hit.

Return to userspace

It should be clear now how system call is implemented from user to kernel space, to return from kernel to userspace process, just reverse what we have done when entering the kernel.

	call	do_syscall_64		/* returns with IRQs disabled */

	// Skip some checks here

	/*
	 * We win! This label is here just for ease of understanding
	 * perf profiles. Nothing jumps here.
	 */
syscall_return_via_sysret:
	IBRS_EXIT
	POP_REGS pop_rdi=0 // restore from r15 to rsi

	/*
	 * Now all regs are restored except RSP and RDI.
	 * Save old stack pointer and switch to trampoline stack.
	 */
	movq	%rsp, %rdi
	movq	PER_CPU_VAR(cpu_tss_rw + TSS_sp0), %rsp
	UNWIND_HINT_END_OF_STACK

	pushq	RSP-RDI(%rdi)	/* RSP */
	pushq	(%rdi)		/* RDI */

	/*
	 * We are on the trampoline stack.  All regs except RDI are live.
	 * We can do future final exit work right here.
	 */
	STACKLEAK_ERASE_NOCLOBBER

	SWITCH_TO_USER_CR3_STACK scratch_reg=%rdi // Switch back to user pagetable

	popq	%rdi // restore rdi
	popq	%rsp // restore rsp
SYM_INNER_LABEL(entry_SYSRETQ_unsafe_stack, SYM_L_GLOBAL)
	ANNOTATE_NOENDBR
	swapgs // swap to user GSBase
	sysretq // return from kernel

After the do_syscall_64 function returns, the context switch happens again to restore the userspace context. Then, the pagetable is switched back to the user pagetable, and the GSBase is also swapped to the user base address. Finally, the sysretq instruction is executed, which performs the reverse operations of the syscall instruction.

The full operations of sysret can be found at here, and there are only few matter:

IF (operand size is 64-bit)
    THEN (* Return to 64-Bit Mode *)
        IF (RCX is not canonical) THEN #GP(0);
        RIP := RCX; (* Restore the next instruction address in the userspace *)
    ELSE (* Return to Compatibility Mode *)
        RIP := ECX; (* Same but in 32-bit *)
FI;
RFLAGS := (R11 & 3C7FD7H) | 2; (* Clear RF, VM, reserved bits; set bit 1 *)
CPL := 3; (* Set current protect ring to 3, the userspace *)

After the sysretq instruction, the CPU will continue executing the instruction that comes after the original syscall instruction in userspace. This marks the end of our journey through the system call process.

Summary

In this blog post, we first introduced some background knowledge about the Application Binary Interface(ABI) and the C Calling Convention. Then, we wrote a simple program that calls the write system call and used GDB to walk through the assembly code before jumping into the kernel space.

Next, we built a bootable kernel image from source and created an initramfs for it. Finally, we used QEMU and GDB to investigate what happens in the kernel space and how the execution returns from the kernel to user space.

Throughout this journey, we uncovered the whole picture of the system call mechanism and the interaction between userspace and kernel.

**Protocol Buffers(protobuf)**有两种序列化编码方式：二进制格式(wire format) 和 文本格式(text format)

其中wire format是protobuf的默认编码格式，可以高效地通过网络传输； text format是一种类似JSON的表示方式，可以方便地用于调试和手动编写/编辑消息。

这篇文章记录一下wire format的编码格式

这篇Blog是protobuf系列的第二篇，整个系列目录如下

• Protocol Buffers系列：格式介绍(一)
• Protocol Buffers系列：编码方式(二)

Base128 Varints

变长整数varint允许在不同范围内的整数编码结果具有不同的字节长度， 且整数越小字节长度越短(仅对非负数而言)

在varint二进制表示中，每个字节都有一个continuation bit(CB)用于指示后续是否还有连续的字节， 以及一个7-bit的payload。由于7-bit能表示的范围是$[0,127]$，所以这种编码方式也叫做Base128编码

对于一个64位的整数，要完整表示其64个bit则最少需要10个这样的字节， 因此varint类型的整数最多需要10个字节来表示，但最少只需要1个字节就可以表示， 因此在编码范围不定的非负整数时varint类型具有非常高的空间效率。

把这些连续字节的所有payload按little-endian的顺序组合到一起得到的结果就是该整数的二进制表示

比如，整数1只需要1bit就能表示，因此只需要一个字节，且其CB为0

0 0000001
^ CB

0000001即为最后的结果1的二进制表示

整数150最少需要8个bit表示，由于一个字节的payload只有7个bit， 所以需要两个字节来表示，并且第一个字节的CB为1，第二个字节的CB为0

1 0010110  0 0000001
^ CB       ^ CB

将这两个字节的payload按照little-endian顺序组合到一起得到的00000010010110 就是最后的结果150的二进制表示，具体操作如下

10010110 00000001        // Original bits.
 0010110  0000001        // Drop continuation bits.
 0000001  0010110        // Put into little-endian order.
 10010110                // Concatenate.
 128 + 16 + 4 + 2 = 150  // Interpret as integer.

负数编码

intN类型使用补码的方式编码负数，因此任何负数都需要使用全部的10个字节来表示，空间效率很低

sintN类型使用ZigZag方式编码负数，把LSB作为signed bit， 即正数n编码为2 * n，负数-n编码为2 * n + 1，这样虽然正数需要额外的1bit来表示， 但是解码一个负数并不需要知道该类型所能表示的最大范围， 这样不是所有的负数都需要用到全部的10个字节，相比补码方式能节省更多的空间

实际上，如果按照2 * n + 1来计算负数，会发现0和1都表示0， 所以Protobuf将负数往前移了一位，即使用2 * n - 1来计算负数(n > 0)

其他数字类型

Field type为bool和enum的value都按照int32类型来编码， 并且bool类型的值总是0或1，其protoscope别名为false和true

Protoscope

为了增加二进制编码的可读性，protobuf使用一种叫做Protoscope的描述语言来描述wire format的二进制表示。其包括以下语法：

Value

• `70726f746f6275660a`: 表示该十六进制字符串对应的字节序列
• 150: 表示一个varint类型的数字150，其等效于该varint的底层二进制表示`9601`
• "Hello, Protobuf!": 表示一个UTF-8字符串

Field

Wire format编码的field使用Type-Length-Value(TLV)结构，用protoscope表示就是

<field number>:<wire type> <value>

wire type用于指示value的长度，不同的field类型会使用不同的wire type，其共有六种：

• [0]VARINT: int32, int64, uint32, uint64, sint32, sint64, bool, enum
• [1]I64: fixed64, sfixed64, double
• [2]LEN: string, bytes, embedded messages, packed repeated fields
• [3]SGROUP: group start(deprecated)
• [4]EGROUP: group end(deprecated)
• [5]I32: fixed32, sfixed32, float

比如1:VARINT 150表示wire type为VARINT，field number为1的field，且其值为150

此外，当wire type为LEN时，后面还会跟一个varint用于表示value的长度

比如2:LEN 16 {"Hello, Protobuf!"}，其表示一个wire type为LEN，field number为2的field, 且其字节长度为16，值为"Hello, Protobuf!"

类型推断

使用protoscope表示的field不一定要表明wire type，可以通过value的形式隐含地指示wire type，比如

• 当value是一个varint类型的数字123时，其wire type为VARINT
• 当value的第一个字符为{时，其wire type为LEN
• 当value显式标注了类型后缀时，比如5i32的wire type为I32、3.1415i64的wire type为I64等等

Message

Protobuf message中的每个field都可以看作是一个key-value pair， 其中key是该field的field number，value就是该field的值。 每个field的名称和类型只在解码的时候才会决定，wire format本身不包含名称和类型信息。

在编码消息时，这样的一个key-value pair称为一个record，其具体表现形式就是上面提到的TLV结构。

在一个record中field number和wire type会编码在同一个varint中， 这个varint称为这个record的tag，并且该varint的lower 3-bit就是wire type， 其余的为field number

例如下面这个record

1:VARINT 150

表示其field number和wire type的varint是`08`，拆开来看就是

0    0001           000
^    ^^^^           ^^^
CB   field number   wire type([0]VARINT)

这个record加上value的完整表示如下

|   tag   |   | value: varint 150 |
0  0001 000   1 0010110   0 0000001
^  ^^^^ ^^^   ^           ^
CB FN   WT    CB          CB

Length-Delimited Records

Wire type为LEN的field使用一个额外的varint来表示其value的字节长度， 因此这种表示方式叫做Length-Delimited

例如，对于下面这个record

2:LEN 5 {"hello"}

其二进制表示如下

|   tag   |  | Len 5 |  |    string "hello"    |
0  0010 010  0000 0101  0x68 0x65 0x6c 0x6c 0x6f
^  ^^^^ ^^^  ^
CB FN   WT   CB

对于其他wire type为LEN的类型也是一样的，其value就是该类型的二进制表示

Record Order

Field number与该field对应的record在编码结果中的位置没有关系。 此外，序列化的格式也并不是固定的，protobuf只保证被编码的对象和解码出的对象对同一个.proto生成的代码具有相同的值， 因此不能对编码后的wire format内容做任何假设

repeated

类型为repeated的field的每一个元素都会按照singular方式编码为一个record， 此时编码结果中有多个具有相同key的record

例如，对于以下field

repeated int32 arr = 3;

当arr = [1,2,3]时，会被编码为

3: 1
3: 2
3: 3

在编码repeated类型的时候不需要保证其所有元素的record都分布在一起， 其他field的record可以穿插在这些元素的record之间。 只有具有相同key的record之间的相对顺序会被作为该repeated类型field元素之间的顺序

Packed Encoding

在proto3中，使用VARINT、I32或I64作为元素wire type的repeated类型默认会以"packed"的形式编码， 即用一个wire type为LEN的record代替多个wire type为VARINT、I32或I64的record

例如，上面的arr用"packed"的方式可以编码为如下形式

3: {1 2 3}

同样，以packed形式编码的field也可以有多个相同key的record， 这些record的值在解码的时候会被拼接在一起

map

如果一个map类型的field的定义如下

map<string, int32> count = 4;

则其与以下定义的编码方式相同

message count_Entry {
    optional string key = 1;
    optional int32 value = 2;
}
repeated count_Entry count = 4;

oneof

oneof类型的field编码方式与message相同，但一个oneof只能有一个field的record

repeated to singluar

当反序列化的时候某个singular类型的field出现了多个record， 即该field表现出repeated的性质，则按照以下方式处理

1. scalar类型会取最后一个record的值作为该field的值
2. embeded message会合并，即递归按照以下方式处理该embeded message中的每个field
   • singular scalar按照(1)处理
   • singular embedded message按照(2)处理
   • repeated类型会把值拼接到一起

Reference Card

以下是wire format的一个速查表

message    := (tag value)*

tag        := (field << 3) bit-or wire_type;
                encoded as varint
value      := varint      for wire_type == VARINT,
              i32         for wire_type == I32,
              i64         for wire_type == I64,
              len-prefix  for wire_type == LEN,
              <empty>     for wire_type == SGROUP or EGROUP

varint     := int32 | int64 | uint32 | uint64 | bool | enum | sint32 | sint64;
                encoded as varints (sintN are ZigZag-encoded first)
i32        := sfixed32 | fixed32 | float;
                encoded as 4-byte little-endian;
                memcpy of the equivalent C types (u?int32_t, float)
i64        := sfixed64 | fixed64 | double;
                encoded as 8-byte little-endian;
                memcpy of the equivalent C types (u?int32_t, float)

len-prefix := size (message | string | bytes | packed);
                size encoded as varint
string     := valid UTF-8 string (e.g. ASCII);
                max 2GB of bytes
bytes      := any sequence of 8-bit bytes;
                max 2GB of bytes
packed     := varint* | i32* | i64*,
                consecutive values of the type specified in `.proto`

最近组里的项目需要自己写一套消息序列化协议来代替protobuf， 所以顺便研究了一下protobuf的原理，以便在造轮子的时候借鉴一下其设计上的优雅之处。

首先介绍一下Protocol Buffers， 这是由Google开发的一种无关语言、无关平台、高可扩展性、轻量级的结构化数据序列化格式。 它使用一种模式定义语言来描述数据结构，可以通过protoc编译器将其编译成对应语言的代码。

Protocol Buffers数据格式在不同语言和平台间具有高度一致性， 生成的代码也具有相似的功能，因此其通常作为分布式组件和微服务之间数据通信的序列化协议。

这篇Blog是protobuf系列的第一篇，整个系列目录如下

• Protocol Buffers系列：格式介绍(一)
• Protocol Buffers系列：编码方式(二)

–

代码生成

Protocol Buffers的工作流程如下所示

file.proto => protoc => language-specified code

其中protoc是用来将.proto文件编译成对应语言平台代码的编译器。在编译时需要安装对应语言的编译器后端

以Go为例

# 安装用于生成Go代码的编译器后端protoc-gen-go
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
# 在DST_DIR目录中生成对应的Go代码
protoc --proto-path=<IMPORT_PATH> --go_out=<DST_DIR> path/to/file.proto

示例目录结构如下

.
├── go.mod
├── proto
│   └── example.proto
└── src
    ├── cmd
    │   └── main.go
    └── model

其中example.proto的内容为

syntax = "proto3";
example.

执行命令

```shell
# 注意，由于没有导入其他.proto文件，所以这里省略了--proto-path参数
protoc --go_out=src proto/example.proto

此时目录结构如下

.
├── go.mod
├── proto
│   └── example.proto
└── src
    ├── cmd
    │   └── main.go
    └── model
        └── example.pb.go

可以看到，./src/model目录中新增了一个名为example.pb.go的文件，该文件内容大致如下

package model

// ...

type Hello struct {
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	Id   uint32 `protobuf:"varint,1,opt,name=id,proto3" json:"id,omitempty"`
	Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"`
}

// ...

这时在代码中就可以通过model.Hello访问这个结构体

其他语言可以用以下参数来生成对应的代码

• --cpp_out=<DST_DIR> generates C++ code in DST_DIR.
• --java_out=<DST_DIR> generates Java code in DST_DIR.
• --kotlin_out=<DST_DIR> generates additional Kotlin code in DST_DIR.
• --python_out=<DST_DIR> generates Python code in DST_DIR.
• --ruby_out=<DST_DIR> generates Ruby code in DST_DIR.
• --objc_out=<DST_DIR> generates Objective-C code in DST_DIR.
• --csharp_out=<DST_DIR> generates C# code in DST_DIR.
• --php_out=<DST_DIR> generates PHP code in DST_DIR.

.proto 文件格式

syntax = <proto.version>;

import *public "other_proto_file_path"; // 可以导入其他.proto文件来使用其中的定义

**package PackageName; // 定义该.proto文件的命名空间

**option FileOption;

message MessageName {
  // 将某个FieldName或FieldNumber设为保留值，这两种类型的保留值需写在不同的reserved语句中
  **reserved FieldName...;
  **reserved FieldNumber...;

  // 默认为singular, optional通常用于指针类型
  *optional *repeated FieldType FieldName = FieldNumber;

  // map类型
  *optional <KeyType, ValueType> FieldName = FieldNumber;
}

enum EnumName {
  **option allow_alias = true; // 允许不同的case具有相同的enum number
  CaseName = EnumNumber;
}

// 只能定义在message内部
oneof OneofName {
  // 不能出现repeated和map类型的field
  FieldType FieldName = FieldNumber;
}

• 在一个结构内部可以嵌套定义其他的结构，而且嵌套层数可以无限深，并且可以从外部通过__Parent__.__Type__访问
• map类型实际上是一个别名，以下两种定义是一样的

// Define 1
message MapFieldEntry {
  optional key_type key = 1;
  optional value_type value = 2;
}
repeated MapFieldEntry map_field = N;

// Define 2
map<key_type, value_type> map_field = N;

标量类型

Numeric

Others

默认值

在反序列化过程中，缺失的field会被设置为默认值，同样，消息对象中具有默认值的field不会被序列化。 对于不同的field类型，其默认值定义如下

• string: 空字符串
• bytes: 空字节序列
• bool: False
• numeric: 0/0.0
• enum: 第一个enum case
• composite: language-dependent

注意：protobuf在反序列化时无法区分一个field的值被显式设置为了默认值还是没有设置， 所以不要把这个默认值作为重要的判断条件

复合类型

message类型

Example

message Person {
  string name = 1;
  int32 id = 2;
  repeated string email = 3;
  optional Organization org = 4;
}

Field Number

Field number用于在反序列化时识别不同的field，从1开始。 为了保证兼容性，在更新.proto文件时不要修改已有field的field number， 而是以增量更新的方式为新创建或更新的field分配一个新的field number。

当需要删除某个field时，需要将其对应的field number或field name声明在reserved语句中， 以避免未来的更新可能会重复使用这一field number或field name (如果不需要考虑兼容性可以不使用reserved声明)

注意：field number和field名必须声明在不同的reserved语句中

其中field number的范围为[1,2^29-1]，此外[19000,19999]是protobuf保留范围，不能使用。 可以通过FieldDescriptor::kFirstReservedNumber和FieldDescriptor::kLastReservedNumber 获取这两个上下限。声明为reserved的field number也不能使用

不同范围的field number会被编码成不同长度的字节串，其中1-15范围内的field number只占用1字节， 因此应该将1-15分配给最有可能不是默认值的field,并且还要考虑到将来可能出现的符合这种条件的field

Field Type

message支持以下field类型

• singular 非数组类型，该类型为默认类型，无需显式声明
• repeated 有序数组类型，长度可以为0
• optional 通常用于指针类型
• map

enum类型

• enum的EnumNumber从0开始，并且0对应的case为默认值
• 当设置option allow_alias = true;时，多个case可以有同一个EnumNumber
• enum使用int32编码，因此一个enum结构中case的数量不能超过int32所能表示的范围， 此外有些语言也限制了一个enum结构所能包含的case数量

在反序列化过程中，未定义的case在不同的语言会有不同的表示

• open enum type语言(如C++, Go)中会被直接储存为以该值为底层表示的enum对象，
• closed enum type语言(如Java)中会有一个特殊的unrecognized case来表示未定义的值

enum类型的reserved语法为

reserved EnumNumber, RangeBegin to RangeEnd, RangeBegin to max;
reserved CaseName;

oneof类型

oneof严格来说并不算是一种类型，因为它只能定义在message内部， 并且会占用该message的field number，它只能用来说明这个message中的某几个field只会出现其中一个

• 如果重复设置某个oneof类型的值，只会保留最后一次设置的值
• 反序列化时如果某个oneof有多个类型的值，则只取最后一个
• oneof本身不能是repeated
• 即使一个oneof的值是默认值，其也会被序列化
• 更新一个oneof结构时会遇到很多兼容性的问题，如果一定要更新确保仔细阅读文档避免出现问题

map类型

• key_type只能是整数类型(不包括enum和bool类型, 即使他们是兼容的)或者string类型
• value_type不能是map类型
• map类型的field不能是repeated类型
• map类型在序列化到wire format时其键值对的顺序取决于具体语言的实现；序列化到text format时会按照key进行排序
• 从wire format反序列化时重复出现的键值对只会保留最后一个，从text format反序列化时重复的key会报错

更新定义

• 不要修改已有field的field number
• 删除某个field时需确保field number不会被复用，可以通过reserved语句或者添加OBSOLETE_前缀来实现
• 未知的field会被跳过或储存在名为unknown的field中

类型兼容性

类型之间兼容意味着一个类型的编码结果可以正确的解码出另一个类型的值

• 这些集合内部是相互兼容的:
  • varint = {int32, uint32, int64, uint64}
  • {varint, bool} {varint, enum}
  • {sint32, sint64} {fixed32, sfixed32} {fixed64, sfixed64}
• 只要string和bytes都是合法的UTF-8字符串，那么他们也是兼容的
• 如果一个bytes是某个message的序列化表示，那么其与这个message也是兼容的
• string、bytes和message的singular类型与其对应的repeated类型是兼容的
  • primitive类型会取该repeated中的最后一个元素
  • message类型会将所有元素合并
• 由于repeated标量数字类型使用packed序列化格式，无法从repeated类型中解析出正确的singular值
• optional和oneof是二进制兼容的，但是在语言层面可能不兼容

Service Type

service类型用于定义RPC接口，

service <ServiceName> {
  rpc <RPCName>(<parameter type>) returns (<return type>);
}

service类型定义了一个RPC服务和其支持的RPC接口，需要对应的编译器来生成对应的RPC代码，参考gRPC

TCP是面向连接的协议，因此在数据传输之前需要先建立连接状态，传输完成后需要断开连接。

在建立连接、数据传输和断开连接时TCP会发生一系列同步过程，TCP连接的状态也随之发生变化， 因此可以将TCP连接看作一个状态机模型。 事实上，TCP标准为TCP连接规定了若干种状态，并且随着各个过程发生转换。

建立连接: 三步握手

在TCP连接中，连接双方符合C/S通信模型，即客户端主动发起建立连接的请求， 服务器则时刻监听其服务端口以便能够及时收到客户端的连接请求并完成建立连接的过程。 实际上TCP连接双方都有可能发起TCP连接，因此这里的C/S只是一个相对的概念， 即C端表示发起连接的一方，S端表示接收连接请求的一方。

要完成建立连接，需要通信双方(即C端和S端)互相进行共计3次通信，因此这三次通信过程也叫做三步握手 (Three-way handshake)。

监听端口

并不是所有主机之间都能够直接进行通信，而是需要S端首先对通信端口(Port)进行监听(Listen)， 然后才能收到来自发送端的消息。接收端监听的端口称为开放服务端口或开放端口， 大部分应用层开源协议都会与某个端口号相关联并规定在标准文件中，这些端口也称为对应协议的端口。 比如HTTP端口(80)、HTTPS端口(443)、SSH端口(22)等等。

端口协议列表详见[IANA]Service Name and Transport Protocol Port Number Registry

由于不是所有应用层协议都以TCP为基础，比如有些应用层协议使用UDP或SCTP， 因此开放端口并不都提供TCP服务，有的端口只提供UDP服务，还有的端口能同时提供多种协议的服务。

当TCP的S端开始监听其服务端口后，其状态从CLOSED切换为LISTEN

发送连接请求C -> S

C端首先向S端发送一封SYN包，表示发起TCP连接请求。该SYN包具有以下属性

• 设置了SYN标志位
• 序号为ISS(Initial Send Sequence number)
• 不携带数据

状态切换：CLOSED->SYN_SENT

回应连接请求C <- S

S端在接收到来自C端的SYN包后，将判断该连接请求是否有效。

若请求无效(比如目标端口不提供TCP服务)，则回应设置了RST标志的RST包，表示拒绝(Reject)连接。

若请求有效，则回应SYN-ACK包，表示同意建立连接。SYN-ACK包具有以下属性

• 设置了SYN和ACK标志位
• 序号为IRS(Initial Receive Sequence number)
• 确认号为SYN包的SEG.SEQ+1
• 不携带数据

状态切换：LISTEN->SYN_RCVD

此时TCP连接处于半连接(Half-Open Connection)状态。

在收到合法的连接请求后，S端往往会为本次TCP连接分配各种资源，如缓冲区(buffer)和各种变量。 这种在连接建立之前就分配资源的行为会导致TCP的安全问题，比如SYN洪水(SYN flood)， 是拒绝服务攻击(Denial of Service, DoS)的一种形式。 因此目前的TCP的S端通常会使用叫做SYN cookie的技术来抵御SYN洪水。

连接建立C -> S

若C端收到RST包，则中断(Abort)建立连接的过程，将状态重置为CLOSED。

若C端收到SYN-ACK包，则回应的ACK包，表示TCP连接建立完成。ACK包具有以下属性

• 设置了ACK标志位
• 设置确认号为SYN-ACK包的SEG.SEQ+1
• 可能携带数据

状态切换：SYN_SENT->ESTABLISHED

C端在建立连接后将为本次连接分配资源。

S端在收到该ACK包后进行状态切换：SYN_RCVD->ESTABLISHED

当双方均处于ESTABLISHED状态时，连接建立完成，可以开始数据传输。 此时SND.NXT为ISS+1，即第一个数据包的SEG.SEQ为ISS+1

注意，TCP数据传输可能发生在建立连接时期，但TCP层必须在连接建立完成后才能将数据上传给应用层， 在此之前需将数据缓存¹。

关闭连接

TCP连接的双方都有可能要求关闭连接，对任意一方来说其同步都是相同的， 这里考虑C端主动要求关闭连接的情况。

1. C端发送给S端一个设置了FIN标志位的FIN包表示关闭连接
   • 状态切换: ESTABLISHED -> FIN_WAIT_1
2. S端收到C端发来的FIN包后返回一个ACK包
   • 状态切换: ESTABLISHED -> CLOSE_WAIT
3. C端收到S端的ACK包
   • 状态切换: FIN_WAIT_1 -> FIN_WAIT_2
4. S端再次发送一个FIN包给C端
   • 状态切换: CLOSE_WAIT -> LAST_ACK
5. C端收到S端发来的FIN包后返回一个ACK包
   • 状态切换: FIN_WAIT_2 -> TIME_WAIT -等待一段时间后²-> CLOSED
   • 状态切换为CLOSED后释放所有资源并解除端口占用
6. S端收到C端发来的ACK包后正式关闭连接
   • 状态切换: LAST_ACK -> CLOSED
   • 释放所有资源并解除端口占用

状态转换图

下面的状态转换图来自RFC 793

                              +---------+ ---------\      active OPEN
                              |  CLOSED |            \    -----------
                              +---------+<---------\   \   create TCB
                                |     ^              \   \  snd SYN
                   passive OPEN |     |   CLOSE        \   \
                   ------------ |     | ----------       \   \
                    create TCB  |     | delete TCB         \   \
                                V     |                      \   \
                              +---------+            CLOSE    |    \
                              |  LISTEN |          ---------- |     |
                              +---------+          delete TCB |     |
                   rcv SYN      |     |     SEND              |     |
                  -----------   |     |    -------            |     V
 +---------+      snd SYN,ACK  /       \   snd SYN          +---------+
 |         |<-----------------           ------------------>|         |
 |   SYN   |                    rcv SYN                     |   SYN   |
 |   RCVD  |<-----------------------------------------------|   SENT  |
 |         |                    snd ACK                     |         |
 |         |------------------           -------------------|         |
 +---------+   rcv ACK of SYN  \       /  rcv SYN,ACK       +---------+
   |           --------------   |     |   -----------
   |                  x         |     |     snd ACK
   |                            V     V
   |  CLOSE                   +---------+
   | -------                  |  ESTAB  |
   | snd FIN                  +---------+
   |                   CLOSE    |     |    rcv FIN
   V                  -------   |     |    -------
 +---------+          snd FIN  /       \   snd ACK          +---------+
 |  FIN    |<-----------------           ------------------>|  CLOSE  |
 | WAIT-1  |------------------                              |   WAIT  |
 +---------+          rcv FIN  \                            +---------+
   | rcv ACK of FIN   -------   |                            CLOSE  |
   | --------------   snd ACK   |                           ------- |
   V        x                   V                           snd FIN V
 +---------+                  +---------+                   +---------+
 |FINWAIT-2|                  | CLOSING |                   | LAST-ACK|
 +---------+                  +---------+                   +---------+
   |                rcv ACK of FIN |                 rcv ACK of FIN |
   |  rcv FIN       -------------- |    Timeout=2MSL -------------- |
   |  -------              x       V    ------------        x       V
    \ snd ACK                 +---------+delete TCB         +---------+
     ------------------------>|TIME WAIT|------------------>| CLOSED  |
                              +---------+                   +---------+

TCP是一个有状态的，面向连接的协议，其在下层不可靠传输的基础上提供了一层抽象， 使连接双方可将TCP看作一个能够可靠传输数据的端到端通道， 保证接收方从TCP层收到的数据与发送方送往TCP层的数据的内容和顺序一致。

理论上讲，TCP/IP五层模型的任何一层都可以提供可靠的数据传输，但由于层间通信仍有可能会引入错误， 使得以提供可靠传输服务的下层为基础的传输协议仍有可能是非可靠的传输， 因此在紧邻应用层的传输层提供可靠传输服务将使得该服务的价值最大化。 另一方面，也可以基于UDP(不提供可靠传输服务)直接在应用层实现数据的可靠传输。

“functions placed at the lower levels may be redundant or of litle value when compared to the cost of providing them at the higher level.” ¹

可靠传输服务主要包括以下三部分

• 乱序重组
• 错误检测
• 丢包处理

针对这三种需求，TCP分别使用对应的机制来实现。

乱序重组: 序号

TCP使用序号(Sequence Number)对发送的字节流进行编号，接收方可以根据序号对收到的数据重新排序。 接收方收到数据后会发送确认包(Acknowledgment Segment)， 其中包含声明自己已经无误接收到的数据流的确认号(Acknowledgment Number)。 通过使用序号和确认号可以确保发送方送入TCP层的数据与接收方从TCP层收到的数据顺序一致。

具体来说，发送方为每个数据包附上一个序号SEG.SEQ，其表示发送数据的开头在数据流中的字节位置； 接收方接收到该数据包后会将序号加上数据长度(Payload Length)作为确认号， 并回应一个含有确认号SEG.ACK的确认包。

考虑一个单向传输数据的TCP连接

1. 发送方维护两个变量SND.NXT和SND.UNA并初始化为相同的序号初始值(Initial Send Sequence number, ISS)。 ISS通常会随机选取²以防御TCP序号预测攻击。 把这两个变量的值减去ISS的结果作为相对值，从而能够将ISS当作相对0来考虑， 相对值只与发送的字节数有关而与ISS的取值无关。其中
   • SND.NXT表示发送方下个数据包的序号，其相对值表示尚未发送的数据在数据流中的字节位置
   • SND.UNA表示尚未被确认的最小序号，其相对值表示尚未被确认的数据在数据流中的字节位置
2. 接收方维护一个变量RCV.NXT并使用收到的第一个数据包的SEG.SEQ作为初始值(即ISS)
3. TCP连接建立完成后数据流的第一个字节的序号为ISS+1，详见TCP连接
4. 发送方发送序号为SND.NXT，包含数据长度为n的报文，并将SND.NXT更新为SND.NXT+n
5. 接受方收到序号为RCV.NXT的数据包并确认无误后将RCV.NXT更新为RCV.NXT+n 并延时等待最多500ms³ (若已处于等待状态则结束等待并立即发送所有等待中的确认包)， 然后发送确认号为RCV.NXT的确认包，表示下个数据包的期望序号为RCV.NXT
6. 当发送方收到的确认包满足SEG.ACK>SND.UNA时将SND.UNA更新为SEG.ACK

当接收方接收到SEG.SEQ与RCV.NXT不一致的数据包时(out-of-order segment)⁴

• 若SEG.SEQ<RCV.NXT，则重新发送确认号为RCV.NXT的确认包并丢掉接受到的内容
• 若SEG.SEQ>RCV.NXT，TCP标准没有规定暂存还是丢掉该数据包， 但通常情况会暂存该数据包并记录SEG.SEQ与RCV.NXT的间隔(Gap)。 若后续数据包完全或部分填补间隔(Gap)， 则立即发送确认号为间隔最左端(lower end of the gap)的确认包

实际上，TCP通信双方都会各自维护发送方和接收方两套变量， 因此在TCP数据传输中会有两个序号和两个响应号。

错误检测: 校验和

为了确保通信双方发送和接受到的内容一致，TCP使用校验和(Checksum)机制来对TCP数据包进行校验。

计算校验和

发送方首先对以下三部分进行求和(以16 bits为单位，不足16 bits的右端补0)

• IP伪首部
• TCP的首部字段(Checksum字段初始为0x0000)
• TCP的数据内容(Data Payload)

再对求和结果求其反码，就得到了校验和，并将其填入数据包的Checksum字段中。 接收者在收到数据包后连同Checksum字段按相同的算法再计算一次校验和。 如果计算结果为0xFFFF，那么就表明数据包没有检测出错误和完整性缺失。

错误处理

当检测出错误时: