CPU hotplugging is the ability of an operating system to dynamically bring a CPU core online or take it offline while the system is running. This mechanism is useful for power management, fault isolation, and adapting the number of active CPUs to the current workload. In this project, you will extend xv6 so that individual harts can be turned on and off at runtime in a controlled and safe manner. The goal is to understand the low-level interaction between the scheduler, interrupts, and multiprocessor hardware while implementing a practical OS mechanism for dynamic CPU management.

RISC-V supports multiple privilege levels, most notably machine mode (M-mode), supervisor mode (S-mode), and user mode (U-mode), each with its own set of capabilities and restrictions. In a typical operating system environment, user applications execute in U-mode, while the operating system kernel runs in S-mode. M-mode is reserved for low-level machine control, such as system bootstrapping, early-stage initialization, and other platform-specific setup operations. Since CPU hotplugging requires controlling hart execution at a very low level, its implementation often relies on cooperation between software running in S-mode and software running in M-mode.

In RISC-V, a trap is a general term that encompasses both exceptions and interrupts (see Chap. 4 of the xv6 book). Exceptions are typically generated by the CPU itself in response to events such as illegal instructions, memory access faults, system call invocations via the ecall instruction, etc. Interrupts, by contrast, are triggered by external signals from devices or timers that indicate they require immediate attention.

Traps can be configured to be handled at different privilege levels depending on their type and the processor's current operating mode. When a trap occurs, the RISC-V hart automatically fills a register (mcause or scause depending on the privilege level) with a value that indicates the reason for the trap, as shown in the following table.

Besides the mcause (or scause) register, various additional registers are used to handle traps; when a trap is taken into M-mode (or S-mode), mepc (or sepc) register is written with the virtual address of the instruction that was interrupted or that encountered the exception. The mtvec (or stvec) register holds the start address of the trap handler in M-mode (or S-mode). Also, the mstatus (or sstatus) register keeps track of important information such as M-mode or S-mode interrupt-enable bits (MIE or SIE bit), the value of the interrupt-enable bit active prior to the trap (MPIE or SPIE bit), and the previous privilege mode (MPP or SPP bits).

To increase performance, certain exceptions and interrupts can be handled at a lower privilege level. For example, setting a bit in medeleg or mideleg register will delegate the corresponding trap or interrupt, when occurring in S-mode or U-mode, to the S-mode trap handler. By default, xv6 delegates all interrupts and exceptions to S-mode.

The RISC-V architecture supports two types of interrupts: local and global. Global interrupts are routed through a Platform-Level Interrupt Controller (PLIC), which can direct interrupts to any hart in the system via the external interrupt. Local interrupts, by contrast, are signaled directly to an individual hart. Software and timer interrupts are local interrupts generated by the Core-Local Interruptor (CLINT).

CLINT holds memory-mapped control and status registers associated with software and timer interrupts. The following shows the memory map for CLINT.

A machine software interrupt (MSI) is an inter-hart signaling mechanism supported by the RISC-V platform. One hart can trigger an MSI to another hart by writing a value of 1 to the memory-mapped control register msip in the CLINT. Each msip register is 32 bits wide, although only the least significant bit is used; the upper 31 bits are hardwired to zero, and the least significant bit is reflected in the MSIP bit of the mip register. When this bit is set, and interrupts are enabled on the target hart, the target hart traps into its machine-mode interrupt handler. After handling the interrupt, the receiving hart should clear the pending interrupt by resetting its own msip bit to 0.

MSIs are primarily used for inter-processor communication in multi-hart systems, since one hart can write another hart's msip register to generate an inter-processor interrupt (IPI). Because msip is level-triggered and not queued, writing 1 to an msip while it is already 1 does not trigger an additional delivery. To avoid lost IPI requests, it may be necessary to issue a new IPI only after the corresponding msip has cleared to 0. In this project, MSIs are used to coordinate CPU hotplugging, serving as the signaling mechanism for both offlining and waking up a hart.

For more information on CLINT, please refer to Chap. 12 in the SiFive's Freedom U740 Manual.

Turning off a hart does not mean physically powering down the processor core. Instead, it means placing the hart into a parked state, where it stops executing normal work and remains blocked until an interrupt arrives. RISC-V provides the wfi (wait for interrupt) instruction for this purpose. When a hart executes wfi, it suspends execution until an interrupt occurs. From the operating system's point of view, a hart parked in this way behaves as if it is offline, even though the underlying hardware is still present.

A machine software interrupt (MSI) must be used as the mechanism for delivering a hart-on or hart-off request to the target hart. Accordingly, a design in which such requests are conveyed only by setting a shared per-hart variable, without using an MSI, does not satisfy this requirement. Note that correct CPU hotplugging requires the target hart to enter the parked state only at a safe scheduling point. A safe scheduling point is one at which the current process has already been context-switched out, and control has returned to the scheduler, so that the hart is no longer executing any process and is not holding any lock, even though other runnable processes may still remain. Therefore, after receiving the MSI, the target hart should defer parking until it reaches the next such safe scheduling point. Once it does, it should transition to M-mode and execute the wfi instruction, until another MSI arrives to bring the hart back online. Your implementation should minimize the interval between receipt of the hart-off request and entry into the parked state.

A hart is turned on again by delivering an MSI to it, causing wfi to return and transfer control to the appropriate trap handler. After waking up, the hart must restore the appropriate execution context, leave the parked state, and resume normal kernel execution. Conceptually, turning a hart off and on requires a coordinated software protocol: the kernel must mark the hart as unavailable for ordinary scheduling, park it safely, later signal it with an interrupt, and finally mark it online again.

NAME

hartid -- return the current hart ID
hartmask -- return the current online hart mask
hartpin -- pin the current process to a specified hart mask

SYNOPSIS

int hartid(void);
int hartmask(void);
int hartpin(int mask);

DESCRIPTION

In RISC-V, a hart is a hardware execution context, roughly corresponding to a CPU core as seen by the operating system. These system calls allow user programs in xv6 to query and control hart placement and availability. In kernel/syscall.h, the system call numbers for hartid(), hartmask(), and hartpin() are assigned to 30, 31, and 32, respectively.

• hartid()

  hartid() returns the ID of the hart on which the calling process is currently executing.

  The returned value is the logical hart ID used by the kernel. Because a process may migrate between harts when it is scheduled, repeated calls to hartid() may return different values unless the process has been pinned to a specific hart with hartpin(), or only one eligible hart is available.

• hartmask()

  hartmask() returns a bit mask describing which harts are currently online.

  Bit i of the returned value is 1 if hart i is online and available for scheduling, and 0 if hart i is offline. For example, the return value 0b0101 means that only harts 0 and 2 are online, while all other harts are offline.

• hartpin(int mask)

  hartpin(mask) restricts the calling process so that it may be scheduled only on the harts selected by mask.

  If bit i of mask is 1, hart i is allowed for the calling process; if bit i is 0, then hart i is disallowed. If mask is 0, the process is not pinned to any specific set of harts and may be scheduled on any currently available online hart.

  After a successful call, the scheduler must run the calling process only on harts that are both (1) currently online, and (2) allowed by mask. If the current hart is not permitted by the new mask, the calling process should immediately yield the CPU so that it can be rescheduled on one of the allowed harts.

  hartpin() affects only the calling process. It does not change the online/offline state of any hart. In addition, when a new process is created, its hart mask is initialized to 0 by default, meaning that it may run on any online hart unless restricted later by hartpin().

RETURN_VALUES

hartid() returns the ID of the current hart.

hartmask() returns the current online hart mask.

hartpin() returns 0 on success. It returns -1 if mask does not allow any valid hart, if mask refers only to non-existent harts, if no currently online hart is permitted by mask, or if mask contains bits outside the range of supported harts.

NAME

hartoff -- take an online hart offline
harton -- bring an offline hart back online

SYNOPSIS

int hartoff(int hartid);
int harton(int hartid);

DESCRIPTION

These system calls allow a user process to request that a hart be brought online or taken offline at runtime. In kernel/syscall.h, the system call numbers for hartoff() and harton() are assigned to 33 and 34, respectively.

• hartoff(int hartid)

  hartoff(hartid) requests that hart hartid, if currently online, be taken offline.

  A correct implementation must use a machine software interrupt (MSI) to deliver the hart-off request to the target hart. The target hart must be taken offline only after it reaches a point where it is no longer executing any process and is not holding any lock. After reaching such a point, the target hart should transition to M-mode and enter the state of executing the wfi instruction.

  Once the operation completes successfully, the hart must no longer be used for ordinary scheduling, and the system-wide hart mask returned by hartmask() must reflect that the hart is offline.

• harton(int hartid)

  harton(hartid) requests that hart hartid, if currently offline, be brought back online so that it can participate in normal kernel execution and scheduling again.

  A correct implementation must use a machine software interrupt (MSI) to deliver the hart-on request to the target hart. After the target hart leaves the parked state and restores the required execution context, it should rejoin normal kernel execution. Once the operation completes successfully, the hart must be reflected as online in the system-wide hart mask returned by hartmask().

Simultaneous invocations of harton() and hartoff() from different harts must be properly serialized with locks so that online/offline transitions occur atomically and without races.

RETURN_VALUES

hartoff() returns 0 on success. It returns -1 if hartid is invalid, -2 if the hartid is the same as the current hart, or -3 if the target hart is already offline.

harton() returns 0 on success. It returns -1 if hartid is invalid, -2 if the hartid is the same as the current hart, or -3 if the target hart is already online.

Prepare a design document detailing your implementation in a single PDF file. Your document should include the following sections.

1. New data structures
   • Provide details about any newly introduced data structures or modifications made to existing ones.
   • Explain why these data structures/modifications were necessary and how they contribute to the implementation.
2. Algorithm design
   • Describe the overall flow of hartoff() and harton().
   • Describe all corner cases you considered and the strategies you used to address them.
   • Discuss any optimizations you applied to improve code efficiency, both in terms of time and space.
3. Testing and validation
   • Visualize the process scheduling resulting from running the task program using the graph.py script. (see Example 2 below)
   • Outline the test cases you created to validate your implementation, if any.
   • Describe how you verified the correct handling of the corner cases mentioned in Section 2.
   • Explain which part of the project consumed most of your time and why.

• You only need to change the files in the ./kernel directory. Put any new functions or definitions required to implement CPU hotplugging into the kernel/hotplug.c and kernel/hotplug.h files. Also, place any assembly code that must run in M-mode into kernel/machinevec.S.
• For this project assignment, you should use the qemu version 8.2.0 or higher. To determine the qemu version, use the command: $ qemu-system-riscv64 --version

• Read Chap. 4.1 of the xv6 book to understand RISC-V's privileged modes (supervisor mode and machine mode) and trap handling mechanism.

• Read Chap. 4.2 ~ 4.5 of the xv6 book to see how traps (system calls and interrupts) are handled in xv6.

• Read Chap. 8 of the xv6 book to understand the scheduling subsystem of xv6.

• Read Chap. 7.1 ~ 7.4 of the xv6 book to understand locks and lock ordering.

• For your reference, the following roughly shows the required code changes; each + denotes about 1~10 lines to add, remove, or modify.

  kernel/hotplug.c     |  ++++++++++++++++++
  kernel/hotplug.h     |  ++
  kernel/machinevec.S  |  +++++++
  kernel/proc.h        |  +
  kernel/proc.c        |  ++
  kernel/riscv.h       |  +++
  kernel/trap.c        |  ++
  

The skeleton code for this project assignment (PA3) is available as a branch named pa3. Therefore, you should work on the pa3 branch as follows:

$ git clone https://github.com/snu-csl/xv6-riscv-snu
$ git checkout pa3

After downloading, you must first set your STUDENTID in the Makefile again.

The address of CLINT's msip register for hart hartid is defined as follows in memlayout.h.

#define CLINT               0x2000000L
#define CLINT_MSIP(hartid)  (CLINT + 4*(hartid))

The skeleton code also provides hotplug.h and hotplug.c. These files are intentionally left mostly empty, and you are expected to fill them in as part of this project. In particular, you should use them to define the interfaces and implement the core logic required for CPU hotplugging. In addition, the skeleton includes machinevec.S. Any RISC-V assembly code that must execute in machine mode (M-mode) should be placed in this file.

The pa3 branch includes two user-level example programs, hart.c and task.c, to help you understand and test the CPU hotplugging interface.

The first example, hart.c, is a simple test program for the new hart-management system calls. It checks whether hartid(), hartmask(), harton(), and hartoff() behave correctly. The command $ hart id prints the hart on which the calling process is currently running, while $ hart mask prints the current on/off status of all harts as a bitmask. The commands $ hart on <hartid> and $ hart off <hartid> request that a specific hart be turned on or off, respectively. The following example shows how hart.c can be used to test the basic hart-management system calls. In this run, xv6 is started with four harts (CPUS=4 in Makefile). Initially, all harts are online, so user processes may execute on any of harts 0 through 3.

qemu-system-riscv64 -machine virt -bios none -m 128M -smp 4 -nographic -global virtio-mmio.force-legacy=false -drive file=fs.img,if=none,format=raw,id=x0 -device virtio-blk-device,drive=x0,bus=virtio-mmio-bus.0 -kernel kernel/kernel

xv6 kernel is booting

hart 2 starting
hart 1 starting
hart 3 starting
init: starting sh

Running $ hart id multiple times shows that the hart process may be scheduled on different harts while all of them are online.

$ hart id
Hart id: 3
$ hart id
Hart id: 2
$ hart id
Hart id: 2
$ hart id
Hart id: 0
$ hart id
Hart id: 1

The $ hart mask command displays the current online/offline state of each hart. Here, harts 0, 1, 2, and 3 are online.

$ hart mask
Hart mask [7-0]: . . . . O O O O

Next, hart 1 is turned off successfully, and the mask changes accordingly.

$ hart off 1
Hart 1 is turned off                   
$ hart mask
Hart mask [7-0]: . . . . O O . O

Hart 0 is then turned off as well.

$ hart off 0
Hart 0 is turned off
$ hart mask
Hart mask [7-0]: . . . . O O . .

A process cannot turn off the hart on which it is currently running. The following command fails for that reason.

$ hart off 3
Hart 3: can't turn on/off this hart

After the hart process is scheduled on a different hart, turning off hart 3 succeeds.

$ hart off 3
Hart 3 is turned off
$ hart mask
Hart mask [7-0]: . . . . . O . .

At this point, only hart 2 remains online. Therefore, every subsequent $ hart id invocation must report hart 2.

$ hart id
Hart id: 2
$ hart id
Hart id: 2

The example then turns the offline harts back on one by one.

$ hart on 1
Hart 1 is turned on
$ hart mask
Hart mask [7-0]: . . . . . O O .
$ hart on 0
Hart 0 is turned on
$ hart mask
Hart mask [7-0]: . . . . . O O O
$ hart on 3
Hart 3 is turned on
$ hart mask
Hart mask [7-0]: . . . . O O O O

This example illustrates several important properties of the interface. First, hartid() reflects the hart currently executing the calling process. Second, hartmask() correctly reports which harts are online. Third, hartoff() and harton() update the system state as expected. Finally, the example demonstrates an important safety rule: a process must not be allowed to turn off the hart on which it is currently running.

The skeleton code also includes a second example program, user/task.c, which is designed to demonstrate the scheduling effect of CPU hotplugging. Unlike hart.c, which focuses on testing the correctness of the hart-management system calls themselves, task.c is intended to show that once a hart is turned off, the scheduler no longer places runnable processes on that hart, and scheduling resumes on the hart after it is turned back on.

The program creates several CPU-bound child processes whose running times differ according to the span[] array. Each child executes a busy loop for a different duration, so the scheduler has multiple runnable tasks to distribute over time. Before forking the children, the parent process pins itself to hart 0 by calling hartpin(1). Each child then pins itself to harts 1, 2, and 3 by calling hartpin(14), where 14 corresponds to the bitmask 0b1110. As a result, the child processes may run only on harts 1, 2, and 3, while the parent remains on hart 0 and controls the hotplug events.

After creating the children, the parent waits briefly in a busy loop and then turns hart 2 off by calling hartoff(2). It continues executing for a while, then turns hart 2 back on with harton(2). Later, it performs the same sequence for hart 3. The expected behavior is that, during the interval when hart 2 is offline, no child process should be scheduled on hart 2, and similarly no child process should be scheduled on hart 3 while hart 3 is offline. Since the children are pinned to harts 1, 2, and 3, any temporary disappearance of scheduling activity on hart 2 or hart 3 should be directly visible.

To make this behavior observable, the skeleton code provides a logging facility enabled by the -DLOG compile-time option. When this option is used, the kernel records scheduling events, including when a process starts running on a hart and when it stops running due to preemption, blocking, exit, or other scheduling decisions. The Makefile includes a convenience target, make qemu-log, which builds the logging-enabled kernel and runs xv6 under QEMU, saving the console output to xv6.log. To produce a meaningful log, run the task program inside xv6 and then quit QEMU. After QEMU exits, the Python script graph.py is invoked to process the log and generate a scheduling visualization in graph.png, as shown below.

qemu-system-riscv64 -machine virt -bios none -m 128M -smp 4 -nographic -global virtio-mmio.force-legacy=false -drive file=fs.img,if=none,format=raw,id=x0 -device virtio-blk-device,drive=x0,bus=virtio-mmio-bus.0 -kernel kernel/kernel-log | tee xv6.log

xv6 kernel is booting

1335840 1 starts on 0
hart 2 starting
hart 3 starting
hart 1 starting
1345500 1 starts on 1
1350860 1 ends on 1
1354100 1 starts on 1
1361640 1 ends on 1
1365500 1 starts on 1
1371880 1 ends on 1
...
init: starting sh
...
$ task                            <--- Run the task program
11659280 2 starts on 3
11676600 2 starts on 3
11678970 3 starts on 2
11682440 2 ends on 3
11738470 3 ends on 2
11748040 3 starts on 1
11770220 3 ends on 1
11777410 3 starts on 1
11790940 3 ends on 1
11796800 3 starts on 3
11810480 3 ends on 3
11817530 3 starts on 3
11876150 3 ends on 3
...
QEMU: Terminated                   <--- Quit qemu using ^a-x
*** The output of xv6 is logged in the 'xv6.log' file.
graph saved in the 'graph.png' file
*** See graph.png file.

A typical log entry has the form:

11659280 2 starts on 3
11676600 2 starts on 3

This means that process 2 was scheduled on hart 3 starting at time 11659280 and stopped running at time 11676600. The script graph.py reads these records, groups them by hart, and draws a timeline in which each horizontal bar represents an interval during which a particular process ran on a particular hart. Different processes are shown in different colors, and each hart is displayed on its own row. In the generated figure, gaps in a hart's row indicate that no logged process ran there during that period.

When task.c is executed with logging enabled, the resulting graph.png should clearly show that the rows corresponding hart 2 and hart 3 become inactive during the periods when those harts are turned off. After each hart is turned bak on, scheduling activity should reappear on that hart as show below. This makes task.c a useful visual validation tool: it not only exercises the hartoff() and harton() system calls, but also helps confirm that CPU hotplugging has been correctly integrated with the scheduler and hart-affinity mechanism.

• First, make sure you are on the pa3 branch in your xv6-riscv-snu directory. And then perform the make submit command to generate a compressed tar file named xv6-{PANUM}-{STUDENTID}.tar.gz in the ../xv6-riscv-snu directory. Upload this file to the submission server. In addition, you must also upload your design document as a PDF file for this project assignment.

• The total number of submissions for this project assignment will be limited to 30. Only the version marked as FINAL will be considered for the project score. Please remember to designate the version you wish to submit using the FINAL button.

• Note that the submission server is only accessible inside the SNU campus network. If you want off-campus access (from home, cafe, etc.), you can add your IP address by submitting a Google Form whose URL is available in the eTL. Now, adding your new IP address is automated by a script that periodically checks the Google Form at minutes 0, 20, and 40 during the hours between 09:00 and 00:40 the following day, and at minute 0 every hour between 01:00 and 09:00.

  • If you cannot reach the server a minute after the update time, check your IP address, as you might have sent the wrong IP address.
  • If you still cannot access the server after some time, it is likely due to an error in the automated process. The TAs will verify whether the script is running correctly, but since this check must be performed manually, please understand that it may not be completed immediately.

• You will work on this project alone.
• Only the upload submitted before the deadline will receive the full credit. 25% of the credit will be deducted for every single day delayed.
• You can use up to 3 slip days during this semester. If your submission is delayed by one day and you decide to use one slip day, there will be no penalty. In this case, you should explicitly declare the number of slip days you want to use on the QnA board of the submission server before the next project assignment is announced. Once slip days have been used, they cannot be canceled later, so saving them for later projects is highly recommended!
• Any attempt to copy others' work will result in a heavy penalty (for both the copier and the originator). Don't take a risk.

Have fun!

Jin-Soo Kim
Systems Software and Architecture Laboratory
Dept. of Computer Science and Engineering
Seoul National University