Track upstream OpenSBI Smrnmi support; adapt for Blackhole X280 (non-standard CSRs) #167

What

Track the OpenSBI mainline Smrnmi work currently in active review and, once it lands, adapt it for Blackhole's X280, which has non-standard RNMI CSRs and an MMIO-based per-hart entry-vector register.

This is the long-term path to recover an L2CPU hart wedged in M-mode with mstatus.MIE=0. For the more common operator-driven force-reboot (kernel wedged in S-mode with sstatus.SIE=0), see Phase 5 of #166 — a custom M-mode IPI event covers that case without needing RNMI at all.

Background — the hardware bug this exists for

From tt-isa-documentation/BlackholeA0/L2CPUTile/README.md:

Due to a hardware bug, the harts within each L2CPU tile can only be brought out of reset once. Once running, putting them back into reset requires resetting the entire Blackhole ASIC. As such, software is encouraged to build a mechanism for seizing control away from a running hart and parking it in an idle state in machine mode. One viable approach is to use RNMIs, with the RNMI trap handler set to the external peripherals general purpose scratch memory.

For everything except the very rare "wedge inside OpenSBI itself with mstatus.MIE=0" case, the Phase 5 custom IPI event in #166 is sufficient. RNMI is the strict superset that also handles M-mode wedges.

Upstream OpenSBI status (snapshot 2026-05-04)

Active series: [PATCH v2 0/5] Add RISC-V Smrnmi extension support by Evgeny Voevodin (evvoevod@tenstorrent.com, Tenstorrent), posted 2026-05-01. v2 review comments from Anup Patel (OpenSBI maintainer) came back 2026-05-04.

Diff size: +273/-34 across 9 files.

Notable from the v2 cover:

• v1 patches 6/7 + 7/7 (Tenstorrent platform glue) dropped from v2; they're moving through Voevodin's separate Atlantis platform support series.
• v2 dropped a separate _rnme_handler entry; RNME (Restricted Non-Maskable Exception) reuses the existing _trap_handler since it's taken as a regular M-mode trap with NMIE=0.
• v2 introduced a refactor of __stack_chk_guard_init so it runs after Smrnmi handlers are installed and NMIE=1 — necessary because mainline's Zkr seed-CSR access in _start would otherwise trap to a not-yet-installed NMEVEC on Smrnmi platforms (which boot with NMIE=0).
• Extending sbi_irqchip for NMIs is left for a follow-up.

Related v1 reference:
[PATCH 0/7] Add RISC-V Smrnmi extension support — the dropped 7/7 was [PATCH 7/7] platform: tenstorrent: Add Smrnmi support for Atlantis, which says: "Atlantis platform code installs the generic Ascalon RNMI/E handlers with ATLANTIS_MMR_BASE."

Adjacent series in flight:

• OpenSBI: [PATCH v3] platform: generic: Tenstorrent Atlantis support by Nicholas Piggin.
• QEMU: [PATCH v4 00/13] hw/riscv: Add the Tenstorrent Atlantis machine by Joel Stanley.

Atlantis ≠ Blackhole — Atlantis is Tenstorrent's next-gen chip with Ascalon cores. Blackhole uses X280. Some of the upstream work (CSR macros, fw_base.S entry stub) is generic and will work on both. The platform-specific glue (entry-vector install, trigger register) won't.

Why we can't just drop the upstream patches in for Blackhole

Per tt-isa-documentation/BlackholeA0/L2CPUTile/RNMIs.md:

The upstream macros target the standard addresses; the upstream _start flow assumes the CSR-driven entry. Both need overrides for X280.

Proposed adaptation plan

When the upstream Smrnmi v2 series lands in mainline OpenSBI master and is included in a release we vendor:

1. Bump third_party/opensbi/ to the first release that contains the merged series.
2. Override CSR addresses for X280. Either via a small bhx-blackhole-csr-override.patch on top of upstream's macros, or — depending on how the upstream macros end up structured — a platform-selected pair of header files. Keep this scoped tightly so mainline rebases stay easy.
3. Override the entry-vector install path. Upstream installs RNMI vectors via CSR; Blackhole needs an MMIO write to 0x2001_0418 + 16 * hartid per hart. Most likely lives in our patched generic_final_init (or a new blackhole_smrnmi_init if the platform layer factors that out).
4. Wire the host-side trigger. New bhx debug subcommand or daemon RPC: bhx force-rnmi -l N writes 1 << hart to 0x0000_2001_0414 over the per-L2CPU NoC TLB. Reuses the existing TLB-window machinery in src/l2cpu.rs. Trivial — single u32 write.
5. Connect the RNMI handler to the soft-reboot park path (Architecture: OpenSBI purgatory + host-handshake for guest shutdown/reboot (no chip reset) #166). The handler's "park the hart" exit drops into the same sbi_hsm_hart_wait loop that Architecture: OpenSBI purgatory + host-handshake for guest shutdown/reboot (no chip reset) #166 already uses. Daemon-side release-from-purgatory then works identically to the SBI-SRST case.

Estimated total size after upstream lands: ~150–200 LOC of bhx-side OpenSBI overrides + a one-line host kick. Most of that is the address overrides and the entry-vector install; the handler logic itself comes free with upstream.

Open questions to settle empirically when we land this

• Is the trigger bit at 0x2001_0414 W1C, or does the handler need to clear it explicitly with a store of 0, or is it auto-deasserted by mnret? Doc is silent.
• Is the trigger register writable via the host's PCIe NoC TLB? bhx already touches the adjacent 0x10000 register region (per-hart reset vectors), so the fabric path is fine; just need to confirm this specific offset isn't gated.
• BEU wiring: does Blackhole's BEU drive the same RNMI vector (cause 3)? If so, we either ignore cause 3 or stub it.

Acceptance

• Upstream Smrnmi series landed in OpenSBI master, vendored into bhx via a third_party/opensbi bump.
• bhx-side adaptation patch(es) applied on top.
• New bhx force-rnmi -l N subcommand fires the trigger; daemon waits for the parked-magic from the soft-reboot status block; release-from-purgatory completes successfully.
• Reproducible end-to-end test against a deliberately-wedged kernel (e.g., a guest running an infinite csrw mstatus, 0; 1: j 1b).