*.xdavidwu.link had been solely backed by my homelab-kind-of-thing, where I did interesting experiments and mostly just leave it there when it worked. The architecture was messy, tangled, overly reused for multiple unrelated purposes. Security was considered heavily, but availability was almost always ignored, excused by lack of redundancy at hardware level. The hardware was also aged, quirky and not really reliable.

I have been slowly refactoring *.xdavidwu.link to make it more portable and resilient, actually since before the downtime. *.xdavidwu.link was once constructed by foss (ignoring firmware, of course) and fully self-hosted, but now I am adding public resources into the mix, as long as it does not create much vendor lock-in and is still fairly easy to reimplement in foss. What you are seeing is a part of this effort: xdavidwu.link blog is now partially up via public static site hosting, sans the Matrix-powered commenting system and the Gemini protocol variant.

The server did have virtual CD-ROM and KVM feature (although it requires ancient version of Java on browsers, which is painful), and by that, I booted an ISO to get a working temporary Linux environment for inspecting the situation. To boot with virtual CD-ROM, I needed to skip the HBA option ROM loop via CTRL+C before that to make it not to try booting and go to its settings utility instead. With the temporary Linux environment, I checked that all the previously working disks are fine, and should be able to boot it if I loaded the kernel and initrd from something that did not go through the HBA.

Initially, I thought using an external GRUB from an ISO to load kernel from disk may work, and experimented with grub-mkrescue. GRUB did not seem to recognize any disks, which made sense without a dedicated driver for the HBA. I then tried to pack kernel and initrd into the ISO, but booting them got me a 452: out of range pointer.

Then I tried ISOLINUX. Creating a kernel-loading ISO with it was easy and straightforward. Booting with it worked perfectly. This workaround works except that I might need to wire up the virtual CD-ROM again if it needs to boot another time.

But there are ways around it, without actually packing tools into the container or copying them into the working filesysystem.

If you know the internals, you may be familiar with nsenter command and setns syscall to poke around the namespaces. The goal here is to enter mount namespace of the containers, but not losing our debugging tools. Imagine if we have those tools already in memory, after those setns syscalls, instead of exec’ing a program from the target mount namespace, we just invoke the planted code?

Yes, this works, and I have patched busybox nsenter implementation to run ash_main instead of exec’ing to test it out. Combined with the “nofork” trick toggle in busybox that trys to avoid fork-and-exec but just run code for commands in the shell itself like shell builtins, this gives me a comfy debug experience without throwing any file into the container. But there is a much simpler way than this, even without involving any coding.

Quoting from pid_namespaces(7),

A /proc filesystem shows (in the /proc/pid directories) only processes visible in the PID namespace of the process that performed the mount, even if the /proc filesystem is viewed from processes in other namespaces.

and also considering the fact that /proc/[pid]/exe are not really symbolic links but representing kernel file handles, with PID namespace of the container also entered, /proc/self/exe may work as a way to slide an executable in.

Busybox also has a build-time toggle to prefer its applets when exec’ing, that is, it detects if the name is any of the command it implements, and execute /proc/self/exe instead. So all you need is just a statically-linked busybox with nsenter, ash applets and CONFIG_FEATURE_PREFER_APPLETS, then

busybox nsenter -t <container pid> -m -p ash

and you get a shell with all the applets you chose, inside the nearly-empty container.

以此為契機挖出了一連串各種奇妙知識

首先因為工作性質的關係，跟使用者溝通以郵件為主，所以作法上選擇減少溝通成本盡量自行調查， 不過其實這次在仔細調查前對方也先回報換個瀏覽器就沒事了

各種版號們與 WebKitGTKPermalink

在 access log 中找到了事發的 UA

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.3 Safari/605.1.15

第一眼看起來應該就是 Safari，不過也有人測過最新的 Safari 沒問題。 因為這次事件先順勢帶出了缺乏公用 Safari 測試環境的議題，去補充了一下 Safari 的知識， 對 Safari, macOS, WebKit 版本對應 有初步的印象， 發現不對勁，WebKit 605.1.15 相對於 macOS 10.15.7, Safari 15.3 也太老了，對應表上的 WebKit 版本實際上是 612.4.9。 到 WebKit GitHub 上搜尋 605.1.15，確實找到在各平台上產 UA strings 都把這個版號寫死， 找到相關的 tracker item， 為了防更多看 UA 特判產生的歪風所以乾脆把版號 freeze 了。 對於實際 Safari 與 WebKit 的版本對應，MDN 有更齊全的表， 不過沒有 macOS 版本資訊

對於重現，因為沒有夠新的公用蘋果硬體，加上 macOS EULA 對於 VM 據說也只允許跑在蘋果硬體上， 所以打算放棄 macOS 的環境，採用 WebKitGTK 盡力重現。WebKitGTK 雖然跟 WebKit 活在同一個 repo 下， 版號跟 release cycle 也都是獨立的，勢必要想辦法在找出 WebKitGTK 與 WebKit 間的版本對應。 WebKitGTK 每個 minor 都是從 WebKit 本人再 branch 出去維護， 可以在 GitHub Wiki 找到整理， 老一些的版本（轉 GitHub 前？）則是整理在 trac 上， 在 GitHub 上可以看到 branch 點的 commit 包含在哪些 tags 裡，找最小就可以大略對應到 WebKit 版本

Take 1: FlatpakPermalink

有了以上這些資訊，對於在非 macOS 下重現環境就只差要怎麼獲取與使用特定版本的 WebKitGTK。 使用上我選最知名的 client – GNOME Web / Epiphany，對於獲取，第一個想法就是 Flatpak， 找到了 Flatpak 安裝特定版本 app 的方法：

flatpak remote-info --log flathub org.gnome.Epiphany
flatpak update --commit=<desired commit from above> org.gnome.Epiphany

再來需要的就是 Epiphany Flatpak package 跟 WebKitGTK 版本對應， 找到了 Flathub 上 Epiphany 的 manifest， 但上面沒有 WebKitGTK 的影子，往上追發現做在 org.gnome.Platform。 GNOME platform 的作法比較複雜， 用了 BuildStream，WebKitGTK 的引入在 elements/sdk/webkitgtk.inc， 在對於各個 GNOME major 的 branch 上看 history 就可以知道哪個 GNOME major 對到哪個 WebKitGTK minor， Epiphany 的 release 是 follow GNOME 的，所以就是直接找同個 major

對應版號後發現我需要的 WebKitGTK 大概落在 2.32 與 2.34，2.34 再對到 Epiphany Flatpak package 在 40.x， 結果 flatpak remote-info --log 一下，發現 Flathub 上已經沒有存這麼久以前的版本了，這次沒那麼簡單

Take 2: OCI containersPermalink

換個傳統一點的想法就是從 stable distribution 去抓套件，但老東西抓 binary 丟上新系統 ABI 大概會接不上死一片， 裝個 VM 又有點累，於是用了 container:

podman run -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY --privileged -it alpine:3.15

用 X11 而不是 Wayland 是因為 GTK 不知為何貌似在 Wayland 下會先需要連 DBus，但 DBus 比較難搞定， socket 丟進去 address 設好後還有 protocol 上的 auth 要處理，所以偷懶用 XWayland / X11。 privileged 則是因為 WebKitGTK 用 Bubblewrap 做 sandboxing，貌似需要些權限去開 namespaces， 但也懶得挖需要哪些 capabilities 了就乾脆丟個 privileged

apk add epiphany font-noto-cjk mesa-dri-gallium
epiphany

跑起來就能測了

有了這些功夫，遇到 Safari 使用者回報問題，大致上也都能用開放的系統重現個大概了

Off-topic 的後記：

這次撞到的是舊版 WebKit JS Date parsing 不吃 YYYY-MM-DD HH:mm， 其實在 ECMAScript 下用空格分隔不怎麼標準，不過因為其他 JS 實做有做， 後來還是做了

這次拿來升級的環境只有一個 CentOS Stream 8 的 lxc，用來架 GitLab

首先要蓋掉 dnf repos，手動抓 el9 的 centos-stream-release, centos-stream-repos, centos-gpg-keys，rpm -U --force 下去

這次很順 dnf makecache dnf upgrade --allowerasing 下去就差不多了

但接下來手動刪 el8 殘留物的時候就撞到了 dnf remove 不會動

Error message 看起來像 rpm 嘗試用 sqlite 這個 backend，但看到現有 repo db 的長相給改成了 bdb_ro （ro 猜是 readonly，看來 rpm 有換過格式）

查了一下確實 rpm 從 Berkeley DB 轉成了 SQLite，用 rpmdb --rebuilddb rebuild db in SQLite 就可以解決

再來 dnf modules 噴了一堆東西被自動搬到 @modulefailsafe 並且底下都炸 platform(el:8) 之類的 dependency 解不開

這個反而單純，因為我沒有在用那些 dnf modules，一次把它 disable 光光就解決了： dnf module disable virt nodejs perl perl-IO-Socket-SSL perl-libwww-perl postgresql python27 python36

再來將 gitlab-ce 轉到 el9 時撞到 package signature 驗證不過，爬了一下該有的 config， key 也都跟原本 el8 的一樣，但在驗 package 時有出現 warning Signature not supported. Hash algorithm SHA1 not available.

查了一下 el9 把 SHA1 預設關了，可以用 update-crypto-policies --set DEFAULT:SHA1 先開回來解決，而 update-crypto-policies rpmfind 找出是在 crypto-policies-scripts 裡

除了這些之外沒遇到多少問題

另外注意到了 DBus implementation 換成了 dbus-broker

看來這次 el8 -> el9 比上次 el7 -> el8 小很多

In AOSP, DocumentsUI has a string config trusted_quick_viewer_package. This predefined package name is used for preview (or “quick view”, as the source also suggests) the file. DocumentsUI contains two ways of opening a file (in AbstractActionHandler), preview (via that package) and regular (via a normal view intent). The code supports setting a primary method and fallback. When opening a file by clicking it from the UI, the primary is preview, and will fallback to regular. Disabling the quick viewer package (in this case, Google Drive) makes the preview intent fail and fallback to regular.

There is also another way around it. DocumentsUI contains a debug mode that can be triggered even on a build of type user. In that mode, various debugging commands are available (some of which are especially useful for developing a DocumentsProvider, by the way), including overriding the quick viewer package name. I don’t want to ruin the fun of reading source code, but here’s a hint: CommandInterceptor.

A quick search on AOSPXRef (and AndroidXRef) shows that trusted_quick_viewer_package is availble at lease since Nougat (7.0.0_r1).

Note that this “quick view” mechanism is available in AOSP. Even if a ROM looks very AOSP-ish and have source code published, an overlay in the device part may set this to something evil that transparently build a regular view intent for you, making you unaware of its existence, but also do shady things with your files.

ceph-volume is packaged separately, and not depended by ceph-osd (for Ubuntu it is under Recommends). BlueStore OSDs created with ceph-volume lvm create needs it to start.

Here’s how I understand how starting BlueStore OSDs work: systemd service ceph-volume@lvm-{osd id}-{uuid}.service (under multi-user.targets.want) runs ceph-volume-systemd lvm-{osd id}-{uuid}, which is a wrapper for ceph-volume lvm trigger {osd id}-{uuid}, which is currently equivalent to ceph-volume lvm activate --auto-detect-objectstore {osd id} {uuid}. It will find the LVM devices, mount tmpfs at /var/lib/ceph/osd/{cluster}-{osd id}, populate it, and start ceph-osd@{osd id}.service which runs the osd daemon.

If it fails, we can use ceph-volume lvm activate --all to start them manually.

When upgrading Ceph, restarting ceph-{daemon}.target will restart all those daemons, which is especially useful when upgrading OSDs.

Package managementPermalink

This is about what package management tool is used. What differs among them are usually under these three categories: packaging scripts, management features, and package format.

Packaging scriptsPermalink

The scripts that are read by package building tools for building packages.

PKGBUILD in pacman/makepkg and APKBUILD in apk/abuild are two examples of easily understandable packaging scripts. Both of them are in shell scripts (bash and busybox ash) and thus very flexible. PKGBUILD supports checking out various version control repositories, and in combination of determining package version from a shell function, it provides an easy way to construct “VCS packages”, that is, to build packages from a changing, bleeding edge source code (like a git branch, or svn trunk), without making changes in script, while still be able to distinguish exactly which built packages is newer by generated version strings.

APKBUILD looks like when PKGBUILD is written in busybox ash, but there are many differences. abuild supports automatic dependency discovery. In APKBUILD, typically you only need to specify build-time dependency in makedepends, and linked shared libraries will be detected and added to run-time dependencies. It also add provides of libraries and commands automatically. Package split also works differently in APKBUILD. In PKGBUILD, we have pkgbase for a name of that PKGBUILD, and pkgname array splits the package, packaging functions install files into $pkgdir of each package. In APKBUILD, pkgname is the name of a main package, and subpackages list is a list of subpackage names. The package function of main package is called first, where files are installed into $pkgdir, and then subpackage functions are called, where files are futher moved from $pkgdir to $subpkgdir, making package splits more natural and easier. abuild additionally provides common subpackage functions for things like development files and document, most package splits will work with just adding the names in subpackages.

Management featuresPermalink

This is about user-facing package manager features. Some package managers selects packages in repositories order then versions order, while some package managers consider versions order first. Some support repository pinning of packages. Some support creating dummy package with a single command. These differences and additional features may or may not make your life easier depending on how you use them.

Package formatPermalink

A simple internal format for packages is easier to debug, and may makes things easier when fixing a broken installation. Packages of pacman (*.pkg.tar(.*)) and apk (*.apk) are basically tarball archives of its content with package metadata written in fixed paths of the archives. For lazy, temporary fixes of broken package management tools install, we can just extract them as tarballs to root directory, without actually involving package management tools, then reinstall the packages using normal workflow with now working tools.

Packaging conventionPermalink

This is about a distribution’s policy and convention on subjects like how packages should be splitted, how to name a package and how non-free software are handled. Distributions like Alpine and Debian split documentation and development files into subpackages, while Arch Linux usually package the software without splitting. Some distributions have naming convention like naming libraries packages in lib*, while some distributions just use the name of software. Some distributions isolates non-free packages into an optional repository, or provide packaging scripts only, while some distribution allow non-free software in their core repositories.

Selection of softwarePermalink

Distributions act differently on selecting alternative software, implementation or forks. For libc, some use glibc while some use musl. For init system, some use systemd while some uses OpenRC. For default privilege escalation tool, most distro use sudo while some are discussing about switching to OpenDoas.

Release patternPermalink

Some distributions are rolling release, some are stable, and some provide stable releases on top of rolling releases. Depending on what you use the OS for, you may need rolling or stable.

Under specific use cases, there are also a few more things to consider, like availability of kernel live-patching, presence of specific software packages directly provided by upstream, and other features.

I had been using Arch Linux for desktop usage as daily driver for years, for its rolling release and simplicity on packaging scripts. Recently, I switched to Alpine Linux edge with systemd added on top of it by me. I will write about how I did it in another post. I choosed Alpine Linux for its simplicity on package management, fast rolling release (on edge), easier software packaging, and musl. I have also been using stable version of Alpine Linux for container usage, choosing the same distro for desktop leaves me fewer things to care about. I still rely on journald, networkd, resolved on host environment, and I like them, so I added systemd.

以這段期間的使用經驗, freenom 提供的 nameserver 功能偏少, 不能添加 wildcard record, 能加入的 record type 偏少, 並且不太可靠, 偶爾會 SRVFAIL

評估之後, 終究是得付費的, 所以找了別家買了個 xdavidwu.link, 不過我的 infrastructure 是與別人共用的, 這個網域還是可能會幫忙架些我信任的朋友的服務

注意我的 Matrix 和 ActivityPub 也一律轉移了, 最近上線的 Matrix 留言區也是

Random thoughts:

在選擇域名時, 如果不想很認真的去檢查歷史, 盡量選擇較獨特的命名會比較安全

獨特的命名比較不會有上個擁有者使用期間還很近, 還握有著有效的 TLS 憑證的疑慮

另外, 對於網域和 TLS 憑證間這種可能產生擁有者可以暫時不符的問題, 如果 DANE 或類似的系統被大量採用, 可以很根本的解決這個問題, 但推行 DNSSEC 的使用和驗證應該還有很長一段路要走

There are Matrix rooms for each posts, and you can comment by either the embedded web frontend or any Matrix client. Room aliases are located above the embedded web frontend.

You can also browse all comment rooms from Matrix Space at #comments_xdavidwu.link:xdavidwu.link

Here is the story behind this:

I had been considering about moving from Disqus to an open platfrom that statisfies following criteria:

• Accessible in some other way to those who disables JavaScript in their browsers
• Does not depend on external proprietary service
• Does not require an account just for commenting on a few sites
• Still applicable on static sites
• Still support moderation to mitigate abuses

Staticman looked like a nice choice, but it seemed to depend on source hosting platform. I was using a self-hosted GitLab instance which it supported, but I might change my hosting platform and I did not want this to be a blocker. I would like my visitors able to see new comments as soon as possible, but if I made it fully automatic with an approach that alter the site source code, when abused it can make version control history messy and also introduce high load on building systems. I preferred a solution that neither introduce source code changes nor trigger a new build of the site when comments arrive.

It turned out that my ideal solution would still inject content dynamically. To be friendly for visitors without JavaScript enabled, using <iframe> was the next mechanism that came to my mind. But then styling would become another problem. I would like to have my source all in one place, so the platform needs to either be configured to use style sheets from my site or have an API to update style sheets in my build process. With all these requirements it was hard to search for eligible solutions. I planned to write one by myself but more questions appeared. To prevent easy abuses, I needed to implement auth, but requiring an account for my site only was obviously a bad choice, so I might need external identity providers. Depends on specific identity providers also looked bad to me. The idea to build my own platform was put on hold.

Later, I learned about Matrix, the federated messaging protocol that supported lots of modern fancy features one would expect from an IM software. I hosted a homeserver and shared it with some friends of mine. After some frequent usage I came up with the idea of using Matrix room for comments. Federation makes it easily accessible around the world, and there is also built-in moderation feature. For visitors without JavaScript enabled, they can still find the rooms by aliases and read or leave comments with clients they like. I can also guide visitors of Gemini version of my site to comment rooms by leaving room aliases there.

To make this work, I need an embeddable web client with a simple UI and ability to view as guest. This had been in my todo (with low priority though) until I discovered Cactus Comments. They had the same idea of using Matrix room for commenting platform and they already built it! But I do not like thier method for creating the rooms. If I understand the code correctly, they hook into homeserver by appservice API, and create the room when an alias that does not exists is queried. This can be abused to create many unwanted rooms and bring troubles to site owner and homeserver admin. The use of appservice API also make it harder to deploy.

So I use their web client, and create the rooms in my own way. I write a hook for the static-site generator I am using (Jekyll), and it provisions rooms when building the site. This is done under a new user, and it will invite me to become an admin in newly created room. All room creations happen automatically under my control. If interested, see _plugins/matrix-room-provision.rb of the site source code.

There are still many spaces for improvements, like more CSS work to make the experience more immersive to this site, and support of login mechanisms other than native username / password.