Git through a jump host

The VPS is always on and already in the tailnet, so it can be the jump host:

Host vps
  HostName <vps-hostname>
  User <user>
  IdentityFile ~/.ssh/id_ed25519_vps

Host forgejo
  HostName <forgejo-ip>
  User git
  ProxyJump vps
  IdentityFile ~/.ssh/id_ed25519_forgejo
  IdentitiesOnly yes

ProxyJump only makes the VPS open a TCP socket to port 22. The SSH session stays end-to-end, so the Forgejo deploy key authenticates directly and the VPS never sees plaintext git traffic.

id_ed25519_forgejo is a deploy key, added on the repository itself under Settings -> Deploy Keys with write access enabled. That scopes it to this one repo -- an account SSH key would hand the sprite every repository I have.

The sprite's key is restricted on the VPS side:

restrict,port-forwarding,permitopen="<forgejo-ip>:22" ssh-ed25519 AAAA... sprite-jump

restrict disables port forwarding as well, and permitopen only narrows forwarding that is already allowed -- it does not enable it. Without port-forwarding in that list the jump fails with administratively prohibited. With both, the sprite gets a TCP path to Forgejo and nothing else: no shell, no PTY, no other destination.

Services on a sprite

PID 1 is tini and there is no systemctl, which I first read as "no daemons". There is an in-VM CLI called sprite-env instead, and it does the part of systemd that matters here:

sprite-env services create <name> --cmd <binary> --args "a,b,c"
sprite-env services list | get | start | stop | restart | delete
sprite-env services signal <name> TERM

Services restart automatically on boot, which on a sprite means after every cold pause. --cmd takes the binary only, arguments go into a comma separated --args; there is also --env, --dir and --needs for dependencies. State comes back as JSON, and stdout and stderr land in /.sprite/logs/services/<name>.log:

{"name":"tailscaled","state":{"status":"running","pid":1167,
 "started_at":"2026-08-02T19:37:05Z","next_restart_at":"0001-01-01T00:00:00Z"}}

Exactly one service may claim --http-port, and that service is auto-started when an HTTP request arrives at the sprite's public URL -- the same request that wakes a paused sprite. The documentation is explicit that you should not start a background process yourself next to a service, because the service manager owns the process lifecycle.

Tailscale works until the sprite pauses

Kernel mode works, so no userspace networking is needed. /dev/net/tun exists and opens O_RDWR as the unprivileged sprite user, even though the mode has no read bits:

$ ls -l /dev/net/tun
c-w--wx-wT 1 root root 10, 200 Aug  2 19:27 /dev/net/tun

CAP_NET_ADMIN is in CapEff and ip link add dummy0 type dummy works. A fresh sprite has no egress policy -- /.sprite/policy/network.json does not exist -- and tailscale netcheck reports UDP: true.

So tailscaled becomes a service, wrapped in sudo because it needs root:

sprite-env services create tailscaled --cmd /usr/bin/sudo \
  --args "-n,/usr/sbin/tailscaled,--state=/var/lib/tailscale/tailscaled.state,--socket=/var/run/tailscale/tailscaled.sock,--port=41641"

Logging the node in is a separate step, and the sprite has no browser, so it needs an auth key:

sudo tailscale up --auth-key=tskey-auth-... --hostname=sprite

Use a non-ephemeral key -- an ephemeral node gets reaped while the sprite sleeps. The state lives in /var/lib/tailscale/tailscaled.state on the persistent disk, so this is a one-off.

/etc/resolv.conf sits on a read-only overlay, so tailscaled cannot install MagicDNS. *.ts.net names then fall through to public DNS, which answers with addresses that are not my tailnet and do reply to ping:

$ dig +short A forgejo.<tailnet>.ts.net @1.1.1.1
185.40.234.37
185.40.234.172
185.40.234.210

/etc/hosts is writable, so pin the peer there, or put the tailnet IP straight into ~/.ssh/config.

That holds until the first pause:

$ tailscale status | grep forgejo
<forgejo-ip>  forgejo  linux  active; relay "fra", tx 2340 rx 0

tx 2340 rx 0 -- the peer is still listed as active, packets go out, nothing comes back. Every connection hangs for 2m13s and then times out, and it does not recover on its own. tailscale ping still answers via DERP while real traffic is black-holed, so it is not a usable health check. Restarting the service fixes it, and it comes back with a direct connection instead of DERP:

$ tailscale ping forgejo
pong from forgejo (<forgejo-ip>) via <home-ip>:62264 in 22ms

This is not a Tailscale bug. A pause freezes the process and drops its TCP connections; tailscaled thaws with dead socket state. From inside the sprite you can only detect it and restart, never prevent it.

With the jump host there is no daemon left in the sprite that could keep broken sockets after a pause. When the sprite pauses, it aborts the git command that is running, and the next one opens a new SSH connection and works. That is what makes the extra hop the right trade here.

Endpoints and stations

The API is documented on one page and lives at https://niwis-online.de/api/daten, no authentication, JSON, dates as YYYY-MM-DD, decimal point. Values are daily. For sub-daily data PEGELONLINE is the source, and the 70 WSV stations in NIWIS point back to it in their urlInstitution.

There are 698 stations, 397 of them on a named river with level or discharge data, the rest groundwater and springs. /messstelle lists them without the river name, so the river comes from /stammdaten, one request per station. I cache the result in a JSON file.

Derived quantities

/abgeleiteteGroesse returns a catalogue of 43 statistics. Each entry names the endpoint to call it from and the parameters it needs:

{
  "abgeleiteteGroesse": "MQ",
  "messgroesse": "ABFLUSS",
  "benoetigtReferenzzeitraum": true,
  "benoetigtZeitintervall": false,
  "endpunkt": "berechneEinzelwertNummer"
}

A client needs no hardcoded list of statistics, it can loop over the catalogue and build each request from the two boolean flags. The ones I use are MQ/MW (mean discharge and level), MNQ/MNW (mean annual low), Median Q pro Kalendertag and the deciles per calendar day. Reference period and year definition -- calendar year, hydrologisches Jahr or Wasserhaushaltsjahr -- are parameters.

The reference period cannot start before 1991. 1961--1990 returns 400 with Beginn des Referenzzeitraums darf nicht vor dem Jahr 1991 liegen, so a comparison against the older climate normal is not possible. Any window inside the allowed range works, 1991--2019 and 1996--2025 both compute.

In late summer the per-calendar-day median is the useful reference, not the annual mean.

The classification has four steps: "extrem niedrig", "sehr niedrig", "niedrig" and "Kein Niedrigwasser". The last one is open ended upwards and only means "not low water", so a gauge at its long-term average falls into it and so does a gauge in flood. NIWIS classifies low water and nothing else.

Missing values

Missing data appears in three different forms.

A statistic that cannot be computed returns 200 with an empty payload and hatZuvieleFehlwerte: true. That happens when more than 10% of the underlying values are missing, today for 49 of the 359 discharge stations.

Invalid parameters return 400 with a JSON message.

In the measurement series a missing value is -777 with a flag of Fehlwert or BfGAdded, not null:

{"datum": "2026-08-02", "messwert": -777.0, "einheit": "m³/s", "flag": "Fehlwert"}

Filtering on null does not catch these, and filtering by sign is wrong: -777 m³/s is impossible, a water level of -3 cm is not, and Emmerich is at -3 cm. In a sample of 60 stations every -777 carried a flag and no real value did, so the flag is the reliable signal.

Licences

lizenz is per station, with six different values across the 698 stations:

dl-zero-de/2.0   220
dl-by-de/2.0     210
cc-by/4.0        183
dl-de/by-2-0      46
ccbync/4.0        29
cc by-sa 3.0      10

dl-by-de/2.0 and dl-de/by-2-0 are the same Datenlizenz Deutschland spelled two ways, and 29 stations are non-commercial. Republishing values needs a per-station check.

A water level has no useful percentage

Every gauge counts from its own zero mark, the Pegelnullpunkt. That mark is neither the river bed nor sea level, it was fixed at some point in the past, and it sits at a different height at every gauge: 97.72 m above sea level at Maxau, 8.00 m at Emmerich. So 0 cm does not mean "no water", it means "the water is exactly at that mark".

A water level is therefore like a temperature in Celsius. You can work out what percentage of the yearly average today's 15 °C is, and the answer tells you nothing, because 0 °C is not "no temperature". The same thing happens on the Rhine:

station             W cm   %MW    Q m3/s   %MQ
Worms                  9    4%       480   35%
Duisburg-Ruhrort     149   36%       656   30%

Same river, same day, and NIWIS puts both in the worst class, "extrem niedrig". By water level one is at 4% of its average and the other at 36%, nine times as much. By discharge they are at 35% and 30%, which is what two gauges on one river should look like.

Discharge is different because 0 m³/s really does mean no water. That makes it the number to compare between gauges, and the water level the number a ship's captain reads.

Emmerich shows the problem from the other end: it is at -3 cm today, below its own mark, so its percentage comes out negative.

The Rhine on 2026-08-03

Against the 1991--2020 reference period:

station             Q m3/s   %MQ  %day    W cm
Maxau                  428   35%   37%     320
Speyer                 415   34%   36%     173
Worms                  480   35%   38%       9
Mainz                  532   33%   37%     126
Kaub                   557   34%   38%      28
Andernach              614   30%   39%      30
Bonn                   607   30%   39%      83
Köln                   638   31%   40%      69
Düsseldorf             649   31%   39%      22
Duisburg-Ruhrort       656   30%   38%     149
Wesel                  647   29%   38%      87
Rees                   666   30%   38%      33
Emmerich               695   31%   39%      -3

Q is the discharge, the volume of water passing the gauge every second. It grows downstream as tributaries join, which is why Emmerich carries more than Maxau. %MQ compares it to the long-term annual mean, %day to the median for this same date, and %day is the fairer number because a river runs lower in August than in March anyway.

Every gauge on the river is at 36 to 40% of its normal 3 August flow, so a bit over a third of the usual water, and NIWIS puts all 13 of them in its worst class, "extrem niedrig". Every one of them is also below the lowest discharge measured on a 3 August anywhere in the reference period, which makes today a record low for the date along the whole German Rhine. The decile endpoint supplies those minima, its outermost bounds being the measured extremes of the period. At Kaub, the gauge that Rhine shipping uses to work out how much cargo a vessel can load, the previous minimum for this date is 769 m³/s and the river is running 557.

The values are from 2 August, except the discharge at Emmerich, which is from 30 July because the series has gaps since.

Neckar: level and discharge disagree

Neckar
station                   Q m3/s   %MQ  %day Q class           W cm   %MW  %day W class
Horb                           3   22%   44% sehr niedrig        34   50%   72% niedrig
Wendlingen-Kläranlage         11   29%   56% niedrig             47   57%   75% niedrig
Plochingen                     -     -     - -                  154   93%   99% kein NW
Lauffen                        -     -     - -                  221   89%   95% kein NW
Rockenau SKA                  32   24%   48% sehr niedrig       214   91%   99% kein NW

NIWIS classifies water level and discharge separately, and at Rockenau the two verdicts contradict each other. By water level it is "kein Niedrigwasser", no low water at all. By discharge it is "sehr niedrig", and at 32.5 m³/s it is below the 34.5 m³/s that is the lowest discharge ever measured there on a 3 August. Same gauge, same day.

Below Plochingen the Neckar is canalized. The weirs hold the level while the discharge drops, and Plochingen and Lauffen report no discharge at all. The level measures how full the impoundments are, which is what navigation needs. The discharge measures what the catchment delivers, and at a quarter of the normal volume the water warms up faster and holds less oxygen, and less is available for cooling and abstraction.

Nationwide

Of the 359 stations with discharge data, 310 could be classified for today:

extrem niedrig       108   34.8%
sehr niedrig          64   20.6%
niedrig               81   26.1%
Kein Niedrigwasser    57   18.4%

55% of the classified stations are at "sehr niedrig" or worse. Three days earlier the same query returned 129 at "extrem niedrig" and 35 at "kein Niedrigwasser", and the upper Neckar gauges are on steigend again.

Missing features

A gewaesser filter on /messstelle would remove the 698 extra requests, and sub-daily values would remove the need for a second source. The reference period being a parameter instead of a fixed number baked into a published figure is the reason I would use NIWIS for this kind of question.

I explored the endpoints with Claude Code, one call per endpoint with the raw response dumped to a file. The -777 sentinel, the 1991 limit and the licence spread all came out of that and none of them are in the documentation.