Fabrig status
In flight
Landing now
Nothing is landing right now.
Recent set runs
| Set | Result | Machine | Ended | Run | Landed | Reason |
|---|---|---|---|---|---|---|
| forgejo-elsewhere-3/m4 | landed | hetzner | 10-25 04:06Z | 17:35 | skein, smith, hearth, jig, temper | |
| optimistic-gates-catch-up | stopped in temper | hetzner3 | 10-25 03:49Z | 11:19 | – | land-set: temper: land --no-push failed (4) |
| forgejo-elsewhere-3/m4 | stopped in skein | hetzner | 10-25 03:37Z | 0:43 | – | land-set: skein: land --no-push failed (4) |
| optimistic-gates-catch-up | stopped in smith | hetzner3 | 10-25 03:36Z | 6:55 | – | land-set: smith: land --no-push failed (4) |
Missions
Missions the record holds as active or paused, any mission with something running now (a heavy command, a Codex run or owner, a Claude session named for it), and any the record doesn’t call ended with heavy commands in the last 24 hours. Last activity is its last command, run or owner turn.
| Mission | Record | Machines | Now | Commands 24 h · running | Last activity | Owner |
|---|---|---|---|---|---|---|
| web-look | not recorded | hetzner | session idle | – | – | – |
| kache-align | not recorded | hetzner3 | – | 3 · 0:10 | 2 min ago | – |
| status-page | not recorded | hetzner4 | – | 1 · 0:25 | 15 min ago | – |
| forgejo-elsewhere-3 | active | hetzner | – | 408 · 5:02:25 | 17 min ago | o/forgejo-elsewhere-3@hetzner |
| chatgpt-paste | not recorded | hetzner2 | – | 339 · 2:48:13 | 7 h 16 min ago | – |
| fake-backends | not recorded | hetzner4 | – | 32 · 1:28:40 | 7 h 19 min ago | – |
| first-run-ux-6 | not recorded | hetzner5 | – | 218 · 2:36:35 | 7 h 30 min ago | – |
| github-land-2 | not recorded | hetzner3 | – | 337 · 4:59:00 | 7 h 31 min ago | – |
| landing-flow-2 | not recorded | hetzner3 | – | 3 · 0:16 | 7 h 36 min ago | – |
| upgrades-4 | not recorded | hetzner2 | – | 32 · 21:44 | 7 h 42 min ago | – |
| configuring-3 | not recorded | hetzner5 | – | 57 · 54:58 | 7 h 48 min ago | – |
| case-5 | not recorded | hetzner4 | – | 11 · 14:58 | 7 h 50 min ago | – |
| configuring-3-prep | not recorded | hetzner5 | – | 6 · 2:07 | 10 h 23 min ago | – |
| landing-flow | paused | hetzner3 | – | 6 · 5:51 | 16 h 27 min ago | o/landing-flow@hel8 |
| sign-in-land | not recorded | hetzner | – | 8 · 8:47 | 23 h 26 min ago | – |
| change-page | paused | hetzner4 | – | – | – | o/change-page@hetzner4 |
| hearth | paused | hetzner | – | – | – | o/hearth@hetzner |
| hearth-skein-files | paused | hetzner | – | – | – | o/hearth-skein-files@hetzner |
| hearth-skein-trees | paused | hetzner | – | – | – | o/hearth-skein-trees@hetzner |
| site | paused | hetzner5 | – | – | – | o/site@hetzner5 |
Machines
hel8 reachable
debian-8gb-hel1-1 · load 0.1 / 4 CPUs · 6 GB memory free · 46 GB disk free
last heavy command 2 h 8 min ago
Claude sessions
- idle d/render planner-5 · background · up 7 d
- idle d/render planner-7 · background · up 7 d
- idle render supervisor · background · up 7 d
Unread post
- 1 unread d/roadmap-2 · oldest 1 h 28 min ago
- 2 unread director · oldest 4 h 52 min ago
hetzner reachable
debian-16gb-hel1-1 · load 0.1 / 8 CPUs · 13 GB memory free · 220 GB disk free
last heavy command 17 min ago
Claude sessions
- idle parity supervisor · background · up 10 d
- idle o/web-look · background · up 36 h 56 min
hetzner2 reachable
debian-16gb-hel1-2 · load 0.1 / 8 CPUs · 14 GB memory free · 209 GB disk free
last heavy command just now
Idle: nothing queued, running or unread.
hetzner3 reachable
debian-16gb-hel1-3 · load 1.2 / 8 CPUs · 8 GB memory free · 240 GB disk free
last heavy command 2 min ago
Claude sessions
- busy interactive · up 2 h 48 min
hetzner4 reachable
debian-16gb-hel1-4 · load 1.0 / 8 CPUs · 13 GB memory free · 186 GB disk free
last heavy command just now
Claude sessions
- busy director · background · up 7 min
- busy p/temper-on-temper · background · up <1 min
- busy p/real-backends · background · up <1 min
- busy p/first-release · background · up <1 min
hetzner5 reachable
debian-16gb-hel1-5 · load 0.1 / 8 CPUs · 14 GB memory free · 249 GB disk free
last heavy command 18 min ago
Idle: nothing queued, running or unread.
Costs
Command time on the fleet, last 24 hours
Hours of heavy command time per hour, every machine together: running, and waiting in heavy’s queue before it.
As a table
| From | Running | Waiting | Commands ended |
|---|---|---|---|
| 10-24 06:00Z | 5.7 h | 6.9 h | 331 |
| 10-24 07:00Z | 4.9 h | 8.2 h | 407 |
| 10-24 08:00Z | 4.4 h | 5.9 h | 338 |
| 10-24 09:00Z | 4.2 h | 5.3 h | 278 |
| 10-24 10:00Z | 4.0 h | 8.7 h | 207 |
| 10-24 11:00Z | 4.3 h | 4.1 h | 291 |
| 10-24 12:00Z | 4.2 h | 14.3 h | 223 |
| 10-24 13:00Z | 5.2 h | 6.3 h | 264 |
| 10-24 14:00Z | 3.3 h | 5.3 h | 235 |
| 10-24 15:00Z | 3.9 h | 5.4 h | 231 |
| 10-24 16:00Z | 3.2 h | 3.2 h | 192 |
| 10-24 17:00Z | 3.8 h | 2.2 h | 198 |
| 10-24 18:00Z | 3.4 h | 2.2 h | 245 |
| 10-24 19:00Z | 4.3 h | 2.3 h | 298 |
| 10-24 20:00Z | 3.3 h | 2.5 h | 240 |
| 10-24 21:00Z | 3.9 h | 1.6 h | 237 |
| 10-24 22:00Z | 0.7 h | 0.1 h | 17 |
| 10-24 23:00Z | 0.3 h | 0.0 h | 3 |
| 10-25 00:00Z | 0.2 h | 0.0 h | 12 |
| 10-25 01:00Z | 0.4 h | 0.0 h | 15 |
| 10-25 03:00Z | 0.6 h | 0.3 h | 47 |
| 10-25 04:00Z | 0.1 h | 0.0 h | 5 |
| 10-25 05:00Z | 0.1 h | 0.0 h | 20 |
Builds and tests by kind
Every heavy command, by the kind record runs counts (records.md 5.1); land’s own
commands are its prebuild and gate. Failed is a nonzero exit.
| Kind | Commands | Failed | Run median | Run p90 | Wait median | Wait p90 | Running | Waiting |
|---|---|---|---|---|---|---|---|---|
| Tests focused | 526 | 164 | 0:09 | 0:41 | 0:03 | 3:24 | 4.1 h | 10.2 h |
| Tests package | 1,226 | 415 | 0:16 | 1:41 | 0:03 | 2:57 | 14.1 h | 23.3 h |
| Tests workspace | 299 | 60 | 0:57 | 3:21 | 0:03 | 2:24 | 7.3 h | 3.9 h |
| Builds and lints clippy | 467 | 222 | 0:17 | 1:02 | 0:04 | 2:55 | 3.6 h | 8.5 h |
| Builds and lints check | 69 | 46 | 0:23 | 1:03 | 0:02 | 1:54 | 0.6 h | 0.8 h |
| Builds and lints fmt | 436 | 31 | 0:10 | 0:13 | 0:03 | 2:36 | 1.1 h | 8.7 h |
| Builds and lints check.sh | 24 | 8 | 2:08 | 4:57 | 0:05 | 1:00 | 0.9 h | 0.2 h |
| Builds and lints other | 1,116 | 219 | 0:13 | 5:14 | 0:05 | 4:29 | 28.5 h | 27.2 h |
| Landing land prebuild | 154 | 13 | 2:30 | 4:53 | 0:03 | 2:13 | 6.3 h | 2.3 h |
| Landing land gate | 153 | 30 | 1:26 | 2:08 | 0:27 | 4:47 | 4.0 h | 4.6 h |
By machine
| Machine | Commands | Failed | Running | Waiting | Wait median | Wait p90 | Land gates |
|---|---|---|---|---|---|---|---|
| hel8 | 12 | 2 | 0.1 h | 0.0 h | 0:02 | 0:02 | – |
| hetzner | 1,078 | 256 | 14.1 h | 12.9 h | 0:04 | 1:57 | 42 |
| hetzner2 | 1,197 | 338 | 10.5 h | 9.2 h | 0:03 | 1:00 | 42 |
| hetzner3 | 626 | 184 | 13.5 h | 38.4 h | 0:52 | 10:09 | 33 |
| hetzner4 | 635 | 116 | 14.2 h | 15.1 h | 0:05 | 5:08 | 20 |
| hetzner5 | 922 | 312 | 18.2 h | 14.0 h | 0:02 | 2:57 | 16 |
Landing: gates by repository
Every land on any machine, plain or under land-set, from heavy’s
rows: its prebuild in the build lane, then its gate.
| Repository | Gates | Failed | Gate median | Gate p90 | Gate wait | Prebuild median | Prebuild wait | Prebuild failed |
|---|---|---|---|---|---|---|---|---|
| agents | 15 | 4 | 0:27 | 9:11 | 0:05 | – | – | – |
| hearth | 3 | – | 0:30 | 0:32 | 0:06 | 0:22 | 0:03 | 1 |
| jig | 21 | 1 | 1:09 | 1:49 | 0:06 | 3:08 | 0:02 | – |
| r1-repo | 2 | – | 1:32 | 1:49 | 0:18 | 3:01 | 0:02 | – |
| skein | 22 | 1 | 1:44 | 2:17 | 1:05 | 1:23 | 0:03 | 1 |
| smith | 13 | 1 | 0:20 | 0:24 | 0:13 | 1:17 | 0:03 | 1 |
| temper | 77 | 23 | 1:28 | 2:04 | 0:44 | 3:13 | 0:03 | 10 |
Landing: gate steps under land-set
From the landings feed’s results (landing-flow.md 10): the medians of each
gate-step.
| Repository | Gates run | Cached | Failed | Gate median | Gate p90 | Step medians |
|---|---|---|---|---|---|---|
| hearth | 2 | 1 | – | 0:31 | 0:32 | fmt 0:01 · clippy 0:00 · build 0:01 · focused 0:07 · fuzzy 0:19 · other 0:03 |
| jig | 2 | 1 | – | 0:38 | 0:40 | fmt 0:06 · clippy 0:02 · wasm 0:04 · build 0:02 · focused 0:12 · fuzzy 0:10 · other 0:03 |
| skein | 1 | 1 | 1 | 0:37 | 0:37 | fmt 0:04 · clippy 0:01 · wasm 0:02 · build 0:01 · focused 0:08 · fuzzy 0:14 · other 0:07 |
| smith | 3 | 1 | – | 0:22 | 0:24 | fmt 0:03 · clippy 0:01 · build 0:01 · focused 0:08 · fuzzy 0:06 · other 0:03 |
| temper | 1 | 1 | 1 | 0:50 | 0:50 | fmt 0:08 · clippy 0:01 · wasm 0:02 · build 0:02 · focused 0:13 · fuzzy 0:16 · other 0:07 |
Landing: set runs
Lock wait is waiting to started; the run is started to
landed or stopped.
| Set | Runs | Landed | Stopped | Lock wait median | Longest | Run median | Longest |
|---|---|---|---|---|---|---|---|
| forgejo-elsewhere-3/m4 | 2 | 1 | 1 | 5:00 | 5:17 | 9:09 | 17:35 |
| optimistic-gates-catch-up | 2 | – | 2 | 0:00 | 0:00 | 9:07 | 11:19 |
Landing: the landing turn
How long each machine’s land waited for, and held, its repository’s turn.
| Repository | Turns | Wait median | Wait p90 | Held median | Held p90 |
|---|---|---|---|---|---|
| temper | 82 | 0:18 | 23:03 | 5:30 | 17:20 |
Codex runs
Every codex-run task that ended, from each machine’s runs.tsv.
| Model and effort | Runs | Failed | Median | p90 | Input tokens | Cached | Output tokens |
|---|---|---|---|---|---|---|---|
| gpt-6.1-sol high | 1 | – | 9:38 | 9:38 | 6.5M | 99% | 16.4K |
| gpt-6.1-sol medium | 4 | – | 1:54 | 2:45 | 22.6M | 97% | 59.6K |
Command time on the fleet, last 7 days
Hours of heavy command time per 4 hours, every machine together: running, and waiting in heavy’s queue before it.
As a table
| From | Running | Waiting | Commands ended |
|---|---|---|---|
| 10-18 16:00Z | 0.9 h | 0.0 h | 31 |
| 10-18 20:00Z | 0.6 h | 0.1 h | 96 |
| 10-19 00:00Z | 0.1 h | 0.0 h | 11 |
| 10-19 04:00Z | 0.3 h | 0.0 h | 11 |
| 10-19 08:00Z | 0.1 h | 0.0 h | 10 |
| 10-19 12:00Z | 0.7 h | 0.2 h | 88 |
| 10-19 16:00Z | 1.7 h | 0.2 h | 197 |
| 10-20 00:00Z | 4.3 h | 0.7 h | 447 |
| 10-20 04:00Z | 1.7 h | 0.5 h | 230 |
| 10-20 08:00Z | 5.2 h | 9.1 h | 619 |
| 10-20 12:00Z | 6.8 h | 13.5 h | 1279 |
| 10-20 16:00Z | 13.5 h | 20.0 h | 1905 |
| 10-20 20:00Z | 10.9 h | 6.4 h | 1595 |
| 10-21 00:00Z | 8.6 h | 7.0 h | 931 |
| 10-21 04:00Z | 8.9 h | 17.2 h | 1024 |
| 10-21 08:00Z | 1.1 h | 0.6 h | 28 |
| 10-21 16:00Z | 0.7 h | 0.5 h | 145 |
| 10-21 20:00Z | 4.9 h | 4.6 h | 807 |
| 10-22 00:00Z | 8.2 h | 6.6 h | 1172 |
| 10-22 04:00Z | 15.0 h | 15.7 h | 1691 |
| 10-22 08:00Z | 14.4 h | 11.0 h | 1554 |
| 10-22 12:00Z | 14.3 h | 16.8 h | 1713 |
| 10-22 16:00Z | 13.5 h | 23.0 h | 981 |
| 10-22 20:00Z | 7.9 h | 5.3 h | 761 |
| 10-23 00:00Z | 10.2 h | 11.5 h | 1089 |
| 10-23 04:00Z | 12.7 h | 23.7 h | 1305 |
| 10-23 08:00Z | 15.7 h | 13.8 h | 1382 |
| 10-23 12:00Z | 10.2 h | 13.2 h | 895 |
| 10-23 16:00Z | 17.2 h | 22.4 h | 1681 |
| 10-23 20:00Z | 14.8 h | 11.7 h | 1684 |
| 10-24 00:00Z | 20.2 h | 29.6 h | 1955 |
| 10-24 04:00Z | 19.5 h | 48.4 h | 1322 |
| 10-24 08:00Z | 16.9 h | 24.1 h | 1114 |
| 10-24 12:00Z | 16.6 h | 31.3 h | 953 |
| 10-24 16:00Z | 14.7 h | 9.9 h | 933 |
| 10-24 20:00Z | 8.2 h | 4.1 h | 497 |
| 10-25 00:00Z | 1.2 h | 0.3 h | 74 |
| 10-25 04:00Z | 0.2 h | 0.0 h | 25 |
Builds and tests by kind
Every heavy command, by the kind record runs counts (records.md 5.1); land’s own
commands are its prebuild and gate. Failed is a nonzero exit.
| Kind | Commands | Failed | Run median | Run p90 | Wait median | Wait p90 | Running | Waiting |
|---|---|---|---|---|---|---|---|---|
| Tests focused | 3,415 | 1380 | 0:09 | 0:38 | 0:02 | 2:13 | 18.2 h | 43.5 h |
| Tests package | 9,982 | 3886 | 0:08 | 0:47 | 0:02 | 2:04 | 59.9 h | 120.3 h |
| Tests workspace | 1,572 | 443 | 0:51 | 3:01 | 0:02 | 2:10 | 34.0 h | 18.2 h |
| Builds and lints clippy | 4,390 | 2238 | 0:09 | 0:42 | 0:02 | 2:10 | 21.8 h | 53.8 h |
| Builds and lints check | 987 | 584 | 0:10 | 0:50 | 0:01 | 1:27 | 5.2 h | 7.6 h |
| Builds and lints fmt | 3,474 | 228 | 0:05 | 0:10 | 0:03 | 2:35 | 6.0 h | 52.8 h |
| Builds and lints check.sh | 107 | 43 | 1:44 | 4:07 | 0:03 | 2:54 | 3.6 h | 1.1 h |
| Builds and lints other | 5,410 | 1034 | 0:10 | 3:10 | 0:02 | 2:39 | 92.5 h | 78.3 h |
| Landing land prebuild | 1,478 | 61 | 1:32 | 3:43 | 0:01 | 1:33 | 46.7 h | 11.8 h |
| Landing land gate | 1,426 | 155 | 1:24 | 2:08 | 0:01 | 1:54 | 34.8 h | 16.0 h |
By machine
| Machine | Commands | Failed | Running | Waiting | Wait median | Wait p90 | Land gates |
|---|---|---|---|---|---|---|---|
| hel8 | 2,248 | 631 | 20.4 h | 4.0 h | 0:01 | 0:11 | 94 |
| hetzner | 7,597 | 2345 | 80.9 h | 73.0 h | 0:02 | 1:57 | 305 |
| hetzner2 | 10,491 | 3225 | 79.5 h | 106.5 h | 0:02 | 1:52 | 507 |
| hetzner3 | 4,610 | 1514 | 52.5 h | 92.4 h | 0:02 | 3:32 | 192 |
| hetzner4 | 5,249 | 1619 | 57.4 h | 90.9 h | 0:03 | 2:59 | 241 |
| hetzner5 | 2,046 | 718 | 32.1 h | 36.5 h | 0:02 | 3:24 | 87 |
Landing: gates by repository
Every land on any machine, plain or under land-set, from heavy’s
rows: its prebuild in the build lane, then its gate.
| Repository | Gates | Failed | Gate median | Gate p90 | Gate wait | Prebuild median | Prebuild wait | Prebuild failed |
|---|---|---|---|---|---|---|---|---|
| agents | 15 | 4 | 0:27 | 9:11 | 0:05 | – | – | – |
| hearth | 4 | – | 0:30 | 0:32 | 0:06 | 0:36 | 0:03 | 1 |
| jig | 145 | 14 | 1:34 | 2:19 | 0:17 | 3:05 | 0:02 | 6 |
| p1-ordinary | 3 | – | 1:02 | 1:47 | 0:02 | 0:42 | 0:01 | 1 |
| p1-single | 3 | – | 1:55 | 4:22 | 0:02 | – | – | – |
| p2-ordinary | 3 | – | 1:00 | 1:49 | 0:07 | 1:18 | 0:00 | – |
| p2-single | 3 | – | 1:51 | 4:04 | 0:02 | – | – | – |
| p3-ordinary | 3 | – | 1:03 | 1:46 | 0:07 | 1:20 | 0:01 | – |
| p3-single | 3 | – | 1:55 | 4:25 | 0:02 | – | – | – |
| r1-repo | 2 | – | 1:32 | 1:49 | 0:18 | 3:01 | 0:02 | – |
| skein | 359 | 16 | 1:31 | 2:22 | 0:01 | 1:06 | 0:01 | 18 |
| skein-atomic-absent | 1 | – | 1:26 | 1:26 | 0:00 | 1:47 | 0:01 | – |
| smith | 92 | 17 | 0:35 | 1:39 | 0:01 | 2:02 | 0:01 | 3 |
| temper | 790 | 104 | 1:19 | 1:59 | 0:01 | 1:42 | 0:01 | 32 |
Landing: gate steps under land-set
From the landings feed’s results (landing-flow.md 10): the medians of each
gate-step.
| Repository | Gates run | Cached | Failed | Gate median | Gate p90 | Step medians |
|---|---|---|---|---|---|---|
| hearth | 2 | 1 | – | 0:31 | 0:32 | fmt 0:01 · clippy 0:00 · build 0:01 · focused 0:07 · fuzzy 0:19 · other 0:03 |
| jig | 2 | 1 | – | 0:38 | 0:40 | fmt 0:06 · clippy 0:02 · wasm 0:04 · build 0:02 · focused 0:12 · fuzzy 0:10 · other 0:03 |
| skein | 1 | 1 | 1 | 0:37 | 0:37 | fmt 0:04 · clippy 0:01 · wasm 0:02 · build 0:01 · focused 0:08 · fuzzy 0:14 · other 0:07 |
| smith | 3 | 1 | – | 0:22 | 0:24 | fmt 0:03 · clippy 0:01 · build 0:01 · focused 0:08 · fuzzy 0:06 · other 0:03 |
| temper | 1 | 1 | 1 | 0:50 | 0:50 | fmt 0:08 · clippy 0:01 · wasm 0:02 · build 0:02 · focused 0:13 · fuzzy 0:16 · other 0:07 |
Landing: set runs
Lock wait is waiting to started; the run is started to
landed or stopped.
| Set | Runs | Landed | Stopped | Lock wait median | Longest | Run median | Longest |
|---|---|---|---|---|---|---|---|
| forgejo-elsewhere-3/m4 | 2 | 1 | 1 | 5:00 | 5:17 | 9:09 | 17:35 |
| optimistic-gates-catch-up | 2 | – | 2 | 0:00 | 0:00 | 9:07 | 11:19 |
Landing: the landing turn
How long each machine’s land waited for, and held, its repository’s turn.
| Repository | Turns | Wait median | Wait p90 | Held median | Held p90 |
|---|---|---|---|---|---|
| temper | 265 | 0:51 | 11:41 | 4:46 | 10:54 |
Codex runs
Every codex-run task that ended, from each machine’s runs.tsv.
| Model and effort | Runs | Failed | Median | p90 | Input tokens | Cached | Output tokens |
|---|---|---|---|---|---|---|---|
| gpt-6.1-sol high | 1 | – | 9:38 | 9:38 | 6.5M | 99% | 16.4K |
| gpt-6.1-sol medium | 4 | – | 1:54 | 2:45 | 22.6M | 97% | 59.6K |