Agreguesi i feed

If you have recently installed a very up-to-date Linux distribution with a desktop environment, or upgraded your system on a rolling-release distribution, you might have noticed that your home directory has a new folder: “Projects”

Why?

With the recent 0.20 release of xdg-user-dirs we enabled the “Projects” directory by default. Support for this has already existed since 2007, but was never formally enabled. This closes a more than 11 year old bug report that asked for this feature.

The purpose of the Projects directory is to give applications a default location to place project files that do not cleanly belong into one of the existing categories (Documents, Music, Pictures, Videos). Examples of this are software engineering projects, scientific projects, 3D printing projects, CAD design or even things like video editing projects, where project files would end up in the “Projects” directory, with output video being more at home in “Videos”.

By enabling this by default, and subsequently in the coming months adding support to GLib, Flatpak, desktops and applications that want to make use of it, we hope to give applications that do operate in a “project-centric” manner with mixed media a better default storage location. As of now, those tools either default to the home directory, or will clutter the “Documents” folder, both of which is not ideal. It also gives users a default organization structure, hopefully leading to less clutter overall and better storage layouts.

This sucks, I don’t like it!

As usual, you are in control and can modify your system’s behavior. If you do not like the “Projects” folder, simply delete it! The xdg-user-dirs utility will not try to create it again, and instead adjust the default location for this directory to your home directory. If you want more control, you can influence exactly what goes where by editing your ~/.config/user-dirs.dirs configuration file.

If you are a system administrator or distribution vendor and want to set default locations for the default XDG directories, you can edit the /etc/xdg/user-dirs.defaults file to set global defaults that affect all users on the system (users can still adjust the settings however they like though).

What else is new?

Besides this change, the 0.20 release of xdg-user-dirs brings full support for the Meson build system (dropping Automake), translation updates, and some robustness improvements to its code. We also fixed the “arbitrary code execution from unsanitized input” bug that the Arch Linux Wiki mentions here for the xdg-user-dirs utility, by replacing the shell script with a C binary.

Thanks to everyone who contributed to this release!

Table of Contents

• Introduction
• The problem
• Architecture overview
• The VCL layer
• The POST-to-GET conversion
• Protecting private repositories
• The Lua layer
• Debugging the rollout
• How we got here
• Conclusions

Introduction

One of the most visible signs that GNOME’s infrastructure has grown over the years is the amount of CI traffic that flows through gitlab.gnome.org on any given day. Hundreds of pipelines run in parallel, most of them starting with a git clone or git fetch of the same repository, often at the same commit. All that traffic was landing directly on GitLab’s webservice pods, generating redundant load for work that was essentially identical.

GNOME’s infrastructure runs on AWS, which generously provides credits to the project. Even so, data transfer is one of the largest cost drivers we face, and we have to operate within a defined budget regardless of those credits. The bandwidth costs associated with this Git traffic grew significant enough that for a period of time we redirected unauthenticated HTTPS Git pulls to our GitHub mirrors as a short-term cost mitigation. That measure bought us some breathing room, but it was never meant to be permanent: sending users to a third-party platform for what is essentially a core infrastructure operation is not a position we wanted to stay in. The goal was always to find a proper solution on our own infrastructure.

This post documents the caching layer we built to address that problem. The solution sits between the client and GitLab, intercepts Git fetch traffic, and routes it through Fastly’s CDN so that repeated fetches of the same content are served from cache rather than generating a fresh pack every time. The design went through several iterations — this post presents the final architecture first, then walks through how we got here for readers interested in the evolution.

The problem

The Git smart HTTP protocol uses two endpoints: info/refs for capability advertisement and ref discovery, and git-upload-pack for the actual pack generation. The second one is the expensive one. When a CI job runs git fetch origin main, GitLab has to compute and send the entire pack for that fetch negotiation. If ten jobs run the same fetch within a short window, GitLab does that work ten times.

The tricky part is that git-upload-pack is a POST request with a binary body that encodes what the client already has (have lines) and what it wants (want lines). Traditional HTTP caches ignore POST bodies entirely. Building a cache that actually understands those bodies and deduplicates identical fetches requires some work at the edge.

For a fresh clone the body contains only want lines — one per ref the client is requesting:

0032want 7d20e995c3c98644eb1c58a136628b12e9f00a78 0032want 93e944c9f728a4b9da506e622592e4e3688a805c 0032want ef2cbad5843a607236b45e5f50fa4318e0580e04 ...

For an incremental fetch the body is a mix of want lines (what the client needs) and have lines (commits the client already has locally), which the server uses to compute the smallest possible packfile delta:

00a4want 51a117587524cbdd59e43567e6cbd5a76e6a39ff 0000 0032have 8282cff4b31dce12e100d4d6c78d30b1f4689dd3 0032have be83e3dae8265fdc4c91f11d5778b20ceb4e2479 0032have 7d46abdf9c5a3f119f645c8de6d87efffe3889b8 ...

The leading four hex characters on each line are the pkt-line length prefix. The server walks back through history from the wanted commits until it finds a common ancestor with the have set, then packages everything in between into a packfile. Two CI jobs running the same pipeline at the same commit will produce byte-for-byte identical request bodies and therefore identical responses — exactly the property a cache can help with.

Architecture overview

The current architecture has three components:

• Fastly as the user-facing CDN for gitlab.gnome.org, with custom VCL that intercepts git-upload-pack traffic, hashes the request body, converts the POST to a GET, and caches the response at edge POPs worldwide
• OpenResty (Nginx + LuaJIT) running as the origin server, with a Lua script that restores the original POST, checks a Valkey denylist for private repositories, and signals cacheability back to Fastly
• Valkey + webhook — a small Valkey instance stores a denylist of private repository paths, kept in sync by a webhook service that listens for GitLab project visibility changes

flowchart TD client["Git client / CI runner"] edge["Fastly Edge POP (nearest)"] shield["Fastly Shield POP (IAD)"] nginx["OpenResty Nginx (origin)"] lua["Lua: git_upload_pack.lua"] valkey["Valkey denylist"] gitlab["GitLab webservice"] webhook["gitlab-git-cache-webhook"] gitlab_events["GitLab project events"] client -- "POST /git-upload-pack" --> edge edge -- "HIT → serve from edge" --> client edge -- "MISS → forward to shield" --> shield shield -- "HIT → return to edge (edge caches)" --> edge shield -- "MISS → fetch from origin" --> nginx nginx --> lua lua -- "authenticated? check denylist" --> valkey lua -- "denied/error: keep auth, skip cache" --> gitlab lua -- "allowed: keep auth, signal cacheable" --> gitlab gitlab -- "packfile response" --> nginx nginx -- "X-Git-Cacheable: 1 (if allowed)" --> shield gitlab_events --> webhook webhook -- "SET/DEL git:deny:" --> valkey

The request flow:

1. The POST /git-upload-pack arrives at the nearest Fastly edge POP.
2. VCL checks the body: if Content-Length exceeds 8 KB (the limit of what Fastly can read from req.body), or the body does not contain command=fetch, the request is passed through uncached.
3. VCL hashes the body with SHA256 to build the cache key, base64-encodes the body into X-Git-Original-Body, and converts the request to GET. If the request carries authentication headers (Authorization, PRIVATE-TOKEN, Job-Token), VCL sets X-Git-Auth-Passthrough to flag it — but the request still enters the cache lookup.
4. On a cache hit at the edge, the packfile is served immediately — regardless of whether the request is authenticated or not.
5. On a miss, the request routes to the IAD shield POP. If the shield has it cached, it returns the object and the edge caches it locally.
6. On a shield miss, the request reaches Nginx at the origin. Lua detects X-Git-Original-Body and restores the POST body. If X-Git-Auth-Passthrough is set, Lua checks the Valkey denylist: if the repo is private (or Valkey is unreachable), the Authorization header is preserved and cacheability is not signaled — the response passes through uncached. If the repo is not on the denylist, Authorization is preserved (internal repos need it for GitLab to return 200) and cacheability is signaled.
7. For unauthenticated requests (no passthrough flag), Lua strips Authorization and signals cacheability unconditionally — these are by definition accessing public repositories.
8. The response flows back through the shield and the edge. If X-Git-Cacheable: 1 is present, both nodes cache the response. Subsequent requests — authenticated or not — for the same cache key are served directly from cache.

The VCL layer

The vcl_recv snippet runs at priority 9, before the existing enable_segmented_caching snippet at priority 10 which would otherwise return(pass) for non-asset URLs:

# Snippet git-cache-vcl-recv : 9 # Edge: convert POST to GET, hash body, encode body in header if (req.url ~ "/git-upload-pack$" && req.request == "POST") { if (std.atoi(req.http.Content-Length) > 8192) { return(pass); } if (req.body !~ "command=fetch") { return(pass); } set req.http.X-Git-Cache-Key = "v3:" digest.hash_sha256(req.body); set req.http.X-Git-Original-Body = digest.base64(req.body); # Flag authenticated requests — they still enter the cache lookup, # but on a miss Lua uses this to decide whether to cache the response if (req.http.Authorization || req.http.PRIVATE-TOKEN || req.http.Job-Token) { set req.http.X-Git-Auth-Passthrough = "1"; } set req.request = "GET"; set req.backend = F_Host_1; if (req.restarts == 0) { set req.backend = fastly.try_select_shield(ssl_shield_iad_va_us, F_Host_1); } return(lookup); } # Shield: request already converted to GET by the edge if (req.http.X-Git-Cache-Key) { set req.backend = F_Host_1; return(lookup); }

Authenticated requests — CI runners with Authorization: Basic <gitlab-ci-token:TOKEN>, API clients with PRIVATE-TOKEN or Job-Token — are no longer sent straight to origin. Instead, VCL flags them with X-Git-Auth-Passthrough and lets them enter the cache lookup. On a cache hit, the packfile is served directly from the edge — no origin contact, no credential validation needed, because the cached object can only exist if a previous request already established that the repository is public (see Protecting private repositories). On a cache miss, the flagged request reaches origin where Lua checks the Valkey denylist to decide whether the response should be cached.

The command=fetch filter means only Git protocol v2 fetch commands are cached. The ls-refs command is excluded because its request body is essentially static — caching it with a long TTL would serve stale ref listings after a push. Fetch bodies encode exactly the SHAs the client wants and already has, making them safe to cache indefinitely.

The v3: prefix is a cache version string. Bumping it invalidates all existing cache entries without touching Fastly’s purge API.

The second if block handles the shield. When a cache miss at the edge forwards the request to the shield POP, the shield runs vcl_recv again. At that point the request is already a GET (the edge converted it), so the first block’s req.request == "POST" check will not match. Without the second block, the request would fall through to the enable_segmented_caching snippet, which returns pass for any URL that is not an artifact or archive — effectively preventing the shield from ever caching git traffic.

The vcl_hash snippet overrides the default URL-based hash when a cache key is present:

# Snippet git-cache-vcl-hash : 10 if (req.http.X-Git-Cache-Key) { set req.hash += req.http.X-Git-Cache-Key; return(hash); }

The vcl_fetch snippet caches 200 responses that carry the X-Git-Cacheable signal from Nginx:

# Snippet git-cache-vcl-fetch : 100 if (req.http.X-Git-Cache-Key) { if (beresp.status == 200 && beresp.http.X-Git-Cacheable == "1") { set beresp.http.Surrogate-Key = "git-cache " regsub(req.url.path, "/git-upload-pack$", ""); set beresp.cacheable = true; set beresp.ttl = 30d; set beresp.http.X-Git-Cache-Key = req.http.X-Git-Cache-Key; unset beresp.http.Cache-Control; unset beresp.http.Pragma; unset beresp.http.Expires; unset beresp.http.Set-Cookie; return(deliver); } set beresp.ttl = 0s; set beresp.cacheable = false; return(deliver); }

The Surrogate-Key line tags each cached object with both a global git-cache key and the repository path. This enables targeted purging — a single repository’s cache can be flushed with fastly purge --key "/GNOME/glib", or all git cache at once with fastly purge --key "git-cache".

The 30-day TTL is deliberately long. Git pack data is content-addressed: a pack for a given set of want/have lines will always be the same. As long as the objects exist in the repository, the cached pack is valid. The only case where a cached pack could be wrong is if objects were deleted (force-push that drops history, for instance), which is rare and, on GNOME’s GitLab, made even rarer by the Gitaly custom hooks we run to prevent force-pushes and history rewrites on protected namespaces. In those cases the cache version prefix would force a key change rather than relying on TTL expiry.

The X-Git-Cacheable header is intentionally not unset in vcl_fetch. This is important for the shielding architecture: when the shield caches the object, the stored headers include X-Git-Cacheable: 1. When the edge later fetches this object from the shield, the edge’s own vcl_fetch sees the header and knows it is safe to cache locally. If vcl_fetch stripped the header, the edge would never cache — every request would be a local miss that has to travel back to the shield.

The cleanup happens in vcl_deliver, which runs last before the response reaches the client:

# Snippet git-cache-vcl-deliver : 100 if (req.http.X-Git-Cache-Key) { set resp.http.X-Git-Cache-Status = if(fastly_info.state ~ "HIT(?:-|\z)", "HIT", "MISS"); unset resp.http.X-Git-Original-Body; if (!req.http.Fastly-FF) { unset resp.http.X-Git-Cacheable; unset resp.http.X-Git-Cache-Key; } }

The Fastly-FF check distinguishes between inter-POP traffic (shield-to-edge) and the final client response. Fastly-FF is set when the request comes from another Fastly node. On the shield, where the request came from the edge, internal headers like X-Git-Cacheable and X-Git-Cache-Key are preserved — the edge’s vcl_fetch needs them. On the edge, where the request came from the actual client, those headers are stripped from the final response. Only X-Git-Cache-Status is exposed to clients for observability.

The POST-to-GET conversion

This is probably the most unusual part of the design. Fastly’s consistent hashing and shield routing only works for GET requests. POST requests always go straight to origin. Fastly does provide a way to force POST responses into the cache — by returning pass in vcl_recv and setting beresp.cacheable in vcl_fetch — but it is a blunt instrument: there is no consistent hashing, no shield collapsing, and no guarantee that two nodes in the same POP will ever share the cached result.

By converting the POST to a GET in VCL, encoding the body in a header (X-Git-Original-Body), and using a body-derived SHA256 as the cache key, we get consistent hashing and shield-level request collapsing for free. The VCL uses the X-Git-Cache-Key header (not the URL or method) as the cache key, so the GET conversion is invisible to the caching logic.

Fastly’s shield feature routes cache misses through a designated shield node before going to origin. When two different edge nodes both get a MISS for the same cache key simultaneously, the shield node collapses them into a single origin request. This is important because without it, a burst of CI jobs fetching the same commit would all miss, all go to origin in parallel, and GitLab would end up generating the same pack multiple times.

Protecting private repositories

Private repository traffic must never be cached — that would mean storing authenticated git content in a third-party cache and serving it to arbitrary clients. The protection relies on two independent layers.

Layer 1: cache population is restricted. The cache can only be populated when Lua signals cacheability via X-Git-Cacheable: 1. Lua only signals cacheability when the request is either unauthenticated (by definition accessing a public repo) or authenticated for a repo that is not on the Valkey denylist. For private repos, Lua does not signal cacheability, so vcl_fetch sets ttl=0 and cacheable=false — the response is delivered but never stored.

Layer 2: the Valkey denylist. A webhook service listens for GitLab project_create and project_update system hooks. When a project’s visibility is set to private (level 0), the webhook sets a git:deny:<path> key in Valkey. When visibility changes to internal (level 10) or public (level 20), the key is removed. A periodic reconciliation job (reconcile.py) syncs the full denylist against the GitLab API to correct any drift from missed events.

On a cache miss for an authenticated request, Lua checks the denylist:

• Repo is on the denylist (private): Authorization is preserved, cacheability is not signaled. The request proxies to GitLab with credentials intact, GitLab validates the token, the response is returned but never cached.
• Repo is not on the denylist (public/internal): Authorization is preserved (internal repos require it for GitLab to return 200), cacheability is signaled. The response is cached for future requests.
• Valkey is unreachable or returns an error: treated the same as denied — Authorization is preserved, cacheability is not signaled. This fail-closed design means infrastructure failures result in cache misses, never in data leaks.

The denylist only needs to track private repositories, which are a small fraction of the total on GNOME’s GitLab instance. A private repo’s packfile can never enter the cache through two independent mechanisms: the denylist prevents Lua from signaling cacheability, and even if the denylist were somehow wrong, an unauthenticated request to a private repo returns a 401 from GitLab — which vcl_fetch does not cache (it only caches 200 + X-Git-Cacheable).

The Lua layer

With the VCL handling body hashing, the POST-to-GET conversion, and the cache lookup for all requests, the Lua script runs on cache misses that reach origin. Both authenticated and unauthenticated requests can arrive here. The script’s responsibilities are:

1. Detect that the request arrived from Fastly with an encoded body (the X-Git-Original-Body header).
2. Decode and restore the original POST.
3. For authenticated requests, check the Valkey denylist to determine if the repository is private.
4. Signal back to Fastly whether the response is safe to cache.

local redis_helper = require("redis_helper") local redis_host = os.getenv("REDIS_HOST") local redis_port = os.getenv("REDIS_PORT") local encoded_body = ngx.req.get_headers()["X-Git-Original-Body"] if not encoded_body then return end local body = ngx.decode_base64(encoded_body) ngx.req.read_body() ngx.req.set_method(ngx.HTTP_POST) ngx.req.set_body_data(body) ngx.req.set_header("Content-Length", tostring(#body)) ngx.req.clear_header("X-Git-Original-Body") if ngx.req.get_headers()["X-Git-Auth-Passthrough"] then ngx.req.clear_header("X-Git-Auth-Passthrough") local uri = ngx.var.uri local repo_path = uri:match("^/(.+)/git%-upload%-pack$") if repo_path then repo_path = repo_path:gsub("%.git$", "") end local denied, err = redis_helper.is_denied(redis_host, redis_port, repo_path) if err then ngx.log(ngx.WARN, "git-cache: Redis error for ", repo_path, ": ", err, " — keeping auth, skipping cache") end if err or denied then return end ngx.ctx.git_cacheable = true else ngx.req.clear_header("Authorization") ngx.ctx.git_cacheable = true end

The two branches handle the authenticated and unauthenticated paths. When X-Git-Auth-Passthrough is present, the request came from a CI runner or API client. Lua checks the denylist: if the repo is private or Valkey is unreachable, the script returns early — Authorization stays on the request (so GitLab can validate it), and git_cacheable is never set (so the response is not cached). If the repo is not denied, Authorization is preserved and cacheability is signaled. The Authorization header is kept rather than stripped because internal repositories (visibility level 10) require authentication for git operations — stripping it would cause GitLab to return a 401. Public repos work with or without credentials, so keeping the header is safe for both.

For unauthenticated requests (no passthrough flag), Authorization is stripped and cacheability is signaled unconditionally — these are by definition accessing public repositories.

The early return for denied or errored lookups is the fail-closed behavior. The request still proxies to GitLab (the proxy_pass directive in the Nginx location block runs after Lua), but without the cacheable signal, vcl_fetch will not store the response.

The ngx.ctx.git_cacheable flag is picked up by the header_filter_by_lua_block in the Nginx configuration, which translates it into the X-Git-Cacheable: 1 response header that vcl_fetch checks:

location ~ /git-upload-pack$ { client_body_buffer_size 5m; client_max_body_size 5m; access_by_lua_file /etc/nginx/lua/git_upload_pack.lua; header_filter_by_lua_block { if ngx.ctx.git_cacheable then ngx.header["X-Git-Cacheable"] = "1" end } proxy_pass http://gitlab-webservice; ... } Debugging the rollout

The rollout surfaced a few issues worth documenting for anyone building a similar setup on Fastly.

Shielding introduces a second vcl_recv execution. When the edge forwards a cache miss to the shield, the shield runs the entire VCL pipeline from scratch. The POST-to-GET conversion in vcl_recv checks for req.request == "POST", but on the shield the request is already a GET. Without the fallback if (req.http.X-Git-Cache-Key) block, the shield’s vcl_recv would fall through to the segmented caching snippet and return(pass) — making the shield unable to cache anything.

Response headers must survive the shield-to-edge hop. vcl_fetch and vcl_deliver both run on each node independently. If vcl_fetch on the shield strips a header after caching the object, the stored object will not have that header. When the edge fetches from the shield, the edge’s vcl_fetch will not see it. The solution is to only strip internal headers in vcl_deliver on the final client response, using Fastly-FF to distinguish inter-POP traffic from client traffic.

Fastly’s req.body is limited to 8 KB. VCL can only inspect the first 8192 bytes of a request body. For the vast majority of git fetch negotiations — especially shallow clones and CI pipelines fetching recent commits — the body is well under this limit. Requests with larger bodies (deep fetches with many have lines) fall through to return(pass) and are handled directly by GitLab without caching. This is an acceptable tradeoff: those large-body requests are typically unique negotiations that would not benefit from caching anyway.

Git protocol v1 clients are not cached. The VCL filters on command=fetch, which is a Git protocol v2 construct. Protocol v1 uses a different body format (want/have lines without the command= prefix). Since protocol v2 has been the default since git 2.26 (March 2020), the vast majority of traffic benefits from caching. Protocol v1 clients still work correctly — they simply bypass the cache.

Internal repositories require authentication for git operations. An early version of the Lua script stripped Authorization for any repo not on the denylist, assuming that “not private” meant “accessible without credentials.” Internal repositories (visibility level 10) are not on the denylist — their content is not sensitive — but GitLab still requires authentication for git clone/fetch operations on them. Stripping credentials produced a 401 from GitLab. The fix was to preserve Authorization for all authenticated requests that pass the denylist check, regardless of whether the repo is public or internal. Public repos accept the header harmlessly; internal repos require it.

How we got here

The current architecture is the result of two iterations. The sections above describe the final design; this section documents the path we took to get there.

Iteration 1: Separate CDN service with Lua-driven caching

The first version used a separate Fastly CDN service (cdn.gitlab.gnome.org) as the cache layer, with Nginx doing most of the heavy lifting in Lua:

flowchart TD client["Git client / CI runner"] gitlab_gnome["gitlab.gnome.org (Nginx reverse proxy)"] nginx["OpenResty Nginx"] lua["Lua: git_upload_pack.lua"] cdn_origin["/cdn-origin internal location"] fastly_cdn["Fastly CDN"] origin["gitlab.gnome.org via its origin (second pass)"] gitlab["GitLab webservice"] valkey["Valkey denylist"] webhook["gitlab-git-cache-webhook"] gitlab_events["GitLab project events"] client --> gitlab_gnome gitlab_gnome --> nginx nginx --> lua lua -- "check denylist" --> valkey lua -- "private repo: BYPASS" --> gitlab lua -- "public/internal: internal redirect" --> cdn_origin cdn_origin --> fastly_cdn fastly_cdn -- "HIT" --> cdn_origin fastly_cdn -- "MISS: origin fetch" --> origin origin --> gitlab gitlab_events --> webhook webhook -- "SET/DEL git:deny:" --> valkey

In this design, the Lua script did everything: read the POST body, SHA256-hash it to build a cache key, check a Valkey denylist to exclude private repositories, convert the POST to a GET, encode the body in a header, and perform an internal redirect to a /cdn-origin location that proxied to the CDN. On a cache miss, the CDN would fetch from gitlab.gnome.org directly (the “second pass”), where Lua would detect the origin fetch, decode the body, restore the POST, and proxy to GitLab.

Private repositories were protected by a denylist stored in Valkey. A small FastAPI webhook service (gitlab-git-cache-webhook) listened for GitLab system hooks on project_create and project_update events, maintaining git:deny:<path> keys for private repositories (visibility level 0). Internal repositories (level 10) were treated the same as public (level 20) since they are accessible to any authenticated user on the instance.

The Lua script for this design was substantially more complex:

local resty_sha256 = require("resty.sha256") local resty_str = require("resty.string") local redis_helper = require("redis_helper") local redis_host = os.getenv("REDIS_HOST") or "localhost" local redis_port = os.getenv("REDIS_PORT") or "6379" -- Second pass: request arriving from CDN origin fetch. if ngx.req.get_headers()["X-Git-Cache-Internal"] then local encoded_body = ngx.req.get_headers()["X-Git-Original-Body"] if encoded_body then ngx.req.read_body() local body = ngx.decode_base64(encoded_body) ngx.req.set_method(ngx.HTTP_POST) ngx.req.set_body_data(body) ngx.req.set_header("Content-Length", tostring(#body)) ngx.req.clear_header("X-Git-Original-Body") end return end

And on the first pass, it handled hashing, denylist checks, and the CDN redirect:

if not body:find("command=fetch", 1, true) then ngx.header["X-Git-Cache-Status"] = "BYPASS" return end local sha256 = resty_sha256:new() sha256:update(body) local body_hash = resty_str.to_hex(sha256:final()) local cache_key = "v2:" .. repo_path .. ":" .. body_hash local denied, err = redis_helper.is_denied(redis_host, redis_port, repo_path) if denied then return end ngx.req.clear_header("Authorization") ngx.req.set_header("X-Git-Original-Body", ngx.encode_base64(body)) ngx.req.set_method(ngx.HTTP_GET) ngx.req.set_body_data("") return ngx.exec("/cdn-origin" .. uri)

The CDN’s VCL was relatively simple — it used X-Git-Cache-Key for the hash, routed through a shield, and cached 200 responses for 30 days.

This architecture worked, but it had two significant limitations that led to the current design.

Iteration 2: Edge caching with CI runner participation

The first problem with the separate CDN service was geographic. Nginx runs in AWS us-east-1, so from Fastly’s perspective the only client of the CDN was that single instance in Virginia. Every request entered through the IAD POP, which meant the CDN’s edge POPs around the world were never populated. A CI runner in Europe would have its request travel from a European Fastly POP to IAD, then to Nginx, then back to Fastly IAD, and then all the way back — crossing the Atlantic twice for every cache miss.

The fix was to eliminate the separate CDN service and move all the caching logic into the gitlab.gnome.org Fastly service itself. The key insight was that the POST-to-GET conversion and body hashing could happen in Fastly’s VCL rather than in Lua — Fastly provides digest.hash_sha256() and digest.base64() functions that operate directly on req.body. By doing the conversion at the CDN edge, every POP in the network became a potential cache node for git traffic.

The second problem was that the original denylist approach had two flaws. First, its error handling was fail-open: a Valkey connection error would cause the Lua script to assume the repo was public and strip credentials — the wrong default. Second, even after briefly replacing the denylist with a simple VCL auth bypass (return(pass) for any request with Authorization), CI runners were left completely uncached. GitLab CI always injects a CI_JOB_TOKEN into every job, and the runner authenticates with Authorization: Basic <gitlab-ci-token:TOKEN> regardless of whether the repository is public or private. With the auth bypass, every CI clone skipped the cache entirely — safe, but it left the biggest source of redundant traffic unserved.

The current design solves both problems. VCL flags authenticated requests with X-Git-Auth-Passthrough instead of bypassing the cache, letting them participate in cache lookups. On a hit, the cached packfile is served immediately. On a miss, the request reaches Lua at origin, where the flag triggers a denylist check against Valkey — the same denylist and webhook infrastructure from iteration 1, re-deployed with one critical change: fail-closed error handling. A Valkey error or missing connection causes Lua to preserve Authorization and skip cacheability signaling. The request still works (GitLab validates the token and serves the packfile), but the response is not cached. Infrastructure failures result in cache misses, never in data leaks.

The denylist only tracks private repositories (visibility level 0), which are a small fraction of the total on GNOME’s GitLab. Public and internal repositories pass the denylist check, and Lua signals cacheability while preserving the Authorization header — internal repos require it for GitLab to return 200, and public repos accept it harmlessly.

Conclusions

The system has been running in production since April 2026 and has gone through two iterations to reach its current form. Packfiles are cached at Fastly edge POPs worldwide — a CI runner in Europe gets a cache hit served from a European POP rather than making a round trip to the US East coast.

The moving parts are Fastly’s VCL, an OpenResty Nginx instance with a ~30-line Lua script, a Valkey instance storing the private repository denylist, and a small webhook service that keeps the denylist synchronized with GitLab. Private repositories are protected by two independent layers: the Valkey denylist (which prevents cacheability signaling) and GitLab’s own authentication (which rejects unauthenticated access).

If something goes wrong with the cache layer, requests fall through to GitLab directly — the same path they took before caching existed. There is no failure mode where caching breaks git operations. This also means we don’t redirect any traffic to github.com anymore.

That should be all for today, stay tuned!

In our previous episode we wrote a merge sort implementation that runs a bit faster than the one in stdlibc++. The question then becomes, could it be made even faster. If you go through the relevant literature one potential improvement is to do a multiway merge. That is, instead of merging two arrays into one, you merge four into one using, for example, a priority queue.

This seems like a slam dunk for performance.

• Doubling the number of arrays to merge at a time halves the number of total passes needed
• The priority queue has a known static maximum size, so it can be put on the stack, which is guaranteed to be in the cache all the time
• Processing an element takes only log(#lists) comparisons

Implementing multimerge was conceptually straightforward but getting all the gritty details right took a fair bit of time. Once I got it working the end result was slower. And not by a little, either, but more than 30% slower. Trying some optimizations made it a bit faster but not noticeably so.

Why is this so? Maybe there are bugs that cause it to do extra work? Assuming that is not the case, what actually is? Measuring seems to indicate that a notable fraction of the runtime is spent in the priority queue code. Beyond that I got very little to nothing.

The best hypotheses I could come up with has to with the number of comparisons made. A classical merge sort does two if statements per output elements. One to determine which of the two lists has a smaller element at the front and one to see whether removing the element exhausted the list. The former is basically random and the latter is always false except when the last element is processed. This amounts to 0.5 mispredicted branches per element per round.

A priority queue has to do a bunch more work to preserve the heap property. The first iteration needs to check the root and its two children. That's three comparisons for value and two checks whether the children actually exist. Those are much less predictable than the comparisons in merge sort. Computers are really efficient at doing simple things, so it may be that the additional bookkeeping is so expensive that it negates the advantage of fewer rounds.

Or maybe it's something else. Who's to say? Certainly not me. If someone wants to play with the code, the implementation is here. I'll probably delete it at some point as it does not have really any advantage over the regular merge sort.

This winter I was bored and needed something new, so I spent lots of my free time disassembling and analysing Monster World IV for the SEGA Mega Drive. More specifically, I looked at the 2008 Virtual Console revision of the game, which adds an English translation to the original 1994 release.

My long term goal would be to fully disassemble and analyse the game, port it to C or Rust as I do, and then port it to the Game Boy Advance. I don’t have a specific reason to do that, I just think it’s a charming game from a dated but charming series, and I think the Monaster World series would be a perfect fit on the Game Boy Advance. Since a long time, I also wanted to experiment with disassembling or decompiling code, understanding what doing so implies, understanding how retro computing systems work, and understanding the inner workings of a game I enjoy. Also, there is not publicly available disassembly of this game as far as I know.

As Spring is coming, I sense my focus shifting to other projets, but I don’t want this work to be gone forever and for everyone, especially not for future me. Hence, I decided to publish what I have here, so I can come back to it later or so it can benefit someone else.

First, here is the Ghidra project archive. It’s the first time I used Ghidra and I’m certain I did plenty of things wrong, feedback is happily welcome! While I tried to rename things as my understanding of the code grew, it is still quite a mess of clashing name conventions, and I’m certain I got plenty of things wrong.

Then, here is the Rust-written data extractor. It documents how some systems work, both as code and actual documentation. It mainly extracts and documents graphics and their compression methods, glyphs and their compression methods, character encodings, and dialog scripts. Similarly, I’m not a Rust expert, I did my best but I’m certain there is area for improvement, and everything was constantly changing anyway.

There is more information that isn’t documented and is just floating in my head, such as how the entity system works, but I yet have to refine my understanding of it. Same goes for the optimimzations allowed by coding in assembly, such as using specific registers for commonly used arguments. Hopefully I will come back to this project and complete it, at least when it comes to disassembling and documenting the game’s code.

Red Hat just published the Accessibility Conformance Report (ACR) for Red Hat Enterprise Linux 10.

Accessibility Conformance Reports basically document how our software measures up against accessibility standards like WCAG and Section 508. Since RHEL 10 is built on GNOME 47, this report is a good look at how our stack handles various accessibility things from screen readers to keyboard navigation.

Getting a desktop environment to meet these requirements is a huge task and it’s only possible because of the work done by our community in projects like: Orca, GTK, Libadwaita, Mutter, GNOME Shell, core apps, etc…

Kudos to everyone in the GNOME project that cares about improving accessibility. We all know there’s a long way to go before desktop computing is fully accessible to everyone, but we are surely working on that.

If you’re curious about the state of accessibility in the 47 release or how these audits work, you can find the full PDF here.

This post attempts to explain how Huion tablet devices currently integrate into the desktop stack. I'll touch a bit on the Huion driver and the OpenTablet driver but primarily this explains the intended integration[1]. While I have access to some Huion devices and have seen reports from others, there are likely devices that are slightly different. Huion's vendor ID is also used by other devices (UCLogic and Gaomon) so this applies to those devices as well.

This post was written without AI support, so any errors are organic artisian hand-crafted ones. Enjoy.

The graphics tablet stack

First, a short overview of the ideal graphics tablet stack in current desktops. At the bottom is the physical device which contains a significant amount of firmware. That device provides something resembling the HID protocol over the wire (or bluetooth) to the kernel. The kernel typically handles this via the generic HID drivers [2] and provides us with an /dev/input/event evdev node, ideally one for the pen (and any other tool) and one for the pad (the buttons/rings/wheels/dials on the physical tablet). libinput then interprets the data from these event nodes, passes them on to the compositor which then passes them via Wayland to the client. Here's a simplified illustration of this:

Unlike the X11 api, libinput's API works both per-tablet and per-tool basis. In other words, when you plug in a tablet you get a libinput device that has a tablet tool capability and (optionally) a tablet pad capability. But the tool will only show up once you bring it into proximity. Wacom tools have sufficient identifiers that we can a) know what tool it is and b) get a unique serial number for that particular device. This means you can, if you wanted to, track your physical tool as it is used on multiple devices. No-one [3] does this but it's possible. More interesting is that because of this you can also configure the tools individually, different pressure curves, etc. This was possible with the xf86-input-wacom driver in X but only with some extra configuration, libinput provides/requires this as the default behaviour.

The most prominent case for this is the eraser which is present on virtually all pen-like tools though some will have an eraser at the tail end and others (the numerically vast majority) will have it hardcoded on one of the buttons. Changing to eraser mode will create a new tool (the eraser) and bring it into proximity - that eraser tool is logically separate from the pen tool and can thus be configured differently. [4]

Another effect of this per-tool behaviour is also that we know exactly what a tool can do. If you use two different styli with different capabilities (e.g. one with tilt and 2 buttons, one without tilt and 3 buttons), they will have the right bits set. This requires libwacom - a library that tells us, simply: any tool with id 0x1234 has N buttons and capabilities A, B and C. libwacom is just a bunch of static text files with a C library wrapped around those. Without libwacom, we cannot know what any individual tool can do - the firmware and kernel always expose the capability set of all tools that can be used on any particular tablet. For example: wacom's devices support an airbrush tool so any tablet plugged in will announce the capabilities for an airbrush even though >99% of users will never use an airbrush [5].

The compositor then takes the libinput events, modifies them (e.g. pressure curve handling is done by the compositor) and passes them via the Wayland protocol to the client. That protocol is a pretty close mirror of the libinput API so it works mostly the same. From then on, the rest is up to the application/toolkit.

Notably, libinput is a hardware abstraction layer and conversion of hardware events into others is generally left to the compositor. IOW if you want a button to generate a key event, that's done either in the compositor or in the application/toolkit. But the current versions of libinput and the Wayland protocol do support all hardware features we're currently aware of: the various stylus types (including Wacom's lens cursor and mouse-like "puck" devices) and buttons, rings, wheels/dials, and touchstrips on pads. We even support the rather once-off Dell Canvas Totem device.

Huion devices

Huion's devices are HID compatible which means they "work" out of the box but they come in two different modes, let's call them firmware mode and tablet mode. Each tablet device pretends to be three HID devices on the wire and depending on the mode some of those devices won't send events.

Firmware mode

This is the default mode after plugging the device in. Two of the HID devices exposed look like a tablet stylus and a keyboard. The tablet stylus is usually correct (enough) to work OOTB with the generic kernel drivers, it exports the buttons, pressure, tilt, etc. The buttons and strips/wheels/dials on the tablet are configured to send key events. For example, the Inspiroy 2S I have sends b/i/e/Ctrl+S/space/Ctrl+Alt+z for the buttons and the roller wheel sends Ctrl-/Ctrl= depending on direction. The latter are often interpreted as zoom in/out so hooray, things work OOTB. Other Huion devices have similar bindings, there is quite some overlap but not all devices have exactly the same key assignments for each button. It does of course get a lot more interesting when you want a button to do something different - you need to remap the key event (ideally without messing up your key map lest you need to type an 'e' later).

The userspace part is effectively the same, so here's a simplified illustration of what happens in kernel land: Any vendor-specific data is discarded by the kernel (but in this mode that HID device doesn't send events anyway).

Tablet mode

If you read a special USB string descriptor from the English language ID, the device switches into tablet mode. Once in tablet mode, the HID tablet stylus and keyboard devices will stop sending events and instead all events from the device are sent via the third HID device which consists of a single vendor-specific report descriptor (read: 11 bytes of "here be magic"). Those bits represent the various features on the device, including the stylus features and all pad features as buttons/wheels/rings/strips (and not key events!). This mode is the one we want to handle the tablet properly. The kernel's hid-uclogic driver switches into tablet mode for supported devices, in userspace you can use e.g. huion-switcher. The device cannot be switched back to firmware mode but will return to firmware mode once unplugged.

Once we have the device in tablet mode, we can get true tablet data and pass it on through our intended desktop stack. Alas, like ogres there are layers.

hid-uclogic and udev-hid-bpf

Historically and thanks in large parts to the now-discontinued digimend project, the hid-uclogic kernel driver did do the switching into tablet mode, followed by report descriptor mangling (inside the kernel) so that the resulting devices can be handled by the generic HID drivers. The more modern approach we are pushing for is to use udev-hid-bpf which is quite a bit easer to develop for. But both do effectively the same thing: they overlay the vendor-specific data with a normal HID report descriptor so that the incoming data can be handled by the generic HID kernel drivers. This will look like this:

Notable here: the stylus and keyboard may still exist and get event nodes but never send events[6] but the uclogic/bpf-enabled device will be proper stylus/pad event nodes that can be handled by libinput (and thus the rest), with raw hardware data where buttons are buttons.

Challenges

Because in true manager speak we don't have problems, just challenges. And oh boy, we collect challenges as if we'd be organising the olypmics.

hid-uclogic and libinput

First and probably most embarrassing is that hid-uclogic has a different way of exposing event nodes than what libinput expects. This is largely my fault for having focused on Wacom devices and internalized their behaviour for long years. The hid-uclogic driver exports the wheels and strips on separate event nodes - libinput doesn't handle this correctly (or at all). That'd be fixable but the compositors also don't really expect this so there's a bit more work involved but the immediate effect is that those wheels/strips will likely be ignored and not work correctly. Buttons and pens work.

udev-hid-bpf and huion-switcher

hid-uclogic being a kernel driver has access to the underlying USB device. The HID-BPF hooks in the kernel currently do not, so we cannot switch the device into tablet mode from a BPF, we need it in tablet mode already. This means a userspace tool (read: huion-switcher) triggered via udev on plug-in and before the udev-hid-bpf udev rules trigger. Not a problem but it's one more moving piece that needs to be present (but boy, does this feel like the unix way...).

Huion's precious product IDs

By far the most annoying part about anything Huion is that until relatively recently (I don't have a date but maybe until 2 years ago) all of Huion's devices shared the same few USB product IDs. For most of these devices we worked around it by matching on device names but there were devices that had the same product id and device name. At some point libwacom and the kernel and huion-switcher had to implement firmware ID extraction and matching so we could differ between devices with the same 0256:006d usb IDs. Luckily this seems to be in the past now with modern devices now getting new PIDs for each individual device. But if you have an older device, expect difficulties and, worse, things to potentially break after firmware updates when/if the firmware identification string changes. udev-hid-bpf (and uclogic) rely on the firmware strings to identify the device correctly.

edit: and of course less than 24h after posting this I process a bug report about two completely different new devices sharing one of the product IDs

udev-hid-bpf and hid-uclogic

Because we have a changeover from the hid-uclogic kernel driver to the udev-hid-bpf files there are rough edges on "where does this device go". The general rule is now: if it's not a shared product ID (see above) it should go into udev-hid-bpf and not the uclogic driver. Easier to maintain, much more fire-and-forget. Devices already supported by udev-hid-bpf will remain there, we won't implement BPFs for those (older) devices, doubly so because of the aforementioned libinput difficulties with some hid-uclogic features.

Reverse engineering required

The newer tablets are always slightly different so we basically need to reverse-engineer each tablet to get it working. That's common enough for any device but we do rely on volunteers to do this. Mind you, the udev-hid-bpf approach is much simpler than doing it in the kernel, much of it is now copy-paste and I've even had quite some success to get e.g. Claude Code to spit out a 90% correct BPF on its first try. At least the advantage of our approach to change the report descriptor means once it's done it's done forever, there is no maintenance required because it's a static array of bytes that doesn't ever change.

Plumbing support into userspace

Because we're abstracting the hardware, userspace needs to be fully plumbed. This was a problem last year for example when we (slowly) got support for relative wheels into libinput, then wayland, then the compositors, then the toolkits to make it available to the applications (of which I think none so far use the wheels). Depending on how fast your distribution moves, this may mean that support is months and years off even when everything has been implemented. On the plus side these new features tend to only appear once every few years. Nonetheless, it's not hard to see why the "just sent Ctrl=, that'll do" approach is preferred by many users over "probably everything will work in 2027, I'm sure".

So, what stylus is this?

A currently unsolved problem is the lack of tool IDs on all Huion tools. We cannot know if the tool used is the two-button + eraser PW600L or the three-button-one-is-an-eraser-button PW600S or the two-button PW550 (I don't know if it's really 2 buttons or 1 button + eraser button). We always had this problem with e.g. the now quite old Wacom Bamboo devices but those pens all had the same functionality so it just didn't matter. It would matter less if the various pens would only work on the device they ship with but it's apparently quite possible to use a 3 button pen on a tablet that shipped with a 2 button pen OOTB. This is not difficult to solve (pretend to support all possible buttons on all tools) but it's frustrating because it removes a bunch of UI niceties that we've had for years - such as the pen settings only showing buttons that actually existed. Anyway, a problem currently in the "how I wish there was time" basket.

Summary

Overall, we are in an ok state but not as good as we are for Wacom devices. The lack of tool IDs is the only thing not fixable without Huion changing the hardware[7]. The delay between a new device release and driver support is really just dependent on one motivated person reverse-engineering it (our BPFs can work across kernel versions and you can literally download them from a successful CI pipeline). The hid-uclogic split should become less painful over time and the same as the devices with shared USB product IDs age into landfill and even more so if libinput gains support for the separate event nodes for wheels/strips/... (there is currently no plan and I'm somewhat questioning whether anyone really cares). But other than that our main feature gap is really the ability for much more flexible configuration of buttons/wheels/... in all compositors - having that would likely make the requirement for OpenTabletDriver and the Huion tablet disappear.

OpenTabletDriver and Huion's own driver

The final topic here: what about the existing non-kernel drivers?

Both of these are userspace HID input drivers which all use the same approach: read from a /dev/hidraw node, create a uinput device and pass events back. On the plus side this means you can do literally anything that the input subsystem supports, at the cost of a context switch for every input event. Again, a diagram on how this looks like (mostly) below userspace:

Note how the kernel's HID devices are not exercised here at all because we parse the vendor report, create our own custom (separate) uinput device(s) and then basically re-implement the HID to evdev event mapping. This allows for great flexibility (and control, hence the vendor drivers are shipped this way) because any remapping can be done before you hit uinput. I don't immediately know whether OpenTabletDriver switches to firmware mode or maps the tablet mode but architecturally it doesn't make much difference.

From a security perspective: having a userspace driver means you either need to run that driver daemon as root or (in the case of OpenTabletDriver at least) you need to allow uaccess to /dev/uinput, usually via udev rules. Once those are installed, anything can create uinput devices, which is a risk but how much is up for interpretation.

[1] As is so often the case, even the intended state does not necessarily spark joy
[2] Again, we're talking about the intended case here...
[3] fsvo "no-one"
[4] The xf86-input-wacom driver always initialises a separate eraser tool even if you never press that button
[5] For historical reasons those are also multiplexed so getting ABS_Z on a device has different meanings depending on the tool currently in proximity
[6] In our udev-hid-bpf BPFs we hide those devices so you really only get the correct event nodes, I'm not immediately sure what hid-uclogic does
[7] At which point Pandora will once again open the box because most of the stack is not yet ready for non-Wacom tool ids

Because I am bad at giving up on things, I’ve been running my own email server for over 20 years. Some of that time it’s been a PC at the end of a DSL line, some of that time it’s been a Mac Mini in a data centre, and some of that time it’s been a hosted VM. Last year I decided to bring it in house, and since then I’ve been gradually consolidating as much of the rest of my online presence as possible on it. I mentioned this on Mastodon and a couple of people asked for more details, so here we are.

First: my ISP doesn’t guarantee a static IPv4 unless I’m on a business plan and that seems like it’d cost a bunch more, so I’m doing what I described here: running a Wireguard link between a box that sits in a cupboard in my living room and the smallest OVH instance I can, with an additional IP address allocated to the VM and NATted over the VPN link. The practical outcome of this is that my home IP address is irrelevant and can change as much as it wants - my DNS points at the OVH IP, and traffic to that all ends up hitting my server.

The server itself is pretty uninteresting. It’s a refurbished HP EliteDesk which idles at 10W or so, along 2TB of NVMe and 32GB of RAM that I found under a pile of laptops in my office. We’re not talking rackmount Xeon levels of performance, but it’s entirely adequate for everything I’m doing here.

So. Let’s talk about the services I’m hosting.

Web

This one’s trivial. I’m not really hosting much of a website right now, but what there is is served via Apache with a Let’s Encrypt certificate. Nothing interesting at all here, other than the proxying that’s going to be relevant later.

Email

Inbound email is easy enough. I’m running Postfix with a pretty stock configuration, and my MX records point at me. The same Let’s Encrypt certificate is there for TLS delivery. I’m using Dovecot as an IMAP server (again with the same cert). You can find plenty of guides on setting this up.

Outbound email? That’s harder. I’m on a residential IP address, so if I send email directly nobody’s going to deliver it. Going via my OVH address isn’t going to be a lot better. I have a Google Workspace, so in the end I just made use of Google’s SMTP relay service. There’s various commerical alternatives available, I just chose this one because it didn’t cost me anything more than I’m already paying.

Blog

My blog is largely static content generated by Hugo. Comments are Remark42 running in a Docker container. If you don’t want to handle even that level of dynamic content you can use a third party comment provider like Disqus.

Mastodon

I’m deploying Mastodon pretty much along the lines of the upstream compose file. Apache is proxying /api/v1/streaming to the websocket provided by the streaming container and / to the actual Mastodon service. The only thing I tripped over for a while was the need to set the “X-Forwarded-Proto” header since otherwise you get stuck in a redirect loop of Mastodon receiving a request over http (because TLS termination is being done by the Apache proxy) and redirecting to https, except that’s where we just came from.

Mastodon is easily the heaviest part of all of this, using around 5GB of RAM and 60GB of disk for an instance with 3 users. This is more a point of principle than an especially good idea.

Bluesky

I’m arguably cheating here. Bluesky’s federation model is quite different to Mastodon - while running a Mastodon service implies running the webview and other infrastructure associated with it, Bluesky has split that into multiple parts. User data is stored on Personal Data Servers, then aggregated from those by Relays, and then displayed on Appviews. Third parties can run any of these, but a user’s actual posts are stored on a PDS. There are various reasons to run the others, for instance to implement alternative moderation policies, but if all you want is to ensure that you have control over your data, running a PDS is sufficient. I followed these instructions, other than using Apache as the frontend proxy rather than nginx, and it’s all been working fine since then. In terms of ensuring that my data remains under my control, it’s sufficient.

Backups

I’m using borgmatic, backing up to a local Synology NAS and also to my parents’ home (where I have another HP EliteDesk set up with an equivalent OVH IPv4 fronting setup). At some point I’ll check that I’m actually able to restore them.

Conclusion

Most of what I post is now stored on a system that’s happily living under a TV, but is available to the rest of the world just as visibly as if I used a hosted provider. Is this necessary? No. Does it improve my life? In no practical way. Does it generate additional complexity? Absolutely. Should you do it? Oh good heavens no. But you can, and once it’s working it largely just keeps working, and there’s a certain sense of comfort in knowing that my online presence is carefully contained in a small box making a gentle whirring noise.

gedit 50.0 has been released! Here are the highlights since version 49.0 from January. (Some sections are a bit technical).

No Large Language Models AI tools

The gedit project now disallows the use of LLMs for contributions.

The rationales:

Programming can be seen as a discipline between art and engineering. Both art and engineering require practice. It's the action of doing - modifying the code - that permits a deep understanding of it, to ensure correctness and quality.

When generating source code with an LLM tool, the real sources are the inputs given to it: the training dataset, plus the human commands.

Adding something generated to the version control system (e.g., Git) is usually frown upon. Moreover, we aim for reproducible results (to follow the best-practices of reproducible builds, and reproducible science more generally). Modifying afterwards something generated is also a bad practice.

Releasing earlier, releasing more often

To follow more closely the release early, release often mantra, gedit aims for a faster release cadence in 2026, to have smaller deltas between each version. Future will tell how it goes.

The website is now responsive

Since last time, we've made some efforts to the website. Small-screen-device readers should have a more pleasant experience.

libgedit-amtk becomes "The Good Morning Toolkit"

Amtk originally stands for "Actions, Menus and Toolbars Kit". There was a desire to expand it to include other GTK extras that are useful for gedit needs.

A more appropriate name would be libgedit-gtk-extras. But renaming the module - not to mention the project namespace - is more work. So we've chosen to simply continue with the name Amtk, just changing its scope and definition. And - while at it - sprinkle a bit of fun :-)

So there are now four libgedit-* modules:

• libgedit-gfls, aka "libgedit-glib-extras", currently for "File Loading and Saving";
• libgedit-amtk, aka "libgedit-gtk-extras" - it extends GTK for gedit needs at the exception of GtkTextView;
• libgedit-gtksourceview - it extends GtkTextView and is a fork of GtkSourceView, to evolve the library for gedit needs;
• libgedit-tepl - the Text Editor Product Line library, it provides a high-level API, including an application framework for creating more easily new text editors.

Note that all of these are still constantly in construction.

Some code overhaul

Work continues steadily inside libgedit-gfls and libgedit-gtksourceview to streamline document loading.

You might think that it's a problem solved (for many years), but it's actually not the case for gedit. Many improvements are still possible.

Another area of interest is the completion framework (part of libgedit-gtksourceview), where changes are still needed to make it fully functional under Wayland. The popup windows are sometimes misplaced. So between gedit 49.0 and 50.0 some progress has been made on this. The Word Completion gedit plugin works fine under Wayland, while the LaTeX completion with Enter TeX is still buggy since it uses more features from the completion system.

I published three Rust crates:

• name-to-handle-at: Safe, low-level Rust bindings for Linux name_to_handle_at and open_by_handle_at system calls
• pidfd-util: Safe Rust wrapper for Linux process file descriptors (pidfd)
• listen-fds: A Rust library for handling systemd socket activation

They might seem like rather arbitrary, unconnected things – but there is a connection!

systemd socket activation passes file descriptors and a bit of metadata as environment variables to the activated process. If the activated process exec’s another program, the file descriptors get passed along because they are not CLOEXEC. If that process then picks them up, things could go very wrong. So, the activated process is supposed to mark the file descriptors CLOEXEC, and unset the socket activation environment variables. If a process doesn’t do this for whatever reason however, the same problems can arise. So there is another mechanism to help prevent it: another bit of metadata contains the PID of the target. Processes can check it against their own PID to figure out if they were the target of the activation, without having to depend on all other processes doing the right thing.

PIDs however are racy because they wrap around pretty fast, and that’s why nowadays we have pidfds. They are file descriptors which act as a stable handle to a process and avoid the ID wrap-around issue. Socket activation with systemd nowadays also passes a pidfd ID. A pidfd ID however is not the same as a pidfd file descriptor! It is the 64 bit inode of the pidfd file descriptor on the pidfd filesystem. This has the advantage that systemd doesn’t have to install another file descriptor in the target process which might not get closed. It can just put the pidfd ID number into the $LISTEN_PIDFDID environment variable.

Getting the inode of a file descriptor doesn’t sound hard. fstat(2) fills out struct stat which has the st_ino field. The problem is that it has a type of ino_t, which is 32 bits on some systems so we might end up with a process identifier which wraps around pretty fast again.

We can however use the name_to_handle syscall on the pidfd to get a struct file_handle with a f_handle field. The man page helpfully says that “the caller should treat the file_handle structure as an opaque data type”. We’re going to ignore that, though, because at least on the pidfd filesystem, the first 64 bits are the 64 bit inode. With systemd already depending on this and the kernel rule of “don’t break user-space”, this is now API, no matter what the man page tells you.

So there you have it. It’s all connected.

Obviously both pidfds and name_to_handle have more exciting uses, many of which serve my broader goal: making Varlink services a first-class citizen. More about that another time.

On March 17 we released systemd v260 into the wild.

In the weeks leading up to that release (and since then) I have posted a series of serieses of posts to Mastodon about key new features in this release, under the #systemd260 hash tag. In case you aren't using Mastodon, but would like to read up, here's a list of all 21 posts:

• Post #1: NvPCR Measurements for Activated DDIs
• Post #2: Varlink Transport Plugins
• Post #3: Well-Known Varlink Services
• Post #4: .mstack Overlay Mount Stacks
• Post #5: RefreshOnReload= in Service Units
• Post #6: FANCY_NAME= in /etc/os-release
• Post #7: BindNetworkInterface= in Service Units
• Post #8: importctl pull-oci for Acquiring OCI Containers
• Post #9: systemd-report and Metrics API
• Post #10: udev's tpm2_id built-in and the TPM2 Quirks Database
• Post #11: Devicetree/CHID Database
• Post #12: Varlink IPC for systemd-networkd
• Post #13: systemd-vmspawn knows --ephemeral now
• Post #14: systemd-loginds's xaccess Concept
• Post #15: Unprivileged Portable Services
• Post #16: Image Policy Improvements
• Post #17: LUKS Volume Key Fixation
• Post #18: Journal Varlink Access
• Post #19: Nested UID Range Delegation
• Post #20: PrivateUsers=managed
• Post #21: bootctl install as Varlink API

I intend to do a similar series of serieses of posts for the next systemd release (v261), hence if you haven't left tech Twitter for Mastodon yet, now is the opportunity.

My series for v261 will begin in a few weeks most likely, under the #systemd261 hash tag.

In case you are interested, here is the corresponding blog story for systemd v259, here for v258, here for v257, and here for v256.

Suddenly I have been hearing the term Landlock more in (agent) security circles. To me this is a bit weird because while Landlock is absolutely a useful Linux security tool, it’s been a bit obscure and that’s for good reason. It feels to me a lot like the how weird prevalence of the word delve became a clear tipoff that LLMs were the ones writing, not a human.

Here’s my opinion: Agentic LLM AI security is just security.

We do not need to reinvent any fundamental technologies for this. Most uses of agents one hears about provide the ability to execute arbitrary code as a feature. It’s how OpenCode, Claude Code, Cursor, OpenClaw and many more work.

Especially let me emphasize since OpenClaw is popular for some reason right now: You should absolutely not give any LLM tool blanket read and write access to your full user account on your computer. There are many issues with that, but everyone using an LLM needs to understand just how dangerous prompt injection can be. This post is just one of many examples. Even global read access is dangerous because an attacker could exfiltrate your browser cookies or other files.

Let’s go back to Landlock – one prominent place I’ve seen it mentioned is in this project nono.sh pitches itself as a new sandbox for agents. It’s not the only one, but indeed it heavily leans on Landlock on Linux. Let’s dig into this blog post from the author. First of all, I’m glad they are working on agentic security. We both agree: unsandboxed OpenClaw (and other tools!) is a bad idea.

Here’s where we disagree:

With AI agents, the core issue is access without boundaries. We give agents our full filesystem permissions because that’s how Unix works. We give them network access because they need to call APIs. We give them access to our SSH keys, our cloud credentials, our shell history, our browser cookies – not because they need any of that, but because we haven’t built the tooling to say “you can have this, but not that.”

No. We have had usable tooling for “you can have this, but not that” for well over a decade. Docker kicked off a revolution for a reason: docker run <app> is “reasonably completely isolated” from the host system. Since then of course, there’s many OCI runtime implementations, from podman to apple/container on MacOS and more.

If you want to provide the app some credentials, you can just use bind mounts to provide them like docker|podman|ctr -v ~/.config/somecred.json:/etc/cred.json:ro. Notice there the ro which makes it readonly. Yes, it’s that straightforward to have “this but not that”.

Other tools like Flatpak on Linux have leveraged Linux kernel namespacing similar to this to streamline running GUI apps in an isolated way from the host. For a decade.

There’s far more sophisticated tooling built on top of similar container runtimes since then, from having them transparently backed by virtual machines, Kubernetes and similar projects are all about running containers at scale with lots of built up security knowledge.

That doesn’t need reinventing. It’s generic workload technology, and agentic AI is just another workload from the perspective of kernel/host level isolation. There absolutely are some new, novel risks and issues of course: but again the core principle here is we don’t need to reinvent anything from the kernel level up.

Security here really needs to start from defaulting to fully isolating (from the host and other apps), and then only allow-listing in what is needed. That’s again how docker run worked from the start. Also on this topic, Flatpak portals are a cool technology for dynamic resource access on a single host system.

So why do I think Landlock is obscure? Basically because most workloads should already be isolated already per above, and Landlock has heavy overlap with the wide variety of Linux kernel security mechanisms already in use in containers.

The primary pitch of Landlock is more for an application to further isolate itself – it’s at its best when it’s a complement coarse-grained isolation techniques like virtualization or containers. One way to think of it is that often container runtimes don’t grant privileges needed for an application to further spawn its own sub-containers (for kernel attack surface reasons), but Landlock is absolutely a reasonable thing for an app to use to e.g. disable networking from a sub-process that doesn’t need it, etc.

Of course the challenge is that not every app is easy to run in a container or virtual machine. Some workloads are most convenient with that “ambient access” to all of your data (like an IDE or just a file browser).

But giving that ambient access by default to agentic AI is a terrible idea. So don’t do it: use (OCI) containers and allowlist in what you need.

(There’s other things nono is doing here that I find dubious/duplicative; for example I don’t see the need for a new filesystem snapshotting system when we have both git and OCI)

But I’m not specifially trying to pick on nono – just in the last two weeks I had to point out similar problems in two different projects I saw go by also pitched for AI security. One used bubblewrap, but with insufficient sandboxing, and the other was also trying to use Landlock.

On the other hand, I do think the credential problem (that nono and others are trying to address in differnet ways) is somewhat specific to agentic AI, and likely does need new tooling. When deploying a typical containerized app usually one just provisions a few relatively static credentials. In contrast, developer/user agentic AI is often a lot more freeform and dynamic, and while it’s hard to get most apps to leak credentials without completely compromising it, it’s much easier with agentic AI and prompt injection. I have thoughts on credentials, and absolutely more work here is needed.

It’s great that people want to work on FOSS security, and AI could certainly use more people thinking about security. But I don’t think we need “next generation” security here: we should build on top of the “previous generation”. I actually use plain separate Unix users for isolation for some things, which works quite well! Running OpenShell in a secondary user account where one only logs into a select few things (i.e. not your email and online banking) is much more reasonable, although clearly a lot of care is still needed. Landlock is a fine technology but is just not there as a replacement for other sandboxing techniques. So just use containers and virtual machines because these are proven technologies. And if you take one message away from this: absolutely don’t wire up an LLM via OpenShell or a similar tool to your complete digital life with no sandboxing.

When you’re looking at source code it can be helpful to have some evidence indicating who wrote it. Author tags give a surface level indication, but it turns out you can just lie and if someone isn’t paying attention when merging stuff there’s certainly a risk that a commit could be merged with an author field that doesn’t represent reality. Account compromise can make this even worse - a PR being opened by a compromised user is going to be hard to distinguish from the authentic user. In a world where supply chain security is an increasing concern, it’s easy to understand why people would want more evidence that code was actually written by the person it’s attributed to.

git has support for cryptographically signing commits and tags. Because git is about choice even if Linux isn’t, you can do this signing with OpenPGP keys, X.509 certificates, or SSH keys. You’re probably going to be unsurprised about my feelings around OpenPGP and the web of trust, and X.509 certificates are an absolute nightmare. That leaves SSH keys, but bare cryptographic keys aren’t terribly helpful in isolation - you need some way to make a determination about which keys you trust. If you’re using someting like GitHub you can extract that information from the set of keys associated with a user account1, but that means that a compromised GitHub account is now also a way to alter the set of trusted keys and also when was the last time you audited your keys and how certain are you that every trusted key there is still 100% under your control? Surely there’s a better way.

SSH Certificates

And, thankfully, there is. OpenSSH supports certificates, an SSH public key that’s been signed by some trusted party and so now you can assert that it’s trustworthy in some form. SSH Certificates also contain metadata in the form of Principals, a list of identities that the trusted party included in the certificate. These might simply be usernames, but they might also provide information about group membership. There’s also, unsurprisingly, native support in SSH for forwarding them (using the agent forwarding protocol), so you can keep your keys on your local system, ssh into your actual dev system, and have access to them without any additional complexity.

And, wonderfully, you can use them in git! Let’s find out how.

Local config

There’s two main parameters you need to set. First,

1 git config set gpg.format ssh

because unfortunately for historical reasons all the git signing config is under the gpg namespace even if you’re not using OpenPGP. Yes, this makes me sad. But you’re also going to need something else. Either user.signingkey needs to be set to the path of your certificate, or you need to set gpg.ssh.defaultKeyCommand to a command that will talk to an SSH agent and find the certificate for you (this can be helpful if it’s stored on a smartcard or something rather than on disk). Thankfully for you, I’ve written one. It will talk to an SSH agent (either whatever’s pointed at by the SSH_AUTH_SOCK environment variable or with the -agent argument), find a certificate signed with the key provided with the -ca argument, and then pass that back to git. Now you can simply pass -S to git commit and various other commands, and you’ll have a signature.

Validating signatures

This is a bit more annoying. Using native git tooling ends up calling out to ssh-keygen2, which validates signatures against a file in a format that looks somewhat like authorized-keys. This lets you add something like:

1 * cert-authority ssh-rsa AAAA…

which will match all principals (the wildcard) and succeed if the signature is made with a certificate that’s signed by the key following cert-authority. I recommend you don’t read the code that does this in git because I made that mistake myself, but it does work. Unfortunately it doesn’t provide a lot of granularity around things like “Does the certificate need to be valid at this specific time” and “Should the user only be able to modify specific files” and that kind of thing, but also if you’re using GitHub or GitLab you wouldn’t need to do this at all because they’ll just do this magically and put a “verified” tag against anything with a valid signature, right?

Haha. No.

Unfortunately while both GitHub and GitLab support using SSH certificates for authentication (so a user can’t push to a repo unless they have a certificate signed by the configured CA), there’s currently no way to say “Trust all commits with an SSH certificate signed by this CA”. I am unclear on why. So, I wrote my own. It takes a range of commits, and verifies that each one is signed with either a certificate signed by the key in CA_PUB_KEY or (optionally) an OpenPGP key provided in ALLOWED_PGP_KEYS. Why OpenPGP? Because even if you sign all of your own commits with an SSH certificate, anyone using the API or web interface will end up with their commits signed by an OpenPGP key, and if you want to have those commits validate you’ll need to handle that.

In any case, this should be easy enough to integrate into whatever CI pipeline you have. This is currently very much a proof of concept and I wouldn’t recommend deploying it anywhere, but I am interested in merging support for additional policy around things like expiry dates or group membership.

Doing it in hardware

Of course, certificates don’t buy you any additional security if an attacker is able to steal your private key material - they can steal the certificate at the same time. This can be avoided on almost all modern hardware by storing the private key in a separate cryptographic coprocessor - a Trusted Platform Module on PCs, or the Secure Enclave on Macs. If you’re on a Mac then Secretive has been around for some time, but things are a little harder on Windows and Linux - there’s various things you can do with PKCS#11 but you’ll hate yourself even more than you’ll hate me for suggesting it in the first place, and there’s ssh-tpm-agent except it’s Linux only and quite tied to Linux.

So, obviously, I wrote my own. This makes use of the go-attestation library my team at Google wrote, and is able to generate TPM-backed keys and export them over the SSH agent protocol. It’s also able to proxy requests back to an existing agent, so you can just have it take care of your TPM-backed keys and continue using your existing agent for everything else. In theory it should also work on Windows3 but this is all in preparation for a talk I only found out I was giving about two weeks beforehand, so I haven’t actually had time to test anything other than that it builds.

And, delightfully, because the agent protocol doesn’t care about where the keys are actually stored, this still works just fine with forwarding - you can ssh into a remote system and sign something using a private key that’s stored in your local TPM or Secure Enclave. Remote use can be as transparent as local use.

Wait, attestation?

Ah yes you may be wondering why I’m using go-attestation and why the term “attestation” is in my agent’s name. It’s because when I’m generating the key I’m also generating all the artifacts required to prove that the key was generated on a particular TPM. I haven’t actually implemented the other end of that yet, but if implemented this would allow you to verify that a key was generated in hardware before you issue it with an SSH certificate - and in an age of agentic bots accidentally exfiltrating whatever they find on disk, that gives you a lot more confidence that a commit was signed on hardware you own.

Conclusion

Using SSH certificates for git commit signing is great - the tooling is a bit rough but otherwise they’re basically better than every other alternative, and also if you already have infrastructure for issuing SSH certificates then you can just reuse it4 and everyone wins.

1. Did you know you can just download people’s SSH pubkeys from github from https://github.com/<username>.keys? Now you do ↩︎

2. Yes it is somewhat confusing that the keygen command does things other than generate keys ↩︎

3. This is more difficult than it sounds ↩︎

4. And if you don’t, by implementing this you now have infrastructure for issuing SSH certificates and can use that for SSH authentication as well. ↩︎

It’s clear LLMs are one of the biggest changes in technology ever. The rate of progress is astounding: recently due to a configuration mistake I accidentally used Claude Sonnet 3.5 (released ~2 years ago) instead of Opus 4.6 for a task and looked at the output and thought “what is this garbage”?

But daily now: Opus 4.6 is able to generate reasonable PoC level Rust code for complex tasks for me. It’s not perfect – it’s a combination of exhausting and exhilarating to find the 10% absolutely bonkers/broken code that still makes it past subagents.

So yes I use LLMs every day, but I will be clear: if I could push a button to “un-invent” them I absolutely would because I think the long term issues in larger society (not being able to trust any media, and many of the things from Dario’s recent blog etc.) will outweigh the benefits.

But since we can’t un-invent them: here’s my opinion on how they should be used. As a baseline, I agree with a lot from this doc from Oxide about LLMs. What I want to talk about is especially around some of the norms/tools that I see as important for LLM use, following principles similar to those.

On framing: there’s “core” software vs “bespoke”. An entirely new capability of course is for e.g. a nontechnical restaurant owner to use an LLM to generate (“vibe code”) a website (excepting hopefully online orderings and payments!). I’m not overly concerned about this.

Whereas “core” software is what organizations/businesses provide/maintain for others. I work for a company (Red Hat) that produces a lot of this. I am sure no one would want to run for real an operating system, cluster filesystem, web browser, monitoring system etc. that was primarily “vibe coded”.

And while I respect people and groups that are trying to entirely ban LLM use, I don’t think that’s viable for at least my space.

Hence the subject of this blog is my perspective on how LLMs should be used for “core” software: not vibe coding, but using LLMs responsibly and intelligently – and always under human control and review.

Agents should amplify and be controlled by humans

I think most of the industry would agree we can’t give responsibility to LLMs. That means they must be overseen by humans. If they’re overseen by a human, then I think they should be amplifying what that human thinks/does as a baseline – intersected with the constraints of the task of course.

On “amplification”: Everyone using a LLM to generate content should inject their own system prompt (e.g. AGENTS.md) or equivalent. Here’s mine – notice I turn off all the emoji etc. and try hard to tune down bulleted lists because that’s not my style. This is a truly baseline thing to do.

Now most LLM generated content targeted for core software is still going to need review, but just ensuring that the baseline matches what the human does helps ensure alignment.

Pull request reviews

Let’s focus on a very classic problem: pull request reviews. Many projects have wired up a flow such that when a PR comes in, it gets reviewed by a model automatically. Many projects and tools pitch this. We use one on some of my projects.

But I want to get away from this because in my experience these reviews are a combination of:

• Extremely insightful and correct things (there’s some amazing fine-tuning and tool use that must have happened to find some issues pointed out by some of these)
• Annoying nitpicks that no one cares about (not handling spaces in a filename in a shell script used for tests)
• Broken stuff like getting confused by things that happened after its training cutoff (e.g. Gemini especially seems to get confused by referencing the current date, and also is unaware of newer Rust features, etc)

In practice, we just want the first of course.

How I think it should work:

• A pull request comes in
• It gets auto-assigned to a human on the team for review
• A human contributing to that project is running their own agents (wherever: could be local or in the cloud) using their own configuration (but of course still honoring the project’s default development setup and the project’s AGENTS.md etc)
• A new containerized/sandboxed agent may be spawned automatically, or perhaps the human needs to click a button to do so – or perhaps the human sees the PR come in and thinks “this one needs a deeper review, didn’t we hit a perf issue with the database before?” and adds that to a prompt for the agent.
• The agent prepares a draft review that only the human can see.
• The human reviews/edits the draft PR review, and has the opportunity to remove confabulations, add their own content etc. And to send the agent back to look more closely at some code (i.e. this part can be a loop)
• When the human is happy they click the “submit review” button.
• Goal: it is 100% clear what parts are LLM generated vs human generated for the reader.

I wrote this agent skill to try to make this work well, and if you search you can see it in action in a few places, though I haven’t truly tried to scale this up.

I think the above matches the vision of LLMs amplifying humans.

Code Generation

There’s no doubt that LLMs can be amazing code generators, and I use them every day for that. But for any “core” software I work on, I absolutely review all of the output – not just superficially, and changes to core algorithms very closely.

At least in my experience the reality is still there’s that percentage of the time when the agent decided to reimplement base64 encoding for no reason, or disable the tests claiming “the environment didn’t support it” etc.

And to me it’s still a baseline for “core” software to require another human review to merge (per above!) with their own customized LLM assisting them (ideally a different model, etc).

FOSS vs closed

Of course, my position here is biased a bit by working on FOSS – I still very much believe in that, and working in a FOSS context can be quite different than working in a “closed environment” where a company/organization may reasonably want to (and be able to) apply uniform rules across a codebase.

While for sure LLMs allow organizations to create their own Linux kernel filesystems or bespoke Kubernetes forks or virtual machine runtime or whatever – it’s not clear to me that it is a good idea for most to do so. I think shared (FOSS) infrastructure that is productized by various companies, provided as a service and maintained by human experts in that problem domain still makes sense. And how we develop that matters a lot.

In Chapter 1 I gave the context for this project and in Chapter 2 I showed the bare minimum: an ELF that Open Firmware loads, a firmware service call, and an infinite loop.

That was July 2024. Since then, the project has gone from that infinite loop to a bootloader that actually boots Linux kernels. This post covers the journey.

The filesystem problem

The Boot Loader Specification expects BLS snippets in a FAT filesystem under loaders/entries/. So the bootloader needs to parse partition tables, mount FAT, traverse directories, and read files. All #![no_std], all big-endian PowerPC.

I tried writing my own minimal FAT32 implementation, then integrating simple-fatfs and fatfs. None worked well in a freestanding big-endian environment.

Hadris

The breakthrough was hadris, a no_std Rust crate supporting FAT12/16/32 and ISO9660. It needed some work to get going on PowerPC though. I submitted fixes upstream for:

• thiserror pulling in std: default features were not disabled, preventing no_std builds.
• Endianness bug: the FAT table code read cluster entries as native-endian u32. On x86 that’s invisible; on big-endian PowerPC it produced garbage cluster chains.
• Performance: every cluster lookup hit the firmware’s block I/O separately. I implemented a 4MiB readahead cache for the FAT table, made the window size parametric at build time, and improved read_to_vec() to coalesce contiguous fragments into a single I/O. This made kernel loading practical.

All patches were merged upstream.

Disk I/O

Hadris expects Read + Seek traits. I wrote a PROMDisk adapter that forwards to OF’s read and seek client calls, and a Partition wrapper that restricts I/O to a byte range. The filesystem code has no idea it’s talking to Open Firmware.

Partition tables: GPT, MBR, and CHRP

PowerVM with modern disks uses GPT (via the gpt-parser crate): a PReP partition for the bootloader and an ESP for kernels and BLS entries.

Installation media uses MBR. I wrote a small mbr-parser subcrate using explicit-endian types so little-endian LBA fields decode correctly on big-endian hosts. It recognizes FAT32, FAT16, EFI ESP, and CHRP (type 0x96) partitions.

The CHRP type is what CD/DVD boot uses on PowerPC. For ISO9660 I integrated hadris-iso with the same Read + Seek pattern.

Boot strategy? Try GPT first, fall back to MBR, then try raw ISO9660 on the whole device (CD-ROM). This covers disk, USB, and optical media.

The firmware allocator wall

This cost me a lot of time.

Open Firmware provides claim and release for memory allocation. My initial approach was to implement Rust’s GlobalAlloc by calling claim for every allocation. This worked fine until I started doing real work: parsing partitions, mounting filesystems, building vectors, sorting strings. The allocation count went through the roof and the firmware started crashing.

It turns out SLOF has a limited number of tracked allocations. Once you exhaust that internal table, claim either fails or silently corrupts state. There is no documented limit; you discover it when things break.

The fix was to claim a single large region at startup (1/4 of physical RAM, clamped to 16-512 MB) and implement a free-list allocator on top of it with block splitting and coalescing. Getting this right was painful: the allocator handles arbitrary alignment, coalesces adjacent free blocks, and does all this without itself allocating. Early versions had coalescing bugs that caused crashes which were extremely hard to debug – no debugger, no backtrace, just writing strings to the OF console on a 32-bit big-endian target.

And the kernel boots!

March 7, 2026. The commit message says it all: “And the kernel boots!”

The sequence:

1. BLS discovery: walk loaders/entries/*.conf, parse into BLSEntry structs, filter by architecture (ppc64le), sort by version using rpmvercmp.

2. ELF loading: parse the kernel ELF, iterate PT_LOAD segments, claim a contiguous region, copy segments to their virtual address offsets, zero BSS.

3. Initrd: claim memory, load the initramfs.

4. Bootargs: set /chosen/bootargs via setprop.

5. Jump: inline assembly trampoline – r3=initrd address, r4=initrd size, r5=OF client interface, branch to kernel:

core::arch::asm!( "mr 7, 3", // save of_client "mr 0, 4", // r0 = kernel_entry "mr 3, 5", // r3 = initrd_addr "mr 4, 6", // r4 = initrd_size "mr 5, 7", // r5 = of_client "mtctr 0", "bctr", in("r3") of_client, in("r4") kernel_entry, in("r5") initrd_addr as usize, in("r6") initrd_size as usize, options(nostack, noreturn) )

One gotcha: do NOT close stdout/stdin before jumping. On some firmware, closing them corrupts /chosen and the kernel hits a machine check. We also skip calling exit or release – the kernel gets its memory map from the device tree and avoids claimed regions naturally.

The boot menu

I implemented a GRUB-style interactive menu:

• Countdown: boots the default after 5 seconds unless interrupted.
• Arrow/PgUp/PgDn/Home/End navigation.
• ESC: type an entry number directly.
• e: edit the kernel command line with cursor navigation and word jumping (Ctrl+arrows).

This runs on the OF console with ANSI escape sequences. Terminal size comes from OF’s Forth interpret service (#columns / #lines), with serial forced to 80×24 because SLOF reports nonsensical values.

Secure boot (initial, untested)

IBM POWER has its own secure boot: the ibm,secure-boot device tree property (0=disabled, 1=audit, 2=enforce, 3=enforce+OS). The Linux kernel uses an appended signature format – PKCS#7 signed data appended to the kernel file, same format GRUB2 uses on IEEE 1275.

I wrote an appended-sig crate that parses the appended signature layout, extracts an RSA key from a DER X.509 certificate (compiled in via include_bytes!), and verifies the signature (SHA-256/SHA-512) using the RustCrypto crates, all no_std.

The unit tests pass, including an end-to-end sign-and-verify test. But I have not tested this on real firmware yet. It needs a PowerVM LPAR with secure boot enforced and properly signed kernels, which QEMU/SLOF cannot emulate. High on my list.

The ieee1275-rs crate

The crate has grown well beyond Chapter 2. It now provides: claim/release, the custom heap allocator, device tree access (finddevice, getprop, instance-to-package), block I/O, console I/O with read_stdin, a Forth interpret interface, milliseconds for timing, and a GlobalAlloc implementation so Vec and String just work.

Published on crates.io at github.com/rust-osdev/ieee1275-rs.

What’s next

I would like to test the Secure Boot feature on an end to end setup but I have not gotten around to request access to a PowerVM PAR. Beyond that I want to refine the menu. Another idea would be to perhaps support the equivalent of the Unified Kernel Image using ELF. Who knows, if anybody finds this interesting let me know!

The source is at the powerpc-bootloader repository. Contributions welcome, especially from anyone with POWER hardware access.

Last week, Igalia finally announced Moonforge, a project we’ve been working on for basically all of 2025. It’s been quite the rollercoaster, and the announcement hit various news outlets, so I guess now is as good a time as any to talk a bit about what Moonforge is, its goal, and its constraints.

Of course, as soon as somebody announces a new Linux-based OS, folks immediately think it’s a new general purpose Linux distribution, as that’s the square shaped hole where everything OS-related ends up. So, first things first, let’s get a couple of things out of the way about Moonforge:

• Moonforge is not a general purpose Linux distribution
• Moonforge is not an embedded Linux distribution

What is Moonforge

Moonforge is a set of feature-based, well-maintained layers for Yocto, that allows you to assemble your own OS for embedded devices, or single-application environments, with specific emphasys on immutable, read-only root file system OS images that are easy to deploy and update, through tight integration with CI/CD pipelines.

Why?

Creating a whole new OS image out of whole cloth is not as hard as it used to be; on the desktop (and devices where you control the hardware), you can reasonably get away with using existing Linux distributions, filing off the serial numbers, and removing any extant packaging mechanism; or you can rely on the containerised tech stack, and boot into it.

When it comes to embedded platforms, on the other hand, you’re still very much working on bespoke, artisanal, locally sourced, organic operating systems. A good number of device manufacturers coalesced their BSPs around the Yocto Project and OpenEmbedded, which simplifies adaptations, but you’re still supposed to build the thing mostly as a one off.

While Yocto has improved leaps and bounds over the past 15 years, putting together an OS image, especially when it comes to bundling features while keeping the overall size of the base image down, is still an exercise in artisanal knowledge.

A little detour: Poky

Twenty years ago, I moved to London to work for this little consultancy called OpenedHand. One of the projects that OpenedHand was working on was taking OpenEmbedded and providing a good set of defaults and layers, in order to create a “reference distribution” that would help people getting started with their own project. That reference was called Poky.

We had a beaver mascot before it was cool

These days, Poky exists as part of the Yocto Project, and it’s still the reference distribution for it, but since it’s part of Yocto, it has to abide to the basic constraint of the project: you still need to set up your OS using shell scripts and copy-pasting layers and recipes inside your own repository. The Yocto project is working on a setup tool to simplify those steps, but there are alternatives…

Another little detour: Kas

One alternative is kas, a tool that allows you to generate the local.conf configuration file used by bitbake through various YAML fragments exported by each layer you’re interested in, as well as additional fragments that can be used to set up customised environments.

Another feature of kas is that it can spin up the build environment inside a container, which simplifies enourmously its set up time. It avoids unadvertedly contaminating the build, and it makes it very easy to run the build on CI/CD pipelines that already rely on containers.

What Moonforge provides

Moonforge lets you create a new OS in minutes, selecting a series of features you care about from various available layers.

Each layer provides a single feature, like:

• support for a specific architecture or device (QEMU x86_64, RaspberryPi)
• containerisation (through Docker or Podman)
• A/B updates (through RAUC, systemd-sysupdate, and more)
• graphical session, using Weston
• a WPE environment

Every layer comes with its own kas fragment, which describes what the layer needs to add to the project configuration in order to function.

Since every layer is isolated, we can reason about their dependencies and interactions, and we can combine them into a final, custom product.

Through various tools, including kas, we can set up a Moonforge project that generates and validates OS images as the result of a CI/CD pipeline on platforms like GitLab, GitHub, and BitBucket; OS updates are also generated as part of that pipeline, just as comprehensive CVE reports and Software Bill of Materials (SBOM) through custom Yocto recipes.

More importantly, Moonforge can act both as a reference when it comes to hardware enablement and support for BSPs; and as a reference when building applications that need to interact with specific features coming from a board.

While this is the beginning of the project, it’s already fairly usable; we are planning a lot more in this space, so keep an eye out on the repository.

Trying Moonforge out

If you want to check out Moonforge, I will point you in the direction of its tutorials, as well as the meta-derivative repository, which should give you a good overview on how Moonforge works, and how you can use it.

Last week was the end of Malika' internship within Papers about signatures that I had the pleasure to mentor. After a post about the first phase of Outreachy, here is the sequel of the story.

Nowadays, people expect to be able to fill and sign PDF documents. We previously worked on features to insert text into documents and signatures needed to be improved.

There is actually some ambiguity when speaking about signatures in PDFs: there are cryptographic signatures that guarantee that a certificate owner approved a document (now denoted by "digital" signatures) and there are also signatures that are just drawings on the document. These latter ones of course do not guarantee any authenticity but are more or less accepted in various situations, depending on the country. Moreover, getting a proper certificate to digitally sign documents may be complicated or costly (with the notable exception of a few countries providing them to their residents such as Spain).

Papers lacked any support for this second category (that I will call "visual" signatures from now on). On the other hand, digital signing was implemented a few releases ago, but it heavily relies on Firefox certificate database 1 and in particular there is no way to manage personal certificates within Papers.

During her three months internship, Malika implemented a new visual signatures management dialog and the corresponding UI to insert them, including nice details such as image processing to import signature pictures properly. She also contributed to the poppler PDF rendering library to compress signature data.

Then she looked into digital signatures and improved the insertion dialog, letting users choose visual signatures for them as well. If all goes well, all of this should be merged before Papers 51!

Malika also implemented a prototype that allows users to import certificates and also deal with multiple NSS databases. While this needs more testing and code review2, it should significantly simplify digital signing.

I would like to thank everyone who made this internship possible, and especially everyone who took the time to do calls and advise us during the internship. And of course, thanks to Malika for all the work she put into her internship!

1

or on NSS command line tools.

2

we don't have enough NSS experts, so help is very welcomed.

Another slow cycle, same as last time. Still, a few new things to showcase.

Sidebars

The most visible addition is the new sidebar widget. This is a bit confusing, because we already had widgets for creating windows with sidebars - AdwNavigationSplitView and AdwOverlaySplitView, but nothing to actually put into the sidebar pane. The usual recommendation is to build your own sidebar using GtkListBox or GtkListView, combined with the .navigation-sidebar style class.

This isn't too difficult, but the result is zero consistency between different apps, not unlike what we had with GtkNotebook-based tabs in the past:

It's even worse on mobile. In the best scenario it will just be a strangely styled flat list. Sometimes it will also have selection, and depending on how it's implemented it may be impossible to activate the selected row, like in libadwaita demo.

So we have a pre-built one now. It doesn't aim to support every single use case (sidebars can get very complex, see e.g. GNOME Builder), but just to be good enough for the basic situations.

How basic is basic? Well, it has selection, sections (with or without titles), tooltips, context menus, a drop target, suffix widgets at the end of each item's row, auto-activation when hovered during drag-n-drop.

A more advanced feature is built-in search filter - via providing a GtkFilter and a placeholder page.

And that's about it. There will likely be more features in future, like collapsible sections and drag source on items, rather than just a drop target, but this should already be enough for quite a lot of apps. Not everything, but that's not the goal here.

Internally, it's using GtkListBox. This means that it doesn't scale to thousands of items the way GtkListView would, but we can have much tighter API and mobile integration.

Now, let's talk about mobile. Ideally sidebars on mobile wouldn't really be sidebars at all. This pattern inherently requires a second pane, and falls apart otherwise. AdwNavigationSplitView already presents the sidebar pane as a regular page, so let's go further and turn sidebars into boxed lists. We're already using GtkListBox, after all.

So - AdwSidebar has the mode property. When set to ADW_SIDEBAR_MODE_PAGE, it becomes a page of boxed lists - indistinguishable from any others. It hides item selection, but it's still tracked internally. It can still be changed programmatically, and changes when an item is activated. Once the sidebar mode is set back to ADW_SIDEBAR_MODE_SIDEBAR, it will reappear.

Internally it's nothing special, as it just presents the same data using different widgets.

The adaptive layouts page has a detailed example for how to create UIs like this, as well as the newly added section about overlay sidebars that don't change as drastically.

View switcher sidebar

Once we have a sidebar, a rather obvious thing to do is to provide a GtkStackSidebar replacement. So AdwViewSwitcherSidebar is exactly that.

It works with AdwViewStack rather than GtkStack, and has all the same features as existing view switcher, as well as an extra one - sections.

To support that, AdwViewStackPage has new API for defining sections - the :starts-section and :section-title properties, while the AdwViewStack:pages) model is now a section model.

Like regular sidebars, it supports the boxed list mode and search filtering.

Unlike other view switchers or GtkStackSidebar, it also exposes AdwSidebar's item activation signal. This is required to make it work on mobile.

Demo improvements

The lack of sidebar was the main blocker for improving libadwaita demo in the past. Now that it's solved, the demo is at last, fully adaptive. The sidebar has been reorganized into sections, and has icons and search now.

This also unblocks other potential improvements, such as having a more scalable preferences dialog.

Reduced motion

While there isn't any new API, most widgets with animations have been updated to respect the new reduced motion preference - mostly by replacing sliding/scaling animations with crossfades, or otherwise toning down animations when it's impossible:

• AdwDialog open/close transitions are crossfades except for the swipe-to-close gesture
• AdwBottomSheet transition is a crossfade when there's no bottom bar, and a slide without overshooting if there is
• AdwNavigationView transition is a crossfade except when using the swipe gestures
• AdwTabOverview transition is a crossfade

AdwOverlaySplitView is unaffected for now. Same for toasts, those are likely small enough to not cause motion sickness. If it turns out to be a problem, it can be changed later.

I also didn't update any of the deprecated widgets, like AdwLeaflet. Applications still using those should switch to the modern alternatives.

The prefers-reduced-motion media feature is available for use from app CSS as well, following the GTK addition.

Other changes

• AdwAboutDialog rows that contain links have a context menu now. Link rows may become a public widget in future if there's interest.

• GTK_DEBUG=builder diagnostics are now supported for all libadwaita widgets. This can be used to find places where <child> tags are used in UI when equivalent properties exist.

• Following GTK, all GListModel implementations now come with :item-type and :n-item properties, to make it easier to use them from expressions.

• The AdwTabView:pages model implements sections now: one for pinned pages and one for everything else.

• AdwToggle has a new :description property that can be used to set accessible description for individual toggles separately from tooltips.

• Adrien Plazas improved accessibility in a bunch of widgets. The majority of this work has been backported to 1.8.x as well. For example, AdwViewSwitcher and AdwInlineViewSwither now read out number badges and needs attention status.

• AdwNoneAnimationTarget now exists for situations where animations are used as frame clock-based timers, as an alternative to using AdwCallbackAnimationTarget with empty callback.

• AdwPreferencesPage will refuse to add children of types other than AdwPreferencesGroup, instead of overlaying them over the page and then leaking them after the page is destroyed. This change was backported to 1.8.2 and subsequently reverted in 1.8.3 as it turned out multiple apps were relying on the broken behavior.

• Maximiliano made non-nullable string setter functions automatically replace NULL parameters with empty strings, since allowing NULL breaks Rust bindings, while rejecting them means apps using expressions get unexpected criticals - for example, when accessing a non-nullable string property on an object, and that object itself is NULL.

• As mentioned in the 1.8 blog post, style-dark.css, style-hc.css and style-hc-dark.css resources are now deprecated and apps using them will get warnings on startup. Apps are encouraged to switch to a single style.css and conditionally load styles using media queries instead.

• While not a user-visible change (hopefully!), the internal stylesheet has been refactored to use prefers-contrast media queries for high contrast styles instead of 2 conditionally loaded variants - further reducing the need on SCSS, even if not entirely replacing it just yet. (the main blocker is @extend, as well nesting and a few mixins, such as focus ring)

Future

A big change in works is a revamp of icon API. GTK has a new icon format that supports stateful icons with animated transitions, variable stroke weight, and many other capabilities. Currently, libadwaita doesn't make use of this, but it will in future.

In fact, a few smaller changes are already in 1.9: all of the internal icons in libadwaita itself, as well as in the demo and docs, have been updated to use the new format.

Thanks to the GNOME Foundation for their support and thanks to all the contributors who made this release possible.

Because 2026 is such an interesting period of time to live in, I feel I should explicitly say that libadwaita does not contain any AI slop, nor does allow such contributions, nor do I have any plans to change that. Same goes for all of my other projects, including this website.