Planet GNOME

JJ-FZF is a TUI (Terminal-based User Interface) for Jujutsu, built on top of fzf. It centers around the jj log view, providing key bindings for common operations on JJ/Git repositories. About six months ago, I revisited JJ, drawn in by its promise of Automatic rebase and conflict resolution. I have…

It’s been more than 2 years since the last time I wrote something here, and in that time a lot of things happened. Among those, one of the main highlights was me moving back to Igalia‘s WebKit team, but this time I moved as part of Igalia’s support infrastructure to help with other types of tasks such as general coordination, team facilitation and project management, among other things.

On top of those things, I’ve been also presenting our work around WebKit in different venues, such as in the Embedded Open Source Summit or in the Embedded Recipes conference, for instance. Of course, that included presenting our work in the WebKit community as part of the WebKit Contributors Meeting, a small and technically focused event that happens every year, normally around the Bay Area (California). That’s often a pretty dense presentation where, over the course of 30-40 minutes, we go through all the main areas that we at Igalia contribute to in WebKit, trying to summarize our main contributions in the previous 12 months. This includes work not just from the WebKit team, but also from other ones such as our Web Platform, Compilers or Multimedia teams.

So far I did that a couple of times only, both last year on October 24rth as well as this year, just a couple of weeks ago in the latest instance of the WebKit Contributors meeting. I believe the session was interesting and informative, but unfortunately it does not get recorded so this time I thought I’d write a blog post to make it more widely accessible to people not attending that event.

This is a long read, so maybe grab a cup of your favorite beverage first…

Igalia and WebKit

So first of all, what is the relationship between Igalia and the WebKit project?

In a nutshell, we are the lead developers and the maintainers of the two Linux-based WebKit ports, known as WebKitGTK and WPE. These ports share a common baseline (e.g. GLib, GStreamer, libsoup) and also some goals (e.g. performance, security), but other than that their purpose is different, with WebKitGTK being aimed at the Linux desktop, while WPE is mainly focused on embedded devices.

This means that, while WebKitGTK is the go-to solution to embed Web content in GTK applications (e.g. GNOME Web/Epiphany, Evolution), and therefore integrates well with that graphical toolkit, WPE does not even provide a graphical toolkit since its main goal is to be able to run well on embedded devices that often don’t even have a lot of memory or processing power, or not even the usual mechanisms for I/O that we are used to in desktop computers. This is why WPE’s architecture is designed with flexibility in mind with a backends-based architecture, why it aims for using as few resources as possible, and why it tries to depend on as few libraries as possible, so you can integrate it virtually in any kind of embedded Linux platform.

Besides that port-specific work, which is what our WebKit and Multimedia teams focus a lot of their effort on, we also contribute at a different level in the port-agnostic parts of WebKit, mostly around the area of Web standards (e.g. contributing to Web specifications and to implement them) and the Javascript engine. This work is carried out by our Web Platform and Compilers team, which tirelessly contribute to the different parts of WebCore and JavaScriptCore that affect not just the WebKitGTK and WPE ports, but also the rest of them to a bigger or smaller degree.

Last but not least, we also devote a considerable amount of our time to other topics such as accessibility, performance, bug fixing, QA... and also to make sure WebKit works well on 32-bit devices, which is an important thing for a lot of WPE users out there.

Who are our users?

At Igalia we distinguish 4 main types of users of the WebKitGTK and WPE ports of WebKit:

Port users: this category would include anyone that writes a product directly against the port’s API, that is, apps such as a desktop Web browser or embedded systems that rely on a fullscreen Web view to render its Web-based content (e.g. digital signage systems).

Platform providers: in this category we would have developers that build frameworks with one of the Linux ports at its core, so that people relying on such frameworks can leverage the power of the Web without having to directly interface with the port’s API. RDK could be a good example of this use case, with WPE at the core of the so-called Thunder plugin (previously known as WPEFramework).

Web developers: of course, Web developers willing to develop and test their applications against our ports need to be considered here too, as they come with a different set of needs that need to be fulfilled, beyond rendering their Web content (e.g. using the Web Inspector).

End users: And finally, the end user is the last piece of the puzzle we need to pay attention to, as that’s what makes all this effort a task worth undertaking, even if most of them most likely don’t need what WebKit is, which is perfectly fine :-)

We like to make this distinction of 4 possible types of users explicit because we think it’s important to understand the complexity of the amount of use cases and the diversity of potential users and customers we need to provide service for, which is behind our decisions and the way we prioritize our work.

Strategic goals

Our main goal is that our product, the WebKit web engine, is useful for more and more people in different situations. Because of this, it is important that the platform is homogeneous and that it can be used reliably with all the engines available nowadays, and this is why compatibility and interoperability is a must, and why we work with the the standards bodies to help with the design and implementation of several Web specifications.

With WPE, it is very important to be able to run the engine in small embedded devices, and that requires good performance and being efficient in multiple hardware architectures, as well as great flexibility for specific hardware, which is why we provided WPE with a backend-based architecture, and reduced dependencies to a minimum.

Then, it is also important that the QA Infrastructure is good enough to keep the releases working and with good quality, which is why I regularly maintain, evolve and keep an eye on the EWS and post-commit bots that keep WebKitGTK and WPE building, running and passing the tens of thousands of tests that we need to check continuously, to ensure we don’t regress (or that we catch issues soon enough, when there’s a problem). Then of course it’s also important to keep doing security releases, making sure that we release stable versions with fixes to the different CVEs reported as soon as possible.

Finally, we also make sure that we keep evolving our tooling as much as possible (see for instance the release of the new SDK earlier this year), as well as improving the documentation for both ports.

Last, all this effort would not be possible if not because we also consider a goal of us to maintain an efficient collaboration with the rest of the WebKit community in different ways, from making sure we re-use and contribute to other ports as much code as possible, to making sure we communicate well in all the forums available (e.g. Slack, mailing list, annual meeting).

Contributions to WebKit in numbers

Well, first of all the usual disclaimer: number of commits is for sure not the best possible metric,  and therefore should be taken with a grain of salt. However, the point here is not to focus too much on the actual numbers but on the more general conclusions that can be extracted from them, and from that point of view I believe it’s interesting to take a look at this data at least once a year.

With that out of the way, it’s interesting to confirm that once again we are still the 2nd biggest contributor to WebKit after Apple, with ~13% of the commits landed in this past 12-month period. More specifically, we landed 2027 patches out of the 15617 ones that took place during the past year, only surpassed by Apple and their 12456 commits. The remaining 1134 patches were landed mostly by Sony, followed by RedHat and several other contributors.

Now, if we remove Apple from the picture, we can observe how this year our contributions represented ~64% of all the non-Apple commits, a figure that grew about ~11% compared to the past year. This confirms once again our commitment to WebKit, a project we started contributing about 14 years ago already, and where we have been systematically being the 2nd top contributor for a while now.

Main areas of work

The 10 main areas we have contributed to in WebKit in the past 12 months are the following ones:

• Web platform
• Graphics
• Multimedia
• JavaScriptCore
• New WPE API
• WebKit on Android
• Quality assurance
• Security
• Tooling
• Documentation

In the next sections I’ll talk a bit about what we’ve done and what we’re planning to do next for each of them.

Web Platform

content-visibility:auto

This feature allows skipping painting and rendering of off-screen sections, particularly useful to avoid the browser spending time rendering parts in large pages, as content outside of the view doesn’t get rendered until it gets visible.

We completed the implementation and it’s now enabled by default.

Navigation API

This is a new API to manage browser navigation actions and examine history, which we started working on in the past cycle. There’s been a lot of work happening here and, while it’s not finished yet, the current plan is that Apple will continue working on that in the next months.

hasUAVisualTransition

This is an attribute of the NavigateEvent interface, which is meant to be True if the User Agent has performed a visual transition before a navigation event. It was something that we have also finished implementing and is now also enabled by default.

Secure Curves in the Web Cryptography API

In this case, we worked on fixing several Web Interop related issues, as well as on increasing test coverage within the Web Platform Tests (WPT) test suites.

On top of that we also moved the X25519 feature to the “prepare to ship” stage.

Trusted Types

This work is related to reducing DOM-based XSS attacks. Here we finished the implementation and this is now pending to be enabled by default.

MathML

We continued working on the MathML specification by working on the support for padding, border and margin, as well as by increasing the WPT score by ~5%.

The plan for next year is to continue working on core features and improve the interaction with CSS.

Cross-root ARIA

Web components have accessibility-related issues with native Shadow DOM as you cannot reference elements with ARIA attributes across boundaries. We haven’t worked on this in this period, but the plan is to work in the next months on implementing the Reference Target proposal to solve those issues.

Canvas Formatted Text

Canvas has not a solution to add formatted and multi-line text, so we would like to also work on exploring and prototyping the Canvas Place Element proposal in WebKit, which allows better text in canvas and more extended features.

Graphics

Completed migration from Cairo to Skia for the Linux ports

If you have followed the latest developments, you probably already know that the Linux WebKit ports (i.e. WebKitGTK and WPE) have moved from Cairo to Skia for their 2D rendering library, which was a pretty big and important decision taken after a long time trying different approaches and experiments (including developing our own HW-accelerated 2D rendering library!), as well as running several tests and measuring results in different benchmarks.

The results in the end were pretty overwhelming and we decided to give Skia a go, and we are happy to say that, as of today, the migration has been completed: we covered all the use cases in Cairo, achieving feature parity, and we are now working on implementing new features and improvements built on top of Skia (e.g. GPU-based 2D rendering).

On top of that, Skia is now the default backend for WebKitGTK and WPE since 2.46.0, released on September 17th, so if you’re building a recent version of those ports you’ll be already using Skia as their 2D rendering backend. Note that Skia is using its GPU-based backend only on desktop environments, on embedded devices the situation is trickier and for now the default is the CPU-based Skia backend, but we are actively working to narrow the gap and to enable GPU-based rendering also on embedded.

Architecture changes with buffer sharing APIs (DMABuf)

We did a lot of work here, such as a big refactoring of the fencing system to control the access to the buffers, or the continued work towards integrating with Apple’s DisplayLink infrastructure.

On top of that, we also enabled more efficient composition using damaging information, so that we don’t need to pass that much information to the compositor, which would slow the CPU down.

Enablement of the GPUProcess

On this front, we enabled by default the compilation for WebGL rendering using the GPU process, and we are currently working in performance review and enabling it for other types of rendering.

New SVG engine (LBSE: Layer-Based SVG Engine)

If you are not familiar with this, here the idea is to make sure that we reuse the graphics pipeline used for HTML and CSS rendering, and use it also for SVG, instead of having its own pipeline. This means, among other things, that SVG layers will be supported as a 1st-class citizen in the engine, enabling HW-accelerated animations, as well as support for 3D transformations for individual SVG elements.

On this front, on this cycle we added support for the missing features in the LBSE, namely:

• Implemented support for gradients & patterns (applicable to both fill and stroke)
• Implemented support for clipping & masking (for all shapes/text)
• Implemented support for markers
• Helped review implementation of SVG filters (done by Apple)

Besides all this, we also improved the performance of the new layer-based engine by reducing repaints and re-layouts as much as possible (further optimizations still possible), narrowing the performance gap with the current engine for MotionMark. While we are still not at the same level of performance as the current SVG engine, we are confident that there are several key places where, with the right funding, we should be able to improve the performance to at least match the current engine, and therefore be able to push the new engine through the finish line.

General overhaul of the graphics pipeline, touching different areas (WIP):

On top of everything else commented above, we also worked on a general refactor and simplification of the graphics pipeline. For instance, we have been working on the removal of the Nicosia layer now that we are not planning to have multiple rendering implementations, among other things.

Multimedia

DMABuf-based sink for HW-accelerated video

We merged the DMABuf-based sink for HW-accelerated video in the GL-based GStreamer sink.

WebCodecs backend

We completed the implementation of  audio/video encoding and decoding, and this is now enabled by default in 2.46. As for the next steps, we plan to keep working on the integration of WebCodecs with WebGL and WebAudio.

GStreamer-based WebRTC backends

We continued working on GstWebRTC, bringing it to a point where it can be used in production in some specific use cases, and we will still be working on this in the next months.

Other

Besides the points above, we also added an optional text-to-speech backend based on libspiel to the development branch, and worked on general maintenance around the support for Media Source Extensions (MSE) and Encrypted Media Extensions (EME), which are crucial for the use case of WPE running in set-top-boxes, and is a permanent task we will continue to work on in the next months.

JavaScriptCore

ARMv7/32-bit support:

A lot of work happened around 32-bit support in JavaScriptCore, especially around WebAssembly (WASM): we ported the WASM BBQJIT and ported/enabled concurrent JIT support, and we also completed 80% of the implementation for the OMG optimization level of WASM, which we plan to finish in the next months. If you are unfamiliar with what the OMG and BBQ optimization tiers in WASM are, I’d recommend you to take a look at this article in webkit.org: “Assembling WebAssembly“.

We also contributed to the JIT-less WASM, which is very useful for embedded systems that can’t support JIT for security or memory related constraints, and also did some work on the In-Place Interpreter (IPInt), which is a new version of the WASM Low-level interpreter (LLInt) that uses less memory and executes WASM bytecode directly without translating it to LLInt bytecode  (and should therefore be faster to execute).

Last, we also contributed most of the implementation for the WASM GC, with the exception of some Kotlin tests.

As for the next few months, we plan to investigate and optimize heap/JIT memory usage in 32-bit, as well as to finish several other improvements on ARMv7 (e.g. IPInt).

New WPE API

The new WPE API is a new API that aims at making it easier to use WPE in embedded devices, by removing the hassle of having to handle several libraries in tandem (i.e. WPEWebKit, libWPE and WPEBackend-FDO, for instance), available from WPE’s releases page, and providing a more modern API in general, better aimed at the most common use cases of WPE.

A lot of effort happened this year along these lines, including the fact that we finally upstreamed and shipped its initial implementation with WPE 2.44, back in the first half of the year. Now, while we recommend users to give it a try and report feedback as much as possible, this new API is still not set in stone, with regular development still ongoing, so if you have the chance to try it out and share your experience, comments are welcome!

Besides shipping its initial implementation, we also added support for external platforms, so that other ones can be loaded beyond the Wayland, DRM and “headless” ones, which are the default platforms already included with WPE itself. This means for instance that a GTK4 platform, or another one for RDK could be easily used with WPE.

Then of course a lot of API additions were included in the new API in the latest months:

• Screens management API:  API to handle different screens, ask the display for the list of screens with their device scale factor, refresh rate, geometry…
• Top level management API: This API allows a greater degree of control, for instance by allowing more than one WebView for the same top level, as well as allowing to retrieve properties such as size, scale or state (i.e. full screen, maximized…).
• Maximized and minimized windows API: API to maximize/minimize a top level and monitor its state. mainly used by WebDriver.
• Preferred DMA-BUF formats API: enables asking the platform (compositor or DRM) for the list of preferred formats and their intended use (scanout/rendering).
• Input methods API: allows platforms to provide an implementation to handle input events (e.g. virtual keyboard, autocompletion, auto correction…).
• Gestures API: API to handle gestures (e.g. tap, drag).
• Buffer damaging: WebKit generates information about the areas of the buffer that actually changed and we pass that to DRM or the compositor to optimize painting.
• Pointer lock API: allows the WebView to lock the pointer so that the movement of the pointing device (e.g. mouse) can be used for a different purpose (e.g. first-person shooters).

Last, we also added support for testing automation, and we can support WebDriver now in the new API.

With all this done so far, the plan now is to complete the new WPE API, with a focus on the Settings API and accessibility support, write API tests and documentation, and then also add an external platform to support GTK4. This is done on a best-effort basis, so there’s no specific release date.

WebKit on Android

This year was also a good year for WebKit on Android, also known as WPE Android, as this is a project that sits on top of WPE and its public API (instead of developing a fully-fledged WebKit port).

In case you’re not familiar with this, the idea here is to provide a WebKit-based alternative to the Chromium-based Web view on Android devices, in a way that leverages HW acceleration when possible and that it integrates natively (and nicely) with the several Android subsystems, and of course with Android’s native mainloop. Note that this is an experimental project for now, so don’t expect production-ready quality quite yet, but hopefully something that can be used to start experimenting with selected use cases.

If you’re adventurous enough, you can already try the APKs yourself from the releases page in GitHub at https://github.com/Igalia/wpe-android/releases.

Anyway, as for the changes that happened in the past 12 months, here is a summary:

• Updated WPE Android to WPE 2.46 and NDK 27 LTS
• Added support for WebDriver and included WPT test suites
• Added support for instrumentation tests, and integrated with the GitHub CI
• Added support for the remote Web inspector, very useful for debugging
• Enabled the Skia backend, bringing HW-accelerated 2D rendering to WebKit on Android
• Implemented prompt delegates, allowing implementing things such as alert dialogs
• Implemented WPEView client interfaces, allowing responding to things such as HTTP errors
• Packaged a WPE-based Android WebView in its own library and published in Maven Central. This is a massive improvement as now apps can use WPE Android by simply referencing the library from the gradle files, no need to build everything on their own.
• Other changes: enabled HTTP/2 support (via the migration to libsoup3), added support for the device scale factor, improved the virtual on-screen keyboard, general bug fixing…

On top of that, we published 3 different blog posts covering different topics, from a general intro to a more deep dive explanation of the internals, and showing some demos. You can check them out in Jani’s blog at https://blogs.igalia.com/jani

As for the future, we’ll focus on stabilization and regular maintenance for now, and then we’d like to work towards achieving production-ready quality for specific cases if possible.

Quality Assurance

On the QA front, we had a busy year but in general we could highlight the following topics.

• Fixed a lot of API tests failures in the bots that were limiting our test coverage.
• Fixed lots of assertions-related crashes in the bots, which were slowing down the bots as well as causing other types of issues, such as bots exiting early due too many failures.
• Enabled assertions in the release bots, which will help prevent crashes in the future, as well as with making our debug bots healthier.
• Moved all the WebKitGTK and WPE bots to building now with Skia instead of Cairo. This means that all the bots running tests are now using Skia, and there’s only one bot still using Cairo to make sure that the compilation is not broken, but that bot does not run tests.
• Moved all the WebKitGTK bots to use GTK4 by default. As with the move to Skia, all the WebKit bots running tests now use GTK4 and the only one remaining building with GTK3 does not run tests, it only makes sure we don’t break the GTK3 compilation for now.
• Working on moving all the bots to use the new SDK. This is still work in progress and will likely be completed during 2025 as it’s needed to implement several changes in the infrastructure that will take some time.
• General gardening and bot maintenance

In the next months, our main focus would be a revamp of the QA infrastructure to make sure that we can get all the bots (including the debug ones) to a healthier state, finish the migration of all the bots to the new SDK and, ideally, be able to bring back the ready-to-use WPE images that we used to have available in wpewebkit.org.

Security

The current release cadence has been working well, so we continue issuing major releases every 6 months (March, September), and then minor and unstable development releases happening on-demand when needed.

As usual, we kept aligning releases for WebKitGTK and WPE, with both of them happening at the same time (see https://webkitgtk.org/releases and https://wpewebkit.org/release), and then also publishing WebKit Security Advisories (WSA) when necessary, both for WebKitGTK and for WPE.

Last, we also shortened the time before including security fixes in stable releases this year, and we have removed support for libsoup2 from WPE, as that library is no longer maintained.

Tooling & Documentation

On tooling, the main piece of news is that this year we released the initial version of the new SDK,  which is developed on top of OCI-based containers. This new SDK fixes the issues with the current existing approaches based on JHBuild and flatpak, where one of them was great for development but poor for testing and QA, and the other one was great for testing and QA, but not very convenient for development.

This new SDK is regularly maintained and currently runs on Ubuntu 24.04 LTS with GCC 14 & Clang 18. It has been made public on GitHub and announced to the public in May 2024 in Patrick’s blog, and is now the officially recommended way of building WebKitGTK and WPE.

As for documentation, we didn’t do as much as we would have liked here, but we still landed a few contributions in docs.webkit.org, mostly related to WebKitGTK (e.g. Releases and Versioning, Security Updates, Multimedia). We plan to do more on this regard in the next months, though, mostly by writing/publishing more documentation and perhaps also some tutorials.

Final thoughts

This has been a fairly long blog post but, as you can see, it’s been quite a year for WebKit here at Igalia, with many exciting changes happening at several fronts, and so there was quite a lot of stuff to comment on here. This said, you can always check the slides of the presentation in the WebKit Contributors Meeting here if you prefer a more concise version of the same content.

In any case, what’s clear it’s that the next months are probably going to be quite interesting as well with all the work that’s already going on in WebKit and its Linux ports, so it’s possible that in 12 months from now I might be writing an equally long essay. We’ll see.

Thanks for reading!

A couple years ago the Fedora council denied a request by Meta engineers to build the distribution with frame-pointers. Pretty immediately I pushed back by writing a number of articles to inform the council members why frame-pointers were necessary for a good profiling experience.

Profiling is used by developers, system administrators, and when we’re lucky by bug reporters!

Since then, many people have discussed other options. For example in the not too distant future we’ll probably see SFrame unwinding provide a reasonable way to unwind stacks w/o frame-pointers enabled and more importantly, without copying the contents of the stack.

Until then, it can be helpful to have a way to unwind stacks even without the presence of frame-pointers. This past week I implemented that for Sysprof based on a prototype put together by Serhei Makarov in the elfutils project called eu-stacktrace.

This prototype works by taking samples of the stack from perf (say 16KB-32KB worth) and resolving enough of the ELF data for DWARF/CFI (Call-frame-information)/etc to unwind the stacks in memory using a copy of the registers. From this you create a callchain (array of instruction pointers) which can be sent to Sysprof for recording.

I say “in memory” because the stack and register content doesn’t hit disk. It only lands inside the mmap()-based ring buffer used to communicate with Linux’s perf event subsystem. The (much smaller) array of instruction pointers eventually lands on disk if you’re not recording to a memfd.

I expanded upon this prototype with a new sysprof-live-unwinder process which does roughly the same thing as eu-stacktrace while fitting into the Sysprof infrastructure a bit more naturally. It consumes a perf data stream directly (eu-stacktrace consumed Sysprof-formatted data) and then provides that to Sysprof to help reduce overhead.

Additionally, eu-stacktrace only unwinds the user-space side of things. On x86_64, at least, you can convince perf to give you both callchains (PERF_SAMPLE_CALLCHAIN) as well as sample stack/registers (PERF_SAMPLE_STACK_USER|PERF_SAMPLE_REGS_USER). If you peek for the location of PERF_CONTEXT_USER to find the context switch, blending them is quite simple. So, naturally, Sysprof does that. The additional overhead for frame-pointer unwinding user-space is negligible when you don’t have frame-pointers to begin with.

I should start by saying that this still has considerable overhead compared to frame-pointers. Locally on my test machine (a Thinkpad X1 Carbon Gen 3 from around 2015, so not super new) that is about 10% of samples. I imagine I can shave a bit of that off by tracking the VMAs differently than libdwfl, so we’ll see.

Here is an example of it working on CentOS Stream 10 which does not have frame-pointers enabled. Additionally, this build is debuginfod-enabled so after recording it will automatically locate enough debug symbols to get appropriate function names for what was captured.

This definitely isn’t the long term answer to unwinding. But if you don’t have frame-pointers on your production operating system of choice, it might just get you by until SFrame comes around.

The code is at wip/chergert/translate but will likely get cleaned up and merged this next week.

Approximately four years ago, I published the first release of ASHPD, one of my first Rust libraries, with the simple goal of making it easy to use XDG portals from Rust.

Since then, the library has grown to support all available portals and even includes a demo application showcasing some of these features.

Let's look at an example: the org.freedesktop.portal.Account portal. From the client side, an API end-user can request user information with the following code:

use ashpd::desktop::account::UserInformation; async fn run() -> ashpd::Result<()> { let response = UserInformation::request() .reason("App would like to access user information") .send() .await? .response()?; println!("Name: {}", response.name()); println!("ID: {}", response.id()); Ok(()) }

This code calls the org.freedesktop.portal.Account.GetUserInformation D-Bus method, which xdg-desktop-portal will "redirect" to any portal frontend implementing the org.freedesktop.impl.portal.Account D-Bus interface.

So, how can you provide an implementation of org.freedesktop.impl.portal.Account in Rust? That's exactly what Maximiliano and I have been working on, building on the solid foundations we established earlier. I’m thrilled to announce that we finally shipped this functionality in the 0.10 release!

The first step is to implement the D-Bus interface, which we hide from the API’s end-user using traits.

use ashpd::{ async_trait, backend::{ account::{AccountImpl, UserInformationOptions}, request::RequestImpl, Result, }, desktop::account::UserInformation, AppID, WindowIdentifierType, }; pub struct Account; #[async_trait] impl RequestImpl for Account { async fn close(&self) { // Close the dialog } } #[async_trait] impl AccountImpl for Account { async fn get_user_information( &self, _app_id: Option<AppID>, _window_identifier: Option<WindowIdentifierType>, _options: UserInformationOptions, ) -> Result<UserInformation> { Ok(UserInformation::new( "user", "User", url::Url::parse("file://user/icon").unwrap(), )) } }

Pretty straightforward! With the D-Bus interface implemented using ASHPD wrapper types, the next step is to export it on the bus.

use futures_util::future::pending; async fn main() -> ashpd::Result<()> { ashpd::backend::Builder::new("org.freedesktop.impl.portal.desktop.mycustomportal")? .account(Account) .build() .await?; loop { pending::<()>().await; } }

And that’s it—you’ve implemented your first portal frontend!

Currently, the backend feature doesn't yet support session-based portals, but we hope to add that functionality in the near future.

With over 1 million downloads, ASHPD has come a long way, and it wouldn’t have been possible without the support and contributions from the community. A huge thank you to everyone who has helped make this library what it is today.

 The Bargain-Finder-inator 5000: One programmer's quest for a new flatOr how I managed to get a reasonably priced apartment offer despite estate agencies

I think every one of us had to go through the hell that's searching for a new place to live. The reasons may be of all kinds, starting with moving between jobs or random life events, ending with your landlord wanting to raise your rent for fixing his couch despite your 3 years of begging for him to do so. You can guess my reasoning from that totally not suspiciously specific example, one thing's for certain - many of us, not lucky enough to be on their own yet, have to go through that not very delightful experience.

One major problem when scraping those online market websites, is that you're not the only one desperately doing so. And if it was only for the fellow lost souls who are trying to make ends meet, oh no - many real estate agencies say hello there as well. So when a very good offer finally comes up, one that you've been dreaming your whole life kind of one, you grab that phone and call them not maybe, but may they please-oh-lord pick up. Despite you wasting no breath, chances are that when you enthusiastically call them (after correcting the typos in the phone number you made out of excitement), you're already too late. Even though you ended up manually checking the damn website every 20 minutes (yup, I set an alarm), and you called after only a quarter, you were still not fast enough and there are already four people in line before you. Which in case of a good offer means it's as good as doughnuts at work you heard they were giving out to buy your sympathy for the corporate - gone even faster than they have probably arrived. Yup, that's basically the housing market situation in Poland, yay \o/

But do not abandon all hope ye who enter here -  after having only a couple of mental break downs my friend sent me a link to a program on github, that was supposed to scrap our local market website and give instance notice about new offers. The web page did have a similar function, but it only worked in theory - the emails about the "latest" offers came only once a day, not to mention the fact that they were from the day before. Oh well, in that case saying goodbye to the 20 minute alarm sounded like a dream come true, so I tried to configure the program olx-scraper to my needs. However, it turned out to be pretty useless as well - it would repeatedly fetch a whole list of offers from only one page of search results, and compare its size between iterations. If the length of such list increased, it would theoretically mean that there are new offers, and the program would send a mail notification that contained the whole list. While this approach kinda worked for searches that returns only a few results, the whole idea fell apart when there were more than could fit in one page. In that case the number of offers would seem to remain constant, and new offers would be missed. Another room for improvement was in lack of ability to ignore certain kinds of offers, such as ads, and not so helpful emails, which could just give you what you're looking for - the newest offer, instead of the whole list.

Here comes the sun in the form of the Bargain-Finder-inator 5000 to the rescue! I quickly realized that a few patches was not enough to fix the old program for my (or frankly saying anyone's) use case and re-wrote the whole searching algorithm, eventually leading to a whole new program. The original name was "Wyszukiwator-Mieszkań 5000", inspired by Dr. Doofenschmirtz various schemes and inventions, and roughly translates to "Searcher-Of-Flats 5000". However, as the project grew beyond the real estate market, I needed a new name that would reflect that - it also needed to be slightly more accessible for foreigners than our oh how beautiful polish words. So I came up with the current one, with the best fitting abbreviation: bf5000. I think it's kind of neat :)

Totally accurate photograph of me giving birth to Bargain-Finder-inator 5000 circa 2024, colorized

What Bargain-Finder-inator 5000 dutifully does is monitor a link you serve to it, pointing to an online marketplace, be it for a real estate market or any other you can think of. The catch is that it needs to be supported, but writing a new backend shouldn't be too much of a hassle, and when it is you can simply copy paste the URL of your search with all the necessary filters specified, and  give it to bf5000. You also need to specify the delay between each check for new offers, which consists of fetching only the latest offer, and comparing it with the previous "latest". If they don't match, then we are in for some goodies - an email notification with the link to the latest offer will be sent, so you need to specify the email title, addresses and the whole provider too. For more information, check out the repository on gitlab.

So, don't wait no more for better days, and be part of the change now! We can take back what's rightfully ours from those money-hungry real estate agencies! When I say Bargain, you say Finder-inator 5000! You get the idea.

A few months ago this happened.

Which, for those of you not up to date on your 1960s British television, is to say that I've resigned. I'm currently enjoying the unemployed life style. No, that is not me being cheeky or ironic. I'm actually enjoying being able to focus on my own free time projects and sleeping late.

Since I'm not a millionaire at some point I'll probably have to get a job again. But not for at least six months. Maybe more, maybe less, we'll see what happens.

This should not affect Meson users in any significant way. I plan to spend some time to work on some fundamental issues in the code base to make things better all round. But the most important thing for now is to land the option refactor monster.

Recently, I got a Steam Deck OLED. Obviously, one of the main reasons for that is to run a certain yet to be announced here emulation app on it, so I installed Bazzite instead of SteamOS, cleaned up the preinstalled junk and got a clean desktop along with the Steam session/gaming mode.

For the most part, it just works (in desktop mode, at least), but there was one problematic area: input.

Gamepad input

Gamepads in general are difficult. While you can write generic evdev code dealing with, say, keyboard input and be reasonably sure it will work with at least the majority of keyboards, that’s not the case for gamepads. Buttons will use random input codes. Gamepads will assign different input types for the same control. (for example, D-pad can be presented as 4 buttons, 2 hat axes or 2 absolute axes). Linux kernel includes specialized hid drivers for some gamepads which will work reasonably well out of the box, but in general all bets are off.

Projects like SDL have gamepad mapping databases – normalizing input for all gamepads into a standardized list of inputs.

However, even that doesn’t guarantee they will work. Gamepads will pretend to be other gamepads (for example, it’s very common to emulate an Xbox gamepad) and will use incorrect mapping as a result. Some gamepads will even use identical IDs and provide physically different sets of buttons, meaning there’s no way to map both at the same time.

As such, apps have to expect that gamepad may or may not work correctly and user may or may not need to remap their gamepad.

Steam controllers

Both the standalone Steam Controller and Steam Deck’s internal gamepad pose a unique challenge: in addition to being gamepads with every problem mentioned above, they also emulate keyboard and pointer input. To make things more complicated, Steam has a built-in userspace HID driver for these controllers, with subtly different behavior between it and the Linux kernel driver. SteamOS and Bazzite both autostart Steam in background in desktop mode.

If one tries to use evdev in a generic way, same as for other gamepads, the results will not be pretty:

In desktop mode Steam emulates a virtual XInput (Xbox) gamepad. This gamepad works fine, except it lacks access to Steam and QAM buttons, as well as the 4 back buttons (L4, L5, R4, R5). This works perfectly fine for most games, but fails for emulators where in addition to the in-game controls you need a button to exit the game/open menu.

It also provides 2 action sets: Desktop and Gamepad. In desktop action set none of the gamepad buttons will even act like gamepad buttons, and instead will emulate keyboard and mouse. D-pad will act as arrow keys, A button will be Enter, B button will be Esc and so on. This is called “lizard mode” for some reason, and on Steam Deck is toggled by holding the Menu (Start) button. Once you switch to gamepad action set, gamepad buttons will act as a gamepad, with the caveat mentioned above.

Gamepad action set also makes the left touchpad behave differently: instead of scrolling and performing a middle click on press, it does a right click on press while moving finger on it does nothing.

hid-steam

Linux kernel includes a driver for these controllers, called hid-steam, so you don’t have to be running Steam for it to work. While it does most of the same things Steam’s userspace driver does, it’s not identical.

Lizard mode is similar, the only difference is that haptic feedback on the right touchpad stops right after lifting finger instead of after the cursor stops, while left touchpad scrolls with a different speed and does nothing on press.

The gamepad device is different tho – it’s now called “Steam Deck” instead of “Microsoft X-Box 360 pad 0” and this time every button is available, in addition to touchpads – presented as a hat and a button each (tho there’s no feedback when pressing).

The catch? It disables touchpads’ pointer input.

The driver was based on Steam Deck HID code from SDL, and in SDL it made sense – it’s made for (usually fullscreen) games, if you’re playing it with a gamepad, you don’t need a pointer anyway. It makes less sense in emulators or otherwise desktop apps tho. It would be really nice if we could have gamepad input AND touchpads. Ideally automatically, without needing to toggle modes manually.

libmanette

libmanette is the GNOME gamepad library, originally split from gnome-games. It’s very simple and basically acts as a wrapper around evdev and SDL mappings database, and has API for mapping gamepads from apps.

So, I decided to add support for Steam deck properly. This essentially means writing our own HID driver.

Steam udev rules

First, hidraw access is currently blocked by default and you need an udev rule to allow it. This is what the well known Steam udev rules do for Valve devices as well as a bunch of other well known gamepads.

There are a few interesting developments in kernel, logind and xdg-desktop-portal, so we may have easier access to these devices in future, but for now we need udev rules. That said, it’s pretty safe to assume that if you have a Steam Controller or Steam Deck, you already have those rules installed.

Writing a HID driver

Finally, we get to the main part of the article, everything before this was introduction.

We need to do a few things:

1. Disable lizard mode on startup
2. Keep disabling it every now and then, so that it doesn’t get reenabled (this is unfortunately necessary and SDL does the same thing)
3. Handle input ourselves
4. Handle rumble

Both SDL and hid-steam will be excellent references for most of this, and we’ll be referring to them a lot.

For the actual HID calls, we’ll be using hidapi.

Before that, we need to find the device itself. Raw HID devices are exposed differently from evdev ones, as /dev/hidraw* instead of /dev/input/event*, so first libmanette needs to search for those (either using gudev, or monitoring /dev when in flatpak).

Since we’re doing this for a very specific gamepad, we don’t need to worry about filtering out other input devices – this is an allowlist, so we just don’t include those. So we just match by vendor ID and product ID. Steam Deck is 28DE:1205 (at least OLED, but as far as I can tell the PID is the same for LCD).

However, there are 3 devices like that: the gamepad itself, but also its emulated mouse and keyboard. Well, sort of. Only hid-steam uses those devices, Steam instead sends them via XTEST. Since that obviously doesn’t work on Wayland, there’s instead a uinput device provided by extest.

SDL code tells us that only the gamepad device can actually receive HID reports, so the right device is the one that allows to read from it.

Disabling lizard mode

Next, we need to disable lizard mode. SDL sends an ID_CLEAR_DIGITAL_MAPPINGS report to disable keyboard/mouse emulation, then changes a few settings: namely, disables touchpads. As mentioned above, hid-steam does the same thing – it was based on this code.

However, we don’t want to disable touchpads here.

What we want to do instead is to send a ID_LOAD_DEFAULT_SETTINGS feature report to reset settings changed by hid-steam, and then only disable scrolling for the left touchpad. We’ll make it right click instead, like Steam does.

This will keep the right touchpad moving pointer, but the previous ID_CLEAR_DIGITAL_MAPPINGS report had disabled touchpad clicking, so we also need to restore it. For that, we need to use the ID_SET_DIGITAL_MAPPINGS report. SDL does not have an existing struct for its payload (likely because of struct padding issues), so I had to figure it out myself. The structure is as follows, after the standard zero byte and the header:

• 8 bytes: buttons bitmask
• 1 byte: emulated device type
• 1 byte: a mouse button for DEVICE_MOUSE, a keyboard key for DEVICE_KEYBOARD, etc. Note that the SDL MouseButtons struct starts from 0 while the IDs Steam Deck accepts start from 1, so MOUSE_BTN_LEFT should be 1, MOUSE_BTN_RIGHT should be 2 and so on.

Then the structure repeats, up to 6 times in the same report.

ID_GET_DIGITAL_MAPPINGS returns the same structure.

So, setting digital mappings for:

• STEAM_DECK_LBUTTON_LEFT_PAD, DEVICE_MOUSE, MOUSE_BTN_RIGHT
• STEAM_DECK_LBUTTON_RIGHT_PAD, DEVICE_MOUSE, MOUSE_BTN_LEFT

(with the mouse button enum fixed to start from 1 instead of 0)

reenables clicking. Now we have working touchpads even without Steam running, with the rest of gamepad working as a gamepad, automatically.

Keeping it disabled

We also need to periodically do this again to prevent hid-steam from reenabling it. SDL does it every 200 updates, so about every 800 ms (update rate is 4 ms), and the same rate works fine here. Note that SDL doesn’t reset the same settings as initially, but only SETTING_RIGHT_TRACKPAD_MODE. I don’t know why, and doing the same thing did not work for me, so I just use the same code as detailed above instead and it works fine. It does mean that clicks from touchpad presses are ended and immediately restarted every 800 ms, but it doesn’t seem to cause any issues in practice, even with e.g. drag-n-drop)

Handling gamepad input

This part was straightforward. Every 4 ms we poll the gamepad and receive the entire state in a single struct: buttons as a bitmask, stick coordinates, trigger values, but also touchpad coordinates, touchpad pressure, accelerometer and gyro.

Right now we only expose a subset of buttons, as well as stick coordinates. There are some very interesting values in the button mask though – for example whether sticks are currently being touched, and whether touchpads are currently being touched and/or pressed. We may expose that in future, e.g. having API to disable touchpads like SDL does and instead offer the raw coordinates and pressure. Or do things on touch and/or click. Or send haptic feedback. We’ll see.

libmanette event API is pretty clunky, but it wasn’t very difficult to wrap these values and send them out.

Rumble

For rumble we’re doing the same thing as SDL: sending an ID_TRIGGER_RUMBLE_CMD report. There are a few magic numbers involved, e.g. for the left and right gain values – originated presumably in SDL, copied into hid-steam and now into libmanette as well ^^

Skipping duplicate devices

The evdev device for Steam Deck is still there, as is the virtual gamepad if Steam is running. We want to skip both of them. Thankfully, that’s easily done via checking VID/PID: Steam virtual gamepad is 28DE:11FF, while the evdev device has the same PID as the hidraw one. So, now we only have the HID device.

Behavior

So, how does all of this work now?

When Steam is not running, libmanette will automatically switch to gamepad mode, and enable touchpads. Once the app exits, it will revert to how it was before.

When Steam is running, libmanette apps will see exactly the same gamepad instead of the emulated one. However, we cannot disable lizard mode automatically in this state, so you’ll have to hold Menu button, or you’ll get input from both the gamepad and keyboard. Since Steam doesn’t disable touchpads in gamepad mode, they will still work as expected, so the only caveat is needing to hold Menu button.

So, it’s not perfect, but it’s a big improvement from how it was before.

Mappings

Now that libmanette has bespoke code specifically for Steam Deck, there are a few more questions. This gamepad doesn’t use mappings, and apps can safely assume it has all the advertised controls and nothing else. They can also know exactly what it looks like. So, libmanette now has ManetteDeviceType enum, currently with 2 values: MANETTE_DEVICE_GENERIC for evdev devices, and MANETTE_DEVICE_STEAM_DECK, for Steam Deck. In future we’ll likely have more dedicated HID drivers and as such more device types. For now though, that’s it.

The code is here, though it’s not merged yet.

Big thanks to people who wrote SDL and the hid-steam driver – I would definitely not be able to do this without being able to reference them. ^^

(I worked on this feature last year, before being moved off desktop related projects, but I never saw it documented anywhere other than in the original commit messages, so here's the opportunity to shine a little light on a feature that could probably see more use)

    The new usb_set_wireless_status() driver API function can be used by drivers of USB devices to export whether the wireless device associated with that USB dongle is turned on or not.

    To quote the commit message:

This will be used by user-space OS components to determine whether the battery-powered part of the device is wirelessly connected or not, allowing, for example: - upower to hide the battery for devices where the device is turned off but the receiver plugged in, rather than showing 0%, or other values that could be confusing to users - Pipewire to hide a headset from the list of possible inputs or outputs or route audio appropriately if the headset is suddenly turned off, or turned on - libinput to determine whether a keyboard or mouse is present when its receiver is plugged in.This is not an attribute that is meant to replace protocol specific APIs [...] but solely for wireless devices with an ad-hoc “lose it and your device is e-waste” receiver dongle.  

    Currently, the only 2 drivers to use this are the ones for the Logitech G935 headset, and the Steelseries Arctis 1 headset. Adding support for other Logitech headsets would be possible if they export battery information (the protocols are usually well documented), support for more Steelseries headsets should be feasible if the protocol has already been reverse-engineered.

    As far as consumers for this sysfs attribute, I filed a bug against Pipewire (link) to use it to not consider the receiver dongle as good as unplugged if the headset is turned off, which would avoid audio being sent to headsets that won't hear it.

    UPower supports this feature since version 1.90.1 (although it had a bug that makes 1.90.2 the first viable release to include it), and batteries will appear and disappear when the device is turned on/off.

A turned-on headset

Registration for GNOME Asia 2024 is now open! This year’s summit will be held from December 6-8, 2024, in the dynamic city of Bangalore, India, with both in-person and remote participation options.

GNOME Asia 2024 will feature a fantastic lineup of presentations and workshops centered around the latest innovations in the GNOME ecosystem and its community. Whether you’re attending on-site in Bangalore or joining online from anywhere in the world, there’s something for everyone.

The full conference schedule, including session and speaker details, will soon be available on the event website.

Registration is open to everyone—whether you’re an experienced developer, new to the open-source world, or simply curious about what’s happening in GNOME. We look forward to welcoming you, both in person and online, from December 6-8!

Register Now Become a GNOME Asia 2024 Sponsor!

We’re still looking for sponsors for this year’s summit. If you or your company are interested in sponsoring GNOME Asia 2024, please find more details and our sponsorship brochure on the event website or reach out to asia@gnome.org.

The systemd docs talk about UsrMerge, and while bootc works nicely with this, it does not require it and never will. In this blog we’ll touch on the rationale for that a bit.

The first stumbling block is pretty simple: For many people shipping “/usr merge” systems, a a lot of backwards compatibility symlinks are required, like /bin → /usr/bin etc. Those symbolic links are pretty load bearing, and we really want them to also not just be sitting there as random mutable state.

This problem domain really scope creeps into “how does / (aka the root filesystem)” work?

There are multiple valid models; one that is viable for many use cases is where it’s ephemeral (i.e. a tmpfs) as encouraged by things like systemd-volatile-root. One thing I don’t like about that is that / is just sitting there mutable, given how important those symlinks are. It clashes a bit with things like wanting to ensure all read files are only from verity-protected paths and things like that. These things are closer to quibbles though, and I’m sure some folks are successfully shipping systems where they don’t have those compatibility symlinks at all.

The bigger problem though is all the things that never did “/usr move”, such as /opt. And for many things in there we actually really do want it to be read-only at runtime (and more generally, versioned with the operating system content).

Finally, /opt is just a symptom of a much larger issue that there’s no “/usr merge” requirement for building application containers (docker/podman/kube style) and a toplevel, explicit goal of bootc is to be compatible with that world.

It’s for these reasons that while historically the ostree project encouraged “/usr merge”, it never required it and in fact the default / is versioned with the operating system – defining /etc and /var as the places to put persistent machine local state.

The way bootc works by default is to continue that tradition, but as of recently we default to composefs which provides a strong and consistent story for immutability for everything under / (including /usr and /opt and arbitrary toplevels). There’s more about this in our filesystem docs.

In conclusion I think what we’re doing in bootc is basically more practical, and I hope it will make it easier for people to adopt image-based systems!

It was long overdue, but better late than never! Shortwave 4.0 is now available on Flathub:

General

• New MPRIS media controls implementation with improved CPU usage
• Song notifications are disabled by default now
• No more loading on startup, stations now get directly retrieved from cached data
• Fixed issue which sometimes prevented loading more than 8 stations from library
• Refreshed user interface by making use of new Libadwaita widgets
• Large parts of the app were reworked, providing a solid foundation for the next upcoming features

Playback

• Last station now gets restored on app launch
• Redesigned player sidebar, allowing to control volume more easily
• New recording indicator showing whether the current playback is being recorded
• Fixed buffering issue which prevented playing new stations, especially after switching stations too fast
• Fixed issues which sometimes prevented that a song gets recorded
• Fixed issue that volume remains muted after unmuting

Station Covers

• More supported image file format for station covers
• Enhanced security by loading station covers using sandboxed Glycin image library
• Non square covers automatically get a blurred background
• New generated fallback for stations without any cover image
• Improved disk usage by automatically purging no longer needed cached data

Browse / Search

• More useful station suggestions by respecting configured system language / region
• Suggestions now get updated with every start, no longer always showing the same stations
• More accessible search feature, no longer hidden in a subpage
• Search results are no longer limited at 250 stations
• Faster and more efficient search by using new grid widgets

Chromecast

• Shortwave is now a registered Google Cast app, no longer relying on the generic media player
• New backend which greatly improves communication stability with cast devices
• Improved discovery of cast devices with lower CPU and memory usage
• Now possible to change the volume of a connected cast device

Enjoy!

1. Some historical background

The GNOME Infrastructure has been hosted as part of one of Red Hat’s datacenters for over 15 years now. The “community cage”, which is how we usually define the hosting platform that backs up multiple Open Source projects including OSCI, is made of a set of racks living within the RAL3 (located in Raleigh) datacenter. Red Hat has not only been contributing to GNOME by maintaining the Red Hat’s Desktop Team operational, sponsoring events (such as GUADEC) but has also been supporting the project with hosting, internet connectivity, machines, RHEL (and many other RH products subscriptions). When the infrastructure was originally stood up it was primarily composed of a set of bare metal machines, workloads were not yet virtualized at the time and many services were running directly on top of the physical nodes. The advent of virtual machines and later containers reshaped how we managed and operated every component. What however remained the same over time was the networking layout of these services: a single L2 and a shared (with other tenants) public internet L3 domains (with both IPv4 and IPv6).

Recent challenges

When GNOME’s Openshift 4 environment was built back in 2020 we had to make specific calls:

1. We’d have ran an Openshift Hyperconverged setup (with storage (Ceph), control plane, workloads running on top of the same subset of nodes)
2. The total amount of nodes we received budget for was 3, this meant running with masters.schedulable=true
3. We’d have kept using our former Ceph cluster (as it had slower disks, a good combination for certain workloads we run), this is however not supported by ODF (Openshift Data Foundation) and would have required some glue to make it completely functional
4. Migrating GNOME’s private L2 network to L3 would have required an effort from Red Hat’s IT Network Team who generally contributes outside of their working hours, no changes were planned in this regard
5. No changes were planned on the networking equipment side to make links redundant, that means a code upgrade on switches would have required a full services downtime

Over time and with GNOME’s users and contributors base growing (46k users registered in GitLab, 7.44B requests and 50T of traffic per month on services we host on Openshift and kindly served by Fastly’s load balancers) we started noticing some of our original architecture decisions weren’t positively contributing to platform’s availability, specifically:

1. Every time an Openshift upgrade was applied, it resulted in a cluster downtime due to the unsupported double ODF cluster layout (one internal and one external to the cluster). The behavior was stuck block devices preventing the machines to reboot with associated high IO (and general SELinux labeling mismatches), with the same nodes also hosting OCP’s control plane it was resulting in API and other OCP components becoming unavailable
2. With no L3 network, we had to create a next-hop on our own to effectively give internet access through NAT to machines without a public internet IP address, this was resulting in connectivity outages whenever the target VM would go down for a quick maintenance

Migration to AWS

With budgets season for FY25 approaching we struggled finding the necessary funds in order to finally optimize and fill the gaps of our previous architecture. With this in mind we reached out to AWS Open Source Program and received a substantial amount for us to be able to fully transition GNOME’s Infrastructure to the public cloud.

What we achieved so far:

1. Deployed and configured VPC related resources, this step will help us resolve the need to have a next-hop device we have to maintain
2. Deployed an Openshift 4.17 cluster (which uses a combination of network and classic load balancers, x86 control plane and arm64 workers)
3. Deployed IDM nodes that are using a Wireguard tunnel between AWS and RAL3 to remain in sync
4. Migrated several applications including SSO, Discourse, Hedgedoc

What’s upcoming:

1. Migrating away from Splunk and use a combination of rsyslog/promtail/loki
2. Keep migrating further applications, the idea is to fully decommission the former cluster and GNOME’s presence within Red Hat’s community cage during Q1FY25
3. Introduce a replacement for master.gnome.org and GNOME tarballs installation
4. Migrate applications to GNOME’s SSO
5. Retire services such as GNOME’s wiki (MoinMoin, a static copy will instead be made available), NSD (authoritative DNS servers were outsourced and replaced with ClouDNS and GitHub’s pipelines for DNS RRs updates), Nagios, Prometheus Blackbox (replaced by ClouDNS endpoints monitoring service), Ceph (replaced by EBS, EFS, S3)
6. Migrate smtp.gnome.org to OSCI in order to maintain current public IP’s reputation

And benefits of running GNOME’s services in AWS:

1. Scalability, we can easily scale up our worker nodes pool
2. We run our services on top of AWS SDN and can easily create networks, routing tables, benefit from faster connectivity options, redundant networking infrastructure
3. Use EBS/EFS, don’t have to maintain a self-managed Ceph cluster, easily scale volumes IOPS
4. Use a local to-the-VPC load balancer, less latency for traffic to flow between the frontend and our VPC
5. Have access to AWS services such as AWS Shield for advanced DDOS protection (with one bringing down GNOME’s GitLab just a week ago)

I’d like to thank AWS (Tom “spot” Callaway, Mila Zhou) for their sponsorship and the massive opportunity they are giving to the GNOME’s Infrastructure to improve and provide resilient, stable and highly available workloads to GNOME’s users and contributors base. And a big thank you to Red Hat for the continued sponsorship over more than 15 years on making the GNOME’s Infrastructure run smoothly and efficiently, it’s crucial for me to emphatise how critical Red Hat’s long term support has been.

I’ve participated in two internships this year, and interns — who are usually busy full-time students — often ask “How do you get time to contribute to open source?”.

And the truth is that there’s no secret formula. It’s tricky to get paid to work on something that you give away for free, isn’t it? Mostly I contribute to open source in free time, either after work hours, or occasionally during periods of downtime.

To my complete surprise I managed to buy a house this year and so I suddenly don’t have any time after work. During the day most of my time is spent on proprietary customer-specific work, and after work I go to look at the house and try to figure out where to start with the whole thing. (By the way, does anyone around Santiago need a load of 1980s-style furniture made from chipboard?).

I’ll still be participating in GNOME around desktop search and the openQA tests, answering questions and triaging bug reports, but I won’t be driving any new stuff forwards.

Anyway, why is it interesting to blog about things I’m not doing?

I read this quote in LWN the other day:

Make it easy to quit – Actively celebrate people who step back from maintainer positions. Celebrate what they accomplished and what they are moving on to. Don’t punish or otherwise shame quitting. This also incentivizes other people to step up, knowing that they don’t necessarily have to do it forever.

— Rich Bowen, “Open Source Summit Vienna 2024”

At least in GNOME, we often don’t do this. We don’t celebrate what people *have achieved*, with I think one exception (the legendary “Pants of Thanks” ceremony).

We should do better at this. It’s not that we don’t appreciate each others work. But mostly we require the person doing the work to also be the one shouting loudly about it, before we notice. Is there a better way?

Another thing we don’t do, by the way, is celebrate corporate participation. The great exception to this is the STF grant, and everyone involved in that did an excellent job of highlighting work which the STF grant enabled. We’re less good at crediting all the work that happens thanks to paid engineers from Red Hat, Endless, Canonical, SUSE, and so on.

Another quote from this article:

Each generation of a project (ie open source but not only open source) is responsible for mentoring the next generation. When you mentor someone, spend time emphasizing that it’s their job to mentor the next person, otherwise they will assume that it’s your job. A failure to commuincate this will result in the eventual attrition and death of the community.

— Rich Bowen, “Open Source Summit Vienna 2024”

I quite like giving conference talks and I’ve been wondering what I could speak about, if I’m not driving any new development myself.

We now have 25 years of history in GNOME and it would be nice to give some talks about “How $thing works.” Desktop search comes to mind here, of course. I also learned (against my will) a lot about initial-setup this year. So I might propose some talks along these lines. It seems like also a nice way to look back at work that’s been done over the years, and give credit the people who have worked on these things over time, doing stuff that’s often invisible.

On that topic, I want to highlight the excellent work done over the summer by our two GSoC interns Divyansh Jain and Rachel Tam, adding a web-based IDE to TinySPARQL that can run queries against the GNOME search database. You can read more about that both on Rachel’s blog and on Demigod’s blog. The idea behind this was making it easier to visualize how the LocalSearch index actually works, what is stored there, and what you can do with it. Hopefully this can lead into some interesting talks about search!

If you like this post, please leave a comment! You use the form below, or reply on the Fediverse to @samthursfield.wordpress.com@samthursfield.wordpress.com. I’m also on LinkedIn.

Last weekend I went to Prague to represent Fedora at LinuxDays 2024. It’s the biggest Linux event in the country with more than a thousand attendees and the Fedora booth is busy there every year.

Like last year the Fedora booth was colocated with the Red Hat booth. It made sense not only because there is a relationship between the two, but it had very practical reasons: I was the only person representing and staffing the Fedora booth and I appreciated help from my colleagues who watch over the Fedora booth when I took a break to have a meal or give a talk.

Post by @fedoracz@floss.social View on Mastodon

The biggest magnet at our booth was again a macbook running Fedora Asahi Remix. I gave a talk about it which was only 20 minutes long and was intended as a teaser: here is an overview of the project and if you’d like to know and see more, come to your booth.

Fortunately just two days before the conference, the Asahi Linux project announced support for Steam via the Fex/muvm emulation, so I could utilize a large library of games I own have a license for on Steam. During the talk someone asked if it could run the Factorio game and it could, indeed.

Post by @fedoracz@floss.social View on Mastodon

We also had a Fedora conference box which includes a Fedora Slimbook laptop. It was a nice contrast to the Macbook because Slimbook focuses on Linux whereas Apple doesn’t care about Linux at all.

The booth was so busy that I was making a post about our presence for 2 hours because I couldn’t find even a few minutes to finish it.

I also did a bit of user support. An older gentleman approached our booth stating that he had traveled 100km to get help. He had a dual boot of Fedora and Ubuntu and an Ubuntu update had broken the bootloader. Regenerating the GRUB resolved the issue.

Pavel Píša, a doctor from Czech University of Technology, invited me to their booth to check out Fedora Linux running on a Milk-V box with a RISC-V CPU. I left a flyer regarding an open Fedora QA position for RISC-V because Red Hat is currently looking for someone to test Fedora Linux on RISC-V.

Me with the RISC-V box. Original post.

Overall, the conference was a great experience, albeit tiring. I hope to attend next year again.

Based on some initial work by Barnabás Pőcze Sysprof gained support for symbolizing stack traces using debuginfod.

If you don’t want to install debuginfo packages for your entire system but still want really useful function names, this is for you. The system-configured debuginfod servers will provide you access to those debuginfo-enabled ELF binaries so that we can discover the appropriate symbol names.

Much like in gdb, you can cancel them if you don’t care and those symbols will fallback to the “In File lib.so+offset” you’re used to.

I expect a bit more UI work around this before GNOME 48 like preferences to disable it or configure alternate debuginfod servers.

Happy profiling!

We’re trying to increase the fwupd coverage score, so we can mercilessly refactor and improve code upstream without risks of regressions. To do this we run thousands of unit tests for each part of the libfwupd public API and libfwupdplugin private API. This gets us a long way, but what we really want to do is emulate the end-to-end firmware update of every real device we support.

It’s not trivial (or quick) connecting hundreds of devices to a specific CI machine, and so for some time we’ve supported recording USB device enumeration, re-plug, firmware write, re–re-plug and re-enumeration. For fwupd 2.0.0 we added support for all sysfs-based devices too, which allows us emulate a real world NVMe disk doing actual ioctls() and reads() in every submitted CI job. We’re now going to ask vendors to record emulations for existing plugins of the firmware update so we can run those in CI too.

The device emulation docs are complicated and there’s lots of things that the user can do wrong. What I really wanted was a “click, click, save-as, click” user experience that doesn’t need to use the command line. The tl;dr: is that we’ve now added the needed async API in fwupd 2.0.1 (probably going to be released on Monday) and added the click, click UI to gnome-firmware:

There’s a slight niggle when the user starts recording the first “internal” device (e.g. a NVMe disk) that we need to ask the user to restart the daemon or the computer. This is because we can’t just hotplug the internal non-removable device, and need to “start recording” then “enumerate device(s)” rather than the other way around. Recording all the device enumeration isn’t free in CPU or RAM (and is possibly a security problem too), and so we don’t turn it on by default. All the emulation is also all controlled using polkit now, so you need the root password to do anything remotely interesting.

Some of the strings are a bit unhelpful, and some a bit clunky, so if you see anything that doesn’t look awesome or is hard to translate please tell us and we can fix it up. Of course, even better would be a merge request with a better string.

If you want to try it out there’s a COPR with all the right bits for Fedora 41. It’ll might also work on Fedora 40 if you remove gnome-software. I’ll probably switch the Flathub build to 48.alpha when fwupd 2.0.1 is released too. Feedback welcome.

Dear community members,

As promised in the previous communication the Board would like to share some more details on our current financial situation and the budget for our 2024-2025 financial year, which runs from 1st October 2024 to 30th September 2025.

Background

• The Foundation needs an approved budget in place because our spending policies use the budget to authorise what staff and committees are allowed to spend money on. This year we passed the budget on time for the start of the financial year, which was thanks to a lot of detailed and particularly challenging work by Richard, which the board is grateful for.
• We consider the budget in 2 distinct parts:
  • Budget for our fiscally-sponsored projects. We consider their income, but not their expenses. The reason for that is that the Foundation takes a small part of the income as the fiscal sponsorship fee, supporting our administrative and operating costs. Funds received on behalf of other projects are tracked separately, called “reserved funds”, and the Foundation cannot spend money that belongs to the other projects.
  • General operating budget for the GNOME Foundation, which is what this post is all about! At any later point, when talking about the budget, we’re talking about the general/unrestricted operating funds and it is safe to assume that income for fiscally-sponsored projects is not included.
• The budget for the previous 2023-2024 fiscal year was presented to the board as a roughly balanced break-even budget, anticipating $1.201M of revenue and $1.195M of expenses. The board considered two fundraising scenarios proposed by our previous ED, with the most ambitious scenario planning to raise an additional $2M for the Foundation, and one more conservative which anticipated an additional $475k of revenue from various sources (donations, grants, event sponsorship). This more conservative scenario was included in the budget, but in practice things did not work out as planned. This additional funding was not raised, meaning that in practice the Foundation once again ran at a deficit over the past year and used funds from our reserves.
• The new 2024-2025 budget considers a total income of $586k, and total expense of $550k. Two things are clearly different from last year: the expenses have been greatly reduced, and we have aimed for a surplus instead of the deficit we ended up with last year. Both things were a consequence of the budget from previous year not being executed as expected. Since our reserve policy requires us to retain enough money to sustain core operations without income for another year (specifically, 1.1 times core spending), we’ve had to reduce expenses to save money and restore our reserves.

So, let’s dig into the details:

Income

• $205,100 in donations. This number is based on previous years income, of individual contributions ($75,000), Advisory Board fees ($105,800), and other small contributions ($7,800) like matching donations (where companies double what employees donate). It also includes $16,500 currently pending from Wau Holland Stiftung, an organization we had a historic agreement with to collect funds from European donors that is tax deductible. We believe that there is a great potential for the GNOME Foundation to increase the amount of individual contributions received, and this has been included in the Strategic Plan and many board discussions. Unfortunately, without a permanent Executive Director, we cannot guarantee that we will be able to establish a program to do so in the short-term, so we have decided to budget conservatively to ensure economic sustainability.
• $64,500 from event sponsorship. Most of that money comes from GUADEC ($61,000), with some from LAS and GNOME Asia, which is one of the main reasons why we are able to maintain our events: because they are sponsored separately, they are mostly self-sustaining.
• $65,500 in fiscal sponsorship fees. This is based on a % fee the GNOME Foundation takes for our operational costs from hosting GIMP and Black Python Devs. This number is uncommonly high due as we have been workng with the GIMP on financial and legal arrangements to receive approx $1M of historical Bitcoin donations. (And sell them immediately – holding Bitcoin assets creates a regulatory/reporting problem for US nonprofits and our accountants have advised us against it.)
• $1,000 in interest from money in the bank account. This is budgeted higher than previous years, as work is already in progress to change bank accounts to increase this income, as recommended by our auditors.
• $500 profit from selling T-shirts and other goods ($2,500 income, $2,000 in expenses).
• $250,000 from the 2nd year of an Endless grant that was approved last year. This grant provides $50,000 for general funds that the Foundation can use at its discretion, and $200,000 that need to be spent on specific tasks. Currently, those are assigned to Flathub, Parental Controls, GNOME Software maintenance, and internships. Some of those will be detailed in the expense section.

Expenditures

• $10,000 interim ED salary. This is to be able to pay Richard to continue managing the Foundation and staff team until 10th December.
• $100,000 for development contractors for work associated with the Endless grant. This work includes improvements in Parental Controls and GNOME Software, and is being executed by Philip Withnall (development), Sam Hewitt (design) and potentially one more developer over the coming year. Philip gave an update on the work in his presentation at GUADEC.
• $110,600 in contractor costs for program staff, including events and infrastructure. This covers Kristi’s work which is the backbone of events such as GUADEC, LAS and GNOME.Asia, and Bart’s work running GNOME and Flathub infrastructure. The Flathub portion of this work is funded by the Endless grant.
• $32,000 in Outreachy interships. This is a long-term partnership with Conservancy and commitment by the GNOME Foundation as the original birthplace of the Outreachy initiative. They are supported this year by reallocating some of the Endless grant, with their permission. This will pay for a total 4 interns between the winter and summer cohort.
• $20,000 in contractor support. This is allocated for part-time contracting of Thibault Martin and Dawid Jankowiak to support the STF team and work on a crowdfunding platform for our development fundraising. Some of this is funded by the Endless grant and will be spent on coordinating the next steps of the Flathub payments/donations launch.
• $158,000 in employment/contractor costs for operations and admin staff, supporting the GNOME Foundation across finances, events and community initiatives.
• $47,500 in professional services, ie legal and accounting. These include a reserve for legal fees ($10,000), an external accounts audit for the previous financial year ($17,500), which is required due to our income (mostly due to STF) being over the $2M threshold, and accounting fees ($20,000). Some of the financial and legal costs are driven by work setting up Flathub LLC and are covered by the Endless grant.
• $3,200 in office expenses, mostly related to postal expenses required for sending material between contractors, staff, and event organisers.
• $54,000 in conferences and travel. These include the budget for the conferences themselves ($30,000), which includes GUADEC, GNOME Asia, and hackathons around the globe, but also travel for staff ($12,000) and community ($12,000). Travel particularly has been significantly reduced from previous year, but should still allow for staff/organisers to attend our events, and for the travel committee to support some community travel to GUADEC and GNOME Asia.
• $15,000 in other fees. These include banking costs for sending money from the US to Europe, PayPal fees, and insurance. They might seem high, but are in total less than 1.5% of the cash flow of the Foundation, which is within the expected value for any organization.

Balance

• As of the preparation of this budget, we have approx $140,000 in GNOME Foundation reserves. There’s a lot more money in the bank, but they are reserved funds held for GIMP and BPD.
• We need to ensure that we meet our reserve policy of retaining 1.1 times core spending. Unfortunately, core spending is fairly loosely defined. This year, we have considered: Events and minimal staff travel, part-time infrastructure support, minimal staff, and some fees and professional services. In total, we accounted that we would need at least $158,000 at the end of the year to be able meet the policy.
• The approved budget should put our reserves around $176,000 at the year end, which is slightly above our reserve policy. Considering we used a very limited interpretation of the reserves policy, it’s better to include a small safety margin for any unanticipated costs.

Conclusion

With limited time from our interim Executive Director (ED), Richard Littauer, who is working part-time, the board is prioritising: recruiting our new ED, delivering our current project/grant commitments (to STF and to Endless), and fundraising for development work. This includes working with the community to launch our development fund crowdfunder/platform and plan a follow-up project for STF grant, so that the GNOME Foundation can support and grow its direct investment in project development.

Keen readers will note that there is nothing in the current budget for the ED’s salary. We are in discussions with a potential donor to see whether we can find support for the salary for the ED for the first year. In any case, transparently sharing our financial situation and fundraising needs is an essential part of any ED recruitment process, so we could still recruit somebody with “raise money for your own salary” being their first priority.

Hopefully this additional detail helps to show the challenges of our current situation, and why we had to make really tough decisions, like parting ways with some greatly appreciated members of our staff team. We hope this sheds some more light on why those decisions were taken, provides confidence on the work done by the board and the ED, and where we currently stand. We are also very relieved to be able to provide a surplus budget for the first time in many years, and doing so while still being able to support the community: events, infrastructure, internships, travel funding, and meeting our commitment to donors for work done in some parts of the stack, e.g.: Flathub, parental controls and GNOME Software.

We welcome any feedback and questions from the GNOME community. Thanks to all of our GNOME members, contributors, donors, sponsors and advisory board members!

The GNOME Foundation Board of Directors

This year PyCon Spain happened just down the road from me in Vigo, and I was able to attend with a few folk from Codethink. Python conferences are usually great as there’s so much variety in the talks and the bar for quality is also quite high.

I’m not sure when the videos will be out and they’ll anyway be in Spanish, but here are some notes for the talks I found most interesting. (Codethink sponsors our attendence to conferences on the basis that we write an internal report afterwards, which is where most of these notes came from).

Mapping illegal tourist apartments with Python

https://pretalx.com/pycones-2024/talk/PWEFGB/

Presented by Juan Luis Cano Rodríguez, who is a co-founder of Python España and PyCon ES. This talk was activism with some data engineering, in the context of the housing crisis that’s affecting most Spanish cities and tourist destinations.

This talk is the story of how he accidentally got involved as an activist after he noticed someone in his own appt building in Madrid was doing short-term rents to tourists, and wondered whether they had a license.

He started by asking the audience to guess how many long-term rental properties are available on Idealista for Madrid. Is it 7,000, 21,000 or 43,000? Then he asked to guess how many short-term rentals are available on AirBnB.

Tourist apartments in Madrid are licensed only after a professional inspection. The council publish data on how many licenses they’ve issued, and there are 1008 properties listed. Meanwhile the property register, which doesn’t require any inspection, lists 11,000 short-term rental properties. AirBnB meanwhile has 26,000 offers in Madrid. He realized that illegal apartment rentals are far worse than he had imagined.

He used Python (of course) to compare the register of properties against the license data to identify which properties were unlicensed, and participated in a group effort to send 10,000 individual reports to the council in one day. This made the national news back in June. The mayor of Madrid even praised this “citizens effort”.

AirBnB has a “license” field for short-term rentals in Spain, but they don’t do any checking of whether it’s valid. He experimented with their “Report” button with an apartment he knew was unlicensed, and AirBnB’s internal “investigation team” reported back a couple of days later that they had found nothing wrong.

Towards the end of the talk he offered the audience a choice, he could explain the data wrangling with Python in more detail, or give his thoughts on the housing crisis. The vote was around 90% for the latter, and his message was that (unsurprisingly) a lot more pressure is needed before anything is going to change. He finished by announcing a protest happening on the 13th October in Madrid.

The talk was very well received and could probably have filled 3 hours instead of 35 minutes.

How we are removing the GIL in Python

https://pretalx.com/pycones-2024/talk/ESYBVA/

Presented by Pablo Galindo Salgado, a Python core developer and Bloomberg engineer, this was a summary of the incredible work going on to remove CPython’s “Global Interpeter Lock” or GIL. The full technical details of this are in PEP 703, this is my short summary.

Again this talk could have been a full-day workshop but Pablo packed a lot of details into the time, using the simple of trick of talking incredibly fast.

CPython’s global lock means that writing multithreaded Python code is a waste of time, as the threads just get stuck waiting for the global lock. Today people use horrible workarounds such as the ‘multiprocessing’ library.

However, the GIL has advantages too, mainly its simplicitly. Deadlocks are a common problem in multithreaded code but currently CPython is immune to all that — a deadlock is caused when two mutexes block each other, but of course CPython only has the one lock.

So the big challenge is replacing the GIL with many separate locks, without making CPython slower, and avoiding deadlocks. CPython uses reference counting to manage memory, and this is the hardest piece of the puzzle as sharing objects between threads means every refcount change must be atomic and this can require locking, which limits throughput.

People have been trying to solve this since the mid 1990s and noone has yet succeeded, but the work here is actually being released this week in Python 3.13. It’s behind a compile-time configure flag, and the expectation is to spent around 5 years testing this before it’s enabled by default for everyone.

A minimal overview of how they’ve done it:

• switch to biased ref counting for Python objects: this means there are now two refcounts per object, a thread-local refcount and a global refcount. The global one can be slow to access, but most of the time a local refcount is good enough and this is a fast path.
• make common objects immortal: remember Python dates from the golden age of object-oriented programming, and everything is an object including True, False, None, the number 4 and so on. By setting the highest 2 bits of the refcount, these are now marked as immortal.
• fast locking: using spinlocks from WebKit (the excellently named WTF::Lock) where possible
• new memory allocator: use mimalloc, which allows linked list segments to be allocated contiguously instead of all over the place.

It seems a lot of this work has been funded by corporations, mainly Meta as noted in the Python 3.13 release notes, and we can assume Bloomberg are somewhat involved as well.

I hope Pablo does the same talk in English at some point because it was excellent. In the meantime you can hear him on the Changelog podcast:

The tsunami of disinformation: how Generative AI can help us in the fight against Fake News.

https://pretalx.com/pycones-2024/talk/8ZXZ8Z/

Presented by Rubén Míguez and Agustín Cañas from Newtral, a fact-checking organization in Spain.

There are many bullshit talks around about how generative AI will magically solve some or other intractable problem, as if a text synthesis machine built from Reddit comments is somehow going to succeed where our best scientists and leaders have so far failed. I recommend reading https://pivot-to-ai.com/ on that topic, but let me get back to
the talk.

The talk began with an overview of fake news, which I’m sure I don’t need to repeat here, except the excellent statistic taken from Washington Post fact-checkers that during 2016-2020, the then US president made around 30,000 provably false statements, which works out at 21 lies a day, every day for four years including Sundays and bank holidays. So it’s been an interesting time in the world of fact-checkers.

Newtral exists since 2013 and began in a very analogue way: human fact checkers watching news programs with big notebooks and writing down things to be verified. The goal of the small technology team was not to replace the human fact checkers but to build tools that could make them work 30x as fast.

(So if you were wondering how generative AI tools could possibly detect fake news, when Google’s “state of the art” AI is telling people to throw car batteries into the ocean and put glue on pizza, the answer is of course: it’s not going to, there will still be a human fact
checker in the loop).

The fact checking process is roughly as follows:

1. Monitor media
2. Spot facts
3. Verify them
4. Publish (“exploit”) the result

The talk detailed some tools to automate the first two parts: using speech-to-text models to transcribe political speeches and so on as they happen, and then text analysis tools to identify potentially “verifiable” statements. As well as deciding whether a statement can be
verified, they have to analyse whether it matters, whether the misinformation could be dangerous, whether it’s a joke, and so on. There wasn’t much detail how they do this in the talk, unfortunately.

Then they mentioned a multimodal model named ClaimCheck which has run as a WhatsApp bot since 2020 where people can send forwarded messages to see if they are true. This works using RAG (Retrieval Augmented Generation) to search an existing database of human-verified facts. Again I’d have loved more details on how this works.

Finally they mentioned that Newtral will be launching a technology-focused spinoff named Trueflag, aiming to build the existing tech into a software-as-a-service tool for fact checking that can be used by news organisations, social media platforms and so on.

I found this article online which gives some more info:

• https://www.wired.com/story/fact-checkers-ai-chatgpt-misinformation/

I’m still very curious about how far they can get with this tech and will be following closely.

Other note

There was some talks I didn’t enjoy so much. One about using Pyomo to model the game Age of Empires 2, delivered by some mathmeticians who unfortunately didn’t actually hook up their model to the real game, and the second half the talk was just graphs when what we obviously wanted was footage from actual Age of Empires 2 games. Pyomo looks interesting though.

I also saw a talk on the Robot Framework for testing. The talk was well delivered but I fundamentally disagree with the premise that by hiding Python code behind a bunch of free form text, “non-programmers” will be able to write and adapt test suites. The tool itself looks fine, but I wonder when we’ll learn that there’s no tool that can magically make testing easy, people just have to learn to write excellent test suites using code, and that’s the only way anyone will be able to produce a good test suite. (I’m considering doing a talk myself called “How to write a good test suite using regular programming tools”.)

Vigo was of course fantastic despite the intermittent rain, minor travel issues and increasingly strange procedures for checking in to tourist apartments. (This is the first time I’ve stayed in a place with a “Virtual key”, a web page linked to a smart lock on the apartment door which means you need working internet to enter the place). At least the rental apartment did have a license.

I’d recommend Python community events to everyone. People use Python for all kinds of different things – science, journalism, art, activism, and so on – you always get an interesting mix of talks and not just dry technical presentations of screenshots of code. Even the very technical talk on the GIL was not dry at all. Wherever you live you probably have a local Python community and they may well organize an annual conference, worth investigating.Those of us outside the UK no doubt also have a national Python
community who probably hold a conference from time to time. Get
involved!

Thanks as always to Codethink for sponsoring accommodation, food and sponsoring the event itself.

A long overdue dev log. The last one was for September 2023. That's a year. Stuff in life has happened.

Compiano

In November I switched Compiano to use pipewire directly for sound. This mean removing bits of UI too.

I should look at a release, but I have a few blockers I need to tackle. One key element is that there is a mechanism to download the soundbanks and for now it doesn't ask consent to do so. I don't want a release without this.

Raw thumbnailer

I already posted about it.

libopenraw

Lot has happened on that front. I want it to be a foundation of Niepce and others.

First adding it to glycin triggered an alpha release on crates.io. There started a long cycle of alpha release, we are at alpha 8 as of now. Various changes:

• Added the mp4parse crate directly as a module. A key reason is that I already used a fork so it complicate things. Maybe I should make these few bits upstreamable and use upstream.
• Added a mime types API
• Save up a lot of RAM when doing colour interpolation: it's done in place instead of allocating a new buffer. This is significant as this is done using a 64-bit float per component.
• Fixed the rendering in many ways. The only thing it needs it to apply the colour balance.
• Fixed unpack or decompression of Olympus and Fuji files.
• Got an external contribution: Panasonic decompression. This made me fix the loading of uncompressed Panasonic raw too. The more recent Panasonic cameras are still failing though, there is a subtle variant that needs to be handled.

Still missing from rendering: recent Nikon and all their exotic variants and compression scheme, Canon CR3, GoPro, Sony.

Niepce

Not so much work directly done on Niepce in the last few month, but still.

Ongoing features

Started a while ago some work towards the import and a rework of the catalog (the main data storage).

The former is the implementation of a workflow that allow immporting images into the catalog.

The latter involve reworking the catalog to become a self contained storage as a sqlite3 database. One step I already did was to use it to store the catalog preferences instead of a separate file. This should also include fixing the UI open, creating, switching.

This big two things are user visible and are a stop forward what I want to happen as an internal milestone. Then I can start pluging the library import and maybe import my picture vault. A good starting point towards managing the collection, but not really for photo editing yet. Gotta make choices.

Images

Implemented support for HEIF which is being adopted by camera manufacturers.

I updated the RT engine to 5.11 which came with RawTherapee 5.11. This is still a soft work of the code base to use strip out Gtk3 and a more recent version of glibmm. The latter patch might no longer be needed as I have since removed gtkmm from Niepce.

I also implemented the GEGL pipeline using gegl-rs, which I took over to make it useful. At one I shall try to figure out how to write a loader in Rust to use libopenraw with GEGL.

Cleanups

The UI is slowly moving to use blueprint, and I removed all the first-party C++ code outside of bindings, no more Gtkmm and Glibmm is only here because RT engine needs it.

Other

Stuff I contributed to.

STF

I took part to the STF effort and worked on fixing issues with the desktop portal. The big chunk of the work related to the USB portal, taking over code by Georges that is itself based on code by Ryan. It spreads through multiple component of the stack: flatpak, xdg-desktop-portal, xdg-desktop-portal-gnome, libportal and ashpd.

I also did a bunch of bug fixes, crashes, memory leaks, etc in flatpak, flatpak-builder, and the rest of the stack.

I also implemented issue #1 for flatpak-builder: easy renaming of MIME files and icons, and also properly fixing the id in the appstream file, a common problem in flatpak.

Glycin

Glycin is a sandboxed image loader. I did implement the raw camera loader using libopenraw. It's written in Rust, so is libopenraw now. Thank you Sophie for merging it.

Poppler

Jeff was complaining about a file being super slow, with sysprof flamegraph. That picked my curiosity and looked at it. The peculiarity of the document is that it has 16000 pages and a lot of cross references.

This lead to two patches:

• The first one is in 24.06. There was a loop, calling for the length of the container at each iteration. Turns out this is protected by a mutex and that's a lot of time spend for nothing since the value is immutable. Call it once before the loop and voila.

• The other one, merged for 24.09 change that loop to be a hash table lookup. The problem is that it want to locate the page by object reference, but iterate through the page list. Lots of ref and lots of a page mean even more iterator. The more complex approach is when building the page cache (it's done on demand), we build a reference to page index map. And the slow code is no longer slow and almost disappear from the flamegraphs.

This make Evince and Okular faster to open that document and any presenting similar attribute: a lot of bookmarks.

Last weekend we had another edition of last year’s post-All Systems Go hackfest in Berlin. This year it was even more of a collaborative event with friends from other communities, particularly postmarketOS. Topics included GNOME OS, postmarketOS, systemd, Android app support, hardware enablement, app design, local-first sync, and many other exciting things.

This left us with an awkward branding question, since we didn’t want to name the event after one specific community or project. Initially we had a very long and unpronounceable acronym (LMGOSRP), but I couldn’t bring myself to use that on the announcement post so I went with something a bit more digestible :)

“Boiling The Ocean” refers to the fact that this is what all the hackfest topics share in common: They’re all very difficult long-term efforts that we expect to still be working on for years before they fully bear fruit. A second, mostly incidental, connotation is that the the ocean (and wider biosphere) are currently being boiled thanks to the climate crisis, and that much of our work has a degrowth or resilience angle (e.g. running on older devices or local-first).

I’m not going to try to summarize all the work done at the event since there were many different parallel tracks, many of which I didn’t participate in. Here’s a quick summary of a few of the things I was tangentially involved in, hopefully others will do their own write-ups about what they were up to.

Mobile

Mainline Linux on ex-Android phones was a big topic, since there were many relevant actors from this space present. This includes the postmarketOS crew, Robert with his camera work, and Jonas and Caleb who are still working on Android app support via Alien Dalvik.

To me, one of the most exciting things here is that we’re seeing more well-supported Qualcomm devices (in addition to everyone’s favorite, the Oneplus 6) these days thanks to all the work being done by Caleb and others on that stack. Between this, the progress on cameras, and the Android app support maybe we can finally do the week-long daily driving challenge we’ve wanted to do for a while at GUADEC 2025 :)

Design

On Thursday night we already did a bit of pre-event hacking at a cafe, and I had an impromptu design session with Luca about eSIM support. He has an app for this at the moment, though of course ideally this should just be in Settings longer-term. For now we discussed how to clean up the UI a bit and bring it more in line with the HIG, and I’ll push some updates to the cellular settings mockups based on this soon.

On Friday I looked into a few Papers things with Pablo, in particular highlights/annotations. I pushed the new mockups, including a new way to edit annotations. It’s very exciting to see how energetic the Papers team is, huge kudos to Pablo, Qiu, Markus, et al for revitalizing this app <3

On Saturday I sat down with fellow GNOME design contributor Philipp, and looked at a few design questions in Decibels and Calendar. One of my main takeaways is that we should take a fresh look at the adaptive Calendar layout now that we have Adwaita breakpoints and multi-layout.

47 Release Party

On Saturday night we had the GNOME 47 release party, featuring a GNOME trivia quiz. Thanks to Ondrej for preparing it, and congrats to the winners: Adrian, Marvin, and Stefan :)

Local-First

Adrian and Andreas from p2panda had some productive discussions about a longer-term plan for a local-first sync system, and immediate next steps in that direction.

We have a first collaboration planned in the form of a Hedgedoc-style local-first syncing pad, codenamed “Aardvark” (initial mockups). This will be based on a new, more modular version of p2panda (still WIP, but to be released later this year). Longer-term the idea is to have some kind of shared system level daemon so multiple apps can use the same syncing infrastructure, but for now we want to test this architecture in a self-contained app since it’s much easier to iterate on. There’s no clear timeline for this yet, but we’re aiming to start this work around the end of the year.

GNOME OS

On Sunday we had a GNOME OS planning meeting with Adrian, Abderrahim, and the rest of the GNOME OS team (remote). The notes are here if you’re interested in the details, but the upshot is that the transition to the next-generation stack using systemd sysupdate and homed is progressing nicely (thanks to the work Adrian and Codethink have been doing for our Sovereign Tech Fund project).

If all goes to plan we’ll complete both of these this cycle, making GNOME OS 48 next spring a real game changer in terms of security and reliability.

Community

Despite the very last minute announcement and some logistical back and forth the event worked out beautifully, and we had over 20 people joining across the various days. In addition to the usual suspects I was happy to meet some newcomers, including from outside Berlin and outside the typical desktop crowd. Thanks for joining everyone!

Thanks also to Caleb and Zeeshan for helping with organization, and the venues we had hosting us across the various days:

• offline, a community space in Neukölln
• JUCR, for hosting us in their very cool Kreuzberg office and even paying for drinks and food
• The x-hain hackerspace in Friedrichshain

See you next time!