Two weeks ago, I released AppStream 1.2.0. This release contains a lot of great changes, but one of the most important ones concerns how media are being handled, and AppStream’s default image export format.
AppStream is a Freedesktop metadata standard to describe software components. That can be anything from system services over fonts to console and graphical applications. AppStream metadata is supposed to give users enough information to decide whether they want to install a piece of software, to represent that piece of software, and to give the operating system enough information to decide whether a software component should be installed automatically and (to some extent) what capabilities and relations it has, to provide the user with sensible options.
Especially for the first two goals, and especially for GUI applications, AppStream supports icons and screenshots, which are used to showcase applications. Today, AppStream is used by all kinds of services, from Linux distributions over firmware updates to Flatpak and desktops directly. AppStream’s original design however comes from the perspective of Linux distributions in 2011, where you may want to browse the software catalog offline, without delay, and without pinging an external server (which could be a privacy concern).
Therefore, a common way to deploy an AppStream-enabled software repository is to ship all icons of all applications in the repository to the user as part of the repository metadata download. AppStream does support remote icon downloads nowadays, and for a while I thought that this would become the default eventually. However, especially in today’s world, having a bandwidth-saving, instantly responsive, privacy-protecting application browsing experience seems more important that ever.
PNG images are great!The only format that AppStream supports for icons and screenshots (which are downloaded on-demand from your distributor’s CDN) has always been exclusively PNG. PNG images are perfect for icons, because they compress well (especially for common icon shapes), are fast and simple to load, and can be loaded anywhere, by any toolkit or webbrowser. They also ensure we deliver faithful screenshot images, even though we may have scaled or re-rendered them. Still though, PNG images are less great for screenshots, as they are not very efficient, which puts strain on any CDN that has to deliver them, as well as on people’s internet connections when browsing screenshots. Having smaller thumbnails alleviates that problem a little, but does not fully solve it.
But even for icons, PNG could be improved upon: In many cases, icons are re-downloaded with the repository metadata again and again, so having a large icon tarball adds up to the data transferred during metadata refreshes. AppStream also now supports large 128x128px icons, which nobody in 2012 expected we would need, adding even more data that will be re-downloaded. Saving some space here translates directly to lower bandwidth costs as well as faster downloads for users.
To improve PNG file sizes, the AppStream Compose library, which handles all image processing and metadata catalog composition, was running optipng on all generated PNG images. That does create smaller PNG images, but they were still relatively large compared to other image formats.
For a long time though, there was no alternative to PNG images for icons: There was no lossless image compression format that could give us the same quality as PNG images and that was also widely supported.
JPEG-XL vs PNG in AppStreamSince 2021 we have JPEG-XL (JXL), which offers a true lossless mode with often better compression than PNG. The issue was that JPEG-XL wasn’t widely supported. Then, in 2025, the PDF Association selected JPEG-XL as the preferred image format for HDR images in PDFs, and now we are finally getting browser support and more ubiquitous availability of the format (you can try it right now in Firefox!).
For screenshots, using JXL’s lossy mode, it has obvious and extreme size advantages over PNG, so supporting JXL or WebP for screenshot images was an obvious choice. If JXL would support the lossless case very well as well though, we could serve many use cases with the same exported image format, which is very attractive to me.
So, the obvious next question was whether it was worth the pain of switching the icon format, so I did some measurements on real icons. For that I used the AppStream component icon pool that Debian Unstable ships, which is almost 5000 application icons of various sizes, and converted them to PNG:
Icon sizeIconsPNG totalJXL totalPool savedPNG avgJXL avgMedian savedMean saved Worst BestLarger as JXL 48×48 1544 3.7 MiB 3.0 MiB 17.8%2.4 KiB2.0 KiB 17.9% 16.7%-118.7%60.0% 206 64×64 2018 7.0 MiB 5.8 MiB 17.8%3.6 KiB2.9 KiB 18.0% 15.8%-112.7%70.0% 279 128×128 1411 11.2 MiB 8.7 MiB 22.0%8.1 KiB6.3 KiB 20.1% 17.5% -89.7%61.0% 209 TOTAL 4973 21.9 MiB 17.5 MiB 19.9%4.5 KiB3.6 KiB 18.6% 16.6%-118.7%70.0% 694PNG images saved with libpng at effort=4, compression=9, then optimized using optipng -o2, JXL images encoded using vips jxlsave lossless=1 effort=7 strip=1 via VIPS/libjxl.
As the table shows, using lossless JXL images over size-optimized PNG images (using optipng’s default settings) provides a roughly 20% gain. This does not look like much, until you consider how often these files are downloaded: A 20% file size reduction may only save 1-2 MiB of disk space, but if they are downloaded over and over again by many clients, it will save a lot of bandwidth.
Interesting JXL encoding findingsAs a sidequest, I was curious why some images were larger than their PNG counterparts when encoded with JXL, and what the ones that were significantly smaller were.
In short, the biggest size reductions for JXL existed on images that were already small as PNG, and contained large, flat color surfaces with hard edges and simple shapes. They were not very interesting, and much of JXL’s wins come from accumulating smaller gains across all files, which compound the bigger icons get (especially at 128x128px, where JXL truly shines).
The events were JXL loses to PNG are more interesting: For example, it does quite poorly with pixel-art images that have a lot of repeating patterns. Those are encoded well by PNG, but less efficiently by JXL. Take for example Vonsh:
Icon of Vonsh, an SDL-based snake game, which PNG compresses better than JXLMy guess is that while PNG can exploit the repeating pixel patterns for compression, JXL’s predicts surrounding pixels from its neighbours, which fails too often and makes it pay almost full entropy per pixel. In this single rare case, the PNG is at 5.4 KiB, while the JXL is almost 8 KiB in size.
Other cases I looked at were arguably buggy input data, where color channels were hidden under the alpha channel of the input image. PNG could probably again exploit repeats, while we were forcing JXL to encode pixels that were invisible in the final image. This is arguably a problem with the original input data. Currently, AppStream does not make any changes to icons at all, but in future we might add a filter that removes invisible colors from images to solve this pathological case (it was only two icons out of 5000 though, so it is not a high priority).
The third case I found where JXL loses to PNG were icons with checkerboard-like patterns:
Icon of x3270, an IBM 3270 Terminal EmulatorFor those, PNG can likely again exploit the repeating patterns, while a checkerboard layout is pretty bad for left/top predictors like JXL’s. However, in this case the size difference (and loss for JXL) is only 450 bytes, so even though JXL loses to PNG, it does so not by much.
JXL in AppStreamGiven these findings, JPEG-XL is the default image format starting with AppStream 1.2.0. AppStream Compose will encode all images losslessly as JXL, while screenshots are encoded in lossy mode at Q=90 effort=7. Since the optipng step does not happen for JXL images, this comes at no speed penalty and is even a bit faster on modern x86_64 CPUs (where libjxl can use SIMD). PNG is still available, and Compose can be told to switch between the two formats.
Upsides of JXL in AppStream right nowIf you use JXL in Compose or the recent release of appstream-generator, you will get much smaller images and, for screenshots, will benefit from other JPEG-XL features such as progressive decoding, providing a far nicer user experience. libAppStream has supported JXL icons since version 1.1.3, so your clients will need that version or a newer one, and all software centers will have to support loading JXL images (which all of them do, provided the right plugins are installed).
Downsides of switching to JXL too quicklyJXL is a very new format, so web browsers might not yet display it if you are serving webpages. Your clients may also have bugs in processing JXL images, as the format is still “new”. For example, switching on JXL in Debian sent KDE Discover into an infinite loop on startup while trying to load the icons (an issue which has been fixed, but clients will need that patch first before JXL is switched on).
This currently makes JXL enablement only possible when you know that your clients can support it. This is the case for me in Debian Unstable and Debian 14, which are using JXL images for a few weeks now, but not for any older releases. Platforms like Flatpak have it even harder, because they do know even less about their clients. So, even though it has big advantages, you may want to hold off on using JXL right away, and force PNG by setting the ImageFormat key to png in appstream-generator‘s configuration, or passing --image-format=png to appstreamcli compose.
It is also worth mentioning that JPEG-XL is much, much slower on systems that do not have SIMD instructions or for which the libjxl/jxl-rs library does not have them (such as apparently riscv64 right now). If this is a concern, you might not want to switch to JXL right away.
Media pipeline improvementsBesides the JXL default change, AppStream 1.2.0 also comes with a complete overhaul of its media processing pipeline. While libappstream, AppStream’s main library, does not do any media processing and comes with very minimal dependencies to be embedded in client applications and used on servers, the same can not be said about libappstream-compose, AppStream’s library to build metadata generating applications (the server-side part, usually).
The compose library has to render fonts into font specimen cards, inspect translation files, render SVG images, decode all kinds of raster images, inspect video files, etc. Especially the fonts, and the fact that fonts can appear in SVG images, has caused issues in the past, as libappstream-compose is a heavily threaded library and most font libraries can only work from a single thread. This forced the library to essentially go into single-thread mode anytime anything that could touch a font was being processed.
AppStream also originally was created for a “safe world” where applications were vetted by the distributors before their metadata was processed. This is increasingly not the case, so it made sense to put at least a few guardrails on the most complex part of the pipeline: The media processing. As part of the change, media processing was split out into a separate worker process. This solved two problems at once: Font handling was isolated in a single-threaded binary – if we wanted to handle fonts in parallel, we could simply spawn more workers. And, being in a separate process, the media processing could now be sandboxed.
As part of the multiprocess changes, Compose also switched from using GdkPixbuf to VIPS for image processing. The latter allows for much more fine-grained control over the image output and encoding, and comes with a lot of well-maintained filters and operations, which made it possible to eliminate a fair chunk of AppStream’s hand-rolled image processing operations. As part of this transition, we unfortunately lost the ability to read XPM images, which dropped about 20-30 applications from the pool at Debian. But in the name of security, this is a sensible choice, especially since most XPM icons were very small and low-resolution, and applications using them could benefit from adding a high-quality PNG icon anyway. With VIPS, we also now restrict the amount of image formats we can load to a sensible set, so extremely niche or unexpected formats will be outright rejected (this includes sane-but-unusual formats for screenshots and icons, such as TIFF images).
The Compose library, with all of these changes, will now just request high-level operations (e.g. “render a font card for this font to a JXL image”) from the worker, and provide it with input data in sealed memfds and output locations as FDs as well. On Linux systems, the worker will use Landlock if available, to block all write access to the filesystem, deny device access and deny TCP and UDP as well. The sandbox can certainly be tightened a fair bit in future, but this was a good and safe start to gain some experience with it without having things break too easily, given the many places Compose is used in (also, Landlock’s API is surprisingly nice to use, so it was easier than I thought to add in this early version).
With all of these changes, the libappstream-compose library is now also officially marked API-stable, so you should be able to rely on it in future to build new things (its API has barely changed in the past, and now with the new media API and defaults change in place, it was time to declare it stable).
I want to see / try this!Currently, the easiest way to have a look at the new data is to check out Debian Unstable. If you have a JXL-enabled browser, you can also see the icons in AppStream Generator’s HTML pages for Debian Sid. If you are using appstream-generator for your distribution, you will also get much more pleasant statistics and HTML pages, as well as fully deterministic media output and a whole bunch of security updates, so, update to its recent 1.0 release.
Please keep in mind that if you switch to JXL, the client tools receiving the image data have to support it. Support varies depending on the Linux distribution, so, test it first and switch the default back to PNG in case you encounter any issues.
What’s next?With so many features and changes landed, the next changes in AppStream will focus on improving what already exists and fixing any issues (there will be more blogposts about the other features 1.2.x delivers!). Testing with the entire Debian archive as data source makes me fairly confident though that there will not be many problems. In the longer term, tightening the media processing sandbox will also be something we might want to do, e.g. by hiding parts of the filesystem tree or filtering syscalls.
For JPEG-XL, one obvious question is “Will you add support for it to the Freedesktop icon-theme specification as supported format alongside PNG, SVG(Z), and XPM?”. For on-disk icon repositories, JXL’s space-savings are less compelling, and it being HDR-capable is also not necessarily a killer feature (PNG can go a long way!). However, JPEG-XL’s ability to immediately decode larger images at reduced resolution without resampling could legitimately be very powerful here, as applications could ship a single large image and quickly decode it at 1/2, 1/4 or 1/8 the size for different purposes in their UI. JPEG-XL also supports spot-color extra channels, which applications could use as masks to recolor raster icons at render time. This could be incredibly nice to color symbolic icons on-the-fly without any SVG and CSS. JXL also provides richer metadata, which might be neat for (license/author) documentation. So, the answer here is: Maybe it makes sense to allow another format, but this will have to be discussed first, as it would force JXL into every toolkit and desktop, which is a much bigger ask than supporting it only in AppStream.
As always, let me know what you think and please report any issues or bugs directly against AppStream or AppStream Generator if you encounter problems that are with the tools, and not with a project’s metadata.
It’s been a long time since I last posted anything here, huh.
Well, a few hours ago I was preparing the release notes for the next release of GNOME Calendar. It is yet to be reviewed, but this is how it reads at the time I write this blog post:
This is a remarkable release for us, as it is one of the biggest releases in the history of the project, and we're excited to share a slightly longer update on it. The first thing many users will notice is how GNOME Calendar will feel snappier now. During the past six months, a lot of work was put in optimizing GNOME Calendar from the inside out. This includes a major change in how it handles events internally, vastly reducing the amount of data transferred between GNOME Calendar and other components of the desktop, and applying many different tricks and strategies to make it render faster. Really, this is probably the most optimized the project has ever been. Another front in which GNOME Calendar has been consistently improving is accessibility and keyboard navigation. During this development cycle, another big batch of improvements on these fronts were merged. You can now navigate between events and days in the Month view using only your keyboard. Notification bubbles are properly read out loud (thanks also to Orca developers for accommodating our use case!). The Week view is now properly styled when the high-contrast setting is enabled. […] On the non-technical side, in the past few months the project received contributions from many new contributors, as well as long time contributors. Our issue tracker continues to be in excellent shape, well triaged, and properly labeled. Our three latest releases were the biggest releases in the history of the project. Thank you all very much for using, developing, documenting, translating, testing, and fixing GNOME Calendar!This release of Calendar has lots to talk about. It is, as mentioned, the biggest release in the history of the project. Not in numbers of line added, or patch count, but certainly in terms of contributor involvement, code reviews, code quality, and features. We’re not just a bunch of bored university students pushing unreviewed patches non-stop to the main branch anymore!
For the next few weeks, I’ll be writing more focused blog posts about the work I’ve done in Calendar this cycle. I’ve focused mostly on performance and reorganizing the internals of the application to be more resilient. It’s not glorious work, but I do love working on optimization problems!
GNOME Calendar will complete 15 years in a few months from now. The project is one of the few lucky projects in GNOME – and, I’d argue, in the free software scene in general – that has such a thriving community of contributors. It’s one of the few GNOME core apps that survived the great purge. It’s a super rare example of a GNOME app with a product manager.
It is also the project that brought me in in GNOME, so pardon me if I get a little emotional when I see the project thriving as it is, and think back of all the good friends that came and went, the hard lessons from maintaining it over over a third of my life, and the prospects for the future.
GNOME Calendar is entirely developed and maintained by volunteers. We have never received any kind of funding, be it corporate, from grants, or other forms of patronage. This gives us freedom from these kinds of influences (mostly to complain about how so many big companies fail to meet the calendaring standards that they themselves helped create), but the reality is that it is really damn hard to pitch for funds for a calendaring application.
Please consider donating to GNOME, or to the individual contributors of your choice. It makes a difference. All the difference.
In 1993, I stood at the top of the World Trade Center feeling like being on top of everything. It was the culmination of my first proper trip west, a stark contrast to a country behind an iron curtain or even a small-town Amherst, New Hampshire, where I had to earn my way into that adventure.
Many people can't wrap their heads around how such enormous buildings could collapse on September 11. What I can't wrap my head around is that they survived the first attack, which happened in February of '93, when I was, entirely oblivious to what had happened on the ground floor a few months earlier, soaking in those panoramic views.
The attack was carried out by a group of radicals led by mastermind Ramzi Yousef. In the underground parking garage of the North Tower, they detonated a yellow Ford van packed with explosives. It's often claimed that the terrorists used Czechoslovak Semtex, but in reality it was a massive, roughly 600-kilogram homemade explosive charge, further reinforced with pressurized hydrogen tanks. Semtex was only a trigger explosive.
The explosion was devastating. The blast tore a 30-meter crater through five floors of underground parking and damaged several support columns, though the main structural frame of the tower held. Though the tower didn't collapse as the terrorists had originally planned, the shockwave destroyed the main electrical wiring and emergency lighting, and smoke rose as high as the 93rd floor. Six people lost their lives and more than a thousand were injured, most from smoke inhalation during the grueling evacuation through dark stairwells. Operations in both towers were completely paralyzed and the complex had to be shut down for nearly a full month. Total damages and subsequent repairs cost roughly half a billion dollars.
Because of the '93 bombing, the Port Authority installed photoluminescent safety markings along the steps, landings, and handrails throughout the towers. When the planes struck eight years later and emergency lights flickered or failed, these glow-in-the-dark strips guided occupants downward. According to National Institute of Standards and Technology, 33% of survivors in the North Tower and 17% in the South Tower directly credited these markings with aiding their escape. The stairs were well-lit by battery-pack emergency lights and photoluminescent guides, and people moved much faster. Survivors who had been in the building during both attacks noted that the 2001 descent took roughly half the time it did in 1993.
I only know about all of this because of the internet — a firehose of news and dangers pouring at me every hour of every day. Could I stand up there today, soaking in those fantastic views, still so oblivious and happy?
I’m looking at something entirely unrelated, but tripped over some search results that made me realise that a lot of people still think getting errors like ACPI Warning: SystemIO range 0x0000000000001828-0x000000000000182F conflicts with OpRegion 0x0000000000001800-0x000000000000187F indicate a firmware bug. This is generally untrue. We need to dive a little into what ACPI is to clarify why.
The Advanced Configuration and Power Interface1 specification defines a whole bunch of stuff, but what’s interesting to us here is the hardware abstraction it performs. While PCs are nominally a well-defined platform that’s really not true at the hardware level once you get beyond a certain level of complexity. When you suspend a system you want to power down the hardware in the correct order, for instance, and knowing what that order is requires you to know details about the specific motherboard design. The approach taken in the embedded world is to just bake that knowledge into the OS in some form, which is how we end up with Devicetree. ACPI takes an alternative approach - rather than provide that information as data that has to be consumed by OS drivers, it distributes it as code.
The ACPI Source Language, or ASL, is a simple language that gets compiled into a bytecode that’s then interpreted by the OS at runtime. One of the features of this language is the ability to define “Operation Regions”, effectively structure definitions that describe access to underlying hardware. Let’s imagine a simple device with two exposed registers. The first is an index register - it describes which internal register we want to access. The second is a data register, where reading it gives us the value of the internal register whose address is currently in the index register, and writing to it modifies that register. An example operation region declaration would look something like
1 2 3 4 5 6 OperationRegion(OPR1, SystemIO, 0x400, 0x2) Field(OPR1, ByteAcc, NoLock, Preserve) { INDX, 8 DATA, 8 }This defines an operation region called “OPR1” at IO port 0x400, 2 bytes long. Inside it are two 8-bit fields, INDX and DATA. These are to be accessed one at a time, do not need the ACPI interpreter to take a global lock when accessing them, and if a subset of the register is modified then the other values should be preserved (irrelevant in this case since the fields are only a byte wide). Now any references to INDX or DATA in this scope will trigger accesses to those registers. So, a method to read the value of register 0x03 would look something like:
1 2 3 4 Method (RD03) { INDX = 0x3 Return (DATA) }ie, set INDX to 3, and then read the value of DATA and return it. But! What if another ACPI method is running at the same time? Let’s say we have one that writes to register 0x05:
1 2 3 4 Method (WR05, 1) { INDX = 0x05 DATA = Arg1 }What happens if RD03 executes while we’re part-way through WR05? INDX might get reset to 0x03, and now WR05 will modify register 0x03 instead of 0x05. Oh no! But we can avoid this - we declare a mutex (Mutex (MUTX, 0x00)), and update our methods to be something like:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 Method (RD03) { Acquire (MUTX, 0xFFFF) INDX = 0x3 Local0 = DATA Release (MUTX) Return (Local0) } Method (WR05, 1) { Acquire (MUTX, 0xFFFF) INDX = 0x05 DATA = Arg1 Release (MUTX) }Each method takes a lock (waiting up to 0xffff milliseconds and then erroring out if it doesn’t), and performs the access. There’s now no chance of a race. Phew!
Now suppose someone writes a Linux driver for this piece of hardware. It accesses the hardware directly, with no knowledge of ACPI. What stops the driver from racing against one of the ACPI access methods? Nothing at all. Oh no! Again! This isn’t hypothetical, by the way - here’s a relatively harmless example, but back in the day we did trip over cases where temperature monitoring chips would be accessed by the firmware and Linux simultaneously and as a result you might end up thinking you’re reading a temperature when you’re actually reading a status flag, resulting in an impossibly high temperature and an immediate thermal shutdown.
In this case, the kernel saves you from this (potentially hardware damaging) outcome by printing a message like ACPI Warning: SystemIO range 0x0000000000000400-0x000000000000401 conflicts with OpRegion 0x0000000000000400-0x0000000000000401 (OPR1), telling you that the kernel has detected that a driver is attempting to allocate IO ports 0x400-0x401, but that there’s an ACPI operation region called OPR1 that is claiming the same addresses. The kernel isn’t in a position to know what type of access the firmware might perform in that region, so assumes that it might be dangerous and blocks the driver from loading.
But all is not lost! The kernel also prints some helpful advice, ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver. And ACPI tables will often actually have a definition that looks like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Device (HDW1) { Name (_HID, "VEND0001") OperationRegion(OPR1, SystemIO, 0x400, 0x2) Field(OPR1, ByteAcc, NoLock, Preserve) { INDX, 8 DATA, 8 } Mutex (MUTX, 0) Method (RD03) { Acquire (MUTX, 0xFFFF) INDX = 0x3 Local0 = DATA Release (MUTX) Return (Local0) } Method (WR05, 1) { Acquire (MUTX, 0xFFFF) INDX = 0x05 DATA = Arg1 Release (MUTX) } }which defines an ACPI device and associated methods. The _HID field defines the device type, and a Linux driver can be written that will be automatically loaded if a device with type VEND0001 is seen. That driver can then call ACPI methods associated with the device and access the resources in a way that matches the firmware’s expectations.
(Interested in writing such a driver? I wrote a guide back in 2009)
The firmware did absolutely nothing wrong here2, but trying to load the native ddriver will generate an error and the internet will tell you that PC firmware developers are incompetent3 and you should pass a kernel argument that overrides this behaviour and it never did them any harm, and it probably won’t do you any harm either but it might and you might never know why your system occasionally wedges or catches fire.
The ACPI spec used to live at acpi.info, but sadly that seems to have vanished some time after UEFI took over stewardship of the spec ↩︎
You might argue that the firmware should simply not do anything at runtime because it is not the firmware’s job to do that, and I do understand that and you can certainly boot with acpi=off if you want to and no ACPI code will be executed at runtime. Let me know how that goes. ↩︎
I’m not going to present an opinion on that here, merely say that this provides no supporting evidence for that assertion ↩︎
Read more of this story at Slashdot.
Read more of this story at Slashdot.
Read more of this story at Slashdot.
Read more of this story at Slashdot.
Read more of this story at Slashdot.
Read more of this story at Slashdot.
Read more of this story at Slashdot.
Read more of this story at Slashdot.
Read more of this story at Slashdot.
Read more of this story at Slashdot.
Read more of this story at Slashdot.
Read more of this story at Slashdot.