Read more of this story at Slashdot.
Read more of this story at Slashdot.
It's probably not your fault.
On a cache miss, there are two things a reverse proxy (which Fastly is to us) can do. It can make the client wait until the proxy itself fetches the requested content and then serve it, with subsequent requests being served from the cache. From a user's perspective, it means staring at "hung" process, and people tend not to be understanding when a program is stuck seemingly doing nothing.
Instead, the proxy can stream the response from the origin, caching it at the end. This makes the client receive the data right away, although it's not without drawbacks.
In a streaming setup like Flathub's, an all-MISS path adds some upstream latency before the first byte, but also limits the download speed to what the slowest link can deliver. As we don't run servers in the same datacenter or on a single backbone network, the hop from Fastly through the caching proxy to the master server incurs a penalty that may affect how quickly the data gets back.
In order to cache files larger than 20MB, Fastly expects customers who use streaming misses to use segmented caching. Anything larger than that gets broken down into smaller chunks. When Fastly wants the data from us, it will add a Range header specifying which bytes we should respond with. Fastly will then serve the request after reconstructing the file from various chunks. Our caching proxies also use the value of the Range header in the caching key to avoid requesting the full file over and over again from the master server as well.
While great for caching, many concurrent range MISSes can turn what would be a sequential file read into scattered, random reads. It wouldn't matter with SSD or NVMe, but as the repository is stored on HDDs, when combined with streaming misses, it can turn cold transfer speed into min(network bottleneck, ZFS random-read bottleneck).
Counterintuitively, you may improve your download speeds by aborting the ongoing Flatpak operation and starting it again. While the initial request was slow, there's a non-zero chance it went through all the caching layers and it will become a cache hit in the meantime.
FlatpakLet's talk Flatpak. When installing or upgrading applications, Flatpak will try to use delta files. A typical delta is an update file that contains only the difference between versions. There are also from-scratch deltas, which effectively are an archive with all files required to install an app from scratch, thus the name.
Flathub generates a single upgrade delta and a from-scratch delta for the latest version. Delta generation is an expensive process in terms of disk reads and writes, but also disk space. Because our ZFS setup isn't exactly the fastest, generating more delta files also affects how quickly we can publish an update. Yes, in theory we could be doing this out of band but we don't. In hindsight, Titanic wasn't unsinkable after all.
What happens if you are not updating often enough? A lot of suffering. Flatpak will download each missing file between the version you are on and the one you want to upgrade to, separately. This is an almost certain cache miss causing even more random seeks on the master server. At this point Flatpak would be better off downloading the from-scratch delta but it can't. The behaviour is controlled by OSTree, which doesn't offer any knobs to affect it. It is the right choice if the goal is to limit the bandwidth used by the client to fetch updates, but an incredibly bad one for anyone on a reliable connection; downloading a single large file is almost always faster than fetching multiple smaller ones.
What do? Some brave soul could fix OSTree to apply a better heuristic on when to use from-scratch deltas for upgrades, or at least make it expose an API that lets Flatpak choose. For the rest of us mere mortals, we can only update regularly or wait patiently for the update to finish.
I often write about how when stuff works well, you take it for granted.
It’s true for technology: when’s the last time you hit a compiler bug in GCC? Once upon a time these were a common thing and you had to choose your C compiler wisely. Yet I haven’t recently seen an article that says “GCC is going great” .
It’s true for people too. When someone does an excellent job maintaining an open source project then, they do occasionally get some gratitude, but — if you do a bad job, it’s amazing how quickly the negative comments pile up in the issue tracker, many of which taking subtle or not-so-subtle digs at the project owners. Maybe we created this situation for ourselves by having a prominent “report issue” button but no corresponding “send flowers to the maintainer” button.
On that note, a hat tip to Carlos Garnacho for all his work on the Localsearch extractor sandbox which recently got a shout out its “extremely strong” design.
(It’s worth noting that Localsearch also stopped using GStreamer to parse media files altogether, which the discussion in that thread missed. We love GStreamer but it isn’t the right tool for metadata scanning. The 3.9 and 3.10 series use libav/ffmpeg instead, but given that US software patent laws make it tricky for USA folk to distribute that, the plan is to move to using MediaInfoLib)
Fairphone 5It’s coming up to two years since I switched to a Fairphone 5. The real proof of this device will be in 2033 when I manage ten years of using the same phone.
Meanwhile, I recently had some issues with it not charging via the USB-C port. I thought it might be a bit tricky to fix, but it really is easy: buy the replacement part (about 20€), take off the back cover, remove a few small screws and switch over the whole USB port + speaker unit.
I hear some fellow Android users complaining about Alphabet/Google’s intrusive AI integration. Apparently the power button is now the AI button? I use the stock Android, and I know vendors have their hands tied somewhat by Alphabet/Google, so its worth noting that disabling the AI integration on the Fairphone 5 is a single config setting.
I’d be interested to know more about the kernel version as it is old as hell. I guess this is a vendor/Android thing, and hopefully most of the many known vulnerabilities in this old version of Linux are mitigated by sandboxing higher up in Android. If you’re a high risk cybercrime target then I would definitely not recommend using the vendor Android OS on this device. (Probably best to avoid Android altogether if this is your situation!)
So its not perfect, but I just wanted to shout out again that there are some good people doing good work here. If only all smartphones were built like this one.
Korg Minilogue XDOne reason I’m not writing much about open source software is that I’m spending a lot of my time outside work making music in various guises, these days mainly as part of soon to be huge Galician disco revival group Muaré. This band needs a website, so in future I don’t have to link you to Instagram, but you know how the world is at the moment. We do at least have a Bandcamp page.
When it comes to music gear, I seem to be a Yamaha guy. It’s amazing actually that the same company that made my trombone also makes excellent digital pianos, and if and when I need a motorbike, Yamaha also sells those.
When it comes to synths though I’ve been really enjoying the Korg Minilogue XD. It’s cheap, built like a tank and its ten years old so there are plenty of second hand models around. It’s not fucking Behringer (please don’t give money to Behringer). It’s simple and sounds great.
But most impressively, it support plugins via a freely available SDK. You can develop your own custom digital oscillators and effects for this thing and deploy them over USB. Of all major pro audio manufacturers, Korg are the only company I know to support this. So even though the hardware is now 10 years old, it can still learn new tricks, and there is an active scene of both free and commercial plugins for the platform. Perhaps the most active commercial outfit is Sinevibes. There is, of course, reddit. The SDK is not truly open source (and few things in pro audio ever are) but it’s free from any licensing fees, and the whole thing is sat here in a Git repo. Pretty good.
If I’d had more time to prepare I might have a video here of some cool Minilogue XD tunes I made. But I guess you’ll have to wait til next month for that. Until then!