You are here

Agreguesi i feed

Montreal's Debian & Stuff - November 2018

Planet Debian - Pre, 02/11/2018 - 5:00pd

November's wet, My socks are too, Away from keyboard; still on the net, Let's fix /usr/bin/$foo.

November can be a hard month in the Northen Hemisphere. It tends to be dark, rainy and cold. Montreal sure has been dark, rainy and cold lately.

That's why you should join us at our next Debian & Stuff later this month. Come by and work on Debian-related stuff - or not! Hanging out and chatting with folks is also perfectly fine. As always, everyone's welcome.

The date hasn't been decided yet, so be sure to fill out this poll before November 10th. This time we'll be hanging out at Koumbit.

What else can I say; if not for the good company, the bad poutine from the potato shack next door or the nice craft beer from the very hipster beer shop a little bit further down the street, you should drop by to keep November from creeping in too far.

Louis-Philippe Véronneau https://veronneau.org/ Louis-Philippe Véronneau

Written too much javascript and python feels less liberal.

Planet Debian - Pre, 02/11/2018 - 1:16pd
Written too much javascript and python feels less liberal. Dict is not an Object.

Junichi Uekawa http://www.netfort.gr.jp/~dancer/diary/201811.html.en Dancer's daily hackings

October 2018 report: LTS, Monkeysphere, Flatpak, Kubernetes, CD archival and calendar project

Planet Debian - Enj, 01/11/2018 - 9:12md
Debian Long Term Support (LTS)

This is my monthly Debian LTS report.

GnuTLS

As discussed last month, one of the options to resolve the pending GnuTLS security issues was to backport the latest 3.3.x series (3.3.30), an update proposed then uploaded as DLA-1560-1. I after a suggestion, I've included an explicit NEWS.Debian item warning people about the upgrade, a warning also included in the advisory itself.

The most important change is probably dropping SSLv3, RC4, HMAC-SHA384 and HMAC-SHA256 from the list of algorithms, which could impact interoperability. Considering how old RC4 and SSLv3 are, however, this should be a welcome change. As for the HMAC changes, those are mandatory to fix the targeted vulnerabilities (CVE-2018-10844, CVE-2018-10845, CVE-2018-10846).

Xen

Xen updates had been idle for a while in LTS, so I bit the bullet and made a first discovery of the pending vulnerabilities. I sent the result to the folks over at Credativ who maintain the 4.4 branch and they came back with a set of proposed updates which I briefly review. Unfortunately, the patches were too deep for me: all I was able to do was to confirm consistency with upstream patches.

I also brought up a discussion regarding the viability of Xen in LTS, especially regarding the "speculative execution" vulnerabilities (XSA-254 and related). My understanding is upstream Xen fixes are not (yet?) complete, but apparently that is incorrect as Peter Dreuw is "condident in the Xen project to provide a solution for these issues". I nevertheless consider, like RedHat that the simpler KVM implementation might provide more adequate protection against those kind of attacks and LTS users should seriously consider switching to KVM for hosing untrusted virtual machines, even if only because that code is actually mainline in the kernel while Xen is unlikely to ever be. It might be, as Dreuw said, simpler to upgrade to stretch than switch virtualization systems...

When all is said and done, however, Linux and KVM are patches in Jessie at the time of writing, while Xen is not (yet).

Enigmail

I spent a significant amount of time working on Enigmail this month again, this time specifically working on reviewing the stretch proposed update to gnupg from Daniel Kahn Gillmor (dkg). I did not publicly share the code review as we were concerned it would block the stable update, which seemed to be in jeopardy when I started working on the issue. Thankfully, the update went through but it means it might impose extra work on leaf packages. Monkeysphere, in particular, might fail to build from source (FTBFS) after the gnupg update lands.

In my tests, however, it seems that packages using GPG can deal with the update correctly. I tested Monkeysphere, Password Store, git-remote-gcrypt and Enigmail, all of which passed a summary smoke test. I have tried to summarize my findings on the mailing list. Basically our options for the LTS update are:

  1. pretend Enigmail works without changing GnuPG, possibly introducing security issues

  2. ship a backport of GnuPG and Enigmail through jessie-sloppy-backports

  3. package OpenPGP.js and backport all the way down to jessie

  4. remove Enigmail from jessie

  5. backport the required GnuPG patchset from stretch to jessie

So far I've taken that last step as my favorite approach...

Firefox / Thunderbird and finding work

... which brings us to the Firefox and Thunderbird updates. I was assuming those were going ahead, but the status of those updates currently seems unclear. This is a symptom of a larger problem in the LTS work organization: some packages can stay "claimed" for a long time without an obvious status update.

We discussed ways of improving on this process and, basically, I will try to be more proactive in taking over packages from others and reaching out to others to see if they need help.

A note on GnuPG

As an aside to the Enigmail / GnuPG review, I was struck by the ... peculiarities in the GnuPG code during my review. I discovered that GnuPG, instead of using the standard resolver, implements its own internal full-stack DNS server, complete with UDP packet parsing. That's 12 000 lines of code right there. There are also abstraction leaks like using "1" and "0" as boolean values inside functions (as opposed to passing an integer and converting as string on output).

A major change in the proposed patchset are changes to the --with-colons batch output, which GnuPG consumers (like GPGME) are supposed to use to interoperate with GnuPG. Having written such a parser myself, I can witness to how difficult parsing those data structures is. Normally, you should always be using GPGME instead of parsing those directly, but unfortunately GPGME does not do everything GPG does: signing operations and keyring management, for example, has long been considered out of scope, so users are force to parse that output.

Long story short, GPG consumers still use --with-colons directly (and that includes Enigmail) because they have to. In this case, critical components were missing from that output (e.g. knowing which key signed which UID) so they were added in the patch. That's what breaks the Monkeysphere test suite, which doesn't expect a specific field to be present. Later versions of the protocol specification have been updated (by dkg) to clarify that might happen, but obviously some have missed the notice, as it came a bit late.

In any case, the review did not make me confident in the software architecture or implementation of the GnuPG program.

autopkgtest testing

As part of our LTS work, we often run tests to make sure everything is in order. Starting with Jessie, we are now seeing packages with autopkgtest enabled, so I started meddling with that program. One of the ideas I was hoping to implement was to unify my virtualization systems. Right now I'm using:

Because sbuild can talk with autopkgtest, and autopkgtest can talk with qemu (which can use KVM images), I figured I could get rid of schroot. Unfortunately, I met a few snags;

  • #911977: how do we correctly guess the VM name in autopkgtest?
  • #911963: qemu build fails with proxy_cmd: parameter not set (fixed and provided a patch)
  • #911979: fails on chown in autopkgtest-qemu backend
  • #911981: qemu server warns about missing CPU features

So I gave up on that approach. But I did get autopkgtest working and documented the process in my quick Debian development guide.

Oh, and I also got sucked down into wiki stylesheet (#864925) after battling with the SystemBuildTools page.

Spamassassin followup

Last month I agreed we could backport the latest upstream version of SpamAssassin (a recurring pattern). After getting the go from the maintainer, I got a test package uploaded but the actual upload will need to wait for the stretch update (#912198) to land to avoid a versioning conflict.

Salt Stack

My first impression of Salt was not exactly impressive. The CVE-2017-7893 issue was rather unclear: first upstream fixed the issue, but reverted the default flag which would enable signature forging after it was discovered this would break compatibility with older clients.

But even worse, the 2014 version of Salt shipped in Jessie did not have master signing in the first place, which means there was simply no way to protect from master impersonation, a worrisome concept. But I assumed this was expected behavior and triaged this away from jessie, and tried to forgot about the horrors I had seen.

phpLDAPadmin with sunweaver

I looked next at the phpLDAPadmin (or PHPLDAPadmin?) vulnerabilities, but could not reproduce the issue using the provided proof of concept. I have also audited the code and it seems pretty clear the code is protected against such an attack, as was explained by another DD in #902186. So I asked Mitre for rejection, and uploaded DLA-1561-1 to fix the other issue (CVE-2017-11107). Meanwhile the original security researcher acknowledged the security issue was a "false positive", although only in a private email.

I almost did a NMU for the package but the security team requested to wait, and marked the package as grave so it gets kicked out of buster instead. I at least submitted the patch, originally provided by Ubuntu folks, upstream.

Smarty3

Finally, I worked on the smart3 package. I confirmed the package in jessie is not vulnerable, because Smarty hadn't yet had the brilliant idea of "optimizing" realpath by rewriting it with new security vulnerabilities. Indeed, the CVE-2018-13982 proof of content and CVE-2018-16831 proof of content both fail in jessie.

I have tried to audit the patch shipped with stretch to make sure it fixed the security issue in question (without introducing new ones of course) abandoned parsing the stretch patch because this regex gave me a headache:

'%^(?<root>(?:<span class="createlink"><a href="/ikiwiki.cgi?do=create&amp;from=blog%2F2018-11-01-report&amp;page=%3Aalpha%3A" rel="nofollow">?</a>:alpha:</span>:[\\\\]|/|[\\\\]{2}<span class="createlink"><a href="/ikiwiki.cgi?do=create&amp;from=blog%2F2018-11-01-report&amp;page=%3Aalpha%3A" rel="nofollow">?</a>:alpha:</span>+|<span class="createlink"><a href="/ikiwiki.cgi?do=create&amp;from=blog%2F2018-11-01-report&amp;page=%3Aprint%3A" rel="nofollow">?</a>:print:</span>{2,}:[/]{2}|[\\\\])?)(?<path>(?:<span class="createlink"><a href="/ikiwiki.cgi?do=create&amp;from=blog%2F2018-11-01-report&amp;page=%3Aprint%3A" rel="nofollow">?</a>:print:</span>*))$%u', "who is supporting our users?"

I finally participated in a discussion regarding concerns about support of cloud images for LTS releases. I proposed that, like other parts of Debian, responsibility of those images would shift to the LTS team when official support is complete. Cloud images fall in that weird space (ie. "Installing Debian") which is not traditionally covered by the LTS team.

Hopefully that will become the policy, but only time will tell how this will play out.

Other free software work irssi sandbox

I had been uncomfortable running irssi as my main user on my server for a while. It's a constantly running network server, sometimes connecting to shady servers too. So it made sense to run this as a separate user and, while I'm there, start it automatically on boot.

I created the following file in /etc/systemd/system/irssi@.service, based on this gist:

[Unit] Description=IRC screen session After=network.target [Service] Type=forking User=%i ExecStart=/usr/bin/screen -dmS irssi irssi ExecStop=/usr/bin/screen -S irssi -X stuff '/quit\n' NoNewPrivileges=true [Install] WantedBy=multi-user.target

A whole apparmor/selinux/systemd profile could be written for irssi of course, but I figured I would start with NoNewPrivileges. Unfortunately, that line breaks screen, which is sgid utmp which is some sort of "new privilege". So I'm running this as a vanilla service. To enable, simply enable the service with the right username, previously created with adduser:

systemctl enable irssi@foo.service systemctl start irssi@foo.service

Then I join the session by logging in as the foo user, which can be configured in .ssh/config as a convenience host:

Host irc.anarc.at Hostname shell.anarc.at User foo IdentityFile ~/.ssh/id_ed25519_irc # using command= in authorized_keys until we're all on buster #RemoteCommand screen -x RequestTTY force

Then the ssh irc.anarc.at command rejoins the screen session.

Monkeysphere revival

Monkeysphere was in bad shape in Debian buster. The bit rotten test suite was failing and the package was about to be removed from the next Debian release. I filed and worked on many critical bugs (Debian bug #909700, Debian bug #908228, Debian bug #902367, Debian bug #902320, Debian bug #902318, Debian bug #899060, Debian bug #883015) but the final fix came from another user. I was also welcome on the Debian packaging team which should allow me to make a new release next time we have similar issues, which was a blocker this time round.

Unfortunately, I had to abandon the Monkeysphere FreeBSD port. I had simply forgotten about that commitment and, since I do not run FreeBSD anywhere anymore, it made little sense to keep on doing so, especially since most of the recent updates were done by others anyways.

Calendar project

I've been working on a photography project since the beginning of the year. Each month, I pick the best picture out of my various shoots and will collect those in a 2019 calendar. I documented my work in the photo page, but most of my work in October was around finding a proper tool to layout the calendar itself. I settled on wallcalendar, a beautiful LaTeX template, because the author was very responsive to my feature request.

I also figured out which events to include in the calendar and a way to generate moon phases (now part of the undertime package) for the local timezone. I still have to figure out which other astronomical events to include. I had no response from the local Planetarium but (as always) good feedback from NASA folks which pointed me at useful resources to top up the calendar.

Kubernetes

I got deeper into Kubernetes work by helping friends setup a cluster and share knowledge on how to setup and manage the platforms. This led me to fix a bug in Kubespray, the install / upgrade tool we're using to manage Kubernetes. To get the pull request accepted, I had to go through the insanely byzantine CLA process of the CNCF, which was incredibly frustrating, especially since it was basically a one-line change. I also provided a code review of the Nextcloud helm chart and reviewed the python-hvac ITP, one of the dependencies of Kubespray.

As I get more familiar with Kubernetes, it does seem like it can solve real problems especially for shared hosting providers. I do still feel it's overly complex and over-engineered. It's difficult to learn and moving too fast, but Docker and containers are such a convenient way to standardize shipping applications that it's hard to deny this new trend does solve a problem that we have to fix right now.

CD archival

As part of my work on archiving my CD collection, I contributed three pull requests to fix issues I was having with the project, mostly regarding corner cases but also improvements on the Dockerfile. At my suggestion, upstream also enabled automatic builds for the Docker image which should make it easier to install and deploy.

I still wish to write an article on this, to continue my series on archives, which could happen in November if I can find the time...

Flatpak conversion

After reading a convincing benchmark I decided to give Flatpak another try and ended up converting all my Snap packages to Flatpak.

Flatpak has many advantages:

  • it's decentralized: like APT or F-Droid repositories, anyone can host their own (there is only one Snap repository, managed by Canonical)

  • it's faster: the above benchmarks hinted at this, but I could also confirm Signal starts and runs faster under Flatpak than Snap

  • it's standardizing: many of the work Flatpak is doing to make sense of how to containerize desktop applications is being standardized (and even adopted by Snap)

Much of this was spurred by the breakage of Zotero in Debian (Debian bug #864827) due to the Firefox upgrade. I made a wiki page to tell our users how to install Zotero in Debian considering Zotero might take a while to be packaged back in Debian (Debian bug #871502).

Debian work

Without my LTS hat, I worked on the following packages:

Other work

Usual miscellaneous:

Antoine Beaupré https://anarc.at/tag/debian-planet/ pages tagged debian-planet

Linux Audio Miniconf 2018 report

Planet Debian - Enj, 01/11/2018 - 6:12md

The audio miniconference was held on the 21st in the offices of Cirrus Logic in Edinburgh with 15 attendees from across the industry including userspace and kernel developers, with people from several OS vendors and a range of silicon companies.

Community

We started off with a discussion of community governance lead by Takashi Iwai. We decided that for the ALSA-hosted projects we’ll follow the kernel and adopt the modified version of the contributor covenant that they have adopted, Sound Open Firmware already has a code. We also talked a bit about moving to use some modern hosted git with web based reviews. While this is not feasible for the kernel components we decided to look at doing this for the userspace components, Takashi will start a discussion on alsa-devel. Speaking of the lists Vinod Koul also volunteered to work with the Linux Foundation admin team to get them integrated with lore.kernel.org.

Liam Girdwood presenting virtualization (photo: Arun Raghavan) Kernel

Liam Girdwood then kicked off the first technical discussion of the day, covering virtualization. Intel have a new hypervisor called ACRN which they are using as part of a solution to expose individual PCMs from their DSPs to virtual clients, they have a virtio specification for control. There were a number of concerns about the current solution being rather specific to both the hardware and use case they are looking at, we need to review that this can work on architectures that aren’t cache coherent or systems where rather than exposing a DSP the host system is using a sound server.

We then moved on to AVB, several vendors have hardware implementations already but it seems clear that these have been built by teams who are not familiar with audio hardware, hopefully this will improve in future but for now there are some regrettable real time requirements. Sakamoto-san suggested looking at FireWire which has some similar things going on with timestamps being associated with the audio stream.

For SoundWire, basic functionality for x86 systems is now 90% there – we still need support for multiple CPU DAIs in the ASoC core (which is in review on the lists) and the Intel DSP drivers need to plumb in the code to instantiate the drivers.

We also covered testing, there may be some progress here this year as Intel have a new hypervisor called ACRN and some out of tree QEMU models for some of their newer systems both of which will help with the perennial problem that we need hardware for a lot of the testing we want to do. We also reviewed the status with some other recurring issues, including PCM granularity and timestamping, for PCM granularity Takashi Iwai will make some proposals on the list and for timestamping Intel will make sure that the rest of their driver changes for this are upstreamed. For dimen we agreed that Sakamoto-san’s work is pretty much done and we just need some comments in the header, and that his control refactoring was a good idea. There was discussion of user defined card elements, there were no concerns with raising the number of user defined elements that can be created but some fixes needed for cleanup of user defined card elements when applications close. The compressed audio userspace is also getting some development with the focus on making things easier to test, integrating with ffmpeg to give something that’s easier for user to work with.

Charles Keepax covered his work on rate domains (which we decided should really be much more generic than just covering sample rates), he’d posted some patches on the list earlier in the week and gave a short presentation about his plans which sparked quite a bit of discussion. His ideas are very much in line with what we’ve discussed before in this area but there’s still some debate as to how we configure the domains – the userspace interface is of course still there but how we determine which settings to use once we pass through something that can do conversions is less clear. The two main options are that the converters can expose configuration to userspace or that we can set constraints on other widgets in the card graph and then configure converters automatically when joining domains. No firm conclusion was reached, and since substantial implementation will be required it is not yet clear what will prove most sensible in practical systems.

Userspace

Sakamoto-san also introduced some discussion of new language bindings. He has been working on a new library designed for use with GObject introspection which people were very interested in, especially with the discussion of testing – having something like this would simplify a lot of the boilerplate that is involved in using the C API and allow people to work in a wider variety of languages without needing to define specific bindings or use the individual language’s C adaptations. People also mentioned the Rust bindings that David Henningsson had been working on, they were particularly interesting for the ChromeOS team as they have been adopting Rust in their userspace.

We talked a bit about higher level userspace software too. PulseAudio development has been relatively quiet recently, Arun talked briefly about his work on native compressed audio support and we discussed if PulseAudio would be able to take advantage of the new timestamping features added by Pierre-Louis Bossart. There’s also the new PipeWire sound server stack, this is a new stack which was originally written for video but now also has some audio support. The goal is to address architectural limitations in the existing JACK and PulseAudio stacks, offering the ability to achieve low latencies in a stack which is more usable for general purpose applications than JACK is.

DSPs

Discussions of DSP related issues were dominated by Sound Open Firmware which is continuing to progress well and now has some adoption outside Intel. Liam gave an overview of the status there and polled interest from the DSP vendors who were present. We talked about how to manage additions to the topology ABI for new Sound Open Firmware features including support for loading and unloading pieces of the DSP topology separately when dynamically adding to the DSP graph at runtime, making things far more flexible. The issues around downloading coefficient data were also covered, the discussion converged on the idea of adding something to hwdep and extending alsa-lib and tinyalsa to make this appear integrated with the standard control API. This isn’t ideal but it seems unlikely that anything will be. Techniques for handling long sequences of RPC calls to DSPs efficiently were also discussed, the conclusion was that the simplest thing was just to send commands asynchronously and then roll everything back if there are any errors.

Summary

Thanks again to all the attendees for their time and contributions and to Cirrus Logic for their generosity in hosting this in their Edinburgh office. It was really exciting to see all the active development that’s going on these days, it’ll be great to see some of that bear fruit over the next year!

Group photo broonie https://blog.sirena.org.uk Technicalities

Carlos Soriano: GNOME Internships interns has been elected

Planet GNOME - Enj, 01/11/2018 - 5:37md

Hello all,

GNOME Internships projects and interns has been elected!

We had have strong applicants and quite a big amount of applications. If you are not the elected don’t be discouraged, it wasn’t an easy choice.

This round we have to congratulate Ludovico de Nittis who will work in the “USB Protection” project with his mentor Tobias Mueller. Congrats Ludovico!

The project goal is to increase the robustness against attacks via malicious USB devices. Certainly a challenging goal! You can read extensive information in the project wiki linked above, it’s definitely quite interesting.

This round of internships is fund by the privacy and security campaign we did at GNOME a few years ago, and we are happy we can put them into good use to improve the security and privacy of GNOME users and donors.

Deeply thanks to all the donors! Withouth you this initiative wouldn’t have been possible. Also thanks to the mentors and applicants.

More on the admin side, it has been a little bumpy since its the first time doing these internships. And well, this last week I have been a bit worried by other stuff you already probably know about :)

If you have any question or feedback, don’t hesitate to contact me on IRC as csoriano or send an email to internships-admin@gnome.org

My Work on Debian LTS (October 2018)

Planet Debian - Enj, 01/11/2018 - 4:13md

after some nice family vacation in Tuscany, I did four hours of work on the Debian LTS project as a paid contributor at the end of this month. Thanks to all LTS sponsors for making this possible.

I move over a backlog of 4h from October to November (so I will work 12h on Debian LTS in November 2018).

Furthermore, I have signed up for Debian ELTS work with another 4h (as a start, more availability planned for upcoming months).

This is my list of work done in October 2018:

  • Upload of poppler (DLA 1562-1 [1]), fixing 4 CVEs
  • Discuss my research on CVE-2018-12689 in phpldapadmin from August 2018 with Antoine Beaupré who identified the published exploit as 'false positive' (for details, see his monthly LTS report for Octobre 2018).

light+love
Mike

References

[1] https://lists.debian.org/debian-lts-announce/2018/10/msg00024.html

sunweaver http://sunweavers.net/blog/blog/1 sunweaver's blog

FLOSS Activities October 2018

Planet Debian - Enj, 01/11/2018 - 11:45pd
Changes Issues Review Administration
  • Debian: investigate MSA issue, investigate Fastly outage, investigate buildd issue, forwarded mail bounce info
  • Debian wiki: unblacklist IP range, clean up stray tmp files, whitelist email addresses, ask on IRC about accounts with bouncing email, ping accounts with bouncing email, disable accounts with bouncing email
  • Debian derivatives census: merge patches, deploy changes, clean up cruft, delete giant source packages
Communication Sponsors

All work was done on a volunteer basis.

Paul Wise http://bonedaddy.net/pabs3/log/ Log

Time for an official MIME type for patches?

Planet Debian - Enj, 01/11/2018 - 8:15pd

As part of my involvement in the Nikita archive API project, I've been importing a fairly large lump of emails into a test instance of the archive to see how well this would go. I picked a subset of my notmuch email database, all public emails sent to me via @lists.debian.org, giving me a set of around 216 000 emails to import. In the process, I had a look at the various attachments included in these emails, to figure out what to do with attachments, and noticed that one of the most common attachment formats do not have an official MIME type registered with IANA/IETF. The output from diff, ie the input for patch, is on the top 10 list of formats included in these emails. At the moment people seem to use either text/x-patch or text/x-diff, but neither is officially registered. It would be better if one official MIME type were registered and used everywhere.

To try to get one official MIME type for these files, I've brought up the topic on the media-types mailing list. If you are interested in discussion which MIME type to use as the official for patch files, or involved in making software using a MIME type for patches, perhaps you would like to join the discussion?

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Petter Reinholdtsen http://people.skolelinux.org/pere/blog/ Petter Reinholdtsen - Entries tagged english

Montreal's Debian &amp; Stuff - November 2018

Planet Debian - Enj, 01/11/2018 - 5:00pd

November's wet, My socks are too, Away from keyboard; still on the net, Let's fix /usr/bin/$foo.

November can be a hard month in the Northen Hemisphere. It tends to be dark, rainy and cold. Montreal sure has been dark, rainy and cold lately.

That's why you should join us at our next Debian & Stuff later this month. Come by and work on Debian-related stuff - or not! Hanging out and chatting with folks is also perfectly fine. As always, everyone's welcome.

The date hasn't been decided yet, so be sure to fill out this poll before November 10th. This time we'll be hanging out at Koumbit.

What else can I say; if not for the good company, the bad poutine from the potato shack next door or the nice craft beer from the very hipster beer shop a little bit further down the street, you should drop by to keep November from creeping in too far.

Louis-Philippe Véronneau https://veronneau.org/ Louis-Philippe Véronneau

Sean Davis: Xubuntu Development Update November 2018

Planet Ubuntu - Enj, 01/11/2018 - 4:28pd

Aaaaaaaaaaaand, we’re back! After skipping last month’s development update, there’s a lot of new developments to unpack for the previous 2 months. Let’s get right to it.

Xubuntu 18.10 “Cosmic Cuttlefish”

We wrapped up development on Xubuntu 18.10 throughout September and October, landing the following changes in the last month and a half of work.

This release includes 6 new GTK+ 3 Xfce components, giving users a snapshot of the Xfce 4.14 development. More information about the release can be found in the release notes.

Upcoming Fixes

Since the 18.10 release, we’ve identified fixes for two of our documented bugs. We’ll be pushing these fixes to our users via the stable release updates.

  • Panel: Window buttons are not clickable at the top pixel of the screen (LP: #1795135)
    • Resolution: Export GDK_CORE_DEVICE_EVENTS = 1 (Xfce Git)
  • Settings Manager: Mouse fails to scroll embedded panels (LP: #1653448)
    • Resolution: Export GDK_CORE_DEVICE_EVENTS = 1 (Xfce Git)
Xfce September New Releases October New Releases 4.14 Roadmap Updates

The Xfce development team has worked on tidying up the Xfce 4.14 roadmap over the last few days. Statuses have been updated, pending work has been moved to the top of each section, and completion percents have been adjusted to better reflect each project’s progress. With these updates, we can now see that…

Xfce 4.14 is now approximately 83% complete.

Of course, we need all the help we can get to get this milestone out the door. Check out the Xfce Contribute page to find out how you can help.

What’s Next?

With the 18.10 release now behind us, and the 19.04 cycle starting today, it’s time to get back to work! No release goals have been determined yet, so stay tuned to the Xubuntu Development mailing list for updates about Xubuntu 19.04 “Disco Dingo” development.

Review: In Pursuit of the Traveling Salesman

Planet Debian - Enj, 01/11/2018 - 4:25pd

Review: In Pursuit of the Traveling Salesman, by William J. Cook

Publisher: Princeton University Copyright: 2012 ISBN: 0-691-15270-5 Format: Kindle Pages: 272

In Pursuit of the Traveling Salesman is a book-length examination of the traveling salesman problem (TSP) in computer science, written by one of the foremost mathematicians working on solutions to the TSP. Cook is Professor of Applied Mathematics and Statistics at Johns Hopkins University and is one of the authors of the Concorde TSP Solver.

First, a brief summary of the TSP for readers without a CS background. While there are numerous variations, the traditional problem is this: given as input a list of coordinates on a two-dimensional map representing cities, construct a minimum-length path that visits each city exactly once and then returns to the starting city. It's famous in computer science in part because it's easy to explain and visualize but still NP-hard, which means that not only do we not know of a way to exactly solve this problem in a reasonable amount of time for large numbers of cities, but also that a polynomial-time solution to the TSP would provide a solution to a vast number of other problems. (For those familiar with computability theory, the classic TSP is not NP-complete because it's not a decision problem and because of some issues with Euclidean distances, but when stated as a graph problem and converted into a decision problem by, for example, instead asking if there is a solution with length less than n, it is NP-complete.)

This is one of those books where the quality of the book may not matter as much as its simple existence. If you're curious about the details of the traveling salesman problem specifically, but don't want to read a lot of mathematics and computer science papers, algorithm textbooks, or books on graph theory, this book is one of your few options. Thankfully, it's also fairly well-written. Cook provides a history of the problem, a set of motivating problems (the TSP doesn't come up as much and isn't as critical as some NP-complete problems, but optimal tours are still more common than one might think), and even a one-chapter tour of the TSP in art. The bulk of the book, though, is devoted to approximation methods, presented in roughly chronological order of development.

Given that the TSP is NP-hard, we obviously don't know a good exact solution, but I admit I was a bit disappointed that Cook spent only one chapter exploring the exact solutions and explaining to the reader what makes the problem difficult. Late in the book, he does describe the Held-Karp dynamic programming algorithm that gets the work required for an exact solution down to exponential in n, provides a basic introduction to complexity theory, and explains that the TSP is NP-complete by reduction from the Hamiltonian path problem, but doesn't show the reduction of 3SAT to Hamiltonian paths. Since my personal interest ran a bit more towards theory and less towards practical approximations, I would have appreciated a bit more discussion of the underlying structure of the problem and why it's algorithmically hard. (I did appreciate the explanation of why it's not clear whether the general Euclidean TSP is even in NP due to problems with square roots, though.)

That said, I suppose there isn't as much to talk about in exact solutions (the best one we know dates to 1962) and much more to talk about in approximations, which is where Cook has personally spent his time. That's the topic of most of this book, and includes a solid introduction to the basic concept of linear programming (a better one than I ever got in school) and some of its other applications, as well as other techniques (cutting planes, branch-and-bound, and others). The math gets a bit thick here, and Cook skips over a lot of the details to try to keep the book suitable for a general audience, so I can't say I followed all of it, but it certainly satisfied my curiosity about practical approaches to the TSP. (It also made me want to read more about linear programming.)

If you're looking for a book like this, you probably know that already, and I can reassure you that it delivers what it promises and is well-written and approachable. If you aren't already curious about a brief history of practical algorithms for one specific problem, I don't think this book is sufficiently compelling to worth seeking out anyway. This is not a general popularization of interesting algorithms (see Algorithms to Live By if you're looking for that), or (despite Cook's efforts) particularly approachable if this is your first deep look at computer algorithms. It's a niche book that delivers on its promise, but probably won't convince you the topic is interesting if you don't see the appeal.

Rating: 7 out of 10

Russ Allbery https://www.eyrie.org/~eagle/ Eagle's Path

Debian LTS work, October 2018

Planet Debian - Mër, 31/10/2018 - 11:26md

I was assigned 15 hours of work by Freexian's Debian LTS initiative and carried over 4 hours from September. I worked all 19 hours.

I released security updates for the linux (DLA 1529-1) and linux-4.9 (DLA 1531-1) packages. I prepared and released another stable update for Linux 3.16 (3.16.60), but have not yet included this in a Debian upload. I also released a security update for libssh (DLA 1548-1).

Ben Hutchings https://www.decadent.org.uk/ben/blog Better living through software

RHL'19 St-Cergue, Switzerland, 25-27 January 2019

Planet Debian - Mër, 31/10/2018 - 10:06md

(translated from original French version)

The Rencontres Hivernales du Libre (RHL) (Winter Meeting of Freedom) takes place 25-27 January 2019 at St-Cergue.

Swisslinux.org invites the free software community to come and share workshops, great meals and good times.

This year, we celebrate the 5th edition with the theme «Exploit».

Please think creatively and submit proposals exploring this theme: lectures, workshops, performances and other activities are all welcome.

RHL'19 is situated directly at the base of some family-friendly ski pistes suitable for beginners and more adventurous skiers. It is also a great location for alpine walking trails.

Why, who?

RHL'19 brings together the forces of freedom in the Leman basin, Romandy, neighbouring France and further afield (there is an excellent train connection from Geneva airport). Hackers and activists come together to share a relaxing weekend and discover new things with free technology and software.

If you have a project to present (in 5 minutes, an hour or another format) or activities to share with other geeks, please send an email to rhl-team@lists.swisslinux.org or submit it through the form.

If you have any specific venue requirements please contact the team.

You can find detailed information on the event web site.

Please ask if you need help finding accommodation or any other advice planning your trip to the region.

Daniel.Pocock https://danielpocock.com/tags/debian DanielPocock.com - debian

Daniel Pocock: RHL'19 St-Cergue, Switzerland, 25-27 January 2019

Planet Ubuntu - Mër, 31/10/2018 - 10:06md

(translated from original French version)

The Rencontres Hivernales du Libre (RHL) (Winter Meeting of Freedom) takes place 25-27 January 2019 at St-Cergue.

Swisslinux.org invites the free software community to come and share workshops, great meals and good times.

This year, we celebrate the 5th edition with the theme «Exploit».

Please think creatively and submit proposals exploring this theme: lectures, workshops, performances and other activities are all welcome.

RHL'19 is situated directly at the base of some family-friendly ski pistes suitable for beginners and more adventurous skiers. It is also a great location for alpine walking trails.

Why, who?

RHL'19 brings together the forces of freedom in the Leman basin, Romandy, neighbouring France and further afield (there is an excellent train connection from Geneva airport). Hackers and activists come together to share a relaxing weekend and discover new things with free technology and software.

If you have a project to present (in 5 minutes, an hour or another format) or activities to share with other geeks, please send an email to rhl-team@lists.swisslinux.org or submit it through the form.

If you have any specific venue requirements please contact the team.

You can find detailed information on the event web site.

Please ask if you need help finding accommodation or any other advice planning your trip to the region.

Arun Raghavan: Update from the PipeWire hackfest

Planet GNOME - Mër, 31/10/2018 - 5:06md

As the third and final day of the PipeWire hackfest draws to a close, I thought I’d summarise some of my thoughts on the goings-on and the future.

Thanks

Before I get into the details, I want to send out a big thank you to:

  • Christian Schaller for all the hard work of organising the event and Wim Taymans for the work on PipeWire so far (and in the future)
  • The GNOME Foundation, for sponsoring the event as a whole
  • Qualcomm, who are funding my presence at the event
  • Collabora, for sponsoring dinner on Monday
  • Everybody who attended and participate for their time and thoughtful comments
Background

For those of you who are not familiar with it, PipeWire (previously Pinos, previously PulseVideo) was Wim’s effort at providing secure, multi-program access to video devices (like webcams, or the desktop for screen capture). As he went down that rabbit hole, he wrote SPA, a lightweight general-purpose framework for representing a streaming graph, and this led to the idea of expanding the project to include support for low latency audio.

The Linux userspace audio story has, for the longest time, consisted of two top-level components: PulseAudio which handles consumer audio (power efficiency, wide range of arbitrary hardware), and JACK which deals with pro audio (low latency, high performance). Consolidating this into a good out-of-the-box experience for all use-cases has been a long-standing goal for myself and others in the community that I have spoken to.

An Opportunity

From a PulseAudio perspective, it has been hard to achieve the 1-to-few millisecond latency numbers that would be absolutely necessary for professional audio use-cases. A lot of work has gone into improving this situation, most recently with David Henningsson’s shared-ringbuffer channels that made client/server communication more efficient.

At the same time, as application sandboxing frameworks such as Flatpak have added security requirements of us that were not accounted for when PulseAudio was written. Examples including choosing which devices an application has access to (or can even know of) or which applications can act as control entities (set routing etc., enable/disable devices). Some work has gone into this — Ahmed Darwish did some key work to get memfd support in PulseAudio, and Wim has prototyped an access-control mechanism module to enable a Flatpak portal for sound.

All this said, there are still fundamental limitations in architectural decisions in PulseAudio that would require significant plumbing to address. With Wim’s work on PipeWire and his extensive background with GStreamer and PulseAudio itself, I think we have an opportunity to revisit some of those decisions with the benefit of a decade’s worth of learning deploying PulseAudio in various domains starting from desktops/laptops to phones, cars, robots, home audio, telephony systems and a lot more.

Key Ideas

There are some core ideas of PipeWire that I am quite excited about.

The first of these is the graph. Like JACK, the entities that participate in the data flow are represented by PipeWire as nodes in a graph, and routing between nodes is very flexible — you can route applications to playback devices and capture devices to applications, but you can also route applications to other applications, and this is notionally the same thing.

The second idea is a bit more radical — PipeWire itself only “runs” the graph. The actual connections between nodes are created and managed by a “session manager”. This allows us to completely separate the data flow from policy, which means we could write completely separate policy for desktop use cases vs. specific embedded use cases. I’m particularly excited to see this be scriptable in a higher-level language, which is something Bastien has already started work on!

A powerful idea in PulseAudio was rewinding — the ability to send out huge buffers to the device, but the flexibility to rewind that data when things changed (a new stream got added, or the stream moved, or the volume changed). While this is great for power saving, it is a significant amount of complexity in the code. In addition, with some filters in the data path, rewinding can break the algorithm by introducing non-linearity. PipeWire doesn’t support rewinds, and we will need to find a good way to manage latencies to account for low power use cases. One example is that we could have the session manager bump up the device latency when we know latency doesn’t matter (Android does this when the screen is off).

There are a bunch of other things that are in the process of being fleshed out, like being able to represent the hardware as a graph as well, to have a clearer idea of what is going on within a node. More updates as these things are more concrete.

The Way Forward

There is a good summary by Christian about our discussion about what is missing and how we can go about trying to make a smooth transition for PulseAudio users. There is, of course, a lot to do, and my ideal outcome is that we one day flip a switch and nobody knows that we have done so.

In practice, we’ll need to figure out how to make this transition seamless for most people, while folks with custom setup will need to be given a long runway and clear documentation to know what to do. It’s way to early to talk about this in more specifics, however.

Configuration

One key thing that PulseAudio does right (I know there are people who disagree!) is having a custom configuration that automagically works on a lot of Intel HDA-based systems. We’ve been wondering how to deal with this in PipeWire, and the path we think makes sense is to transition to ALSA UCM configuration. This is not as flexible as we need it to be, but I’d like to extend it for that purpose if possible. This would ideally also help consolidate the various methods of configuration being used by the various Linux userspaces.

To that end, I’ve started trying to get a UCM setup on my desktop that PulseAudio can use, and be functionally equivalent to what we do with our existing configuration. There are missing bits and bobs, and I’m currently focusing on the ones related to hardware volume control. I’ll write about this in the future as the effort expands out to other hardware.

Onwards and upwards

The transition to PipeWire is unlikely to be quick or completely-painless or free of contention. For those who are worried about the future, know that any switch is still a long way away. In the mean time, however, constructive feedback and comments are welcome.

Free software activities in October 2018

Planet Debian - Mër, 31/10/2018 - 4:47md

Here is my monthly update covering what I have been doing in the free software world during October 2018 (previous month):

We intend to maintain changes to these modules under their original open source licenses and applying only free and open fixes and updates. You can find out more at goodformcode.com.

  • My activities as the current Debian Project Leader are covered in my monthly "Bits from the DPL" email to the debian-devel-announce mailing list.

  • I created Github-esque ribbons to display on Salsa-hosted websites. (Salsa being the collaborative development server for Debian and is the replacement for the now-deprecated Alioth service.)

  • Started a highly work-in-progress "Debbugs Enhancement Suite" Chrome browser extension to enhance various parts of the bugs.debian.org web interface.

  • Even more hacking on the Lintian static analysis tool for Debian packages:

    • New features:

      • Warn about packages that use PIUPARTS_* in maintainer scripts. (#912040)
      • Check for packages that parse /etc/passwd in maintainer scripts. (#911157)
      • Emit a warning for packages that do not specify Build-Depends-Package in symbol files. (#911451)
      • Check for non-Python files in top-level Python module directories. [...]
      • Check packages missing versioned dependencies on init-system-helpers. (#910594)
      • Detect calls to update-inetd(1) that use --group without --add, etc. (#909511)
      • Check for packages that encode a Python version number in their source package name. [...]
    • Bug fixes:

    • Misc:

      • Also show the maintainer name on the tag-specific reporting HTML. [...]
      • Tidy a number of references regarding the debhelper-compat virtual package. [...]
Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws almost all software is distributed pre-compiled to end users.

The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

This month:

  • I attended the Tandon School of Engineering (part of New York University) to speak and work with students from the Application Security course on the topic of reproducible builds.

  • Wrote and forwarded patch for Fontconfig to ensure the cache filenames are determinstic. [...]

  • I sent two previously-authored patches for GNU mtools to ensure the Debian Installer images could become reproducible. (1 & 2)

  • Submitted 11 Debian patches to fix reproducibility issues in fast5, libhandy, lmfit-py, mp3fs, opari2, pjproject, radon, sword, syndie, wit & zsh-antigen. I also submitted an upstream pull request for python-changelog.

  • Made a large number of changes to our website, including adding step-by-step instructions and screenshots on how to signup to our project on Salsa and migrating the TimestampsProposal page on the Debian Wiki to our website.

  • Fixed an issue in disorderfs — our FUSE-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues — where touch -m and touch -a were not working as expected (#911281). In addition, ensured that failing an XFail test should in-itself be a failure [...].

  • Made the following changes to diffoscope, our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues to:

    • Add support for comparing OCaml files via ocamlobjinfo. (#910542)

    • Add support for comparing PDF metadata using PyPDF2. (#911446)

    • Support gnumeric 1.12.43. [...]

    • Use str.startswith(...) over str.index(...) == 0 in the Macho comparator to prevent tracebacks if text cannot be found on the line. (#910540).

    • Add note on how to regenerate debian/tests/control.in and regenerate debian/tests/control with no material changes to add the regeneration comment itself. (1, 2)

    • Prevent test failures when running under stretch-backports by checking the OCaml version number. (#911846)

    • I also added a Salsa ribbon to the diffoscope.org website. [...]

  • Categorised a huge number of packages and issues in the Reproducible Builds "notes" repository and kept isdebianreproducibleyet.com up to date [...].

  • Worked on publishing our weekly reports. (#180, #181, #182 & #183)

  • Lastly, I fixed an issue in our Jenkins-based testing framework that powers tests.reproducible-builds.org to suppress some warnings from the cryptsetup initramfs hook which were causing some builds to be marked as "unstable". [...]


Debian


Debian bugs & patches filed
  • debbugs: Correct "favicon" location in <link/> HTML header. (#912186)

  • ikiwiki: "po" plugin can insert raw file contents with [[!inline]] directives. (#911356)

  • kitty: Please update homepage. (#911848)

  • pipenv: Bundles a large number of third-party libraries. (#910107)

  • mailman: Please include List-Id header on confirmation mails. (#910378)

  • fswatch: Clarify Files-Excluded entries. (#910330)

  • fuse3: Please obey nocheck build profile. (#910029)

  • gau2grid: Please add a non-boilerplate long description. (#911532)

  • hiredis: Please backport to stretch-backports. (#911732)

  • Please remove unnecessary overrides in fuse3 (#910030), puppet-module-barbican (#910374), python-oslo.vmware (#910011) & python3-antlr3(#910012)

  • python3-pypdf2: Python 3.x package ships non-functional Python 2.x examples. (#911649)

  • mtools: New upstream release. (#912285)

I also a filed requests with the stable release managers to update lastpass-cli (#911767) and python-django (#910821).


Debian LTS

This month I have worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project.

  • Multiple "frontdesk" shifts, triaging upstream CVEs, liasing with the Security Team, etc.

  • Issued DLA 1528-1 to prevent a denial-of-service (DoS) vulnerability in strongswan, a virtual private network (VPN) client and server where verification of an RSA signature with a very short public key caused an integer underflow in a length check that resulted in a heap buffer overflow.

  • Issued DLA 1547-1 for the Apache PDFBox library to fix a potential DoS issue where a malicious file could have triggered an extremely long running computation when parsing the PDF page tree.

  • Issued DLA 1550-1 for src:drupal7 to close remote code execution and an external URL injection exploit in the Drupal web-based content management framework as part of Drupal's SA-CORE-2018-006 security release.

  • Issued ELA-49-1 for the Adplug sound library to fix potential DoS attack due to double-free vulnerability.


Uploads
  • redis:

    • 5.0~rc5-2 — Use the Debian hiredis library now that #907259 has landed. (#907258)
    • 5.0.0-1 — New upstream release.
    • 5.0.0-2 — Update patch to sentinel.conf to ensure the correct runtime PID file location (#911407), listen on ::1 interfaces too for redis-sentinel to match redis-server, & run the new LOLWUT command in the autopkgtests.
  • python-django:

    • 1.11.16-1 — New upstream bugfix release.
    • 1.11.16-2 — Fix some broken README.txt symlinks. (#910120)
    • 1.11.16-3 — Default to supporting Spatialite 4.2. (#910240)
    • 2.1.2-1 — New upstream security release.
    • 2.1.2-2 — Default to supporting Spatialite 4.2. (#910240)
  • libfiu:

  • 0.96-5 — Apply patch from upstream to write fiu_ctrl.py atomically to avoid a.parallel build failure. (#909843)

  • 0.97-1 — New upstream release.
  • 0.97-2 — Mangle return offset sizes for 64-bit variants to prevent build failures on 32-bit architectures. (#911733)

  • adminer (4.6.3-2) — Use continue 2 to avoid a switch/continue warning in PHP 7.3, thus preventing an autopkgtest regression. (#911825)

  • bfs (1.2.4-1) — New upstream release.

  • django-auto-one-to-one (3.1.1-1) — New upstream release.

  • lastpass-cli (1.3.1-5) — Add ca-certificates to Depends.

  • python-redis (2.10.6-5) — Fix debian/watch file.

  • python-daiquiri (1.5.0-1) — New upstream release.


I also sponsored uploads of elpy (1.25.0-1) and hiredis (0.14.0-1).

FTP Team


As a Debian FTP assistant I ACCEPTed 95 packages: barrier, cct, check-pgactivity, cloudkitty-dashboard, cmark-gfm, eclipse-emf, eclipse-jdt-core, eclipse-platform-team, eclipse-platform-ua, eclipse-platform-ui, eos-sdk, equinox-p2, fontcustom, fonts-fork-awesome, fswatch, fuse3, gau2grid, gitlab, glom, grapefruit, grub-cloud, gsequencer, haskell-base-compat-batteries, haskell-invariant, haskell-parsec-numbers, haskell-reinterpret-cast, haskell-resolv, haskell-shelly, haskell-skylighting-core, haskell-wcwidth, hollywood, intelhex, javapoet, libgpg-error, libjsoncpp, libnbcompat, lintian-brush, llvm-toolchain-snapshot, mando, mat2, mini-httpd-run, modsecurity, mtree-netbsd, neutron-tempest-plugin, ngspice, openstack-cluster-installer, pg-checksums, pg-cron, pg-dirtyread, pg-qualstats, pg-repack, pg-similarity, pg-stat-kcache, pgaudit, pgextwlist, pgfincore, pgl-ddl-deploy, pgmemcache, pgpool2, pgrouting, pgsql-ogr-fdw, pgstat, pipenv, postgresql-hll, postgresql-plproxy, postgresql-plsh, puppet-module-barbican, puppet-module-icann-quagga, puppet-module-icann-tea, puppet-module-rodjek-logrotate, pykwalify, pyocd, python-backports.csv, python-fastfunc, python-httptools, python-redmine, python-tld, python-yaswfp, python3-simpletal, r-cran-eaf, r-cran-emoa, r-cran-ggally, r-cran-irace, r-cran-parallelmap, r-cran-popepi, r-cran-pracma, r-cran-spp, radon, rust-semver-parser-0.7, syndie, unicycler, vitetris, volume-key, weston & zram-tools.

I additionally filed 14 RC bugs against packages that had potentially-incomplete debian/copyright files against fontcustom, fuse3, intelhex, libnbcompat, mat2, modsecurity, mtree-netbsd, puppet-module-barbican, python-redmine, r-cran-eaf, r-cran-emoa, r-cran-pracma, radon & syndie.

Chris Lamb https://chris-lamb.co.uk/blog/category/planet-debian lamby: Items or syndication on Planet Debian.

Lubuntu Blog: Disco Dingo: The development cycle has started!

Planet Ubuntu - Mër, 31/10/2018 - 4:16md
The development cycle for the Disco Dingo (which will be the codename for the 19.04 release) has started for the Lubuntu team! Translated into: español UPDATE: Daily images are now up, and are available on our downloads page, for the adventurous. Also, an update to Perl 5.28 is being done prior to opening as well. […]

Bastien Nocera: Pipewire Hackfest 2018

Planet GNOME - Mër, 31/10/2018 - 12:44md
Good morning from Edinburgh, where the breakfast contains haggis, and the charity shops have some interesting finds.

My main goal in attending this hackfest was to discuss Pipewire integration in the desktop, and how it will eventually replace PulseAudio as the audio daemon.

The main problem GNOME has had over the years with PulseAudio relate mostly to how PulseAudio was a black box when it came to its routing policy. What happens when you plug in an HDMI cable into your laptop? Or turn on your Bluetooth headset? I've heard the stories of folks with highly mobile workstations having to constantly visit the Sound settings panel.

PulseAudio has policy scattered in a number of places (do a "git grep routing" inside the sources to see that): some are in the device manager, then modules themselves can set priorities for their outputs and inputs. But there's nothing to take all the information in, and take a decision based on the hardware that's plugged in, and the applications currently in use.

For Pipewire, the policy decisions would be split off from the main daemon. Pipewire, as it gains PulseAudio compatibility layers, will grow a default/example policy engine that will try to replicate PulseAudio's behaviour. At the very least, that will mean that Pipewire won't regress compared to PulseAudio, and might even be able to take better decisions in the short term.

For GNOME, we still wanted to take control of that part of the experience, and make our own policy decisions. It's very possible that this engine will end up being featureful and generic enough that it will be used by more than just GNOME, or even become the default Pipewire one, but it's far too early to make that particular decision.

In the meanwhile, we wanted the GNOME policies to not be written in C, difficult to experiment with for power users, and for edge use cases. We could have started writing a configuration language, but it would have been too specific, and there are plenty of embeddable languages around. It was also a good opportunity for me to finally write the helper library I've been meaning to write for years, based on my favourite embedded language, Lua.

So I'm introducing Anatole. The goal of the project is to make it trivial to write chunks of programs in Lua, while the core of your project is written in C (we might even be able to embed it in Python or Javascript, once introspection support is added).

It's still in the very early days, and unusable for anything as of yet, but progress should be pretty swift. The code is mostly based on Victor Toso's incredible "Lua factory" plugin in Grilo. (I'm hoping that, once finished, I won't have to remember on which end of the stack I need to push stuff for Lua to do something with it ;)

SAT solvers for fun and fairness

Planet Debian - Mar, 30/10/2018 - 10:55md

Trøndisk 2018, the first round of the Norwegian ultimate series (the frisbee sport, not the fighting style) is coming up this weekend! Normally that would mean that I would blog something about all the new and exciting things we are doing with Nageru for the stream, but for now, I will just point out that the stream is on plastkast.no and will be live from 0945–1830 CET on Saturday (group stage) and 1020–1450 (playoffs) on Sunday.

Instead, I wanted to talk about a completely different but interesting subproblem we had to solve; how do you set up a good schedule for the group stages? There are twelve teams, pre-seeded and split into two groups (call them A0–A5 and B0–B5) that are to play round-robin, but there are only two fields—and only one of them is streamed. You want a setup that maximizes fairness in the sense that people get adequate rest between matches, and also more or less equal number of streamed games. Throw in that one normally wants the more exciting games last, and it starts to get really tricky to make something good by hand. Could we do it programmatically?

My first thought was that since this is all about the ordering, it sounded like a variant of the infamous travelling salesman problem. It's well-known that TSP is NP-hard (or NP-complete, but I won't bother with the details), but there are excellent heursitic implementations in practice. In particular, I had already used OR-Tools, Google's optimization toolkit, to solve TSP problems in the past; it contains a TSP solver that can deal with all sorts of extra details, like multiple agents to travel (in our case, multiple fields), subconstraints on ordering and so on. (OR-Tools probably doesn't contain the best TSP solver in the world—there are specialized packages that do even better—but it's much better than anything I could throw together myself.)

However, as I tried figuring out something, and couldn't quite get it to fit (there are so many extra nonlocal constraints), I saw that the OR-Tools documentation had a subsection on scheduling problems. It turns out this kind of scheduling can be represented as a so-called SAT (satisfiability) problem, and OR-Tools also has a SAT solver. (SAT, in its general forms, is also NP-hard, but again, there are great heuristics.) I chose the Python frontend, which probably wasn't the best idea in the world (it's poorly documented, and I do wonder when Python will take the step into the 90s and make spelling errors in variables into compile-time errors instead of throwing a runtime exception four hours into a calculation), but that's what the documentation used, and the backend is in C++ anyway, so speed doesn't matter.

The SAT solver works by declaring variables and various constraints between them, and then asking the machine to either come up with a solution that fits, or to prove that it's not possible. Let's have a look of some excerpts to get a feel for how it all works:

We know we have 15 rounds, two fields on each, and every field should contain a match. So let's generate 30 such variables, each containing a match number (we use the convention that match 0, 2, 4, 6, etc. are on the stream field and 1, 3, 5, 7, etc. are played in parallel on the other field):

matchnums = [] for match_idx in range(num_matches): matchnums.append(model.NewIntVar(0, num_matches - 1, "matchnum%d" % (match_idx)))

So this is 30 variables, and each go from 0 to 29, inclusive. We start a fairly obvious constraint; we can only play each match once:

model.AddAllDifferent(matchnums)

The SAT solver might make this into a bunch of special constraints underneath, or it might not. We don't care; it's abstracted away for us.

Now, it's not enough to just find any ordering—after all, we want to find an ordering with some constraints. However, the constraints are rarely about the match numbers, but more about the teams that play in those matches. So we'll need some helper variables. For instance, it would be interesting to know which teams play in each match:

home_teams = [] away_teams = [] for match_idx in range(num_matches): home_teams.append(model.NewIntVar(0, num_teams - 1, "home_team_match%i" % (match_idx))) away_teams.append(model.NewIntVar(0, num_teams - 1, "away_team_match%i" % (match_idx))) model.AddElement(matchnums[match_idx], home_teams_for_match_num, home_teams[match_idx]) model.AddElement(matchnums[match_idx], away_teams_for_match_num, away_teams[match_idx])

AddElement() here simply is an indexing operation; since there's no difference between home and away teams for us, we've just pregenerated all the matches as A0 vs. A1, A0 vs. A2, etc. up until A4 vs. A6, A5 vs. A6 and then similarly for the other gruop. The “element” constraint makes sure that e.g. home_team_match0 = home_teams_for_match_num[matchnum0]. Note that even though I think of this is as an assignment where the home team for match 0 follows logically from which match is being played as match 0, it is a constraint that goes both ways; the solver is free to do inference that way, or instead first pick the home team and then deal with the consequences for the match number. (E.g., if it picks A5 as the home team, the match number most certainly needs to be 14, which corresponds to A5–A6.)

We're not quite done with the helpers yet; we want to explode these variables into booleans:

home_team_in_match_x_is_y = [[ model.NewBoolVar('home_team_in_match_%d_is_%d' % (match_idx, team_idx)) for team_idx in range(num_teams) ] for match_idx in range(num_matches)] for match_idx in range(num_matches): model.AddMapDomain(matchnums[match_idx], match_x_has_num_y[match_idx])

and similarly for away team and match number.

So now we have a bunch of variables of the type “is the home team in match 6 A4 or not?”. Finally we can make some interesting constraints! For instance, we've decided already that the group finals (A0–A1 and B0–B1) should be the last two matches of the day, and on the stream field:

model.AddBoolOr([match_x_has_num_y[28][0], match_x_has_num_y[28][15]]) model.AddBoolOr([match_x_has_num_y[26][0], match_x_has_num_y[26][15]])

This is a hard constraint; we don't have a solution unless match 0 and match 15 are the last two (and we earlier said that they must be different).

We're going to need even more helper variables now. It's useful to know whether a team is playing at all in a given round; that's the case if they are the home or away team on either field:

plays_in_round = {} for team_idx in range(num_teams): plays_in_round[team_idx] = {} for round_idx in range(num_rounds): plays_in_round[team_idx][round_idx] = model.NewBoolVar('plays_in_round_t%d_r%d' % (team_idx, round_idx)) model.AddMaxEquality(plays_in_round[team_idx][round_idx], [ home_team_in_match_x_is_y[round_idx * 2 + 0][team_idx], home_team_in_match_x_is_y[round_idx * 2 + 1][team_idx], away_team_in_match_x_is_y[round_idx * 2 + 0][team_idx], away_team_in_match_x_is_y[round_idx * 2 + 1][team_idx]])

Now we can establish a few other very desirable properties; in particular, each team should never need to play two matches back-to-back:

for round_idx in range(num_rounds - 1): for team_idx in range(num_teams): model.AddBoolOr([plays_in_round[team_idx][round_idx].Not(), plays_in_round[team_idx][round_idx + 1].Not()])

Note that there's nothing here that says the same team can't be assigned to play on both fields at the same time! However, this is taken care of by some constraints on the scheduling that I'm not showing for brevity (in particular, we established that each round must have exactly one game from group A and one from group B).

Now we're starting to get out of the “hard constraint” territory and more into things that would be nice. For this, we need objectives. One such objective is what I call ”tiredness”; playing matches nearly back-to-back (ie., game - rest - game) should have a penalty, and the solution should try to avoid it.

tired_matches = [] for round_idx in range(num_rounds - 2): for team_idx in range(num_teams): tired = model.NewBoolVar('team_%d_is_tired_in_round_%d' % (team_idx, round_idx)) model.AddMinEquality(tired, [plays_in_round[team_idx][round_idx], plays_in_round[team_idx][round_idx + 2]]) tired_matches.append(tired) sum_tiredness = sum(tired_matches)

So here we have helper variables that are being set to the minimum (effectively a logical AND) of “do I play in round N” and “do I play in round N + 2”. Tiredness is simply a sum of those 0–1 variables, which we can seek to minimize:

model.Minimize(sum_tiredness)

You may wonder how we went from a satisfiability problem to an optimization problem. Conceptually, however, this isn't so hard. Just ask the solver to find any solution, e.g. something with sum_tiredness 20. Then simply add a new constraint saying sum_tiredness <= 19 and ask for a re-solve (or continue). Eventually, the solver will either come back with a better solution (in which case you can tighten the constraint further), or the message that you've asked for something impossible, in which case you know you have the optimal solution. (I have no idea whether modern SAT solvers actually work this way internally, but again, conceptually it's simple.)

As an extra bonus, you do get incrementally better solutions as you go. These problems are theoretically very hard—in fact, I let it run for fun for a week now, and it's still not found an optimal solution—and in practice, you just take some intermediate solution that is “good enough”. There are always constraints that you don't bother adding to the program anyway, so there's some eyeballing involved, but still feels like a more fair process than trying to nudge it by hand.

We had many more objectives, some of them contradictory (e.g., games between more closely seeded opponents are more “exciting”, and should be put last—but they should also be put on the stream, so do you put them early on the stream field or late on the non-stream field?). It's hard to weigh all the factors against each other, but in the end, I think we ended up with something pretty nice. Every team gets to play two or three times (out of five) on the stream, only one team needs to be “tired” twice (and I checked; if you ask for a hard maximum of once for every team, it comes back pretty fast as infeasible), many of the tight matches are scheduled near the end… and most importantly, we don't have to play the first matches while I'm still debugging the stream. :-)

You can see final schedule here. Good luck to everyone, and consider using a SAT solver next time you have a thorny scheduling problem!

Steinar H. Gunderson http://blog.sesse.net/ Steinar H. Gunderson

Enabling Wake-on-Lan with the N34 Mini PC

Planet Debian - Mar, 30/10/2018 - 8:58md

There is a room at the top of my house which was originally earmarked for storage (the loft is full of insulation rather than being a useful option). Then I remembered I still had my pico projector and it ended up as a cinema room as well. The pico projector needs really low light conditions with a long throw, so the fact the room only has a single small window is a plus.

I bought an “N34” mini PC to act as a media player - I already had a spare DVB-T2 stick to Freeview enable things, and the Kodi box downstairs has all my DVDs stored on it for easy streaming. It’s a Celeron N3450 based box with 4G RAM and a 32GB internal eMMC (though I’m currently running off an SD card because that’s what I initially used to set it up and I haven’t bothered to copy it onto the internal device yet). My device came from Amazon and is branded “Kodlix” (whose website no longer works) but it appears to be the same thing as the Beelink AP34.

Getting Linux onto it turned out to be a hassle. GRUB does not want to play with the EFI BIOS; it can be operated sometimes if manually called from the EFI Shell, but it does not work as the default EFI image to load. Various forum posts recommended the use of rEFInd, which mostly works fine.

Other than that Debian Stretch worked without problems. I had to pull in a backports kernel in order to make the DVB-T2 stick work properly, but the hardware on the N34 itself was all supported out of the box.

The other issue was trying to get Wake-on-Lan to work. The room isn’t used day to day so I want to be able to tie various pieces together with home automation such that I can have everything off by default and a scene configured to set things up ready for use. The BIOS has an entry for Wake-on-Lan, ethtool reported Supports Wake-on: g which should mean MagicPacket wakeup was enabled, but no joy. Looking at /proc/acpi/wakeup gave:

/proc/acpi/wakeup contents Device S-state Status Sysfs node HDAS S3 *disabled pci:0000:00:0e.0 XHC S3 *enabled pci:0000:00:15.0 XDCI S4 *disabled BRCM S0 *disabled RP01 S4 *disabled PXSX S4 *disabled RP02 S4 *disabled PXSX S4 *disabled RP03 S4 *disabled pci:0000:00:13.0 PXSX S4 *disabled pci:0000:01:00.0 RP04 S4 *disabled PXSX S4 *disabled RP05 S4 *disabled PXSX S4 *disabled RP06 S4 *disabled pci:0000:00:13.3 PXSX S4 *disabled pci:0000:02:00.0 PWRK S4 *enabled platform:PNP0C0C:00

pci:0000:01:00.0 is the network card:

01:00.0 Ethernet controller [0200]: Realtek […] Ethernet Controller [10ec:8168] (rev 0c)

I need this configured to allow wakeups which apparently is done via sysfs these days:

echo enabled > /sys/bus/pci/devices/0000\:01\:00.0/power/wakeup

This has to be done every boot so I just tied it into /etc/network/interfaces.

All of this then enables Home Assistant to control the Kodi box:

Home Assistant Kodi WoL configuration wake_on_lan: media_player: - platform: kodi name: Kodi (Cinema) host: kodi-cinema.here port: 8000 username: kodi password: !secret kodi_cinema_pass enable_websocket: false turn_on_action: service: wake_on_lan.send_magic_packet data: mac: 84:39:be:11:22:33 broadcast_address: 192.168.0.2 turn_off_action: service: media_player.kodi_call_method data: entity_id: media_player.kodi_cinema method: System.Shutdown

My Home Assistant container sits on a different subnet to the media box, and I found that the N34 wouldn’t respond to a Wake-on-Lan packet to the broadcast MAC address. So I’ve configured the broadcast_address for Home Assistant to be the actual IP of the media box, allowed UDP port 9 (discard) through on the firewall and statically nailed the ARP address of the media box on the router, so it transmits the packet with the correct destination MAC:

ip neigh change 192.168.0.2 lladdr 84:39:be:11:22:33 nud permanent dev eth0

I’ve still got some other bits to glue together (like putting the pico projector on a SonOff), but this gets me started on that process.

(And yes, the room is a bit cosier these days than when that photograph was taken.)

Jonathan McDowell https://www.earth.li/~noodles/blog/ Noodles' Emptiness

Faqet

Subscribe to AlbLinux agreguesi