Planet Debian

Debian LTS report for April 2019

After a longer break (~two years) I again took part in the funded Debian LTS project in April 2019.

I was allocated 14 hours and spent all of them (and even a bit more) on the following two issues:

• DLA 1748-1: Several security fixes for apache2
• DLA 1766-1: OpenPGP signature spoofing in evolution

Links

• Debian Wiki: Debian Long Term Support
• Freexian: Debian Long Term Support

Weblate 3.6.1 has been released today. It is a bugfix release fixing several issues reported after the 3.6 release.

Full list of changes:

• Improved handling of monolingual Xliff files.
• Fixed digest notifications in some corner cases.
• Fixed addon script error alert.
• Fixed generating MO file for monolingual PO files.
• Fixed display of uninstalled checks.
• Indicate administered projects on project listing.
• Allow update to recover from missing VCS repository.

If you are upgrading from older version, please follow our upgrading instructions.

You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. Weblate is also being used on https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Turris, FreedomBox, Weblate itself and many other projects.

Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure.

Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Filed under: Debian English SUSE Weblate

We have a new DPL! On 21 April 2019 Sam Hartman started his term as Debian Project Leader.

Join us on the #debian-meeting channel on the OFTC IRC network on 10 May 2019 at 10:00 UTC for an introduction to our new DPL, and also to have the chance to ask him any questions that you may have.

Your IRC nick needs to be registered in order to join the channel. Refer to the Register your account section on the oftc website for more information on how to register your nick.

We plan to have many more project-wide IRC sessions in the future.

You can always refer to the debian-meeting wiki page for the latest information and up to date schedule.

I wanted to be able to test deployment of VMs inside a Kubernetes cluster with KubeVirt (and maybe Kata-Containers too), using a VM-based testbed. An additional requirement was trying to get a similar setup script that could be applied to a regular physical host, should the tests be conclusive.

I’ve adapted an existing installation procedure for installation on Vagrant Ubuntu VMs, and the result is there:
https://github.com/olberger/vagrant-k8s-kubevirt-katacontainers

I runs a Vagrant VM with libvirt+qemu/kvm, installed with Ubuntu 18.04, and provisions a Kubernetes cluster using kubeadm (no longer minikube, unlike what was done in the original repo I’ve forked from).

Inside the cluster, containers are run with CRI-O, which seems to be compatible with KubeVirt. Packages are “official” k8s packages (except for CRI-O and KubeVirt).

I hope it’ll work for you too, provided that you can run nested virtualization.

Next steps will be trying I’ve also added Kata-Containers to the cluster. It seems that kata-containers and KubeVirt are compatible (all running with libvirt below).

Here’s a screencast :

 

Updated: I’ve now added Kata-Containers deployment to the provisioning scripts. Also, it runs 18.04 now (LTS too, but more recent than 16.04). Oh, and the scripts seem to be useable outside Vagrant, on a regular Ubuntu server too.

Next steps, testing various workloads on the cluster for virtual labs deployments.

Since I started working as a full-time Software Engineer at Peerdustry I’ve been having a lot of trouble finding the time to do everything I wanted outside of my work hours. Consequently, I have contributed to FLOSS projects less often than I would like and I have not been able to write a single blog post this year. However, I have had some progress recently in organizing my time. Regarding the writing habit, I will try a more minimalist approach to improve my frequency by writing short texts with some quick notes or tips that are useful to others and myself in the future.

This is my first post where I apply this approach. It has a little tip about how to initialize your Ruby objects passing named arguments with a Hash.

In a PORO - Plain Old Ruby Object, the usual way to assign values to the attributes of a new object is through the arguments of the initialize method, as illustrated by the snippet below:

class Person def initialize(name, email, address, phone) @name = name @email = email @address = address @phone = phone end end person = Person.new("Arthur", "arthur@example.com", nil, "+55 11 999999999")

The initialize method follows the same rules of any other method in Ruby, for instance, the parameters must be passed in exactly the same order in which they were declared. Unlike, the Ruby on Rails framework (RoR), more specifically the ActiveRecord, provides a very convenient way to assign values to the object’s attributes of your model classes with a Hash, as shown below:

person = Person.new(name: "Arthur", phone: "+55 11 999999999", email: "arthur@example.com")

With this approach, we need neither to pass nil values nor to follow any pre-defined order of arguments, since the arguments based on a Hash enable ‘named arguments’. Such behavior is also provided by the module ActiveModel::Model, which is very useful for including in your Rails App’s specific domain objects, such as Service Objects, since this module is packaged within the framework. Besides the support for initializing objects with a hash, the ActiveModel::Model module adds other behaviors to your classes that you may not be interested in, such as model name introspections, conversions, translations and validations.

However, if you are not using Rails, but still want to initialize your objects with Hashes, you can create a generic module with the single purpose of implementing such behavior and include it into your classes.

module HashBasedInit def initialize(args) args.each do |key, value| send("#{key}=", value) end end end class Person include HashBasedInit attr_accessor :name, :email, :address, :phone end person = Person.new(name: "Arthur", phone: "+55 11 999999999", email: "arthur@example.com")

Note that it is necessary to declare accessing methods to your attributes as we did in the Person class through the attr_accessor method. Now, we only need to include the HashBasedInit module in your other classes of your project and that’s it.

Let’s get moving on! ;)

Thanks to support from Article 19, I was able to attend IETF 104 in Prague, Czech Republic this week. Primarily this was to present my Internet Draft which takes safe measurement principles from Tor Metrics work and the Research Safety Board and applies them to Internet Measurement in general.

My IETF badge, complete with additional tag for my nick

I attended with a free one-day pass for the IETF and free hackathon registration, so more than just the draft presentation happened. During the hackathon I sat at the MAPRG table and worked on PATHspider with Mirja Kühlewind from ETH Zurich. We have the code running again with the latest libraries available in Debian testing and this may become the basis of a future Tor exit scanner (for generating exit lists, and possibly also some bad exit detection). We ran a quick measurement campaign that was reported in the hackathon presentations.

During the hackathon I also spoke to Watson Ladd from Cloudflare about his Roughtime draft which could be interesting for Tor for a number of reasons. One would be for verifying if a consensus is fresh, another would be for Tor Browser to detect if a TLS cert is valid, and another would be providing archive signatures for Tor Metrics. (We’ve started looking at archive signatures since our recent work on modernising CollecTor).

On the Monday, this was the first “real” day of the IETF. The day started off for me at the PEARG meeting. I presented my draft as the first presentation in that session. The feedback was all positive, it seems like having the document is both desirable and timely.

The next presentation was from Ryan Guest at Salesforce. He was talking about privacy considerations for application level logging. I think this would also be a useful draft that compliments my draft on safe measurement, or maybe even becomes part of my draft. I need to follow up with him to see what he wants to do. A future IETF hackathon project might be comparing Tor’s safe logging with whatever guidelines we come up with, and also comparing our web server logs setup.

Nick Sullivan was up next with his presentation on Privacy Pass. It seems like a nice scheme, assuming someone can audit the anti-tagging properties of it. The most interesting thing I took away from it is that federation is being explored which would turn this into a system that isn’t just for Cloudflare.

Amelia Andersdotter and Christoffer Långström then presented on differential privacy. They have been exploring how it can be applied to binary values as opposed to continuous, and how it could be applied to Internet protocols like the QUIC spin bit.

The last research presentation was Martin Schanzenbach presenting on an identity provider based on the GNU Name System. This one was not so interesting for me, but maybe others are interested.

I attended the first part of the Stopping Malware and Researching Threats (SMART) session. There was an update from Symantec based on their ISTR report and I briefly saw the start of a presentation about “Malicious Uses of Evasive Communications and Threats to Privacy“ but had to leave early to attend another meeting. I plan to go back and look through all of the slides from this session later.

The next IETF meeting is directly after the next Tor meeting (I had thought for some reason it directly clashed, but I guess I was wrong). I will plan to remotely participate in PEARG again there and move my draft forwards.

Redworth Hall

main talk, thanks Mark Little

Amiga lightning talk, thanks Mark Little

I gave a talk on my research at the Fourth Annual UK System Research Challenges Workshop. This is the second time I've attended this conference. Last year I presented on some Red Hat work.

The conference took place at Redworth Hall, a 17th Century Jacobean Manor House converted into a spa Hotel. The main presentations took place in an ornate hall with high ceilings, candelabra and long curtains (definitely not a drop you can buy at Dunelm)

This is the first time I've presented on my research to a public audience. Here's a copy of my presentation slides, with speaker notes. I've tried to annotate the questions from the session into the notes of the last slide.

• uksystems19.pdf

I also delivered a short lightning talk about my Amiga floppy recovery project which resulted in some really interesting spin-off conversations about historic computing.

Thanks to my employer, Red Hat, for sponsoring the conference and making it possible, and to Jen for doing an excellent job of making sure it could take place.

My project to make an alternative board for Pandemic Rising Tide needed a program to lay out a planar graph, choosing exact coordinates for the vertices.

(The vertices in question are the vertices of the graph which is the dual of the adjacency graph of the board "squares" - what Pandemic Rising Tide calls Regions. For gameplay reasons the layout wants to be a straight line drawing - that is, one where every boundary is a straight line.)
Existing softwareI found that this problem was not well handled by existing Free Software. The leading contender, graphviz, generally produces non-planar layouts even for planar inputs; and it does not provide a way to specify the planar embedding. There are some implementations of "straight line drawing" algorithms from the literature, but these produce layouts which meet the letter of the requirement for the drawing to consist only of nonintersecting straight lines, but they are very ugly and totally unsuitable for use as a game board layout.

My web searches for solutions to this problem yielded only forum postings etc. where people were asking roughly this question and not getting a satisfactory answer.

I have some experience with computer optimisation algorithms and I thought this should be a tractable problem, so I set out to solve it - well, at least well enough for my purposes.
My approachMy plan was to use one of the algorithms from the literature to generate a straight line drawing, and then use cost-driven nonlinear optimisation to shuffle the vertices about into something pretty and useable.

Helpfully Boost provides an implementation of Chrobak & Payne's straight line drawing algorithm. Unfortunately Boost's other planar graph functions were not suitable because they do not remember which face is the outer face. (In planar graph theory and algorithms the region outside the graph drawing is treated as a face, called the outer face.) So I also had to write my own implementations of various preparatory algorithms - yet more yak shaving before I could get to the really hard part.

Having been on a Rust jag recently, I decided on Rust as my implementation language. I don't regret this choice, although it did add a couple of yaks.
Cost function and constraintsMy cost function has a number of components:

• I wanted to minimise the edge lengths.
  
• But there was a minimum edge length (for both gameplay and aesthetic reasons)
  
• Also I wanted to avoid the faces having sharp corners (ie, small angles between edges at the same vertex)
  
• And of course I needed the edges to still come out of each vertex in the right order.
  

You will notice that two of these are not costs, but constraints. Different optimisation algorithms handle this differently.

Also "the edges to still come out of each vertex in the right order" is hard to express as a continuous quantity. (Almost all of these algorithms demand that constraints take the form of a function which is to be nonnegative, or some such.) My solution is, at each vertex, to add up the angles between successive edges (in the intended order, and always treating each direction difference as a positive angle). Ie, to add up the face corner angles. They should sum to tau: if so, we have gone round once and the order is right. If the edges are out of order, we'll end up going round more than once. If the sum was only tau too much, I defined the violation quantity to be tau minus the largest corner angle; this is right because probably it's just that two edges next to each other are out of order and the face angle has become "negative"; this also means that for a non-violating vertex, the violation quantity is negative but still represents how close to violation we are. (For larger corner angle sums, I added half of the additional angle sum as an additional violation quantity. That seemed good enough in the end.)
Simulated annealing - and visual debug of the optimisationMy first attempt used GSL's simulated annealing functions. I have had reasonable success with siman in the past. The constraints are folded into the cost function. (An alternative approach is to somehow deal with them in the random step function, eg by adjusting violating layouts to similar non-violating ones, but that seemed quite tricky here.)

Siman did not seem to be working at all.

I was hampered by not knowing what was going on so I wrote a visual debug utility which would let me observe the candidate layouts being tried, in real time. (I should have taken my first instinct and done it in Tcl/Tk, but initially Qt seemed like it would be easier. But in the end I had to fight several of Qt's built-in behaviours.)

The visual debug showed me the graph randomly jiggling about without any sign of progress. It was clear that if this was going to work at all it would be far too slow.
More suitable optimisation algorithmI felt that a gradient descent algorithm, or something like one, would work well for this problem. It didn't seem to me that there would be troublesome local minima. More web searching led me to Steven G. Johnson's very useful NLopt library. As well as having implementations of algorithms I thought would work well, it offered the ability to change algorithm without having to deal with a whole new API.

I quickly found that NLopt's Sbplx algorithm (T. Rowan's Subplex algorithm, reimplemented) did fairly well. That algorithm does not support constraints but the grandly-named Augmented Lagrangian Method can handle that: it adds the constraint violations to the cost. It then reruns the optimisation, cranking up the constraint violation cost factor until none of the constraints are violated by more than the tolerance.

Unfortunately the Augmented Lagrangian Method can convert a problem with a cost function without local minima, into one which does have bad local minima. The Sbplx algorithm is a kind of descent algorithm so it finds a local minimum and hopes it's what you wanted. But unfortunately for me it wasn't: during the initial optimisation, part of the graph "capsized", violating the edge order constraint and leaving a planar layout impossible. The subsequent cranking up of the constraint violation cost didn't help, I think maybe because my violation cost was not very helpful at guiding the algorithm when things were seriously wrong.

But I fixed this by the simple expedient of adding the edge order constraint with a high cost to my own cost function. The result worked pretty well for my simple tests and for my actual use case. The graph layout optimiation takes a couple of minutes. The results are nice, I think.

I made a screen capture video of the optimisation running. (First the debug build which is slower so captures the early shape better; then again with the release build.)
SoftwareThe planar graph layout tool I wrote is plag-mangler.

It's really not very productised, but I think it will be useful to people who have similar problems. Many of the worst features (eg the bad command line syntax) would be easy to fix. OTOH if you have a graph it does badly on, please do file an issue on salsa, as it will guide me to help make the program more general.
ReferencesSee my first post about this project for some proper references to the academic literature etc.

(Edit 2019-04-04 12:55 +0100: Fixed typos and grammar.)

comments

Over the weekend I spent time putting together a few slides on Linux namespaces, mostly because I wanted to understand better (and putting this together helped a lot!), but also because it will be useful to me later, and finally (and really) because I promised to a few colleagues I’ll explain how all this works :)

So the HTML slides are here, and the source is on github. I put the source up because I’m very sure this has lots of mistakes; not only in the intro where I mention FreeBSD jails and OpenVZ a bit (but I have zero experience with both), but also in the main content, so any corrections are more than welcome.

Writing this, and organising it, was actually much more entertaining than I originally thought. It also made me realise that the kernel-level implementation is very powerful, and—at least to the extent that e.g. Debian uses it by default—it’s basically wasted (a lot of lost opportunity). I know there are some tools to use this, but for example why Firefox is not by default namespaced… I don’t know. Food for later thought. Happy to receive information otherwise, of course.

Most of the information is gathered from man pages, Wikipedia (for the historic bits), blog posts, mailing list archives, etc., so I don’t claim a lot of deep original content; the main idea is just to put all this information together in a single place.

Hope this is useful to somebody else, and again, contributions and re-sharing welcome (CC-BY-SA-4.0).

In March 2019, I have worked on the Debian LTS project for 14 hours (of 10 hours planned plus 4 hours pulled over from February) and on the Debian ELTS project for another 2 hours (of originally planned 6 hours) as a paid contributor.

LTS Work

• CVE triaging (ntp, glib2.0, libjpeg-turbo, cron, otrs2, poppler)
• Sponsor upload to jessie-security (aka LTS): cron (DLA 1723-1 [1])
• Upload to jessie-security (aka LTS): openssh (DLA 1728-1 [2])
• Upload to jessie-security (aka LTS): libssh2 (DLA 1730-1 [3])
• Upload to jessie-security (aka LTS): libav (DLA 1740-1 [4])

ELTS Work

• Create .debdiff for cron src:pkg targetting wheezy (but I failed to build it due to two issues with Debian 10 as build machine)
• Discover and document that kernel boot parameter "vsyscall=emulate" is required for building wheezy packages on Debian 10. (See #844350 and #845942 for details).
• Bug hunt sbuild bug #926161 in sbuild 0.78.1-1 [5]

References

• [1] https://lists.debian.org/debian-lts-announce/2019/03/msg00025.html
• [2] https://lists.debian.org/debian-lts-announce/2019/03/msg00030.html
• [3] https://lists.debian.org/debian-lts-announce/2019/03/msg00032.html
• [4] https://lists.debian.org/debian-lts-announce/2019/03/msg00041.html
• [5] https://bugs.debian.org/926161

ASAT Test

For last few days I was not in Pune as had gone to attend a workshop which was funded by Innovation for change . Unfortunately, I was not able to take part in the workshop as the traveling proved to be a bit too much in too short a time. While I would share more in another blog post for the moment, I would like to share about the ASAT test that India conducted. While it’s a positive development, from my perspective there was no need for the Prime Minister to come on-stage and declare that we can shoot down a Satellite at 3k when China can do the same at 38k . So we have a long way to go, in as far as parity with China is concerned. While I’m not sharing the source of this information, this is for all and anybody to see and figure out if you know how to use the web. There are a few things I would share, I didn’t use any private data-sets to get this information, which means it’s available easily online. I did not use tor, the dark web otherwise I probably could have got far more material. Thirdly and more interestingly, if you wanna start your search from scratch, ORF could be a good starting point from an Indian POV although there are many other such think-tanks which could help you in your research.

The only question I have to ask is if we are the weaker party, which is clearly the case herein, then whom are we trying to sell this idea if not the Indian public ? Chinese military satellites are in varying range from 300 km. to 36,000 km. so there is hardly a chance that we would be able to make any significant dent to their military usage. Also using an ASAT on another country’s satellite would be an act of war. As far as Communication satellites are concerned, they are also at 36,000 km. are at the Geostationary orbit so they will not be harmed. There is also a pretty nice animation of the same at wikimedia .

International Politics

While we can understand that Mr. Modi did it for electioneering, it does have impact internationally. Last year when the Chinese did another ASAT test (which the Pentagon guestimate it reached 36k from sea level from their ground and space-based instruments) . The Chinese statement was quite brief and to the point . They said that they did the test and it performed on all the military objectives. This is a sort of perfect statement which doesn’t reveal either what the Chinese military objectives of the test were and what was accomplished. All other Governments either have to rely on their own instrumentation (if they have in space to spy and on lookout for such activities ) or rely on Pentagon’s guestimates and findings which they chose to public. The Americans are also well to not show their hand and may share some information or even share mis-information as this is and would be considered part of Information warfare. This is also precisely the reasons we have ambassadors, diplomats and others who sit together and are engaged in naunced wording. There were no need of an announcement and even if it needed, it could have been done by some mid-level executive on DRDO saying something similar on the lines of what the Chinese said and probably adding we have a long road ahead of us or something like that.

Update – 04/04/2019 – Somebody on twitter shared a link to Dr. Saraswat’s latest interview which was held a few days back .

The answers were designed in the way so as to show that the UPA govt. didn’t show the interest for the ASAT test while the NDA Govt. Even if we do take Dr. Saraswat’s interpretion of how the event happened, it still raises questions rather than answers.

1. By. Dr. Saraswat’s own admission, it was an informal presentation . While he didn’t go into the details of what he meant by ‘informal presentation’ it could be something akin to somebody asking me to do an informal presentation on Debian. For this, the most I have to do is collect my thoughts, read up a bit onto what’s new, exciting if there is something which catches my eye and at the most have 5-7 pages of slides and depending upon what kind of organization it is, I would share what Debian is. If however, somebody would ask me to make a presentation on a possible Debian deployment, it would consist of knowing and having details of how small or big the network is ? What are the critical points in the network (for e.g. many shops or small businessess have either their custom-designed billing system whose source-code they don’t have and has to be on MS-Windows) while other systems you could potentially do the deployment. Apart from doing the actual deployment, there would be time for training, documentation etc. all of which involve some sort of hard numbers and time which both parties would have to work at to get some sort of understanding of how this different system works.

2. And this is where my question comes in. In the interview it’s also not mentioned what time or date when the presentation was done. Now we all know that 2014 was only a year away, if the presentation was done 6-9 months before elections, it is very much possible that there was no interest because it would be time-consuming and there are no guarantees of a successful test. In fact, before this test which was declared a success, there was another test which was conducted by DRDO which was a failure. This also begs or marks the question as to when did Dr. Saraswat approach NDA or vice-versa and when he started actively working on the project. Did it take 5 years for this to come to this stage or 2 years or less because that would give some more guidance and a way for us to guage future success of the project.

Rumour of Merging DRDO and ISRO

There is also a worrying bit of news that the Government of India is thinking of merging both DRDO and ISRO to be similar structure to what the Chinese have for their space program, which I think will be disastrous for the Indian Space Program, the taxpayer public money as well as the two organizations as well.

DRDO work culture

While my mother had the honor of serving within a sub-set of DRDO and she was friends with few scientists, one of the major grouses for most scientists was the constant shifting of parameters or specifications. To take a very simple example, let’s think that you are told or given a set of specs. of a Maruti 800, a small city car , then a year, year down the half, you are told that the design specifications has changed to now a Station Wagon or a hatchback and when you start to design for those, the specs. are changed again in a year or two to a sports car. Now any car-enthusiast would know that these three are completely different cars having their unique needs, dimensions, center of gravity, steering, fuel consumption, the works. Extrapolate that to a missile or missiles where more often than not, these design changes were at many a times not asked by the Armed forces who would be the actual users but the bureaucracy i.e. civil servants, many from IAS who instead of consulting, using consesus of the people on both sides, instead share and put whatever opinion they have. Of course inter-personality conflicts also do occur and inspite of it DRDO is able to do what it does. Because of quite a few such Inter-personality conflicts, many a brilliant scientist have been forced to leave DRDO and are now either serving private Indian interests or some foreign ones and they repent why they spent their best productive years at DRDO or whatever sub-unit they were into.

ISRO Work culture

While I do not have relatives working in ISRO, I do and did have friends who work or have worked in ISRO. Due to the nature of the work itself, which is more exploratory and peaceful in nature, they are able to collaborate with lot of educational institutions within India and worldwide and even collaborate with organizations like NASA, ESA and others. The civilian beaureacracy has had a more hands-off approach which has resulted in ISRO being able to carry out whatever fantastic achievements they have been able to achieve. The only thing, if they need to learn from this Government, is the ability to find money and do more of promotion of the good work they are doing. Even if ISRO were to do 1% of the promotion that NASA does in promotion with merchandising, they would get more than money back while at the same time inspire millions of young children to take up challenges in space sciences.

So from the above, it is pretty clear it would be disastrous as both have a very different mind-set and ways of working. I remember hearing or conversing with some military gentleman couple of years ago and we were talking on some similar topics. This was on a short train trip. The gentleman remarked, it’s not often that we get things to work right the first time, in any of the fields of endevour the military does. If we do, even some small part, we make sure not to disturb or change it and would make changes around it so it works and fix all the other things and processes till there is cohesion. He went on to share some real-life examples from his work which I have since forgotten but the principle seems good, solid enough at least to me.

Making Organizations Fun

At the very end, I would like to draw attention to Jonathan Carter’s blog post where he shares about Debian and Fun . I found both the art peices most appropriate not just for the organizations listed above, but should be the calling points of any organization which believes in genuine stewardship of whatever organization they have or hope to take forward.

While I would invite everybody who has more than a passing interest in the world of computer science to see Jonathan’s and other potential DPL (Debian Project Leader) platforms as well as their rebuttals, the difference between the two is statements or pictures above is that while the first one is an employer-employee model, the second is more on the volunteer, contributor-steward model. Although as DPL , the only perks the DPL enjoys are speaking about Debian in sometimes exotic locations, although that is more than tempered by being part of Debian Politics and Free software politics which comes with its own rewards, risk scenario and is and can be pretty tricky as has been observed over the years.

Here’s what happened in the Reproducible Builds project between March 24th and March 30th 2019:

• On our mailing list this week, Vagrant Cascadian posted a request for suggestions for Reproducible Builds-related ideas that students from Portland State University could work on.

• In addition, Holger Levsen made an announcement that registration for a MiniDebConf in Hamburg during June 2019 is now open and will likely involve a number of people involved in Reproducible Builds, both from Debian and from other projects.

• Bernhard M. Wiedemann posted his monthly reproducible builds update for the openSUSE distribution.

• Reproducible builds were mentioned in the 2018 annual report from the Huawei Cyber Security Evaluation Centre Oversight Board, a report to the National Security Adviser of the United Kingdom.

• 17 reviews of Debian packages were added, 2 were updated and 10 were removed in this week, adding to our knowledge about identified issues.

• There were a number of updates to the reproducible-builds.org project website, including Stefano Zacchiroli adding the Reproducible Builds Steering Committee to the Who is involved? page  […] and jajajasalu2 dropping invalid links to the trydiffoscope and reprotest issue trackers […].

Don’t forget that Reproducible Builds is part of May/August 2019 round of Outreachy which offers paid internships to work on free software. Internships are open to applicants around the world and are paid a stipend for the three month internship with an additional travel stipend to attend conferences. So far, we received more than ten initial requests from candidates and the closing date for applicants is April 2nd. More information is available on the application page.

Packages reviewed and fixed, and bugs filed

• Chris Lamb:
  • #925575 filed against tclxml.
• Bernhard M. Wiedemann:
  • mpich: date/time and filesystem ordering issues.
  • cobbler (date/time).
  • llvm7/clang (sort table to fix gnustep-libobjc2, already upstream).
  • meep (FTBFS CPU).

Test framework development

• We operate a comprehensive Jenkins-based testing framework that powers tests.reproducible-builds.org. The following changes were done this week:

  • Mattia Rizzolo built a static list of SSH host keys […] so we could build the ssh_config file based on this file […], leading to being able to enable OpenSSH’s StrictHostKeyChecking option […][…][…].
  • Holger Levsen added a number of links to pages, including Guix’s challenge command […], the F-Droid tests […] as well as NixOS and openSUSE tests […].

This week’s edition was written by Bernhard M. Wiedemann, Chris Lamb & Holger Levsen and was reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

I was assigned 20 hours of work by Freexian's Debian LTS initiative and carried over 16.5 hours from February. I worked 22.5 hours and so will carry over 14 hours.

I merged changes from stretch's linux package into the linux-4.9 package, uploaded that, and issued DLA-1715. I made another stable update to Linux 3.16 (3.16.64). I then rebased Debian's linux package on that version, uploaded it, and issued DLA-1731. This unfortunately introduced a regression, which I fixed in a second update.

I also reviewed and merged Emilio Pozuelo Monfort's changes to the firmware-nonfree package to address CVE-2018-5383.

Hello.

Three weeks ago I attended DebUtsav-Delhi organized by the Debian and free folks in North India.

Debutsav-Delhi is the third edition of its kind. Initially Mozilla Delhi backed the Debutsav-delhi when they pitched the idea but later they withdrew for some reason and just became a supporting member. I must say Debian India events are happening frequent now. Some years ago in India Debian hang around with other FLOSS events. Now its DebUtsav giving chance to other FLOSS people to meet around Debian.

As the usual way of DebUtsav, this one also was two day event with separate track for Debian related talks and for general FLOSS talk. I gave a talk about Debian LTS project. On first day evening some speakers and organizers gathered for dinner.

Its funny that most of the Debian people gathered there were contributing/contributed to Ruby and JavaScript team . There is a strong reason for that. All the contributors to Debian from India after 2014 were branched out from a single person who do mostly Ruby and JS - Pirate Praveen. You can expect a blog post from him about Debutsav. He is contesting in upcoming Lok Sabha Elections and quite busy with that.

On second day there were talks from SFLC - Digital Security and Privacy. Srud conducted a interactive session with topic Gender diversity in FLOSS projects. We reserved afternoon sessions for Bug Squashing Party and introducing packaging tutorial to newcomers. All together it was a wonderful gathering. I also met isaagar whom with I have corresponded in matrix a lot but finally able to meet him IRL.

Special appreciation to Hamara Linux for sponsoring the event.They are becoming the de facto sponsors of every Debian events in India.

With almost year passed since the previous 0.1.0 release, a nice new release of the tint package arrived on CRAN today. Its name expands from tint is not tufte as the package offers a fresher take on the Tufte-style for html and pdf presentations.

This version adds new features, and a new co-author. Jonathan Gilligan calmly and persistently convinced me that there was ‘life beyond Roboto’ and I overcame the reluctance to offer other fonts. So now we have two additional reference implementations for Lato and Garamond which look stunning, as well as generally enhanced support for fonts, font families and entire LaTeX templates all via the standard YAML headers.

A screenshot for Lato follows:

And another for garamond:

The full list of changes is below.

Changes in tint version 0.1.1 (2019-03-30)

• The two pdf styles have been extended allowing more flexible LaTeX customization particularly for fonts but also link colour. (Jonathan in #30)

• Two example documents where added pre-rendered (and not as vignettes to keep processing lighter)

• Documentation for the HTML style was updates (Jonathan in #30).

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the tint page.

For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Today, we had a little get-together of DDs in València, Spain, with some other DDs.

Most of us were here to attend the Internet Freedom Festival (IFF), plus Héctor and Filippo, who are locals. We missed some DDs (because in a 2500+ people gathering... Well, you cannot ever find everybody you are looking for!) so, sorry guys for not having you attend!

Sadly, we have no further report than having enjoyed a very nice dinner. No bugs were closed, no policy was discussed, no GRs were drafted, no cabals were hatched.

AttachmentSize deb_iff.jpg2.97 MB deb_iff_mini.jpg180.18 KB

For people who use Debian as docker base image...

• Do not use ftp.debian.org as your apt line in docker image, use deb.debian.org instead. It chooses near mirror where users are
• If you use old Jessie, use cdn-fastly.deb.debian.org instead of deb.debian.org. Since old apt in Jessie cannot handle redirect
• Do not use httpredir.debian.org, it is obsolete

A few months ago, we've seen how to write a filtering syntax tree in Python. The idea behind this was to create a data structure — in the form of a dictionary — that would allow to filter data based on conditions.

Our API looked like this:

>>> f = Filter( {"and": [ {"eq": ("foo", 3)}, {"gt": ("bar", 4)}, ] }, ) >>> f(foo=3, bar=5) True >>> f(foo=4, bar=5) False

While such a mechanism is pretty powerful to use, the input data structure format might not be user friendly. It's great to use, for example, with a JSON based REST API, but it's pretty terrible to use for a command-line interface.

A good solution to that problem is to build our own language. That's called a DSL.

Building a DSL

What's a Domain-Specific Language (DSL)? It's a computer language that is specialized to a certain domain. In our case, our domain is filtering, as we're providing a Filter class that allows to filter a set of value.

How do you build a data structure such as {"and": [{"eq": ("foo", 3)}, {"gt": ("bar", 4)}]} from a string? Well, you define a language, parse it, and then convert it to the right format.

In order to parse a language, there are a lot of different solutions, from implementing manual parsers to using regular expression. In this case, we'll use lexical analsysis.

First Iteration

Let's start small and define the base of our grammar. That should be something simple, so we'll go with <identifier><operator><value>. For example "foobar"="baz" is a valid sentence in our grammar and will conver to {"=": ("foobar", "baz")}.

The following code snippet leverages pyparsing for parsing the string and specifying the grammar:

import pyparsing identifier = pyparsing.QuotedString('"') operator = ( pyparsing.Literal("=") | pyparsing.Literal("≠") | pyparsing.Literal("≥") | pyparsing.Literal("≤") | pyparsing.Literal("<") | pyparsing.Literal(">") ) value = pyparsing.QuotedString('"') match_format = identifier + operator + value print(match_format.parseString('"foobar"="123"')) # Prints: # ['foobar', '=', '123']

With that simple grammar, we can parse and get a token list composed of our 3 items: the identifier, the operator and the value.

Transforming the Data

The list above in the format [identifier, operator, value] is not really what we need in the end. We need something like {operator: (identifier, value)}. We can leverage pyparsing API to help us with that.

def list_to_dict(pos, tokens): return {tokens[1]: (tokens[0], tokens[2])} match_format = (identifier + operator + value).setParseAction(list_to_dict) print(match_format.parseString('"foobar"="123"')) # Prints: # [{'=': ('foobar', '123')}]

The parseString method allows to modify the returned value of a grammar token. In that case, we transform the list of the dict we need.

Plugging the Parser and the Filter

In the following code, we'll reuse the Filter class we wrote in our previous post. We'll just add the following code to our previous example:

def parse_string(s): return match_format.parseString(s, parseAll=True)[0] f = Filter(parse_string('"foobar"="baz"')) print(f(foobar="baz")) print(f(foobar="biz")) # Prints: # True # False

Now, we have a pretty simple parser and a good way to build a Filter object from a string.

As our Filter object supports complex and nested operations, such as and and or, we could also add it to the grammar — I'll leave that to you reader as an exercise!

Building your own Grammar

pyparsing makes it easy to build one's own grammar. However, it should not be abused: building a DSL means that your users will have to discover and learn it. If it's way different that what they know and already exists, it might be cumbersome for them.

Finally, if you're curious and want to see a real world usage, Mergify condition system leverages pyparsing to implement its parser. Check it out!

Changes

• icns: https, copyright notice
• myrepos: typo
• spelling dictionaries: codespell (1 2 3 4 5 6 7), lintian (1 2 3 4 5 6 7)
• Debian derivatives census: explicitness, cleanup
• Debian package tracker: naming
• Debian LE: typo
• Debian wiki pages: ChrootOnAndroid, DebConf/20, DebianMedTodo, DebianUnstable, Derivatives/Census (QA, AlienVault-OSSIM, Aptosid), DesktopEnvironment, Diagrams, Glossary, i3, InstallingDebianOn (Cavium/ThunderXStation, Microsoft/Windows/SubsystemForLinux), LocalGroups, qa.debian.org, Teams/Dpkg/Spec/InstallBootstrap, Wayland
• Wikipedia: Message-ID

Issues

• Incorrectness in Ubuntu archive
• Template handling Linux
• Underlinking in libtheora0
• Broken symlinks in android-sdk
• Overprotection in apparmor-profiles-extra
• Missing info in chooseafoundation.com (1 2)

Review

• Spam: reported 233 Debian mailing list posts
• Debian wiki: RecentChanges for the month
• Debian screenshots:
  • approved a2ps amphetamine birdtray eom ewipe ffrenzy fonts-eurofurence fractgen gnat-gps gnome-control-center gnome-tweaks gnujump goodvibes gpa how-can-i-help jumpnbump launchy lemonldap-ng lsmount lugaru mate-applets mate-backgrounds nvtop pdfarranger screenfetch seahorse-adventures systemsettings tilix wings3d xscavenger
  • rejected coyim (logo), jumpnbump (another game), anfo/andi (help output), liblemonldap-ng-portal-perl (same as lemonldap-ng ones), gnome-tweaks/gnome-terminal (not Debian)
  • rejected deletion of xgalaga++/phpbb3/viewvc (spam reason), qbittorrent/setbfree/aptitude (nonsense reason), gnome-terminal (incorrect comment doesn't justify deletion)

Administration

• Debian wiki: update email addresses, whitelist email addresses, whitelist domains
• Debian security tracker: merge patches

Communication

• Initiate discussions about the status of ArcheOS, Canaima
• Invite Mentor Embedded Linux (Debian edition), AlienVault OSSIM to the Debian derivatives census
• Respond to queries from Debian users and developers on the mailing lists and IRC

Sponsors

All work was done on a volunteer basis.

SAMD21 USB vs Windows 7

I'm mostly used to USB being really hard to get working on a new SoC, everything from generating a stable 48MHz clock to diving through thousands of register definitions to get the device programmed to receive that first SETUP packet. However, I'm used to having that part be the hardest section of the work, and once the first SETUP packet has been received and responded to successfully, it's usually down hill from there.

Not this time.

I've written about Snek on the SAMD21G18A before, and this is about the same board. USB on this device is medium-complicated, as the device supports both host and device modes, plus has a range of 'optimizations' which always makes simple operation harder. It took a few hours of hacking to get SETUP packets flowing, but after that (at least when talking to Linux and Mac OS X), the rest of the USB driver was pretty simple.

Enter Windows 7

I'm pushing towards a Snek 1.0 release and was testing snekde on Windows 7. It's working great with the classic Arduino Duemilanove, but when I plugged in the Metro M0 board, it got stuck after I typed one character. "That's Odd", I thought.

I figured it'd be a simple matter of a stuck interrupt or other minor mistake in the SAMD21 USB driver that I wrote. So, I broke out my trusty Beagle USB analyzer to see where the USB link was getting stuck.

IN-NAK ... IN DATAx ...

USB is an odd protocol; data from the device to the host has to sit in the device waiting for the host to come and ask for it. When the device is in use, the host polls for data by sending an IN packet. When there's no data to send back, the device sends a NAK reply. When there is data, the device sends a DATAx packet and the host replies with an ACK packet.

In my case, the host sends thousands of IN packets waiting for data, and the device responds with an equally huge number of NAK packets. The first time data was queued from the device to the host, the device responded to the IN packet with a DATAx packet and the host ACK'd that. After that, the host never sent another IN packet again. It would happily send it's own data using OUT packets, and the device would receive that data, and of course the usual stream of SOF (start of frame) packets were streaming along. But, not a single IN packet to be seen.

Differential Debugging

Well, I've got a lot of USB devices around here, so I hooked up one of our TeleBTv3.0 devices. That worked just fine, which was good as we've sold hundreds of those and it would kinda suck to discover that some Windows boxes weren't compatible.

A visual examination of the traces as seen captured by the Beagle analyzer didn't show anything obvious. But, it's often the little details that break things.

So, I hacked up the SAMD21 board to appear to be the same device as the TeleBT -- same VID/PID, same names, same serial number. Everything.

Now windows can't seem to tell the difference. It uses the same COM port for both at least.

I devised a simple test — plug-in the device, start PuTTY and then type two characters ('a', or 0x61). Because both devices echo whatever you send to them, this means I should get two characters back. Because they're typed separately, those two characters will be sent in separate OUT transactions, and the echos should be sent back in two IN transactions.

I captured traces from both devices:

TeleBT-v3.0 (STM32L151):

• Binary Beagle capture file (.tdc)
• CSV format
• Trimmed of timing information

Metro M0 (SAMD21G18A):

• Binary Beagle capture file (.tdc)
• CSV format
• Trimmed of timing information

The 'trimmed' versions elide timing and packet sequence information which can't be easily replicated exactly between the two tests; that "can't" matter, at least according to my understanding of USB. With those versions, I can do a text diff of the packet traces to find that, aside from a different number of SOF and IN-NAK transactions, the only difference appears at the end

$ diff -u stm32l.trim samd21.trim | tail +231 0 1 B 01 04 OUT txn 61 1 3 B 01 04 OUT packet E1 01 BA 1 4 B 01 04 DATA0 packet C3 61 81 57 1 1 B 01 04 ACK packet D2 -0 1 B 01 05 IN txn [57536 POLL] 61 -1 01 05 [57536 IN-NAK] +0 1 B 01 05 IN txn [50387 POLL] 61 +1 01 05 [50387 IN-NAK] 1 3 B 01 05 IN packet 69 81 0A 1 4 B 01 05 DATA0 packet C3 61 81 57 1 1 B 01 05 ACK packet D2 -0 [1004 SOF] [Frames: 853 - 1856] +0 [2000 SOF] [Frames: 138 - 89] [Periodic Timeout] +0 [2000 SOF] [Frames: 90 - 41] [Periodic Timeout] +0 [572 SOF] [Frames: 42 - 613] 0 1 B 01 04 OUT txn 61 1 3 B 01 04 OUT packet E1 01 BA 1 4 B 01 04 DATA1 packet 4B 61 81 57 1 1 B 01 04 ACK packet D2 -0 1 B 01 05 IN txn [83901 POLL] 61 -1 01 05 [83901 IN-NAK] -1 3 B 01 05 IN packet 69 81 0A -1 4 B 01 05 DATA1 packet 4B 61 81 57 -1 1 B 01 05 ACK packet D2 -0 01 01 [16 IN-NAK] [Periodic Timeout] -0 01 05 [178185 IN-NAK] [Periodic Timeout] -0 [2000 SOF] [Frames: 1857 - 1808] [Periodic Timeout] -0 01 01 [16 IN-NAK] [Periodic Timeout] -0 01 05 [147487 IN-NAK] [Periodic Timeout] -0 [2000 SOF] [Frames: 1809 - 1760] [Periodic Timeout] -0 [474 SOF] [Frames: 1761 - 186] -0 01 05 [34876 IN-NAK] -0 ! 01 05 [1 ORPHANED] -1 U 01 05 [1 IN] -0 01 01 [16 IN-NAK] -0 Capture stopped [Sun 31 Mar 2019 02:25:32 PM PDT] +0 [2000 SOF] [Frames: 614 - 565] [Periodic Timeout] +0 [1163 SOF] [Frames: 566 - 1728] +0 Capture stopped [Sun 31 Mar 2019 02:36:23 PM PDT]

You can see both boards receiving the first 'a' character and then send that back. Then both boards receive the second 'a' character, but only the stm32l gets the IN packets which it can respond with the DATAx packet containing the 'a' character. The samd21 board gets only SOF packets.

Next Steps?

I'm heading out of town on Tuesday to help with the NASA Student Launch, so I think I'll let this sit until I get back. Maybe I'll come up with a new debugging idea, or maybe I'll hear about a fancier USB monitoring device that might capture details that I'm missing.

Anyone with suggestions or comments is welcome to send them along; I'd like to get this bug squashed and finish the rest of the Snek 1.0 release process.