Agreguesi i feed

Here’s what happened in the Reproducible Builds effort between Sunday December 9th and Saturday December 15th 2018:

• We had an extremely rewarding and productive Reproducible Builds summit in Paris, France. Thank you to all who participated and helped organise the event.

  A full and in-depth report about the event is being prepared but in the meantime Jelle van der Waa published a blog post with a brief report of the event from an Arch Linux perspective and Bernhard M. Wiedemann posted his own report to his openSUSE blog.

  We would also like to thank Mozilla, Ford Foundation, Open Technology Fund, Google Open Source Research, Debian and Protocol Labs for sponsoring the event and for Aspiration Tech for their facilitation work.

  T-shirts with the Reproducible Builds logo were also distributed at the event. If you are a contributor to Reproducible Builds and want a t-shirt, please send an email to Holger as outlined in this mail. The design is freely available in the reproducible-promo-material Git repository if you wish to print one yourself.

• Following on from a session at our summit, Bernhard M. Wiedemann started a thread on our mailing list discussing Reproducible Builds through a mathematical formalism perspective.

• Chris Lamb updated the Lintian quality-assurance tool for Debian packages to check for files that reference the build path. (#916021)

• Bernhard M. Wiedemann noticed that Qt’s rcc had finally merged the patch to use SOURCE_DATE_EPOCH for reproducible .qrc files. In addition, Chris Lamb filed Debian bug #916425 with a patch for python-ruffus to fix a reproducibility issue there too.

• Jelle van der Waa, inspired by Bernhard’s recent RPM ecosystem update, posted an equivalent update of Reproducible Builds in Arch Linux to our mailing list.

• Bernhard M. Wiedemann submitted strip-nondeterminism-1.0.0 to openSUSE.

• Chris Lamb started to merge in new style guide to our project website, adding an example “work in progress” typography page […] and also corrected some references to “non-determinism” […]. In addition, Holger Levsen updated our project’s website to add Microsoft as an attendee of our summit in Paris […] and added a group photo taken by Morten Linderud […].

diffoscope development

diffoscope is our in-depth “diff-on-steroids” utility which helps us diagnose reproducibility issues in packages. There was considerable activity this week, including contributions from:

• Chris Lamb:
  • Fix a test_mozzip_compressed_files test failure under Alpine Linux. (#916353)
  • Calculate the path to a test .icc file using data() to avoid a “Fixtures are not meant to be called directly” warning/error. (#916226)
  • Drop debbindiff Breaks/Replaces. […]
  • Use File.file_header to simplify file detection. ([…], […] & […])
  • Correct a “positives” typo. […]
• Holger Levsen:
  • Clarify that upstream issues should be now reported via Salsa. […]
• Joachim Breitner:
  • Add support for comparing WebAssembly modules. (!17)
• Mattia Rizzolo:
  • Try matching for MozillaZipFile before ZipFile. […]

Chris Lamb also overhauled the diffoscope.org website, updating the design […] as well as informing users that they should file issues on salsa […] and adding corresponding link to our Salsa registration instruction […].

Test framework development

There were a number of updates to our Jenkins-based testing framework that powers tests.reproducible-builds.org this week, including:

• Chris Lamb:
  • Add a PureOS default installation package set. (!24)
  • Avoid double spaces in IRC output. (!23)
• Mattia Rizzolo:
  • Arch Linux-specific changes:
    • Define DISTROID variable before using it. […]
    • Check for the correct distribution when querying the database. […]
  • Debian-specific changes:
    • Only build the index_notify page for Debian. […]
    • Adjust the APT pinning on arm64 to install Munin from backports.debian.org. […]
  • openSUSE-specific changes:
    • Decode the input before giving it to json […]
    • Notify IRC and Bernhard of job failures. […]
    • Add a job importing the openSUSE status into the database to build some HTML pages. […]
    • Use --ignore-missing-files to not warn about missing build logs. […]
  • Misc/generic changes:
    • Add openSUSE, Arch Linux and Alpine Linux to the list of “known” distributions. […]
    • Convert the stats_build.build_duration field type from TEXT to INTEGER. […]
    • Add a distribution field to the sources. […]
    • Change the data type used to store timestamps. […]
    • Various bits node maintenance. (eg. […])

This week’s edition was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Intro

Welcome to another of my monthly summaries on my work in the free software world. My mission is to make engineering and science available for everyone, and Debian, the Universal Operating System, is my weapon of choice.

I received some feedback on my last post that I made it seem I would be stepping away from my role maintaining FreeCAD and other packages on the Debian Science Team, which was an unfortunate miscommunication on my part. Mainly, I just would like to reduce the proportion of my overall free software time on it, from its current amount, nearly 100%, to a roughly even 1/3rd split between Debian Science, FreeCAD, and PostCAD. The latter is a promising personal project to make an OpenCASCADE-powered CAD extension for PostgreSQL, bringing support for CAD file formats, datatypes, and algorithms to the powerful Postgres ecosystem, similar to what PostCAD has done for geospatial analysis. This could serve as a backend for both FreeCAD in the short term, and in the long term, it could power a web-based version of FreeCAD, perhaps with some Django-powered middleware serving a REST API for some WebGL-based frontend.

So, besides summarizing my work this month, I also plan to give a synopsis on my Debian packaging, for both what's in-progress and on my wishlist. As you'll see, it's quite extensive. My hope is that by whittling away at both lists, my Debian Science work can focus more on maintenance of existing packages, and free up some time for other things.

Sponsors

My work on Debian Science, FreeCAD, and PostCAD is supported by my patrons on Patreon <https://patreon.com/kkremitzki>. While I had created a Liberapay a while back, I never got any traction with it, and I found out recently that it was because I had not set it up to actually receive payments. So, if you don't like Patreon for whatever reason, you can also support me on Liberapay. Just to round things out, if you don't like either of those platforms, you can also help support my work via PayPal.

Transition for coin3

Coin3D is a scene graph library and a fundamental part of FreeCAD. Unfortunately, it hasn't had a release since 2011, but development has been picking up recently, so it's likely that a release is not too far off. Even without a formal release, several improvements have been made including CMake support, so it's time to prepare a transition in advance of the Debian 10 release. Luckily, Leo Palomo-Avellaneda has taken the initiative of getting this transition started.

Currently, the new Coin3D package is available in Debian Experimental as we prepare the reverse-dependencies to build against it. For FreeCAD, we directly depend on Pivy, which are Python bindings for Coin. Pivy, in turn, depends on Coin and SoQt, which is a Qt GUI component toolkit library for Coin. A pre-release version of SoQt is also being packaged since, like Coin, CMake support has been added, as well as support for Qt 5.

Unfortunately, I've been grinding my gears on building Pivy against the new Coin and SoQt for a good part of this month, which is especially troublesome since FreeCAD's transition to Python 3 is blocked by the upload of a new Pivy, which I prepared earlier this year.

With any luck, I'll be able to help Leo finish the Coin and SoQt transition and have Pivy prepared in December.

Gmsh Update

Gmsh is a 3D finite element mesh generator with built-in pre- and post-processing facilities. It's also one of the main meshers used by FreeCAD's Finite Element Modeling Workbench, besides Netgen.

Currently Debian has Gmsh 3.0.6, but a new major version was released in August. I've already prepared this new major version for testing in FreeCAD's Community Extras PPA, but I hadn't cleaned up the packaging yet to submit to Debian. However, in November, there were two point releases, 4.0.5 and 4.0.6, so I wasn't able to complete this work this month, but I'm sure it'll be done in early December.

One major piece of information regarding Gmsh 4 is a change in the API: the libjava-gmsh3 package in Debian was never meant to be a public API, and so Gmsh upstream has requested that we no longer ship it. To offset this, there have been refinements for the actual public API, which officially comes in C, Python 3 and (new) Julia flavors. However, I haven't found much information on Julia packaging in Debian, so I'll likely hold off on that package.

PySide 2 Rebuild & PPA Plans

Since FreeCAD is an LGPL-licensed Qt project, we must use PySide and not PyQt to do Qt things with Python. Because of this, the FreeCAD migration to Qt 5 on Debian was blocked by the packaging of PySide 2, which was completed by Freexian SARL over the summer. In Debian, we now have a Qt 5-enabled FreeCAD, although our daily builds PPA is still using Qt 4.

Besides the Qt 4->5 transition, we're also finishing up a Python 2->3 transition. At the end of the summer I published a freecad-python3 package in the PPA which also used Qt 5. However, it wasn't really fully usable, moreso a proof-of-concept that such a build indeed buildable.

At this point, the Debian FreeCAD package has begun to diverge from the FreeCAD PPAs; besides Qt 5 builds not being available currently, the Debian package has also been split into several packages (e.g. libfreecad, freecad-common, etc. packages) in order to better comply with Debian Policy and the Filesystem Hierarchy Standard.

So, there's a bit of work to do to catch the PPAs up. First, the package split needs to be done. Then, I need to upload an alternative freecad-daily package for Qt 5 builds, separate from Python 3. Once that is done and has undergone some testing, freecad-daily can be replaced by it, and it in turn can be replaced by a freecad-python3 package for further testing. Since FreeCAD's 0.18 release is imminent, I'll need to get this taken care of during December, so stay tuned.

Packaging in Progress

My first packaging list is all the software I've already started packaging. For some, it's almost complete, and for others, I've only just begun.

The purpose of these lists is not to give status updates, but to announce what all I'm interested in to anyone reading this, and to give an idea of how much packaging work I have in mind to improve this usage of Debian.

cantera Homepage, GitHub Cantera is an open-source suite of tools for problems involving chemical kinetics, thermodynamics, and transport processes.

I had this package working and waiting to be sponsored, but it looks like it currently fails to build from source, so this just requires some maintenance.

coolprop Homepage, GitHub CoolProp is a thermophysical property database and wrappers for a selection of programming environments. It offers similar functionality to REFPROP, although CoolProp is open-source and free.

This package was previously building completely, but failing when one would attempt to do an import coolprop. Now that it's been a while since I worked with it, it seems to be failing to build.

elmerfem Homepage, GitHub Elmer is a finite element software for numerical solution of partial differential equations. Elmer is capable of handling any number of equations and is therefore ideally suited for the simulation of multiphysical problems. It includes models, for example, of structural mechanics, fluid dynamics, heat transfer and electromagnetics. Users can also write their own equations that can be dynamically linked with the main program.

This was previously in Debian but removed due to abandonment, so a great deal of the work is already done, but it also requires quite a bit of updating to current Debian standards.

if97 PDF Standard, GitHub Open-source C++ implementation of the IAPWS-IF97 equations to calculate properties of the pure water substance.

This is a dependency of CoolProp, and I already have it packaged and waiting for sponsorship at https://salsa.debian.org/science-team/if97.

ifcopenshell Homepage, GitHub IfcOpenShell is an open source (LGPL) software library that helps users and software developers to work with the IFC file format. The IFC file format can be used to describe building and construction data. The format is commonly used for Building Information Modelling. IfcOpenShell uses OpenCASCADE internally to convert the implicit geometry in IFC files into explicit geometry that any software CAD or modelling package can understand.

I already have this packaged and awaiting sponsorship at https://salsa.debian.org/kkremitzki-guest/ifcopenshell.

It's also available on the FreeCAD Community Extras PPA.

ifcplusplus Homepage, GitHub IfcPlusPlus is an open source C++ class model, as well as a reader and writer for IFC files in STEP format. It features easy and efficient memory management using smart pointers, a parallel reader for fast parsing on multi-core CPU's, and a simple IFC viewer application using Qt and OpenSceneGraph.

I already have this packaged and awaiting sponsorship at https://salsa.debian.org/kkremitzki-guest/ifcplusplus.

It's also available on the FreeCAD Community Extras PPA.

opencamlib Homepage, GitHub OpenCAMLib (OCL) is a C++ library with Python bindings for creating 3D toolpaths for CNC-machines such as mills and lathes.

I already have this packaged and awaiting sponsorship at https://salsa.debian.org/science-team/opencamlib.

It's also available on the FreeCAD Community Extras PPA.

openvoronoi Homepage, GitHub 2D voronoi diagram for point and line-segment sites using incremental topology-oriented algorithm. C++ with Python bindings.

I already have this packaged and awaiting sponsorship at https://salsa.debian.org/kkremitzki-guest/openvoronoi.

It's also available on the FreeCAD Community Extras PPA.

projectchrono Homepage, GitHub C++ library for multi-physics simulation. The applications areas in which Chrono is most often used are vehicle dynamics, robotics, and machine design. In vehicle dynamics, Chrono has mature support for tire/terrain interaction modeling and simulation.

I've only roughly begun packaging this, and I'm already tired of typing libprojectchrono. Anyway, it's a rather large set of components which will be broken up into several packages. Luckily, things are done in a pretty normal way so I don't imagine this will be difficult to finish packaging, just a little time-costly.

smesh GitHub A stand-alone library of the mesh framework from the Salome Platform

I've gotten this standalone version of SMESH packaged and awaiting sponsorship at https://salsa.debian.org/kkremitzki-guest/salome-smesh. Eventually, I want to package the entire Salome Platform, but it's extremely large and really several source packages. Packaging this as an intermediate step allows us to remove SMESH from FreeCAD's included sources.

It's also available on the FreeCAD Community Extras PPA.

xcfem Homepage, GitHub XC is an open source FEA program designed to solve structural analysis problems.

This library is supposed to be an alternative to the not-quite-freely licensed OpenSees, which is used in seismic research and analysis. There has been some interest in the FreeCAD forums about using this, so I'm beginning packaging it in advance. However, it seems a bit complicated as it requires multiple sources, the GitHub xcfem/xc repo as well as xcfem/xc_utils.

Wishlist Packages 2geom Homepage, GitLab lib2geom (2Geom in private life) was initially a library developed for Inkscape but will provide a robust computational geometry framework for any application. It is not a rendering library, instead concentrating on high level algorithms such as computing arc length.

I looked at this package and it seemed like it will be straightforward to package, and with the parent project's popularity, someone else may get to it first.

bimserver Homepage, GitHub The Building Information Model server (short: BIMserver) enables you to store and manage the information of a construction (or other building related) project. Data is stored in the open standard IFC. The BIMserver is not a fileserver, but it uses a model-driven architecture approach. This means that IFC data is stored in an underlying database. The main advantage of this approach is the ability to query, merge and filter the BIM-model and generate IFC files on the fly.

The integration of BIM with FreeCAD is a very promising endeavor, and letting FreeCAD be the client in a client-server model provides many potential benefits. (This is the reason I'm working on PostCAD.) Packaging BIMserver is a natural decision, then. However, it's a Java application, which I have little experience with language-wise and none in terms of packaging it in Debian, so this one has a bit of a difficulty associated with it.

cadquery Homepage, GitHub CadQuery is an intuitive, easy-to-use python based language for building parametric 3D CAD models. CadQuery is for 3D CAD what jQuery is for javascript. Imagine selecting Faces of a 3d object the same way you select DOM objects with JQuery!

CadQuery is an interesting project which actually makes use of FreeCAD, and indeed FreeCAD even has a CadQuery Workbench. This would be nice to package as a way of extending the FreeCAD ecosystem on Debian.

Unfortunately, CadQuery 2 is planning on moving away from FreeCAD to PythonOCC, which is based on the now behind-the-times OpenCASCADE Community Edition fork, based on OpenCASCADE 6.9.1; FreeCAD and other projects are moving back to the mainline OpenCASCADE Technology project which is about to release version 7.4.0. It would be nice if both CadQuery and FreeCAD could instead move to use PyOCCT as a middle-layer between itself and OpenCASCADE.

libreoffice-code-highlighter Homepage, GitHub This extension highlights the code snippets for over 350 languages in LibreOffice.

I have packaged a LibreOffice extension before, and it was fairly easy, so I expect this one will be too. However its priority is rather low.

lib3mf Homepage, GitHub Lib3MF is a C++ implementation of the 3D Manufacturing Format file standard.

This seems like a straightforward library to package, but there is no pressing need as FreeCAD does not support it yet.

muesli Homepage, BitBucket MUESLI, a Material UnivErSal LIbrary, is a collection of C++ classes and functions designed to model material behavior at the continuum level. Developed at IMDEA Materials, it is available to the material science and computational mechanics community as a suite of standard models and as a platform for developing new ones.

This seems like a great candidate package for Debian Science but I have had some difficulty building it, which I need to conquer before packaging can begin.

cling Homepage, GitHub Cling is an interactive C++ interpreter, built on top of Clang and LLVM compiler infrastructure. Cling realizes the read-eval-print loop (REPL) concept, in order to leverage rapid application development. Implemented as a small extension to LLVM and Clang, the interpreter reuses their strengths such as the praised concise and expressive compiler diagnostics.

cling is an incredible project which should have been packaged already. Hopefully someone else gets to it first.

dpkg-licenses GitHub A command line tool which lists the licenses of all installed packages in a Debian-based system (like Ubuntu)

This is a small script which gives a summary of the licenses used by the installed packages on your system--a good way to audit packages, e.g. forbidding AGPL.

landlab Homepage, GitHub

Landlab is a Python-based modeling environment that allows scientists and students to build numerical landscape models. Designed for disciplines that quantify earth surface dynamics such as geomorphology, hydrology, glaciology, and stratigraphy, it can also be used in related fields.

Landlab provides components to compute flows (such as water, sediment, glacial ice, volcanic material, or landslide debris) across a gridded terrain. With its robust, reusable components, Landlab allows scientists to quickly build landscape model experiments and compute mass balance across scales.

Landlab is another interesting Debian Science candidate but I have no pressing need to package it.

nikola Homepage, GitHub A static website and blog generator, written in Python.

Nikola is what I use to create this blog, but it's somewhat fast moving and a slow maintainer in Debian previously caused problems, so I don't want to pick this up until I've leveled up my package maintenance.

osifont Homepage, GitHub In some European countries, CAD projects must have font which conform to IS0 3O98 specification. Commercial CADs has this font, but free CADs not. There is no available free font yet, so this project will fix this. This font will be created completely from the scratch. Font is created with free tools like FontForge, Inkscape, Gimp. Font is available under 3 licences: GNU GPL licence version 3 with GPL font exception, GNU GPL licence version 2 with GPL font exception, GNU LGPL licence version 3 with GPL font exception.

This is a bundled font with FreeCAD, so I'd like to separate into its own package. However, the need to package it is not pressing, so I haven't picked it up.

pigpio Homepage, GitHub pigpio is a C library for the Raspberry which allows control of the General Purpose Input Outputs (GPIO).

This is an important tool for teaching with Raspberry Pi's and should be packaged as soon as possible, I've just had more pressing concerns.

piscope Homepage, GitHub A logic analyser (digital waveform viewer). piscope uses the services of the pigpio library. pigpio needs to be running on the Pi whose gpios are to be monitored.

Being able to see the waveform of a GPIO pin on a Raspberry Pi is incredibly useful for teaching robotics and electrical engineering classes with them. This also needs to be packaged.

pyocct GitHub The pyOCCT project provides Python bindings to the OpenCASCADE 7.2.0 geometry kernel and SMESH 8.3.0 meshing library via pybind11. Together, this technology stack enables rapid CAD/CAE application development in the popular Python programming language.

This is a very promising library for Python OpenCASCADE development, so I'd like to get it packaged, but it's blocked by getting SMESH packaged.

pyray GitHub A 3D rendering library written completely in Python.

A promising library for integrating raytracing functionality directly into FreeCAD, and for general raytracing in Python.

quarter BitBucket Quarter is a light-weight glue library that provides seamless integration between the Coin high-level 3D visualization library and Qt's 2D user interface library. The functionality in Quarter revolves around QuarterWidget, a subclass of QGLWidget. This widget provides functionality for rendering of Coin scenegraphs and translation of QEvents into SoEvents. Using this widget is as easy as using any other QWidget.

FreeCAD already uses an included (and slightly modified) copy of Quarter in its source, so I'd like to package Quarter in a standalone fashion as part of moving FreeCAD away from bundled copies in its source.

rebound Homepage, GitHub REBOUND is an N-body integrator, i.e. a software package that can integrate the motion of particles under the influence of gravity. The particles can represent stars, planets, moons, ring or dust particles. REBOUND is very flexible and can be customized to accurately and efficiently solve many problems in astrophysics.

This seems like a really great library to have in Debian Science, but it's not a priority.

sqlint GitHub

SQLint is a simple command-line linter which reads your SQL files and reports any syntax errors or warnings it finds.

At this stage, SQLint checks SQL against the ANSI syntax, and uses the PostgreSQL SQL parser to achieve this. In time, we hope to add support for non-standard SQL variants (e.g. MySQL). Contributions are welcome.

This would be a very useful utility to have in Debian, but I always write SQL without flaw the first try. (wink)

swatmodel Homepage, GitHub The Soil & Water Assessment Tool is a small watershed to river basin-scale model used to simulate the quality and quantity of surface and ground water and predict the environmental impact of land use, land management practices, and climate change. SWAT is widely used in assessing soil erosion prevention and control, non-point source pollution control and regional management in watersheds.

SWAT is a powerful research tool in agricultural engineering, among several others I'm interested in eventually packaging for Debian. The planned package will be based on a CMake-enabled fork of the upstream source, which is built with Intel's Fortran compiler by default and also had to be adapted for gfortran.

wger Homepage, GitHub Self hosted FLOSS fitness/workout and weight tracker written with Django

This is a very promising application which could be used as both a fitness tracker as well as a weight/nutrition tracker, something along the lines of a self-hosted MyFitnessPal. However, my other packaging priorities outweigh this at the moment.

Conclusion

So, there you have it! My mostly complete list of in-progress and wishlist items for Debian packaging. If you have any feedback on packages on the list, or want to get in touch with me, you can find me on Twitter or send me an email at kurt at kwk.systems. I'll also be starting to stream my Debian & FreeCAD work very soon, subscribe to me on Twitch to get notified when I go live.

Heya,

Been quite some time since I wrote about anything. This time, it is Debutsav. When it comes to full-fledged FOSS conferences, I usually am an attendee or at most a speaker. I have given some sporadic advices and suggestions to few in the past, but that was it. However, this time I played the role of an organizer.

DebUtsav Kochi is the second edition of Debian Utsavam, the celebration of Free Software by Debian community. We didn’t name it MiniDebConf because it was our requirement for the conference to be not just Debian specific, but should include general FOSS topics too. This is specifically because our target audience aren’t yet Debian-aware to have a Debian-only event. So, DebUtsav Kochi had three tracks - one for general FOSS topics, one for Debian talks and one for hands-on workshops.

As a disclaimer, the description about the talks below are what I gained from my interaction with the speakers and attendees, since I wasn’t able to attend as many talks as I would’ve liked, since I was busy with the organizing stuff.

The event was organized by Free Software Community of India, whom I represented along with Democratic Alliance for Knowledge Freedom (DAKF) and Student Developer Society (SDS). Cochin University of Science and Technology were generous enough to be our venue partners, providing us with necessary infrastructure for conducting the event as well as accommodation for our speakers.

The event span across two days, with a registration count around 150 participants. Day 1 started with a keynote session by Aruna Sankaranarayanan, affiliated with OpenStreetMap. She has been also associated with GNOME Project, Wikipedia and Wikimedia Commons as well as was a lead developer of the Chennai Flood Map that was widely used during the floods that struck city of Chennai.

Sruthi Chandran, Debian Maintainer from Kerala, gave a brief introduction about the Debian project, its ideologies and philosophies, people behind it, process involved in the development of the operating system etc. An intro about DebUtsav, how it came to be, the planning and organizations process that was involved in conducting the event etc were given by SDS members.

After these common talks, the event was split to two parallel tracks - FOSS and Debian.

In the FOSS track, the first talk was by Prasanth Sugathan of Software Freedom Law Centre about the needs of Free Software licenses and ensuring license compliance by projects. Parallely, Raju Devidas discussed about the process behind becoming an official Debian Developer, what does it mean and why it matters to have more and more developers from India etc.

After lunch, Ramaseshan S introduced the audience to Project Vidyalaya, a free software solution for educational institutions to manage and maintain their computer labs using FOSS solutions rather than the conventional proprietary solutions. Shirish Agarwal shared general idea about various teams in Debian and how everyone can contribute to these teams based on their interest and ability.

Subin S showed introduced some nifty little tools and tricks that make Linux desktop cool, and improve the productivity of users. Vipin George shared about the possibility of using Debian as a forensic workstation, and how it can be made more efficient than the proprietary counterparts.

Ompragash V from RedHat talked about using Ansible for automation tasks, its advantages over similar other tools etc. Day 1 ended with Simran Dhamija talking about Apache SQOOP and how it can be used for data transformation and other related usecases.

In the afternoon session of Day 1, two workshops were also conducted parallel to the talks. First one was by Amoghavarsha about reverse engineering, followed by an introduction to machine learning using Python by Ditty.

We also had an informal discussion with few of the speakers and participants about Free Software Community of India, the services it provide and how to get more people aware of such services and how to get more maintainers for them etc. We also discussed the necessity of self-hosted services, onboarding users smoothly to them and evangelizing these services as alternatives to their proprietary and privacy abusing counterparts etc.

Day 2 started with a keynote session by Todd Weaver, founder and CEO of Purism who aims at developing laptops and phones that are privacy focused. Purism also develops PureOS, a Debian Derivative that consists of Free Software only, with further privacy enhancing modifications.

On day 2, the Debian track focused on a hands-on packaging workshop by Pirate Praveen and Sruthi Chandran that covered the basic workflow of packaging, the flow of packages through various suites like Unstable, Testing and Stable, structure of packages. Then it moved to the actual process of packaging by guiding the participants through packaging a javascript module that is used by GitLab package in Debian. Participants were introduced to the tools like npm2deb, lintian, sbuild/pbuilder etc. and the various debian specific files and their functionalities.

In the FOSS track, Biswas T shared his experience in developing keralarescue.in, a website that was heavily used during the Kerala Floods for effective collaboration between authorities, volunteers and public. It was followed by Amoghavarsha’s talk on his journey from Dinkoism to Debian. Abhijit AM of COEP talked about how Free Software may be losing against Open Source and why that may be a problem. Ashish Kurian Thomas shed some knowledge on few *nix tools and tricks that can be a productivity booster for GNU/Linux users. Raju and Shivani introduced Hamara Linux to the audience, along with the development process and the focus of the project.

The event ended with a panel discussion on how Debian India should move forward to organize itself properly to conduct more events, spread awareness about Debian and other FOSS projects out there, prepare for a potential DebConf in India in the near future etc.

The number of registrations and enthusiasms of the attendees for the event is giving positive signs on the probability of having a proper MiniDebConf in Kerala, followed by a possible DebConf in India, for which we have bid for. Thanks to all the participants and speakers for making the event a success.

Thanks to FOSSEE, Hamara Linux and GitLab for being sponsors of the event and thus enabling us to actually do this. And also to all my co-organizers.

A very special thanks to Kiran S Kunjumon, who literally did 99% of the work needed for the event to happen (as you may recall, I am good at sitting on a chair and planning, not actually doing anything. :D ).

Group Photo

I am extremely pleased to announce the public release of pwnedkeys.com – a database of compromised asymmetric encryption keys. I hope this will become the go-to resource for anyone interested in avoiding the re-use of known-insecure keys. If you have a need, or a desire, to check whether a key you’re using, or being asked to accept, is potentially in the hands of an adversary, I would encourage you to take a look.

Pwnage... EVERYWHERE

By now, most people in the IT industry are aware of the potential weaknesses of passwords, especially short or re-used passwords. Using a password which is too short (or, more technically, with “insufficient entropy”) leaves us open to brute force attacks, while re-using the same password on multiple sites invites a credential stuffing attack.

It is rare, however, that anyone thinks about the “quality” of RSA or ECC keys that we use with the same degree of caution. There are so many possible keys, all of which are “high quality” (and thus not subject to “brute force”), that we don’t imagine that anyone could ever compromise a private key except by actually taking a copy of it off our hard drives.

There is a unique risk with the use of asymmetric cryptography, though. Every time you want someone to encrypt something to you, or verify a signature you’ve created, you need to tell them your public key. While someone can’t calculate your private key from your public key, the public key does have enough information in it to be able to identify your private key, if someone ever comes across it.

So what?

The risk here is that, in many cases, a public key truly is public. Every time your browser connects to a HTTPS-protected website, the web server sends a copy of the site’s public key (embedded in the SSL certificate). Similarly, when you connect to an SSH server, you get the server’s public key as part of the connection process. Some services provide a way for anyone to query a user’s public keys.

Once someone has your public key, it can act like an “index” into a database of private keys that they might already have. This is only a problem, of course, if someone happens to have your private key in their stash. The bad news is that there are a lot of private keys already out there, that have either been compromised by various means (accident or malice), or perhaps generated by a weak RNG.

When you’re generating keys, you usually don’t have to worry. The chances of accidentally generating a key that someone else already has is as close to zero as makes no difference. Where you need to be worried is when you’re accepting public keys from other people. Unlike a “weak” password, you can’t tell a known-compromised key just by looking at it. Even if you saw the private key, it would look just as secure as any other key. You cannot know whether a public key you’re being asked to accept is associated with a known-compromised private key. Or you couldn’t, until pwnedkeys.com came along.

The solution!

The purpose of pwnedkeys.com is to try and collect every private key that’s ever gotten “out there” into the public, and warn people off using them ever again. Don’t think that people don’t re-use these compromised keys, either. One of the “Debian weak keys” was used in an SSL certificate that was issued in 2016, some eight years after the vulnerability was made public!

My hope is that pwnedkeys.com will come to be seen as a worthwhile resource for anyone who accepts public keys, and wants to know that they’re not signing themselves up for a security breach in the future.

I guess backwards compatibility is just another form of entropy. So this is all expected, but still…

Case in point, 512e hard drives. Wikipedia has a nice article on this, so I’ll skip the intro, and just go directly to my main point.

It’s 2018. End of 2018, more precisely. Linux has long supported 4K sectors, Windows since Win10 (OK, so recent), but my sampling on existing HDDs:

• All of Western Digital’s hard drives under its own brand (WD rainbow series) is using 512e (or 512n, for ≤4TB or so); not even the (recently retired) “Gold” series, top of the top, provides 4Kn.
• HGST has a large variety for the same size/basic model, but in Switzerland at least, all the 4Kn variants seem to be rare than unicorns; whereas the 512e are available in stock all over the place.

I tried to find HDDs ≥8TB with 4Kn (and ideally ISE), but no luck; mostly “zero available in stock, no availability at supplier, no orders”. Sure, Switzerland is a small market, but at the same time, exact same size/model but with 512e is either available, or already on order. I’ve seen at one shop a 512e model where the order backlog was around 200 HDDs.

I guess customer demand is what drives the actual stocking of models. So why on earth are people still buying hard drives using 512e format? Is it because whatever virtualisation backend they use doesn’t support yet 4Kn (at all or well)? I’ve seen some mentions of this, but they seemed to be from about 2 years ago.

Or is it maybe because most of HDDs are still used as boot drives? Very unlikely, I think, especially at high-end, where the main/only advantage is humongous size (for the price).

I also haven’t seen any valid performance comparisons (if any). I suppose since Linux knows the physical sector size, it can always do I/O in proper sizes, but still, a crutch is a crutch. And 4Kn allows going above 2TB limit for the old-style MBR partition tables, if that matters.

Side-note: to my surprise, the (somewhat old now) SSDs I have in my machine are supposedly “512n”, although everybody knows their minimal block size is actually much, much larger - on the order of 128KB-512KB. I/O size is not same as erase block size, but still, one could think they are more truthful and will report their proper physical block size.

Anyway, rant over. Let’s not increase the entropy even more…

as mentioned in my last blog post, I attended the Bug Squashing Party in bern two weeks ago. I think alltogether we were quite productive, as seen on the list of usertagged bugs. – here's my personal list of release-critical bugs that I touched at the BSP or afterwards:

• #860523 – python-dogtail: "python-dogtail: /bin/sniff fails to start on a merged-/usr system"
  apply patch found in the BTS, QA upload
• #873997 – src:openjpeg2: "FTBFS with Java 9 due to -source/-target only"
  apply patch from the BTS, upload to DELAYED/5
• #879624 – xorg: "xorg: After upgrade to buster: system doesnt start x-server anymore but stop reacting"
  lower severity
• #884658 – dkms: "dkms: Should really depends on dpkg-dev for dpkg-architecture"
  add a comment to the bug log
• #886120 – ctpp2: "makes ctpp2 randomly FTBFS, syntax errors, hides problems"
  apply patch from the BTS, upload to DELAYED/5
• #886836 – libgtkmm-2.4-dev,libgtkmm-2.4-doc: "libgtkmm-2.4-dev,libgtkmm-2.4-doc: both ship usr/share/doc/libgtkmm-2.4-dev/examples/*"
  propose a patch
• #887602 – dia: "dia: Detect freetype via pkg-config"
  add patch from upstream merge request, upload to DELAYED/5
• #890595 – phpmyadmin: "phpmyadmin: warnings when running under php 7.2, apparently fixed by new upstream series 4.7.x"
  prepare a debdiff
• #892121 – src:libxml-saxon-xslt2-perl: "libxml-saxon-xslt2-perl FTBFS with libsaxonhe-java 9.8.0.8+dfsg+1-1"
  upload with patch from racke (pkg-perl)
• #892444 – src:ttfautohint: "ttfautohint: Please use 'pkg-config' to find FreeType 2"
  add patch from Hilko Bengen in BTS, upload to DELAYED/5, then fixed in a maintainer upload
• #898752 – src:node-webpack: "node-webpack: FTBFS and autopkgtest failure on 32-bit"
  apply proposal from BTS, upload to DELAYED/5
• #900395 – xserver-xorg-input-all: "xserver-xorg-input-all: keyboard no longer working after dist-upgrade"
  downgrade and tag moreinfo
• #906187 – facter-dev: "facter-dev: missing Depends: libfacter3.11.0 (= ${binary:Version})"
  add missing Depends, upload to DELAYED/5
• #906977 – src:parley: "parley: FTBFS in buster/sid ('QItemSelection' does not name a type)"
  add patch from upstream git, upload to DELAYED/5
• #907975 – libf2fs-format-dev: "libf2fs-format-dev: missing Breaks+Replaces: libf2fs-dev (<< 1.11)"
  add missing Breaks+Replaces, upload to DELAYED/5
• #908147 – cups-browsed: "restarting cups-browsed deleted print jobs"
  try to reproduce
• #911324 – carton: "carton: Carton fails due to missing Menlo::CLI::Compat dependency"
  package missing new dependencies and depend on them (pkg-perl)
• #912046 – ebtables: "ebtables provides the same executables as iptables packages without using alternatives"
  add a comment to the bug log
• #912380 – man-db: "endless looping, breaks dpkg"
  add a comment to the bug log
• #912685 – src:net-snmp: "debian/rules is not binNMU safe"
  add a comment to the bug log
• #913179 – libprelude: "libprelude: FTBFS with glibc 2.28; cherrypicked patches attached"
  take patch from BTS, upload to DELAYED/10
• #915297 – src:libtest-bdd-cucumber-perl: "libtest-bdd-cucumber-perl: FTBFS: README.pod: No such file or directory"
  remove override from debian/rules (pkg-perl)
• #915806 – src:jabref: "jabref FTBFS"
  add build-dep and jar to classpath (pkg-java)

While I missed the WordPress 5.0 release, it was only a few more days before there was a security release out.

So WordPress 5.0.1 will be available in Debian soon. This is both a security update from 5.0.1 and a huge feature update from the 4.9.x versions to the 5.0 versions.

The WordPress website, in their 5.0 announcement describe all the changes better, but one of the main things is the new editor (which I’m using as I write this).  It’s certainly cleaner, or perhaps more sparse. I’m not sure if I like it yet.

The security fixes (there are 7) are the usual things you expect from a WordPress security update. The usual XSS and permission problems type stuff.

I have also in the 5.0.1 Debian package removed the build dependency to libphp-phpmailer. The issue with that package is there won’t be any more security updates for the version in Debian. WordPress has an embedded version of it which *I hope* they maintain. There is an issue about the phpmailer in WordPress, so hopefully it gets fixed soon.

Our linl package for writing LaTeX letter with (R)markdown had a fairly minor release today, following up on the previous release well over a year ago. This version just contains one change which Mark van der Loo provided a few months ago with a clean PR. As another user was just bitten the same issue when using an included letterhead – which was fixed but unreleased – we decided it was time for a release. So there it is.

linl makes it easy to write letters in markdown, with some extra bells and whistles thanks to some cleverness chiefly by Aaron.

Here is screenshot of the vignette showing the simple input for some moderately fancy output:

The NEWS entry follows:

Changes in linl version 0.0.3 (2018-12-15)

• Correct LaTeX double loading of package color with different options (Mark van der Loo in #18 fixing #17).

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the linl page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

December starts for our family in Argentina, and in our second day here, I was invited to Facultad de Ingeniería y Ciencias Hídricas (FICH) of Universidad Nacional del Litoral (UNL). FICH-UNL has a short (3 year), distance-learning career called Tecnicatura Universitaria en Software Libre (TUSL).

This career opened in 2015, and today we had the graduation exams for three of its students — It's no small feat for a recently created career to start graduating their first bunch! And we had one for each of TUSL's "exit tracks" (Administration, development and education).

The topics presented by the students were:

1. An introductory manual for performing migrations and installations of free software-based systems
2. Design and implementation of a steganography tool for end-users
3. A Lego system implementation for AppBuilder

The TUSL staff is quite well aligned to freedom, transparency and responsibilty, so it's basically a requirement for projects to be freely available. For the curious, here they are:

• Lucas Sebastian Villa: Guía para administradores de sistemas
• Alejandro Bonavita: Stegopy
• Gustavo del Dago: Ambiente Logo (AppBuilder)

AttachmentSize photo4949471233775347771.jpg249.94 KB photo4949471233775347775.jpg295.26 KB

My server runs on a RAID10 of 4x WD RAID 2 TB disks. Basically those disks are fast enough to cope with the disk load of the virtual machines (VMs). But since many users moved away from Facebook and Google, my Friendica installation on Nerdica.Net has a growing user count putting a large disk I/O load with many small reads & writes on the disks, resulting a slowing down the general disk I/O for all the VMs and the server itself. On mdraid-sync-Sunday this month the server needed two full days to sync its RAID10.

So the idea was to remove the high disk I/O load from the rotational disks the something different. For that reason I bought a Samsung Pro 970 512 GB NVMe disk and a matching PCIe 3.0 card to be put into my server in the colocation. On Thursday the Samsung has been installed by the rrbone staff in the colocation. I moved the PostgreSQL and MySQL databases from the RAID10 to the NVMe disk and restarted services again.

Here are some results from Munin monitoring: 

Disk Utilization

Here you can see how the disk utilization dropped after NVMe installation. The red coloured bar symbolizes the average utilization on RAID10 disks and the green bar symbolizes the same RAID10 after the databases were moved to the NVMe disk. There's roughly 20% less utilization now, whch is good.

Disk Latency

Here you can see the same coloured bars for the disk latency. As you can see the latency dropped by 1/3 now.

CPU I/O wait

The most significant graph is maybe the CPU graph where you can see a large portion of iowait of the CPUs. This is no longer true as there is apparently no significant iowait anymore thanks to the low latency and high IOPS nature of SSD/NVMe disks.

Overall I cannot confirm that adding the NVMe disk results in a significant faster page load of Friendica or Mastodon, maybe because other measurements like Redis/Memcached or pgbouncer already helped a lot before the NVMe disk, but it helps a lot with general disk I/O load and improving disk speeds inside of the VMs, like for my regular backups and such.

Ah, one thing to report is: in a quick test pgbench reported >2200 tps on NVMe now. That at least is a real speed improvement, maybe by order of 10 or so.

Kategorie: DebianTags: DebianServerHardware

A fun way to learn how to program Python is to follow the instructions in the book "Learn to program with Minecraft", which introduces programming in Python to people who like to play with Minecraft. The book uses a Python library to talk to a TCP/IP socket with an API accepting build instructions and providing information about the current players in a Minecraft world. The TCP/IP API was first created for the Minecraft implementation for Raspberry Pi, and has since been ported to some server versions of Minecraft. The book contain recipes for those using Windows, MacOSX and Raspian. But a little known fact is that you can follow the same recipes using the free software construction game Minetest.

There is a Minetest module implementing the same API, making it possible to use the Python programs coded to talk to Minecraft with Minetest too. I uploaded this module to Debian two weeks ago, and as soon as it clears the FTP masters NEW queue, learning to program Python with Minetest on Debian will be a simple 'apt install' away. The Debian package is maintained as part of the Debian Games team, and the packaging rules are currently located under 'unfinished' on Salsa.

You will most likely need to install several of the Minetest modules in Debian for the examples included with the library to work well, as there are several blocks used by the example scripts that are provided via modules in Minetest. Without the required blocks, a simple stone block is used instead. My initial testing with a analog clock did not get gold arms as instructed in the python library, but instead used stone arms.

I tried to find a way to add the API to the desktop version of Minecraft, but were unable to find any working recipes. The recipes I found are only working with a standalone Minecraft server setup. Are there any options to use with the normal desktop version?

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Like each month, here comes a report about the work of paid contributors to Debian LTS.

Individual reports

In November, about 209 work hours have been dispatched among 14 paid contributors. Their reports are available:

• Abhijith PA did 13 hours (out of 13 extra hours from October).
• Antoine Beaupré did 24 hours.
• Ben Hutchings did 20 hours.
• Brian May did 10 hours.
• Chris Lamb did 18 hours.
• Emilio Pozuelo Monfort did 38 hours (out of 30 hours allocated + 47.25 extra hours, thus keeping 39.25 extra hours for December).
• Hugo Lefeuvre did 15 hours.
• Lucas Kanashiro did 4 hours.
• Markus Koschany did 30 hours.
• Mike Gabriel did 9 hours (out of 8 hours allocated + 4 extra hours, but gave back 2 hours, thus keeping 1 extra hour for December).
• Ola Lundqvist did 9 hours (out of 8 hours allocated + 8 extra hours, thus keeping 7 extra hours for December).
• Roberto C. Sanchez did 13.75 hours (out of 12 hours allocated + 2.5 extra hours, thus keeping 0.75 extra hours for December).
• Santiago Ruano Rincón did 12 hours (out of 18 extra hours, thus keeping 6 extra hours for December).
• Thorsten Alteholz did 30 hours.

Evolution of the situation

In November we had a few more hours available to dispatch than we had contributors willing and able to do the work and thus we are actively looking for new contributors. Please contact Holger if you are interested to become a paid LTS contributor.

The number of sponsored hours stayed the same at 212 hours per month but we actually lost two sponsors and gained a new one (silver level).

The security tracker currently lists 30 packages with a known CVE and the dla-needed.txt file has 32 packages needing an update.

Thanks to our sponsors

New sponsors are in bold.

• Platinum sponsors:
• TOSHIBA (for 38 months)
• GitHub (for 29 months)
• Civil Infrastructure Platform (CIP) (for 6 months)
• Gold sponsors:
• The Positive Internet (for 54 months)
• Blablacar (for 53 months)
• Linode (for 43 months)
• Babiel GmbH (for 32 months)
• Plat’Home (for 32 months)
• Silver sponsors:
• Domeneshop AS (for 54 months)
• Nantes Métropole (for 48 months)
• Dalenys (for 44 months)
• Univention GmbH (for 39 months)
• Université Jean Monnet de St Etienne (for 39 months)
• Ribbon Communications, Inc. (for 33 months)
• maxcluster GmbH (for 27 months)
• Exonet B.V. (for 23 months)
• Leibniz Rechenzentrum (for 17 months)
• Vente-privee.com (for 14 months)
• CINECA (for 6 months)
• Ministère de l’Europe et des Affaires Étrangères
• Bronze sponsors:
• David Ayers – IntarS Austria (for 54 months)
• Evolix (for 54 months)
• Seznam.cz, a.s. (for 54 months)
• MyTux (for 53 months)
• Intevation GmbH (for 51 months)
• Linuxhotel GmbH (for 51 months)
• Daevel SARL (for 50 months)
• Bitfolk LTD (for 48 months)
• Megaspace Internet Services GmbH (for 48 months)
• NUMLOG (for 48 months)
• Greenbone Networks GmbH (for 47 months)
• WinGo AG (for 47 months)
• Ecole Centrale de Nantes – LHEEA (for 43 months)
• Sig-I/O (for 41 months)
• Entr’ouvert (for 38 months)
• Adfinis SyGroup AG (for 36 months)
• GNI MEDIA (for 30 months)
• Laboratoire LEGI – UMR 5519 / CNRS (for 30 months)
• Quarantainenet BV (for 30 months)
• Bearstech (for 22 months)
• LiHAS (for 21 months)
• People Doc (for 18 months)
• Catalyst IT Ltd (for 16 months)
• Demarcq SAS (for 10 months)
• TrapX Security (for 7 months)
• NCC Group (for 4 months)

No comment | Liked this article? Click here. | My blog is Flattr-enabled.

Update 14 December 2018: After the release of stable 1.31.0 (aka 2018 Edition), it is no longer necessary to switch to the nightly channel to get access to thumb7em-none-eabi / Cortex-M4 and Cortex-M7 components. Updated examples and commands accordingly.
For more details on embedded development using Rust, the official Rust embedded docs site is the place to go, in particular, you can start with The embedded Rust book.
 
 
This is a reminder for myself, if you want to install Rust for a baremetal Cortex-M7 target, this seems to be a tier 3 platform:

https://forge.rust-lang.org/platform-support.html

Higlighting the relevant part:

Target std rustc cargo notes ... msp430-none-elf * 16-bit MSP430 microcontrollers sparc64-unknown-netbsd ✓ ✓ NetBSD/sparc64 thumbv6m-none-eabi * Bare Cortex-M0, M0+, M1 thumbv7em-none-eabi *

Bare Cortex-M4, M7 thumbv7em-none-eabihf * Bare Cortex-M4F, M7F, FPU, hardfloat thumbv7m-none-eabi * Bare Cortex-M3 ... x86_64-unknown-openbsd ✓ ✓ 64-bit OpenBSD
In order to enable the relevant support, use the nightly build and use stable >= 1.31.0 and add the relevant target:
eddy@feodora:~/usr/src/rust-uc$ rustup show
Default host: x86_64-unknown-linux-gnu

installed toolchains
--------------------

stable-x86_64-unknown-linux-gnu
nightly-x86_64-unknown-linux-gnu (default)

active toolchain
----------------

nightly-x86_64-unknown-linux-gnu (default)
rustc 1.28.0-nightly (cb20f68d0 2018-05-21)eddy@feodora:~/usr/src/rust$ rustup show
Default host: x86_64-unknown-linux-gnu

stable-x86_64-unknown-linux-gnu (default)
rustc 1.31.0 (abe02cefd 2018-12-04)
If not using nightly, switch to that:


eddy@feodora:~/usr/src/rust-uc$ rustup default nightly-x86_64-unknown-linux-gnu
info: using existing install for 'nightly-x86_64-unknown-linux-gnu'
info: default toolchain set to 'nightly-x86_64-unknown-linux-gnu'

  nightly-x86_64-unknown-linux-gnu unchanged - rustc 1.28.0-nightly (cb20f68d0 2018-05-21)
Add the needed target:
eddy@feodora:~/usr/src/rust$ rustup target add thumbv7em-none-eabi
info: downloading component 'rust-std' for 'thumbv7em-none-eabi'
info: installing component 'rust-std' for 'thumbv7em-none-eabi'
eddy@feodora:~/usr/src/rust$ rustup show
Default host: x86_64-unknown-linux-gnu

installed targets for active toolchain
--------------------------------------

thumbv7em-none-eabi
x86_64-unknown-linux-gnu

active toolchain
----------------

stable-x86_64-unknown-linux-gnu (default)
rustc 1.31.0 (abe02cefd 2018-12-04)Then compile with --target.

I was assigned 20 hours of work by Freexian's Debian LTS initiative and worked all those hours.

I prepared and released another stable update for Linux 3.16 (3.16.61), but have not yet included this in a Debian upload.

I updated the firmware-nonfree package to fix security issues in wifi firmware and to provide additional firmware that may be requested by drivers in Linux 4.9. I issued DLA 1573-1 to describe this update.

I worked on documenting a bug in the installer that delays installation of updates to the kernel. The installer can only be updated as part of a point release, and these are not done after the LTS team takes on responsibility for a release. Laura Arjona drafted an addition to the Errata section of the official jessie installer page and I reviewed this. I also updated the LTS/Installing wiki page.

I also participated in various discussions on the debian-lts mailing list.

Some background reading

The relationship between open source and free software is fraught with people arguing about meanings and value. In spite of all the things we’ve built up around open source and free software, they reduce down to both being about software freedom.

Open source is about software freedom. It has been the case since “open source” was created.

In 1986 the Four Freedoms of Free Software (4Fs) were written. In 1998 Netscape set its source code free. Later that year a group of people got together and Christine Peterson suggested that, to avoid ambiguity, there was a “need for a better name” than free software. She suggested open source after open source intelligence. The name stuck and 20 years later we argue about whether software freedom matters to open source, because too many global users of the term have forgotten (or never knew) that some people just wanted another way to say software that ensures the 4Fs.

Once there was a term, the term needed a formal definition: how to we describe what open source is? That’s where the Open Source Definition (OSD) comes in.

The OSD is a set of ten points that describe what an open source license looks like. The OSD came from the Debian Free Software Guidelines. The DFSG themselves were created to “determine if a work is free” and ought to be considered a way of describing the 4Fs.

Back to the present

I believe that the OSD is about user freedom. This is an abstraction from “open source is about free software.” As I eluded to earlier, this is an intuition I have, a thing I believe, and an argument I’m have a very hard time trying to make.

I think of free software as software that exhibits or embodies software freedom — it’s software created using licenses that ensure the things attached to them protect the 4Fs. This is all a tool, a useful tool, for protecting user freedom.

The line that connects the OSD and user freedom is not a short one: the OSD defines open source -> open source is about software freedom -> software freedom is a tool to protect user freedom. I think this is, however, a very valuable reduction we can make. The OSD is another tool in our tool box when we’re trying to protect the freedom of users of computers and computing technology.

Why does this matter (now)?

I would argue that this has always mattered, and we’ve done a bad job of talking about it. I want to talk about this now because its become increasingly clear that people simply never understood (or even heard of) the connection between user freedom and open source.

I’ve been meaning to write about this for a while, and I think it’s important context for everything else I say and write about in relation to the philosophy behind free and open source software (FOSS).

FOSS is a tool. It’s not a tool about developmental models or corporate enablement — though some people and projects have benefited from the kinds of development made possible through sharing source code, and some companies have created very financially successful models based on it as well. In both historical and contemporary contexts, software freedom is at the heart of open source. It’s not about corporate benefit, it’s not about money, and it’s not even really about development. Methods of development are tools being used to protect software freedom, which in turn is a tool to protect user freedom. User freedom, and what we get from that, is what’s valuable.

Side note

At some future point, I’ll address why user freedom matters, but in the mean time, here are some talks I gave (with Karen Sandler) on the topic.

• Introduction to user freedom – HOPE XII
• That’s a free software issue – DebConf 18

This week I gave KDE Frameworks a web page after only 4 years of us trying to promote it as the best thing ever since cabogganing without one.  I also updated the theme on the KDE Applications 18.12 announcement to this millennium and even made the images in it have a fancy popup effect using the latest in JQuery Bootstrap CSS.  But my proudest contribution is making the screenshot for the new release of Konsole showing how it can now display all the cat emojis plus one for a poodle.

So far no comments asking why I named my computer thus.

by

I am returning from the reproducible builds summit 2018 in Paris. The latest hottest thing within the reproducible-builds project seems to be bootstrapping: How can we build a whole operating system from just and only source code, using very little, or even no, binary seeds or auto-generated files. This is actually concern that is somewhat orthogonal to reproducibility: Bootstrappable builds help me in trusting programs that I built, while reproducible builds help me in trusting programs that others built.

And while they make good progress bootstrapping a full system from just a C compiler written in Scheme, and a Scheme interpreter written in C, that can build each other (Janneke’s mes project), and there are plans to build that on top of stage0, which starts with a 280 bytes of binary, the situation looks pretty bad when it comes to Haskell.

Unreachable GHC

The problem is that contemporary Haskell has only one viable implementation, GHC. And GHC, written in contemporary Haskell, needs GHC to be build. So essentially everybody out there either just downloads a binary distribution of GHC. Or they build GHC from source, using a possibly older (but not much older) version of GHC that they already have. Even distributions like Debian do nothing different: When they build the GHC package, the builders use, well, the GHC package.

There are other Haskell implementations out there. But if they are mature and active developed, then they are implemented in Haskell themselves, often even using advanced features that only GHC provides. And even those are insufficient to build GHC itself, let alone the some old and abandoned Haskell implementations.

In all these cases, at some point an untrusted binary is used. This is very unsatisfying. What can we do? I don’t have the answers, but please allow me to outline some venues of attack.

Retracing history

Obviously, even GHC does not exist since the beginning of time, and the first versions surely were built using something else than GHC. The oldest version of GHC for which we can find a release on the GHC web page is version 0.29 from July 1996. But the installation instructions write:

GHC 0.26 doesn't build with HBC. (It could, but we haven't put in the effort to maintain it.)

GHC 0.26 is best built with itself, GHC 0.26. We heartily recommend it. GHC 0.26 can certainly be built with GHC 0.23 or 0.24, and with some earlier versions, with some effort.

GHC has never been built with compilers other than GHC and HBC.

So it seems that besides GHC, only ever HBC was used to compiler GHC. HBC is a Haskell compiler where we find the sources of one random version only thanks to archive.org. Parts of it are written in C, so I looked into this: Compile HBC, use it to compile GHC-0.29, and then step for step build every (major) version of GHC until today.

The problem is that it is non-trivial to build software from the 90s using today's compilers. I briefly looked at the HBC code base, and had to change some files from using varargs.h to stdargs.v, and this is surely just one of many similar stumbling blocks trying to build that tools. Oh, and even the hbc source state

# To get everything done: make universe # It is impossible to make from scratch. # You must have a running lmlc, to # recompile it (of course).

So I learned that actually, most of it is written in LML, and the LML compiler is written in LML. So this is a dead end. (Thanks to Lennart for clearing up a misunderstanding on my side here.

Going back, but doing it differently

Another approach is to go back in time, to some old version of GHC, but maybe not all the way to the beginning, and then try to use another, officially unsupported, Haskell compiler to build GHC. This is what rekado tried to do in 2017: He use the most contemporary implementation of Haskell in C, the Hugs interpreter. Using this, he compiled nhc98 (yet another abandoned Haskell implementation), with the hope of building GHC with nhc98. He made impressive progress back then, but ran into a problem where the runtime crashed. Maybe someone is interested in picking up up from there?

Removing, simplifying, extending, in the present.

Both approaches so far focus on building an old version of GHC. This adds complexity: other tools (the shell, make, yacc etc.) may behave different now in a way that causes hard to debug problems. So maybe it is more fun and more rewarding to focus on today’s GHC? (At this point I am starting to hypothesize).

I said before that no other existing Haskell implementation can compile today’s GHC code base, because of features like mutually recursive modules, the foreign function interface etc. And also other existing Haskell implementations often come with a different, smaller set of standard libraries, but GHC assumes base, so we would have to build that as well...

But we don’t need to build it all. Surely there is much code in base that is not used by GHC. Also, much code in GHC that we do not need to build GHC, and . So by removing that, we reduce the amount of Haskell code that we need to feed to the other implementation.

The remaining code might use some features that are not supported by our bootstrapping implementation. Mutually recursive module could be manually merged. GADTs that are only used for additional type safety could be replaced by normal ones, which might make some pattern matches incomplete. Syntactic sugar can be desugared. By simplifying the code base in that way, one might be able a fork of GHC that is within reach of the likes of Hugs or nhc98.

And if there are features that are hard to remove, maybe we can extend the bootstrapping compiler or interpreter to support them? For example, it was mostly trivial to extend Hugs with support for the # symbol in names – and we can be pragmatic and just allow it always, since we don’t need a standards conforming implementation, but merely one that works on the GHC code base. But how much would we have to implement? Probably this will be more fun in Haskell than in C, so maybe extending nhc98 would be more viable?

Help from beyond Haskell?

Or maybe it is time to create a new Haskell compiler from scratch, written in something other than Haskell? Maybe some other language that is reasonably pleasant to write a compiler in (Ocaml? Scala?), but that has the bootstrappability story already sorted out somehow.

But in the end, all variants come down to the same problem: Writing a Haskell compiler for full, contemporary Haskell as used by GHC is hard and really a lot of work – if it were not, there would at least be implementations in Haskell out there. And as long as nobody comes along and does that work, I fear that we will continue to be unable to build our nice Haskell ecosystem from scratch. Which I find somewhat dissatisfying.

Already December. Nice. I tried using tramp for a while but I am back to mosh. tramp is not usable when ssh connection is not reliable.

Like many people, I've been using Dropbox to share files with friends and family for years. It's a super convenient and easy way to get files syncronised between machines you own, and work with others. This morning I was greeted with a lovely message on my Ubuntu desktop.

It says "Can't sync Dropbox until you sign in and move it to a supported file system" with options to "See requirements", "Quit Dropbox" and "Sign in".

Dropbox have reduced the number of file systems they support. We knew this was coming for a while, but it's a pain if you don't use one of the supported filesystems.

Recently I re-installed my Ubuntu 18.04 laptop and chose XFS rather than the default ext4 partition type when installing. That's the reason the error is appearing for me.

I do also use NextCloud and Syncthing for syncing files, but some of the people I work with only use Dropbox, and forcing them to change is tricky.

So I wanted a solution where I could continue to use Dropbox but not have to re-format the home partition on my laptop. The 'fix' is to create a file, format it ext4 and mount it where Dropbox expects your files to be. That's essentially it. Yay Linux. This may be useful to others, so I've detailed the steps below.

Note: I strongly recommend backing up your dropbox folder first, but I'm sure you already did that so let's assume you're good.

This is just a bunch of commands, which you could blindly paste en masse, or, preferably one-by-one, checking it did what it says it should, before moving on. It worked for me, but may not work for you. I am not to blame if this deletes your cat pictures. Before you begin, stop Dropbox completely. Close the client.

I've also put these in a github gist.

# Location of the image which will contain the new ext4 partition DROPBOXFILE="$HOME"/.dropbox.img # Current location of my Dropbox folder DROPBOXHOME="$HOME"/Dropbox # Where we will copy the folder to. If you have little space, you could make this # a folder on a USB drive DROPBOXBACKUP="$HOME"/old_Dropbox # What size is the Dropbox image file going to be. It makes sense to set this # to whatever the capacity of your Dropbox account is, or a little more. DROPBOXSIZE="20G" # Create a 'sparse' file which will start out small and grow to the maximum # size defined above. So we don't eat all that space immediately. dd if=/dev/zero of="$DROPBOXFILE" bs=1 count=0 seek="$DROPBOXSIZE" # Format it ext4, because Dropbox Inc. says so sudo mkfs.ext4 "$DROPBOXFILE" # Move the current Dropbox folder to the backup location mv "$DROPBOXHOME" "$DROPBOXBACKUP" # Make a new Dropbox folder to replace the old one. This will be the mount point # under which the sparse file will be mounted mkdir "$DROPBOXHOME" # Make sure the mount point can't be written to if for some reason the partition # doesn't get mounted. We don't want dropbox to see an empty folder and think 'yay, let's delete # all his files because this folder is empty, that must be what they want' sudo chattr +i "$DROPBOXHOME" # Mount the sparse file at the dropbox mount point sudo mount -o loop "$DROPBOXFILE" "$DROPBOXHOME" # Copy the files from the existing dropbox folder to the new one, which will put them # inside the sparse file. You should see the file grow as this runs. sudo rsync -a "$DROPBOXBACKUP"/ "$DROPBOXHOME"/ # Create a line in our /etc/fstab so this gets mounted on every boot up echo "$DROPBOXFILE" "$DROPBOXHOME" ext4 loop,defaults,rw,relatime,exec,user_xattr 0 0 | sudo tee -a /etc/fstab # Let's unmount it so we can make sure the above line worked sudo umount "$DROPBOXHOME" # This will mount as per the fstab sudo mount -a # Set ownership and permissions on the new folder so Dropbox has access sudo chown $(id -un) "$DROPBOXHOME" sudo chgrp $(id -gn) "$DROPBOXHOME"

Now start Dropbox. All things being equal, the error message will go away, and you can carry on with your life, syncing files happily.

Hope that helps. Leave a comment here or over on the github gist.

Newt: A Tiny Embeddable Python Subset

I've been helping teach robotics programming to students in grades 5 and 6 for a number of years. The class uses Lego models for the mechanical bits, and a variety of development environments, including Robolab and Lego Logo on both Apple ][ and older Macintosh systems. Those environments are quite good, but when the Apple ][ equipment died, I decided to try exposing the students to an Arduino environment so that they could get another view of programming languages.

The Arduino environment has produced mixed results. The general nature of a full C++ compiler and the standard Arduino libraries means that building even simple robots requires a considerable typing, including a lot of punctuation and upper case letters. Further, the edit/compile/test process is quite long making fixing errors slow. On the positive side, many of the students have gone on to use Arduinos in science research projects for middle and upper school (grades 7-12).

In other environments, I've seen Python used as an effective teaching language; the direct interactive nature invites exploration and provides rapid feedback for the students. It seems like a pretty good language to consider for early education -- "real" enough to be useful in other projects, but simpler than C++/Arduino has been. However, I haven't found a version of Python that seems suitable for the smaller microcontrollers I'm comfortable building hardware with.

How Much Python Do We Need?

Python is a pretty large language in embedded terms, but there's actually very little I want to try and present to the students in our short class (about 6 hours of language introduction and another 30 hours or so of project work). In particular, all we're using on the Arduino are:

• Numeric values
• Loops and function calls
• Digital and analog I/O

Remembering my childhood Z-80 machine with its BASIC interpreter, I decided to think along those lines in terms of capabilities. I think I can afford more than 8kB of memory for the implementation, and I really do want to have "real" functions, including lexical scoping and recursion.

I'd love to make this work on our existing Arduino Duemilanove compatible boards. Those have only 32kB of flash and 2kB of RAM, so that might be a stretch...

What to Include

Exploring Python, I think there's a reasonable subset that can be built here. Included in that are:

• Lists, numbers and string types
• Global functions
• For/While/If control structures.

What to Exclude

It's hard to describe all that hasn't been included, but here's some major items:

• Objects, Dictionaries, Sets
• Comprehensions
• Generators (with the exception of range)
• All numeric types aside from single-precision float

Implementation

Newt is implemented in C, using flex and bison. It includes the incremental mark/sweep compacting GC system I developed for my small scheme interpreter last year. That provides a relatively simple to use and efficient memory system.

The Newt “Compiler”

Instead of directly executing a token stream as my old BASIC interpreter did, Newt is compiling to a byte coded virtual machine. Of course, we have no memory, so we don't generate a parse tree and perform optimizations on that. Instead, code is generated directly in the grammar productions.

The Newt “Virtual Machine”

With the source compiled to byte codes, execution is pretty simple -- read a byte code, execute some actions related to it. To keep things simple, the virtual machine has a single accumulator register and a stack of other values.

Global and local variables are stored in 'frames', with each frame implemented as a linked list of atom/value pairs. This isn't terribly efficient in space or time, but was quick to implement the required Python semantics for things like 'global'.

Lists and tuples are simple arrays in memory, just like C Python. I use the same sizing heuristic for lists that Python does; no sense inventing something new for that. Strings are C strings.

When calling a non-builtin function, a new frame is constructed that includes all of the formal names. Those get assigned values from the provided actuals and then the instructions in the function are executed. As new locals are discovered, the frame is extended to include them.

Testing

Any new language implementation really wants to have a test suite to ensure that the desired semantics are implemented correctly. One huge advantage for Newt is that we can cross-check the test suite by running it with Python.

Current Status

I think Newt is largely functionally complete at this point; I just finished adding the limited for statement capabilities this evening. I'm sure there are a lot of bugs to work out, and I expect to discover additional missing functionality as we go along.

I'm doing all of my development and testing on my regular x86 laptop, so I don't know how big the system will end up on the target yet.

I've written 4836 lines of code for the implementation and another 65 lines of Python for simple test cases. When compiled -Os for x86_64, the system is about 36kB of text and another few bytes of initialized data.

Links

The source code is available from my server at https://keithp.com/cgit/newt.git/, and also at github https://github.com/keith-packard/newt. It is licensed under the GPLv2 (or later version).