You are here

Agreguesi i feed

Less parentheses

Planet Debian - Dje, 10/09/2017 - 12:10md

Yesterday, at the Haskell Implementers Workshop 2017 in Oxford, I gave a lightning talk titled ”syntactic musings”, where I presented three possibly useful syntactic features that one might want to add to a language like Haskell.

The talked caused quite some heated discussions, and since the Internet likes heated discussion, I will happily share these ideas with you

Context aka. Sections

This is probably the most relevant of the three proposals. Consider a bunch of related functions, say analyseExpr and analyseAlt, like these:

analyseExpr :: Expr -> Expr analyseExpr (Var v) = change v analyseExpr (App e1 e2) = App (analyseExpr e1) (analyseExpr e2) analyseExpr (Lam v e) = Lam v (analyseExpr flag e) analyseExpr (Case scrut alts) = Case (analyseExpr scrut) (analyseAlt <$> alts) analyseAlt :: Alt -> Alt analyseAlt (dc, pats, e) = (dc, pats, analyseExpr e)

You have written them, but now you notice that you need to make them configurable, e.g. to do different things in the Var case. You thus add a parameter to all these functions, and hence an argument to every call:

type Flag = Bool analyseExpr :: Flag -> Expr -> Expr analyseExpr flag (Var v) = if flag then change1 v else change2 v analyseExpr flag (App e1 e2) = App (analyseExpr flag e1) (analyseExpr flag e2) analyseExpr flag (Lam v e) = Lam v (analyseExpr (not flag) e) analyseExpr flag (Case scrut alts) = Case (analyseExpr flag scrut) (analyseAlt flag <$> alts) analyseAlt :: Flag -> Alt -> Alt analyseAlt flag (dc, pats, e) = (dc, pats, analyseExpr flag e)

I find this code problematic. The intention was: “flag is a parameter that an external caller can use to change the behaviour of this code, but when reading and reasoning about this code, flag should be considered constant.”

But this intention is neither easily visible nor enforced. And in fact, in the above code, flag does “change”, as analyseExpr passes something else in the Lam case. The idiom is indistinguishable from the environment idiom, where a locally changing environment (such as “variables in scope”) is passed around.

So we are facing exactly the same problem as when reasoning about a loop in an imperative program with mutable variables. And we (pure functional programmers) should know better: We cherish immutability! We want to bind our variables once and have them scope over everything we need to scope over!

The solution I’d like to see in Haskell is common in other languages (Gallina, Idris, Agda, Isar), and this is what it would look like here:

type Flag = Bool section (flag :: Flag) where analyseExpr :: Expr -> Expr analyseExpr (Var v) = if flag then change1 v else change2v analyseExpr (App e1 e2) = App (analyseExpr e1) (analyseExpr e2) analyseExpr (Lam v e) = Lam v (analyseExpr e) analyseExpr (Case scrut alts) = Case (analyseExpr scrut) (analyseAlt <$> alts) analyseAlt :: Alt -> Alt analyseAlt (dc, pats, e) = (dc, pats, analyseExpr e)

Now the intention is clear: Within a clearly marked block, flag is fixed and when reasoning about this code I do not have to worry that it might change. Either all variables will be passed to change1, or all to change2. An important distinction!

Therefore, inside the section, the type of analyseExpr does not mention Flag, whereas outside its type is Flag -> Expr -> Expr. This is a bit unusual, but not completely: You see precisely the same effect in a class declaration, where the type signature of the methods do not mention the class constraint, but outside the declaration they do.

Note that idioms like implicit parameters or the Reader monad do not give the guarantee that the parameter is (locally) constant.

More details can be found in the GHC proposal that I prepared, and I invite you to raise concern or voice support there.

Curiously, this problem must have bothered me for longer than I remember: I discovered that seven years ago, I wrote a Template Haskell based implementation of this idea in the seal-module package!

Less parentheses 1: Bulleted argument lists

The next two proposals are all about removing parentheses. I believe that Haskell’s tendency to express complex code with no or few parentheses is one of its big strengths, as it makes it easier to visualy parse programs. A common idiom is to use the $ operator to separate a function from a complex argument without parentheses, but it does not help when there are multiple complex arguments.

For that case I propose to steal an idea from the surprisingly successful markup language markdown, and use bulleted lists to indicate multiple arguments:

foo :: Baz foo = bracket • some complicated code that is evaluated first • other complicated code for later • even more complicated code

I find this very easy to visually parse and navigate.

It is actually possible to do this now, if one defines (•) = id with infixl 0 •. A dedicated syntax extension (-XArgumentBullets) is preferable:

  • It only really adds readability if the bullets are nicely vertically aligned, which the compiler should enforce.
  • I would like to use $ inside these complex arguments, and multiple operators of precedence 0 do not mix. (infixl -1 • would help).
  • It should be possible to nest these, and distinguish different nesting levers based on their indentation.
Less parentheses 1: Whitespace precedence

The final proposal is the most daring. I am convinced that it improves readability and should be considered when creating a new language. As for Haskell, I am at the moment not proposing this as a language extension (but could be convinced to do so if there is enough positive feedback).

Consider this definition of append:

(++) :: [a] -> [a] -> [a] [] ++ ys = ys (x:xs) ++ ys = x : (xs++ys)

Imagine you were explaining the last line to someone orally. How would you speak it? One common way to do so is to not read the parentheses out aloud, but rather speak parenthesised expression more quickly and add pauses otherwise.

We can do the same in syntax!

(++) :: [a] -> [a] -> [a] [] ++ ys = ys x:xs ++ ys = x : xs++ys

The rule is simple: A sequence of tokens without any space is implicitly parenthesised.

The reaction I got in Oxford was horror and disgust. And that is understandable – we are very used to ignore spacing when parsing expressions (unless it is indentation, of course. Then we are no longer horrified, as our non-Haskell colleagues are when they see our code).

But I am convinced that once you let the rule sink in, you will have no problem parsing such code with ease, and soon even with greater ease than the parenthesised version. It is a very natural thing to look at the general structure, identify “compact chunks of characters”, mentally group them, and then go and separately parse the internals of the chunks and how the chunks relate to each other. More natural than first scanning everything for ( and ), matching them up, building a mental tree, and then digging deeper.

Incidentally, there was a non-programmer present during my presentation, and while she did not openly contradict the dismissive groan of the audience, I later learned that she found this variant quite obvious to understand and more easily to read than the parenthesised code.

Some FAQs about this:

  • What about an operator with space on one side but not on the other? I’d simply forbid that, and hence enforce readable code.
  • Do operator sections still require parenthesis? Yes, I’d say so.
  • Does this overrule operator precedence? Yes! a * b+c == a * (b+c).
  • What is a token? Good question, and I am not not decided. In particular: Is a parenthesised expression a single token? If so, then (Succ a)+b * c parses as ((Succ a)+b) * c, otherwise it should probably simply be illegal.
  • Can we extend this so that one space binds tighter than two spaces, and so on? Yes we can, but really, we should not.
  • This is incompatible with Agda’s syntax! Indeed it is, and I really like Agda’s mixfix syntax. Can’t have everything.
  • Has this been done before? I have not seen it in any language, but Lewis Wall has blogged this idea before.

Well, let me know what you think!

Joachim Breitner mail@joachim-breitner.de nomeata’s mind shares

PHP 7.2 is coming… mcrypt extension isn’t

Planet Debian - Dje, 10/09/2017 - 10:56pd

Early September, it’s about 3 months before PHP 7.2 is expected to be release (schedule here). One of the changes is the removal of the mcrypt extension after it was deprecated in PHP 7.1. The main problem with mcrypt extension is that it is based on libmcrypt that was abandoned by it’s upstream since 2007. That’s 10 years of keeping a library alive, moving the burden to distribution’s security teams. But this isn’t new, Remi already wrote about this two years ago: “About libmcrypt and php-mcrypt“.

But with removal of the extension from the PHP code base (about F**King time), it would force the recommendation was done “nicely” till now. And forcing people means some noise, although an alternative is PHP’s owns openssl extension. But as many migrations that require code change – it’s going slow.

The goal of this post is to reach to the PHP eco system and map the components (mostly frameworks and applications) to still require/recommend mcyrpt and to pressure them to fix it before PHP 72 is released. I’ll appreciate the readers’ help with this mapping in the comments.

For example, Laravel‘s release notes for 5.1:

In previous versions of Laravel, encryption was handled by the mcrypt PHP extension. However, beginning in Laravel 5.1, encryption is handled by the openssl extension, which is more actively maintained.

Or, on the other hand Joomla 3 requirements still mentions mcrypt.

mcrypt safe:

mcrypt dependant:

For those who really need mcrypt, it is part of PECL, PHP’s extensions repository. You’re welcome to compile it on your own risk.


Filed under: Debian GNU/Linux, PHP Kaplan https://liorkaplan.wordpress.com Free Software Universe

Observing Reliability

Planet Debian - Sht, 09/09/2017 - 12:18md

Last year I wrote about how great my latest Thinkpad is [1] in response to a discussion about whether a Thinkpad is still the “Rolls Royce” of laptops.

It was a few months after writing that post that I realised that I omitted an important point. After I had that laptop for about a year the DVD drive broke and made annoying clicking sounds all the time in addition to not working. I removed the DVD drive and the result was that the laptop was lighter and used less power without missing any feature that I desired. As I had installed Debian on that laptop by copying the hard drive from my previous laptop I had never used the DVD drive for any purpose. After a while I got used to my laptop being like that and the gaping hole in the side of the laptop where the DVD drive used to be didn’t even register to me. I would prefer it if Lenovo sold Thinkpads in the T series without DVD drives, but it seems that only the laptops with tiny screens are designed to lack DVD drives.

For my use of laptops this doesn’t change the conclusion of my previous post. Now the T420 has been in service for almost 4 years which makes the cost of ownership about $75 per year. $1.50 per week as a tax deductible business expense is very cheap for such a nice laptop. About a year ago I installed a SSD in that laptop, it cost me about $250 from memory and made it significantly faster while also reducing heat problems. The depreciation on the SSD about doubles the cost of ownership of the laptop, but it’s still cheaper than a mobile phone and thus not in the category of things that are expected to last for a long time – while also giving longer service than phones usually do.

One thing that’s interesting to consider is the fact that I forgot about the broken DVD drive when writing about this. I guess every review has an unspoken caveat of “this works well for me but might suck badly for your use case”. But I wonder how many other things that are noteworthy I’m forgetting to put in reviews because they just don’t impact my use. I don’t think that I am unusual in this regard, so reading multiple reviews is the sensible thing to do.

Related posts:

  1. Is a Thinkpad Still Like a Rolls-Royce For a long time the Thinkpad has been widely regarded...
  2. PC prices drop again! A few weeks ago Dell advertised new laptops for $849AU,...
  3. Laptop Reliability Update: TumbleDry has a good analysis of the Square Trade...
etbe https://etbe.coker.com.au etbe – Russell Coker

TLS Authentication on Freenode and OFTC

Planet Debian - Sht, 09/09/2017 - 6:52pd

In order to easily authenticate with IRC networks such as OFTC and Freenode, it is possible to use client TLS certificates (also known as SSL certificates). In fact, it turns out that it's very easy to setup both on irssi and on znc.

Generate your TLS certificate

On a machine with good entropy, run the following command to create a keypair that will last for 10 years:

openssl req -nodes -newkey rsa:2048 -keyout user.pem -x509 -days 3650 -out user.pem -subj "/CN=<your nick>"

Then extract your key fingerprint using this command:

openssl x509 -sha1 -noout -fingerprint -in user.pem | sed -e 's/^.*=//;s/://g' Share your fingerprints with NickServ

On each IRC network, do this:

/msg NickServ IDENTIFY Password1! /msg NickServ CERT ADD <your fingerprint>

in order to add your fingerprint to the access control list.

Configure ZNC

To configure znc, start by putting the key in the right place:

cp user.pem ~/.znc/users/<your nick>/networks/oftc/moddata/cert/

and then enable the built-in cert plugin for each network in ~/.znc/configs/znc.conf:

<Network oftc> ... LoadModule = cert ... </Network> <Network freenode> ... LoadModule = cert ... </Network> Configure irssi

For irssi, do the same thing but put the cert in ~/.irssi/user.pem and then change the OFTC entry in ~/.irssi/config to look like this:

{ address = "irc.oftc.net"; chatnet = "OFTC"; port = "6697"; use_tls = "yes"; tls_cert = "~/.irssi/user.pem"; tls_verify = "yes"; autoconnect = "yes"; }

and the Freenode one to look like this:

{ address = "chat.freenode.net"; chatnet = "Freenode"; port = "7000"; use_tls = "yes"; tls_cert = "~/.irssi/user.pem"; tls_verify = "yes"; autoconnect = "yes"; }

That's it. That's all you need to replace password authentication with a much stronger alternative.

François Marier http://feeding.cloud.geek.nz/tags/debian/ pages tagged debian

Extract many attachement from many mails in one go using ripmime

Planet Debian - Pre, 08/09/2017 - 11:42md
I was recently looking for a way to extract many attachments from a series of emails. I first had a look at the AttachmentExtractor thunderbird plugin, but it seems very old and not maintained anymore. So I've come up with another very simple solution that also works with any other mail client.

Just copy all the mails you want to extract attachments from to a single (temporary) mail folder, find out which file holds the mail folder and use ripmime on that file (ripmime is packaged for Debian). For my case, it looked like:

~ ripmime -i .icedove/XXXXXXX.default/Mail/pop.xxxx/tmp -d target-directory

Simple solution, but it saved me quite some time. Hope it helps !

Vincent Fourmond noreply@blogger.com YANUB: yet another (nearly) useless blog

munin with TLS

Planet Debian - Pre, 08/09/2017 - 4:41md

Primarily a note for my future self so I don't have to find out what I did in the past once more.

If you're running some smaller systems scattered around the internet, without connecting them with a VPN, you might want your munin master and nodes to communicate with TLS and validate certificates. If you remember what to do it's a rather simple and straight forward process. To manage the PKI I'll utilize the well known easyrsa script collection. For this special purpose CA I'll go with a flat layout. So it's one root certificate issuing all server and client certificates directly. Some very basic docs can be also found in the munin wiki.

master setup

For your '/etc/munin/munin.conf':

tls paranoid tls_verify_certificate yes tls_private_key /etc/munin/master.key tls_certificate /etc/munin/master.crt tls_ca_certificate /etc/munin/ca.crt tls_verify_depth 1

A node entry with TLS will look like this:

[node1.stormbind.net] address [2001:db8::] use_node_name yes

Important points here:

  • "tls_certificate" is a Web Client Authentication certificate. The master connects to the nodes as a client.
  • "tls_ca_certificate" is the root CA certificate.
  • If you'd like to disable TLS connections, for example for localhost, set "tls disabled" in the node block.

For easy-rsa the following command invocations are relevant:

./easyrsa init-pki ./easyrsa build-ca ./easrsa gen-req master ./easyrsa sign-req client master ./easyrsa set-rsa-pass master nopass node setup

For your '/etc/munin/munin-node.conf':

tls paranoid tls_verify_certificate yes tls_private_key /etc/munin/node1.key tls_certificate /etc/munin/node1.crt tls_ca_certificate /etc/munin/ca.crt tls_verify_depth 1

For easy-rsa the following command invocations are relevant:

./easyrsa gen-req node1 ./easyrsa sign-req server node1 ./easyrsa set-rsa-pass node1 nopass

Important points here:

  • "tls_certificate" on the node must be a server certificate.
  • You've to provide the CA here as well so we can verify the client certificate provided by the munin master.
Sven Hoexter http://sven.stormbind.net/blog/ a blog

Canonical Design Team: Command-line usability: A terminal user’s thought process

Planet Ubuntu - Pre, 08/09/2017 - 12:43md

I’ve been thinking about the usability of command-line terminals a lot recently.

Command-line interfaces remain mystifying to many people. Usability hobbyists seem as inclined to ask why the terminal exists, as how to optimise it. I’ve also had it suggested to me that the discipline of User Experience (UX) has little to offer the Command-Line Interface (CLI), because the habits of terminal users are too inherent or instinctive to be defined and optimised by usability experts.

As an experienced terminal user with a keen interest in usability, I disagree that usability has little to offer the CLI experience. I believe that the experience can be improved through the application of usability principles just as much as for more graphical domains.

Steps to learn a new CLI tool

To help demystify the command-line experience, I’m going to lay out some of the patterns of thinking and behaviour that define my use of the CLI.

New CLI tools I’ve learned recently include snap, kubectl and nghttp2, and I’ve also dabbled in writing command-line tools myself.

Below I’ll map out an example of the steps I might go through when discovering a new command-line tool, as a basis for exploring how these tools could be optimised for CLI users.

  1. Install the tool
    • First, I might try apt install {tool} (or brew install {tool} on a mac)
    • If that fails, I’ll probably search the internet for “Install {tool}” and hope to find the official documentation
  2. Check it is installed, and if tab-complete works
    • Type the first few characters of the command name (sna for snap) followed by <tab> <tab>, to see if the command name auto-completes, signifying that the system is aware of its existence
    • Hit space, and then <tab> <tab> again, to see if it shows me a list of available sub-commands, indicating that tab completion is set up correctly for the tool
  3. Try my first command
    • I’m probably following some documentation at this point, which will be telling me the first command to run (e.g. snap install {something}), so I’ll try that out and expect prompt succinct feedback to show me that it’s working
    • For basic tools, this may complete my initial interaction with the tool. For more complex tools like kubectl or git I may continue playing with it
  4. Try to do something more complex
    • Now I’m likely no longer following a tutorial, instead I’m experimenting on my own, trying to discover more about the tool
    • If what I want to do seems complex, I’ll straight away search the internet for how to do it
    • If it seems more simple, I’ll start looking for a list of subcommands to achieve my goal
    • I start with {tool} <tab> <tab> to see if it gives me a list of subcommands, in case it will be obvious what to do next from that list
    • If that fails I’ll try, in order, {tool} <enter>, {tool} -h, {tool} --help, {tool} help or {tool} /?
    • If none of those work then I’ll try man {tool}, looking for a Unix manual entry
    • If that fails then I’ll fall back to searching the internet again
UX recommendations

Considering my own experience of CLI tools, I am reasonably confident the following recommendations make good general practice guidelines:

  • Always implement a --help option on the main command and all subcommands, and if appropriate print out some help when no options are provided ({tool} <enter>)
  • Provide both short- (e.g. -h) and long- (e.g. --help) form options, and make them guessable
  • Carefully consider the naming of all subcommands and options, use familiar words where possible (e.g. help, clean, create)
  • Be consistent with your naming – have a clear philosophy behind your use of subcommands vs options, verbs vs nouns etc.
  • Provide helpful, readable output at all times – especially when there’s an error (npm I’m looking at you)
  • Use long-form options in documentation, to make commands more self-explanatory
  • Make the tool easy to install with common software management systems (snap, apt, Homebrew, or sometimes NPM or pip)
  • Provide tab-completion. If it can’t be installed with the tool, make it easy to install and document how to set it up in your installation guide
  • Command outputs should use the appropriate output streams (STDOUT and STDERR) and should be as user-friendly and succinct as possible, and ideally make use of terminal colours

Some of these recommendations are easier to implement than others. Ideally every command should consider their subcommands and options carefully, and implement --help. But writing auto-complete scripts is a significant undertaking.

Similarly, packaging your tool as a snap is significantly easier than, for example, adding software to the official Ubuntu software sources.

Although I believe all of the above to be good general advice, I would very much welcome research to highlight the relative importance of addressing each concern.

Outstanding questions

There are a number of further questions for which the answers don’t seem obvious to me, but I’d love to somehow find out the answers:

  • Once users have learned the short-form options (e.g. -h) do they ever use the long-form (e.g. --help)?
  • Do users prefer subcommands (mytool create {something}) or options (mytool --create {something})?
  • For multi-level commands, do users prefer {tool} {object} {verb} (e.g. git remote add {remote_name}), or {tool} {verb} {object} (e.g. kubectl get pod {pod_name}), or perhaps {tool} {verb}-{object} (e.g. juju remove-application {app_name})?
  • What patterns exist for formatting command output? What’s the optimal length for users to read, and what types of formatting do users find easiest to understand?

If you know of either authoritative recommendations or existing research on these topics, please let me know in the comments below.

I’ll try to write a more in-depth follow-up to this post when I’ve explored a bit further on some of these topics.

Licensing woes

Planet Debian - Pre, 08/09/2017 - 1:37pd

On releasing modified versions of GPLv3 software in binary form only (quote anonymized):

And in my opinion it's perfectly ok to give out a binary release of a project, that is a work in progress, so that people can try it out and coment on it. It's easier for them to have it as binary and not need to compile it themselfs. If then after a (long) while the code is still only released in binary form, then it's ok to start a discussion. But only for a quick test, that is unneccessary. So people, calm down and enjoy life!

I wonder at what point we got here.

Steinar H. Gunderson http://blog.sesse.net/ Steinar H. Gunderson

It was thirty years ago today... (and a bit more): My first ever public speech!

Planet Debian - Enj, 07/09/2017 - 8:35md

I came across a folder with the most unexpected treasure trove: The text for my first ever public speech! (and some related materials)
In 1985, being nine years old, I went to the IDESE school, to learn Logo. I found my diploma over ten years ago and blogged about it in this same space. Of course, I don't expect any of you to remember what I wrote twelve years ago about a (then) twenty years old piece of paper!

I add to this very old stuff about Gunnar the four pages describing my game, Evitamono ("Avoid the monkey", approximately). I still remember the game quite vividly, including traumatic issues which were quite common back then; I wrote that «the sprites were accidentally deleted twice and the game once». I remember several of my peers telling about such experiences. Well, that is good if you account for the second system syndrome!

I also found the amazing course material for how to program sound and graphics in the C64 BASIC. That was a course taken by ten year old kids. Kids that understood that you had to write [255,129,165,244,219,165,0,102] (see pages 3-5) into a memory location starting at 53248 to redefine a character so it looked as the graphic element you wanted. Of course, it was done with a set of POKEs, as everything in C64. Or that you could program sound by setting the seven SID registers for each of the three voices containing low frequency, high frequency, low pulse, high pulse, wave control, wave length, wave amplitude in memory locations 54272 through 54292... And so on and on and on...

And as a proof that I did take the course:

...I don't think I could make most of my current BSc students make sense out of what is in the manual. But, being a kid in the 1980s, that was the only way to get a computer to do what you wanted. Yay for primitivity! :-D

AttachmentSize Speech for "Evitamono"1.29 MB Coursee material for sound and graphics programming in C64 BASIC15.82 MB Proof that I was there!4.86 MB gwolf http://gwolf.org Gunnar Wolf

FOSScamp Syros 2017 – day 3

Planet Debian - Enj, 07/09/2017 - 5:13md

The 3rd day should have started with a Debian sprint and then a LibreOffice one, taking advantage I’m still attending, as that’s my last day. But plans don’t always work out and we started 2 hours later. When everybody arrive we got everyone together for a short daily meeting (scrum style). The people were divided to 3 teams for translating:  Debian Installer, LibreOffice and Gnome. For each team we did a short list of what left and with what to start. And in the end – how does what so there will be no toe stepping. I was really proud with this and felt it was time well spent.

The current translation percentage for Albanian in LibreOffice is 60%. So my recommendation to the team is translate master only and do not touch the help translation. My plans ahead would be to improve the translation as much as possible for LibreOffice 6.0 and near the branching point (Set to November 20th by the release schedule) decide if it’s doable for the 6.0 life time or to set the goal at 6.1. In the 2nd case, we might try to backport translation back to 6.0.

For the translation itself, I’ve mentioned to the team about KeyID language pack and referred them to the nightly builds. These tools should help with keeping the translation quality high.

For the Debian team, after deciding who works on what, I’ve asked Silva to do review for the others, as doing it myself started to take more and more of my time. It’s also good that the reviewer know the target language and not like me, can catch more the syntax only mistakes. Another point, as she’s available more easily to the team while I’m leaving soon, so I hope this role of reviewer will stay as part of the team.

With the time left I mostly worked on my own tasks, which were packaging the Albanian dictionary, resulting in https://packages.debian.org/sid/myspell-sq and making sure the dictionary is also part of LibreOffice resulting in https://gerrit.libreoffice.org/#/c/41906/ . When it is accepted, I want to upload it to the LibreOffice repository so all users can download and use the dictionary.

During the voyage home (ferry, bus, plain and train), I mailed Sergio Durigan Junior, my NM applicant, with a set of questions. My first action as an AM (:

Overall FOSScamp results for Albanian translation were very close to the goal I set (100%):

  • Albanian (sq) level1 – 99%
  • Albanian (sq) level2 – 25% (the rest is pending at #874497)
  • Albanian (sq) level3 – 100%

That’s the result of work by Silva Arapi, Eva Vranici, Redon Skikuli, Anisa Kuci and Nafie Shehu.


Filed under: Debian GNU/Linux, i18n & l10n, LibreOffice Kaplan https://liorkaplan.wordpress.com Free Software Universe

My recent FAI activities

Planet Debian - Enj, 07/09/2017 - 5:03md

During DebConf 17 in Montréal I had a FAI demo session (video), where I showed how to create a customized installation CD and how to create a diskimage using the same configuration. This diskimage is ready for use with a VM software or can be booted inside a cloud environment.

During the last weeks I was working on FAI 5.4 which will be released in a few weeks. I you want to test it use

deb https://fai-project.org/download beta-testing koeln

in your sources.list file.

The most important new feature will be the cross architecture support. I managed to create an ARM64 diskimage on a x86 host and boot this inside Qemu. Currently I learn how to flash images onto my new Hikey960 board for booting my own Debian images on real hardware. The embedded world is still new for me and very different in respect to the boot process.

At DebConf, I also worked on debootstrap. I produced a set of patches which can speedup debootstrap by a factor of 2. See #871835 for details.

FAI debootstrap ARM

Thomas Lange http://blog.fai-project.org/ FAI (Fully Automatic Installation) / Plan your Installation and FAI installs your Plan

Ubuntu Studio: 17.10 Beta 1 Release

Planet Ubuntu - Enj, 07/09/2017 - 2:32md
Ubuntu Studio 17.10 Artful Aardvark Beta 1 is released! It’s that time of the release cycle again. The first beta of the upcoming release of Ubuntu Studio 17.10 is here and ready for testing. You may find the images at cdimage.ubuntu.com/ubuntustudio/releases/artful/beta-1/. More information can be found in the Beta 1 Release Notes. Reporting Bugs If […]

Reproducible Builds: Weekly report #123

Planet Debian - Enj, 07/09/2017 - 11:54pd

Here's what happened in the Reproducible Builds effort between Sunday August 27 and Saturday September 2 2017:

Talks and presentations

Holger Levsen talked about our progress and our still-far goals at BornHack 2017 (Video).

Toolchain development and fixes

The Debian FTP archive will now reject changelogs where different entries have the same timestamps.

UDD now uses reproducible-tracker.json (~25MB) which ignores our tests for Debian unstable, instead of our full set of results in reproducible.json. Our tests for Debian unstable uses a stricter definition of "reproducible" than what was recently added to Debian policy, and these stricter tests are currently more unreliable.

Packages reviewed and fixed, and bugs filed

Patches sent upstream:

Debian bugs filed:

Debian packages NMU-uploaded:

Reviews of unreproducible packages

25 package reviews have been added, 50 have been updated and 86 have been removed in this week, adding to our knowledge about identified issues.

Weekly QA work

During our reproducibility testing, FTBFS bugs have been detected and reported by:

  • Adrian Bunk (46)
  • Martín Ferrari (1)
  • Steve Langasek (1)
diffoscope development

Version 86 was uploaded to unstable by Mattia Rizzolo. It included previous weeks' contributions from:

  • Mattia Rizzolo
    • tests/binary: skip a test if the 'distro' module is not available.
    • Some code quality and style improvements.
  • Guangyuan Yang
    • tests/iso9660: support both cdrtools' genisoimage's versions of isoinfo.
  • Chris Lamb
    • comparators/xml: Use name attribute over path to avoid leaking comparison full path in output.
    • Tidy diffoscope.progress a little.
  • Ximin Luo
    • Add a --tool-prefix-binutils CLI flag. Closes: #869868
    • On non-GNU systems, prefer some tools that start with "g". Closes: #871029
    • presenters/html: Don't traverse children whose parents were already limited. Closes: #871413
  • Santiago Torres-Arias
    • diffoscope.progress: Support the new fork of python-progressbar. Closes: #873157
reprotest development

Development continued in git with contributions from:

  • Ximin Luo:
    • Add -v/--verbose which is a bit more popular.
    • Make it possible to omit "auto" when building packages.
    • Refactor how the config file works, in preparation for new features.
    • chown -h for security.
Misc.

This week's edition was written by Ximin Luo, Chris Lamb, Bernhard M. Wiedemann and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Reproducible builds folks https://reproducible.alioth.debian.org/blog/ Reproducible builds blog

Switching to xmonad + Gnome – and ditching a Mac

Planet Debian - Enj, 07/09/2017 - 4:43pd

I have been using XFCE with xmonad for years now. I’m not sure exactly how many, but at least 6 years, if not closer to 10. Today I threw in the towel and switched to Gnome.

More recently, at a new job, I was given a Macbook Pro. I wasn’t entirely sure what to think of this, but I thought I’d give it a try. I found MacOS to be extremely frustrating and confining. It had no real support for a tiling window manager, and although projects like amethyst tried to approximate what xmonad can do on Linux, they were just too limited by the platform and were clunky. Moreover, the entire UI was surprisingly sluggish; maybe that was an induced effect from animations, but I don’t think that explains it. A Debisn stretch install, even on inferior hardware, was snappy in a way that MacOS never was. So I have requested to swap for a laptop that will run Debian. The strange use of Command instead of Control for things, combined with the overall lack of configurability of keybindings, meant that I was going to always be fighting muscle memory moving from one platform to another. Not only that, but being back in the world of a Free Software OS means a lot.

Now then, back to xmonad and XFCE situation. XFCE once worked very well with xmonad. Over the years, this got more challenging. Around the jessie (XFCE 4.10) time, I had to be very careful about when I would let it save my session, because it would easily break. With stretch, I had to write custom scripts because the panel wouldn’t show up properly, and even some application icons would be invisible, if things were started in a certain order. This took much trial and error and was still cumbersome.

Gnome 3, with its tightly-coupled Gnome Shell, has never been compatible with other window managers — at least not directly. A person could have always used MATE with xmonad — but a lot of people that run XFCE tend to have some Gnome 3 apps (for instance, evince) anyhow. Cinnamon also wouldn’t work with xmonad, because it is simply another tightly-coupled shell instead of Gnome Shell. And then today I discovered gnome-flashback. gnome-flashback is a Gnome 3 environment that uses the traditional X approach with a separate window manager (metacity of yore by default). Sweet.

It turns out that Debian’s xmonad has built-in support for it. If you know the secret: apt-get install gnome-session-flashback (OK, it’s not so secret; it’s even in xmonad’s README.Debian these days) Install that, plus gnome and gdm3 and things are nice. Configure xmonad with GNOME support and poof – goodness right out of the box, selectable from the gdm sessions list.

I still have some gripes about Gnome’s configurability (or lack thereof). But I’ve got to say: This environment is the first one I’ve ever used that got external display switching very nearly right without any configuration, and I include MacOS in that. Plug in an external display, and poof – it’s configured and set up. You can hit a toggle key (Windows+P by default) to change the configurations, or use the Display section in gnome-control-center. Unplug it, and it instantly reconfigures itself to put everything back on the laptop screen. Yessss! I used to have scripts to do this in the wheezy/jessie days. XFCE in stretch had numerous annoying failures in this area which rendered the internal display completely dark until the next reboot – very frustrating. With Gnome, it just works. And, even if you have “suspend on lid closed” turned on, if the system is powered up and hooked up to an external display, it will keep running even if the lid is closed, figuring you must be using it on the external screen. Another thing the Mac wouldn’t do well.

All in all, some pretty good stuff here. I continue to be impressed by stretch. It is darn impressive to put this OS on generic hardware and have it outshine the closed-ecosystem Mac!

John Goerzen http://changelog.complete.org The Changelog

Canonical Design Team: Webteam development summary

Planet Ubuntu - Mër, 06/09/2017 - 10:15md
Iteration 6 dating between 14th to the 25th of August

This iteration saw a lot of work on tutorials.ubuntu.com and on the migration of design.ubuntu.com from WordPress to a fresh new Jekyll site project. Continued research and planning into the new snapcraft.io site, with some beginnings of the development framework.

Vanilla Framework put a lot of emphasis into polishing the existing components and porting the old theme concept patterns into the code base.

Websites issues: 66 closed, 33 opened (551 in total)

Some highlights include:
– Fixing content of card touching card edge in tutorials – https://github.com/canonical-websites/tutorials.ubuntu.com/issues/312
– Migrate canonical.com to Vanilla: Polish and custom patterns – https://github.com/canonical-websites/www.canonical.com/issues/172
– Prepare for deploy of design.ubuntu.comhttps://github.com/canonical-websites/design.ubuntu.com/issues/54
– Redirect from https://www.ubuntu.com/usn/ to https://usn.ubuntu.com/usn were broken – https://github.com/canonical-websites/www.ubuntu.com/issues/2128
design.ubuntu.com/web-style-guide build page and then hide pages – https://github.com/canonical-websites/design.ubuntu.com/issues/66
– Snapcraft prototype: Snap page – https://github.com/canonical-websites/snapcraft.io/issues/346
– Create Flask skeleton application –https://github.com/canonical-websites/snapcraft-flask/issues/2

Vanilla Framework issues: 24 closed, 16 opened (43 in total)

Some highlights include:
– Combine the entire suite of brochure theme patterns to Vanilla’s code base – https://github.com/vanilla-framework/vanilla-framework/issues/1177
– Many improvements to the documentation theme – https://github.com/vanilla-framework/vanilla-docs-theme/issues/45
– External link icon seems stretched – https://github.com/vanilla-framework/vanilla-framework/issues/1058
– .p-heading–icon pattern remove text color – https://github.com/vanilla-framework/vanilla-framework/issues/1272
– Remove margin rules on card content – https://github.com/vanilla-framework/vanilla-framework/issues/1277

All of these projects are open source. So please file issues if you find any bugs or even better propose a pull request. See you in two weeks for the next update from the web team here at Canonical.

Sebastian Dröge: Exporting a GObject C API from Rust code and using it from C, Python, JavaScript and others

Planet Ubuntu - Mër, 06/09/2017 - 3:47md

During the last days I was experimenting a bit with implementing a GObject C API in Rust. The results can be found in this repository, and this is something like an overview of the work, code walkthrough and status report. Note that this is quite long, a little bit further down you can find a table of contents and then jump to the area you’re interested in. Or read it chapter by chapter.

GObject is a C library that allows to write object-oriented, cross-platform APIs in C (which does not have support for that built-in), and provides a very expressive runtime type system with many features known from languages like Java, C# or C++. It is also used by various C library, most notably the cross-platform GTK UI toolkit and the GStreamer multimedia framework. GObject also comes with strong conventions about how an API is supposed to look and behave, which makes it relatively easy to learn new GObject based APIs as compared to generic C libraries that could do anything unexpected.

I’m not going to give a full overview about how GObject works internally and how it is used. If you’re not familiar with that it might be useful to first read the documentation and especially the tutorial. Also some C & Rust (especially unsafe Rust and FFI) knowledge would be good to have for what follows.

If you look at the code, you will notice that there is a lot of unsafe code, boilerplate and glue code. And especially code duplication. I don’t expect anyone to manually write all this code, and the final goal of all this is to have Rust macros to make the life easier. Simple Rust macros that make it as easy as in C are almost trivial to write, but what we really want here is to be able to write it all only in safe Rust in code that looks a bit like C# or Java. There is a prototype for that already written by Niko Matsakis, and a blog post with further details about it. The goal for this code is to work as a manual example that can be integrated one step at a time into the macro based solution. Code written with that macro should in the end look similar to the following

gobject_gen! { class Counter { struct CounterPrivate { f: Cell<u32> } fn add(&self, x: u32) -> u32 { let private = self.private(); let v = private.f.get() + x; private.f.set(v); v } fn get(&self) -> u32 { self.private().f.get() } } }

and be usable like

let c = Counter::new(); c.add(2); c.add(20);

The code in my repository is already integrated well into GTK-rs, but the macro generated code should also be integrated well into GTK-rs and work the same as other GTK-rs code from Rust. In addition the generated code should of course make use of all the type FFI conversion infrastructure that already exists in there and was explained by Federico in his blog post (part 1, part 2).
In the end, I would like to see such a macro solution integrated directly into the GLib bindings.

Table of Contents
  1. Why?
  2. Simple (boxed) types
  3. Object types
    1. Inheritance
    2. Virtual Methods
    3. Properties
    4. Signals
  4. Interfaces
  5. Usage from C
  6. Usage from Rust
  7. Usage from Python, JavaScript and Others
  8. What next?
Why?

Now one might ask why? GObject is yet another C library and Rust can export plain C API without any other dependencies just fine. While that is true, C is not very expressive at all and there are no conventions about how C APIs should look like and behave, so everybody does their own stuff. With GObject you would get all kinds of object-oriented programming features and strong conventions about API design. And you actually get a couple of features (inheritance, properties/signals, full runtime type system) that Rust does not have. And as bonus points, you get bindings for various other languages (Python, JavaScript, C++, C#, …) for free. More on the last point later.

Another reason why you might want to do this, is to be able to interact with existing C libraries that use GObject. For example if you want to create a subclass of some GTK widget to give it your own custom behaviour or modify its appearance, or even writing a completely new GTK widget that should be placed together with other widgets in your UI, or for implementing a new GStreamer element that implements some fancy filter or codec or … that you want to use.

Simple (boxed) types

Let’s start with the simple and boring case, which already introduces various GObject concepts. Let’s assume you already have some simple Rust type that you want to expose a C API for, and it should be GObject-style to get all the above advantages. For that, GObject has the concept of boxed types. These have to come with a “copy” and “free” function, which can do an actual copy of the object or just implement reference counting, and GObject allows to register these together with a string name for the type and then gives back a type ID (GType) that allows referencing this type.

Boxed types can then be automatically used, together with any C API they provide, from C and any other languages for which GObject support exists (i.e. basically all). It allows to use instances of these boxed types to be used in signals and properties (see further below), allows them to be stored in GValue (a container type that allows to store an instance of any other type together with its type ID), etc.

So how does all this work? In my repository I’m implementing a boxed type around a Option, one time as a “copy” type RString, another time reference counted (SharedRString). Outside Rust, both are just passed as pointers and the implementation of them is private/opaque. As such, it is possible to use any kind of Rust struct or enum and e.g. marking them as #[repr(C)] is not needed. It is also possible to use #[repr(C)] structs though, in which case the memory layout could be public and any struct fields could be available from C and other languages.

RString

The actual implementation of the type is in the imp.rs file, i.e. in the imp module. I’ll cover the other files in there at a later time, but mod.rs is providing a public Rust API around all this that integrates with GTK-rs.

The following is the whole implementation, in safe Rust:

#[derive(Clone)] pub struct RString(Option<String>); impl RString { fn new(s: Option<String>) -> RString { RString(s) } fn get(&self) -> Option<String> { self.0.clone() } fn set(&mut self, s: Option<String>) { self.0 = s; } }

Type Registration

Once the macro based solution is complete, this would be more or less all that would be required to also make this available to C via GObject, and any other languages. But we’re not there yet, and the goal here is to do it all manually. So first of all, we need to register this type somehow to GObject, for which (by convention) a C function called ex_rstring_get_type() should be defined which registers the type on the first call to get the type ID, and on further calls just returns that type ID. If you’re wondering what ex is: this is the “namespace” (C has no built-in support for namespaces) of the whole library, short for “example”. The get_type() function looks like this:

#[no_mangle] pub unsafe extern "C" fn ex_rstring_get_type() -> glib_ffi::GType { callback_guard!(); static mut TYPE: glib_ffi::GType = gobject_ffi::G_TYPE_INVALID; static ONCE: Once = ONCE_INIT; ONCE.call_once(|| { let type_name = CString::new("ExRString").unwrap(); TYPE = gobject_ffi::g_boxed_type_register_static( type_name.as_ptr(), Some(mem::transmute(ex_rstring_copy as *const c_void)), Some(mem::transmute(ex_rstring_free as *const c_void)), ); }); TYPE }

This is all unsafe Rust and calling directly into the GObject C library. We use std::sync::Once for the one-time registration of the type, and store the result in a static mut called TYPE (super unsafe, but OK here as we only ever write to it once). For registration we call g_boxed_type_register_static() from GObject (provided to Rust via the gobject-sys crate) and provide the name (via std::ffi::CString for C interoperability) and the copy and free functions. Unfortunately we have to cast them to a generic pointer, and then transmute them to a different function pointer type as the arguments and return value pointers that GObject wants there are plain void * pointers but in our code we would at least like to use RString *. And that’s all that there is to the registration. We mark the whole function as extern “C” to use the C calling conventions, and use #[no_mangle] so that the function is exported with exactly that symbol name (otherwise Rust is doing symbol name mangling), and last we make sure that no panic unwinding happens from this Rust code back to the C code via the callback_guard!() macro from the glib crate.

Memory Managment Functions

Now let’s take a look at the actual copy and free functions, and the actual constructor function called ex_rstring_new():

#[no_mangle] pub unsafe extern "C" fn ex_rstring_new(s: *const c_char) -> *mut RString { callback_guard!(); let s = Box::new(RString::new(from_glib_none(s))); Box::into_raw(s) } #[no_mangle] pub unsafe extern "C" fn ex_rstring_copy(rstring: *const RString) -> *mut RString { callback_guard!(); let rstring = &*rstring; let s = Box::new(rstring.clone()); Box::into_raw(s) } #[no_mangle] pub unsafe extern "C" fn ex_rstring_free(rstring: *mut RString) { callback_guard!(); let _ = Box::from_raw(rstring); }

These are also unsafe Rust functions that work with raw pointers and C types, but fortunately not too much is happening here.

In the constructor function we get a C string (char *) passed as argument, convert this to a Rust string (actually Option as this can be NULL) via from_glib_none() from the glib crate and then pass that to the Rust constructor of our type. from_glib_none() means that we don’t take ownership of the C string passed to us, the other variant would be from_glib_full() in which case we would take ownership. We then pack up the result in a Rust Box to place the new RString in heap allocated memory (otherwise it would be stack allocated), and use Box’s into_raw() function to get a raw pointer to the memory and not have its Drop implementation called anymore. This is then returned to the caller.

Similarly in the copy and free functions we just do some juggling with Boxes: copy take a raw pointer to our RString, calls the compiler generated clone() function to copy it all, and then packs it up in a new Box to return to the caller. The free function converts the raw pointer back to a Box, and then lets the Drop implementation of Box take care of freeing all memory related to it.

Actual Functionality

The two remaining functions are C wrappers for the get() and set() Rust functions:

#[no_mangle] pub unsafe extern "C" fn ex_rstring_get(rstring: *const RString) -> *mut c_char { callback_guard!(); let rstring = &*rstring; rstring.get().to_glib_full() } #[no_mangle] pub unsafe extern "C" fn ex_rstring_set(rstring: *mut RString, s: *const c_char) { callback_guard!(); let rstring = &mut *rstring; rstring.set(from_glib_none(s)); }

These only call the corresponding Rust functions. The set() function again uses glib’s from_glib_none() to convert from a C string to a Rust string. The get() function uses ToGlibPtrFull::to_glib_full() from GLib to convert from a Rust string (Option to be accurate) to a C string, while passing ownership of the C string to the caller (which then also has to free it at a later time).

This was all quite verbose, which is why a macro based solution for all this would be very helpful.

Corresponding C Header

Now if this API would be used from C, the header file to do so would look something like this. Probably no surprises here.

#define EX_TYPE_RSTRING (ex_rstring_get_type()) typedef struct _ExRString ExRString; GType ex_rstring_get_type (void); ExRString * ex_rstring_new (const gchar * s); ExRString * ex_rstring_copy (const ExRString * rstring); void ex_rstring_free (ExRString * rstring); gchar * ex_rstring_get (const ExRString * rstring); void ex_rstring_set (ExRString *rstring, const gchar *s);

Ideally this would also be autogenerated from the Rust code in one way or another, maybe via rusty-cheddar or rusty-binder.

SharedRString

The shared, reference counted, RString works basically the same. The only differences are in how the pointers between C and Rust are converted. For this, let’s take a look at the constructor, copy (aka ref) and free (aka unref) functions again:

#[no_mangle] pub unsafe extern "C" fn ex_shared_rstring_new(s: *const c_char) -> *mut SharedRString { callback_guard!(); let s = SharedRString::new(from_glib_none(s)); Arc::into_raw(s) as *mut _ } #[no_mangle] pub unsafe extern "C" fn ex_shared_rstring_ref( shared_rstring: *mut SharedRString, ) -> *mut SharedRString { callback_guard!(); let shared_rstring = Arc::from_raw(shared_rstring); let s = shared_rstring.clone(); // Forget it and keep it alive, we will still need it later let _ = Arc::into_raw(shared_rstring); Arc::into_raw(s) as *mut _ } #[no_mangle] pub unsafe extern "C" fn ex_shared_rstring_unref(shared_rstring: *mut SharedRString) { callback_guard!(); let _ = Arc::from_raw(shared_rstring); }

The only difference here is that instead of using a Box, std::alloc::Arc is used, and some differences in the copy (aka ref) function. Previously with the Box, we were just creating a immutable reference from the raw pointer and cloned it, but with the Arc we want to clone the Arc itself (i.e. have the same underlying object but increase the reference count). For this we use Arc::from_raw() to get back an Arc, and then clone the Arc. If we wouldn’t do anything else, at the end of the function our original Arc would get its Drop implementation called and the reference count decreased, defeating the whole point of the function. To prevent that, we convert the original Arc to a raw pointer again and “leak” it. That is, we don’t destroy the reference owned by the caller, which would cause double free problems later.

Apart from this, everything is really the same. And also the C header looks basically the same.

Object types

Now let’s start with the more interesting part: actual subclasses of GObject with all the features you know from object-oriented languages. Everything up to here was only warm-up, even if useful by itself already to expose normal Rust types to C with a slightly more expressive API.

In GObject, subclasses of the GObject base class (think of Object in Java or C#, the most basic type from which everything inherits) all get the main following features from the base class: reference counting, inheritance, virtual methods, properties, signals. Similarly to boxed types, some functions and structs are registered at runtime with the GObject library to get back a type ID but it is slightly more involved. And our structs must be #[repr(C)] and be structured in a very specific way.

Struct Definitions

Every GObject subclass has two structs: 1) one instance struct that is used for the memory layout of every instance and could contain public fields, and 2) one class struct which is storing the class specific data and the instance struct contains a pointer to it. The class struct is more or less what in C++ the vtable would be, i.e. the place where virtual methods are stored, but in GObject it can also contain fields for example. We define a new type Foo that inherits from GObject.

#[repr(C)] pub struct Foo { pub parent: gobject_ffi::GObject, } #[repr(C)] pub struct FooClass { pub parent_class: gobject_ffi::GObjectClass, }

The first element of the structs must be the corresponding struct of the class we inherit from. This later allows casting pointers of our subclass to pointers of the base class, and re-use all API implemented for the base class. In our example here we don’t define any public fields or virtual methods, in the repository the version has them but we get to that later.

Now we will actually need to be able to store some state with our objects, but we want to have that state private. For that we define another struct, a plain Rust struct this time

struct FooPrivate { name: RefCell<Option<String>>, counter: RefCell<i32>, }

This uses RefCell for each field, as in GObject modifications of objects are all done conceptually via interior mutability. For a thread-safe object these would have to be Mutex instead.

Type Registration

In the end we glue all this together and register it to the GObject type system via a get_type() function, similar to the one for boxed types

#[no_mangle] pub unsafe extern "C" fn ex_foo_get_type() -> glib_ffi::GType { callback_guard!(); static mut TYPE: glib_ffi::GType = gobject_ffi::G_TYPE_INVALID; static ONCE: Once = ONCE_INIT; ONCE.call_once(|| { let type_info = gobject_ffi::GTypeInfo { class_size: mem::size_of::<FooClass>() as u16, base_init: None, base_finalize: None, class_init: Some(FooClass::init), class_finalize: None, class_data: ptr::null(), instance_size: mem::size_of::<Foo>() as u16, n_preallocs: 0, instance_init: Some(Foo::init), value_table: ptr::null(), }; let type_name = CString::new("ExFoo").unwrap(); TYPE = gobject_ffi::g_type_register_static( gobject_ffi::g_object_get_type(), type_name.as_ptr(), &type_info, gobject_ffi::GTypeFlags::empty(), ); }); TYPE }

The main difference here is that we call g_type_register_static(), which takes a struct as parameter that contains all the information about our new subclass. In that struct we provide sizes of the class and instance struct (GObject is allocating them for us), various uninteresting fields for now and two function pointers: 1) class_init for initializing the class struct as allocated by GObject (here we would also override virtual methods, define signals or properties for example) and 2) instance_init to do the same with the instance struct. Both structs are zero-initialized in the parts we defined, and the parent parts of both structs are initialized by the code for the parent class already.

Struct Initialization

These two functions look like the following for us (the versions in the repository already do more things)

impl Foo { unsafe extern "C" fn init(obj: *mut gobject_ffi::GTypeInstance, _klass: glib_ffi::gpointer) { callback_guard!(); let private = gobject_ffi::g_type_instance_get_private( obj as *mut gobject_ffi::GTypeInstance, ex_foo_get_type(), ) as *mut Option<FooPrivate>; // Here we initialize the private data. By default it is all zero-initialized // but we don't really want to have any Drop impls run here so just overwrite the // data ptr::write( private, Some(FooPrivate { name: RefCell::new(None), counter: RefCell::new(0), }), ); } } impl FooClass { unsafe extern "C" fn init(klass: glib_ffi::gpointer, _klass_data: glib_ffi::gpointer) { callback_guard!(); // This is an Option<_> so that we can replace its value with None on finalize() to // release all memory it holds gobject_ffi::g_type_class_add_private(klass, mem::size_of::<Option<FooPrivate>>() as usize); } }

During class initialization, we tell GObject about the size of our private struct but we actually wrap it into an Option. This allows us to later replace it simply with None to deallocate all memory related to it. During instance initialization this private struct is already allocated for us by GObject (and zero-initialized), so we simply get a raw pointer to it via g_type_instance_get_private() and write an initialized struct to that pointer. Raw pointers must be used here so that the Drop implementation of Option is not called for the old, zero-initialized memory when replacing the struct.

As you might’ve noticed, we currently never set the private struct to None to release the memory, effectively leaking memory, but we get to that later when talking about virtual methods.

Constructor

With what we have so far, it’s already possible to create new instances of our subclass, and for that we also define a constructor function now

#[no_mangle] pub unsafe extern "C" fn ex_foo_new() -> *mut Foo { callback_guard!(); let this = gobject_ffi::g_object_newv( ex_foo_get_type(), 0, ptr::null_mut(), ); this as *mut Foo }

There is probably not much that has to be explained here: we only tell GObject to allocate a new instance of our specific type (by providing the type ID), which then causes the memory to be allocated and our initialization functions to be called. For the very first time, class_init would be called, for all times instance_init is called.

Methods

All this would be rather boring at this point because there is no way to actually do something with our object, so various functions are defined to work with the private data. For example to get the value of the counter

impl Foo { fn get_counter(_this: &FooWrapper, private: &FooPrivate) -> i32 { *private.counter.borrow() } } #[no_mangle] pub unsafe extern "C" fn ex_foo_get_counter(this: *mut Foo) -> i32 { callback_guard!(); let private = (*this).get_priv(); Foo::get_counter(&from_glib_borrow(this), private) }

This gets the private struct from GObject (get_priv() is a helper function that does the same as we did in instance_init), and then calls a safe Rust function implemented on our struct to actually get the value. Notable here is that we don’t pass &self to the function, but something called FooWrapper. This is a GTK-rs style wrapper type that directly allows to use any API implemented on parent classes and provides various other functionality. It is defined in mod.rs but we will talk about that later.

Inheritance

GObject allows single-inheritance from a base class, similar to Java and C#. All behaviour of the base class is inherited, and API of the base class can be used on the subclass.

I shortly hinted at how that works above already: 1) instance and class struct have the parent class’ structs as first field, so casting to pointers of the parent class work just fine, 2) GObject is told what the parent class is in the call to g_type_register_static(). We did that above already, as we inherited from GObject.

By inheriting from GObject, we e.g. can call g_object_ref() to do reference counting, or any of the other GObject API. Also it allows the Rust wrapper type defined in mod.rs to provide appropriate API for the base class to us without any casts, and to do memory management automatically. How that works is probably going to be explained in one of the following blog posts on Federico’s blog.

In the example repository, there is also another type defined which inherits from our type Foo, called Bar. It’s basically the same code again, except for the name and parent type.

#[repr(C)] pub struct Bar { pub parent: foo::imp::Foo, } #[repr(C)] pub struct BarClass { pub parent_class: foo::imp::FooClass, } #[no_mangle] pub unsafe extern "C" fn ex_bar_get_type() -> glib_ffi::GType { [...] TYPE = gobject_ffi::g_type_register_static( foo::imp::ex_foo_get_type(), type_name.as_ptr(), &type_info, gobject_ffi::GTypeFlags::empty(), ); [...] }

Virtual Methods Overriding Virtual Methods

Inheritance alone is already useful for reducing code duplication, but to make it really useful virtual methods are needed so that behaviour can be adjusted. In GObject this works similar to how it’s done in e.g. C++, just manually: you place function pointers to the virtual method implementations into the class struct and then call those. As every subclass has its own copy of the class struct (initialized with the values from the parent class), it can override these with whatever function it wants. And as it’s possible to get the actual class struct of the parent class, it is possible to chain up to the implementation of the virtual function of the parent class. Let’s look at the example of the GObject::finalize virtual method, which is called at the very end when the object is to be destroyed and which should free all memory. In there we will free our private data struct with the RefCells.

As a first step, we need to override the function pointer in the class struct in our class_init function and replace it with another function that implements the behaviour we want

impl FooClass { unsafe extern "C" fn init(klass: glib_ffi::gpointer, _klass_data: glib_ffi::gpointer) { [...] { let gobject_klass = &mut *(klass as *mut gobject_ffi::GObjectClass); gobject_klass.finalize = Some(Foo::finalize); } [...] } } impl Foo { unsafe extern "C" fn finalize(obj: *mut gobject_ffi::GObject) { callback_guard!(); // Free private data by replacing it with None let private = gobject_ffi::g_type_instance_get_private( obj as *mut gobject_ffi::GTypeInstance, ex_foo_get_type(), ) as *mut Option<FooPrivate>; let _ = (*private).take(); (*PRIV.parent_class).finalize.map(|f| f(obj)); } }

This new function could call into a safe Rust implementation, like it’s done for other virtual methods (see a bit later) but for finalize we have to do manual memory management and that’s all unsafe Rust. The way how we free the memory here is by replacing, that is take()ing the Some value out of the Option that contains our private struct, and then let it be dropped. Afterwards we have to chain up to the parent class’ implementation of finalize, which is done by calling map() on the Option that contains the function pointer.

All the function pointers in glib-sys and related crates is stored in Options to be able to handle the case of a NULL function pointer and an actual function pointer to a function.

Now for chaining up to the parent class’ finalize implementation, there’s a static, global variable containing a pointer to the parent class’ class struct, called PRIV. This is also initialized in the class_init function

struct FooClassPrivate { parent_class: *const gobject_ffi::GObjectClass, } static mut PRIV: FooClassPrivate = FooClassPrivate { parent_class: 0 as *const _, }; impl FooClass { unsafe extern "C" fn init(klass: glib_ffi::gpointer, _klass_data: glib_ffi::gpointer) { [...] PRIV.parent_class = gobject_ffi::g_type_class_peek_parent(klass) as *const gobject_ffi::GObjectClass; } }

While this is a static mut global variable, this is fine as it’s only ever written to once from class_init, and can only ever be accessed after class_init is done.

Defining New Virtual Methods

For defining new virtual methods, we would add a corresponding function pointer to the class struct and optionally initialize it to a default implementation in the class_init function, or otherwise keep it at NULL/None.

#[repr(C)] pub struct FooClass { pub parent_class: gobject_ffi::GObjectClass, pub increment: Option<unsafe extern "C" fn(*mut Foo, inc: i32) -> i32>, } impl FooClass { unsafe extern "C" fn init(klass: glib_ffi::gpointer, _klass_data: glib_ffi::gpointer) { { let foo_klass = &mut *(klass as *mut FooClass); foo_klass.increment = Some(Foo::increment_trampoline); } } }

The trampoline function provided here is responsible for converting from the C types to the Rust types, and then calling a safe Rust implementation of the virtual method.

impl Foo { unsafe extern "C" fn increment_trampoline(this: *mut Foo, inc: i32) -> i32 { callback_guard!(); let private = (*this).get_priv(); Foo::increment(&from_glib_borrow(this), private, inc) } fn increment(this: &FooWrapper, private: &FooPrivate, inc: i32) -> i32 { let mut val = private.counter.borrow_mut(); *val += inc; *val } }

To make it possible to call these virtual methods from the outside, a C function has to be defined again similar to the ones for non-virtual methods. Instead of calling the Rust implementation directly, this gets the class struct of the type that is passed in and then calls the function pointer for the virtual method implementation of that specific type.

#[no_mangle] pub unsafe extern "C" fn ex_foo_increment(this: *mut Foo, inc: i32) -> i32 { callback_guard!(); let klass = (*this).get_class(); (klass.increment.as_ref().unwrap())(this, inc) }

Subclasses would override this default implementation (or provide an actual implementation) exactly the same way, and also chain up to the parent class’ implementation like we saw before for GObject::finalize.

Properties

Similar to Objective-C and C#, GObject has support for properties. These are registered per type, have some metadata attached to them (property type, name, description, writability, valid value range, etc) and subclasses are inheriting them and can override them. The main difference between properties and struct fields is that setting/getting the property values is executing some code instead of just pointing at a memory location, and you can connect a callback to the property to be notified whenever its value changes. And they can be queried at runtime from a specific type, and set/get via their string names instead of actual C API. Allowed types for properties are everything that has a GObject type ID assigned, including all GObject subclasses, many fundamental types (integers, strings, …) and boxed types like our RString and SharedRString above.

Defining Properties

To define a property, we have to register it in the class_init function and also implement the GObject::get_property() and GObject::set_property() virtual methods (or only one of them for read-only / write-only properties). Internally inside the implementation of our GObject, the properties are identified by an integer index for which we define a simple enum, and when registered we get back a GParamSpec pointer that we should also store (for notifying about property changes for example).

#[repr(u32)] enum Properties { Name = 1, } struct FooClassPrivate { parent_class: *const gobject_ffi::GObjectClass, properties: *const Vec<*const gobject_ffi::GParamSpec>, } static mut PRIV: FooClassPrivate = FooClassPrivate { parent_class: 0 as *const _, properties: 0 as *const _, };

In class_init we then override the two virtual methods and register a new property, by providing the name, type, value of our enum corresponding to that property, default value and various other metadata. We then store the GParamSpec related to the property in a Vec, indexed by the enum value. In our example we add a string-typed “name” property that is readable and writable, but can only ever be written to during object construction.

impl FooClass { // Class struct initialization, called from GObject unsafe extern "C" fn init(klass: glib_ffi::gpointer, _klass_data: glib_ffi::gpointer) { { [...] { let gobject_klass = &mut *(klass as *mut gobject_ffi::GObjectClass); gobject_klass.finalize = Some(Foo::finalize); gobject_klass.set_property = Some(Foo::set_property); gobject_klass.get_property = Some(Foo::get_property); let mut properties = Vec::new(); let name_cstr = CString::new("name").unwrap(); let nick_cstr = CString::new("Name").unwrap(); let blurb_cstr = CString::new("Name of the object").unwrap(); properties.push(ptr::null()); properties.push(gobject_ffi::g_param_spec_string( name_cstr.as_ptr(), nick_cstr.as_ptr(), blurb_cstr.as_ptr(), ptr::null_mut(), gobject_ffi::G_PARAM_READWRITE | gobject_ffi::G_PARAM_CONSTRUCT_ONLY, )); gobject_ffi::g_object_class_install_properties( gobject_klass, properties.len() as u32, properties.as_mut_ptr() as *mut *mut _, ); PRIV.properties = Box::into_raw(Box::new(properties)); } } }

Afterwards we define the trampoline implementations for the set_property and get_property virtual methods.

impl Foo { unsafe extern "C" fn set_property( obj: *mut gobject_ffi::GObject, id: u32, value: *mut gobject_ffi::GValue, _pspec: *mut gobject_ffi::GParamSpec, ) { callback_guard!(); let this = &*(obj as *mut Foo); let private = (*this).get_priv(); // FIXME: How to get rid of the transmute? match mem::transmute::<u32, Properties>(id) { Properties::Name => { // FIXME: Need impl FromGlibPtrBorrow for Value let name = gobject_ffi::g_value_get_string(value); Foo::set_name( &from_glib_borrow(obj as *mut Foo), private, from_glib_none(name), ); } _ => unreachable!(), } } unsafe extern "C" fn get_property( obj: *mut gobject_ffi::GObject, id: u32, value: *mut gobject_ffi::GValue, _pspec: *mut gobject_ffi::GParamSpec, ) { callback_guard!(); let private = (*(obj as *mut Foo)).get_priv(); // FIXME: How to get rid of the transmute? match mem::transmute::<u32, Properties>(id) { Properties::Name => { let name = Foo::get_name(&from_glib_borrow(obj as *mut Foo), private); // FIXME: Need impl FromGlibPtrBorrow for Value gobject_ffi::g_value_set_string(value, name.to_glib_none().0); } _ => unreachable!(), } } }

In there we decide based on the index which property is meant, and then convert from/to the GValue container provided by GObject, and then call into safe Rust getters/setters.

impl Foo { fn get_name(_this: &FooWrapper, private: &FooPrivate) -> Option<String> { private.name.borrow().clone() } fn set_name(_this: &FooWrapper, private: &FooPrivate, name: Option<String>) { *private.name.borrow_mut() = name; } }

This property can now be used via the GObject API, e.g. its value can be retrieved via g_object_get(obj, “name”, &pointer_to_a_char_pointer) in C.

Construct Properties

The property we defined above had one special feature: it can only ever be set during object construction. Similarly, every property that is writable can also be set during object construction. This works by providing a value to g_object_new() in the constructor function, which then causes GObject to pass this to our set_property() implementation.

#[no_mangle] pub unsafe extern "C" fn ex_foo_new(name: *const c_char) -> *mut Foo { callback_guard!(); let prop_name_name = "name".to_glib_none(); let prop_name_str: Option<String> = from_glib_none(name); let prop_name_value = glib::Value::from(prop_name_str.as_ref()); let mut properties = [ gobject_ffi::GParameter { name: prop_name_name.0, value: prop_name_value.into_raw(), }, ]; let this = gobject_ffi::g_object_newv( ex_foo_get_type(), properties.len() as u32, properties.as_mut_ptr(), ); gobject_ffi::g_value_unset(&mut properties[0].value); this as *mut Foo }

Signals

GObject also supports signals. These are similar to events in e.g. C#, Qt or the C++ Boost signals library, and not to be confused with UNIX signals. GObject signals allow you to connect a callback that is called every time a specific event happens.

Signal Registration

Similarly to properties, these are registered in class_init together with various metadata, can be queried at runtime and are usually used by string name. Notification about property changes is implemented with signals, the GObject::notify signal.

Also similarly to properties, internally in our implementation the signals are used by an integer index. We also store that globally, indexed by a simple enum.

#[repr(u32)] enum Signals { Incremented = 0, } struct FooClassPrivate { parent_class: *const gobject_ffi::GObjectClass, properties: *const Vec<*const gobject_ffi::GParamSpec>, signals: *const Vec<u32>, } static mut PRIV: FooClassPrivate = FooClassPrivate { parent_class: 0 as *const _, properties: 0 as *const _, signals: 0 as *const _, };

In class_init we then register the signal for our type. For that we provide a name, the parameters of the signal (anything that can be stored in a GValue can be used for this again), the return value (we don’t have one here) and various other metadata. GObject then tells us the ID of the signal, which we store in our vector. In our case we define a signal named “incremented”, that is emitted every time the internal counter of the object is incremented and provides the current value of the counter and by how much it was incremented.

impl FooClass { // Class struct initialization, called from GObject unsafe extern "C" fn init(klass: glib_ffi::gpointer, _klass_data: glib_ffi::gpointer) { [...] let mut signals = Vec::new(); let name_cstr = CString::new("incremented").unwrap(); let param_types = [gobject_ffi::G_TYPE_INT, gobject_ffi::G_TYPE_INT]; // FIXME: Is there a better way? let class_offset = { let dummy: FooClass = mem::uninitialized(); ((&dummy.incremented as *const _ as usize) - (&dummy as *const _ as usize)) as u32 }; signals.push(gobject_ffi::g_signal_newv( name_cstr.as_ptr(), ex_foo_get_type(), gobject_ffi::G_SIGNAL_RUN_LAST, gobject_ffi::g_signal_type_cclosure_new(ex_foo_get_type(), class_offset), None, ptr::null_mut(), None, gobject_ffi::G_TYPE_NONE, param_types.len() as u32, param_types.as_ptr() as *mut _, )); PRIV.signals = Box::into_raw(Box::new(signals)); } }

One special part here is the class_offset. GObject allows to (optionally) define a default class handler for the signal. This is always called when the signal is emitted, and is usually a virtual method that can be overridden by subclasses. During signal registration, the offset in bytes to the function pointer of that virtual method inside the class struct is provided.

#[repr(C)] pub struct FooClass { pub parent_class: gobject_ffi::GObjectClass, pub increment: Option<unsafe extern "C" fn(*mut Foo, inc: i32) -> i32>, pub incremented: Option<unsafe extern "C" fn(*mut Foo, val: i32, inc: i32)>, } impl Foo { unsafe extern "C" fn incremented_trampoline(this: *mut Foo, val: i32, inc: i32) { callback_guard!(); let private = (*this).get_priv(); Foo::incremented(&from_glib_borrow(this), private, val, inc); } fn incremented(_this: &FooWrapper, _private: &FooPrivate, _val: i32, _inc: i32) { // Could do something here. Default/class handler of the "incremented" // signal that could be overriden by subclasses } }

This is all exactly the same as for virtual methods, just that it will be automatically called when the signal is emitted.

Signal Emission

For emitting the signal, we have to provide the instance and the arguments in an array as GValues, and then emit the signal by the ID we got back during signal registration.

impl Foo { fn increment(this: &FooWrapper, private: &FooPrivate, inc: i32) -> i32 { let mut val = private.counter.borrow_mut(); *val += inc; unsafe { let params = [this.to_value(), (*val).to_value(), inc.to_value()]; gobject_ffi::g_signal_emitv( params.as_ptr() as *mut _, (*PRIV.signals)[Signals::Incremented as usize], 0, ptr::null_mut(), ); } *val } }

While all parameters to the signal are provided as a GValue here, GObject calls our default class handler and other C callbacks connected to the signal with the corresponding C types directly. The conversion is done inside GObject and then the corresponding function is called via libffi. It is also possible to directly get the array of GValues instead though, by using the GClosure API, for which there are also Rust bindings.

Connecting to the signal can now be done via e.g. g_object_connect() from C.

C header

Similarly to the boxed types, we also have to define a C header for the exported GObject C API. This ideally would also be autogenerated from the macro based solution (e.g. with rusty-cheddar), but here we write it manually. This is mostly GObject boilerplate and conventions.

#include <glib-object.h> G_BEGIN_DECLS #define EX_TYPE_FOO (ex_foo_get_type()) #define EX_FOO(obj) (G_TYPE_CHECK_INSTANCE_CAST((obj),EX_TYPE_FOO,ExFoo)) #define EX_IS_FOO(obj) (G_TYPE_CHECK_INSTANCE_TYPE((obj),EX_TYPE_FOO)) #define EX_FOO_CLASS(klass) (G_TYPE_CHECK_CLASS_CAST((klass) ,EX_TYPE_FOO,ExFooClass)) #define EX_IS_FOO_CLASS(klass) (G_TYPE_CHECK_CLASS_TYPE((klass) ,EX_TYPE_FOO)) #define EX_FOO_GET_CLASS(obj) (G_TYPE_INSTANCE_GET_CLASS((obj) ,EX_TYPE_FOO,ExFooClass)) typedef struct _ExFoo ExFoo; typedef struct _ExFooClass ExFooClass; struct _ExFoo { GObject parent; }; struct _ExFooClass { GObjectClass parent_class; gint (*increment) (ExFoo * foo, gint inc); void (*incremented) (ExFoo * foo, gint val, gint inc); }; GType ex_foo_get_type (void); ExFoo * ex_foo_new (const gchar * name); gint ex_foo_increment (ExFoo * foo, gint inc); gint ex_foo_get_counter (ExFoo * foo); gchar * ex_foo_get_name (ExFoo * foo); G_END_DECLS

Interfaces

While GObject only allows single inheritance, it provides the ability to implement any number of interfaces on a class to provide a common API between independent types. These interfaces are similar to what exists in Java and C#, but similar to Rust traits it is possible to provide default implementations for the interface methods. Also similar to Rust traits, interfaces can declare pre-requisites: interfaces an implementor must also implement, or a base type it must inherit from.

In the repository, a Nameable interface with a get_name() method is implemented. Generally it all works exactly the same as with non-interface types and virtual methods. You register a type with GObject that inherits from G_TYPE_INTERFACE. This type only has a class struct, no instance struct. And instead of an instance struct, a typedef’d void * pointer is used. Behind that pointer would be the instance struct of the actual type implementing the interface. A default implementation of methods can be provided the same way as with virtual methods in class_init.

There are two main differences though. One is for calling an interface method

impl Nameable { // Helper functions fn get_iface(&self) -> &NameableInterface { unsafe { let klass = (*(self as *const _ as *const gobject_ffi::GTypeInstance)).g_class; let interface = gobject_ffi::g_type_interface_peek(klass as *mut c_void, ex_nameable_get_type()); &*(interface as *const NameableInterface) } } } #[no_mangle] pub unsafe extern "C" fn ex_nameable_get_name(this: *mut Nameable) -> *mut c_char { callback_guard!(); let iface = (*this).get_iface(); iface.get_name.map(|f| f(this)).unwrap_or(ptr::null_mut()) }

Instead of directly getting the class struct from the instance, we have to call some GObject API to get the interface struct of a specific interface type ID with the virtual methods.

The other difference is for implementation of the interface. Inside the get_type() function a new set of functions is registered, which are used similar to class_init for initialization of the interface struct

#[no_mangle] pub unsafe extern "C" fn ex_foo_get_type() -> glib_ffi::GType { [...] // Implement Nameable interface here let nameable_info = gobject_ffi::GInterfaceInfo { interface_init: Some(FooClass::init_nameable_interface), interface_finalize: None, interface_data: ptr::null_mut(), }; gobject_ffi::g_type_add_interface_static( TYPE, ::nameable::imp::ex_nameable_get_type(), &nameable_info, ); }); } impl FooClass { unsafe extern "C" fn init_nameable_interface( iface: glib_ffi::gpointer, _iface_data: glib_ffi::gpointer, ) { let iface = &mut *(iface as *mut ::nameable::imp::NameableInterface); iface.get_name = Some(Foo::nameable_get_name_trampoline); } }

The interface also gets a C header, which looks basically the same as for normal classes.

Usage from C

As mentioned above a few times, we export a normal (GObject) C API. For that various headers have to be written, or ideally be generated later. These can be all found here.

Nothing special has to be taken care off for using this API from C, you simply link to the generated shared library, use the headers and then use it like any other GObject based C API.

Usage from Rust

I mentioned shortly above that in the mod.rs there are gtk-rs-style Rust bindings. And these are also what would be passed (the “Wrapper” arguments) to the safe Rust implementations of the methods.

Ideally these would be autogenerated from a macro, similarly how the gir tool can do already for C based GObject libraries (this is the tool to generate most of the GLib, GTK, etc bindings for Rust).

For usage of those bindings, I’ll just let the code speak for itself

#[test] fn test_counter() { let foo = Foo::new(Some("foo's name")); let incremented = Rc::new(RefCell::new((0i32, 0i32))); let incremented_clone = incremented.clone(); foo.connect_incremented(move |_, val, inc| { *incremented_clone.borrow_mut() = (val, inc); }); assert_eq!(foo.get_counter(), 0); assert_eq!(foo.increment(1), 1); assert_eq!(*incremented.borrow(), (1, 1)); assert_eq!(foo.get_counter(), 1); assert_eq!(foo.increment(10), 11); assert_eq!(*incremented.borrow(), (11, 10)); assert_eq!(foo.get_counter(), 11); } #[test] fn test_new() { let s = RString::new(Some("bla")); assert_eq!(s.get(), Some("bla".into())); let mut s2 = s.clone(); s2.set(Some("blabla")); assert_eq!(s.get(), Some("bla".into())); assert_eq!(s2.get(), Some("blabla".into())); }

This does automatic memory management, allows to call base-class methods on instances of a subclass, provides access to methods, virtual methods, signals, properties, etc.

Usage from Python, JavaScript and Others

Now all this was a lot of boilerplate, but here comes the reason why it is probably all worth it. By exporting a GObject-style C API, we automatically get support for generating bindings for dozens of languages, without having to write any more code. This is possible thanks to the strong API conventions of GObject, and the GObject-Introspection project. Supported languages are for example Rust (of course!), Python, JavaScript (GJS and Node), Go, C++, Haskell, C#, Perl, PHP, Ruby, …

GObject-Introspection achieves this by scanning the C headers, introspecting the GObject types and then generating an XML based API description (which also contains information about ownership transfer!). This XML based API description can then be used by code generators for static, compiled bindings (e.g. Rust, Go, Haskell, …), but it can also be compiled to a so-called “typelib”. The typelib provides a C ABI that allows bindings to be generated at runtime, mostly used by scripting languages (e.g. Python and JavaScript).

To show the power of this, I’ve included a simple Python and JavaScript (GJS) application that uses all the types we defined above, and a Makefile that generates the GObject-Introspection metadata and can directly run the Python and JavaScript applications (“make run-python” and “make run-javascript”).

The Python code looks as follows

#! /usr/bin/python3 import gi gi.require_version("Ex", "0.1") from gi.repository import Ex def on_incremented(obj, val, inc): print("incremented to {} by {}".format(val, inc)) foo = Ex.Foo.new("foo's name") foo.connect("incremented", on_incremented) print("foo name: " + str(foo.get_name())) print("foo inc 1: " + str(foo.increment(1))) print("foo inc 10: " + str(foo.increment(10))) print("foo counter: " + str(foo.get_counter())) bar = Ex.Bar.new("bar's name") bar.connect("incremented", on_incremented) print("bar name: " + str(bar.get_name())) print("bar inc 1: " + str(bar.increment(1))) print("bar inc 10: " + str(bar.increment(10))) print("bar counter: " + str(bar.get_counter())) print("bar number: " + str(bar.get_number())) print("bar number (property): " + str(bar.get_property("number"))) bar.set_number(10.0) print("bar number: " + str(bar.get_number())) print("bar number (property): " + str(bar.get_property("number"))) bar.set_property("number", 20.0) print("bar number: " + str(bar.get_number())) print("bar number (property): " + str(bar.get_property("number"))) s = Ex.RString.new("something") print("rstring: " + str(s.get())) s2 = s.copy() s2.set("something else") print("rstring 2: " + str(s2.get())) s = Ex.SharedRString.new("something") print("shared rstring: " + str(s.get())) s2 = s.ref() print("shared rstring 2: " + str(s2.get()))

and the JavaScript (GJS) code as follows

#!/usr/bin/gjs const Lang = imports.lang; const Ex = imports.gi.Ex; let foo = new Ex.Foo({name: "foo's name"}); foo.connect("incremented", function(obj, val, inc) { print("incremented to " + val + " by " + inc); }); print("foo name: " + foo.get_name()); print("foo inc 1: " + foo.increment(1)); print("foo inc 10: " + foo.increment(10)); print("foo counter: " + foo.get_counter()); let bar = new Ex.Bar({name: "bar's name"}); bar.connect("incremented", function(obj, val, inc) { print("incremented to " + val + " by " + inc); }); print("bar name: " + bar.get_name()); print("bar inc 1: " + bar.increment(1)); print("bar inc 10: " + bar.increment(10)); print("bar counter: " + bar.get_counter()); print("bar number: " + bar.get_number()); print("bar number (property): " + bar["number"]); bar.set_number(10.0) print("bar number: " + bar.get_number()); print("bar number (property): " + bar["number"]); bar["number"] = 20.0; print("bar number: " + bar.get_number()); print("bar number (property): " + bar["number"]); let s = new Ex.RString("something"); print("rstring: " + s.get()); let s2 = s.copy(); s2.set("something else"); print("rstring2: " + s2.get()); let s = new Ex.SharedRString("something"); print("shared rstring: " + s.get()); let s2 = s.ref(); print("shared rstring2: " + s2.get());

Both are doing the same and nothing useful, they simple use all of the available API.

What next?

While everything here can be used as-is already (and I use a variation of this in gst-plugin-rs, a crate to write GStreamer plugins in Rust), it’s rather inconvenient. The goal of this blog post is to have a low-level explanation about how all this works in GObject with Rust, and to have a “template” to use for Nikos’ gnome-class macro. Federico is planning to work on this in the near future, and step by step move features from my repository to the macro. Work on this will also be done at the GNOME/Rust hackfest in November in Berlin, which will hopefully yield a lot of progress on the macro but also on the bindings in general.

In the end, this macro would ideally end up in the glib-rs bindings and can then be used directly by anybody to implement GObject subclasses in Rust. At that point, this blog post can hopefully help a bit as documentation to understand how the macro works.

MATE 1.18 landed in Debian testing

Planet Debian - Mër, 06/09/2017 - 11:04pd

This is to announce that finally all MATE Desktop 1.18 components have landed in Debian testing (aka buster).

Credits

Again a big thanks to the packaging team (esp. Vangelis Mouhtsis and Martin Wimpress, but also to Jeremy Bicha for constant advice and Aron Xu for joining the Debian+Ubuntu MATE Packaging Team and merging all the Ubuntu zesty and artful branches back to master).

Fully Available on all Debian-supported Architectures

The very special thing about this MATE 1.18 release for Debian is that MATE is now available on all Debian hardware architectures. See "Buildd" column on our DDPO overview page [1]. Thanks to all the people from the Debian porters realm for providing feedback to my porting questions.

References sunweaver http://sunweavers.net/blog/blog/1 sunweaver's blog

Kees Cook: security things in Linux v4.13

Planet Ubuntu - Mër, 06/09/2017 - 1:01pd

Previously: v4.12.

Here’s a short summary of some of interesting security things in Sunday’s v4.13 release of the Linux kernel:

security documentation ReSTification
The kernel has been switching to formatting documentation with ReST, and I noticed that none of the Documentation/security/ tree had been converted yet. I took the opportunity to take a few passes at formatting the existing documentation and, at Jon Corbet’s recommendation, split it up between end-user documentation (which is mainly how to use LSMs) and developer documentation (which is mainly how to use various internal APIs). A bunch of these docs need some updating, so maybe with the improved visibility, they’ll get some extra attention.

CONFIG_REFCOUNT_FULL
Since Peter Zijlstra implemented the refcount_t API in v4.11, Elena Reshetova (with Hans Liljestrand and David Windsor) has been systematically replacing atomic_t reference counters with refcount_t. As of v4.13, there are now close to 125 conversions with many more to come. However, there were concerns over the performance characteristics of the refcount_t implementation from the maintainers of the net, mm, and block subsystems. In order to assuage these concerns and help the conversion progress continue, I added an “unchecked” refcount_t implementation (identical to the earlier atomic_t implementation) as the default, with the fully checked implementation now available under CONFIG_REFCOUNT_FULL. The plan is that for v4.14 and beyond, the kernel can grow per-architecture implementations of refcount_t that have performance characteristics on par with atomic_t (as done in grsecurity’s PAX_REFCOUNT).

CONFIG_FORTIFY_SOURCE
Daniel Micay created a version of glibc’s FORTIFY_SOURCE compile-time and run-time protection for finding overflows in the common string (e.g. strcpy, strcmp) and memory (e.g. memcpy, memcmp) functions. The idea is that since the compiler already knows the size of many of the buffer arguments used by these functions, it can already build in checks for buffer overflows. When all the sizes are known at compile time, this can actually allow the compiler to fail the build instead of continuing with a proven overflow. When only some of the sizes are known (e.g. destination size is known at compile-time, but source size is only known at run-time) run-time checks are added to catch any cases where an overflow might happen. Adding this found several places where minor leaks were happening, and Daniel and I chased down fixes for them.

One interesting note about this protection is that is only examines the size of the whole object for its size (via __builtin_object_size(..., 0)). If you have a string within a structure, CONFIG_FORTIFY_SOURCE as currently implemented will make sure only that you can’t copy beyond the structure (but therefore, you can still overflow the string within the structure). The next step in enhancing this protection is to switch from 0 (above) to 1, which will use the closest surrounding subobject (e.g. the string). However, there are a lot of cases where the kernel intentionally copies across multiple structure fields, which means more fixes before this higher level can be enabled.

NULL-prefixed stack canary
Rik van Riel and Daniel Micay changed how the stack canary is defined on 64-bit systems to always make sure that the leading byte is zero. This provides a deterministic defense against overflowing string functions (e.g. strcpy), since they will either stop an overflowing read at the NULL byte, or be unable to write a NULL byte, thereby always triggering the canary check. This does reduce the entropy from 64 bits to 56 bits for overflow cases where NULL bytes can be written (e.g. memcpy), but the trade-off is worth it. (Besdies, x86_64’s canary was 32-bits until recently.)

IPC refactoring
Partially in support of allowing IPC structure layouts to be randomized by the randstruct plugin, Manfred Spraul and I reorganized the internal layout of how IPC is tracked in the kernel. The resulting allocations are smaller and much easier to deal with, even if I initially missed a few needed container_of() uses.

randstruct gcc plugin
I ported grsecurity’s clever randstruct gcc plugin to upstream. This plugin allows structure layouts to be randomized on a per-build basis, providing a probabilistic defense against attacks that need to know the location of sensitive structure fields in kernel memory (which is most attacks). By moving things around in this fashion, attackers need to perform much more work to determine the resulting layout before they can mount a reliable attack.

Unfortunately, due to the timing of the development cycle, only the “manual” mode of randstruct landed in upstream (i.e. marking structures with __randomize_layout). v4.14 will also have the automatic mode enabled, which randomizes all structures that contain only function pointers.

A large number of fixes to support randstruct have been landing from v4.10 through v4.13, most of which were already identified and fixed by grsecurity, but many were novel, either in newly added drivers, as whitelisted cross-structure casts, refactorings (like IPC noted above), or in a corner case on ARM found during upstream testing.

lower ELF_ET_DYN_BASE
One of the issues identified from the Stack Clash set of vulnerabilities was that it was possible to collide stack memory with the highest portion of a PIE program’s text memory since the default ELF_ET_DYN_BASE (the lowest possible random position of a PIE executable in memory) was already so high in the memory layout (specifically, 2/3rds of the way through the address space). Fixing this required teaching the ELF loader how to load interpreters as shared objects in the mmap region instead of as a PIE executable (to avoid potentially colliding with the binary it was loading). As a result, the PIE default could be moved down to ET_EXEC (0x400000) on 32-bit, entirely avoiding the subset of Stack Clash attacks. 64-bit could be moved to just above the 32-bit address space (0x100000000), leaving the entire 32-bit region open for VMs to do 32-bit addressing, but late in the cycle it was discovered that Address Sanitizer couldn’t handle it moving. With most of the Stack Clash risk only applicable to 32-bit, fixing 64-bit has been deferred until there is a way to teach Address Sanitizer how to load itself as a shared object instead of as a PIE binary.

early device randomness
I noticed that early device randomness wasn’t actually getting added to the kernel entropy pools, so I fixed that to improve the effectiveness of the latent_entropy gcc plugin.

That’s it for now; please let me know if I missed anything. As a side note, I was rather alarmed to discover that due to all my trivial ReSTification formatting, and tiny FORTIFY_SOURCE and randstruct fixes, I made it into the most active 4.13 developers list (by patch count) at LWN with 76 patches: a whopping 0.6% of the cycle’s patches. ;)

Anyway, the v4.14 merge window is open!

© 2017, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

security things in Linux v4.13

Planet Debian - Mër, 06/09/2017 - 1:01pd

Previously: v4.12.

Here’s a short summary of some of interesting security things in Sunday’s v4.13 release of the Linux kernel:

security documentation ReSTification
The kernel has been switching to formatting documentation with ReST, and I noticed that none of the Documentation/security/ tree had been converted yet. I took the opportunity to take a few passes at formatting the existing documentation and, at Jon Corbet’s recommendation, split it up between end-user documentation (which is mainly how to use LSMs) and developer documentation (which is mainly how to use various internal APIs). A bunch of these docs need some updating, so maybe with the improved visibility, they’ll get some extra attention.

CONFIG_REFCOUNT_FULL
Since Peter Zijlstra implemented the refcount_t API in v4.11, Elena Reshetova (with Hans Liljestrand and David Windsor) has been systematically replacing atomic_t reference counters with refcount_t. As of v4.13, there are now close to 125 conversions with many more to come. However, there were concerns over the performance characteristics of the refcount_t implementation from the maintainers of the net, mm, and block subsystems. In order to assuage these concerns and help the conversion progress continue, I added an “unchecked” refcount_t implementation (identical to the earlier atomic_t implementation) as the default, with the fully checked implementation now available under CONFIG_REFCOUNT_FULL. The plan is that for v4.14 and beyond, the kernel can grow per-architecture implementations of refcount_t that have performance characteristics on par with atomic_t (as done in grsecurity’s PAX_REFCOUNT).

CONFIG_FORTIFY_SOURCE
Daniel Micay created a version of glibc’s FORTIFY_SOURCE compile-time and run-time protection for finding overflows in the common string (e.g. strcpy, strcmp) and memory (e.g. memcpy, memcmp) functions. The idea is that since the compiler already knows the size of many of the buffer arguments used by these functions, it can already build in checks for buffer overflows. When all the sizes are known at compile time, this can actually allow the compiler to fail the build instead of continuing with a proven overflow. When only some of the sizes are known (e.g. destination size is known at compile-time, but source size is only known at run-time) run-time checks are added to catch any cases where an overflow might happen. Adding this found several places where minor leaks were happening, and Daniel and I chased down fixes for them.

One interesting note about this protection is that is only examines the size of the whole object for its size (via __builtin_object_size(..., 0)). If you have a string within a structure, CONFIG_FORTIFY_SOURCE as currently implemented will make sure only that you can’t copy beyond the structure (but therefore, you can still overflow the string within the structure). The next step in enhancing this protection is to switch from 0 (above) to 1, which will use the closest surrounding subobject (e.g. the string). However, there are a lot of cases where the kernel intentionally copies across multiple structure fields, which means more fixes before this higher level can be enabled.

NULL-prefixed stack canary
Rik van Riel and Daniel Micay changed how the stack canary is defined on 64-bit systems to always make sure that the leading byte is zero. This provides a deterministic defense against overflowing string functions (e.g. strcpy), since they will either stop an overflowing read at the NULL byte, or be unable to write a NULL byte, thereby always triggering the canary check. This does reduce the entropy from 64 bits to 56 bits for overflow cases where NULL bytes can be written (e.g. memcpy), but the trade-off is worth it. (Besdies, x86_64’s canary was 32-bits until recently.)

IPC refactoring
Partially in support of allowing IPC structure layouts to be randomized by the randstruct plugin, Manfred Spraul and I reorganized the internal layout of how IPC is tracked in the kernel. The resulting allocations are smaller and much easier to deal with, even if I initially missed a few needed container_of() uses.

randstruct gcc plugin
I ported grsecurity’s clever randstruct gcc plugin to upstream. This plugin allows structure layouts to be randomized on a per-build basis, providing a probabilistic defense against attacks that need to know the location of sensitive structure fields in kernel memory (which is most attacks). By moving things around in this fashion, attackers need to perform much more work to determine the resulting layout before they can mount a reliable attack.

Unfortunately, due to the timing of the development cycle, only the “manual” mode of randstruct landed in upstream (i.e. marking structures with __randomize_layout). v4.14 will also have the automatic mode enabled, which randomizes all structures that contain only function pointers.

A large number of fixes to support randstruct have been landing from v4.10 through v4.13, most of which were already identified and fixed by grsecurity, but many were novel, either in newly added drivers, as whitelisted cross-structure casts, refactorings (like IPC noted above), or in a corner case on ARM found during upstream testing.

lower ELF_ET_DYN_BASE
One of the issues identified from the Stack Clash set of vulnerabilities was that it was possible to collide stack memory with the highest portion of a PIE program’s text memory since the default ELF_ET_DYN_BASE (the lowest possible random position of a PIE executable in memory) was already so high in the memory layout (specifically, 2/3rds of the way through the address space). Fixing this required teaching the ELF loader how to load interpreters as shared objects in the mmap region instead of as a PIE executable (to avoid potentially colliding with the binary it was loading). As a result, the PIE default could be moved down to ET_EXEC (0x400000) on 32-bit, entirely avoiding the subset of Stack Clash attacks. 64-bit could be moved to just above the 32-bit address space (0x100000000), leaving the entire 32-bit region open for VMs to do 32-bit addressing, but late in the cycle it was discovered that Address Sanitizer couldn’t handle it moving. With most of the Stack Clash risk only applicable to 32-bit, fixing 64-bit has been deferred until there is a way to teach Address Sanitizer how to load itself as a shared object instead of as a PIE binary.

early device randomness
I noticed that early device randomness wasn’t actually getting added to the kernel entropy pools, so I fixed that to improve the effectiveness of the latent_entropy gcc plugin.

That’s it for now; please let me know if I missed anything. As a side note, I was rather alarmed to discover that due to all my trivial ReSTification formatting, and tiny FORTIFY_SOURCE and randstruct fixes, I made it into the most active 4.13 developers list (by patch count) at LWN with 76 patches: a whopping 0.6% of the cycle’s patches. ;)

Anyway, the v4.14 merge window is open!

© 2017, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

kees https://outflux.net/blog Debian – codeblog

Made with Creative Commons: Over half translated, yay!

Planet Debian - Mar, 05/09/2017 - 9:05md

An image speaks for a thousand words...

And our translation project is worth several thousand words!
I am very happy and surprised to say we have surpassed the 50% mark of the Made with Creative Commons translation project. We have translated 666 out of 1210 strings (yay for 3v1l numbers)!
I have to really thank Weblate for hosting us and allowing for collaboration to happen there. And, of course, I have to thank the people that have jumped on board and helped the translation — We are over half way there! Lets keep pushing!



PS - If you want to join the project, just get in Weblate and start translating right away, either to Spanish or other languages! (Polish, Dutch and Norwegian Bokmål are on their way) If you translate into Spanish, *please* read and abide by the specific Spanish translation guidelines.

gwolf http://gwolf.org Gunnar Wolf

Faqet

Subscribe to AlbLinux agreguesi