You are here

Agreguesi i feed

Reproducible builds folks: Reproducible Builds: Weekly report #192

Planet Debian - Mar, 01/01/2019 - 8:44md

Here’s what happened in the Reproducible Builds effort between Sunday December 23 and Saturday December 29 2018:

Packages reviewed and fixed, and bugs filed

In addition, Mattia Rizzolo filed a build failure bug.

reproducible-builds.org website development

Chris Lamb made a huge number of updates to our reproducible-builds.org project website this week:

  • Set a temporary logo for Christmas. [][]

  • Move our homepage to the new visual style. [][]

  • Split, tidy and expand footer. [][][] and link the main heading element of blog posts “back” to themselves [].

  • Move the tools, resources and events pages to new visual style. [][][][]

  • Update the support mechanisms for the weekly reports, such as dropping the migrate-blog-posts script [] as well as fixing some title handling [] [].

  • Improve a number of styles, such as blockquotes, linked headings should not have “link” styling, unpublished blog post drafts, etc. [][][]

  • Tidy and highlight the display of our sponsors. [][]

  • Add the ablity to override the entire <head> title [] and improve spacing etc. on mobile browsers [].

In addition, Arnout Engelen and Hervé Boutemy made an initial stab at documenting the JVM “buildinfo” format. [][]

Test framework development

There were a number of updates to our Jenkins-based testing framework that powers tests.reproducible-builds.org this week, including:

  • Holger Levsen updated the Coreboot support to point to the new Git repositories to new URI. []

  • Mattia Rizzolo updated the “wrong future” check for the 2019 (part of our many build variations). [][] as well as a various bits of node maintenance. [][][]

This week’s edition was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Russ Allbery: 2018 Book Reading in Review

Planet Debian - Mar, 01/01/2019 - 8:38md

Despite the best of intentions to spread my reading out more evenly across the year, much of 2018's reading happened in concentrated bursts during vacation (particularly my fall vacation, during which I read eleven books in a little over two weeks). Politics and other online reading continued to be an irritating distraction, although I made some forward progress at picking up a book instead of Twitter.

My reading goal for last year was to make time and energy for deeper, more demanding, and more rewarding books. I think the verdict is mixed, but I didn't do too poorly. I finished Jemisin's Broken Earth trilogy (more on that below), which certainly qualifies and which was one of the year's highlights, and dug deep into a few other rewarding books. For 2019, my goal is to maintain my current reading pace (hopefully including the gradual improvement year over year) and focus on catching up on award winners and nominees to broaden my reading beyond favorite authors.

Two books, both fiction, received 10 out of 10 ratings from me this year: My Grandmother Asked Me to Tell You She's Sorry, by Fredrik Backman, and Record of a Spaceborn Few, by Becky Chambers. Backman's novel is a delightful character story — funny, open-hearted, and gracious — with a wonderful seven-year-old protagonist (and that's something you'll rarely hear me say). It was the best book I read this year. Record of a Spaceborn Few was the most emotionally affecting book I read in 2018 (by far): a deeply moving story about community and belonging and not belonging, and about culture and why it's important. The narrative structure is unusual and the writing is less evenly high quality than Backman's, but it was exactly the book I needed to read when I read it. I think it's Chambers's best work to date, and that's saying a lot.

The novels that received 9 out of 10 ratings from me in 2018 were The Obelisk Gate and The Stone Sky, the second and third books in N.K. Jemisin's Broken Earth trilogy. Given Jemisin's three Hugo awards for this series and the wealth of online reviews, you probably don't need me to tell you how good they are. I found the series hard to read, since it's full of strong negative emotions and takes a very sharp look at pain, loss, and oppression, but I also thought it was worth the emotional effort. This trilogy is something very special in SFF and fully deserves the attention that it's gotten.

There was one more fiction 9 out of 10 rating this year, which also came as a complete surprise to me: walkingnorth's online graphic novel Always Human. This was one of the year's pure delights: gentle, kind, thoughtful, empathetic, and sweet. I am very grateful to James Nicoll for reviewing it; I never would have discovered it otherwise, and was able to share it with several other people.

The sole non-fiction 9 out of 10 this year was Zeynep Tufekci's excellent Twitter and Tear Gas, a thoughtful, critical, and deep look at the intersection of politics and online social networks that avoids facile moralizing and embraces the complex interactions we have with for-profit web sites that have far outgrown the understanding of the corporations that run them. I think (or at least hope) there's more awareness now, at the end of 2018, of the way that totalitarian regimes undermine political engagement not via suppression but via flooding networks with garbage news, fake personas, heated opinions, and made-up stories. Tufekci was studying this before it was widely talked about, and Twitter and Tear Gas is still a reliable guide to how political engagement works in online spaces.

The full analysis includes some additional personal reading statistics, probably only of interest to me.

Rhonda D'Vine: Political Correct Communication

Planet Debian - Mar, 01/01/2019 - 4:33md

It seems almost as if being political correct is something people do not want to be. As a matter of fact, to move forward as humanity, we though need it very much. Let's take a look at the why and what it actually means, shall we?

I think we all have heard of the Golden Rule: "The Golden Rule is the principle of treating others as one's self would wish to be treated." I hope that we can agree on that. The idea behind is to envision oneself in the other person's shoes, figuratively, and see what it would do to yourself. If you don't like it, don't do it. Sounds easy?

Well, it isn't. When it comes to discrimination, which is something systematic, that doesn't work. There is also a power difference involved in discrimination, and here it starts: It's not possible to envision what some words might do unto others. Most of of the people within the Debian community are most probably white, able-bodied, cis (identifying with the gender assigned at birth), hetero, and male. Just to name a few most prominent categories. So even if we try to envision oneself in the place of the other person, we haven't experienced systematic discrimination like racial profiling, not able to enter a restaurant, being looked strange at whatever toilet we go to, have heads turned on us and people whispering when walking down the street hand-in-hand with our partners, or being cat called. And we might envision that being called "fag" isn't the nicest thing, people forget one thing: There is a huge power difference especially also in language.

How many discriminatory words can you come up for black people? Disabled people? Non-hetero people? Trans people? Women? And then take a step back ... and try to think about how many discriminatory words you can come up with for white, able-bodied, hetero, cis and male people. And then try to realize how even language plays into that power imbalance. Especially on the internet where the only thing you get from others is written language. So the one way to work with that is to actually listen to those facing discrimination and acknowledging that some words are off limit.

So next time you tell someone they are just a special snowflake, or that they should just swallow it down because that's the way things work ... think about this. And think about what you actually are transporting when you oppose to a political correct approach: When you consider political correctness something awful to strive for because it seemingly limits how you speak to and about others. Because honey, no, it doesn't. Anytime you belittle a political correct approach you are just showing one thing: That you are unwilling to be a safe space for the people around you, and simply don't care.

Oh, and one more thing: Free Software and Debian in specific always was political. Don't tell me that's news to you. Working on Free Software is an extremely strong political statement. It is to improve the world for everyone through making software available to everyone. And yes, that everyone includes non-white, non-cis, non-ablebodied, non-hetero and non-male people too, surprisingly to some it seems.

Enjoy, and happy new year!

P.S.: Part of this content is inspired by the German language book: Eine Frage der Moral from Anatol Stefanowitsch. If you understand German I urge you to read it. It gives a good insight.

/debian | permanent link | Comments: 9 | Flattr this

Paul Wise: FLOSS Activities December 2018

Planet Debian - Mar, 01/01/2019 - 1:29md
Changes Issues Review Administration
  • Debian: answer query about LDAP signing
  • Debian wiki: unblacklist IP addresses, fix login issue, whitelist email addresses
Communication Sponsors

The purple-discord work was sponsored by my employer. All other work was done on a volunteer basis.

Junichi Uekawa: 2019 came.

Planet Debian - Mar, 01/01/2019 - 10:11pd
2019 came. I am enjoying that I can run Debian on my chromebook (Crostini). It's much better than termux and making my development more convenient. I am using Crostini and GCE environment for development. However the best thing I bought last year was probably double-seater stroller.

Russ Allbery: Review: The Dragon's Path

Planet Debian - Mar, 01/01/2019 - 5:09pd

Review: The Dragon's Path, by Daniel Abraham

Series: Dagger and the Coin #1 Publisher: Orbit Copyright: June 2011 ISBN: 0-316-13467-8 Format: Kindle Pages: 579

I read this book as a free bonus included in a Kindle edition of Leviathan Wakes by James S.A. Corey (a pen name for Daniel Abraham and Ty Franck). The ISBN information is for that book.

Cithrin bel Sarcour is a ward of the Medean Bank branch in Vanai and has been since she was four years old. She's a teenager, half-Firstblood and half-Cinnae and therefore not entirely welcome in either group, secretly in love with Besel, and being trained in economics by Magister Immaniel. War is coming to Vanai and with it demands from the prince of Vanai for the bank's money. When Besel is murdered, Cithrin is the only one left to secretly smuggle the bank's riches and account books out of the city.

Captain Marcus Wester is working as a caravan guard. Some would consider this a huge step down from his past as a military leader and a killer of kings, but after the death of his wife and daughter, he has no interest in war. He particularly has no interest in being drafted by the prince of Vanai into fighting for the city, even though he can't hire men to fill out his company. That's how he ends up guarding, with only his long-time lieutenant and a hired troop of actors, the same caravan that secretly includes Cithrin.

The war between the Severed Throne of Antea and Vanai is just part of larger political maneuvering between several adjacent kingdoms and the Free Cities (which seem modeled after Italian cities). The reader sees that part of the story through the eyes of Dawson, a member of the royal court, and the hapless Geder, a minor noble who is an officer in the Antean army but who would much rather be searching out and translating speculative essays. These separate strands do cross eventually, but they don't merge, at least in this book.

The reviews I saw of this book were somewhat mixed, but I decided to read it anyway because I was promised fantasy based on medieval banking. And, indeed, the portions with Cithrin are often satisfyingly different than normal fantasy fare and are the best part of the book. Unfortunately, the reviews were right in another respect: The Dragon's Path is very slow. There are pages and pages of setup, pages more of Cithrin being scared and uncertain, lots of Dawson's political maneuvering and Geder's ineptness, and not a tremendous amount of plot for the first half of the book. Things do eventually start happening, but Abraham is clearly not interested in hurrying the story along.

The Dragon's Path is what I'll call George R.R. Martin fantasy, since The Song of Ice and Fire is probably the most famous example of the style. There's a large, multi-threaded story with multiple viewpoint characters, each told in tight third person. Chapters cycle between viewpoint characters and are long enough to be a substantial chunk of story. And, with relatively little narrative signaling, several of the viewpoint characters turn out to be awful, horrible people. Unlike Martin, though, Abraham doesn't pull off sudden reversals of perspective where the reader starts to like characters they previously hated. Rather the contrary: the more I learned about Dawson and Geder, the more I disliked them, albeit for far different reasons.

I'm not sure what to make of this book. The finance parts, and the times when Cithrin was able to show how much she learned from spending her formative years in a bank, were fun and refreshingly different from typical epic fantasy. But then Abraham sharply undermined Cithrin's expertise in a way that is understandable and probably realistic, but which wasn't at all pleasant to read about. I enjoyed the world backstory, with its dragon wars and strangely permanent dragon jade, apparently magical draconic genetic engineering that created multiple variations of humanity, and sense of hinted-at history. I'd like to learn more about it, but the details are so slow in coming. The writing is solid, the details believable, and the world vivid and complex, but Abraham keeps pulling the rug out from under my plot expectations, and not in the good way. Characters showing unexpectedly successful expertise is an old trope but one that I enjoy; characters unexpectedly turning out to be self-centered asses isn't as fun. Abraham repeatedly promises catharsis and then undermines it.

Dawson and Geder are excellent examples of my mixed feelings. Abraham writes Dawson as a rather likable, principled person at first, a close friend and defender of the king. His later actions, and the details of his political positions shown over the course of this book, slowly paint a far different picture without changing the narrative tone. I'm fairly sure Abraham is doing this on purpose and the reader is intended to slowly change their mind about Dawson; indeed, I suspect it's subtle commentary on the sort of monarchy-supporting characters show up in traditional fantasy. But it's still disconcerting. I wanted to like Dawson, and particularly his wife, despite disagreeing with everything they stand for. That can be an enjoyable and challenging reading experience, but it wasn't for me in this book.

Geder is a more abrupt case. It's hard not to be sympathetic to him at the beginning of the book: he just wants to read and translate histories and speculation, and is bullied by other nobles and miserable on campaign. I thought Abraham was setting up a coming-of-age story or an opportunity for Geder to unexpectedly turn out to be more competent than he expected. I won't spoil what actually happens but it's... not that, not at all, and leaves Geder as another character who is deeply disturbing to read about.

The Dragon's Path is well-written, deep, realistic in feel, and caught my interest with its world-building. I'm invested in the story and do want to know what happens next. I'm also rooting for Cithrin (and for Wester's lieutenant, who's probably my favorite character). But it took me a long time to read this book, and I'm not sure it was worth the investment. I'm even less sure that the investment of reading another four books in this world will be worth the payoff. If I had more confidence that good people would rise to the occasion and there would be a satisfactory conclusion for all the horrible things that happen in this book, I'd be more tempted, but the tone of this first volume doesn't make me optimistic.

I still want to read a series about banking and finance set against an epic fantasy background. I want to learn more about the dragons and the jade and the wars Abraham hints at. But I suspect this will be one of those series that I occasionally think about but never get around to reading.

Followed by The King's Blood.

Rating: 5 out of 10

Chris Lamb: Free software activities in December 2018

Planet Debian - Hën, 31/12/2018 - 7:06md

Here is my monthly update covering what I have been doing in the free software world during December 2018 (previous month):

Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws almost all software is distributed pre-compiled to end users.

The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

This month I:


I also made the following changes to our tooling:


diffoscope

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues.

  • Compare .zip file comments with zipnote(1). (#901757)
  • Fix a test_mozzip_compressed_files failure under Alpine Linux. (#916353)
  • Use file_header to simplify magic detection and version parsing. [...][...][...]
  • Calculate the path to test .icc file to avoid a error with new versions of Pytest. (#916226)
  • Drop old debbindiff Breaks/Replaces. [...]
  • Correct a "positives" typo. [...]


strip-nondeterminism

strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build.

  • Remove javaproperties handler after Emmanuel Bourg's patch was shipped in OpenJDK 11. (#914289)
  • Drop .ar handler; binutils output should now be reproducible. (#781262, #843811)
  • Ignore encrypted .zip files; we can never normalise them. (#852207)


Debian


Patches contributed Debian LTS

This month I have worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project.


Uploads
  • redis:

    • 5:5.0.3-1 — New upstream release.
    • 5:5.0.3-2 — Pass --no-as-needed to ensure linking to the Lua libraries on systems where this the default. (#916831)
  • python-django:

    • 1.11.16-4 — Cherry-pick two patches from upstream to fix test failures under Python 3.7. (#891753)
    • 1.11.17-1 — New upstream bugfix release.
    • 2.1.4-1 — New upstream release.
    • 1.11.17-2 & 2.1.4-2 — Apply patch from upstream to fix compatibility with SQLite 3.26. (#915626)
  • libfiu (0.98-1) — New upstream release.

  • python-hiredis (0.3.0-1 & 0.3.1-1) — New upstream releases.

  • python-redis (3.0.1-2 & 3.0.1-3) — Mark a number of failing autopkgtests as XFAIL.

  • lastpass-cli (1.3.1-6) — Add missing pkg-config to Build-Depends. (#916268)

  • creoleparser (0.7.4-2) & django-pagination (1.0.7-2) — Completely overhaul packaging.

  • adminer (4.7.0-2) — Additionally depend on the php-fpm virtual package. (#906692)


Debian bugs filed
  • ITS (Intent to Salvage): mtools. (#916127)

  • busybox: "Too many levels of symbolic links". (#915830)

  • dpkg-mergechangelogs: Strips vim modelines. (#916056)

  • fonts-roboto: Please ship .woff files (eg. Roboto-Light-webfont.woff). (#915360)

  • jenkins.debian.org: Lintian test jobs have not run since November. (#917119)

  • netplan.io: Please add a Homepage field. (#917233)

  • python-envs: Please replace Homepage: reference. (#917230)

  • usrmerge: Please handle aborted conversions more gracefully. (#917226)

I also filed bugs against packages that use vendor-specific patch series files for deluge, fail2ban, filezilla, hexchat, libfreenect, libxfce4util, liferea, mate-power-manager, mate-terminal, mixxx, numix-gtk-theme, packagekit, smuxi, xchat & xfce4-smartbookmark-plugin.

FTP Team


As a Debian FTP assistant I ACCEPTed 141 packages: ansible, bambootracker, birdtray, bitlbee-mastodon, blis, capnproto, centreon-broker, chargebee-python, chargebee2-python, dar, darknet, dask-sphinx-theme, dav4tbsync, davs2, displaycal, django-anymail, dsmidiwifi, eas4tbsync, emerald, emerald-themes, erlang-horse, fusion-icon, ghostwriter, gitlab, go-cpe-dictionary, go-exploitdb, golang-1.12, golang-github-datadog-zstd, golang-github-justinas-alice, golang-github-namsral-flag, google-compute-image-packages, grim, grpc, haskell-gi-atk, haskell-gi-cairo, haskell-gi-dbusmenu, haskell-gi-dbusmenugtk3, haskell-gi-gdk, haskell-gi-gdkpixbuf, haskell-gi-gdkx11, haskell-gi-gio, haskell-gi-glib, haskell-gi-gobject, haskell-gi-gtk, haskell-gi-gtk-hs, haskell-gi-pango, haskell-gi-vte, haskell-gi-xlib, haskell-gtk-sni-tray, haskell-gtk-strut, haskell-status-notifier-item, haskell-system-posix-redirect, haskell-termonad, haskell-xml-html-qq, i3pystatus, jaxb, lablgtk3, libcloudflare-client-perl, libconfig-model-backend-yaml-perl, libcpan-common-index-perl, libhostfile-manager-perl, libhttp-tinyish-perl, libjs-jquery-center, libjs-jquery-markitup, libmenlo-legacy-perl, libmenlo-perl, libmoox-locale-passthrough-perl, libnewlib-nano, libnss-unknown, liborcus, libparse-binary-perl, librtr, libsearch-elasticsearch-client-1-0-perl, libsearch-elasticsearch-client-2-0-perl, libtie-handle-offset-perl, libzstd, lvm2, matplotlib2, med-fichier, meep, meep-lam4, meep-mpi-default, meep-mpich2, meep-openmpi, mir-core, mle, movim, netplan.io, node-lunr, node-ramda, node-react-audio-player, nodejs, oakleaf, olive, openrazer, puppet-module-heini-wait-for, puppet-module-octavia, puppet-module-voxpupuli-ssh-keygen, pylibtiff, pymilter, pyspectral, python-cytoolz, python-dpkt, python-envs, python-flask-cors, python-geotiepoints, python-glad, python-hgapi, python-ifaddr, python-internetarchive, python-markdown2, python-msgpack-numpy, python-netdisco, python-pipx, python-project-generator, python-project-generator-definitions, python-pywebview, python-sparkpost, python-sshoot, python-thinc, python-tornado4, pytroll-schedule, rcm, redberry-pipe, ruby-kitchen-salt, ruby-vcr, rust-crossbeam-channel, rust-crossbeam-utils-0.5, rust-ena, rust-hyphenation, slurp, theme-d-gnome, ticcutils, trollimage, trollsift, ulfius, vim-puppet, vland, voluptuous-serialize, vulkan-tools & xavs2.

I additionally filed 11 RC bugs against packages that had potentially-incomplete debian/copyright files against centreon-broker, dav4tbsync, eas4tbsync, emerald, i3pystatus, lvm2, olive, python-pywebview, ruby-kitchen-salt, rust-crossbeam-channel & trollsift.

Jonathan McDowell: Maxcio W-UK007S Power Monitoring Smart Plug notes

Planet Debian - Hën, 31/12/2018 - 6:57md

The house server I built in 2013 is getting on a bit, so I’d like to replace it. That’s currently held up on availability of Ryzen 7 2700E CPUs, which look to be the best power consumption/performance trade-off available at present. While I wait for stock I figured I should see how the current i3-3220T is doing.

To do so I decided to buy a Smart Plug that advertised energy monitoring, planning to integrate it into my current setup for the monitoring and then being able to use it for general power control once the upgrade comparison is complete. I ended up with a pair of Maxcio Smart Plugs - pricing and availability worked out and I’d found confirmation that the W-US002S was ESP8266 based.

The model I ended up with is externally marked as a W-UK007S. It’s a fairly neat form factor (slightly smaller than the SonOff S26 I already have, which doesn’t do power monitoring). It also turned out to be easy to take apart; there is a circular cover in the middle which can be carefully popped out, revealing the single screw holding the device together.

The back plate has 4 clips holding it together at the corners and can be gently pried off. Inside there’s a main circuit board labelled “W-US0007S-V0.3” which has the relay on it and a perpendicular board with the ESP8266 module and power monitoring chip on it.

Sadly the layout didn’t match anything I was familiar with, or could find any details of. That meant I had to do some detective work to figure out how to talk to the ESP8266. It was easy enough to work out GND + VCC by following PCB tracks. Likewise the relay, the button and the LED (underneath the button, and separately controlled from the relay, unlike the S26). Next move was to hook up power (just a low voltage supply to GND/VCC, I did not engage in any experimentation involving mains voltages!) and monitor each unknown pin in turn in the hope I’d find TX (even if the supplied firmware didn’t print anything out the ESP8266 prints a message on boot, so I’d definitely see something if it was there).

Thankfully TX was brought out to the module connection to the main PCB, so I had something I could monitor.

Maxcio boot log ets Jan 8 2013,rst cause:1, boot mode:(3,7) load 0x40100000, len 1396, room 16 tail 4 chksum 0x89 load 0x3ffe8000, len 776, room 4 tail 4 chksum 0xe8 load 0x3ffe8308, len 540, room 4 tail 8 chksum 0xc0 csum 0xc0 2nd boot version : 1.4(b1) SPI Speed : 40MHz SPI Mode : QIO SPI Flash Size & Map: 8Mbit(512KB+512KB) jump to run user1 @ 1000 OS SDK ver: 1.4.2(23fbe10) compiled @ Sep 22 2016 13:09:03 phy v[notice]user_main.c:268 SDK version:1.4.2(23fbe10) [notice]user_main.c:270 tuya sdk version:1.0.6 [notice]user_main.c:275 tuya sdk compiled at Jul 26 2017 15:27:36 [notice]user_main.c:277 BV:5.20 PV:2.1 LPV:3.1 reset reason: 0 epc1=0x00000000, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000,depc=0x00000000 mode : softAP(de:4f:22:2c:76:93) dhcp server start:(ip:192.168.4.1,mask:255.255.255.0,gw:192.168.4.1) add if1 bcn 100 bcn 0 del if1 usl mode : sta(dc:4f:22:2c:76:93) add if0 scandone [notice]device.c:1694 fireware info name:esp_hys_qx_single_light_jlplug version:1.0.0 [notice]device.c:1695 PID=gDYvLrWhRoqpgHMj [notice]device.c:1696 FW Compiled date:Feb 5 2018 [notice]gw_intf.c:240 Authorization success [notice]device.c:1722 ## AUTH DONE ## [err]bl0937.c:1013 ### get_coefficient get err: 28 [err]device.c:1767 get ele data err... [err]device.c:1772 get tem ele data err... del if0 usl mode : null force slp enable,type: 2 fpm open,type:2 0 [notice]device.c:2041 # STAT_STA_UNCONN STAT_LOW_POWER# [notice]device.c:2042 # STAT_STA_CONN # [notice]device.c:648 #####I_VAL:0##### [notice]device.c:654 ***************************Check Start 1********************************** [notice]device.c:655 ### cur:0 mA power:0 W vol:0 V [notice]device.c:656 ****************************Check Stop**********************************

Next I took the remaining unknown pins and tried grounding them on boot, in an attempt to find GPIO0 (which needs to be grounded to access the ROM serial bootloader). I ended up finding GPIO2 first, and then eventually figuring out the LED was using GPIO0 - learning the lesson not to assume pins don’t have multiple uses. Now I had TX + GPIO0 I could hold GPIO0 on boot and look for RX by probing the remaining pins and seeing if esptool could talk to the bootloader. Again, I was successful.

At that point I was able to download the firmware from flash, and poke it in the hope of working out the GPIO assignments (I’m a software guy, I’m happier with an assembly dump than probing randomly around a board in the hope of enlightenment). I generated a crude .elf from the flash dump using esp-bin2elf, hacking it up to cope better with an OTA ROM image and skip the boot loader. I initially used objdump to examine the result, which wasn’t that helpful, and then found ScratchABit, which made things much easier. What would have been ideal is some way to load in the .a static libraries from the SDK and automatically map those to the code; as well as providing some useful symbols it would have avoided work looking at functions that were irrelevant. The ESP8266 seems to want various levels of indirection to access functions and memory locations so it’s not just a simple matter of looking for an I/O request to a specific location, but I was able to figure out that the button was on GPIO13 and the relay on GPIO15.

All that left was the bit I was actually interested in - the power monitoring. The appropriate chip (clearly attached to a low resistance bridge from one of the AC power pins, and also attached to the other pin) on the PCB was marked “HJL-01 / J1749CYH / D797480E”. Whatever you find on the web this is not the same as the HLW8012. It’s very similar in operation but is actually a BL0937. Electrodragon’s Energy Meter page was the best resource I found - it has a link to the Chinese datasheet for the BL0937, which in combination with Google Translate allows the basic principles to be understood. Both devices work by having a pin (CF) which outputs pulses based on monitored power consumption, and a pin (CF1) which can be switched between monitoring current and voltage via a third pin (SEL). For the BL0937 you can just count pulses; the pulse width is fixed at 38µS and it’s just the frequency which varies. I’d found the GPIO interupt handler in my flash disassembly which indicated that GPIO5 was connected to CF and GPIO14 to CF1. Additionally the handler around GPIO14 needs to check which mode the chip is currently in, which let me discover GPIO12 was connected to this pin.

That resulted in the following pin mapping of the daughter PCB; the remaining 4 pins weren’t of interest once I had the ones I needed, so I didn’t do further investigation:

PCB EDGE +----+ LED / GPIO0 |1 12| 3.3V |2 11| BUTTON / GPIO13 |3 10| GPIO2 TX |4 9| RX RELAY / GPIO15 |5 8| GND |6 7| +----+ PCB CENTER

Of course the frequency values that come out of the BL0937 are not directly usable; there’s a certain amount of conversion/calibration that needs to be done. Thankfully although the datasheet has an equation that includes an odd constant value as well as the internal reference voltage this all boils down to a simple scaling value. I ended up using a multimeter to calibrate the voltage and then a standalone power meter + table lamp to calibrate power/current. Better results could be obtained with a constant voltage source and a known resistance load but this worked out close enough for my needs.

I wrote my own driver to fit within the ESP8266 MQTT framework I use, but if you’re looking for something more off the shelf ESPurna is probably a good start. Ticket #737 talks about adding support for the BL0937 (it’s close enough to the HLW8012 that you can get away with just changing the scaling factors), and the Homecube SP1 seems to use the same GPIOs for the various pieces of hardware.

I’ve put all the images from my teardown into a W-UK007S Teardown Album on Google Photos, in case it’s useful to anyone.

Olivier Berger: Retiring from Debian

Planet Debian - Hën, 31/12/2018 - 4:54md

I've not been able to contribute much to Debian for years, and it seems my free time and energy is mainly dedicated to family duties (and fun) these days.

I've then just announced my retirement as a Debian Developper.

I'll try to stay a contributing user and promoter, but will no longer feel guilty for not taking any duties of a DD.

Maybe I'll comme back some day. When I'm retired (from work) maybe ? ;-) In the meantime it'll be great to meet Debian friends in free software venues, whenever possible.

Dirk Eddelbuettel: 2018: Not so bad for running

Planet Debian - Hën, 31/12/2018 - 4:51md

Long-time readers of this blog (yes, looking at both of you!) may remember that I used to run (i.e. race) more regularly. But it has been rather quiet on that front lately. A sole marathon in 2016 (that was not even mentioned on this blog). One Ragnar Relay (with the same crowd, running once again as an Ultra team of six) in 2017. But in 2018? Nada. No race, no relay. Nuttin.

However, starting in mid-December 2017 I got back to exercising more. A lot on the (really old !!) Nordic track machine in the basement, a fair bit of actual running (which shows up on Strava if you follow me, hi @yanlesin and @kaneplusplus), and some fifty miles cycling (i.e. commuting recorded by GPS and Strava, more overall). All this looks quite alright when aggregated for the year:

Two run-ins with a cold stopped me in the spring and summer, as did some aches from possibly overtraining with too few rest days in the fall. Plus motivation a little sapped late summer—but overall not too bad. I have only four full years in Strava as I used something else back in the days of active marathoning but I suspect this year would have held up well enough in comparison.

We will see how 2019 shapes up. Here’s to some more running.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Dima Kogan: Keeping C OpenCV applications building

Planet Debian - Hën, 31/12/2018 - 12:44md

The OpenCV people decided to break their C API. Except (at least as of OpenCV 3.x) they didn't really remove or deprecate the API, they just stopped testing it. So when things start to break, you just get errors for no reason. Case in point: I have an application that now throws linker error when built without optimizations:

$ gcc -O0 application.c -lopencv_core ... /usr/bin/ld: /tmp/cc8R0Ote.o: in function `cvPointFrom32f': tst.c:(.text+0x557): undefined reference to `cvRound' /usr/bin/ld: tst.c:(.text+0x56d): undefined reference to `cvRound' /usr/bin/ld: /tmp/cc8R0Ote.o: in function `cvReadInt': tst.c:(.text+0xdb1): undefined reference to `cvRound' collect2: error: ld returned 1 exit status

Great. I'm not actually calling cvRound() anywhere, and looking into the sources, this breakage was easily avoidable if they actually cared.

A workaround is to define my own cvRound(). Something like this works:

#include <math.h> static inline int cvRound(float value) { return (int)roundf(value); } #include <opencv2/calib3d/calib3d_c.h>

This compiles and links with or without optimizations. Defining this before the OpenCV #include is significant, and you need to link with -lm to get the implementation of roundf().

This is with OpenCV 3.2, as shipped by Debian. I can imagine later releases either un-breaking this, or actually removing the APIs entirely. That would be the impetus for me to finally dump this library. Which would be a good thing overall, to be honest.

Russ Allbery: New year haul

Planet Debian - Hën, 31/12/2018 - 7:16pd

Accumulated one-off purchases over the past few months. Have to make sure I have plenty to read in the new year (spoiler: this will definitely not be a problem).

Lundy Bancroft — Why Does He Do That? (non-fiction)
Gavin de Becker — The Gift of Fear (non-fiction)
Cennydd Bowles — Future Ethics (non-fiction)
Julie E. Czerneda — Search Image (sff)
Ali Davis — True Porn Clerk Stories (non-fiction)
Abby Franquemont — Respect the Spindle (non-fiction)
N.K. Jemisin — How Long 'til Black Future Month? (sff collection)
Daniel Keys Moran — Tales of the Continuing Time and Other Stories (sff collection)
T. Kingfisher — Swordheart (sff)
Kate Manne — Down Girl (non-fiction)
Barry Marcus — Watches I Have Known (non-fiction)
Claire O'Dell — A Study in Honor (sff)
Michael Oher & Don Yaeger — I Beat the Odds (non-fiction)
Laurie Penny — Everything Belongs to the Future (sff)
Matt Potter — The Last Goodbye (non-fiction)
K Arsenault Rivera — The Phoenix Empress (sff)
John Scalzi — The Consuming Fire (sff)
Ryk E. Spoor — Spheres of Influence (sff)
Rebecca Traister — Good and Mad (non-fiction)
M. Mitchell Waldrop — The Dream Machine (non-fiction)
Martha Wells — Artificial Condition (sff)
Martha Wells — Rogue Protocol (sff)
Martha Wells — Exit Strategy (sff)

Some of these I've already read and reviewed. There's a lot of non-fiction in here that has shown up on my radar recently. I know I'm excessively optimistic about the number of books I pick up, but I do have some hope for 2019 being a strong reading year.

Kunal Mehta: Kiwix in Debian, 2018 update

Planet Debian - Hën, 31/12/2018 - 4:44pd

It's been a relatively productive year for the packaging of Kiwix, an offline Wikipedia reader, in Debian. My minimum expectations for the Buster release scheduled in mid-2019 are that all necessary C/C++ libraries to build the latest versions of Kiwix are in Debian.

  • libzim is at v4.0.4, and basically ready to go.
  • libkiwix is prepared for v3.0.3, but is waiting on the FTP masters to approve the new version that is in the NEW queue (fifth oldest binary-NEW package as of this post...). Once that is approved, I have the v3.1.1 upgrade prepared as well.
  • ctpp2 was a bit of a struggle, but is ready to go as well. However, it looks like the upstream website has vanished (it was inactive for years), so it's good that Kiwix is planning to replace ctpp2 in the future.

There are three main user facing packages that are in the pipeline right now:

  • zimwriterfs is waiting in the NEW queue. I'm hopeful that we can get this included in time for Debian Buster.
  • kiwix-tools is a bundle of command-line utilities, the most useful of which is kiwix-serve. There's one upstream issue relating to how JavaScript code is embedded, but the packaging prep work is basically done. It's blocked on the new libkiwix anyways.
  • kiwix-desktop or just "Kiwix" is the most interesting thing for typical users who want to read offline content, but it's probably the farthest away. I spent some time this week figuring out how to get it to compile, and mostly got it to work. Probably something to aim for the next Debian release. There's also work underway to provide it as a flatpak, which will benefit even more people.

I also created a "Kiwix team" in the Debian Package Tracker, so you can subscribe to different notifications for all of the Kiwix-related packages (though it might be spammy if you're not very interested in Debian internals).

And, thank you to the Kiwix developers for always being receptive to my bug reports and patches - and providing extra chocolate and stickers :-)

Hideki Yamane: wrap up: debootstrap in 2018

Planet Debian - Dje, 30/12/2018 - 12:17md
I've started to contribute to debootstrap in March.

Since then, triaged and squashed a half of bugs (and sometimes made some regressions ;)

Louis-Philippe Véronneau: From Cooperation to Competition or How Code Became Proprietary

Planet Debian - Sht, 29/12/2018 - 6:00pd

After two semesters of hard work, I'm happy to say I've finished all the classes I had to take for my Master's degree. If I work hard enough, I should be able to finish writing my thesis by the end of August.

Last night, I handed out the final paper for my History of the Economic Thought class. Titled "From Cooperation to Competition or How Code Became Proprietary", it is a research paper on the evolution of the Copyright Law in the United States with regards to computer code.

Here's the abstract:

With the emergence of computer science as a academic research domain in the 60s, a new generation of students gain access to computers. This generation will become the first wave of hackers, a community based on a strong ethos promoting curiosity and knowledge sharing. The hazy legal framework of computer code intellectual property at the time enables this community to grow and to put forward cooperation as a mean of economic production. Meanwhile in the 60s, the American politicians take advantage of the revision of the Copyright Act to think about the implications of applying copyright to computer code. After failing to tackle computer issues in the 1976 revision of the Copyright Act, the Senate creates the Commission on New Technological Uses of Copyrighted Works (CONTU). The final report of the CONTU will eventually lead computer code to be recognised as a literary work and thus copyrightable. A few years later, American courts finally create a strong case law on these issues with the 1983 Apple v. Franklin court case.

Sadly, I haven't had time to translate the whole paper to English yet (it's in French). Maybe if enough people are interested by theses issues I'll take the time to do so.

If you want to read my paper in French, you can download the PDF here.

Joey Hess: fridge 0.2

Planet Debian - Mar, 25/12/2018 - 2:34pd

My offgrid, solar powered, zero-battery-use fridge has sucessfully made it through spring, summer, fall, and more than enough winter.

I've proven that it works. I've not gotten food poisoning, though I did lose half a gallon of milk on one super rainy week. I have piles of data, and a whole wiki documenting how I built it. I've developed 3 thousand lines of control software. It purrs along without any assistance.

Fridge0 consists of a standard chest freezer, an added thermal mass, an inverter, and computer control. It ties into the typical offfgrid system of a solar charge controller, battery bank, and photovoltaic panels.

This isn't going to solve global warming or anything, but it does seem much less expensive than traditional offgrid fridge systems, and it ties in with thinking on renewable power such as Low Tech magazine's Redefining Energy Security "To improve energy security, we need to make infrastructures less reliable."

I mostly wanted to share the wiki, in case someone wants to build something like this. And to post some data. Here's the summer and fall temperature data.


(More on temperature ranges here.)

I want to be upfront that this is not guaranteed to work in every situation. Here's that time that the milk spoiled. A tropical storm was involved. Most of the time milk stays good 2 to 3 weeks in my fridge.

Some things I might get around to doing eventually:

  • Using a supercapacitor to provide power while shutting down on loss of solar power, instead of the current few minutes of use of batteries.
  • Also running a freezer, dividing up solar power between them.
  • A self-contained build with its own solar panels and electronics, instead of the current build that uses my house's server and solar panels.
  • A full BOM or kit, just add solar panels and chest freezer to quickly build your own.

I probably won't be devoting much time to this in the upcoming year, but if anyone wants to build one I'm happy to help you.

Kees Cook: security things in Linux v4.20

Planet Ubuntu - Mar, 25/12/2018 - 12:59pd

Previously: v4.19.

Linux kernel v4.20 has been released today! Looking through the changes, here are some security-related things I found interesting:

stackleak plugin

Alexander Popov’s work to port the grsecurity STACKLEAK plugin to the upstream kernel came to fruition. While it had received Acks from x86 (and arm64) maintainers, it has been rejected a few times by Linus. With everything matching Linus’s expectations now, it and the x86 glue have landed. (The arch-specific portions for arm64 from Laura Abbott actually landed in v4.19.) The plugin tracks function calls (with a sufficiently large stack usage) to mark the maximum depth of the stack used during a syscall. With this information, at the end of a syscall, the stack can be efficiently poisoned (i.e. instead of clearing the entire stack, only the portion that was actually used during the syscall needs to be written). There are two main benefits from the stack getting wiped after every syscall. First, there are no longer “uninitialized” values left over on the stack that an attacker might be able to use in the next syscall. Next, the lifetime of any sensitive data on the stack is reduced to only being live during the syscall itself. This is mainly interesting because any information exposures or side-channel attacks from other kernel threads need to be much more carefully timed to catch the stack data before it gets wiped.

Enabling CONFIG_GCC_PLUGIN_STACKLEAK=y means almost all uninitialized variable flaws go away, with only a very minor performance hit (it appears to be under 1% for most workloads). It’s still possible that, within a single syscall, a later buggy function call could use “uninitialized” bytes from the stack from an earlier function. Fixing this will need compiler support for pre-initialization (this is under development already for Clang, for example), but that may have larger performance implications.

raise faults for kernel addresses in copy_*_user()

Jann Horn reworked x86 memory exception handling to loudly notice when copy_{to,from}_user() tries to access unmapped kernel memory. Prior this, those accesses would result in a silent error (usually visible to callers as EFAULT), making it indistinguishable from a “regular” userspace memory exception. The purpose of this is to catch cases where, for example, the unchecked __copy_to_user() is called against a kernel address. Fuzzers like syzcaller weren’t able to notice very nasty bugs because writes to kernel addresses would either corrupt memory (which may or may not get detected at a later time) or return an EFAULT that looked like things were operating normally. With this change, it’s now possible to much more easily notice missing access_ok() checks. This has already caught two other corner cases even during v4.20 in HID and Xen.

spectre v2 userspace mitigation

The support for Single Thread Indirect Branch Predictors (STIBP) has been merged. This allowed CPUs that support STIBP to effectively disable Hyper-Threading to avoid indirect branch prediction side-channels to expose information between userspace threads on the same physical CPU. Since this was a very expensive solution, this protection was made opt-in (via explicit prctl() or implicitly under seccomp()). LWN has a nice write-up of the details.

jump labels read-only after init

Ard Biesheuvel noticed that jump labels don’t need to be writable after initialization, so their data structures were made read-only. Since they point to kernel code, they might be used by attackers to manipulate the jump targets as a way to change kernel code that wasn’t intended to be changed. Better to just move everything into the read-only memory region to remove it from the possible kernel targets for attackers.

VLA removal finished

As detailed earlier for v4.17, v4.18, and v4.19, a whole bunch of people answered my call to remove Variable Length Arrays (VLAs) from the kernel. I count at least 153 commits having been added to the kernel since v4.16 to remove VLAs, with a big thanks to Gustavo A. R. Silva, Laura Abbott, Salvatore Mesoraca, Kyle Spiers, Tobin C. Harding, Stephen Kitt, Geert Uytterhoeven, Arnd Bergmann, Takashi Iwai, Suraj Jitindar Singh, Tycho Andersen, Thomas Gleixner, Stefan Wahren, Prashant Bhole, Nikolay Borisov, Nicolas Pitre, Martin Schwidefsky, Martin KaFai Lau, Lorenzo Bianconi, Himanshu Jha, Chris Wilson, Christian Lamparter, Boris Brezillon, Ard Biesheuvel, and Antoine Tenart. With all that done, “-Wvla” has been added to the top-level Makefile so we don’t get any more added back in the future.

Given the holidays, Linus opened the merge window before v4.20 was released, letting everyone send in pull requests in the week leading up to the release. v4.21 is in the making. :) Happy New Year everyone!

Edit: clarified stackleak details, thanks to Alexander Popov.

© 2018, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

Kees Cook: security things in Linux v4.20

Planet Debian - Mar, 25/12/2018 - 12:59pd

Previously: v4.19.

Linux kernel v4.20 has been released today! Looking through the changes, here are some security-related things I found interesting:

stackleak plugin

Alexander Popov’s work to port the grsecurity STACKLEAK plugin to the upstream kernel came to fruition. While it had received Acks from x86 (and arm64) maintainers, it has been rejected a few times by Linus. With everything matching Linus’s expectations now, it and the x86 glue have landed. (The arch-specific portions for arm64 from Laura Abbott actually landed in v4.19.) The plugin tracks function calls (with a sufficiently large stack usage) to mark the maximum depth of the stack used during a syscall. With this information, at the end of a syscall, the stack can be efficiently poisoned (i.e. instead of clearing the entire stack, only the portion that was actually used during the syscall needs to be written). There are two main benefits from the stack getting wiped after every syscall. First, there are no longer “uninitialized” values left over on the stack that an attacker might be able to use in the next syscall. Next, the lifetime of any sensitive data on the stack is reduced to only being live during the syscall itself. This is mainly interesting because any information exposures or side-channel attacks from other kernel threads need to be much more carefully timed to catch the stack data before it gets wiped.

Enabling CONFIG_GCC_PLUGIN_STACKLEAK=y means almost all uninitialized variable flaws go away, with only a very minor performance hit (it appears to be under 1% for most workloads). It’s still possible that, within a single syscall, a later buggy function call could use “uninitialized” bytes from the stack from an earlier function. Fixing this will need compiler support for pre-initialization (this is under development already for Clang, for example), but that may have larger performance implications.

raise faults for kernel addresses in copy_*_user()

Jann Horn reworked x86 memory exception handling to loudly notice when copy_{to,from}_user() tries to access unmapped kernel memory. Prior this, those accesses would result in a silent error (usually visible to callers as EFAULT), making it indistinguishable from a “regular” userspace memory exception. The purpose of this is to catch cases where, for example, the unchecked __copy_to_user() is called against a kernel address. Fuzzers like syzcaller weren’t able to notice very nasty bugs because writes to kernel addresses would either corrupt memory (which may or may not get detected at a later time) or return an EFAULT that looked like things were operating normally. With this change, it’s now possible to much more easily notice missing access_ok() checks. This has already caught two other corner cases even during v4.20 in HID and Xen.

spectre v2 userspace mitigation

The support for Single Thread Indirect Branch Predictors (STIBP) has been merged. This allowed CPUs that support STIBP to effectively disable Hyper-Threading to avoid indirect branch prediction side-channels to expose information between userspace threads on the same physical CPU. Since this was a very expensive solution, this protection was made opt-in (via explicit prctl() or implicitly under seccomp()). LWN has a nice write-up of the details.

jump labels read-only after init

Ard Biesheuvel noticed that jump labels don’t need to be writable after initialization, so their data structures were made read-only. Since they point to kernel code, they might be used by attackers to manipulate the jump targets as a way to change kernel code that wasn’t intended to be changed. Better to just move everything into the read-only memory region to remove it from the possible kernel targets for attackers.

VLA removal finished

As detailed earlier for v4.17, v4.18, and v4.19, a whole bunch of people answered my call to remove Variable Length Arrays (VLAs) from the kernel. I count at least 153 commits having been added to the kernel since v4.16 to remove VLAs, with a big thanks to Gustavo A. R. Silva, Laura Abbott, Salvatore Mesoraca, Kyle Spiers, Tobin C. Harding, Stephen Kitt, Geert Uytterhoeven, Arnd Bergmann, Takashi Iwai, Suraj Jitindar Singh, Tycho Andersen, Thomas Gleixner, Stefan Wahren, Prashant Bhole, Nikolay Borisov, Nicolas Pitre, Martin Schwidefsky, Martin KaFai Lau, Lorenzo Bianconi, Himanshu Jha, Chris Wilson, Christian Lamparter, Boris Brezillon, Ard Biesheuvel, and Antoine Tenart. With all that done, “-Wvla” has been added to the top-level Makefile so we don’t get any more added back in the future.

Given the holidays, Linus opened the merge window before v4.20 was released, letting everyone send in pull requests in the week leading up to the release. v4.21 is in the making. :) Happy New Year everyone!

Edit: clarified stackleak details, thanks to Alexander Popov.

© 2018, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

Julian Andres Klode: An Introduction to Go

Planet Ubuntu - Hën, 24/12/2018 - 3:15md

(What follows is an excerpt from my master’s thesis, almost all of section 2.1, quickly introducing Go to people familiar with CS)

Go is an imperative programming language for concurrent programming created at and mainly developed by Google, initially mostly by Robert Griesemer, Rob Pike, and Ken Thompson. Design of the language started in 2007, and an initial version was released in 2009; with the first stable version, 1.0 released in 2012 1.

Go has a C-like syntax (without a preprocessor), garbage collection, and, like its predecessors devloped at Bell Labs – Newsqueak (Rob Pike), Alef (Phil Winterbottom), and Inferno (Pike, Ritchie, et al.) – provides built-in support for concurrency using so-called goroutines and channels, a form of co-routines, based on the idea of Hoare’s ‘Communicating Sequential Processes’ 2.

Go programs are organised in packages. A package is essentially a directory containing Go files. All files in a package share the same namespace, and there are two visibilities for symbols in a package: Symbols starting with an upper case character are visible to other packages, others are private to the package:

func PublicFunction() { fmt.Println("Hello world") } func privateFunction() { fmt.Println("Hello package") } Types

Go has a fairly simple type system: There is no subtyping (but there are conversions), no generics, no polymorphic functions, and there are only a few basic categories of types:

  1. base types: int, int64, int8, uint, float32, float64, etc.
  2. struct
  3. interface - a set of methods
  4. map[K, V] - a map from a key type to a value type
  5. [number]Type - an array of some element type
  6. []Type - a slice (pointer to array with length and capability) of some type
  7. chan Type - a thread-safe queue
  8. pointer *T to some other type
  9. functions
  10. named type - aliases for other types that may have associated methods:

    type T struct { foo int } type T *T type T OtherNamedType

    Named types are mostly distinct from their underlying types, so you cannot assign them to each other, but some operators like + do work on objects of named types with an underlying numerical type (so you could add two T in the example above).

Maps, slices, and channels are reference-like types - they essentially are structs containing pointers. Other types are passed by value (copied), including arrays (which have a fixed length and are copied).

Conversions

Conversions are the similar to casts in C and other languages. They are written like this:

TypeName(value) Constants

Go has “untyped” literals and constants.

1 // untyped integer literal const foo = 1 // untyped integer constant const foo int = 1 // int constant

Untyped values are classified into the following categories: UntypedBool, UntypedInt, UntypedRune, UntypedFloat, UntypedComplex, UntypedString, and UntypedNil (Go calls them basic kinds, other basic kinds are available for the concrete types like uint8). An untyped value can be assigned to a named type derived from a base type; for example:

type someType int const untyped = 2 // UntypedInt const bar someType = untyped // OK: untyped can be assigned to someType const typed int = 2 // int const bar2 someType = typed // error: int cannot be assigned to someType Interfaces and ‘objects’

As mentioned before, interfaces are a set of methods. Go is not an object-oriented language per se, but it has some support for associating methods with named types: When declaring a function, a receiver can be provided - a receiver is an additional function argument that is passed before the function and involved in the function lookup, like this:

type SomeType struct { ... } func (s *SomeType) MyMethod() { } func main() { var s SomeType s.MyMethod() }

An object implements an interface if it implements all methods; for example, the following interface MyMethoder is implemented by *SomeType (note the pointer), and values of *SomeType can thus be used as values of MyMethoder. The most basic interface is interface{}, that is an interface with an empty method set - any object satisfies that interface.

type MyMethoder interface { MyMethod() }

There are some restrictions on valid receiver types; for example, while a named type could be a pointer (for example, type MyIntPointer *int), such a type is not a valid receiver type.

Control flow

Go provides three primary statements for control flow: if, switch, and for. The statements are fairly similar to their equivalent in other C-like languages, with some exceptions:

  • There are no parentheses around conditions, so it is if a == b {}, not if (a == b) {}. The braces are mandatory.
  • All of them can have initialisers, like this

    if result, err := someFunction(); err == nil { // use result }

  • The switch statement can use arbitrary expressions in cases

  • The switch statement can switch over nothing (equals switching over true)

  • Cases do not fall through by default (no break needed), use fallthrough at the end of a block to fall through.

  • The for loop can loop over ranges: for key, val := range map { do something }

Goroutines

The keyword go spawns a new goroutine, a concurrently executed function. It can be used with any function call, even a function literal:

func main() { ... go func() { ... }() go some_function(some_argument) } Channels

Goroutines are often combined with channels to provide an extended form of Communicating Sequential Processes 2. A channel is a concurrent-safe queue, and can be buffered or unbuffered:

var unbuffered = make(chan int) // sending blocks until value has been read var buffered = make(chan int, 5) // may have up to 5 unread values queued

The <- operator is used to communicate with a single channel.

valueReadFromChannel := <- channel otherChannel <- valueToSend

The select statement allows communication with multiple channels:

select { case incoming := <- inboundChannel: // A new message for me case outgoingChannel <- outgoing: // Could send a message, yay! } The defer statement

Go provides a defer statement that allows a function call to be scheduled for execution when the function exits. It can be used for resource clean-up, for example:

func myFunc(someFile io.ReadCloser) { defer someFile.close() /* Do stuff with file */ }

It is of course possible to use function literals as the function to call, and any variables can be used as usual when writing the call.

Error handling

Go does not provide exceptions or structured error handling. Instead, it handles errors by returning them in a second or later return value:

func Read(p []byte) (n int, err error) // Built-in type: type error interface { Error() string }

Errors have to be checked in the code, or can be assigned to _:

n0, _ := Read(Buffer) // ignore error n, err := Read(buffer) if err != nil { return err }

There are two functions to quickly unwind and recover the call stack, though: panic() and recover(). When panic() is called, the call stack is unwound, and any deferred functions are run as usual. When a deferred function invokes recover(), the unwinding stops, and the value given to panic() is returned. If we are unwinding normally and not due to a panic, recover() simply returns nil. In the example below, a function is deferred and any error value that is given to panic() will be recovered and stored in an error return value. Libraries sometimes use that approach to make highly recursive code like parsers more readable, while still maintaining the usual error return value for public functions.

func Function() (err error) { defer func() { s := recover() switch s := s.(type) { // type switch case error: err = s // s has type error now default: panic(s) } } } Arrays and slices

As mentioned before, an array is a value type and a slice is a pointer into an array, created either by slicing an existing array or by using make() to create a slice, which will create an anonymous array to hold the elements.

slice1 := make([]int, 2, 5) // 5 elements allocated, 2 initialized to 0 slice2 := array[:] // sliced entire array slice3 := array[1:] // slice of array without first element

There are some more possible combinations for the slicing operator than mentioned above, but this should give a good first impression.

A slice can be used as a dynamically growing array, using the append() function.

slice = append(slice, value1, value2) slice = append(slice, arrayOrSlice...)

Slices are also used internally to represent variable parameters in variable length functions.

Maps

Maps are simple key-value stores and support indexing and assigning. They are not thread-safe.

someValue := someMap[someKey] someValue, ok := someMap[someKey] // ok is false if key not in someMap someMap[someKey] = someValue
  1. Frequently Asked Questions (FAQ) - The Go Programming Language https://golang.org/doc/faq#history [return]
  2. HOARE, Charles Antony Richard. Communicating sequential processes. Communications of the ACM, 1978, 21. Jg., Nr. 8, S. 666-677. [return]

Julian Andres Klode: An Introduction to Go

Planet Debian - Hën, 24/12/2018 - 3:15md

(What follows is an excerpt from my master’s thesis, almost all of section 2.1, quickly introducing Go to people familiar with CS)

Go is an imperative programming language for concurrent programming created at and mainly developed by Google, initially mostly by Robert Griesemer, Rob Pike, and Ken Thompson. Design of the language started in 2007, and an initial version was released in 2009; with the first stable version, 1.0 released in 2012 1.

Go has a C-like syntax (without a preprocessor), garbage collection, and, like its predecessors devloped at Bell Labs – Newsqueak (Rob Pike), Alef (Phil Winterbottom), and Inferno (Pike, Ritchie, et al.) – provides built-in support for concurrency using so-called goroutines and channels, a form of co-routines, based on the idea of Hoare’s ‘Communicating Sequential Processes’ 2.

Go programs are organised in packages. A package is essentially a directory containing Go files. All files in a package share the same namespace, and there are two visibilities for symbols in a package: Symbols starting with an upper case character are visible to other packages, others are private to the package:

func PublicFunction() { fmt.Println("Hello world") } func privateFunction() { fmt.Println("Hello package") } Types

Go has a fairly simple type system: There is no subtyping (but there are conversions), no generics, no polymorphic functions, and there are only a few basic categories of types:

  1. base types: int, int64, int8, uint, float32, float64, etc.
  2. struct
  3. interface - a set of methods
  4. map[K, V] - a map from a key type to a value type
  5. [number]Type - an array of some element type
  6. []Type - a slice (pointer to array with length and capability) of some type
  7. chan Type - a thread-safe queue
  8. pointer *T to some other type
  9. functions
  10. named type - aliases for other types that may have associated methods:

    type T struct { foo int } type T *T type T OtherNamedType

    Named types are mostly distinct from their underlying types, so you cannot assign them to each other, but some operators like + do work on objects of named types with an underlying numerical type (so you could add two T in the example above).

Maps, slices, and channels are reference-like types - they essentially are structs containing pointers. Other types are passed by value (copied), including arrays (which have a fixed length and are copied).

Conversions

Conversions are the similar to casts in C and other languages. They are written like this:

TypeName(value) Constants

Go has “untyped” literals and constants.

1 // untyped integer literal const foo = 1 // untyped integer constant const foo int = 1 // int constant

Untyped values are classified into the following categories: UntypedBool, UntypedInt, UntypedRune, UntypedFloat, UntypedComplex, UntypedString, and UntypedNil (Go calls them basic kinds, other basic kinds are available for the concrete types like uint8). An untyped value can be assigned to a named type derived from a base type; for example:

type someType int const untyped = 2 // UntypedInt const bar someType = untyped // OK: untyped can be assigned to someType const typed int = 2 // int const bar2 someType = typed // error: int cannot be assigned to someType Interfaces and ‘objects’

As mentioned before, interfaces are a set of methods. Go is not an object-oriented language per se, but it has some support for associating methods with named types: When declaring a function, a receiver can be provided - a receiver is an additional function argument that is passed before the function and involved in the function lookup, like this:

type SomeType struct { ... } func (s *SomeType) MyMethod() { } func main() { var s SomeType s.MyMethod() }

An object implements an interface if it implements all methods; for example, the following interface MyMethoder is implemented by *SomeType (note the pointer), and values of *SomeType can thus be used as values of MyMethoder. The most basic interface is interface{}, that is an interface with an empty method set - any object satisfies that interface.

type MyMethoder interface { MyMethod() }

There are some restrictions on valid receiver types; for example, while a named type could be a pointer (for example, type MyIntPointer *int), such a type is not a valid receiver type.

Control flow

Go provides three primary statements for control flow: if, switch, and for. The statements are fairly similar to their equivalent in other C-like languages, with some exceptions:

  • There are no parentheses around conditions, so it is if a == b {}, not if (a == b) {}. The braces are mandatory.
  • All of them can have initialisers, like this

    if result, err := someFunction(); err == nil { // use result }

  • The switch statement can use arbitrary expressions in cases

  • The switch statement can switch over nothing (equals switching over true)

  • Cases do not fall through by default (no break needed), use fallthrough at the end of a block to fall through.

  • The for loop can loop over ranges: for key, val := range map { do something }

Goroutines

The keyword go spawns a new goroutine, a concurrently executed function. It can be used with any function call, even a function literal:

func main() { ... go func() { ... }() go some_function(some_argument) } Channels

Goroutines are often combined with channels to provide an extended form of Communicating Sequential Processes 2. A channel is a concurrent-safe queue, and can be buffered or unbuffered:

var unbuffered = make(chan int) // sending blocks until value has been read var buffered = make(chan int, 5) // may have up to 5 unread values queued

The <- operator is used to communicate with a single channel.

valueReadFromChannel := <- channel otherChannel <- valueToSend

The select statement allows communication with multiple channels:

select { case incoming := <- inboundChannel: // A new message for me case outgoingChannel <- outgoing: // Could send a message, yay! } The defer statement

Go provides a defer statement that allows a function call to be scheduled for execution when the function exits. It can be used for resource clean-up, for example:

func myFunc(someFile io.ReadCloser) { defer someFile.close() /* Do stuff with file */ }

It is of course possible to use function literals as the function to call, and any variables can be used as usual when writing the call.

Error handling

Go does not provide exceptions or structured error handling. Instead, it handles errors by returning them in a second or later return value:

func Read(p []byte) (n int, err error) // Built-in type: type error interface { Error() string }

Errors have to be checked in the code, or can be assigned to _:

n0, _ := Read(Buffer) // ignore error n, err := Read(buffer) if err != nil { return err }

There are two functions to quickly unwind and recover the call stack, though: panic() and recover(). When panic() is called, the call stack is unwound, and any deferred functions are run as usual. When a deferred function invokes recover(), the unwinding stops, and the value given to panic() is returned. If we are unwinding normally and not due to a panic, recover() simply returns nil. In the example below, a function is deferred and any error value that is given to panic() will be recovered and stored in an error return value. Libraries sometimes use that approach to make highly recursive code like parsers more readable, while still maintaining the usual error return value for public functions.

func Function() (err error) { defer func() { s := recover() switch s := s.(type) { // type switch case error: err = s // s has type error now default: panic(s) } } } Arrays and slices

As mentioned before, an array is a value type and a slice is a pointer into an array, created either by slicing an existing array or by using make() to create a slice, which will create an anonymous array to hold the elements.

slice1 := make([]int, 2, 5) // 5 elements allocated, 2 initialized to 0 slice2 := array[:] // sliced entire array slice3 := array[1:] // slice of array without first element

There are some more possible combinations for the slicing operator than mentioned above, but this should give a good first impression.

A slice can be used as a dynamically growing array, using the append() function.

slice = append(slice, value1, value2) slice = append(slice, arrayOrSlice...)

Slices are also used internally to represent variable parameters in variable length functions.

Maps

Maps are simple key-value stores and support indexing and assigning. They are not thread-safe.

someValue := someMap[someKey] someValue, ok := someMap[someKey] // ok is false if key not in someMap someMap[someKey] = someValue
  1. Frequently Asked Questions (FAQ) - The Go Programming Language https://golang.org/doc/faq#history [return]
  2. HOARE, Charles Antony Richard. Communicating sequential processes. Communications of the ACM, 1978, 21. Jg., Nr. 8, S. 666-677. [return]

Faqet

Subscribe to AlbLinux agreguesi