Debian

An updated version of the corels package is now on CRAN! The ‘Certifiably Optimal RulE ListS (Corels)’ learner provides interpretable decision rules with an optimality guarantee. The change is (just like the previous one) chiefly an update to ... [More] configure.ac in order to ensure R on M1 macOS finds the locally-added GNU GMP. Our thanks to the infatiguable Brian Ripley for the heads-up even containing the two needed assignments to LD and CPPFLAGS.. If you like this or other open-source work I do, you can now sponsor me at GitHub. This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings. [Less]

I first started using ledger — the original plain-text accounting software — in 2017. Since then, I had been pretty happy with my accounting routine, but grew a little annoyed by the repetitive manual work I had to do to assign recurring ... [More] transactions to the right account. To make things easier, I had a collection of bash scripts to parse and convert the CSV files from my bank's website1 into ledger entries. They were of course ugly, unreadable piles of sed-grep-regex and did not let met achieve the automation complexity I longed for. Dissatisfied with ledger's features, I decided to migrate to hledger. Contrary to ledger, hledger comes with a CSV parser one can use to import and classify recurring transactions automagically. Having a proper DSL for this makes all the difference: I can easily add new rules and understand the old ones. In the end, I get a very consistent result, something my old bash scripts weren't great at. Here is what my debit.csv.rules file looks like. It is used to tell hledger how CSV data from my debit card should be parsed: # skip the headings line: skip 1 # assign fields fields , , , date, , description, , amount2-in, amount2-out # assign account where the money comes from # 99 to get it at the bottom of the transaction account99 assets:checking include matchers.rules As you can see, the rules matching transactions to specific accounts are imported from another file, matchers.rules. I'm doing this since I want to be able to use the same set of rules for debit and credit and use the cards interchangeably. Here's a snippet of my matchers.rules file: # house if Loyer/bail account2 expenses:rent # leisure if PAYPAL .*STEAM GAMES account2 expenses:leisure:videogame if PAYPAL .*BANDCAMP account2 expenses:leisure:music Using this ruleset, a transaction looking like this: "SOME ACCOUNT DESC","111111","EOP","2022/01/03",00002,"Loyer/bail","",521.00,"","","","","" Would be transformed into: 2022-01-03 Loyer/bail expenses:rent 521.00 assets:checking Sadly, hledger's CSV rules won't let you do arithmetics. This can be useful when you know a certain transaction needs to be split between accounts. This is where auto postings come in. They are a way to specify arbitrary rules when an account is encountered. Going back to my previous rent example, I split it 50/50 with my SO using this rule: = expenses:rent assets:receivable:rent *0.5 assets:checking After it is applied, the final transaction looks like this: 2022-01-03 Loyer/bail ; modified: expenses:rent 521.00 assets:receivable:rent 260.50 ; generated-posting: = expenses:rent assets:checking Neat eh? Here is the little bash script I've written to automate all these steps: #!/bin/bash # # Convert CSV to ledger using hledger declare -a assets=("credit" "debit") # Read the array values with space for i in "${assets[@]}" do if test -f "$i.csv" then # convert to ledger LEDGER=$(hledger -f "$i.csv" --rules-file rules/"$i.csv".rules print) # add auto_postings LEDGER=$(printf "include rules/auto_postings.ledger\n\n$LEDGER\n" | hledger print -f- --auto) # remove superfluous assets:checking lines LEDGER=$(printf "$LEDGER" | sed '/assets:checking \+; generated.\+/d') printf "$LEDGER" > "$i.ledger" else printf "File $i.csv does not exist\n" fi done Migrating to hledger, I've cut down the time I spend on accounting from 1 hour per month to about 10 minutes, all while making my workflow much cleaner. Many thanks to the kind folks on #hledger @irc.libera.chat for the help! One for my checking+savings account and another one for my credit card. ↩ [Less]

The EU Digital Covid Certificate scheme is a format for (digitally signed) vaccination status certificates. Not only EU countries participate - the UK is now a participant in this scheme. I am currently on my way to go skiing in the French Alps. So ... [More] I needed a certificate that would be accepted in France. AFAICT the official way to do this is to get the “international” certificate from the NHS, and take it to a French pharmacy who will convert it into something suitably French. (AIUI the NHS “international” barcode is the same regardless of whether you get it via the NHS website, the NHS app, or a paper letter. NB that there is one barcode per vaccine dose so you have to get the right one - probably that means your booster since there’s a 9 month rule!) I read on an forum somewhere that you could use the French TousAntiCovid app to convert the barcode. So I thought I would try that. The TousAntiCovid is Free Softare and on F-Droid, so I was happy to install and use it for this. I also used the French TAC Verify app to check to see what barcodes were accepted. (I found an official document addressed to French professionals recommending this as an option for verifying the status of visitors to one’s establishment.) Unfortunately this involves a googlified phone, but one could use a burner phone or ask a friend who’s bitten that bullet already. I discovered that, indeed: My own NHS QR code is not accepted by TAC Verify My own NHS QR code can be loaded into TousAntiCovid, and added to my “wallet” in the app If I get TousAntiCovid to display that certificate, it shows a visually different QR code which TAC Verify accepts This made me curious. I used a QR code reader to decode both barcodes. The decodings were identical! A long string of guff starting HC1:. AIUI it is an encoded JWT. But there was a difference in the framing: Binary Eye reported that the NHS barcode used error correction level “M” (medium, aka 15%). The TousAntiCovid barcode used level “L” (low, 7%). I had my QR code software regenerate a QR code at level “M” for the data from the TousAntiCovid code. The result was a QR code which is identical (pixel-wise) to the one from the NHS. So the only difference is the error correction level. Curiously, both “L” (low, generated by TousAntiCovid, accepted by TAC Verify) and “M” (medium, generated by NHS, rejected by TAC Verify) are lower than the “Q” (25") recommended by what I think is the specification. This is all very odd. But the upshot is that I think you can convert the NHS “international” barcode into something that should work in France simply by passing it through any QR code software to re-encode it at error correction level “L” (7%). But if you’re happy to use the TousAntiCovid app it’s probably a good way to store them. I guess I’ll find out when I get to France if the converted NHS barcodes work in real establishments. Thanks to the folks behind sanipasse.fr for publishing some helpful backround info and operating a Free Software backed public verification service. Footnote To compare the QR codes pixelwise, I roughly cropped the NHS PDF image using a GUI tool, and then on each of the two images used pnmcrop (to trim the border), pnmscale (to rescale the one-pixel-per-pixel output from Binary Eye) and pnmarith -difference to compare them (producing a pretty squirgly image showing just the pixel edges due to antialiasing). comments [Less]

The diffoscope maintainers are pleased to announce the release of diffoscope version 203. This version includes the following changes: [ Chris Lamb ] * Improve documentation for --timeout due to a few misconceptions. Add an allowed-to-fail test ... [More] regarding a regression in directory handling. * Tidy control flow in Difference._reverse_self a little. [ Alyssa Ross ] * Fix diffing CBFS names that contain spaces. You find out more by visiting the project homepage. [Less]

tl;dr I ported a part of the python-suntime library to Lua to use it on OpenWRT and RutOS powered devices. suntime.lua There are those unremarkable things which let you pause for a moment, and realize what a great gift of our time open source ... [More] software and open knowledge is. At some point in time someone figured out how to calculate the sunrise and sunset time on the current date for your location. Someone else wrote that up and probably again a different person published it on the internet. The Internet Archive preserved a copy of it so I can still link to it. Someone took this algorithm and published a code sample on StackOverflow, which was later on used by the SatAgro guys to create the python-suntime library. Now I could come along, copy the core function of this library, convert it within a few hours - mostly spent learning a bit of Lua, to a working script fulfilling my needs. [Less]

Mechanical keyboards – the big fat rabbit hole you can disappear … I started using mechanical keyboards about a year ago, with a Drevo Blademaster Pro (review coming up), but recently got a Pulsar PCML TKL keyboard in a build-it-yourself order. ... [More] The Drevo Blademaster Pro I am using is great, but doesn’t allow changing switches at all. So I was contemplating getting a mechanical keyboard that allows for arbitrary switches. My biggest problem here is that I am used to the Japanese JIS layout – which gives you a lot more keys which come in extremely handy in Emacs or when typing various languages. Fortunately, APlusX – a Korean company – manufactures a lot of gear under the Pulsar name, and supports also JIS layout. In addition, they have a great web site to customize your keyboard (layout, color, switch, keycaps) and send you a build-yourself kit for a very reasonable prize – at least in Japan. So I got my first keyboard to put together myself …. how was I nervous getting all the stuff out! Despite being a DIY keyboard, it is very easy (and they offer also pre-assembly options, too!). You don’t need to solder the PCB or similar, the steps are more or less: (i) put in the switches, and (ii) add the key caps. I started with the first, putting the switches (I went with Kailh Box Brown tactile ones) into the PCB board … Well, that was easy … at least I thought until I started testing the keys and realized that about 20 of them didn’t work!! Pulling out the switches again I saw that I twisted a pin on each of them. One by one I straightened the pins and reinserted them very carefully. Lesson learned! At the end all the switches were on the board and reacted to key presses. Next step was adding the key caps. Again, those are not really special key caps, but simply style and sufficient for me. Of course I messed up 0 and O (which have different heights) and at first were confused about different arrow options etc, but since plugging in and pulling out key caps is very easy, at the end all the caps were in place. With the final keyboard assembled (see top photo), I connected it to my Linux system and started typing around. And first of all, the typing experience was nice. The Kailh Box Brown switches have a bit stronger actuation point then the switches I have in the Blademaster Pro (which are Cherry MX Brown ones), but above all the sound is a bit deeper and “thumbier”, which really gives a nice feeling. The keyboard also allows changing the RGB lightening via the keyboard (color, pattern, speed, brightness etc). There is a configuration software for macros etc, unfortunately it only works on Windows (and I couldn’t get it to work with Wine, either), a sour point … One more negative point is that the LED backlight doesn’t have a timeout, that is, it stays on all the time. The Drevo I have turns off after a configured number of seconds, and turns completely black – something I really like and miss on the Pulsar. Another difference to the Blademaster is connectivity: While the Blademaster offers cable, bluetooth, and wireless (with dongle), the Pulsar only offers cable (USB-C). Not a real deal-breaker for me, since I use it at my desktop and really don’t need wireless/bluetooth capabilities, but still. I have been using the Pulsar now for a few days without even touching the Drevo (besides comparing typing sounds and actuation points), and really like the Pulsar. I think it is hard to get a fully configurable and changeable mechanical keyboard for a similar prize. There is one last thing that I really really miss – an ergonomic mechanical keyboard. Of course there are some, like the ErgoDox EZ or the Kinesis Advantage 2, but they don’t offer JIS layout (and are very expensive). Then there is the Truly Ergonomic CLEAVE keyboard, which is really great, but quite expensive. I guess I have to dive down the rabbit hole even more and make my own PCB and ergonomic keyboard! [Less]

Starting just after Christmas, I have been working on CI for all the kernel team's packages on Salsa. The salsa-ci-team has done great work on producing a common pipeline that is usable for most packages with minimal configuration. ... [More] However, for some packages a lot more work was required. Linux I started with the most important package, linux itself. This now has about 1.1 GiB of source spread over 76,000 source files. That turns out to be a problem for the pipeline which currently puts unpacked source in artifacts - it is far beyond the limits of what Salsa allows. I worked around this by using a modified version of the extract-source and build jobs that use packed source package as the artifacts. The output of the build job is compatible with the common test jobs. The linux package also takes a lot of resources to build; around 80 minutes on the fastest PC I have at home (if ccache is not primed). Salsa's shared CI runners seem to be about 10 times slower than that, so it is completely unfeasible to even one full build in CI. Instead I defined a new build profile that includes only the smallest kernel configuration, without debug info, and the user-space packages. This still takes over an hour with the Salsa CI runners, but I don't think we can improve this much without losing a lot of code coverage. Our Git repository for linux also does not contain the upstream source, so the extract-source job has to fetch that. The common extract-source job uses origtargz to do that, and in case the orig tarball is not already in the archive this will run uscan. That led me to a new problem: our debian/watch file could only find tarballs linked from the front of www.kernel.org, and we're sometimes working with different upstream versions. There is actually no single page listing all tarball releases of Linux, and tarballs for release candidates are dynamically generated by CGit and unsigned. So I changed debian/watch to fetch from Git, which is what we were already doing with our own genorig.py script. Unfortunately, running uscan against a Git upstream, with some files excluded (as there are still a few upstream files we consider non-free), is about twice as slow as it could be. Since I had to modify the extract-source job anyway, I've continued using genorig.py there. A full build log for linux is over 200 MiB, and even with the reduced build profile it would be much longer than Salsa's limit of 2 4 MiB. I therefore opted to use the 'terse' build option (which translates to V=0), but made the builds of user-space tools ignore this option so that blhc could still do its work. (The kernel itself cannot use the same hardening options, so blhc is not useful there.) Finally, with the CI pipeline running, blhc and lintian showed a lot of problems that we hadn't been attending to. I've fixed all the blhc errors (with some careful suppressions), all the lintian errors, and the most straightforward lintian warnings. firmware-nonfree The firmware-nonfree package also has huge "source" (about 560 MB) and needed some of the same modifications, but is quick to build so did not require a special build profile. Running lintian over firmware-nonfree reminded me that I needed to sort out the unsuual and inconistent handling of machine-readable copyright information in this source package. I had already done most of that work on a private branch in 2020, so this is mostly ready but I still need to resolve a licensing issue with AppStream metadata. Other packages For kernel-handbook, there was already a trivial "CI" pipeline used to push static pages to the web site. I've replaced this with the common pipeline plus a job that will push the pages from each build on the master branch. For everything else, it was straightforward to enable the common pipeline with a little bit of configuration. [Less]

Starting just after Christmas, I have been working on CI for all the kernel team's packages on Salsa. The salsa-ci-team has done great work on producing a common pipeline that is usable for most packages with minimal configuration. ... [More] However, for some packages a lot more work was required. Linux I started with the most important package, linux itself. This now has about 1.1 GiB of source spread over 76,000 source files. That turns out to be a problem for the pipeline which currently puts unpacked source in artifacts - it is far beyond the limits of what Salsa allows. I worked around this by using a modified version of the extract-source and build jobs that use packed source package as the artifacts. The output of the build job is compatible with the common test jobs. The linux package also takes a lot of resources to build; around 80 minutes on the fastest PC I have at home (if ccache is not primed). Salsa's shared CI runners seem to be about 10 times slower than that, so it is completely unfeasible to even one full build in CI. Instead I defined a new build profile that includes only the smallest kernel configuration, without debug info, and the user-space packages. This still takes over an hour with the Salsa CI runners, but I don't think we can improve this much without losing a lot of code coverage. Our Git repository for linux also does not contain the upstream source, so the extract-source job has to fetch that. The common extract-source job uses origtargz to do that, and in case the orig tarball is not already in the archive this will run uscan. That led me to a new problem: our debian/watch file could only find tarballs linked from the front of www.kernel.org, and we're sometimes working with different upstream versions. There is actually no single page listing all tarball releases of Linux, and tarballs for release candidates are dynamically generated by CGit and unsigned. So I changed debian/watch to fetch from Git, which is what we were already doing with our own genorig.py script. Unfortunately, running uscan against a Git upstream, with some files excluded (as there are still a few upstream files we consider non-free), is about twice as slow as it could be. Since I had to modify the extract-source job anyway, I've continued using genorig.py there. A full build log for linux is over 200 MiB, and even with the reduced build profile it would be much longer than Salsa's limit of 2 MiB. I therefore opted to use the 'terse' build option (which translates to V=0), but made the builds of user-space tools ignore this option so that blhc could still do its work. (The kernel itself cannot use the same hardening options, so blhc is not useful there.) Finally, with the CI pipeline running, blhc and lintian showed a lot of problems that we hadn't been attending to. I've fixed all the blhc errors (with some careful suppressions), all the lintian errors, and the most straightforward lintian warnings. firmware-nonfree The firmware-nonfree package also has huge "source" (about 560 MB) and needed some of the same modifications, but is quick to build so did not require a special build profile. Running lintian over firmware-nonfree reminded me that I needed to sort out the unsuual and inconistent handling of machine-readable copyright information in this source package. I had already done most of that work on a private branch in 2020, so this is mostly ready but I still need to resolve a licensing issue with AppStream metadata. Other packages For kernel-handbook, there was already a trivial "CI" pipeline used to push static pages to the web site. I've replaced this with the common pipeline plus a job that will push the pages from each build on the master branch. For everything else, it was straightforward to enable the common pipeline with a little bit of configuration. [Less]

In January I was assigned 24 hours of work by Freexian's Debian LTS initiative. I worked 16 hours, and will carry over the remaining time to February. I sent various backported security fixes for Linux to the stable mailing list, and ... [More] they have been included in subsequent stable releases. I rebased the linux package on the latest 4.9-stable release, but did not yet upload it. [Less]

I have a backlog of digital stuff to upload to archive.org that I'm finally starting to flush. I'd had some difficulties in getting uploads to work until I tried using the excellent ia CLI tool, which is packaged in Debian. In January 2022 I ... [More] uploaded: two sets of multitracks from the 1981 album My Life in the Bush of Ghosts by Brian Eno and David Byrne, "A Secret Life" and "Help Me Somebody"; 28 cover disks from late 80s/early 90s issues of PC Plus, Pc Today, PC Zone and Personal Computing; and a LEGO Land Billund brochure from 1993. I try to keep a record of my amateur archiving work (and related TODO items) at archiving. In January I also discovered that, late last year, the nine inch nails live archive project have had their download links forcibly removed by their ISP. I've been watching this project for a while because I thought it was "at risk". Years ago the project's creator, Ryan, created a series of torrents covering all the material at the time, although there had been a number of changes since the last one. However, this issue with his ISP spurred him on to create up-to-date torrents. It seems to me that archive.org would be a good place for this material, but there are two problems: the most natural collection would be the Live Music Archive, but it has a rule that musicians must have given explicit permission for live recordings to be deposited there (inherited from the earlier etree.org days) and in this case no such permission exists. The second problem is that ninlive's owner/creator, Ryan, has always resisted the idea. For now the best I can think to do is simply to continue to seed the torrent. [Less]