Posted
about 2 years
ago
This article is part of a series on how to setup a bare-metal CI system for
Linux driver development. Here are the different articles so far:
Part 1:
The high-level view of the whole CI system, and how to fully control test machines remotely
... [More]
(power on, OS to boot, keyboard/screen emulation using a serial console);
Part 2:
Comparing the different ways to generate the rootfs of your test environment, and introducing the boot2container project.
In this article, we will further discuss the role of the CI gateway, and which
steps we can take to simplify its deployment, maintenance, and disaster
recovery.
This work is sponsored by the Valve Corporation.
Requirements for the CI gateway
As seen in the
part 1
of this CI series, the testing gateway is sitting between the test machines
and the public network/internet:
Internet / ------------------------------+
Public network |
+---------+--------+ USB
| +-----------------------------------+
| Testing | Private network |
Main power (120/240 V) -----+ | Gateway +-----------------+ |
| +------+--+--------+ | |
| | | Serial / | |
| Main | | Ethernet | |
| Power| | | |
+-----------+-----------------|--+--------------+ +-------+--------+ +----+----+
| Switchable PDU | | | RJ45 switch | | USB Hub |
| Port 0 Port 1 ...| Port N | | | | |
+----+------------------------+-----------------+ +---+------------+ +-+-------+
| | |
Main | | |
Power| | |
+--------|--------+ Ethernet | |
| +-----------------------------------------+ +----+----+ |
| Test Machine 1 | Serial (RS-232 / TTL) | Serial | |
| +---------------------------------------------+ 2 USB +----+ USB
+-----------------+ +---------+
The testing gateway's role is to expose the test machines to the users, either
directly or via GitLab/Github. As such, it will likely require the following
components:
a host Operating System;
a config file describing the different test machines;
a bunch of services to expose said machines and deploy their test environment on demand.
Since the gateway is connected to the internet, both the OS and the different
services needs to be be kept updated relatively often to prevent your CI farm
from
becoming part of a botnet.
This creates interesting issues:
How do we test updates ahead of deployment, to minimize downtime due to bad updates?
How do we make updates atomic, so that we never end up with a partially-updated system?
How do we rollback updates, so that broken updates can be quickly reverted?
These issues can thankfully be addressed by running all the services in a
container (as systemd units), started using
boot2container.
Updating the operating system and the services would simply be done by generating a new
container, running tests to validate it, pushing it to a container registry, rebooting
the gateway, then waiting while the gateway downloads and execute the new services.
Using boot2container does not however fix the issue of how to update the
kernel or boot configuration when the system fails to boot the current one.
Indeed, if the kernel/boot2container/kernel command line are stored locally,
they can only be modified via an SSH connection and thus require the machine
to always be reachable, the gateway will be bricked until an operator boots an
alternative operating system.
The easiest way not to brick your gateway after a broken update is to power it
through a switchable
PDU (so
that we can power cycle the machine), and to download the kernel,
initramfs (boot2container), and the kernel command line from a remote server
at boot time. This is fortunately possible even through the internet by using
fancy bootloaders, such as
iPXE, and this will be the focus of this article!
Tune in for part 4 to learn more about how to create the container.
iPXE + boot2container: Netbooting your CI infrastructure from anywhere
iPXE is a tiny bootloader that packs a punch! Not only can it boot kernels from
local partitions, but it can also connect to the internet, and download
kernels/initramfs using HTTP(S). Even more impressive is the little scripting
engine which executes boot scripts instead of declarative boot configurations
like grub. This enables creating loops, endlessly trying to boot until one
method finally succeeds!
Let's start with a basic example, and build towards a production-ready
solution!
Netbooting from a local server
In this example, we will focus on netbooting the gateway from a local HTTP
server. Let's start by reviewing a simple script that makes iPXE acquire an IP
from the local DHCP server, then download and execute another iPXE script from
http://:8000/boot/ipxe.
If any step failed, the script will be restarted from the start until a successful
boot is achieved.
#!ipxe
echo Welcome to Valve infra's iPXE boot script
:retry
echo Acquiring an IP
dhcp || goto retry # Keep retrying getting an IP, until we get one
echo Got the IP: $${netX/ip} / $${netX/netmask}
echo
echo Chainloading from the iPXE server...
chain http://:8000/boot.ipxe
# The boot failed, let's restart!
goto retry
Neat, right? Now, we need to generate a bootable ISO image starting iPXE with
the above script run as a default. We will then flash this ISO to a USB
pendrive:
$ git clone git://git.ipxe.org/ipxe.git
$ make -C ipxe/src -j`nproc` bin/ipxe.iso EMBED=
$ sudo dd if=ipxe/src/bin/ipxe.iso of=/dev/sdX bs=1M conv=fsync status=progress
Once connected to the gateway, ensure that you boot from the pendrive, and you
should see iPXE bootloader trying to boot the kernel, but failing to download
the script from
http://:8000/boot.ipxe. So, let's write one:
#!ipxe
kernel /files/kernel b2c.container="docker://hello-world"
initrd /files/initrd
boot
This script specifies the following elements:
kernel: Download the kernel at http://:8000/files/kernel, and set the kernel command line to ask boot2container to start the hello-world container
initrd: Download the initramfs at http://:8000/files/initrd
boot: Boot the specified boot configuration
Assuming your gateway has an architecture supported by boot2container, you may
now download the kernel and initrd from
boot2container's releases page.
In case it is unsupported,
create an issue,
or a merge request
to add support for it!
Now that you have created all the necessary files for the boot, start the
web server on your development machine:
$ ls
boot.ipxe initrd kernel
$ python -m http.server 8080
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
- - [09/Jan/2022 15:32:52] "GET /boot.ipxe HTTP/1.1" 200 -
- - [09/Jan/2022 15:32:56] "GET /kernel HTTP/1.1" 200 -
- - [09/Jan/2022 15:32:54] "GET /initrd HTTP/1.1" 200 -
If everything went well, the gateway should, after a couple of seconds, start
downloading the boot script, then the kernel, and finally the initramfs. Once
done, your gateway should boot Linux, run docker's hello-world container, then
shut down.
Congratulations for netbooting your gateway! However, the current solution has
one annoying constraint: it requires a trusted local network and server because
we are using HTTP rather than HTTPS... On an untrusted network, a man in the
middle could override your boot configuration and take over your CI...
If we were using HTTPS, we could download our boot script/kernel/initramfs
directly from any public server, even GIT forges, without fear of any man in
the middle! Let's try to achieve this!
Netbooting from public servers
In the previous section, we managed to netboot our gateway from the local
network. In this section, we try to improve on it by netbooting using HTTPS.
This enables booting from a public server hosted at places such as
Linode for $5/month.
As I said earlier, iPXE supports HTTPS. However, if you are anyone like me,
you may be wondering how such a small bootloader could know which root
certificates to trust. The answer is that iPXE generates an SSL certificate
at compilation time which is then used to sign all of the root certificates
trusted by Mozilla (default), or any amount of certificate you may want.
See iPXE's crypto page for more information.
WARNING: iPXE currently does not like certificates exceeding 4096 bits. This
can be a limiting factor when trying to connect to existing servers. We hope to
one day fix this bug, but in the mean time, you may be forced to use a 2048 bits
Let's Encrypt certificate on a self-hosted web server.
See our issue
for more information.
WARNING 2: iPXE only supports a limited amount of ciphers. You'll need to make
sure they are listed in nginx's ssl_ciphers configuration:
AES-128-CBC:AES-256-CBC:AES256-SHA256 and AES128-SHA256:AES256-SHA:AES128-SHA
To get started, install NGINX + Let's encrypt on your server, following your
favourite tutorial, copy the boot.ipxe, kernel, and initrd files to the root
of the web server, then make sure you can download them using your browser.
With this done, we just need to edit iPXE's general config C header to enable
HTTPS support:
$ sed -i 's/#undef\tDOWNLOAD_PROTO_HTTPS/#define\tDOWNLOAD_PROTO_HTTPS/' ipxe/src/config/general.h
Then, let's update our boot script to point to the new server:
#!ipxe
echo Welcome to Valve infra's iPXE boot script
:retry
echo Acquiring an IP
dhcp || goto retry # Keep retrying getting an IP, until we get one
echo Got the IP: $${netX/ip} / $${netX/netmask}
echo
echo Chainloading from the iPXE server...
chain https:///boot.ipxe
# The boot failed, let's restart!
goto retry
And finally, let's re-compile iPXE, reflash the gateway pendrive, and boot the gateway!
$ make -C ipxe/src -j`nproc` bin/ipxe.iso EMBED=
$ sudo dd if=ipxe/src/bin/ipxe.iso of=/dev/sdX bs=1M conv=fsync status=progress
If all went well, the gateway should boot and run the hello world container
once again! Let's continue our journey by provisioning and backup'ing the local
storage of the gateway!
Provisioning and backups of the local storage
In the previous section, we managed to control the boot configuration of our
gateway via a public HTTPS server. In this section, we will improve on that by
provisioning and backuping any local file the gateway container may need.
Boot2container has a nice feature that enables you to create a volume, and
provision it from a bucket in a S3-compatible cloud storage, and sync back any
local change. This is done by adding the following arguments to the kernel
command line:
b2c.minio="s3,${s3_endpoint},${s3_access_key_id},${s3_access_key}": URL and credentials to the S3 service
b2c.volume="perm,mirror=s3/${s3_bucket_name},pull_on=pipeline_start,push_on=changes,overwrite,delete": Create a perm podman volume, mirror it from the bucket ${s3_bucket_name} when booting the gateway, then push any local change back to the bucket. Delete or overwrite any existing file when mirroring.
b2c.container="-ti -v perm:/mnt/perm docker://alpine": Start an alpine container, and mount the perm container volume to /mnt/perm
Pretty, isn't it? Provided that your bucket is configured to save all the
revisions of every file, this trick will kill three birds with one stone:
initial provisioning, backup, and automatic recovery of the files in case
the local disk fails and gets replaced with a new one!
The issue is that the boot configuration is currently open for everyone to see,
if they know where to look for. This means that anyone could tamper with your
local storage or even use your bucket to store their files...
Securing the access to the local storage
To prevent attackers from stealing our S3 credentials by simply pointing their
web browser to the right URL, we can authenticate incoming HTTPS requests by
using an SSL client certificate. A different certificate would be embedded in
every gateway's iPXE bootloader and checked by NGINX before serving the boot
configuration for this precise gateway. By limiting access to a machine's boot
configuration to its associated client certificate fingerprint, we even
prevent compromised machines from accessing the data of other machines.
Additionally, secrets should not be kept in the kernel command line, as any
process executed on the gateway could easily gain access to it by reading
/proc/cmdline. To address this issue, boot2container has a
b2c.extra_args_url argument to source additional parameters from this URL.
If this URL is generated every time the gateway is downloading its boot
configuration, can be accessed only once, and expires soon after being created,
then secrets can be kept private inside boot2container and not be exposed to
the containers it starts.
Implementing these suggestions in a blog post is a little tricky, so I suggest
you check out valve-infra's
ipxe-boot-server
component for more details. It provides a
Makefile
that makes it super easy to generate working certificates and create bootable
gateway ISOs, a small python-based web service that will serve the right
configuration to every gateway (including one-time secrets), and step-by-step
instructions to deploy everything!
Assuming you decided to use this component and followed the README, you should
then configure the gateway in this way:
$ pwd
/home/ipxe/valve-infra/ipxe-boot-server/files//
$ ls
boot.ipxe initrd kernel secrets
$ cat boot.ipxe
#!ipxe
kernel /files/kernel b2c.extra_args_url="${secrets_url}" b2c.container="-v perm:/mnt/perm docker://alpine" b2c.ntp_peer=auto b2c.cache_device=auto
initrd /files/initrd
boot
$ cat secrets
b2c.minio="bbz,${s3_endpoint},${s3_access_key_id},${s3_access_key}" b2c.volume="perm,mirror=bbz/${s3_bucket_name},pull_on=pipeline_start,push_on=changes,overwrite,delete"
And that's it! We finally made it to the end, and created a secure way to
provision our CI gateways with the wanted kernel, Operating System, and even
local files!
When Charlie Turner and I started
designing this system, we felt it would be a clean and simple way to solve our
problems with our CI gateways, but the implementation ended up being quite a
little trickier than the high-level view... especially the SSL certificates!
However, the certainty that we can now deploy updates and fix our CI gateways
even when they are physically inaccessible from us (provided the hardware and
PDU are fine) definitely made it all worth it and made the prospect of having
users depending on our systems less scary!
Let us know how you feel about it!
Conclusion
In this post, we focused on provisioning the CI gateway with its boot
configuration, and local files via the internet. This drastically reduces
the risks that updating the gateway's kernel would result in an extended
loss of service, as the kernel configuration can quickly be reverted by
changing the boot config files which is served from a cloud service provider.
The local file provisioning system also doubles as a backup, and disaster
recovery system which will automatically kick in in case of hardware failure
thanks to the constant mirroring of the local files with an S3-compatible
cloud storage bucket.
In the next post, we will be talking about how to create the infra container,
and how we can minimize down time during updates by not needing to reboot the
gateway.
That's all for now, thanks for making it to the end! [Less]
|
Posted
about 2 years
ago
This Is A Serious Blog
I posted some fun fluff pieces last week to kick off the new year, but now it’s time to get down to brass tacks.
Everyone knows adding features is just flipping on the enable button. Now it’s time to see some real work.
If
... [More]
you don’t like real work, stop reading. Stop right now. Now.
Alright, now that all the haters are gone, let’s put on our bisecting snorkels and dive in.
Regressions Suck
The dream of 2022 was that I’d come back and everything would work exactly how I left it. All the same tests would pass, all the perf would be there, and my driver would compile.
I got two of those things, which isn’t too bad.
After spending a while bisecting and debugging last week, I categorized a number of regressions to RADV problems which probably only affect me since there’s no Vulkan CTS cases for them (yet). But today I came to the last of the problem cases: dEQP-GLES31.functional.tessellation_geometry_interaction.feedback.tessellation_output_quads_geometry_output_points.
There’s nothing too remarkable about the test. It’s XFB, so, according to Jason Ekstrand, future Head of Graphic Wows at Pixar, it’s terrible.
What is remarkable, however is that the test passes fine when run in isolation.
Here We Go Again
Anyone who’s anyone knows what comes next.
You find the caselist of the other 499 tests that were run in this block
You run the caselist
You find out that the test still fails in that caselist
You tip your head back to stare at the ceiling and groan
Then it’s another X minutes (where X is usually between 5 and 180 depending on test runtimes) to slowly pare down the caselist to the sequence which actually triggers the failure. For those not in the know, this type of failure indicates a pathological driver bug where a sequence of commands triggers different results if tests are run in a different order.
There is, to my knowledge, no ‘automatic’ way to determine exactly which tests are required to trigger this type of failure from a caselist. It would be great if there was, and it would save me (and probably others who are similarly unaware) considerable time doing this type of caselist fuzzing.
Finally, I was left with this shortened caselist:
dEQP-GLES31.functional.shaders.builtin_constants.tessellation_shader.max_tess_evaluation_texture_image_units
dEQP-GLES31.functional.tessellation_geometry_interaction.feedback.tessellation_output_quads_geometry_output_points
dEQP-GLES31.functional.ubo.random.all_per_block_buffers.25
What Now?
Ideally, it would be great to be able to use something like gfxreconstruct for this. I could record two captures—one of the test failing in the caselist and one where it passes in isolation—and then compare them.
Here’s an excerpt from that attempt:
"[790]vkCreateShaderModule": {
"return": "VK_SUCCESS",
"device": "0x0x4",
"pCreateInfo": {
"sType": "VK_STRUCTURE_TYPE_SHADER_MODULE_CREATE_INFO",
"pNext": null,
"flags": 0,
"codeSize": Unhandled VkFormatFeatureFlagBits2KHR,
"pCode": "0x0x285c8e0"
},
"pAllocator": null,
"[out]pShaderModule": "0x0xe0"
},
Why is it trying to print an enum value for codeSize you might ask?
I’m not the only one to ask, and it’s still an unresolved mystery.
I was successful in doing the comparison with gfxreconstruct, but it yielded nothing of interest.
Puzzled, I decided to try the test out on lavapipe. Would it pass?
No.
It similarly fails on llvmpipe and IRIS.
But my lavapipe testing revealed an important clue. Given that there are no synchronization issues with lavapipe, this meant I could be certain this was a zink bug. Furthermore, the test failed both when the bug was exhibiting and when it wasn’t, meaning that I could actually see the “passing” values in addition to the failing ones for comparison.
Here’s the failing error output:
Verifying feedback results.
Element at index 0 (tessellation invocation 0) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.166663, 0.5, 1)
Element at index 1 (tessellation invocation 1) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.166663, 0.5, 1)
Element at index 2 (tessellation invocation 2) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.166663, 0.5, 1)
Element at index 3 (tessellation invocation 3) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.433337, 0.5, 1)
Element at index 4 (tessellation invocation 4) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.433337, 0.5, 1)
Element at index 5 (tessellation invocation 5) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.433337, 0.5, 1)
Element at index 6 (tessellation invocation 6) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.4, -0.433337, 0.5, 1)
Element at index 7 (tessellation invocation 7) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.4, -0.433337, 0.5, 1)
Omitted 24 error(s).
And here’s the passing error output:
Verifying feedback results.
Element at index 3 (tessellation invocation 1) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.433337, 0, 1)
Element at index 4 (tessellation invocation 2) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.433337, 0, 1)
Element at index 5 (tessellation invocation 3) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.433337, 0, 1)
Element at index 6 (tessellation invocation 4) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.4, -0.433337, 0, 1)
Element at index 7 (tessellation invocation 5) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.4, -0.433337, 0, 1)
Element at index 8 (tessellation invocation 6) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.4, -0.433337, 0, 1)
Element at index 9 (tessellation invocation 7) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.433337, 0, 1)
Element at index 10 (tessellation invocation 8) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.433337, 0, 1)
Omitted 18 error(s).
This might not look like much, but to any zinkologists, there’s an immediate red flag: the Z component of the vertex is 0.5 in the failing case.
What does this remind us of?
Naturally it reminds us of nir_lower_clip_halfz, the compiler pass which converts OpenGL Z coordinate ranges ([-1, 1]) to Vulkan ([0, 1]). This pass is run on the last vertex stage, but if it gets run more than once, a value of -1 becomes 0.5.
Thus, it looks like the pass is being run twice in this test. How can this be verified?
ZINK_DEBUG=spirv will export all spirv shaders used by an app. Therefore, dumping all the shaders for passing and failing runs should confirm that the conversion pass is being run an extra time when they’re compared. The verdict?
@@ -1,7 +1,7 @@
; SPIR-V
; Version: 1.5
; Generator: Khronos; 0
-; Bound: 23
+; Bound: 38
; Schema: 0
OpCapability TransformFeedback
OpCapability Shader
@@ -36,13 +36,28 @@
%_ptr_Output_v4float = OpTypePointer Output %v4float
%gl_Position = OpVariable %_ptr_Output_v4float Output
%v4uint = OpTypeVector %uint 4
+%uint_1056964608 = OpConstant %uint 1056964608
%main = OpFunction %void None %3
%18 = OpLabel
OpBranch %17
%17 = OpLabel
%19 = OpLoad %v4float %a_position
%21 = OpBitcast %v4uint %19
- %22 = OpBitcast %v4float %21
- OpStore %gl_Position %22
+ %22 = OpCompositeExtract %uint %21 3
+ %23 = OpCompositeExtract %uint %21 3
+ %24 = OpCompositeExtract %uint %21 2
+ %25 = OpBitcast %float %24
+ %26 = OpBitcast %float %23
+ %27 = OpFAdd %float %25 %26
+ %28 = OpBitcast %uint %27
+ %30 = OpBitcast %float %28
+ %31 = OpBitcast %float %uint_1056964608
+ %32 = OpFMul %float %30 %31
+ %33 = OpBitcast %uint %32
+ %34 = OpCompositeExtract %uint %21 1
+ %35 = OpCompositeExtract %uint %21 0
+ %36 = OpCompositeConstruct %v4uint %35 %34 %33 %22
+ %37 = OpBitcast %v4float %36
+ OpStore %gl_Position %37
OpReturn
OpFunctionEnd
And, as is the rule for such things, the fix was a simple one-liner to unset values in the vertex shader key.
It wasn’t technically a regression, but it manifested as such, and fixing it yielded another dozen or so fixes for cases which were affected by the same issue.
Blammo. [Less]
|
Posted
about 2 years
ago
PyCook
A few months ago, I went on a quest to better digitize and collect a bunch of the recipes I use on a regular basis. Like most people, I’ve got a 3-ring binder of stuff I’ve printed from the internet, a box with the usual 4x6 cards, most of
... [More]
which are hand-written, and a stack of cookbooks. I wanted something that could be both digital and physical and which would make recipes easier to share. I also wanted whatever storage system I developed to be something stupid simple. If there’s one thing I’ve learned about myself over the years it’s that if I make something too hard, I’ll never get around to it.
This led to a few requirements:
A simple human-writable file format
Recipes organized as individual files in a Git repo
A simple tool to convert to a web page and print formats
That tool should be able to produce 4x6 cards
Enter PyCook. It does pretty much exactly what the above says and not much more. The file format is based on YAML and, IMO, is beautiful to look at all by itself and very easy to remember how to type:
name: Swedish Meatballs
from: Grandma Ekstrand
ingredients:
- 1 [lb] Ground beef (80%)
- 1/4 [lb] Ground pork
- 1 [tsp] Salt
- 1 [tsp] Sugar
- 1 Egg, beaten
- 1 [tbsp] Catsup
- 1 Onion, grated
- 1 [cup] Bread crumbs
- 1 [cup] Milk
instructions:
- Soak crumbs in milk
- Mix all together well
- Make into 1/2 -- 3/4 [in] balls and brown in a pan on the stove
(you don't need to cook them through; just brown)
- Heat in a casserole dish in the oven until cooked through.
Over Christmas this year, I got my mom and a couple siblings together to put together a list of family favorite recipes. The format is simple enough that my non-technical mother was able to help type up recipes and they needed very little editing before PyCook would consume them. To me, that’s a pretty good indicator that the file format is simple enough. 😁
The one bit of smarts it does have is around quantities and units. One thing that constantly annoys me with recipes is the inconsistency with abbreviations of units. Take tablespoons, for instance. I’ve seen it abbreviated “T” (as opposed to “t” for teaspoon), “tbsp”, or “tblsp”, sometimes with a “.” after the abbreviation and sometimes not. To handle this, I have a tiny macro language where units have standard abbreviations and are placed in brackets. This is then substituted with the correct abbreviation to let me change them all in one go if I ever want to. It’s also capable of handling plurals properly so when you type [cup] it will turn into either “cup” or “cups” depending on the associated number. It also has smarts to detect “1/4” and turn that into vulgar fraction character “¼” for HTML output or a nice LaTeX fraction when doing PDF output.
That brings me to the PDF generator. One of my requirements was the ability to produce 4x6 cards that I could use while cooking instead of having an electronic device near all those fluids. I started with the LaTeX template I got from my friend Steve Schulteis for making a PDF cookbook and adapted them to 4x6 cards, one recipe per card. And the results look pretty nice, I think
Meatballs recipe as a 4x6 card
It’s even able to nicely paginate the 4x6 cards into a double-sided 8.5x11 PDF which prints onto Avery 5389 templates.
Other output formats include an 8.5x11 PDF cookbook with as many recipes per page as will fit and an HTML web version generated using Sphinx which is searchable and has a nice index.
P.S. Yes, that’s a real Swedish meatball recipe and it makes way better meatballs than the ones from IKEA. [Less]
|
Posted
about 2 years
ago
New blog!
This week, I’ve taken advantage of some of my funemployment time to we work our website infrastructure. As part of my new job (more on that soon!) I’ll be communicating publicly more and I need a better blog space. I set up a blogger a few
... [More]
years ago but I really hate it. It’s not that Blogger is a terrible platform per se. It just doesn’t integrate well with the rest of my website and the comments I got were 90% spam.
At the time, I used Blogger because I didn’t want to mess implementing a blog on my own website infrastructure. Why? The honest answer is an object lesson in software engineering. The last time I re-built my website I thought that building a website generator sounded like a fantastic excuse to learn some Ruby. Why not? It’s a great programming language for web stuff and learning new languages is good, right? While those are both true, software maintainability is important. When I went to try and add a blog, it’d been nearly 4 years since I’d written a line of ruby code and the static site generation framework I was using (nanoc) had moved to a backwards-incompatible new version and I had no clue how to move my site forward without re-learning Ruby and nanoc and rewriting it from scratch. I didn’t have the time for that.
This time, I learned my lesson and went all C programmer on it. The new infra is built in Python, a language I use nearly daily in my Mesa work and, instead of using someone else’s static site generation infra, I rolled my own. I’m using Jinja2 for templating as well as Pygments for syntax highlighting and Pandoc for Markdown to HTML conversion. They’re all libraries not frameworks so they take a lot less learning to figure out and remember how to use. Jinja2 is something of a framework (it’s a templating language) but it’s so obvious and user-friendly that it’s basically impossible to forget how to use.
Sure, my infra isn’t nearly as nice as some. I don’t have caching and I have to re-build the whole thing every time. I also didn’t bother to implement RSS feed support, comments, or any of those other bloggy features. But, frankly, I don’t care. Even on a crappy raspberry pi, my website builds in a few seconds. As far as RSS goes, who actually uses RSS readers these days? Just follow me on Twitter (@jekstrand_) if you want to know when I blog stuff. Comments? 95% of the comments I got on Blogger were spam anyway. If you want to comment, reply to my tweet.
The old blog lived at jason-blog.jlekstrand.net while the new one lives at www.jlekstrand.net/jason/blog/. I’ve set up my webserver to re-direct from the old one and gone though the painful process of checking every single post to ensure the re-directs work. Any links you may have should still work.
So there you have it! A new website framework written in all of 2 days. Hopefully, this one will be maintainable and, if I need to extend it, I can without re-learning whole programming languages. I’m also hoping that now that blogging is as easy as writing some markdown and doing git push, I’ll actually blog more. [Less]
|
Posted
about 2 years
ago
Chug-a-chug-a-chug-a-chug-a-chug-a
It’s a busy week here at SGC. There’s emails to read, tickets to catch up on, rumors to spread about jekstrand’s impending move to Principal Engineer of Bose’s headphone compiler team, code to unwrite. The usual.
... [More]
Except now I’m actually around to manage everything instead of ignoring it.
Let’s do a brief catchup of today’s work items.
Sparse Textures
I said this was done yesterday, but the main CTS case for the extension is broken, so I didn’t adequately test it. Fortunately, Qiang Yu from AMD is on the case in addition to doing the original Gallium implementations for these extensions, and I was able to use a WIP patch to fix the test. And run it. And then run it again. And then run it in gdb. And then… And then…
Anyway, it all passes now, and sparse texture support is good to go once Australia comes back from vacation to review patches.
Also I fixed sparse buffer support, which I accidentally broke 6+ months ago but never noticed since only RADV implements these features and I have no games in my test list that use them.
Queries
I hate queries. Everyone knows I hate queries. The query code is the worst code in the entire driver. If I never have to open zink_query.c again, I will still have opened it too many times for a single lifetime.
But today I hucked myself back in yet again to try and stop a very legitimate and legal replay of a Switch game from crashing. Everyone knows that anime is the real primary driver of all technology, so as soon as anyone files an anime-related ticket, all driver developers drop everything they’re doing to solve it. Unless they’re on vacation.
In this case, the problem amounted to:
vulkan query pools have a maximum number of queries
exceeding this causes a crash
trying not to exceed it also causes a crash if the person writing the code is dumb
2021 me was much dumber than 2022 me
Rejoice, for you can now play all your weeb games on zink if for some reason that’s where you’re at in your life.
But I’m not judging.
Source Games: Do More Of Them Run On Gallium Nine In 2022?
Yes.
I came back to the gift of a new CS:GO version which adds DXVK support, so now there’s also Gallium Nine support. It works fine.
Does it work better than other engines?
I don’t know, and I have real work to do so I’m not going to test it, but surely someone will take an interest in benchmarking such things now that I’ve heroically git added a 64bit wrapper to my repo that can be used for testing.
A quick reminder that all Gallium Nine blog post references and tests happen with RadeonSI. [Less]
|
Posted
about 2 years
ago
It appears that Google created a handy tool that helps finding the command which causes a GPU hang/crash. It is called Graphics Flight Recorder (GFR) and was open-sourced a year ago but didn’t receive any attention. From the readme:
The Graphics
... [More]
Flight Recorder (GFR) is a Vulkan layer to help trackdown and identify the cause of GPU hangs and crashes. It works by instrumenting command buffers with completion tags. When an error is detected a log file containing incomplete command buffers is written. Often the last complete or incomplete commands are responsible for the crash.
It requires VK_AMD_buffer_marker support; however, this extension is rather trivial to implement - I had only to copy-paste the code from our vkCmdSetEvent implementation and that was it. Note, at the moment of writing, GFR unconditionally usesVK_AMD_device_coherent_memory, which could be manually patched out for it to run on other GPUs.
GFR already helped me to fix hangs in “Alien: Isolation” and “Digital Combat Simulator”. In both cases the hang was in a compute shader and the output from GFR looked like:
...
- # Command:
id: 6/9
markerValue: 0x000A0006
name: vkCmdBindPipeline
state: [SUBMITTED_EXECUTION_COMPLETE]
parameters:
- # parameter:
name: commandBuffer
value: 0x000000558CFD2A10
- # parameter:
name: pipelineBindPoint
value: 1
- # parameter:
name: pipeline
value: 0x000000558D3D6750
- # Command:
id: 6/9
message: '>>>>>>>>>>>>>> LAST COMPLETE COMMAND <<<<<<<<<<<<<<'
- # Command:
id: 7/9
markerValue: 0x000A0007
name: vkCmdDispatch
state: [SUBMITTED_EXECUTION_INCOMPLETE]
parameters:
- # parameter:
name: commandBuffer
value: 0x000000558CFD2A10
- # parameter:
name: groupCountX
value: 5
- # parameter:
name: groupCountY
value: 1
- # parameter:
name: groupCountZ
value: 1
internalState:
pipeline:
vkHandle: 0x000000558D3D6750
bindPoint: compute
shaderInfos:
- # shaderInfo:
stage: cs
module: (0x000000558F82B2A0)
entry: "main"
descriptorSets:
- # descriptorSet:
index: 0
set: 0x000000558E498728
- # Command:
id: 8/9
markerValue: 0x000A0008
name: vkCmdPipelineBarrier
state: [SUBMITTED_EXECUTION_NOT_STARTED]
...
After confirming that corresponding vkCmdDispatch is indeed the call which hangs, in both cases I made an Amber test which fully simulated the call. For a compute shader, this is relatively easy to do since all you need is to save the decompiled shader and buffers being used by it. Luckily in both cases these Amber tests reproduced the hangs.
With standalone reproducers, the problems were much easier to debug, and fixes were made shortly: MR#14044 for “Alien: Isolation” and MR#14110 for “Digital Combat Simulator”.
Unfortunately this tool is not a panacea:
It likely would fail to help with unrecoverable hangs where it would be impossible to read the completion tags back.
Or when the mere addition of the tags could “fix” the issue which may happen with synchronization issues.
If draw/dispatch calls run in parallel on the GPU, writing tags may force them to execute sequentially or to be imprecise.
Anyway, it’s easy to use so you should give it a try. [Less]
|
Posted
about 2 years
ago
We Back
The blog is back. I know everyone’s been furiously spamming F5 to see if there were any secret new posts, but no. There were not.
Today’s the first day of the new year, so I had to dig deep to remember how to do basic stuff like shitpost on
... [More]
IRC. And then someone told me jekstrand was going to Broadcom to work on Windows network drivers?
I’m just gonna say it now:
2022 has gone too far.
I know it’s early, I know some people are seeing this as a hot take, but I’m throwing the statement down before things get worse.
Knock it off, 2022.
Zink
Somehow the driver is still in the tree, still builds, and still runs. It’s a miracle.
Thus, since there were obviously no other matters more pressing than not falling behind on MesaMatrix, I spent the morning figuring out how to implement ARB_sparse_texture.
Was this the best decision when I didn’t even remember how to make meson clear its dependency cache? No. No it wasn’t.
But I did it anyway because here at SGC, we take bad ideas and turn them into code.
Your move, 2022. [Less]
|
Posted
over 2 years
ago
Starting with kernel 5.17 the kernel supports the builtin privacy screens built into the LCD panel of some new laptop models.This means that the drm drivers will now return -EPROBE_DEFER from their probe() method on models with a builtin privacy
... [More]
screen when the privacy screen provider driver has not been loaded yet.To avoid any regressions distors should modify their initrd generation tools to include privacy screen provider drivers in the initrd (at least on systems with a privacy screen), before 5.17 kernels start showing up in their repos.If this change is not made, then users using a graphical bootsplash (plymouth) will get an extra boot-delay of up to 8 seconds (DeviceTimeout in plymouthd.defaults) before plymouth will show and when using disk-encryption where the LUKS password is requested from the initrd, the system will fallback to text-mode after these 8 seconds.I've written a patch with the necessary changes for dracut, which might be useful as an example for how to deal with this in other initrd generators, see: https://github.com/dracutdevs/dracut/pull/1666I've also filed bugs for tracking this for Fedora, openSUSE, Arch, Debian and Ubuntu. [Less]
|
Posted
over 2 years
ago
Introduction
One of the big issues I have when working on Turnip driver development is that when compiling either Mesa or VK-GL-CTS it takes a lot of time to complete, no matter how powerful the embedded board is. There are reasons for that:
... [More]
typically those board have limited amount of RAM (8 GB for the best case), a slow storage disk (typically UFS 2.1 on-board storage) and CPUs that are not so powerful compared with x86_64 desktop alternatives.
Photo of the Qualcomm® Robotics RB3 Platform embedded board that I use for Turnip development.
To fix this, it is recommended to do cross-compilation, however installing the development environment for cross-compilation could be cumbersome and prone to errors depending on the toolchain you use. One alternative is to use a distributed compilation system that allows cross-compilation like Icecream.
Icecream is a distributed compilation system that is very useful when you have to compile big projects and/or on low-spec machines, while having powerful machines in the local network that can do that job instead. However, it is not perfect: the linking stage is still done in the machine that submits the job, which depending on the available RAM, could be too much for it (however you can alleviate this a bit by using ZRAM for example).
One of the features that icecream has over its alternatives is that there is no need to install the same toolchain in all the machines as it is able to share the toolchain among all of them. This is very useful as we will see below in this post.
Installation
Debian-based systems
$ sudo apt install icecc
Fedora systems
$ sudo dnf install icecream
Compile it from sources
You can compile it from sources.
Configuration of icecc scheduler
You need to have an icecc scheduler in the local network that will balance the load among all the available nodes connected to it.
It does not matter which machine is the scheduler, you can use any of them as it is quite lightweight. To run the scheduler execute the following command:
$ sudo icecc-scheduler
Notice that the machine running this command is going to be the scheduler but it will not participate in the compilation process by default unless you ran iceccd daemon as well (see next step).
Setup on icecc nodes
Launch daemon
First you need to run the iceccd daemon as root. This is not needed on Debian-based systems, as its systemd unit is enabled by default.
You can do that using systemd in the following way:
$ sudo systemctl start iceccd
Or you can enable the daemon at startup time:
$ sudo systemctl enable iceccd
The daemon will connect automatically to the scheduler that is running in the local network. If that’s not the case, or there are more than one scheduler, you can run it standalone and give the scheduler’s IP as parameter:
sudo iceccd -s
Enable icecc compilation
With ccache
If you use ccache (recommended option), you just need to add the following in your .bashrc:
export CCACHE_PREFIX=icecc
Without ccache
To use it without ccache, you need to add its path to $PATH envvar so it is picked before the system compilers:
export PATH=/usr/lib/icecc/bin:$PATH
Execution
Same architecture
If you followed the previous steps, any time you compile anything on C/C++, it will distribute the work among the fastest nodes in the network. Notice that it will take into account system load, network connection, cores, among other variables, to decide which node will compile the object file.
Remember that the linking stage is always done in the machine that submits the job.
Different architectures (example cross-compiling for aarch64 on x86_64 nodes)
Icemon showing my x86_64 desktop (maxwell) cross-compiling a job for my aarch64 board (rb3).
Preparation on x86_64 machine
In one x86_64 machine, you need to create a toolchain. This is not automatically done by icecc as you can have different toolchains for cross-compilation.
Install cross-compiler
For example, you can install the cross-compiler from the distribution repositories:
For Debian-based systems:
sudo apt install crossbuild-essential-arm64
For Fedora:
$ sudo dnf install gcc-aarch64-linux-gnu gcc--c++-aarch64-linux-gnu
Create toolchain for icecc
Finally, to create the toolchain to share in icecc:
$ icecc-create-env --gcc /usr/bin/aarch64-linux-gnu-gcc /usr/bin/aarch64-linux-gnu-g++
This will create a .tar.gz file. The is used to identify the toolchain to distribute among the nodes in case there is more than one. But don’t worry, once it is copied to a node, it won’t be copied again as it detects it is already present.
Note: it is important that the toolchain is compatible with the target machine. For example, if my aarch64 board is using Debian 11 Bullseye, it is better if the cross-compilation toolchain is created from a Debian Bullseye x86_64 machine (a VM also works), because you avoid incompatibilities like having different glibc versions.
If you have installed Debian 11 Bullseye in your aarch64, you can use my own cross-compilation toolchain for x86_64 and skip this step.
Copy the toolchain to the aarch64 machine
scp .tar.gz aarch64-machine-hostname:
Preparation on aarch64
Once the toolchain (.tar.gz) is copied to the aarch64 machine, you just need to export this on .bashrc:
# Icecc setup for crosscompilation
export CCACHE_PREFIX=icecc
export ICECC_VERSION=x86_64:~/.tar.gz
Execute
Just compile on aarch64 machine and the jobs be distributed among your x86_64 machines as well. Take into account the jobs will be shared among other aarch64 machines as well if icecc decides so, therefore no need to do any extra step.
It is important to remark that the cross-compilation toolchain creation is only needed once, as icecream will copy it on all the x86_64 machines that will execute any job launched by this aarch64 machine. However, you need to copy this toolchain to any aarch64 machines that will use icecream resources for cross-compiling.
Icecream monitor
This is an interesting graphical tool to see the status of the icecc nodes and the jobs under execution.
Install on Debian-based systems
$ sudo apt install icecc-monitor
Install on Fedora
$ sudo dnf install icemon
Install it from sources
You can compile it from sources.
Acknowledgments
Even though icecream has a good cross-compilation documentation, it was the post written 8 years ago by my Igalia colleague Víctor Jáquez the one that convinced me to setup icecream as explained in this post.
Hope you find this info as useful as I did :-) [Less]
|
Posted
over 2 years
ago
On the road to AppStream 1.0, a lot of items from the long todo list have been done so far – only one major feature is remaining, external release descriptions, which is a tricky one to implement and specify. For AppStream 1.0 it needs to be present
... [More]
or be rejected though, as it would be a major change in how release data is handled in AppStream.
Besides 1.0 preparation work, the recent 0.15 release and the releases before it come with their very own large set of changes, that are worth a look and may be interesting for your application to support. But first, for a change that affects the implementation and not the XML format:
1. Completely rewritten caching code
Keeping all AppStream data in memory is expensive, especially if the data is huge (as on Debian and Ubuntu with their large repositories generated from desktop-entry files as well) and if processes using AppStream are long-running. The latter is more and more the case, not only does GNOME Software run in the background, KDE uses AppStream in KRunner and Phosh will use it too for reading form factor information. Therefore, AppStream via libappstream provides an on-disk cache that is memory-mapped, so data is only consuming RAM if we are actually doing anything with it.
Previously, AppStream used an LMDB-based cache in the background, with indices for fulltext search and other common search operations. This was a very fast solution, but also came with limitations, LMDB’s maximum key size of 511 bytes became a problem quite often, adjusting the maximum database size (since it has to be set at opening time) was annoyingly tricky, and building dedicated indices for each search operation was very inflexible. In addition to that, the caching code was changed multiple times in the past to allow system-wide metadata to be cached per-user, as some distributions didn’t (want to) build a system-wide cache and therefore ran into performance issues when XML was parsed repeatedly for generation of a temporary cache. In addition to all that, the cache was designed around the concept of “one cache for data from all sources”, which meant that we had to rebuild it entirely if just a small aspect changed, like a MetaInfo file being added to /usr/share/metainfo, which was very inefficient.
To shorten a long story, the old caching code was rewritten with the new concepts of caches not necessarily being system-wide and caches existing for more fine-grained groups of files in mind. The new caching code uses Richard Hughes’ excellent libxmlb internally for memory-mapped data storage. Unlike LMDB, libxmlb knows about the XML document model, so queries can be much more powerful and we do not need to build indices manually. The library is also already used by GNOME Software and fwupd for parsing of (refined) AppStream metadata, so it works quite well for that usecase. As a result, search queries via libappstream are now a bit slower (very much depends on the query, roughly 20% on average), but can be mmuch more powerful. The caching code is a lot more robust, which should speed up startup time of applications. And in addition to all of that, the AsPool class has gained a flag to allow it to monitor AppStream source data for changes and refresh the cache fully automatically and transparently in the background.
All software written against the previous version of the libappstream library should continue to work with the new caching code, but to make use of some of the new features, software using it may need adjustments. A lot of methods have been deprecated too now.
2. Experimental compose support
Compiling MetaInfo and other metadata into AppStream collection metadata, extracting icons, language information, refining data and caching media is an involved process. The appstream-generator tool does this very well for data from Linux distribution sources, but the tool is also pretty “heavyweight” with lots of knobs to adjust, an underlying database and a complex algorithm for icon extraction. Embedding it into other tools via anything else but its command-line API is also not easy (due to D’s GC initialization, and because it was never written with that feature in mind). Sometimes a simpler tool is all you need, so the libappstream-compose library as well as appstreamcli compose are being developed at the moment. The library contains building blocks for developing a tool like appstream-generator while the cli tool allows to simply extract metadata from any directory tree, which can be used by e.g. Flatpak. For this to work well, a lot of appstream-generator‘s D code is translated into plain C, so the implementation stays identical but the language changes.
Ultimately, the generator tool will use libappstream-compose for any general data refinement, and only implement things necessary to extract data from the archive of distributions. New applications (e.g. for new bundling systems and other purposes) can then use the same building blocks to implement new data generators similar to appstream-generator with ease, sharing much of the code that would be identical between implementations anyway.
2. Supporting user input controls
Want to advertise that your application supports touch input? Keyboard input? Has support for graphics tablets? Gamepads? Sure, nothing is easier than that with the new control relation item and supports relation kind (since 0.12.11 / 0.15.0, details):
pointing
keyboard
touch
tablet
3. Defining minimum display size requirements
Some applications are unusable below a certain window size, so you do not want to display them in a software center that is running on a device with a small screen, like a phone. In order to encode this information in a flexible way, AppStream now contains a display_length relation item to require or recommend a minimum (or maximum) display size that the described GUI application can work with. For example:
360
This will make the application require a display length greater or equal to 300 logical pixels. A logical pixel (also device independent pixel) is the amount of pixels that the application can draw in one direction. Since screens, especially phone screens but also screens on a desktop, can be rotated, the display_length value will be checked against the longest edge of a display by default (by explicitly specifying the shorter edge, this can be changed).
This feature is available since 0.13.0, details. See also Tobias Bernard’s blog entry on this topic.
4. Tags
This is a feature that was originally requested for the LVFS/fwupd, but one of the great things about AppStream is that we can take very project-specific ideas and generalize them so something comes out of them that is useful for many. The new tags tag allows people to tag components with an arbitrary namespaced string. This can be useful for project-internal organization of applications, as well as to convey certain additional properties to a software center, e.g. an application could mark itself as “featured” in a specific software center only. Metadata generators may also add their own tags to components to improve organization. AppStream gives no recommendations as to how these tags are to be interpreted except for them being a strictly optional feature. So any meaning is something clients and metadata authors need to negotiate. It therefore is a more specialized usecase of the already existing custom tag, and I expect it to be primarily useful within larger organizations that produce a lot of software components that need sorting. For example:
vendor-2021q1
featured
This feature is available since 0.15.0, details.
5. MetaInfo Creator changes
The MetaInfo Creator (source) tool is a very simple web application that provides you with a form to fill out and will then generate MetaInfo XML to add to your project after you have answered all of its questions. It is an easy way for developers to add the required metadata without having to read the specification or any guides at all.
Recently, I added support for the new control and display_length tags, resolved a few minor issues and also added a button to instantly copy the generated output to clipboard so people can paste it into their project. If you want to create a new MetaInfo file, this tool is the best way to do it!
The creator tool will also not transfer any data out of your webbrowser, it is strictly a client-side application.
And that is about it for the most notable changes in AppStream land! Of course there is a lot more, additional tags for the LVFS and content rating have been added, lots of bugs have been squashed, the documentation has been refined a lot and the library has gained a lot of new API to make building software centers easier. Still, there is a lot to do and quite a few open feature requests too. Onwards to 1.0! [Less]
|