I Use This!
Very High Activity

News

Analyzed 1 day ago. based on code collected 2 days ago.
Posted about 2 years ago
This article is part of a series on how to setup a bare-metal CI system for Linux driver development. Here are the different articles so far: Part 1: The high-level view of the whole CI system, and how to fully control test machines remotely ... [More] (power on, OS to boot, keyboard/screen emulation using a serial console); Part 2: Comparing the different ways to generate the rootfs of your test environment, and introducing the boot2container project. In this article, we will further discuss the role of the CI gateway, and which steps we can take to simplify its deployment, maintenance, and disaster recovery. This work is sponsored by the Valve Corporation. Requirements for the CI gateway As seen in the part 1 of this CI series, the testing gateway is sitting between the test machines and the public network/internet: Internet / ------------------------------+ Public network | +---------+--------+ USB | +-----------------------------------+ | Testing | Private network | Main power (120/240 V) -----+ | Gateway +-----------------+ | | +------+--+--------+ | | | | | Serial / | | | Main | | Ethernet | | | Power| | | | +-----------+-----------------|--+--------------+ +-------+--------+ +----+----+ | Switchable PDU | | | RJ45 switch | | USB Hub | | Port 0 Port 1 ...| Port N | | | | | +----+------------------------+-----------------+ +---+------------+ +-+-------+ | | | Main | | | Power| | | +--------|--------+ Ethernet | | | +-----------------------------------------+ +----+----+ | | Test Machine 1 | Serial (RS-232 / TTL) | Serial | | | +---------------------------------------------+ 2 USB +----+ USB +-----------------+ +---------+ The testing gateway's role is to expose the test machines to the users, either directly or via GitLab/Github. As such, it will likely require the following components: a host Operating System; a config file describing the different test machines; a bunch of services to expose said machines and deploy their test environment on demand. Since the gateway is connected to the internet, both the OS and the different services needs to be be kept updated relatively often to prevent your CI farm from becoming part of a botnet. This creates interesting issues: How do we test updates ahead of deployment, to minimize downtime due to bad updates? How do we make updates atomic, so that we never end up with a partially-updated system? How do we rollback updates, so that broken updates can be quickly reverted? These issues can thankfully be addressed by running all the services in a container (as systemd units), started using boot2container. Updating the operating system and the services would simply be done by generating a new container, running tests to validate it, pushing it to a container registry, rebooting the gateway, then waiting while the gateway downloads and execute the new services. Using boot2container does not however fix the issue of how to update the kernel or boot configuration when the system fails to boot the current one. Indeed, if the kernel/boot2container/kernel command line are stored locally, they can only be modified via an SSH connection and thus require the machine to always be reachable, the gateway will be bricked until an operator boots an alternative operating system. The easiest way not to brick your gateway after a broken update is to power it through a switchable PDU (so that we can power cycle the machine), and to download the kernel, initramfs (boot2container), and the kernel command line from a remote server at boot time. This is fortunately possible even through the internet by using fancy bootloaders, such as iPXE, and this will be the focus of this article! Tune in for part 4 to learn more about how to create the container. iPXE + boot2container: Netbooting your CI infrastructure from anywhere iPXE is a tiny bootloader that packs a punch! Not only can it boot kernels from local partitions, but it can also connect to the internet, and download kernels/initramfs using HTTP(S). Even more impressive is the little scripting engine which executes boot scripts instead of declarative boot configurations like grub. This enables creating loops, endlessly trying to boot until one method finally succeeds! Let's start with a basic example, and build towards a production-ready solution! Netbooting from a local server In this example, we will focus on netbooting the gateway from a local HTTP server. Let's start by reviewing a simple script that makes iPXE acquire an IP from the local DHCP server, then download and execute another iPXE script from http://:8000/boot/ipxe. If any step failed, the script will be restarted from the start until a successful boot is achieved. #!ipxe echo Welcome to Valve infra's iPXE boot script :retry echo Acquiring an IP dhcp || goto retry # Keep retrying getting an IP, until we get one echo Got the IP: $${netX/ip} / $${netX/netmask} echo echo Chainloading from the iPXE server... chain http://:8000/boot.ipxe # The boot failed, let's restart! goto retry Neat, right? Now, we need to generate a bootable ISO image starting iPXE with the above script run as a default. We will then flash this ISO to a USB pendrive: $ git clone git://git.ipxe.org/ipxe.git $ make -C ipxe/src -j`nproc` bin/ipxe.iso EMBED= $ sudo dd if=ipxe/src/bin/ipxe.iso of=/dev/sdX bs=1M conv=fsync status=progress Once connected to the gateway, ensure that you boot from the pendrive, and you should see iPXE bootloader trying to boot the kernel, but failing to download the script from http://:8000/boot.ipxe. So, let's write one: #!ipxe kernel /files/kernel b2c.container="docker://hello-world" initrd /files/initrd boot This script specifies the following elements: kernel: Download the kernel at http://:8000/files/kernel, and set the kernel command line to ask boot2container to start the hello-world container initrd: Download the initramfs at http://:8000/files/initrd boot: Boot the specified boot configuration Assuming your gateway has an architecture supported by boot2container, you may now download the kernel and initrd from boot2container's releases page. In case it is unsupported, create an issue, or a merge request to add support for it! Now that you have created all the necessary files for the boot, start the web server on your development machine: $ ls boot.ipxe initrd kernel $ python -m http.server 8080 Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ... - - [09/Jan/2022 15:32:52] "GET /boot.ipxe HTTP/1.1" 200 - - - [09/Jan/2022 15:32:56] "GET /kernel HTTP/1.1" 200 - - - [09/Jan/2022 15:32:54] "GET /initrd HTTP/1.1" 200 - If everything went well, the gateway should, after a couple of seconds, start downloading the boot script, then the kernel, and finally the initramfs. Once done, your gateway should boot Linux, run docker's hello-world container, then shut down. Congratulations for netbooting your gateway! However, the current solution has one annoying constraint: it requires a trusted local network and server because we are using HTTP rather than HTTPS... On an untrusted network, a man in the middle could override your boot configuration and take over your CI... If we were using HTTPS, we could download our boot script/kernel/initramfs directly from any public server, even GIT forges, without fear of any man in the middle! Let's try to achieve this! Netbooting from public servers In the previous section, we managed to netboot our gateway from the local network. In this section, we try to improve on it by netbooting using HTTPS. This enables booting from a public server hosted at places such as Linode for $5/month. As I said earlier, iPXE supports HTTPS. However, if you are anyone like me, you may be wondering how such a small bootloader could know which root certificates to trust. The answer is that iPXE generates an SSL certificate at compilation time which is then used to sign all of the root certificates trusted by Mozilla (default), or any amount of certificate you may want. See iPXE's crypto page for more information. WARNING: iPXE currently does not like certificates exceeding 4096 bits. This can be a limiting factor when trying to connect to existing servers. We hope to one day fix this bug, but in the mean time, you may be forced to use a 2048 bits Let's Encrypt certificate on a self-hosted web server. See our issue for more information. WARNING 2: iPXE only supports a limited amount of ciphers. You'll need to make sure they are listed in nginx's ssl_ciphers configuration: AES-128-CBC:AES-256-CBC:AES256-SHA256 and AES128-SHA256:AES256-SHA:AES128-SHA To get started, install NGINX + Let's encrypt on your server, following your favourite tutorial, copy the boot.ipxe, kernel, and initrd files to the root of the web server, then make sure you can download them using your browser. With this done, we just need to edit iPXE's general config C header to enable HTTPS support: $ sed -i 's/#undef\tDOWNLOAD_PROTO_HTTPS/#define\tDOWNLOAD_PROTO_HTTPS/' ipxe/src/config/general.h Then, let's update our boot script to point to the new server: #!ipxe echo Welcome to Valve infra's iPXE boot script :retry echo Acquiring an IP dhcp || goto retry # Keep retrying getting an IP, until we get one echo Got the IP: $${netX/ip} / $${netX/netmask} echo echo Chainloading from the iPXE server... chain https:///boot.ipxe # The boot failed, let's restart! goto retry And finally, let's re-compile iPXE, reflash the gateway pendrive, and boot the gateway! $ make -C ipxe/src -j`nproc` bin/ipxe.iso EMBED= $ sudo dd if=ipxe/src/bin/ipxe.iso of=/dev/sdX bs=1M conv=fsync status=progress If all went well, the gateway should boot and run the hello world container once again! Let's continue our journey by provisioning and backup'ing the local storage of the gateway! Provisioning and backups of the local storage In the previous section, we managed to control the boot configuration of our gateway via a public HTTPS server. In this section, we will improve on that by provisioning and backuping any local file the gateway container may need. Boot2container has a nice feature that enables you to create a volume, and provision it from a bucket in a S3-compatible cloud storage, and sync back any local change. This is done by adding the following arguments to the kernel command line: b2c.minio="s3,${s3_endpoint},${s3_access_key_id},${s3_access_key}": URL and credentials to the S3 service b2c.volume="perm,mirror=s3/${s3_bucket_name},pull_on=pipeline_start,push_on=changes,overwrite,delete": Create a perm podman volume, mirror it from the bucket ${s3_bucket_name} when booting the gateway, then push any local change back to the bucket. Delete or overwrite any existing file when mirroring. b2c.container="-ti -v perm:/mnt/perm docker://alpine": Start an alpine container, and mount the perm container volume to /mnt/perm Pretty, isn't it? Provided that your bucket is configured to save all the revisions of every file, this trick will kill three birds with one stone: initial provisioning, backup, and automatic recovery of the files in case the local disk fails and gets replaced with a new one! The issue is that the boot configuration is currently open for everyone to see, if they know where to look for. This means that anyone could tamper with your local storage or even use your bucket to store their files... Securing the access to the local storage To prevent attackers from stealing our S3 credentials by simply pointing their web browser to the right URL, we can authenticate incoming HTTPS requests by using an SSL client certificate. A different certificate would be embedded in every gateway's iPXE bootloader and checked by NGINX before serving the boot configuration for this precise gateway. By limiting access to a machine's boot configuration to its associated client certificate fingerprint, we even prevent compromised machines from accessing the data of other machines. Additionally, secrets should not be kept in the kernel command line, as any process executed on the gateway could easily gain access to it by reading /proc/cmdline. To address this issue, boot2container has a b2c.extra_args_url argument to source additional parameters from this URL. If this URL is generated every time the gateway is downloading its boot configuration, can be accessed only once, and expires soon after being created, then secrets can be kept private inside boot2container and not be exposed to the containers it starts. Implementing these suggestions in a blog post is a little tricky, so I suggest you check out valve-infra's ipxe-boot-server component for more details. It provides a Makefile that makes it super easy to generate working certificates and create bootable gateway ISOs, a small python-based web service that will serve the right configuration to every gateway (including one-time secrets), and step-by-step instructions to deploy everything! Assuming you decided to use this component and followed the README, you should then configure the gateway in this way: $ pwd /home/ipxe/valve-infra/ipxe-boot-server/files// $ ls boot.ipxe initrd kernel secrets $ cat boot.ipxe #!ipxe kernel /files/kernel b2c.extra_args_url="${secrets_url}" b2c.container="-v perm:/mnt/perm docker://alpine" b2c.ntp_peer=auto b2c.cache_device=auto initrd /files/initrd boot $ cat secrets b2c.minio="bbz,${s3_endpoint},${s3_access_key_id},${s3_access_key}" b2c.volume="perm,mirror=bbz/${s3_bucket_name},pull_on=pipeline_start,push_on=changes,overwrite,delete" And that's it! We finally made it to the end, and created a secure way to provision our CI gateways with the wanted kernel, Operating System, and even local files! When Charlie Turner and I started designing this system, we felt it would be a clean and simple way to solve our problems with our CI gateways, but the implementation ended up being quite a little trickier than the high-level view... especially the SSL certificates! However, the certainty that we can now deploy updates and fix our CI gateways even when they are physically inaccessible from us (provided the hardware and PDU are fine) definitely made it all worth it and made the prospect of having users depending on our systems less scary! Let us know how you feel about it! Conclusion In this post, we focused on provisioning the CI gateway with its boot configuration, and local files via the internet. This drastically reduces the risks that updating the gateway's kernel would result in an extended loss of service, as the kernel configuration can quickly be reverted by changing the boot config files which is served from a cloud service provider. The local file provisioning system also doubles as a backup, and disaster recovery system which will automatically kick in in case of hardware failure thanks to the constant mirroring of the local files with an S3-compatible cloud storage bucket. In the next post, we will be talking about how to create the infra container, and how we can minimize down time during updates by not needing to reboot the gateway. That's all for now, thanks for making it to the end! [Less]
Posted about 2 years ago
This Is A Serious Blog I posted some fun fluff pieces last week to kick off the new year, but now it’s time to get down to brass tacks. Everyone knows adding features is just flipping on the enable button. Now it’s time to see some real work. If ... [More] you don’t like real work, stop reading. Stop right now. Now. Alright, now that all the haters are gone, let’s put on our bisecting snorkels and dive in. Regressions Suck The dream of 2022 was that I’d come back and everything would work exactly how I left it. All the same tests would pass, all the perf would be there, and my driver would compile. I got two of those things, which isn’t too bad. After spending a while bisecting and debugging last week, I categorized a number of regressions to RADV problems which probably only affect me since there’s no Vulkan CTS cases for them (yet). But today I came to the last of the problem cases: dEQP-GLES31.functional.tessellation_geometry_interaction.feedback.tessellation_output_quads_geometry_output_points. There’s nothing too remarkable about the test. It’s XFB, so, according to Jason Ekstrand, future Head of Graphic Wows at Pixar, it’s terrible. What is remarkable, however is that the test passes fine when run in isolation. Here We Go Again Anyone who’s anyone knows what comes next. You find the caselist of the other 499 tests that were run in this block You run the caselist You find out that the test still fails in that caselist You tip your head back to stare at the ceiling and groan Then it’s another X minutes (where X is usually between 5 and 180 depending on test runtimes) to slowly pare down the caselist to the sequence which actually triggers the failure. For those not in the know, this type of failure indicates a pathological driver bug where a sequence of commands triggers different results if tests are run in a different order. There is, to my knowledge, no ‘automatic’ way to determine exactly which tests are required to trigger this type of failure from a caselist. It would be great if there was, and it would save me (and probably others who are similarly unaware) considerable time doing this type of caselist fuzzing. Finally, I was left with this shortened caselist: dEQP-GLES31.functional.shaders.builtin_constants.tessellation_shader.max_tess_evaluation_texture_image_units dEQP-GLES31.functional.tessellation_geometry_interaction.feedback.tessellation_output_quads_geometry_output_points dEQP-GLES31.functional.ubo.random.all_per_block_buffers.25 What Now? Ideally, it would be great to be able to use something like gfxreconstruct for this. I could record two captures—one of the test failing in the caselist and one where it passes in isolation—and then compare them. Here’s an excerpt from that attempt: "[790]vkCreateShaderModule": { "return": "VK_SUCCESS", "device": "0x0x4", "pCreateInfo": { "sType": "VK_STRUCTURE_TYPE_SHADER_MODULE_CREATE_INFO", "pNext": null, "flags": 0, "codeSize": Unhandled VkFormatFeatureFlagBits2KHR, "pCode": "0x0x285c8e0" }, "pAllocator": null, "[out]pShaderModule": "0x0xe0" }, Why is it trying to print an enum value for codeSize you might ask? I’m not the only one to ask, and it’s still an unresolved mystery. I was successful in doing the comparison with gfxreconstruct, but it yielded nothing of interest. Puzzled, I decided to try the test out on lavapipe. Would it pass? No. It similarly fails on llvmpipe and IRIS. But my lavapipe testing revealed an important clue. Given that there are no synchronization issues with lavapipe, this meant I could be certain this was a zink bug. Furthermore, the test failed both when the bug was exhibiting and when it wasn’t, meaning that I could actually see the “passing” values in addition to the failing ones for comparison. Here’s the failing error output: Verifying feedback results. Element at index 0 (tessellation invocation 0) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.166663, 0.5, 1) Element at index 1 (tessellation invocation 1) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.166663, 0.5, 1) Element at index 2 (tessellation invocation 2) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.166663, 0.5, 1) Element at index 3 (tessellation invocation 3) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.433337, 0.5, 1) Element at index 4 (tessellation invocation 4) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.433337, 0.5, 1) Element at index 5 (tessellation invocation 5) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.433337, 0.5, 1) Element at index 6 (tessellation invocation 6) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.4, -0.433337, 0.5, 1) Element at index 7 (tessellation invocation 7) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.4, -0.433337, 0.5, 1) Omitted 24 error(s). And here’s the passing error output: Verifying feedback results. Element at index 3 (tessellation invocation 1) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.433337, 0, 1) Element at index 4 (tessellation invocation 2) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.433337, 0, 1) Element at index 5 (tessellation invocation 3) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.433337, 0, 1) Element at index 6 (tessellation invocation 4) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.4, -0.433337, 0, 1) Element at index 7 (tessellation invocation 5) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.4, -0.433337, 0, 1) Element at index 8 (tessellation invocation 6) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.4, -0.433337, 0, 1) Element at index 9 (tessellation invocation 7) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.433337, 0, 1) Element at index 10 (tessellation invocation 8) expected vertex in range: ( [-0.4, 0.4], [-0.4, 0.4], 0.0, 1.0 ) got: (-0.133337, -0.433337, 0, 1) Omitted 18 error(s). This might not look like much, but to any zinkologists, there’s an immediate red flag: the Z component of the vertex is 0.5 in the failing case. What does this remind us of? Naturally it reminds us of nir_lower_clip_halfz, the compiler pass which converts OpenGL Z coordinate ranges ([-1, 1]) to Vulkan ([0, 1]). This pass is run on the last vertex stage, but if it gets run more than once, a value of -1 becomes 0.5. Thus, it looks like the pass is being run twice in this test. How can this be verified? ZINK_DEBUG=spirv will export all spirv shaders used by an app. Therefore, dumping all the shaders for passing and failing runs should confirm that the conversion pass is being run an extra time when they’re compared. The verdict? @@ -1,7 +1,7 @@ ; SPIR-V ; Version: 1.5 ; Generator: Khronos; 0 -; Bound: 23 +; Bound: 38 ; Schema: 0 OpCapability TransformFeedback OpCapability Shader @@ -36,13 +36,28 @@ %_ptr_Output_v4float = OpTypePointer Output %v4float %gl_Position = OpVariable %_ptr_Output_v4float Output %v4uint = OpTypeVector %uint 4 +%uint_1056964608 = OpConstant %uint 1056964608 %main = OpFunction %void None %3 %18 = OpLabel OpBranch %17 %17 = OpLabel %19 = OpLoad %v4float %a_position %21 = OpBitcast %v4uint %19 - %22 = OpBitcast %v4float %21 - OpStore %gl_Position %22 + %22 = OpCompositeExtract %uint %21 3 + %23 = OpCompositeExtract %uint %21 3 + %24 = OpCompositeExtract %uint %21 2 + %25 = OpBitcast %float %24 + %26 = OpBitcast %float %23 + %27 = OpFAdd %float %25 %26 + %28 = OpBitcast %uint %27 + %30 = OpBitcast %float %28 + %31 = OpBitcast %float %uint_1056964608 + %32 = OpFMul %float %30 %31 + %33 = OpBitcast %uint %32 + %34 = OpCompositeExtract %uint %21 1 + %35 = OpCompositeExtract %uint %21 0 + %36 = OpCompositeConstruct %v4uint %35 %34 %33 %22 + %37 = OpBitcast %v4float %36 + OpStore %gl_Position %37 OpReturn OpFunctionEnd And, as is the rule for such things, the fix was a simple one-liner to unset values in the vertex shader key. It wasn’t technically a regression, but it manifested as such, and fixing it yielded another dozen or so fixes for cases which were affected by the same issue. Blammo. [Less]
Posted about 2 years ago
PyCook A few months ago, I went on a quest to better digitize and collect a bunch of the recipes I use on a regular basis. Like most people, I’ve got a 3-ring binder of stuff I’ve printed from the internet, a box with the usual 4x6 cards, most of ... [More] which are hand-written, and a stack of cookbooks. I wanted something that could be both digital and physical and which would make recipes easier to share. I also wanted whatever storage system I developed to be something stupid simple. If there’s one thing I’ve learned about myself over the years it’s that if I make something too hard, I’ll never get around to it. This led to a few requirements: A simple human-writable file format Recipes organized as individual files in a Git repo A simple tool to convert to a web page and print formats That tool should be able to produce 4x6 cards Enter PyCook. It does pretty much exactly what the above says and not much more. The file format is based on YAML and, IMO, is beautiful to look at all by itself and very easy to remember how to type: name: Swedish Meatballs from: Grandma Ekstrand ingredients: - 1 [lb] Ground beef (80%) - 1/4 [lb] Ground pork - 1 [tsp] Salt - 1 [tsp] Sugar - 1 Egg, beaten - 1 [tbsp] Catsup - 1 Onion, grated - 1 [cup] Bread crumbs - 1 [cup] Milk instructions: - Soak crumbs in milk - Mix all together well - Make into 1/2 -- 3/4 [in] balls and brown in a pan on the stove (you don't need to cook them through; just brown) - Heat in a casserole dish in the oven until cooked through. Over Christmas this year, I got my mom and a couple siblings together to put together a list of family favorite recipes. The format is simple enough that my non-technical mother was able to help type up recipes and they needed very little editing before PyCook would consume them. To me, that’s a pretty good indicator that the file format is simple enough. 😁 The one bit of smarts it does have is around quantities and units. One thing that constantly annoys me with recipes is the inconsistency with abbreviations of units. Take tablespoons, for instance. I’ve seen it abbreviated “T” (as opposed to “t” for teaspoon), “tbsp”, or “tblsp”, sometimes with a “.” after the abbreviation and sometimes not. To handle this, I have a tiny macro language where units have standard abbreviations and are placed in brackets. This is then substituted with the correct abbreviation to let me change them all in one go if I ever want to. It’s also capable of handling plurals properly so when you type [cup] it will turn into either “cup” or “cups” depending on the associated number. It also has smarts to detect “1/4” and turn that into vulgar fraction character “¼” for HTML output or a nice LaTeX fraction when doing PDF output. That brings me to the PDF generator. One of my requirements was the ability to produce 4x6 cards that I could use while cooking instead of having an electronic device near all those fluids. I started with the LaTeX template I got from my friend Steve Schulteis for making a PDF cookbook and adapted them to 4x6 cards, one recipe per card. And the results look pretty nice, I think Meatballs recipe as a 4x6 card It’s even able to nicely paginate the 4x6 cards into a double-sided 8.5x11 PDF which prints onto Avery 5389 templates. Other output formats include an 8.5x11 PDF cookbook with as many recipes per page as will fit and an HTML web version generated using Sphinx which is searchable and has a nice index. P.S. Yes, that’s a real Swedish meatball recipe and it makes way better meatballs than the ones from IKEA. [Less]
Posted about 2 years ago
New blog! This week, I’ve taken advantage of some of my funemployment time to we work our website infrastructure. As part of my new job (more on that soon!) I’ll be communicating publicly more and I need a better blog space. I set up a blogger a few ... [More] years ago but I really hate it. It’s not that Blogger is a terrible platform per se. It just doesn’t integrate well with the rest of my website and the comments I got were 90% spam. At the time, I used Blogger because I didn’t want to mess implementing a blog on my own website infrastructure. Why? The honest answer is an object lesson in software engineering. The last time I re-built my website I thought that building a website generator sounded like a fantastic excuse to learn some Ruby. Why not? It’s a great programming language for web stuff and learning new languages is good, right? While those are both true, software maintainability is important. When I went to try and add a blog, it’d been nearly 4 years since I’d written a line of ruby code and the static site generation framework I was using (nanoc) had moved to a backwards-incompatible new version and I had no clue how to move my site forward without re-learning Ruby and nanoc and rewriting it from scratch. I didn’t have the time for that. This time, I learned my lesson and went all C programmer on it. The new infra is built in Python, a language I use nearly daily in my Mesa work and, instead of using someone else’s static site generation infra, I rolled my own. I’m using Jinja2 for templating as well as Pygments for syntax highlighting and Pandoc for Markdown to HTML conversion. They’re all libraries not frameworks so they take a lot less learning to figure out and remember how to use. Jinja2 is something of a framework (it’s a templating language) but it’s so obvious and user-friendly that it’s basically impossible to forget how to use. Sure, my infra isn’t nearly as nice as some. I don’t have caching and I have to re-build the whole thing every time. I also didn’t bother to implement RSS feed support, comments, or any of those other bloggy features. But, frankly, I don’t care. Even on a crappy raspberry pi, my website builds in a few seconds. As far as RSS goes, who actually uses RSS readers these days? Just follow me on Twitter (@jekstrand_) if you want to know when I blog stuff. Comments? 95% of the comments I got on Blogger were spam anyway. If you want to comment, reply to my tweet. The old blog lived at jason-blog.jlekstrand.net while the new one lives at www.jlekstrand.net/jason/blog/. I’ve set up my webserver to re-direct from the old one and gone though the painful process of checking every single post to ensure the re-directs work. Any links you may have should still work. So there you have it! A new website framework written in all of 2 days. Hopefully, this one will be maintainable and, if I need to extend it, I can without re-learning whole programming languages. I’m also hoping that now that blogging is as easy as writing some markdown and doing git push, I’ll actually blog more. [Less]
Posted about 2 years ago
Chug-a-chug-a-chug-a-chug-a-chug-a It’s a busy week here at SGC. There’s emails to read, tickets to catch up on, rumors to spread about jekstrand’s impending move to Principal Engineer of Bose’s headphone compiler team, code to unwrite. The usual. ... [More] Except now I’m actually around to manage everything instead of ignoring it. Let’s do a brief catchup of today’s work items. Sparse Textures I said this was done yesterday, but the main CTS case for the extension is broken, so I didn’t adequately test it. Fortunately, Qiang Yu from AMD is on the case in addition to doing the original Gallium implementations for these extensions, and I was able to use a WIP patch to fix the test. And run it. And then run it again. And then run it in gdb. And then… And then… Anyway, it all passes now, and sparse texture support is good to go once Australia comes back from vacation to review patches. Also I fixed sparse buffer support, which I accidentally broke 6+ months ago but never noticed since only RADV implements these features and I have no games in my test list that use them. Queries I hate queries. Everyone knows I hate queries. The query code is the worst code in the entire driver. If I never have to open zink_query.c again, I will still have opened it too many times for a single lifetime. But today I hucked myself back in yet again to try and stop a very legitimate and legal replay of a Switch game from crashing. Everyone knows that anime is the real primary driver of all technology, so as soon as anyone files an anime-related ticket, all driver developers drop everything they’re doing to solve it. Unless they’re on vacation. In this case, the problem amounted to: vulkan query pools have a maximum number of queries exceeding this causes a crash trying not to exceed it also causes a crash if the person writing the code is dumb 2021 me was much dumber than 2022 me Rejoice, for you can now play all your weeb games on zink if for some reason that’s where you’re at in your life. But I’m not judging. Source Games: Do More Of Them Run On Gallium Nine In 2022? Yes. I came back to the gift of a new CS:GO version which adds DXVK support, so now there’s also Gallium Nine support. It works fine. Does it work better than other engines? I don’t know, and I have real work to do so I’m not going to test it, but surely someone will take an interest in benchmarking such things now that I’ve heroically git added a 64bit wrapper to my repo that can be used for testing. A quick reminder that all Gallium Nine blog post references and tests happen with RadeonSI. [Less]
Posted about 2 years ago
It appears that Google created a handy tool that helps finding the command which causes a GPU hang/crash. It is called Graphics Flight Recorder (GFR) and was open-sourced a year ago but didn’t receive any attention. From the readme: The Graphics ... [More] Flight Recorder (GFR) is a Vulkan layer to help trackdown and identify the cause of GPU hangs and crashes. It works by instrumenting command buffers with completion tags. When an error is detected a log file containing incomplete command buffers is written. Often the last complete or incomplete commands are responsible for the crash. It requires VK_AMD_buffer_marker support; however, this extension is rather trivial to implement - I had only to copy-paste the code from our vkCmdSetEvent implementation and that was it. Note, at the moment of writing, GFR unconditionally usesVK_AMD_device_coherent_memory, which could be manually patched out for it to run on other GPUs. GFR already helped me to fix hangs in “Alien: Isolation” and “Digital Combat Simulator”. In both cases the hang was in a compute shader and the output from GFR looked like: ... - # Command: id: 6/9 markerValue: 0x000A0006 name: vkCmdBindPipeline state: [SUBMITTED_EXECUTION_COMPLETE] parameters: - # parameter: name: commandBuffer value: 0x000000558CFD2A10 - # parameter: name: pipelineBindPoint value: 1 - # parameter: name: pipeline value: 0x000000558D3D6750 - # Command: id: 6/9 message: '>>>>>>>>>>>>>> LAST COMPLETE COMMAND <<<<<<<<<<<<<<' - # Command: id: 7/9 markerValue: 0x000A0007 name: vkCmdDispatch state: [SUBMITTED_EXECUTION_INCOMPLETE] parameters: - # parameter: name: commandBuffer value: 0x000000558CFD2A10 - # parameter: name: groupCountX value: 5 - # parameter: name: groupCountY value: 1 - # parameter: name: groupCountZ value: 1 internalState: pipeline: vkHandle: 0x000000558D3D6750 bindPoint: compute shaderInfos: - # shaderInfo: stage: cs module: (0x000000558F82B2A0) entry: "main" descriptorSets: - # descriptorSet: index: 0 set: 0x000000558E498728 - # Command: id: 8/9 markerValue: 0x000A0008 name: vkCmdPipelineBarrier state: [SUBMITTED_EXECUTION_NOT_STARTED] ... After confirming that corresponding vkCmdDispatch is indeed the call which hangs, in both cases I made an Amber test which fully simulated the call. For a compute shader, this is relatively easy to do since all you need is to save the decompiled shader and buffers being used by it. Luckily in both cases these Amber tests reproduced the hangs. With standalone reproducers, the problems were much easier to debug, and fixes were made shortly: MR#14044 for “Alien: Isolation” and MR#14110 for “Digital Combat Simulator”. Unfortunately this tool is not a panacea: It likely would fail to help with unrecoverable hangs where it would be impossible to read the completion tags back. Or when the mere addition of the tags could “fix” the issue which may happen with synchronization issues. If draw/dispatch calls run in parallel on the GPU, writing tags may force them to execute sequentially or to be imprecise. Anyway, it’s easy to use so you should give it a try. [Less]
Posted about 2 years ago
We Back The blog is back. I know everyone’s been furiously spamming F5 to see if there were any secret new posts, but no. There were not. Today’s the first day of the new year, so I had to dig deep to remember how to do basic stuff like shitpost on ... [More] IRC. And then someone told me jekstrand was going to Broadcom to work on Windows network drivers? I’m just gonna say it now: 2022 has gone too far. I know it’s early, I know some people are seeing this as a hot take, but I’m throwing the statement down before things get worse. Knock it off, 2022. Zink Somehow the driver is still in the tree, still builds, and still runs. It’s a miracle. Thus, since there were obviously no other matters more pressing than not falling behind on MesaMatrix, I spent the morning figuring out how to implement ARB_sparse_texture. Was this the best decision when I didn’t even remember how to make meson clear its dependency cache? No. No it wasn’t. But I did it anyway because here at SGC, we take bad ideas and turn them into code. Your move, 2022. [Less]
Posted over 2 years ago
Starting with kernel 5.17 the kernel supports the builtin privacy screens built into the LCD panel of some new laptop models.This means that the drm drivers will now return -EPROBE_DEFER from their probe() method on models with a builtin privacy ... [More] screen when the privacy screen provider driver has not been loaded yet.To avoid any regressions distors should modify their initrd generation tools to include privacy screen provider drivers in the initrd (at least on systems with a privacy screen), before 5.17 kernels start showing up in their repos.If this change is not made, then users using a graphical bootsplash (plymouth) will get an extra boot-delay of up to 8 seconds (DeviceTimeout in plymouthd.defaults) before plymouth will show and when using disk-encryption where the LUKS password is requested from the initrd, the system will fallback to text-mode after these 8 seconds.I've written a patch with the necessary changes for dracut, which might be useful as an example for how to deal with this in other initrd generators, see: https://github.com/dracutdevs/dracut/pull/1666I've also filed bugs for tracking this for Fedora, openSUSE, Arch, Debian and Ubuntu. [Less]
Posted over 2 years ago
Introduction One of the big issues I have when working on Turnip driver development is that when compiling either Mesa or VK-GL-CTS it takes a lot of time to complete, no matter how powerful the embedded board is. There are reasons for that: ... [More] typically those board have limited amount of RAM (8 GB for the best case), a slow storage disk (typically UFS 2.1 on-board storage) and CPUs that are not so powerful compared with x86_64 desktop alternatives. Photo of the Qualcomm® Robotics RB3 Platform embedded board that I use for Turnip development. To fix this, it is recommended to do cross-compilation, however installing the development environment for cross-compilation could be cumbersome and prone to errors depending on the toolchain you use. One alternative is to use a distributed compilation system that allows cross-compilation like Icecream. Icecream is a distributed compilation system that is very useful when you have to compile big projects and/or on low-spec machines, while having powerful machines in the local network that can do that job instead. However, it is not perfect: the linking stage is still done in the machine that submits the job, which depending on the available RAM, could be too much for it (however you can alleviate this a bit by using ZRAM for example). One of the features that icecream has over its alternatives is that there is no need to install the same toolchain in all the machines as it is able to share the toolchain among all of them. This is very useful as we will see below in this post. Installation Debian-based systems $ sudo apt install icecc Fedora systems $ sudo dnf install icecream Compile it from sources You can compile it from sources. Configuration of icecc scheduler You need to have an icecc scheduler in the local network that will balance the load among all the available nodes connected to it. It does not matter which machine is the scheduler, you can use any of them as it is quite lightweight. To run the scheduler execute the following command: $ sudo icecc-scheduler Notice that the machine running this command is going to be the scheduler but it will not participate in the compilation process by default unless you ran iceccd daemon as well (see next step). Setup on icecc nodes Launch daemon First you need to run the iceccd daemon as root. This is not needed on Debian-based systems, as its systemd unit is enabled by default. You can do that using systemd in the following way: $ sudo systemctl start iceccd Or you can enable the daemon at startup time: $ sudo systemctl enable iceccd The daemon will connect automatically to the scheduler that is running in the local network. If that’s not the case, or there are more than one scheduler, you can run it standalone and give the scheduler’s IP as parameter: sudo iceccd -s Enable icecc compilation With ccache If you use ccache (recommended option), you just need to add the following in your .bashrc: export CCACHE_PREFIX=icecc Without ccache To use it without ccache, you need to add its path to $PATH envvar so it is picked before the system compilers: export PATH=/usr/lib/icecc/bin:$PATH Execution Same architecture If you followed the previous steps, any time you compile anything on C/C++, it will distribute the work among the fastest nodes in the network. Notice that it will take into account system load, network connection, cores, among other variables, to decide which node will compile the object file. Remember that the linking stage is always done in the machine that submits the job. Different architectures (example cross-compiling for aarch64 on x86_64 nodes) Icemon showing my x86_64 desktop (maxwell) cross-compiling a job for my aarch64 board (rb3). Preparation on x86_64 machine In one x86_64 machine, you need to create a toolchain. This is not automatically done by icecc as you can have different toolchains for cross-compilation. Install cross-compiler For example, you can install the cross-compiler from the distribution repositories: For Debian-based systems: sudo apt install crossbuild-essential-arm64 For Fedora: $ sudo dnf install gcc-aarch64-linux-gnu gcc--c++-aarch64-linux-gnu Create toolchain for icecc Finally, to create the toolchain to share in icecc: $ icecc-create-env --gcc /usr/bin/aarch64-linux-gnu-gcc /usr/bin/aarch64-linux-gnu-g++ This will create a .tar.gz file. The is used to identify the toolchain to distribute among the nodes in case there is more than one. But don’t worry, once it is copied to a node, it won’t be copied again as it detects it is already present. Note: it is important that the toolchain is compatible with the target machine. For example, if my aarch64 board is using Debian 11 Bullseye, it is better if the cross-compilation toolchain is created from a Debian Bullseye x86_64 machine (a VM also works), because you avoid incompatibilities like having different glibc versions. If you have installed Debian 11 Bullseye in your aarch64, you can use my own cross-compilation toolchain for x86_64 and skip this step. Copy the toolchain to the aarch64 machine scp .tar.gz aarch64-machine-hostname: Preparation on aarch64 Once the toolchain (.tar.gz) is copied to the aarch64 machine, you just need to export this on .bashrc: # Icecc setup for crosscompilation export CCACHE_PREFIX=icecc export ICECC_VERSION=x86_64:~/.tar.gz Execute Just compile on aarch64 machine and the jobs be distributed among your x86_64 machines as well. Take into account the jobs will be shared among other aarch64 machines as well if icecc decides so, therefore no need to do any extra step. It is important to remark that the cross-compilation toolchain creation is only needed once, as icecream will copy it on all the x86_64 machines that will execute any job launched by this aarch64 machine. However, you need to copy this toolchain to any aarch64 machines that will use icecream resources for cross-compiling. Icecream monitor This is an interesting graphical tool to see the status of the icecc nodes and the jobs under execution. Install on Debian-based systems $ sudo apt install icecc-monitor Install on Fedora $ sudo dnf install icemon Install it from sources You can compile it from sources. Acknowledgments Even though icecream has a good cross-compilation documentation, it was the post written 8 years ago by my Igalia colleague Víctor Jáquez the one that convinced me to setup icecream as explained in this post. Hope you find this info as useful as I did :-) [Less]
Posted over 2 years ago
On the road to AppStream 1.0, a lot of items from the long todo list have been done so far – only one major feature is remaining, external release descriptions, which is a tricky one to implement and specify. For AppStream 1.0 it needs to be present ... [More] or be rejected though, as it would be a major change in how release data is handled in AppStream. Besides 1.0 preparation work, the recent 0.15 release and the releases before it come with their very own large set of changes, that are worth a look and may be interesting for your application to support. But first, for a change that affects the implementation and not the XML format: 1. Completely rewritten caching code Keeping all AppStream data in memory is expensive, especially if the data is huge (as on Debian and Ubuntu with their large repositories generated from desktop-entry files as well) and if processes using AppStream are long-running. The latter is more and more the case, not only does GNOME Software run in the background, KDE uses AppStream in KRunner and Phosh will use it too for reading form factor information. Therefore, AppStream via libappstream provides an on-disk cache that is memory-mapped, so data is only consuming RAM if we are actually doing anything with it. Previously, AppStream used an LMDB-based cache in the background, with indices for fulltext search and other common search operations. This was a very fast solution, but also came with limitations, LMDB’s maximum key size of 511 bytes became a problem quite often, adjusting the maximum database size (since it has to be set at opening time) was annoyingly tricky, and building dedicated indices for each search operation was very inflexible. In addition to that, the caching code was changed multiple times in the past to allow system-wide metadata to be cached per-user, as some distributions didn’t (want to) build a system-wide cache and therefore ran into performance issues when XML was parsed repeatedly for generation of a temporary cache. In addition to all that, the cache was designed around the concept of “one cache for data from all sources”, which meant that we had to rebuild it entirely if just a small aspect changed, like a MetaInfo file being added to /usr/share/metainfo, which was very inefficient. To shorten a long story, the old caching code was rewritten with the new concepts of caches not necessarily being system-wide and caches existing for more fine-grained groups of files in mind. The new caching code uses Richard Hughes’ excellent libxmlb internally for memory-mapped data storage. Unlike LMDB, libxmlb knows about the XML document model, so queries can be much more powerful and we do not need to build indices manually. The library is also already used by GNOME Software and fwupd for parsing of (refined) AppStream metadata, so it works quite well for that usecase. As a result, search queries via libappstream are now a bit slower (very much depends on the query, roughly 20% on average), but can be mmuch more powerful. The caching code is a lot more robust, which should speed up startup time of applications. And in addition to all of that, the AsPool class has gained a flag to allow it to monitor AppStream source data for changes and refresh the cache fully automatically and transparently in the background. All software written against the previous version of the libappstream library should continue to work with the new caching code, but to make use of some of the new features, software using it may need adjustments. A lot of methods have been deprecated too now. 2. Experimental compose support Compiling MetaInfo and other metadata into AppStream collection metadata, extracting icons, language information, refining data and caching media is an involved process. The appstream-generator tool does this very well for data from Linux distribution sources, but the tool is also pretty “heavyweight” with lots of knobs to adjust, an underlying database and a complex algorithm for icon extraction. Embedding it into other tools via anything else but its command-line API is also not easy (due to D’s GC initialization, and because it was never written with that feature in mind). Sometimes a simpler tool is all you need, so the libappstream-compose library as well as appstreamcli compose are being developed at the moment. The library contains building blocks for developing a tool like appstream-generator while the cli tool allows to simply extract metadata from any directory tree, which can be used by e.g. Flatpak. For this to work well, a lot of appstream-generator‘s D code is translated into plain C, so the implementation stays identical but the language changes. Ultimately, the generator tool will use libappstream-compose for any general data refinement, and only implement things necessary to extract data from the archive of distributions. New applications (e.g. for new bundling systems and other purposes) can then use the same building blocks to implement new data generators similar to appstream-generator with ease, sharing much of the code that would be identical between implementations anyway. 2. Supporting user input controls Want to advertise that your application supports touch input? Keyboard input? Has support for graphics tablets? Gamepads? Sure, nothing is easier than that with the new control relation item and supports relation kind (since 0.12.11 / 0.15.0, details): pointing keyboard touch tablet 3. Defining minimum display size requirements Some applications are unusable below a certain window size, so you do not want to display them in a software center that is running on a device with a small screen, like a phone. In order to encode this information in a flexible way, AppStream now contains a display_length relation item to require or recommend a minimum (or maximum) display size that the described GUI application can work with. For example: 360 This will make the application require a display length greater or equal to 300 logical pixels. A logical pixel (also device independent pixel) is the amount of pixels that the application can draw in one direction. Since screens, especially phone screens but also screens on a desktop, can be rotated, the display_length value will be checked against the longest edge of a display by default (by explicitly specifying the shorter edge, this can be changed). This feature is available since 0.13.0, details. See also Tobias Bernard’s blog entry on this topic. 4. Tags This is a feature that was originally requested for the LVFS/fwupd, but one of the great things about AppStream is that we can take very project-specific ideas and generalize them so something comes out of them that is useful for many. The new tags tag allows people to tag components with an arbitrary namespaced string. This can be useful for project-internal organization of applications, as well as to convey certain additional properties to a software center, e.g. an application could mark itself as “featured” in a specific software center only. Metadata generators may also add their own tags to components to improve organization. AppStream gives no recommendations as to how these tags are to be interpreted except for them being a strictly optional feature. So any meaning is something clients and metadata authors need to negotiate. It therefore is a more specialized usecase of the already existing custom tag, and I expect it to be primarily useful within larger organizations that produce a lot of software components that need sorting. For example: vendor-2021q1 featured This feature is available since 0.15.0, details. 5. MetaInfo Creator changes The MetaInfo Creator (source) tool is a very simple web application that provides you with a form to fill out and will then generate MetaInfo XML to add to your project after you have answered all of its questions. It is an easy way for developers to add the required metadata without having to read the specification or any guides at all. Recently, I added support for the new control and display_length tags, resolved a few minor issues and also added a button to instantly copy the generated output to clipboard so people can paste it into their project. If you want to create a new MetaInfo file, this tool is the best way to do it! The creator tool will also not transfer any data out of your webbrowser, it is strictly a client-side application. And that is about it for the most notable changes in AppStream land! Of course there is a lot more, additional tags for the LVFS and content rating have been added, lots of bugs have been squashed, the documentation has been refined a lot and the library has gained a lot of new API to make building software centers easier. Still, there is a lot to do and quite a few open feature requests too. Onwards to 1.0! [Less]