I Use This!
Very High Activity

News

Analyzed about 22 hours ago. based on code collected 2 days ago.
Posted over 3 years ago by Daniel Stenberg
I write and prefer code that fits within 80 columns in curl and other projects – and there are reasons for it. I’m a little bored by the people who respond and say that they have 400 inch monitors already and they can use them. I too have multiple ... [More] large high resolution screens – but writing wide code is still a bad idea! So I decided I’ll write down my reasoning once and for all! Narrower is easier to read There’s a reason newspapers and magazines have used narrow texts for centuries and in fact even books aren’t using long lines. For most humans, it is simply easier on the eyes and brain to read texts that aren’t using really long lines. This has been known for a very long time. Easy-to-read code is easier to follow and understand which leads to fewer bugs and faster debugging. Side-by-side works better I never run windows full sized on my screens for anything except watching movies. I frequently have two or more editor windows next to each other, sometimes also with one or two extra terminal/debugger windows next to those. To make this feasible and still have the code readable, it needs to fit “wrapless” in those windows. Sometimes reading a code diff is easier side-by-side and then too it is important that the two can fit next to each other nicely. Better diffs Having code grow vertically rather than horizontally is beneficial for diff, git and other tools that work on changes to files. It reduces the risk of merge conflicts and it makes the merge conflicts that still happen easier to deal with. It encourages shorter names A side effect by strictly not allowing anything beyond column 80 is that it becomes really hard to use those terribly annoying 30+ letters java-style names on functions and identifiers. A function name, and especially local ones, should be short. Having long names make them really hard to read and makes it really hard to spot the difference between the other functions with similarly long names where just a sub-word within is changed. I know especially Java people object to this as they’re trained in a different culture and say that a method name should rather include a lot of details of the functionality “to help the user”, but to me that’s a weak argument as all non-trivial functions will have more functionality than what can be expressed in the name and thus the user needs to know how the function works anyway. I don’t mean 2-letter names. I mean long enough to make sense but not be ridiculous lengths. Usually within 15 letters or so. Just a few spaces per indent level To make this work, and yet allow a few indent levels, the code basically have to have small indent-levels, so I prefer to have it set to two spaces per level. Many indent levels is wrong anyway If you do a lot of indent levels it gets really hard to write code that still fits within the 80 column limit. That’s a subtle way to suggest that you should not write functions that needs or uses that many indent levels. It should then rather be split out into multiple smaller functions, where then each function won’t need that many levels! Why exactly 80? Once upon the time it was of course because terminals had that limit and these days the exact number 80 is not a must. I just happen to think that the limit has worked fine in the past and I haven’t found any compelling reason to change it since. It also has to be a hard and fixed limit as if we allow a few places to go beyond the limit we end up on a slippery slope and code slowly grow wider over time – I’ve seen it happen in many projects with “soft enforcement” on code column limits. Enforced by a tool In curl, we have ‘checksrc’ which will yell errors at any user trying to build code with a too long line present. This is good because then we don’t have to “waste” human efforts to point this out to contributors who offer pull requests. The tool will point out such mistakes with ruthless accuracy. Credits Image by piotr kurpaska from Pixabay [Less]
Posted over 3 years ago
Since Firefox Nightly is now using CRLite to determine if enrolled websites’ certificates are revoked, it’s useful to dig into the data to answer why a given certificate issuer gets enrolled or not. Ultimately this is a matter of whether the CRLs ... [More] for a given issuer are available to CRLite, and are valid, but the Internet is a messy place, and sometimes things don’t work as planned. If an issuing CA is not enrolled in CRLite, the Mozilla infrastructure emits enough information to figure out what went wrong. Digging into CRLite’s Data Sets Mozilla’s infrastructure publishes not only the filters and stashes that define the CRLite state-of-the-WebPKI, it also publishes all the intermediate datasets that can be used to both reproduce the filters, and validate them. I have a tool, crlite-status, which barely sticks a toe into the water of these datasets. crlite-status is distributed as a Python package that can be installed via a simple invocation of pip: pip install crlite-status It downloads data directly from the Google Cloud Storage buckets for the production and staging environments, and parses out some of the useful statistics. For a simple example, asking for the last four runs out of the default production environment: Four recent CRLite runs It gets more exciting when enabling CRL auditing, as CRLs dermine whether an issuer is “enrolled” in CRLite or not: crlite-status showing not-enrolled issuers For just the most recent run, we can see which issuers were excluded from CRLite due to CRL issues, marked with a red X. There are actually six of them. Even more details to trace this information is available with the --crl-details command line option. I’ve posted one output from 20201126-0 as an example. Using that 20201126-0, one can see that all three CertCloud issuers are not enrolled as their CRLs are returning a status 403 Forbidden, while the Actalis URL is giving a 404 Not Found. TeleSec’s CRL is hosted at a domain that is down entirely. Starfield and Secom both had recoverable warnings, though the Starfield root failed to recover and details would have to be harvested from the crls and logs directory, while the Secom issuer remained enrolled on the basis of the already-cached CRL still being valid. More to do Lots of the CRLite CRL auditing is manual, still. Since the whole state of the generation task gets uploaded to the Google Cloud Filestore, it’s always possible to determine what state existed for CRLite at the time it made its enrollment decisions, certainly better tools would make it a simpler process. [Less]
Posted over 3 years ago by The Rustup Working Group
The rustup working group is happy to announce the release of rustup version 1.23.0. Rustup is the recommended tool to install Rust, a programming language that is empowering everyone to build reliable and efficient software. If you have a previous ... [More] version of rustup installed, getting rustup 1.23.0 is as easy as closing your IDE and running: rustup self update Rustup will also automatically update itself at the end of a normal toolchain update: rustup update If you don't have it already, you can get rustup from the appropriate page on our website. What's new in rustup 1.23.0 Support for Apple M1 devices Rustup is now natively available for the new Apple M1 devices, allowing you to install it on the new Macs the same way you'd install it on other platforms! Note that at the time of writing this blog post the aarch64-apple-darwin compiler is at Tier 2 target: precompiled binaries are available starting from Rust 1.49 (currently in the beta channel), but no automated tests are executed on them. You can follow issue #73908 to track the work needed to bring Apple Silicon support to Tier 1. Support for installing minor releases The Rust team releases a new version every six weeks, bringing new features and bugfixes on a regular basis. Sometimes a regression slips into a stable release, and the team releases a "point release" containing fixes for that regression. For example, 1.45.1 and 1.45.2 were point releases of Rust 1.45.0, while 1.46.0 and 1.47.0 both had no point releases. With rustup 1.22.1 or earlier if you wanted to use a stable release you were able to either install stable (which automatically updates to the latest one) or a specific version number, such as 1.48.0, 1.45.0 or 1.45.2. Starting from this release of rustup (1.23.0) you can also install a minor version without specifying the patch version, like 1.48 or 1.45. These "virtual" releases will always point to the latest patch release of that cycle, so rustup toolchain install 1.45 will get you a 1.45.2 toolchain. New format for rust-toolchain The rustup 1.5.0 release introduced the rust-toolchain file, allowing you to choose the default toolchain for a project. When the file is present rustup ensures the toolchain specified in it is installed on the local system, and it will use that version when calling rustc or cargo: $ cat rust-toolchain nightly-2020-07-10 $ cargo --version cargo 1.46.0-nightly (fede83ccf 2020-07-02) The file works great for projects wanting to use a specific nightly version, but didn't allow to add extra components (like clippy) or compilation targets. Rustup 1.23.0 introduces a new, optional TOML syntax for the file, with support for specifying components and targets: [toolchain] channel = "nightly-2020-07-10" components = ["rustfmt", "clippy"] targets = ["wasm32-unknown-unknown"] The new syntax doesn't replace the old one, and both will continue to work. You can learn more about overriding the default toolchain in the "Overrides" chapter of the rustup book. Other changes There are more changes in rustup 1.23.0: check them out in the changelog! Rustup's documentation is also available in the rustup book starting from this release. Thanks Thanks to all the contributors who made rustup 1.23.0 possible! Aaron Loucks Aleksey Kladov Aurelia Dolo Camelid Chansuke Carol (Nichols || Goulding) Daniel Silverstone Dany Marcoux Eduard Miller Eduardo Broto Eric Huss Francesco Zardi FR Bimo Ivan Nejgebauer Ivan Tham Jake Goulding Jens Reidel Joshua M. Clulow Joshua Nelson Jubilee Young Leigh McCulloch Lzu Tao Matthias Krüger Matt Kraai Matt McKay Nick Ashley Pascal Hertleif Paul Lange Pietro Albini Robert Collins Stephen Muss Tom Eccles [Less]
Posted over 3 years ago by [email protected] (ClassicHasClass)
TenFourFox Feature Parity Release 30 beta 1 is now available (downloads, hashes, release notes). I managed to make some good progress on backporting later improvements to the network and URL handling code, so there are no UI-facing features in this ... [More] release, but the browser should use a bit less memory and run a little quicker especially on pages that reference a lot of resources (which, admittedly, is a lot of sites these days). There is also a minor update to the host database for basic adblock. Assuming all goes well, this release will come out parallel with Firefox 84 on or around December 15. I'll probably do an SPR-only build for the release immediately following to give myself a break; this will contain just needed security fixes, and there will most likely not be a beta. A few people got bitten by not noticing the locale update, so let me remind everyone that FPR29 needs new locales if you are using a custom langpack. They're linked from the main TenFourFox page and all of them are on SourceForge except for the separately-maintained Japanese version, which I noticed has also been updated to FPR29. If you get a weird error starting TenFourFox and you have a langpack installed, quit the browser and run the new langpack installer and it should fix itself. Finally, in case you missed it, with the right browser and a side-car TLS 1.2 proxy, you can get A/UX, Power MachTen (on any classic MacOS supporting it) and pre-Tiger Mac OS X able to access modern web pages again. The key advance here is that the same machine can also run the proxy all by itself: no cheating with a second system! Sadly, this does not work as-is with all browsers, including with Classilla, which is something I'll think about allowing as a down payment on proper in-browser support at some future date. [Less]
Posted over 3 years ago
Firefox Nightly is now using CRLite to determine if websites’ certificates are revoked — e.g., if the Certificate Authority published that web browsers shouldn’t trust that website certificate. Telemetry shows that querying the local CRLite dataset ... [More] is much faster than making a network connection for OCSP, which makes intuitive sense. It also avoids sending the website’s certificate information in cleartext over the network to check the revocation status: solving one of the remaining cleartext browsing data leakages in Firefox. Mozilla is currently publishing CRLite data to Remote Settings four times per day, keeping a very fresh set of revocation information for the public Web. I’ve provided some direct details on how to get at that data from the CRLite FAQ, and I want to introduce one of my command-line tools I’ve used to analyze and play with the dataset: moz_crlite_query. I’ll introduce crlite_status in a later post. Querying CRLite Status at the CLI moz_crlite_query is a Python tool that more-or-less implements the same algorithm Firefox does to check a website’s certificate against CRLite. It maintains a local database of the CRLite dataset in your ~/.crlite_db/ directory, which for me right now looks like: 0 Nov 25 09:24 .last_updated 5381898 Nov 25 09:24 2020-11-24T00:08:12+00:00Z-full 27363 Nov 25 09:24 2020-11-24T06:08:12+00:00Z-diff 59196 Nov 25 09:24 2020-11-24T12:08:14+00:00Z-diff 56121 Nov 25 09:24 2020-11-24T18:08:15+00:00Z-diff 204863 Nov 25 09:24 2020-11-25T00:08:23+00:00Z-diff 25826 Nov 25 09:24 2020-11-25T06:08:05+00:00Z-diff 66626 Nov 25 09:24 2020-11-25T12:08:22+00:00Z-diff 25826 Nov 25 09:24 2020-11-25T06:08:05+00:00Z-diff 66626 Nov 25 09:24 2020-11-25T12:08:22+00:00Z-diff 53783 Nov 25 22:19 2020-11-25T18:08:11+00:00Z-diff 190353 Nov 25 22:19 2020-11-26T00:08:11+00:00Z-diff 1118208 Nov 25 22:19 intermediates.sqlite You can see here that the last full CRLite filter update happened at the midnight-UTC update on 24 November 2020, and since then there have been eight delta updates (e.g., “stashes”). moz_crlite_query is distributed as a Python package that can be installed via an invocation of pip (Note that this tool requires Python 3.7 or newer because of SQLite features.): pip install moz_crlite_query On invocation, the tool will try and update to the latest CRLite data, unless it already updated in the last several hours. You can query the CRLite status for either a certificate you already have, or the tool can connect to one or more websites and examine the certificates presented there. Experimenting with the Query Tool Let’s download a few certificates into /tmp from crt.sh: 77575263, 1988442812, 1485147627, and 2680822568. These simply are representative – right now – of various states in the Web PKI. ○ → for id in 77575263 1988442812 1485147627 2680822568; do > curl --silent https://crt.sh/?d=${id} > /tmp/${id}.pem > done Now, run them all through CRLite: ○ → moz_crlite_query /tmp/*.pem INFO:crlite_query:CRLite Update: Syncing CRLite filters. 100% (9 of 9) |######################################################| Elapsed Time: 0:00:00 Time: 0:00:00 INFO:query_cli:Status: 2457 Intermediates, Current filter: 2020-11-24T00:08:12+00:00Z-full with 27 layers and 43053008 bit-count, 8 stash files with 43022 stashed revocations, up-to-date as of 2020-11-26 00:08:11+00:00 (5:11:15.467645 ago). /tmp/1485147627.pem Issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US Enrolled in CRLite: ❌ Result: ❔ Not Enrolled ❔ /tmp/1988442812.pem Issuer: CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US Enrolled in CRLite: ✅ Revoked via CRLite filter: 2020-11-24T00:08:12+00:00Z-full Result: ⛔️ Revoked ⛔️ /tmp/2680822568.pem Issuer: CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US Enrolled in CRLite: ✅ Result: 👍 Valid 👍 /tmp/77575263.pem Issuer: CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US Enrolled in CRLite: ✅ Result: ⏰ Expired ⏰ We can see 1485147627 is issued by a CA not participating in CRLite: Let’s Encrypt doesn’t publish CRLs. 1988442812 is for revoked.badssl.com, and is revoked. 2680822568 is currently a certificate used for many Mozilla properties, like firefox.com and taskcluster.net. Finally, 77575263 is an expired certificate for blog.mozilla.com (and some others). We can also supply hosts, either on the command line, as an input file, or both: ○ → cat >/tmp/top4.txt < This is all in good fun, but there’s more possibility here than just a command-line gimmick. Embedding CRLite into Other Tools? Many tools use the Mozilla Root CA program’s list of trust anchors for their trust decisions, particularly on Linux or BSD operating systems. For many such tools, certificate revocations are sufficiently unlikely and the consequences sufficiently low that checking revocation status is simply unimportant. For those tools that do check, leaking the status check in cleartext to network observers is likely to simply be evidence of doing a software update. However, I can imagine that some tools might benefit from CRLite’s infrastructure-as-a-service, and make good use of the privacy benefit. As it’s not particularly difficult to develop a CRLite client, perhaps if this tooling makes it to Firefox Release, we’ll see some libraries for non-Firefox users of CRLite. While Mozilla certainly isn’t intending to have numerous users of the data beyond Firefox — at least they aren’t accounted for in the budget — the data is there, and there to be useful. [Less]
Posted over 3 years ago by [email protected] (ClassicHasClass)
LTO-PGO is still working great in Firefox 83, which expands in-browser PDF support, adds additional features to Picture-in-Picture (which is still one of my favourite tools in Firefox) and some miscellany developer changes. The exact same process ... [More] , configs and patches to build a fully link-time and profile-guided optimized build work that were used in Firefox 82. Dan Horák has filed three bugs (1679271, 1679272 and 1679273) for build failures related to the internal profiler, which is still not supported on ppc64, ppc64le or s390x (or, for that matter, 32-bit PowerPC). These targetted fixes should be landing well before release, but perhaps we should be thinking about how to get it working on OpenPOWER rather than having to play emergency games of whack-a-mole whenever the build blows up. [Less]
Posted over 3 years ago by TWiR Contributors
Hello and welcome to another issue of This Week in Rust! Rust is a systems language pursuing the trifecta: safety, concurrency, and speed. This is a weekly summary of its progress and community. Want something mentioned? Tweet us at @ThisWeekInRust ... [More] or send us a pull request. Want to get involved? We love contributions. This Week in Rust is openly developed on GitHub. If you find any errors in this week's issue, please submit a PR. Updates from Rust Community Official [Inside] What the Error Handling Project Group is Working On Newsletters Tooling Rust Analyzer Changelog #52 Knurling-rs Changelog #7 Observations/Thoughts FlatBuffer as serialization agnostic IDL WTF is Rust? The Illustrated Notes First Lines of Rust Optimizing Benchpress pwintln uwu and other fun with elves and dynamic linkers Rust Walkthroughs Hands-on Rust: Effective Learning through 2D Game Development and Play CBOR IoT Data Serialization for Embedded C and Rust Basic Interactions with Yew Testing your crate C-API Rocket Tutorial 03 part II: Proper Testing Select Syscall in Rust FBSim: football-playing AI agents in Rust Building a Recipe Manager - Part 5 - Data Integrity Bootstrapping support for the STM32WLE with the Embedded Rust ecosystem Recipe for Calling Swift Closures from Asynchronous Rust Code [video] A Cool Generic Concurrency Primitive in Rust [video] Creative coding in Rust: re-creating a retro screensaver [video] (Live Coding) Audio adventures in Rust: Packaging Actix + React app as macOS bundle [video] Prototype (Rust tutorial) - Design Patterns Project Updates cargo-mobile: Rust on mobile made easy! Welcoming the RustFest Project Levenshtein Heuristic in Poi Miscellaneous Strengthening memory safety in Rust: exploring CHERI capabilities for a safe language Why AWS loves Rust, and how we'd like to help Why Work in Blockchain? - Journey from C++ to Rust Developer Flash Animations Live Forever at the Internet Archive [via ruffle.rs, a Flash emulator written in Rust compiled to WASM] [video] How to create an awesome Rust GitHub project Crate of the Week This week's crate is cargo-intraconv, a cargo subcommand to convert links in rust documentation to the newly stable intra-doc-links format. Thanks to Alexis Bourget for the suggestion! Submit your suggestions and votes for next week! Call for Participation Always wanted to contribute to open-source projects but didn't know where to start? Every week we highlight some tasks from the Rust community for you to pick and get started! Some of these tasks may also have mentors available, visit the task page for more information. oapth - Meta issue for clean method uom - Fix issues with uom CI process If you are a Rust project owner and are looking for contributors, please submit tasks here. Updates from Rust Core 345 pull requests were merged in the last week never inline naked functions fix exhaustiveness in case a byte string literal is used at slice type Arena: use specialization to avoid copying data add column number support to Backtrace add support for custom allocators in Vec change slice::to_vec to not use extend_from_slice tighten the bounds on atomic Ordering in std::sys::unix::weak::Weak Add #[cold] attribute to std::process::abort and alloc::alloc::handle_alloc_error impl Default for PhantomPinned add trailing_zeros and leading_zeros to non zero types add f{32, 64}::is_subnormal add core::slice::fill_with implement Index and IndexMut for arrays make as{_mut,}_slice on array::IntoIter public stabilize refcell_take stabilize clamp stabilise then stabilize IpAddr::is_ipv4 and is_ipv6 as const stabilize alloc::Layout const functions futures: stream: unzip operator cargo: allow resolver="1" to explicitly use the old resolver behavior rustdoc: give a better error when rustdoc tests fail semverver: speed compilation by using .rmeta over .rlib files measureme: hardware performance counter support (via rdpmc) Rust Compiler Performance Triage 2020-11-24: 1 Regression, 2 Improvements, 2 mixed This week saw landing of #79237 which by itself provides no wins but opens the door to support for split debuginfo on macOS. This'll eventually show huge wins as we can likely avoid re-collecting debuginfo while retaining support for lldb and Rust backtraces. #79361 tracks the stabilization of the rustc flag, but the precise rollout to stable users is not yet 100% clear. Triage done by @jyn514 and @simulacrum. 4 regressions, 4 improvements, 2 mixed results. 5 of them in rollups. See the full report for more. Approved RFCs Changes to Rust follow the Rust RFC (request for comments) process. These are the RFCs that were approved for implementation this week: No RFCs were approved this week. Final Comment Period Every week the team announces the 'final comment period' for RFCs and key PRs which are reaching a decision. Express your opinions now. RFCs No RFCs are currently in the final comment period. Tracking Issues & PRs No Tracking Issues or PRs are currently in the final comment period. New RFCs Add a new syntax to declare that a trait must always be object-safe Upcoming Events Online November 26, Edinburgh, UK - Rust in the Polymesh Project - Rust Edinburgh November 26, Berlin, DE - Rust Hack and Learn - Berline.rs November 26, Tel Aviv-Yafo, IL - Rust Machine Learning On-line Meetup - ODSC Tel Aviv Data Science December 1, Buffalo, NY, US - Buffalo Rust User Group - Buffalo Rust Meetup December 1, Munich, DE - Rust Remote #4 (CEST) - Rust Munich Meetup December 2, Johannesburg, ZA - Monthly Joburg Rust Chat - Johannesburg Rust Meetup December 2, Indianapolis, IN, US - Indy.rs - with Social Distancing - Indy Rust December 8, Saarbücken, Saarland, DE - Meetup: 6u16 (virtual) - Rust Saar North America December 9, Atlanta, GA, US - Grab a beer with fellow Rustaceans - Rust Atlanta Asia Pacific December 7, Auckland, NZ - Rust AKL - Show and Tell + Introduction to RUst II If you are running a Rust event please add it to the calendar to get it mentioned here. Please remember to add a link to the event too. Email the Rust Community Team for access. Rust Jobs Senior Software Engineer, IOx at InfluxData (Remote) Tweet us at @ThisWeekInRust to get your job offers listed here! Quote of the Week I know noting about the compiler internals but it looks to me as if 90% of the time is spent pretty-printing LayoutError. – Vadzim Dambrouski on github Thanks to mmmmib for the suggestion. Please submit quotes and vote for next week! This Week in Rust is edited by: nellshamrell, llogiq, and cdmistman. Discuss on r/rust [Less]
Posted over 3 years ago by allen
This morning I saw an interesting twitter thread: Sorry, this makes no sense. Most valuable architectural depictions are high-level and abstract and can't be auto-generated from code, for example; hence they're manually created.— Andrei ... [More] Pacurariu (@AndreiPacurariu) November 23, 2020 Tudor is the driving-force behind gtoolkit.com, a modern and innovative programming environment that builds upon and extends classic Smalltalk development concepts. Tudor is someone worth listening to, but his tweet was something that I really couldn’t agree with. I found myself more in agreement with Andrei Pacurariu’s reply. I started to tweet a response but quickly found that my thoughts were too complex to make a good tweet stream. Instead, I wrote the following mini essay. Concretely, software is just bits in electronic storage that control and/or are manipulated by processors.  Abstractions are the building blocks that enable humans to design and build complex software systems out of bits.  Abstractions are products of out minds—they allow us to assign meaning to clusters (some large, some small) of bits. They allow us to build software systems without thinking about billions of bits or how processors work. We manifest some useful and generally simple abstractions (instructions, statements, functions, classes, modules, etc.) as “code” using other abstractions we call “languages.”  Languages give us a common vocabulary for us to communicate about those abstract building blocks and to produce the corresponding bits.  There are many useful tools that can and should be created to help us understand the code-level operation of a system. But most systems we build today are too complex to be fully understood at the level of code. In designing them we must use higher-level abstractions to conceptualize, compose, and organize code. Abstract machines, frameworks, patterns, roles, stereotypes, heuristics, constraints, etc. are examples of such higher-level abstractions. The languages we commonly use provide few, if any, mechanisms for directly identifying such higher-level abstractions.  These abstractions may manifest as naming or other coding conventions but recognizing them as such depends upon a pre-existing shared understanding between the writer and readers of the code. If such conventions have been adequately formalized and followed, a tool may be able to assist a human identify and display the usage of higher-level abstractions within a code base. But traditionally, such tools are rare and often unreliable.  One problem is that abstractions can overlap. So we (or the tools) not only need to identify and classify abstractions but also identify the clustering and organization of abstractions. Sometimes this results in multiple views that can provide totally different conceptions of the operation of the code we are looking at. Software designers/architects often use informal diagrams to capture their conceptualization of the structure and interactions of the higher-level abstractions of their designs. Such diagrams can serve as maps that help developers understand the territory of the code.  But developers are often skeptical of such diagrams. Experience (and folklore) has taught them that design diagrams likely do not accurately reflect the actual state of a code base.  So, how do we familiarize ourselves with a new software code base that we wish to modify or extend? Most commonly we just try to reverse engineer its architecture/design by examining the code. We tell ourselves that the code provides the ground truth and that we or our tools can generate truthful design docs from the code because the code doesn’t lie. When humans read code we can easily miss or misinterpret the higher-level abstractions that are lurking among and above the code. It’s a real challenge to imagine a tool that can do better. Lacking design documents or even a good sense of a system’s overall design we then often make local code changes without correctly understanding the design context within which that code operates. Such local changes may “work” to solve some immediate problem. But as they accumulate over long periods such changes erode the design integrity of the overall system. And they contribute to the invalidation of any pre-existing design documents and diagrams that may have at one point in time provided an accurate map of the system.  We definitely need better tools for integrating higher-level system architecture/design abstraction with code—or at least for integrating documentation.  But don’t tell me that creating such documents are a waste of time or that any such documents should be ignored.  I’ve spent too many decades trying to guess how a complex program was intended to work by trying to use the code to get into the mind of the original designers and developers.  Any design documentation is useful documentation. It will probably need to be verified against the current state of the code but even when the documents are no longer the truth they often provide insights that help our understanding of the system.  When I asked Rebecca Wirfs-Brock to review this little essay, she had an experience to relate: I remember doing a code/architecture review for a company that built software to perform high reliability backups…and I was happy that the architectural documents were say 70% accurate as they gave me a good look into the mind of the architect who engineered these frameworks. And I didn’t think it worthwhile to update it to reflect truth. As it was a snapshot in time. Sometimes updating may be worthwhile, but as in archeological digs, you’ve gotta be careful about this. Diagrams and other design documents probably won’t be up to date when somebody reads them in the future. But that’s not an excuse to not create them. If you care about the design integrity of you software system you need to provide future developers (including yourself) a map of your abstract design as you currently know it. [Less]
Posted over 3 years ago by Daniel Stenberg
The purpose of the curl web site is to inform the world about what curl and libcurl are and provide as much information as possible about the project, the products and everything related to that. The web site has existed in some form for as long ... [More] as the project has, but it has of course developed and changed over time. Independent The curl project is completely independent and stands free from influence from any parent or umbrella organization or company. It is not even a legal entity, just a bunch of random people cooperating over the Internet. And a bunch of awesome sponsors to help us. This means that we have no one that provides the infrastructure or marketing for us. We need to provide, run and care for our own servers and anything else we think we should offer our users. I still do a lot of the work in curl and the curl web site and I work full time on curl, for wolfSSL. This might of course “taint” my opinions and views on matters, but doesn’t imply ownership or control. I’m sure we’re all colored by where we work and where we are in our lives right now. Contents Most of the web site is static content: generated HTML pages. They are served super-fast and very lightweight by any web server software. The web site source exists in the curl-www repository (hosted on GitHub) and the web site syncs itself with the latest repository changes several times per hour. The parts of the site that aren’t static are mostly consisting of smaller scripts that run either on demand at the time of a request or on an interval in a cronjob in the background. That is part of the reason why pushing an update to the web site’s repository can take a little while until it shows up on the live site. There’s a deliberate effort at not duplicating information so a lot of the web pages you can find on the web site are files that are converted and “HTMLified” from the source code git repository. “Design” Some people say the curl web site is “retro”, others that it is plain ugly. My main focus with the site is to provide and offer all the info, and have it be accurate and accessible. The look and the design of the web site is a constant battle, as nobody who’s involved in editing or polishing the web site is really interested in or particularly good at design, looks or UX. I personally have done most of the editing of it, including CSS etc and I can tell you that I’m not good at it and I don’t enjoy it. I do it because I feel I have to. I get occasional offers to “redesign” the web site, but the general problem is that those offers almost always involve rebuilding the entire thing using some current web framework, not just fixing the looks, layout or hierarchy. By replacing everything like that we’d get a lot of problems to get the existing information in there – and again, the information is more important than the looks. The curl logo is designed by a proper designer however (Adrian Burcea). If you want to help out designing and improving the web site, you’d be most welcome! Who I’ve already touched on it: the web site is mostly available in git so “anyone” can submit issues and pull-requests to improve it, and we are around twenty persons who have push rights that can then make a change on the live site. In reality of course we are not that many who work on the site any ordinary month, or even year. During the last twelve month period, 10 persons authored commits in the web repository and I did 90% of those. How Technically, we build the site with traditional makefiles and we generate the web contents mostly by preprocessing files using a C-like preprocessor called fcpp. This is an old and rather crude setup that we’ve used for over twenty years but it’s functional and it allows us to have a mostly static web site that is also fairly easy to build locally so that we can work out and check improvements before we push them to the git repository and then out to the world. The web site is of course only available over HTTPS. Hosting The curl web site is hosted on an origin VPS server in Sweden. The machine is maintained by primarily by me and is paid for by Haxx. The exact hosting is not terribly important because users don’t really interact with our server directly… (Also, as they’re not sponsors we’re just ordinary customers so I won’t mention their name here.) CDN A few years ago we experienced repeated server outages simply because our own infrastructure did not handle the load very well, and in particular not the traffic spikes that could occur when I would post a blog post that would suddenly reach a wide audience. Enter Fastly. Now, when you go to curl.se (or daniel.haxx.se) you don’t actually reach the origin server we admin, you will instead reach one of Fastly’s servers that are distributed across the world. They then fetch the web contents from our origin, cache it on their edge servers and send it to you when you browse the site. This way, your client speaks to a server that is likely (much) closer to you than the origin server is and you’ll get the content faster and experience a “snappier” web site. And our server only gets a tiny fraction of the load. Technically, this is achieved by the name curl.se resolving to a number of IP addresses that are anycasted. Right now, that’s 4 IPv4 addresses and 4 IPv6 addresses. The fact that the CDN servers cache content “a while” is another explanation to why updated contents take a little while to “take effect” for all visitors. DNS When we just recently switched the site over to curl.se, we also adjusted how we handle DNS. I run our own main DNS server where I control and admin the zone and the contents of it. We then have four secondary servers to help us really up our reliability. Out of those four secondaries, three are sponsored by Kirei and are anycasted. They should be both fast and reliable for most of the world. With the help of fabulous friends like Fastly and Kirei, we hope that the curl web site and services shall remain stable and available. DNS enthusiasts have remarked that we don’t do DNSSEC or registry-lock on the curl.se domain. I think we have reason to consider and possibly remedy that going forward. Traffic The curl web site is just the home of our little open source project. Most users out there in the world who run and use curl or libcurl will not download it from us. Most curl users get their software installation from their Linux distribution or operating system provider. The git repository and all issues and pull-requests are done on GitHub. Relevant here is that we have no logging and we run no ads or any analytics. We do this for maximum user privacy and partly because of laziness, since handling logging from the CDN system is work. Therefore, I only have aggregated statistics. In this autumn of 2020, over a normal 30 day period, the web site serves almost 11 TB of data to 360 million HTTP requests. The traffic volume is up from 3.5 TB the same time last year. 11 terabytes per 30 days equals about 4 megabytes per second on average. Without logs we cannot know what people are downloading – but we can guess! We know that the CA cert bundle is popular and we also know that in today’s world of containers and CI systems, a lot of things out there will download the same packages repeatedly. Otherwise the web site is mostly consisting of text and very small images. One interesting specific pattern on the server load that’s been going on for months: every morning at 05:30 UTC, the site gets over 50,000 requests within that single minute, during which 10 gigabytes of data is downloaded. The clients are distributed world wide as I see the same pattern on access points all over. The minute before and the minute after, the average traffic rate remains at 200MB/minute. It makes for a fun graph: An eight hour zoomed in window of bytes transferred from the web site. UTC times. Our servers suffer somewhat from being the target of weird clients like qqgamehall that continuously “hammer” the site with requests at a high frequency many months after we started always returning error to them. An effect they have is that they make the admin dashboard to constantly show a very high error rate. Software The origin server runs Debian Linux and Apache httpd. It has a reverse proxy based on nginx. The DNS server is bind. The entire web site is built with free and open source. Primarily: fcpp, make, roffit, perl, curl, hypermail and enscript. If you curl the curl site, you can see in response headers that Fastly uses Varnish. [Less]
Posted over 3 years ago by [email protected] (Robert)
To show off the power of our Pernosco debugger, we wanted many short demo videos of the application interface. Regular videos are relatively heavyweight and lossy; we wanted something more like Asciinema, but for our Web application, not just a ... [More] terminal. So we created DOMRec, a DOM recorder. The approach is surprisingly straightforward. To record a demo, we inject the DOMRec script into our application. DOMRec captures the current DOM state and uses a DOM Mutation Observer to record all DOM state changes with associated timestamps. To replay the demo, we create an iframe, fill it with the recorded initial DOM state, then replay the DOM state changes over time. Replay inserts links to the original stylesheets but doesn't load or execute the original scripts. (Of course it couldn't be quite that simple ... see below.) The resulting demos are compact (most of ours are less than 100KB gzipped), work on almost any browser/device, and are pixel-perfect at any zoom level. DOMRec supports single-frame screenshots when dynamism is not required. We can't use browser HTML5 video controls but we implement play/pause/fullscreen buttons. Capturing DOM state isn't quite enough because some rendering-relevant state isn't in the DOM. DOMRec logs mouse movement and click events and during replay, displays a fake mouse cursor and click effect. DOMRec tracks focus changes, and during replay sets "fake focus" classes on relevant DOM elements; our application style sheets check those classes in addition to the real :focus and :focus-within pseudoclasses. When necessary DOMRec creates a fake caret. DOMRec captures scroll offsets and makes a best-effort attempt to match scroll positions during recording and replay. Canvas drawing operations don't change the DOM and don't trigger events, so our application manually triggers a "didDrawCanvas" DOM event which DOMRec listens for. Sometimes the Pernosco client needs to trigger a style flush to get CSS transitions to work properly, which also needs to happen during replay, so we fire a special notification event for that too. It's a bit ad-hoc — we implemented just enough for needs, and no more — but it does at least handle Microsoft's Monaco editor correctly, which is pretty complicated. DOMRec can record demos executed manually, but in practice our demos are scripted using Selenium so that we can rebuild the demo movie when our application interface has changed. This kind of thing has certainly been built many times — see Inspectlet and its competitors — so when I first looked into this problem a few years ago I assumed I would find an open-source off-the-shelf solution. Unfortunately I couldn't find one that looked easy to consume. Now we're releasing DOMRec with an MIT license. To tell the truth, we're not trying hard to make DOMRec easy to consume, either; as noted above, some applications need to be tweaked to work with it, and this blog post is about all you're going to get in terms of documentation. Still, it's only 1200 lines of standalone Javascript so people may find it easier than starting from scratch. [Less]