Varnish

Posted on 2015-11-24 This is the only chapter written in first person. I've worked on Varnish since late 2008, first for Redpill Linpro, then Varnish Software, then, after a brief pause, for Redpill Linpro again. Over the years I've written code ... [More] , written Varnish modules and blog posts, tried to push the boundaries of what Varnish can do, debugged or analyzed countless Varnish sites, probably held more training courses than anyone else, written training material, and helped shape the Varnish community. Today I find myself in a position where the training material I once maintained is no longer my responsibility. But I still love writing, and there's an obvious need for documentation for Varnish. I came up with a simple solution: I will write a book. Because I couldn't imagine that I would ever finish it if I attempted writing a whole book in one go, I decided I would publish one chapter at a time on my blog. This is the first chapter of that book. You will find the source on https://github.com/KristianLyng/varnishfoo. This is something I am doing on my spare time, and I hope to get help from the Varnish community in the form of feedback. While the format will be that of a book, I intend to keep it alive with updates as long as I can. I intend to cover as much Varnish-related content as possible, from administration to web development and infrastructure. And my hope is that one day, this will be good enough that it will be worth printing as more than just a leaflet. I am writing this in my spare time, I retain full ownership of the material. For now, the material is available under a Creative Commons "CC-BY-SA-NC" license. The NC-part of that license will be removed when I feel the material has matured enough and the time is right. To clarify, the "non-commercial" clause is aimed at people wanting to sell the book or use it in commercial training (or similar) - it is not intended to prevent you from reading the material at work. Target audience and format This book covers a large spectre of subjects related to Varnish. It is suitable for system administrators, infrastructure architects and web developers. The first few chapters is general enough to be of interest to all, while later chapters specialize on certain aspects of Varnish usage. Each chapter is intended to stand well on its own, but there will be some cross-references. The book focuses on best practices and good habits that will help you beyond what just a few examples or explanations will do. Each chapter provides both theory and practical examples. Each example is tested with a recent Varnish Version where relevant, and are based on experience from real-world Varnish installations. What is Varnish Varnish is a web server. Unlike most web servers, Varnish does not read content from a hard drive, or run programs that generates content from SQL databases. Varnish acquires the content from other web servers. Usually it will keep a copy of that content around in memory for a while to avoid fetching the same content multiple times, but not necessarily. There are numerous reasons you might want Varnish: Your web server/application is a beastly nightmare where performance is measured in page views per hour - on a good day. Your content needs to be available from multiple geographically diverse locations. Your web site consists of numerous different little parts that you need to glue together in a sensible manner. Your boss bought a service subscription and now has to justify the budget post. You like Varnish. ??? Varnish is designed around two simple concepts: Give you the means to fix or work around technical challenges. And speed. Speed was largely handled very early on, and Varnish is quite simply fast. This is achieved by being, at the core, simple. The less you have to do for each request, the more requests you can handle. The name suggests what it's all about: From The Collaborative International Dictionary of English v.0.48 [gcide]: Varnish \Var"nish\, v. t. [imp. & p. p. {Varnished}; p. pr. & vb. n. {Varnishing}.] [Cf. F. vernir, vernisser. See {Varnish}, n.] [1913 Webster] 1. To lay varnish on; to cover with a liquid which produces, when dry, a hard, glossy surface; as, to varnish a table; to varnish a painting. [1913 Webster] 2. To cover or conceal with something that gives a fair appearance; to give a fair coloring to by words; to gloss over; to palliate; as, to varnish guilt. "Beauty doth varnish age." --Shak. [1913 Webster] Varnish can be used to smooth over rough edges in your stack, to give a fair appearance. History The Varnish project began in 2005. The issue to be solved was that of VG, a large Norwegian news site (or alternatively a tiny international site). The first release came in 2006, and worked well for pretty much one site: www.vg.no. In 2008, Varnish 2.0 came, which opened Varnish up to more sites, as long as they looked and behaved similar to www.vg.no. As time progressed and more people started using Varnish, Varnish has been adapted to a large and varied set of use cases. From the beginning, the project was administered through Redpill Linpro, with the majority of development being done by Poul-Henning Kamp through his own company and his Varnish Moral License. In 2010, Varnish Software sprung out from Redpill Linpro. Varnish Cache has always been a free software project, and while Varnish Software has been custodians of the infrastructure and large contributors of code and cash, the project is independent. Varnish Plus was born some time during 2011, all though it didn't go by that name at the time. It was the result of somewhat conflicting interests. Varnish Software had customer obligations that required features, and the development power to implement them, but they did not necessarily align with the goals and time frames of Varnish Cache. Varnish Plus became a commercial test-bed for features that were not /yet/ in Varnish Cache for various reasons. Many of the features have since trickled into Varnish Cache proper in one way or an other (streaming, surrogate keys, and more), and some have still to make it. Some may never make it. This book will focus on Varnish Cache proper, but will reference Varnish Plus where it makes sense. With Varnish 3.0, released in 2011, varnish modules started becoming a big thing. These are modules that are not part of the Varnish Cache code base, but are loaded at run-time to add features such as cryptographic hash functions (vmod-digest) and memcached. The number of vmods available grew quickly, but even with Varnish 4.1, the biggest issue with them were that they required source-compilation for use. That, however, is being fixed almost as I am writing this sentence. Varnish would not be where it is today without a large number of people and businesses. Varnish Software have contributed and continues to contribute numerous tools, vmods, and core features. Poul-Henning Kamp is still the gatekeeper of Varnish Cache code, for better or worse, and does the majority of the architectural work. Over the years, there have been too many companies and individuals involved to list them all in a book, so I will leave that to the official Varnish Cache project. Today, Varnish is used by CDNs and news papers, APIs and blogs. More than just cache Varnish caches content, but can do much more. In 2008, it was used to rewrite URLs, normalize HTTP headers and similar things. Today, it is used to implement paywalls (whether you like them or not), API metering, load balancing, CDNs, and more. Varnish has a powerful configuration language, the Varnish Configuration Language (VCL). VCL isn't parsed the traditional way a configuration file is, but is translated to C code, compiled and linked into the running Varnish. From the beginning, it was possible to bypass the entire translation process and provide C code directly, which was never recommended. With Varnish modules, it's possible to write proper modules to replace the in-line C code that was used in the past. There is also a often overlooked Varnish agent that provides a HTTP REST interface to managing Varnish. This can be used to extract metrics, review or optionally change configuration, stop and start Varnish, and more. The agent lives on https://github.com/varnish/vagent2, and is packaged for most distributions today. There's also a commercial administration console that builds further on the agent. Using Varnish to gracefully handle operational issues is also common. Serving cached content past its expiry time while a web server is down, or switching to a different server, will give your users a better browsing experience. And in a worst case scenario, at least the user can be presented with a real error message instead of a refused or timed out connection. An often overlooked feature of Varnish is Edge Side Includes. This is a means to build a single HTTP object (like a HTML page) from multiple smaller object, with different caching properties. This lets content writers provide more fine-grained caching strategies without having to be too smart about it. Where to get help The official varnish documentation is available both as manual pages (run man -k varnish on a machine with a properly installed Varnish package), and as Sphinx documentation found under http://varnish-cache.org/docs/. Varnish Software has also publish their official training material, which is called "The Varnish Book" (Not to be confused with THIS book about Varnish). This is available freely through their site at http://varnish-software.com, after registration. An often overlooked source of information for Varnish are the flow charts/dot-graphs used to document the VCL state engine. The official location for this is only found in the source code of Varnish, under doc/graphviz/. They can be generated simply, assuming you have graphviz installed: # git clone http://github.com/varnish/Varnish-Cache/ Cloning into 'Varnish-Cache'... (...) # cd Varnish-Cache/ # cd doc/graphviz/ # for a in *dot; do dot -Tpng $a > $(echo $a | sed s/.dot/.png/); done # ls *png Alternatively, replace -Tpng and .png with -Tsvg and .svg respectively to get vector graphics, or -Tpdf/.pdf for pdfs. You've now made three graphs that you might as well print right now and glue to your desk if you will be working with Varnish a lot. For convenience, the graphs from Varnish 4.1 are included. If you don't quite grasp what these tell you yet, don't be too alarmed. These graphs are provided early as they are useful to have around as reference material. A brief explanation for each is included, mostly to help you in later chapters. cache_req_fsm.png This can be used when writing VCL. You want to look for the blocks that read vcl_ to identify VCL functions. The lines tell you how a return-statement in VCL will affect the VCL state engine at large, and which return statements are available where. You can also see which objects are available where. This particular graph details the client-specific part of the VCL state engine. cache_fetch.png This graph has the same format as the cache_req_fsm.png-one, but from the perspective of a backend request. cache_http1_fsm.png Of the three, this is the least practical flow chart, mainly included for completeness. It does not document much related to VCL or practical Varnish usage, but the internal state engine of an HTTP request in Varnish. It can sometimes be helpful for debugging internal Varnish issues. Comments [Less]

Posted on 2015-11-24 This is the only chapter written in first person. I've worked on Varnish since late 2008, first for Redpill Linpro, then Varnish Software, then, after a brief pause, for Redpill Linpro again. Over the years I've written code ... [More] , written Varnish modules and blog posts, tried to push the boundaries of what Varnish can do, debugged or analyzed countless Varnish sites, probably held more training courses than anyone else, written training material, and helped shape the Varnish community. Today I find myself in a position where the training material I once maintained is no longer my responsibility. But I still love writing, and there's an obvious need for documentation for Varnish. I came up with a simple solution: I will write a book. Because I couldn't imagine that I would ever finish it if I attempted writing a whole book in one go, I decided I would publish one chapter at a time on my blog. This is the first chapter of that book. You will find the source on https://github.com/KristianLyng/varnishfoo. This is something I am doing on my spare time, and I hope to get help from the Varnish community in the form of feedback. While the format will be that of a book, I intend to keep it alive with updates as long as I can. I intend to cover as much Varnish-related content as possible, from administration to web development and infrastructure. And my hope is that one day, this will be good enough that it will be worth printing as more than just a leaflet. I am writing this in my spare time, I retain full ownership of the material. For now, the material is available under a Creative Commons "CC-BY-SA-NC" license. The NC-part of that license will be removed when I feel the material has matured enough and the time is right. To clarify, the "non-commercial" clause is aimed at people wanting to sell the book or use it in commercial training (or similar) - it is not intended to prevent you from reading the material at work. Target audience and format This book covers a large spectre of subjects related to Varnish. It is suitable for system administrators, infrastructure architects and web developers. The first few chapters is general enough to be of interest to all, while later chapters specialize on certain aspects of Varnish usage. Each chapter is intended to stand well on its own, but there will be some cross-references. The book focuses on best practices and good habits that will help you beyond what just a few examples or explanations will do. Each chapter provides both theory and practical examples. Each example is tested with a recent Varnish Version where relevant, and are based on experience from real-world Varnish installations. What is Varnish Varnish is a web server. Unlike most web servers, Varnish does not read content from a hard drive, or run programs that generates content from SQL databases. Varnish acquires the content from other web servers. Usually it will keep a copy of that content around in memory for a while to avoid fetching the same content multiple times, but not necessarily. There are numerous reasons you might want Varnish: Your web server/application is a beastly nightmare where performance is measured in page views per hour - on a good day. Your content needs to be available from multiple geographically diverse locations. Your web site consists of numerous different little parts that you need to glue together in a sensible manner. Your boss bought a service subscription and now has to justify the budget post. You like Varnish. ??? Varnish is designed around two simple concepts: Give you the means to fix or work around technical challenges. And speed. Speed was largely handled very early on, and Varnish is quite simply fast. This is achieved by being, at the core, simple. The less you have to do for each request, the more requests you can handle. The name suggests what it's all about: From The Collaborative International Dictionary of English v.0.48 [gcide]: Varnish \Var"nish\, v. t. [imp. & p. p. {Varnished}; p. pr. & vb. n. {Varnishing}.] [Cf. F. vernir, vernisser. See {Varnish}, n.] [1913 Webster] 1. To lay varnish on; to cover with a liquid which produces, when dry, a hard, glossy surface; as, to varnish a table; to varnish a painting. [1913 Webster] 2. To cover or conceal with something that gives a fair appearance; to give a fair coloring to by words; to gloss over; to palliate; as, to varnish guilt. "Beauty doth varnish age." --Shak. [1913 Webster] Varnish can be used to smooth over rough edges in your stack, to give a fair appearance. History The Varnish project began in 2005. The issue to be solved was that of VG, a large Norwegian news site (or alternatively a tiny international site). The first release came in 2006, and worked well for pretty much one site: www.vg.no. In 2008, Varnish 2.0 came, which opened Varnish up to more sites, as long as they looked and behaved similar to www.vg.no. As time progressed and more people started using Varnish, Varnish has been adapted to a large and varied set of use cases. From the beginning, the project was administered through Redpill Linpro, with the majority of development being done by Poul-Henning Kamp through his own company and his Varnish Moral License. In 2010, Varnish Software sprung out from Redpill Linpro. Varnish Cache has always been a free software project, and while Varnish Software has been custodians of the infrastructure and large contributors of code and cash, the project is independent. Varnish Plus was born some time during 2011, all though it didn't go by that name at the time. It was the result of somewhat conflicting interests. Varnish Software had customer obligations that required features, and the development power to implement them, but they did not necessarily align with the goals and time frames of Varnish Cache. Varnish Plus became a commercial test-bed for features that were not /yet/ in Varnish Cache for various reasons. Many of the features have since trickled into Varnish Cache proper in one way or an other (streaming, surrogate keys, and more), and some have still to make it. Some may never make it. This book will focus on Varnish Cache proper, but will reference Varnish Plus where it makes sense. With Varnish 3.0, released in 2011, varnish modules started becoming a big thing. These are modules that are not part of the Varnish Cache code base, but are loaded at run-time to add features such as cryptographic hash functions (vmod-digest) and memcached. The number of vmods available grew quickly, but even with Varnish 4.1, the biggest issue with them were that they required source-compilation for use. That, however, is being fixed almost as I am writing this sentence. Varnish would not be where it is today without a large number of people and businesses. Varnish Software have contributed and continues to contribute numerous tools, vmods, and core features. Poul-Henning Kamp is still the gatekeeper of Varnish Cache code, for better or worse, and does the majority of the architectural work. Over the years, there have been too many companies and individuals involved to list them all in a book, so I will leave that to the official Varnish Cache project. Today, Varnish is used by CDNs and news papers, APIs and blogs. More than just cache Varnish caches content, but can do much more. In 2008, it was used to rewrite URLs, normalize HTTP headers and similar things. Today, it is used to implement paywalls (whether you like them or not), API metering, load balancing, CDNs, and more. Varnish has a powerful configuration language, the Varnish Configuration Language (VCL). VCL isn't parsed the traditional way a configuration file is, but is translated to C code, compiled and linked into the running Varnish. From the beginning, it was possible to bypass the entire translation process and provide C code directly, which was never recommended. With Varnish modules, it's possible to write proper modules to replace the in-line C code that was used in the past. There is also a often overlooked Varnish agent that provides a HTTP REST interface to managing Varnish. This can be used to extract metrics, review or optionally change configuration, stop and start Varnish, and more. The agent lives on https://github.com/varnish/vagent2, and is packaged for most distributions today. There's also a commercial administration console that builds further on the agent. Using Varnish to gracefully handle operational issues is also common. Serving cached content past its expiry time while a web server is down, or switching to a different server, will give your users a better browsing experience. And in a worst case scenario, at least the user can be presented with a real error message instead of a refused or timed out connection. An often overlooked feature of Varnish is Edge Side Includes. This is a means to build a single HTTP object (like a HTML page) from multiple smaller object, with different caching properties. This lets content writers provide more fine-grained caching strategies without having to be too smart about it. Where to get help The official varnish documentation is available both as manual pages (run man -k varnish on a machine with a properly installed Varnish package), and as Sphinx documentation found under http://varnish-cache.org/docs/. Varnish Software has also publish their official training material, which is called "The Varnish Book" (Not to be confused with THIS book about Varnish). This is available freely through their site at http://varnish-software.com, after registration. An often overlooked source of information for Varnish are the flow charts/dot-graphs used to document the VCL state engine. The official location for this is only found in the source code of Varnish, under doc/graphviz/. They can be generated simply, assuming you have graphviz installed: # git clone http://github.com/varnish/Varnish-Cache/ Cloning into 'Varnish-Cache'... (...) # cd Varnish-Cache/ # cd doc/graphviz/ # for a in *dot; do dot -Tpng $a > $(echo $a | sed s/.dot/.png/); done # ls *png Alternatively, replace -Tpng and .png with -Tsvg and .svg respectively to get vector graphics, or -Tpdf/.pdf for pdfs. You've now made three graphs that you might as well print right now and glue to your desk if you will be working with Varnish a lot. For convenience, the graphs from Varnish 4.1 are included. If you don't quite grasp what these tell you yet, don't be too alarmed. These graphs are provided early as they are useful to have around as reference material. A brief explanation for each is included, mostly to help you in later chapters. cache_req_fsm.png This can be used when writing VCL. You want to look for the blocks that read vcl_ to identify VCL functions. The lines tell you how a return-statement in VCL will affect the VCL state engine at large, and which return statements are available where. You can also see which objects are available where. This particular graph details the client-specific part of the VCL state engine. cache_fetch.png This graph has the same format as the cache_req_fsm.png-one, but from the perspective of a backend request. cache_http1_fsm.png Of the three, this is the least practical flow chart, mainly included for completeness. It does not document much related to VCL or practical Varnish usage, but the internal state engine of an HTTP request in Varnish. It can sometimes be helpful for debugging internal Varnish issues. Comments [Less]

Posted on 2015-11-16 I was preparing to upgrade a customer, and ran across a semi-extensive VCL setup. It quickly became a bit hard to get a decent overview of what was going on. The actual VCL is fairly simple. To deal with this, I ended up hacking ... [More] together a tiny awk/shell script to generate a dot graph of how things were glued together. You can find the script at http://kly.no/code/script/vcl-visualizer.sh . The output is somewhat ugly, but useful. (Click for full version) Of note: This is so far Varnish 3.0-ish, mainly because of the error-syntax. (So it'll work for 4.x, just without vcl_error-tracking) Green borders: Found the reference and everything is OK. Black border: The sub was referenced, but not found in any VCL file. Red border: The sub was found, but never referenced (doesn't count for subroutines beginning with vcl_, e.g. vcl_recv, etc) No idea if it's of interest to anyone but me, but I found it useful. Comments [Less]

Posted on 2015-11-16 I was preparing to upgrade a customer, and ran across a semi-extensive VCL setup. It quickly became a bit hard to get a decent overview of what was going on. The actual VCL is fairly simple. To deal with this, I ended up hacking ... [More] together a tiny awk/shell script to generate a dot graph of how things were glued together. You can find the script at http://kly.no/code/script/vcl-visualizer.sh . The output is somewhat ugly, but useful. (Click for full version) Of note: This is so far Varnish 3.0-ish, mainly because of the error-syntax. (So it'll work for 4.x, just without vcl_error-tracking) Green borders: Found the reference and everything is OK. Black border: The sub was referenced, but not found in any VCL file. Red border: The sub was found, but never referenced (doesn't count for subroutines beginning with vcl_, e.g. vcl_recv, etc) No idea if it's of interest to anyone but me, but I found it useful. Comments [Less]

Varnish-4.1.0 was recently released, and as usual, I have patched and wrapped up packages for fedora and epel. As 4.1.0 is not api/abi compatible with varnish-4.0, packages for stable releases of epel and fedora are not updated. Varnish-4.1.x will be ... [More] available in a stable Fedora at latest from f24, though the package recompiles fine on anything from el5 to f23 as well. Prebuilt packages for epel5, epel6, and epel7 are available here: http://users.linpro.no/ingvar/varnish/4.1.0/. If you are a fedora contributor, please test the f23 package. The package should install directly on el7 and all supported fedoras, including f23. Then report feedback and add karma points. With a little luck, varnish-4.1 will go into fedora 23 before it freezes. Ingvar – Varnish Cache is a powerful and feature rich front side web cache. It is also very fast, and that is, fast as in powered by The Dark Side of the Force. On steroids. And it is Free Software. Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, cloud, and data center, contact us at www.redpill-linpro.com. [Less]

Posted on 2015-09-25 I was hacking together a JavaScript varnishstat implementation for a customer a few days ago when I noticed something strange. I have put Varnish in front of the agent delivering stats, but I'm only caching the statistics for 1 ... [More] second. But the cache hit rate was 100%. And the stats were updating? Logically speaking, how can you hit cache 100% of the time and still get fresh content all the time? Enter Grace Grace mode is a feature Varnish has had since version 2.0 back in 2008. It is a fairly simple mechanic: Add a little bit of extra cache duration to an object. This is the grace period. If a request is made for the object during that grace period, the object is updated and the cached copy is used while updating it. This reduces the thundering horde problem when a large amount of users request recently expired content, and it can drastically improve user experience when updating content is expensive. The big change that happened in Varnish 4 was background fetches. Varnish uses a very simple thread model (so to speak). Essentially, each session is handled by one thread. In prior versions of Varnish, requests to the backend were always tied to a client request. Thread 1: Accept request from client 1 Thread 1: Look up content in cache Thread 1: Cache miss Thread 1: Request content from web server Thread 1: Block Thread 1: Get content from web server Thread 1: Respond If the cache is empty, there isn't much of a reason NOT to do this. Grace mode always complicated this. What PHK did to solve this was, in my opinion, quite brilliant in its simplicity. Even if it was a trade-off. With grace mode, you HAVE the content, you just need to make sure it's updated. It looked something like this: Thread 1: Accept request from client 1 Thread 1: Look up content in cache Thread 1: Cache miss Thread 1: Request content from web server Thread 1: Block Thread 1: Get content from web server Thread 1: Respond So ... NO CHANGE. For a single client, you don't have grace mode in earlier Varnish versions. But enter client number 2 (or 3, 4, 5...): Thread 1: Accept request from client 1 Thread 1: Look up content in cache Thread 1: Cache miss Thread 1: Request content from web server Thread 1: Block Thread 2: Accept request from client 2 Thread 2: Look up content in cache Thread 2: Cache hit - grace copy is now eligible - Respond Thread 1: Get content from web server Thread 1: Respond So with Varnish 2 and 3, only the first client will block waiting for new content. This is still an issue, but it does the trick for the majority of use cases. Background fetches! Background fetches changed all this. It's more complicated in many ways, but from a grace perspective, it massively simplifies everything. With Varnish 4 you get: Thread 1: Accept request from client 1 Thread 1: Look up content in cache Thread 1: Cache hit - grace copy is now eligible - Respond Thread 2: Request content from web server Thread 2: Block Thread 3: Accept request from client 2 Thread 3: Look up content in cache Thread 3: Cache hit - grace copy is now eligible - Respond Thread 2: Get content from web server And so forth. Strictly speaking, I suppose this makes grace /less/ magical... In other words: The first client will also get a cache hit, but Varnish will update the content in the background for you. It just works. Statistics? What is a cache hit? If I tell you that I have 100% cache hit rate, how much backend traffic would you expect? We want to keep track of two ratios: Cache hit rate - how much content is delivered directly from cache (same as today). Target value: 100%. Fetch/request ratio: How many backend fetches do you initiate per client request. Target value: 0%. For my application, a single user will result in a 100% cache hit rate, but also a fetch/request ratio of 100%. The cache isn't really offloading the backend load significantly until I have multiple users of the app. Mind you, if the application was slow, this would still benefit that one user. The latter is also interesting from a security point of view. If you find the right type of request, you could end up with more backend fetches than client requests (e.g. due to restarts/retries). How to use grace You already have it, most likely. Grace is turned on by default, using a 10 second grace period. For frequently updated content, this is enough. Varnish 4 changed some of the VCL and parameters related to grace. The important bits are: Use beresp.grace in VCL to adjust grace for an individual object. Use the default_grace parameter to adjust the ... default grace for objects. If you want to override grace mechanics, you can do so in either vcl_recv by setting req.ttl to define a max TTL to be used for an object, regardless of the actual TTL. That bit is a bit mysterious. Or you can look at vcl_hit. Here you'll be able to do: if (obj.ttl + obj.grace > 0s && obj.ttl =< 0s) { // We are in grace mode, we have an object though if (req.http.x-magic-skip-grace-header ~ "yes") { return (miss); } else { return (delier); } } The above example-snippet will evaluate of the object has an expired TTL, but is still in the grace period. If that happens, it looks for a client header called "X-Magic-Skip-Grace-Header" and checks if it contains the string "yes". If so, the request is treated as a cache miss, otherwise, the cached object is delivered. Comments [Less]

Posted on 2015-09-19 I recently went back to working for Redpill Linpro, and thus started working with Varnish again, after being on the side lines for a few years. I've been using Varnish since 2008. And a bit more than just using it too. There's ... [More] been a lot of great change over time, but there are still things missing. I recently read http://kacper.blog.redpill-linpro.com/archives/743 and while I largely agree with Kacper, I think some of the bigger issues are missing from the list. So here's my attempt to add to the debate. TLS/SSL Varnish needs TLS/SSL. It's the elephant in the room that nobody wants to talk about. The world is not the same as it was in 2006. Varnish is used for more and more sensitive sites. A larger percentage of Varnish installations now have some sort of TLS/SSL termination attached to it. TLS/SSL has been a controversial issue in the history of Varnish Cache, with PHK (Principal architect of Varnish Cache - https://www.varnish-cache.org/docs/trunk/phk/ssl.html) being an outspoken opponent of adding TLS in Varnish. There are valid reasons, and heartbleed has most certainly proven many of PHK's grievances right. But what does that matter when we use TLS/SSL anyway? It's already in the stack, we're just closing our eyes to it. Setting up nginx in front of Varnish to get TLS/SSL, then nginx behind Varnish to get TLS/SSL... That's just silly. Why not just use nginx to cache then? The lack of TLS/SSL in Varnish is a great advertisement for nginx. There are a lot of things I dislike about TLS/SSL, but we need it anyway. There's the hitch project (http://hitch-tls.org), but it's not really enough. We also need TLS/SSL to the backends, and a tunnel-based solution isn't enough. How would you do smart load balancing through that? If we don't add TLS/SSL, we might as well just forget about backend directors all together. And it has to be an integral part of all backends. We can't have a situation where some backend directors support TLS/SSL and some don't. Varnish Software is already selling this through Varnish Cache Plus, their proprietary version of Varnish. That is obviously because it's a deal breaker in a lot of situations. The same goes for basically any serious commercial actor out there. So we need TLS/SSL. And we need it ASAP. Note After speaking to PHK, let me clarify: He's not against adding support for TLS, but adding TLS itself. Varnish now supports the PROXY-protool which is added explicitly to improve support for TLS termination. Further such additions would likely be acceptable, always doing the TLS outside of Varnish. Better procedures for VCL changes With every Varnish version, VCL (The configuration language for Varnish) changes either a little bit, or a lot. Some of these changes are unavoidable due to internal Varnish changes. Some changes are to tweak the language to be more accurate (e.g. changing req.request to req.method, to reflect that it's the request method). If Varnish is part of your day-to-day work, then this might not be a huge deal. You probably keep up-to-date on what's going on with Varnish anyway. But most users aren't there. We want Varnish to be a natural part of your stack, not a special thing that requires a "varnish-admin". This isn't necessarily an easy problem to solve. We want to be able to improve VCL and get rid of old mistakes (e.g., changing req.request to req.method is a good thing for VCL). We've also changed the way to do error messages (or custom varnish-generated messages) numerous times. And how to create hitpass objects (a complicated aspect of any cache). A few simple suggestions: All VCL changes reviewed in public as a whole before the release process even starts. To avoid having to change it again two versions down the line. Backward compatibility when possible. With warnings or even requiring an extra option to allow it. E.g.: req.request could easily still work, there's no conflict there. Never for forever, but perhaps to the end of a major version. Not everything will be backwards compatible, but some can. I've had numerous complaints from highly skilled sysadmins who are frustrated by this aspect of Varnish. They just don't want to upgrade because they have to do what feels like arbitrary VCL changes every single time. Let's see if we can at least REDUCE that. Documentation? There's a lot of documentation for Varnish, but there's also a lot of bad documentation. Some issues: People Google and end up on random versions on varnish-cache.org. No, telling people "but there's a version right there so it's your own fault!" is not an acceptable solution. Varnish Software them self recently had a link in their Varnish Book where they used a link to "trunk" instead of "4.0", whereupon the "here is a complete list of changes between Varnish 3 and Varnish 4" link was actually a link to changes betwen Varnish 4.0 and the next version of Varnish. "user guide" and "tutorial" and "installation"? Kill at least two and leave the others for blog posts or whatever. Hard enough to maintain one with decent quality. Generated documentation needs to be improved. Example: Prototype STRING fileread(PRIV_CALL, STRING) Description Reads a file and returns a string with the content. Please note that it is not recommended to send variables to this function the caching in the function doesn't take this into account. Also, files are not re-read. Example set beresp.http.served-by = std.fileread("/etc/hostname"); PRIV_CALL should clearly not be exposed! Other examples are easy enough to find. In addition, the Description is a mixture of reference documentation style and elaboration. Reference documentation should be clearly separated from analysis of consequences so technical users don't have to reverse-engineer a sentence of "don't do this because X" to figure out what the code actually does. And where are the details? What happens if the file can't be opened? What are the memory constraints? It says it returns the content of the file as a string, but what happens with binary content? There's clearly some caching of the file, but how does that work? Per session? Per VCL? Does that cache persist when you do varnishadm stop; varnishadm start? That's completely left out. Rants mixed in with documentation? Get rid of "doc/shpinx/phk" (https://www.varnish-cache.org/docs/4.0/phk/) and instead reference it somewhere else. Varnish-cache.org/doc should not be a weird blog-space. It clutters the documentation space. Varnish is not a small little project any more, it's grown past this. VMOD packages Varnish vmods are awesome. You can design some truly neat solutions using Open Source vmods, or proprietary ones. But there are no even semi-official package repositories for the open source vmods. Varnish Software offers this to customers, but I really want it for the public too. Both for my own needs, and because it's important to improve Varnish and VMOD adaption. Until you can do "apt-get install varnish-vmod-foo" or something like that, VMODS will not get the attention they deserve. There are some projects in the works here, though, so stay tuned. TLS/SSL In case you missed it, I want TLS/SSL. I want to be able to type https:// BTW: Regarding terminology, I decided to go with "TLS/SSL" instead of either "SSL" or "TLS" after some feedback. I suppose "TLS" is correct, but "SSL" is more recognized, whether we like it or not. Comments [Less]

Posted on 2015-09-19 I recently went back to working for Redpill Linpro, and thus started working with Varnish again, after being on the side lines for a few years. I've been using Varnish since 2008. And a bit more than just using it too. There's ... [More] been a lot of great change over time, but there are still things missing. I recently read http://kacper.blog.redpill-linpro.com/archives/743 and while I largely agree with Kacper, I think some of the bigger issues are missing from the list. So here's my attempt to add to the debate. TLS/SSL Varnish needs TLS/SSL. It's the elephant in the room that nobody wants to talk about. The world is not the same as it was in 2006. Varnish is used for more and more sensitive sites. A larger percentage of Varnish installations now have some sort of TLS/SSL termination attached to it. TLS/SSL has been a controversial issue in the history of Varnish Cache, with PHK (Principal architect of Varnish Cache - https://www.varnish-cache.org/docs/trunk/phk/ssl.html) being an outspoken opponent of adding TLS in Varnish. There are valid reasons, and heartbleed has most certainly proven many of PHK's grievances right. But what does that matter when we use TLS/SSL anyway? It's already in the stack, we're just closing our eyes to it. Setting up nginx in front of Varnish to get TLS/SSL, then nginx behind Varnish to get TLS/SSL... That's just silly. Why not just use nginx to cache then? The lack of TLS/SSL in Varnish is a great advertisement for nginx. There are a lot of things I dislike about TLS/SSL, but we need it anyway. There's the hitch project (http://hitch-tls.org), but it's not really enough. We also need TLS/SSL to the backends, and a tunnel-based solution isn't enough. How would you do smart load balancing through that? If we don't add TLS/SSL, we might as well just forget about backend directors all together. And it has to be an integral part of all backends. We can't have a situation where some backend directors support TLS/SSL and some don't. Varnish Software is already selling this through Varnish Cache Plus, their proprietary version of Varnish. That is obviously because it's a deal breaker in a lot of situations. The same goes for basically any serious commercial actor out there. So we need TLS/SSL. And we need it ASAP. Note After speaking to PHK, let me clarify: He's not against adding support for TLS, but adding TLS itself. Varnish now supports the PROXY-protool which is added explicitly to improve support for TLS termination. Further such additions would likely be acceptable, always doing the TLS outside of Varnish. Better procedures for VCL changes With every Varnish version, VCL (The configuration language for Varnish) changes either a little bit, or a lot. Some of these changes are unavoidable due to internal Varnish changes. Some changes are to tweak the language to be more accurate (e.g. changing req.request to req.method, to reflect that it's the request method). If Varnish is part of your day-to-day work, then this might not be a huge deal. You probably keep up-to-date on what's going on with Varnish anyway. But most users aren't there. We want Varnish to be a natural part of your stack, not a special thing that requires a "varnish-admin". This isn't necessarily an easy problem to solve. We want to be able to improve VCL and get rid of old mistakes (e.g., changing req.request to req.method is a good thing for VCL). We've also changed the way to do error messages (or custom varnish-generated messages) numerous times. And how to create hitpass objects (a complicated aspect of any cache). A few simple suggestions: All VCL changes reviewed in public as a whole before the release process even starts. To avoid having to change it again two versions down the line. Backward compatibility when possible. With warnings or even requiring an extra option to allow it. E.g.: req.request could easily still work, there's no conflict there. Never for forever, but perhaps to the end of a major version. Not everything will be backwards compatible, but some can. I've had numerous complaints from highly skilled sysadmins who are frustrated by this aspect of Varnish. They just don't want to upgrade because they have to do what feels like arbitrary VCL changes every single time. Let's see if we can at least REDUCE that. Documentation? There's a lot of documentation for Varnish, but there's also a lot of bad documentation. Some issues: People Google and end up on random versions on varnish-cache.org. No, telling people "but there's a version right there so it's your own fault!" is not an acceptable solution. Varnish Software them self recently had a link in their Varnish Book where they used a link to "trunk" instead of "4.0", whereupon the "here is a complete list of changes between Varnish 3 and Varnish 4" link was actually a link to changes betwen Varnish 4.0 and the next version of Varnish. "user guide" and "tutorial" and "installation"? Kill at least two and leave the others for blog posts or whatever. Hard enough to maintain one with decent quality. Generated documentation needs to be improved. Example: Prototype STRING fileread(PRIV_CALL, STRING) Description Reads a file and returns a string with the content. Please note that it is not recommended to send variables to this function the caching in the function doesn't take this into account. Also, files are not re-read. Example set beresp.http.served-by = std.fileread("/etc/hostname"); PRIV_CALL should clearly not be exposed! Other examples are easy enough to find. In addition, the Description is a mixture of reference documentation style and elaboration. Reference documentation should be clearly separated from analysis of consequences so technical users don't have to reverse-engineer a sentence of "don't do this because X" to figure out what the code actually does. And where are the details? What happens if the file can't be opened? What are the memory constraints? It says it returns the content of the file as a string, but what happens with binary content? There's clearly some caching of the file, but how does that work? Per session? Per VCL? Does that cache persist when you do varnishadm stop; varnishadm start? That's completely left out. Rants mixed in with documentation? Get rid of "doc/shpinx/phk" (https://www.varnish-cache.org/docs/4.0/phk/) and instead reference it somewhere else. Varnish-cache.org/doc should not be a weird blog-space. It clutters the documentation space. Varnish is not a small little project any more, it's grown past this. VMOD packages Varnish vmods are awesome. You can design some truly neat solutions using Open Source vmods, or proprietary ones. But there are no even semi-official package repositories for the open source vmods. Varnish Software offers this to customers, but I really want it for the public too. Both for my own needs, and because it's important to improve Varnish and VMOD adaption. Until you can do "apt-get install varnish-vmod-foo" or something like that, VMODS will not get the attention they deserve. There are some projects in the works here, though, so stay tuned. TLS/SSL In case you missed it, I want TLS/SSL. I want to be able to type https:// BTW: Regarding terminology, I decided to go with "TLS/SSL" instead of either "SSL" or "TLS" after some feedback. I suppose "TLS" is correct, but "SSL" is more recognized, whether we like it or not. Comments [Less]

Posted on 2015-09-19 I recently went back to working for Redpill Linpro, and thus started working with Varnish again, after being on the side lines for a few years. I've been using Varnish since 2008. And a bit more than just using it too. There's ... [More] been a lot of great change over time, but there are still things missing. I recently read http://kacper.blog.redpill-linpro.com/archives/743 and while I largely agree with Kacper, I think some of the bigger issues are missing from the list. So here's my attempt to add to the debate. TLS/SSL Varnish needs TLS/SSL. It's the elephant in the room that nobody wants to talk about. The world is not the same as it was in 2006. Varnish is used for more and more sensitive sites. A larger percentage of Varnish installations now have some sort of TLS/SSL termination attached to it. TLS/SSL has been a controversial issue in the history of Varnish Cache, with PHK (Principal architect of Varnish Cache - https://www.varnish-cache.org/docs/trunk/phk/ssl.html) being an outspoken opponent of adding TLS in Varnish. There are valid reasons, and heartbleed has most certainly proven many of PHK's grievances right. But what does that matter when we use TLS/SSL anyway? It's already in the stack, we're just closing our eyes to it. Setting up nginx in front of Varnish to get TLS/SSL, then nginx behind Varnish to get TLS/SSL... That's just silly. Why not just use nginx to cache then? The lack of TLS/SSL in Varnish is a great advertisement for nginx. There are a lot of things I dislike about TLS/SSL, but we need it anyway. There's the hitch project (http://hitch-tls.org), but it's not really enough. We also need TLS/SSL to the backends, and a tunnel-based solution isn't enough. How would you do smart load balancing through that? If we don't add TLS/SSL, we might as well just forget about backend directors all together. And it has to be an integral part of all backends. We can't have a situation where some backend directors support TLS/SSL and some don't. Varnish Software is already selling this through Varnish Cache Plus, their proprietary version of Varnish. That is obviously because it's a deal breaker in a lot of situations. The same goes for basically any serious commercial actor out there. So we need TLS/SSL. And we need it ASAP. Note After speaking to PHK, let me clarify: He's not against adding support for TLS, but adding TLS itself. Varnish now supports the PROXY-protool which is added explicitly to improve support for TLS termination. Further such additions would likely be acceptable, always doing the TLS outside of Varnish. Better procedures for VCL changes With every Varnish version, VCL (The configuration language for Varnish) changes either a little bit, or a lot. Some of these changes are unavoidable due to internal Varnish changes. Some changes are to tweak the language to be more accurate (e.g. changing req.request to req.method, to reflect that it's the request method). If Varnish is part of your day-to-day work, then this might not be a huge deal. You probably keep up-to-date on what's going on with Varnish anyway. But most users aren't there. We want Varnish to be a natural part of your stack, not a special thing that requires a "varnish-admin". This isn't necessarily an easy problem to solve. We want to be able to improve VCL and get rid of old mistakes (e.g., changing req.request to req.method is a good thing for VCL). We've also changed the way to do error messages (or custom varnish-generated messages) numerous times. And how to create hitpass objects (a complicated aspect of any cache). A few simple suggestions: All VCL changes reviewed in public as a whole before the release process even starts. To avoid having to change it again two versions down the line. Backward compatibility when possible. With warnings or even requiring an extra option to allow it. E.g.: req.request could easily still work, there's no conflict there. Never for forever, but perhaps to the end of a major version. Not everything will be backwards compatible, but some can. I've had numerous complaints from highly skilled sysadmins who are frustrated by this aspect of Varnish. They just don't want to upgrade because they have to do what feels like arbitrary VCL changes every single time. Let's see if we can at least REDUCE that. Documentation? There's a lot of documentation for Varnish, but there's also a lot of bad documentation. Some issues: People Google and end up on random versions on varnish-cache.org. No, telling people "but there's a version right there so it's your own fault!" is not an acceptable solution. Varnish Software them self recently had a link in their Varnish Book where they used a link to "trunk" instead of "4.0", whereupon the "here is a complete list of changes between Varnish 3 and Varnish 4" link was actually a link to changes betwen Varnish 4.0 and the next version of Varnish. "user guide" and "tutorial" and "installation"? Kill at least two and leave the others for blog posts or whatever. Hard enough to maintain one with decent quality. Generated documentation needs to be improved. Example: Prototype STRING fileread(PRIV_CALL, STRING) Description Reads a file and returns a string with the content. Please note that it is not recommended to send variables to this function the caching in the function doesn't take this into account. Also, files are not re-read. Example set beresp.http.served-by = std.fileread("/etc/hostname"); PRIV_CALL should clearly not be exposed! Other examples are easy enough to find. In addition, the Description is a mixture of reference documentation style and elaboration. Reference documentation should be clearly separated from analysis of consequences so technical users don't have to reverse-engineer a sentence of "don't do this because X" to figure out what the code actually does. And where are the details? What happens if the file can't be opened? What are the memory constraints? It says it returns the content of the file as a string, but what happens with binary content? There's clearly some caching of the file, but how does that work? Per session? Per VCL? Does that cache persist when you do varnishadm stop; varnishadm start? That's completely left out. Rants mixed in with documentation? Get rid of "doc/shpinx/phk" (https://www.varnish-cache.org/docs/4.0/phk/) and instead reference it somewhere else. Varnish-cache.org/doc should not be a weird blog-space. It clutters the documentation space. Varnish is not a small little project any more, it's grown past this. VMOD packages Varnish vmods are awesome. You can design some truly neat solutions using Open Source vmods, or proprietary ones. But there are no even semi-official package repositories for the open source vmods. Varnish Software offers this to customers, but I really want it for the public too. Both for my own needs, and because it's important to improve Varnish and VMOD adaption. Until you can do "apt-get install varnish-vmod-foo" or something like that, VMODS will not get the attention they deserve. There are some projects in the works here, though, so stay tuned. TLS/SSL In case you missed it, I want TLS/SSL. I want to be able to type https:// BTW: Regarding terminology, I decided to go with "TLS/SSL" instead of either "SSL" or "TLS" after some feedback. I suppose "TLS" is correct, but "SSL" is more recognized, whether we like it or not. Comments [Less]

I’ve been meaning to write a blog entry about Varnish for years now. The closest I’ve come is to write a blog about how to make Varnish cache your debian repos, make you a WikiLeaks cache and I’ve released Varnish Secure Firewall, but that without a ... [More] word on this blog. So? SO? Well, after years it turns out there is a thing or two to say about Varnish. Read on to find out what annoys me and people I meet the most. Although you could definitely call me a “Varnish expert” and even a sometimes contributor, and I do develop programs, I cannot call myself a Varnish developer because I’ve shamefully never participated in a Monday evening bug wash. My role in the Varnish world is more… operative. I am often tasked with helping ops people use Varnish correctly, justify its use and cost to their bosses, defend it from expensive and inferior competitors, sit up long nites with load tests just before launch days. I’m the guy that explains the low risk and high reward of putting Varnish in front of your critical site, and the guy that makes it actually be low risk, with long nites on load tests and I’ll be the first guy on the scene when the code has just taken a huge dump on the CEO’s new pet Jaguar. I am also sometimes the guy who tells these stories to the Varnish developers, although of course they also have other sources. The consequences of this .. lifestyle choice .. is that what code I do write is either short and to the point or .. incomplete. I know we all love Varnish, which is why after nearly 7 years of working with this software I’d like to share with you my pet peeves about the project. There aren’t many problems with this lovely and lean piece of software but those which are there are sharp edges that pretty much everyone snubs a toe or snags their head on. Some of them are specific to a certain version, while others are “features” present in nearly all versions. And for you Varnish devs who will surely read this, I love you all. I write this critique of the software you contribute to, knowing full well that I haven’t filed bug reports on any of these issues and therefore I too am guilty in contributing to the problem and not the solution. I aim to change that starting now :-) Also, I know that some of these issues are better lived with than fixed, the medicine being more hazardous than the disease, so take this as all good cooking; with a grain of salt. Silent error messages in init scripts Some genious keeps inserting 1>/dev/null 2>&1 into the startup scripts on most Linux distros. This might be in line with some wacko distro policy but makes conf errors and in particular VCL errors way harder to debug for the common man. Even worse, the `service varnish reload` script called `varnish-vcl-reload -q`, that’s q for please-silence-my-fatal-conf-mistakes, and the best way to fix this is to *edit the init script and remove the offender*. Mind your p’s and q’s eh, it makes me sad every time, but where do I file this particular bug report? debug.health still not adequately documented People go YEARS using Varnish without discovering watch varnishadm debug.health. Not to mention that it’s anyone’s guess this has to do with probes, and that there are no other debug.* parameters, except for the totally unrelated debug parameter. Perhaps this was decided to be dev-internal at some point, but the probe status is actually really useful in precisely this form. debug.health is still absent from the param.show list and the man pages, while in 4.0 some probe status and backend info has been put into varnishstat, which I am sure to be not the only one being verry thankful for indeed. Bad naming Designing a language is tricky. Explaining why purge is now ban and what is now purge is something else is mindboggling. This issue will be fixed in 10 years when people are no longer running varnish 2.1 anywhere. Explaining all the three-letter acronyms that start with V is just a gas. Showing someone ban("req.url = "+ req.url) for the first time is bound to make them go “oh” like a racoon just caught sneaking through your garbage. Grace and Saint mode… that’s biblical, man. Understanding what it does and how to demonstrate the functionality is still for Advanced Users, explaining this to noobs is downright futile, and I am still unsure whether we wouldn’t all be better off for just enabling it by default and forgetting about it. I suppose if you’re going to be awesome at architecting and writing software, it’s going to get in the way of coming up with really awesome names for things, and I’m actually happy that’s still the way they prioritize what gets done first. Only for people who grok regex Sometimes you’ll meet Varnish users who do code but just don’t grok regex. It’s weak, I know, but this language isn’t for them. Uncertain current working directory This is a problem on some rigs which have VCL code in stacked layers, or really anywhere where it’s more appropriate to call the VCL a Varnish program, as in “a program written for the Varnish runtime”, rather than simply a configuration for Varnish. You’ll typically want to organize your VCL in such a way that each VCL is standalone with if-wrappend rules and they’re all included from one main vcl file, stacking all the vcl_recv’s and vcl_fetches . Because distros don’t agree on where to put varnishd’s current working directory, which happens to be where it’s been launched from, instead of always chdir $(basename $CURRENT_VCL_FILE), you can’t reliably specify include statements with relative paths. This forces us to use hardcoded absolute paths in includes, which is neither pretty nor portable. Missing default director in 4.0 When translating VCL to 4.0 there is no longer any language for director definitions, which means they are done in vcl_init(), which means your default backend is no longer the director you specified at the top, which means you’ll have to rewrite some logic lest it bite you in the ass. director.backend() is without string representation, instead of backend_hint, so cannot do old style name comparisons, ie backends are first-class objects but directors are another class of objects. VCL doesn’t allow unused backends or probes Adding and removing backends is a routine ordeal in Varnish. Quite often you’ll find it useful to keep backup backends around that aren’t enabled, either as manual failover backups, because you’re testing something or just because you’re doing something funky. Unfortunately, the VCC is a strict and harsh mistress on this matter: you are forced to comment out or delete unused backends :-( Workarounds include using the backends inside some dead code or constructs like vcl_recv{ set req.backend_hint = unused; set req.backend_hint = default; ... } It’s impossible to determine how many bugs this error message has avoided by letting you know that backend you just added, er yes that one isn’t in use sir, but you can definitely count the number of Varnish users inconvenienced by having to “comment out that backend they just temporarily removed from the request flow”. I am sure it is wise to warn about this, but couldn’t it have been just that, a warning? Well, I guess maybe not, considering distro packaging is silencing error messages in init and reload scripts.. To be fair, this is now configurable in Varnish by setting vcc_err_unref to false, but couldn’t this be the default? saintmode_threshold default considered harmful If many different URLs keep returning bad data or error codes, you might concievably want the whole backend to be declared sick instead of growing some huge list of sick urls for this backend. What if I told you your developers just deployed an application which generates 50x error codes triggering your saintmode for an infinite amount of URLs? Well, then you have just DoSed yourself because you hit this threshold. I usually enable saintmode only after giving my clients a big fat warning about this one, because quite frankly this easily comes straight out of left field every time. Either saintmode is off, or the treshold is Really Large™ or even ∞, and in only some special cases do you actually want this set to an actual number. Then again, maybe it is just my clients and the wacky applications they put behind Varnish. What is graceful about the saint in V4? While we are on the subject, grace mode being the most often misunderstood feature of Varnish, the thing has changed so radically in Varnish 4 that it is no longer recognizable by users, and they often make completely reasonable but devestating mistakes trying to predict its behavior. To be clear on what has happened: saint mode is deprecated as a core feature in V4.0, while the new architecture now allows a type of “stale-while-revalidate” logic. A saintmode vmod is slated for Varnish 4.1. But as of 4.0, say you have a bunch of requests hitting a slow backend. They’ll all queue up while we fetch a new one, right? Well yes, and then they all error out when that request times out, or if the backend fetch errors out. That sucks. So lets turn on grace mode, and get “stale-while-revalidate” and even “stale-if-error” logic, right? And send If-Modified-Since headers too, sweet as. Now that’s gonna work when the request times out, but you might be surprised that it does not when the request errors out with 50x errors. Since beresp.saint_mode isn’t a thing anymore in V4, those error codes are actually going to knock the old object outta cache and each request is going to break your precious stale-while-error until the backend probe declares the backend sick and your requests become grace candidates. Ouch, you didn’t mean for it to do that, did you? And if, gods forbid, your apphost returns 404′s when some backend app is not resolving, bam you are in a cascading hell fan fantasy. What did you want it to do, behave sanely? A backend response always replaces another backend response for the same URL – not counting vary-headers. To get a poor mans saint mode back in Varnish 4.0, you’ll have to return (abandon) those erroneous backend responses. Evil grace on unloved objects For frequently accessed URLs grace is fantastic, and will save you loads of grief, and those objects could have large grace times. However, rarely accessed URLs suffer a big penalty under grace, especially when they are dynamic and ment to be updated from backend. If that URL is meant to be refreshed from backend every hour, and Varnish sees many hours between each access, it’s going to serve up that many-hour-old stale object while it revalidates its cache. This diagram might help you understand what happens in the “200 OK” and “50x error” cases of graceful request flow through Varnish 4.0. Language breaks on major versions This is a funny one because the first major language break I remember was the one that I caused myself. We were making security.vcl and I was translating rules from mod_security and having trouble with it because Varnish used POSIX regexes at the time, and I was writing this really godaweful script to translate PCRE into POSIX when Kristian who conceived of security.vcl went to Tollef, who were both working in the same department at the time, and asked in his classical broker-no-argument kind of way "why don’t we just support Perl regexes?". Needless to say, (?i) spent a full 12 months afterwards cursing myself while rewriting tons of nasty client VCL code from POSIX to PCRE and fixing occasional site-devestating bugs related to case-sensitivity. Of course, Varnish is all the better for the change, and would get no where fast if the devs were to hang on to legacy, but there is a lesson in here somewhere. So what's a couple of sed 's/req.method/req.request/'s every now and again? This is actually the main reason I created the VCL.BNF. For one, it got the devs thinking about the grammar itself as an actual thing (which may or may not have resulted in the cleanups that make VCL a very regular and clean language today), but my intent was to write a parser that could parse any version of VCL and spit out any other version of VCL, optionally pruning and pretty-printing of course. That is still really high on my todo list. Funny how my clients will book all my time to convert their code for days but will not spend a dime on me writing code that would basically make the conversion free and painless for everyone forever. Indeed, most of these issues are really hard to predict consequences of implementation decisions, and I am unsure whether it would be possible to predict these consequences without actually getting snagged by the issues in the first place. So again: varnish devs, I love you, what are your pet peeves? Varnish users, what are your pet peeves? Errata: vcc_err_unref has existed since Varnish 3. [Less]