Varnish

Varnish is a high-performance HTTP accelerator, or frontside cache if you like. Working with Varnish is part of my day job. Among other things, I maintain the packages for Fedora and EPEL. To celebrate the release of Varnish version 3, I decided to ... [More] poke around lists again, to look for Varnish in common use. This is more or less a repost, with updated numbers. There is no deep magic here. I just parse some of the available top lists that I know of, and peek at the HTML headers of the sites that are listed. If there are subsites linked from the front page of the site, I scan them too. Subsites with a Varnish match are shown in parenthesis in the results. For the Nordic countries, I have quite good lists, that is, upload result lists from the probably most visited media sites in the respective countries. Remember of course, that these are generally pay-to-be-included lists, and there may exist sites with far more hits than the ones listed. For a global overview, I have used Alexa and Google’s Top 1000 lists. Now for the results. Varnish is sponsored by large Norwegian sites, so it is no big surprise that there are a lot of hits in Norway. Of the TNS Gallup top list, Varnish runs at stunning 51 of the top 100 sites. That’s 15 up since my last probe. For Denmark, I use FDIM‘s list. Sorted on page hits, we now rule 15 of the top 100, and 29 in the top 200. For Sweden, I use the KIA Index list. It is a bit harder to parse, but I think I got it right. Sorting on page hits, in the top 100, we are up to 13, and in the top 200, we find 26 sites running Varnish. Iceland is finally on the list, with one single item on Modernus’ top list. The lucky site is www.vb.is, which looks like a financial publication. I haven’t got results for Finland yet, I have to rebuild my parser, it seems. For what it’s worth, I’ll toss in Germany as well. Four sites in the Google’s top 100 sites for Germany, and 13 on the Netcraft toolbar users’ list sounds like a good start to me. And Der Spiegel and Der Zeit are well-known publications. For the Alexa’s World top 500 list, we have 17 instances of Varnish in the top 500. That is the same result as last year. Still no World domination. Google shows us a similar result, with 32 sites running Varnish in its top 1000 list. We know Facebook, the World’s most visited site, runs Varnish for several of their services, but it is hidden from my probes. All the gory details are available here. Other more or less worth mentioned sites that has been reported to use Varnish but does not show up in my lists, may be Slashdot, The Pirate Bay, e.Republik, WOWwiki, Globo.com, PCWelt.de, BlackPlanet, funnyordie.com, n-tv.de, 20minutos.es, theglobeandmail.com and hackint0sh.org, to name a few. Do you know of other famous sites running Varnish? Use the comments. [Less]

This post is a version of the slideshow presentation I did at Hack and Tell in Fort Lauderdale, Florida at The Collide Factory on Saturday, April 2, 2011. These are 5 minute talks where each slide auto-advances after fifteen seconds which limits the amount of detail that can be conveyed. A brief introduction What makes [...]

<script type="text/javascript"> </script> <script src="http://pagead2.googlesyndication.com/pagead/show_ads.js" type="text/javascript"> </script>Summary of changes from 2.1.5 to 3.0.0 Module support through VMODs. Compression ... [More] and uncompression support, including stiching together compressed ESI fragments. Preliminary streaming support, both on miss and on pass. Much improved documentation. Better default values for parameters. Varnishncsa now has custom log format support. Varnishlog, varnishncsa and varnishhist can now filter out records that match multiple expressions. Download here. [Less]

  read more

Varnish 3.0.0 beta2 was just released, and we’re aiming for 3.0.0 next week. The release date is set for Thursday the 16th (next week(June, 2011, for the potential future archive crawlers)), and several release parties are planned. Varnish Software ... [More] will be present in Santa Clara, London and Oslo, but numerous parties are planned all around the world. This will be a very special day for everyone in Varnish Software. Varnish 2.0.0 was released roughly the same week I started working at Redpill Linpro, before I was really involved with Varnish, and I still regret that I didn’t snatch one of those fancy 2.0-t-shirts. What’s new? While there are too many news to mention, there are two that stand out more than anything. The first is gzip compression and the second is the Varnish module-architecture, or simply vmods. That most of ESI has been re-factored under the hood will be evident in future releases, and the same goes for streaming delivery. Compression - “What, you don’t already have compression?!” Remember, Varnish is a caching service. Most of the time you don’t need it to do the compression, because the web server will do it for you. But there are good reasons for why you want it. ESI is the primary use-case. With ESI, Varnish needs to parse the content, and it can’t do that if it doesn’t understand the transfer encoding. With Varnish 2.1, you have to send uncompressed data to Varnish if you want to use ESI. This means you have to either deliver uncompressed content to your clients, or use yet an other service in front of Varnish. But let me get a bit technical, because Varnish 3.0′s compression is pretty awe-inspiring. So with ESI, you have multiple, individually cached elements that make up a single user-visible page. So what we could do, is glue it all together and compress it before we send it. The downside is that we’ll do the compression over and over. An alternative would be to cache the compressed result as long as the individual elements are unchanged, but that will require more space and more complexity. So what does Varnish 3.0 does? It stores the elements compressed, modifies the right gzip-bits and glues it all together on the fly, without decompressing it. If a single element is changed, only that element needs to be updated. This is probably the best solution, even if the complexity of meddling with binary gzip headers directly can lead to some pretty tricky code. I challenge you to find a solution that handles compression in a smarter way. Varnish 3.0 also does decompression. If your browser doesn’t support compression (Possibly a script or other tool, real browsers support compression), Varnish will decompress the object for you on the fly. This is an other huge improvement over Varnish 2.1. In Varnish 2.1, this is solved using Vary: and storing different copies of the same object based on compression. We can also do the same with the backend-data: If ESI needs to read the data, Varnish 3.0 can decompress it on the fly, parse it, then re-compress it before storing it. And for you, the user, the complexity is fairly non-existent. Push the button, remove that nginx(no hard feelings)server you had doing compression, ????, profit! VMODS! VCL, the Varnish Configuration Language, is a flexible way of configuring Varnish. Since it is already translated from VCL to C, the compiled and linked in to the running Varnish instance, VCL has “always” had in-line C: Anywhere you want to, you can tell the VCL compiler that this is pure C code, and pass it directly to the compiler instead of trying to translate from VCL to C first. This was mainly provided because: 1. It didn’t add any complexity and was actually less work. 2. It provided an escape chute for features we didn’t want in Varnish-proper, but were valid for some users. What features could that be? Syslog-support, geoip-integration, cryptographic verification of authorization headers, etc. It turned out to be very useful, but impractical and difficult to re-use. Since it was all glued to your VCL, it meant that you had to stick C code in the middle of your regular configuration, and it made it very hard to combine two different features if you didn’t know C yourself. And linking towards external libraries required parameter changes. Enter varnish modules. Simply put, vmods are a way of letting you do the same thing you can do with in-line C, but in a more sustainable manner. A vmod will typically have a few functions exposed to VCL, and the VCL just has to declare that it imports the vmod to use the functions without a single line of C-code in your VCL. This also means that the vmod has its own build system, own linking process, flexible development environment and are much easier to share. In the time to come, I expect us to have a community-site for vmods. I also expect that you will see a lot of minor yet important changes to Varnish during the Varnish 3.0-cycle that exposes more of the internal varnish utilities so vmods can use them. Small disclaimer, though: There is no API guarantee. We don’t wish to slow the pace of Varnish-development by restricting what we can change. That said, we wont tear things apart just to see our vmod-contributors bleed. I plan to write a blog post on vmod-hacking in the near future, so expect more detail there. Finishing words Varnish 1.0 was, as any 1.0-release is, important. I was not involved with the project back then, but as I understand it, Varnish 1 was very much tailored to a specific site, or type of site. With Varnish 2.0, Varnish became useful for anyone with a high-traffic site. The 2.0-series was a good release-series, adding a healthy mixture of bug fixes and non-intrusive features with each release. But constantly focused on real-world usage. We also saw the first Varnish User Group meeting during that time. Varnish 2.1 has been a sort of intermittent release. Director-refactoring, ESI and to a certain degree persistent storage paved the way for architectural changes that had to be done. Meanwhile, the user-base really exploded. Now, with Varnish 3.0.0 coming out, we are already seeing how useful vmods are for Varnish Software-customers like Softonic, who’re sponsoring a VMOD to allow header-modification targeted at the Set-Cookie header (which can’t be treated like the RFC2616 specifies, due to how this has been implemented historically). Doing this in a vmod allows the VCL to remain clean. It also lets us share the code that isn’t site-specific but widely useful, since it is all contained in a header vmod(Still under development). Gzip also means there are no drawbacks to using ESI with Varnish. You don’t need to add a second service to compress data. It all boils down to Varnish being a more useful part of your architecture. It’s easy to get fast and maintainable C-code in there if you need it, you can even pay someone to write just that bit for you. Without being confined to Varnish’ own road map and release cycle. There are no longer any downsides to using ESI. It’s fast. It’s free software. There are professional service and support offerings available. Varnish follow standards – unless you tell it not to. And so forth. I’m not usually one to pat myself too much on the back, but as I’m writing this, I feel proud to be part of the team that gives you Varnish 3.0. If you’re in Oslo, I’ll see you next Thursday! [Less]

Posted on 2011-06-09 Varnish 3.0.0 beta2 was just released (http://www.varnish-cache.org/lists/pipermail/varnish-announce/2011-June/000032.html), and we're aiming for 3.0.0 next week. The release date is set for Thursday the 16th (next week(June ... [More] , 2011, for the potential future archive crawlers)), and several release parties are planned. all around the world (http://v3party.varnish-cache.org/). This will be a very special day for everyone in Varnish Software. Varnish 2.0.0 was released roughly the same week I started working at Redpill Linpro, before I was really involved with Varnish, and I still regret that I didn't snatch one of those fancy 2.0-t-shirts. What's new? While there are too many news to mention, there are two that stand out more than anything. The first is gzip compression and the second is the Varnish module-architecture, or simply vmods. That most of ESI has been re-factored under the hood will be evident in future releases, and the same goes for streaming delivery. Compression "What, you don't already have compression?!" Remember, Varnish is a caching service. Most of the time you don't need it to do the compression, because the web server will do it for you. But there are good reasons for why you want it. ESI is the primary use-case. With ESI, Varnish needs to parse the content, and it can't do that if it doesn't understand the transfer encoding. With Varnish 2.1, you have to send uncompressed data to Varnish if you want to use ESI. This means you have to either deliver uncompressed content to your clients, or use yet an other service in front of Varnish. But let me get a bit technical, because Varnish 3.0's compression is pretty awe-inspiring. So with ESI, you have multiple, individually cached elements that make up a single user-visible page. So what we could do, is glue it all together and compress it before we send it. The downside is that we'll do the compression over and over. An alternative would be to cache the compressed result as long as the individual elements are unchanged, but that will require more space and more complexity. So what does Varnish 3.0 does? It stores the elements compressed, modifies the right gzip-bits and glues it all together on the fly, without decompressing it. If a single element is changed, only that element needs to be updated. This is probably the best solution, even if the complexity of meddling with binary gzip headers directly can lead to some pretty tricky code. I challenge you to find a solution that handles compression in a smarter way. Varnish 3.0 also does decompression. If your browser doesn't support compression (Possibly a script or other tool, real browsers support compression), Varnish will decompress the object for you on the fly. This is an other huge improvement over Varnish 2.1. In Varnish 2.1, this is solved using Vary: and storing different copies of the same object based on compression. We can also do the same with the backend-data: If ESI needs to read the data, Varnish 3.0 can decompress it on the fly, parse it, then re-compress it before storing it. And for you, the user, the complexity is fairly non-existent. Push the button, remove that nginx(no hard feelings)server you had doing compression, ????, profit! VMODS! VCL, the Varnish Configuration Language, is a flexible way of configuring Varnish. Since it is already translated from VCL to C, the compiled and linked in to the running Varnish instance, VCL has "always" had in-line C: Anywhere you want to, you can tell the VCL compiler that this is pure C code, and pass it directly to the compiler instead of trying to translate from VCL to C first. This was mainly provided because: It didn't add any complexity and was actually less work. It provided an escape chute for features we didn't want in Varnish-proper, but were valid for some users. What features could that be? Syslog-support, geoip-integration, cryptographic verification of authorization headers, etc. It turned out to be very useful, but impractical and difficult to re-use. Since it was all glued to your VCL, it meant that you had to stick C code in the middle of your regular configuration, and it made it very hard to combine two different features if you didn't know C yourself. And linking towards external libraries required parameter changes. Enter varnish modules. Simply put, vmods are a way of letting you do the same thing you can do with in-line C, but in a more sustainable manner. A vmod will typically have a few functions exposed to VCL, and the VCL just has to declare that it imports the vmod to use the functions without a single line of C-code in your VCL. This also means that the vmod has its own build system, own linking process, flexible development environment and are much easier to share. In the time to come, I expect us to have a community-site for vmods. I also expect that you will see a lot of minor yet important changes to Varnish during the Varnish 3.0-cycle that exposes more of the internal varnish utilities so vmods can use them. Small disclaimer, though: There is no API guarantee. We don't wish to slow the pace of Varnish-development by restricting what we can change. That said, we wont tear things apart just to see our vmod-contributors bleed. I plan to write a blog post on vmod-hacking in the near future, so expect more detail there. Finishing words Varnish 1.0 was, as any 1.0-release is, important. I was not involved with the project back then, but as I understand it, Varnish 1 was very much tailored to a specific site, or type of site. With Varnish 2.0, Varnish became useful for anyone with a high-traffic site. The 2.0-series was a good release-series, adding a healthy mixture of bug fixes and non-intrusive features with each release. But constantly focused on real-world usage. We also saw the first Varnish User Group meeting during that time. Varnish 2.1 has been a sort of intermittent release. Director-refactoring, ESI and to a certain degree persistent storage paved the way for architectural changes that had to be done. Meanwhile, the user-base really exploded. Now, with Varnish 3.0.0 coming out, we are already seeing how useful vmods are for Varnish Software-customers like header vmod (https://github.com/KristianLyng/libvmod-header) (Still under development). Gzip also means there are no drawbacks to using ESI with Varnish. You don't need to add a second service to compress data. It all boils down to Varnish being a more useful part of your architecture. It's easy to get fast and maintainable C-code in there if you need it, you can even pay someone to write just that bit for you. Without being confined to Varnish' own road map and release cycle. There are no longer any downsides to using ESI. It's fast. It's free software. There are professional service and support offerings available (http://www.varnish-software.com). Varnish follow standards - unless you tell it not to. And so forth. I'm not usually one to pat myself too much on the back, but as I'm writing this, I feel proud to be part of the team that gives you Varnish 3.0. If you're in Oslo, I'll see you next Thursday! Comments [Less]

This post is a version of the slideshow presentation I did at Hack and Tell in Fort Lauderdale, Florida at The Whitetable Foundation on Saturday, June 4, 2011. Briefly, I created a Plugin that enabled Fragment Caching with WordPress and Varnish. The problem we ran into with normal page caching methods was related to the [...]

Hi. As an experiment, we've been experimenting with creating short videos about Varnish. We're not quite sure what the format should be so bear with us. First up is Poul Henning Kamp giving us a his view on what he thinks are the two most important ... [More] features of Varnish Cache 3.0. <object data="http://www.youtube.com/v/AN-Gb1jYR30&amp;autoplay=1&amp;fs=1" height="400" type="application/x-shockwave-flash" width="479"> <param name="movie" value="http://www.youtube.com/v/AN-Gb1jYR30&amp;autoplay=1&amp;fs=1"/> <param name="wmode" value="transparent"/> <param name="allowFullScreen" value="true"/> </object> read more [Less]

We're growing!  We're looking for two technical people in Oslo. The specifications aren't very specific. Basically, we're looking for smart people that might help us with one or more of the following: read more

read more