Varnish

Varnish is known to be quite fast. But how fast? My very first Varnish-job was to design a stress testing scheme, and I did so. But it was never really able to push things to the absolute max. Because Varnish is quite fast. In previous posts I’ve ... [More] written I about hitting 27k requests per second on an aging Opteron (see http://kristianlyng.wordpress.com/2009/10/19/high-end-varnish-tuning/) and then again about reaching 143k requests per second on a more modern quad-core using a ton of test-clients. (see http://kristianlyng.wordpress.com/2010/01/13/pushing-varnish-even-further/). Recently, we were going to do a stress test at a customer setup before putting it live. The setup consisted of two dual Xeon x5670 machines. The X5670 is a 2.93GHz six-core cpu with hyperthreading, giving these machines 12 cpu-cores and 24 cpu threads. Quite fast. During our tests, I discovered some httperf secrets (sigh…). And was able to push things quite far. This is what we learned. The hardware and software As described above, we only had two machines for the test. One is the target and one would be the originating machine. The network was gigabit. Varnish 2.1.3 on 64bit Linux. Httperf for client load. The different setups to be tested Our goal was not to reach the maximum limits of Varnish, but to ensure the site was ready for production. That’s quite tricky on many accounts. The machines were originally configured with heartbeat and haproxy. One test I’m quite fond of is site traversal while hitting a “hot” set at the same time. The intention is to test how your site fares if a ruthless search bot hits your site. Does your front page slow down? As far as Varnish goes, it tests the LRU-capabilities and how it deals with possibly overloaded backend servers. We also switched out haproxy in favor of a dual-varnish setup. Why? Two reasons: 1. Our expertise is within the realm of Varnish. 2. Varnish is fast and does keep-alive. When testing a product like Varnish we also have to take the balance between requests and connections into account. You’ll see shortly that this is very important. During our tests, we also finally got httperf to stress the threading model of Varnish. With a tool like siege, concurrency is defined by the threading level. That’s not the case with httperf, and we were able to do several thousand _concurrent_ connections. As the test progressed, we reduced the size of the content and it became more theoretical in nature. As the specifics of the backends is not that relevant, I’ll keep to the Varnish-specific bits for now. I ended up using a 301 redirect as a test this time. Mostly because it was there. Towards the end, I had to remove various varnish-headers to free up bandwidth. Possible bottlenecks The most obvious bottleneck during a test like this is bandwidth. That is the main reason for reducing the size of objects served during testing. An other bottleneck is how fast your web servers are. A realistic test requires cache misses, and cache misses requires responsive web servers. Slow clients are a problem too. Unfortunately testing that synthetically isn’t easy. Lack of test-clients has been an issue in the past, but we’ve solved this now. CPU? Traditionally, the cpu-speed isn’t much of an issue with Varnish, but when you rule out slow backends, bandwidth and slow clients, the cpu is the next barrier. One thing that’s important in this test is that the sheer amount of parallel execution threads is staggering. My last “big” test had 4 execution threads, this one has 24. This means we get to test contention points that only occur if you have  massive parallelization. The most obvious bottleneck is the acceptor-thread. The thread charged with accepting connections and delegating them to a thread. Even if multiple thread pools is designed to leverage this problem, the actual accept()-call is done in a single thread of execution. Connections As Artur Bergman of Wikia has already demonstrated, the amount of TCP connections Varnish is able to accept per second is currently our biggest bottleneck. Fortunately for most users, Artur’s work-load is very different from most other Varnish users. We (Varnish Software) typically see a 1:10 ration between connections and requests. Artur suggested he’s closer to 1:3 or 1:4. During this round of tests I was easily able to reach about 40k connections/s. However, going much above that is hard. For a “normal” workload, that would allow 400k requests/second, which is more than enough. However, it should be noted that the accept-rate goes somewhat down as the general load increases. It was interesting to note that this was largely unaffected by having two varnishes in front of each other. This essentially confirms that the acceptor is the bottleneck. There wasn’t much we could do to affect this limit either. Increasing the listen_depth isn’t going to help you in a synthetic test. The listen_depth defines how many outstanding connections is allowed to queue up before the kernel starts dropping them. In the real world, the connection-rate will be sporadic and on an almost-overloaded system, it might help to increase the listen depth, but in a synthetic test the connection rate is close to constant. That means increasing the listen depth just means there’s a bigger queue to fill – and it will fill anyway. The number of thread pools had little effect too. By the time the connection is delegated to a thread pool, it’s already past the accept() bottleneck. Now, keep in mind that this is still a staggering number. But it’s also an obvious bottleneck for us. Request rate The raw request rate is essentially defined by how big the request is compared to bandwidth, how much CPU power is available and how fast you can get the requests into a thread. As we have already established that the acceptor-thread is a bottleneck, we needed to up the number of requests per connection. I tested mostly with a 1:10 ratio. This is the result of one such test: The above image shows 202832 requests per second while doing roughly 20 000 connections/s. Quite a number. It proved difficult to exceed this. At about 226k req/s the bandwidth limit of 1gbit was easily hit. To reach that, I had to reduce the connection-rate somewhat. The main reason for that, I suspect, is increased latency when the network is saturated. At this point, Varnish was not saturating the CPU. It still had 30-50% idle CPU power. Just for kicks and giggles, I wanted to see how far we could really get, so I threw in a local httperf, thereby ignoring large parts of the network issue. This is a screenshot of Varnish serving roughly 1gbit traffic over network and a few hundred mbit locally: So that’s 275k requests/s. The connection rate at that point was lousy, so not very interesting. And because httperf was running locally, the load on the machine wasn’t very predictable. Still, the machine was snappy. But what about the varnish+varnish setup? The above numbers are for a single Varnish server. However, when we tested with varnish as a load balancer in front of Varnish, the results were pretty identical – except divided by two. It was fairly easy to do 100k requests/second on both the load balancer and the varnish server behind it – even though both were running on the same machine. The good thing about Varnish as load balancer is the keep alive-nature, speed and flexibility. The contention-point of Varnish is long before any balancing is actually done, so you can have a ton of logic in your “Varnish Load balancer” without worrying about load increasing with complexity. We did, however, discover that the number of HTTP header overflows would spike on the second varnish server. We’re investigating this. The good news is that it was not visible on the user-side. The next step I am re-doing part of our internal test infrastructure (or rather: shifting it around a bit) to test the acceptor thread regularly. I also discovered an assert issue during some sort of race at around 220k req/s, but that was only under certain very very specific situations. It was not possible to reproduce on anything that wasn’t massively parallel and almost saturated on CPU. We’re also constantly improving our test routines both for customer setups and internal quality assurance on the Varnish code base. I’ve already written several load-generating scripts for httperf to allow us to test even more realistic work loads on a regular basis. What YOU should care about The only thing that made a real difference while tuning Varnish was the number of threads. And making sure it actually caches. Beyond that, it really doesn’t matter much. Our defaults are good. However, keep in mind that this does NOT address what happens when you start hitting disk. That’s a different matter entirely. [Less]

Summary of changes from 2.1.3 to 2.1.4 A bug in the binary heap layout caused inflated object counts, this has been fixed. Much more comprehensive documentation. A DNS director that uses DNS lookups for choosing which backend to route requests to ... [More] has been added. The client director now uses the variable client.identity for choosing which backend to send a given request to. String representation of now, making it easier to add Expires headers Portability fixes for Solaris. Various bug fixes. Download here. ——————————————————– Take a look at notable new features in Varnish Cache 2.1.4 here. The DNS Director Sponsored by Globo and MercadoLibre the DNS director allows you to select a backend based on the content of a zone in DNS. If you, like the sponsors do, have a really large amount of virtual hosts backends, it becomes a real hassle to maintain and deploy the ever changing configuration. Always miss Varnish has had the option to PURGE content from it’s initial release. The downside of this is that some poor user will have to come along and request the object and if that object takes time to generate the poor user might have to wait awhile. With always miss you can instruct Varnish to fetch a new version of the page. When the page is fetched the old version will be discarded and your site will be updated. Please remember that if your sites uses Vary you will have to do this for all the different variants of the page. The Always miss feature was contributed by Artur Bergman aka “sky” on #varnish, the CTO of wikia.com. The client director The client director, which was introduced in 2.1.0, you could make Varnish hash the IP address of the client and choose a backend according to the result. For people having a load balancer in front of Varnish this didn’t work so well. You can now set the input value directly in VCL giving you the option to use any header or string to choose your backend. Those of you with load balancers setting a session cookie might use the cookie to direct users to the right backend. [Less]

google_ad_client = "pub-8208356787225078"; google_ad_width = 468; google_ad_height = 60; google_ad_format = "468x60_as"; google_ad_type = "text_image"; google_color_border = "FFFFFF"; google_color_link = "0000FF"; google_color_text = "000000"; ... [More] google_color_bg = "FFFFFF"; google_color_url = "008000"; Use this vcl_fetch to add a custom Expires header to objects. This example adds one day (86400 seconds). sub vcl_fetch { set beresp.grace = 4h; [...] [Less]

Yesterday, we have been down for 30 minutes because one of our memcache server has been shutdown during a maintenance. The anti-scraping system was not protected against an error like this. When developing, we done a test with the memcache process ... [More] halted but not the memcache server itself. (unpluged of lan) And it is not the same error for libmemcached. Furthermore, the 0.39 version of libmemcached has a bug for some cases where connect would have issues with timeout. We have deployed libmemcached 0.43 and we have done few adjustments in our code: //Connect timeout 10ms rc = memcached_behavior_set(memc, MEMCACHED_BEHAVIOR_CONNECT_TIMEOUT, 10); if (rc != MEMCACHED_SUCCESS) syslog(LOG_INFO, “Memcached error: %s”, memcached_strerror(memc,rc) ); # Copyright (c) 2010 OUESTFRANCE-MULTIMEDIA # All rights reserved. # # Author: Vincent ROBERT <[email protected]> # Thanks to: Tony FOUCHARD <[email protected]> # Thanks to: Cyrille MAHIEUX <[email protected]> # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # 1. Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in the # documentation and/or other materials provided with the distribution. # # THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE # ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS # OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) # HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. C{ #define HOSTNAME_MAX_SIZE 200 #include <string.h> #include <libmemcached/memcached.h> #include <syslog.h> #include <netdb.h> #include <sys/types.h> #include <sys/socket.h> }C sub check_ip { set bereq.http.whitelisted = "0"; if (bereq.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|js|css|swf|xml|flv)(\?(.)*)?$") { set bereq.http.whitelisted = "1"; } else { if (bereq.http.host == "internalwebSite") { set bereq.http.whitelisted = "1"; } elsif ((bereq.http.X-Forwarded-For == "IPaddress1") || (bereq.http.X-Forwarded-For == "IPaddress2")) { set bereq.http.whitelisted = "1"; } } if (bereq.http.whitelisted=="0") { C{ void ip_to_hostname(char *ipaddr, char *hostname) { struct addrinfo * result; int successfull=0; int error; error = getaddrinfo(ipaddr, NULL, NULL, &result); if (error==0) { error = getnameinfo(result->ai_addr, result->ai_addrlen, hostname, HOSTNAME_MAX_SIZE, NULL, 0, 0); if (error==0) { successfull=1; } } freeaddrinfo(result); if (!successfull){ memcpy(hostname, ipaddr, HOSTNAME_MAX_SIZE*sizeof(char)); hostname[HOSTNAME_MAX_SIZE-1]=0; } } memcached_server_st *servers = NULL; memcached_st *memc; memcached_return rc; char hostname[HOSTNAME_MAX_SIZE]=""; char key[500]= "varnish_"; strcat(key, VRT_GetHdr(sp, HDR_REQ, "\005host:")); strcat(key, "_"); strcat(key, VRT_IP_string(sp, VRT_r_client_ip(sp))); memc= memcached_create(NULL); servers= memcached_server_list_append(servers, "memcacheServer", 11211, &rc); rc= memcached_server_push(memc, servers); if (rc == MEMCACHED_SUCCESS) { //Connect timeout 10ms rc = memcached_behavior_set(memc, MEMCACHED_BEHAVIOR_CONNECT_TIMEOUT, 10); uint64_t intval; rc= memcached_increment(memc, key, strlen(key), (uint64_t)1, &intval); if (rc != MEMCACHED_SUCCESS) { rc= memcached_set(memc, key, strlen(key), "1", 1, (time_t)60, (uint32_t)0); if (rc != MEMCACHED_SUCCESS) syslog(LOG_INFO, "Memcached error: %s", memcached_strerror(memc,rc) ); } else if (intval>30) { ip_to_hostname(VRT_IP_string(sp, VRT_r_client_ip(sp)), hostname); VRT_SetHdr(sp, HDR_BEREQ, "\013X-Scraping:", "1", vrt_magic_string_end); VRT_SetHdr(sp, HDR_BEREQ, "\013X-Hostname:", hostname, vrt_magic_string_end); VRT_SetHdr(sp, HDR_BEREQ, "\013X-HostOrig:", VRT_GetHdr(sp, HDR_REQ, "\005host:"), vrt_magic_string_end); if (intval<300) { rc= memcached_set(memc, key, strlen(key), "500", 3, (time_t)600, (uint32_t)0); if (rc != MEMCACHED_SUCCESS) syslog(LOG_INFO, "Memcached error: %s", memcached_strerror(memc,rc) ); } } } memcached_free(memc); memcached_server_list_free(servers); }C if (bereq.http.X-Scraping=="1") { if (!(bereq.http.X-Hostname ~ "(\.(yahoo\.(net|com))|(exabot\.com)|(googlebot\.com)|(search\.msn\.com))$")) { C{ syslog(LOG_INFO, "Scraping detected from %s %s on %s with %s",VRT_IP_string(sp, VRT_r_client_ip(sp)), VRT_GetHdr(sp, HDR_BEREQ, "\013X-Hostname:"), VRT_GetHdr(sp, HDR_REQ, "\005host:"), VRT_GetHdr(sp, HDR_REQ, "\013User-Agent:")); }C } } } } sub vcl_miss { call check_ip; fetch; } sub vcl_pipe { call check_ip; pipe; } sub vcl_pass { call check_ip; pass; } [Less]

Following on from previous post about VCL tweaks to improve hitrate; there are occasions when a website should not be served from both www.foobar.com and http://foobar.com. In some instances Google will deem the content to be duplicate copy of each other and a website can suffer from dupe content penalty. read more

Show the most often-made requests to the backend varnishtop -b -i TxURL Show a histogram for the past 1000 requests varnishhist Displays all Varnish traffic for a specific client varnishlog -c -o ReqStart Provides an overview of the stats for the current Varnish instance varnishstat

The default Varnish config for Pressflow by Four Kitchens is an excellent starting point and gets you up and running with relatively little pain and effort. Having done a fair amount of Varnish tweaking for my personal and work websites, I came ... [More] across a couple of varnish tweaks that resulted in a phenominal improvement in Varnish Hit rate. Vary User Agent read more [Less]

Some time in the middle of the night before 2.1.0, I implemented a director that used the client IP to direct traffic. The goal was to direct the same machine to the same backend – cheap session stickyness. Took about an hour to hack up and an other ... [More] to pretty up. Some time during roughly the same night, PHK refactored all the director infrastructure and in the process implemented a client director and a hash director as a sort of side-job. PHK’s version obviously entered trunk and mine never saw the light of day (possibly because it was December and there isn’t much daylight to see here in December anyway). At least the duplicated effort wasn’t significant. We’ve kept that somewhat hidden, mostly because the actual code for the hash and client director is about a screenfull of text once the VCL-bits are taken care of. And it’s such a small feature. Truth be told, they both live within the random director and are just special exceptions. The VCL syntax is the same, except now it’s called a client director instead of random. The first up, the client director, will use the ascii-representation of the client IP to pick a “random” backend. (see why it’s the ascii representation here: http://varnish-cache.org/trac/browser/trunk/varnish-cache/bin/varnishd/cache_dir_random.c?rev=5304#L99) There isn’t much to say about it. Your VCL will be the same as if it was a random director. If the “canonical backend” for the client is sick, the regular random algorithm will be used to pick a backend from the healthy ones. In Varnish 2.1.4 you will also get a “client.identifier” VCL variable that you can use to tell varnish what identifies this as a unique client. In theory, you could pass a cookie to it and avoid wondering what happens if an IP changes. If it is set, the client director will use that instead of the IP. The only problem I have come up with using that approach is bootstrapping. He very first request a client makes will have no cookies, so no data can be passed to client.identifier. The backend will presumably set a session-cookie of some sort so the next request could use that session cookie, but that means the first and second request is likely to go to different backends, even if the following requests will all go to the same backend. Typical load balancers solve this by just setting the cookie them self. I suspect that is what it will come to, and I also suspect that it will be a lot nicer to do that in Varnish 3.0.0 when vmods are in place which could easily deal with these sort of things without messing up your regular VCL… But until then, try out the client director for normal IP’s at least. The client director is available in 2.1.3 (I think it’s even in 2.1.0, though I can’t remember. Definitely not documented in 2.1.0, though). http://www.varnish-cache.org/docs/trunk/reference/vcl.html#the-client-director I’m very interested in your take on how to deal with sessions, if just basing it on IP is “close enough” and if you think the client director will be able to solve most session-stickyness scenarios. [Less]

varnish makes it possible to scale as needed, without expensive monthly access fees, and without outrageous per GB transfer fees. As your traffic needs ebb and flow, you decommission your monthly server rentals, or you add more, adjusting your DNS and your config distribution to suit the situation

After having run janaksingh.com website on Drupal 6 with Apache+PHP+MySql, I wanted to move to Pressflow so I could harness the added advantages of Varnish. This is not an in-depth installation guide or a discussion about Varnish or Pressflow, but ... [More] quick setup commands for my own reference. Please see links at the end if you wish to explore Varnish or Pressflow setup in greater depth. I wanted: Varnish > APC > Apache > Pressflow setup.. read more [Less]