Optimize the caching of a multilingual Plone site

Goals and initial setup

Our Plone site should be as fast as possible for the anonymous visitor. In other words, given a warm cache, anonymous visitors should almost never have to deal with Zope directly.

The initial setup comprises Plone 3.3.5 with Products.CacheSetup and Products.LinguaPlone for the backend, Varnish 2.1.2 for caching, and nginx as the frontend.

The languages in use are English and French.

Problems and solutions

After having optimized anonymous pages for caching, removing unused dynamic content, and configured CacheSetup and LinguaPlone, here are the problems we met.

1. Accept-Language header diversity

When caching multilingual content one must add the Accept-Language header to the list of content varying headers (Vary: Accept-Language) and that will cause the caching of a different page for every variation in the header.

Given the Accept-Language header can assume an infinite number of values we're likely to waste a lot of caching space.

We would need to reduce Accept-Language's possible variations to the few values needed to represent our supported languages, that is reduce "fr,fr-fr;q=0.8,en;q=0.6,en-us;q=0.4,it;q=0.2" to "fr" and "da,en-gb;q=0.8,en;q=0.7" to "en".

That's exactly what this great VCL extension for Varnish from Cosimo Streppone is doing for us.

2. The I18N_LANGUAGE cookie

When Varnish gets a Set-Cookie header from the backend, it rightfully does not cache the document.

The problem is that Products.PloneLanguageTool always try to create the I18N_LANGUAGE cookie if it doesn't exist yet. As a result a visitor with cookies disabled will never get a page from cache given that all the Zope generated pages will bear the Set-Cookie header...

We solve this by patching LanguageTool so that it only sets the cookie on language change:

from Products.PloneLanguageTool import LanguageTool

def _setLanguageCookie(self, lang=None, REQUEST=None, noredir=None):
        """Sets a cookie for overriding language negotiation."""
        res = None
        if lang and lang in self.getSupportedLanguages():
            if lang != self.getLanguageCookie():
                if lang not in self.getRequestLanguages():
                    self.REQUEST.RESPONSE.setCookie('I18N_LANGUAGE', lang, path='/')
            res = lang
        if noredir is None:
            if REQUEST:
                REQUEST.RESPONSE.redirect(REQUEST['HTTP_REFERER'])
        return res

LanguageTool.setLanguageCookie = _setLanguageCookie

3. I18N_LANGUAGE and Etags

Another problem arises when a page is cached after a visitor actively changed its default language.

The page is indeed cached by CacheSetup and then Varnish with an Etag containing the browser's language (Accept-Language) rather than the effective language (I18N_LANGUAGE cookie).

To solve this we integrate the I18N_COOKIE to the Accept-Language reduction mechanism in Varnish:

# vcl.conf
sub normalize_accept_language {
    C{
        vcl_rewrite_accept_language(sp);
    }C
    if (req.http.Cookie && req.http.Cookie ~ "I18N_LANGUAGE=") {
        set req.http.X-Varnish-I18N-Language = regsuball(req.http.Cookie,
                            "(^.*I18N_LANGUAGE=)%22([^ ][^;]*)%22(;|$)(.*$)",
                            "\2");
        if (!req.http.X-Varnish-I18N-Language ~ "^(fr|en)$") {
            set req.http.X-Varnish-I18N-Language = "en";
        }
        if (req.http.X-Varnish-I18N-Language != req.http.X-Varnish-Accept-Language) {
            set req.http.X-Varnish-Accept-Language = req.http.X-Varnish-I18N-Language;
        }
    }
    set req.http.Accept-Language = req.http.X-Varnish-Accept-Language;
}

We then call the function from vcl_receive:

sub vcl_receive {
    (....)
    call normalize_accept_language;
    (...)
}

4. Accept-Encoding

After a series of tests, deciding to save memory instead of CPU cycles, we chose not to compress pages with CacheSetup, avoiding Vary: Accept-Encoding between Varnish and Plone, but rather to let nginx handle the compression on-the-fly:

# nginx conf
gzip  on;
gzip_disable "MSIE [1-6]\.(?!.*SV1)";
gzip_vary on;
gzip_proxied any;
gzip_types text/plain text/css application/x-javascript;

5. URL format in the cache

For Varnish and Plone to speak the same language while purging cache entries, we need to handle the conversion to the VirtualHostMonster format beforehand, that at the nginx level:

# nginx conf
location / {
    include /etc/nginx/proxy_base;
    rewrite ^(.*)$ /VirtualHostBase/http/$server_name:443/your_plone_site/VirtualHostRoot$1 break;
    proxy_pass http://your_varnish_host_and_port;
}

Varnish now only handles VHM formatted URLs.

Final setup

So here's our final setup:

nginx

This is the system's entry point and it handles:

• HTTPS
• VHM syntax URLs rewriting
• on-the-fly compression

Varnish

Besides caching it handles:

• The Accept-Language header and I18N_LANGUAGE cookie reduction
• optionally the load balancing between Zope instances
  

Plone

Except the small LanguageTool patch, and CacheSetup and LinguaPlone configuration, no modification is necessary.

Benchmark

Here's a local benchmark:

plone@evax:~$ ab -c 100 -n 1000 http://www.evax.fr/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.evax.fr (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nginx
Server Hostname:        www.evax.fr
Server Port:            80

Document Path:          /
Document Length:        19287 bytes

Concurrency Level:      100
Time taken for tests:   0.369 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      19822264 bytes
HTML transferred:       19287000 bytes
Requests per second:    2712.72 [#/sec] (mean)
Time per request:       36.863 [ms] (mean)
Time per request:       0.369 [ms] (mean, across all concurrent requests)
Transfer rate:          52511.92 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.8      0       8
Processing:     4   34   5.2     36      41
Waiting:        4   34   5.2     36      41
Total:         12   35   4.0     36      41

Percentage of the requests served within a certain time (ms)
  50%     36
  66%     37
  75%     38
  80%     38
  90%     38
  95%     38
  98%     38
  99%     39
 100%     41 (longest request)

Feedback

If you want to participate, please use the contact form.

Since the original publication of this paper we've seen in our logs traces from readers trying to verify the above benchmark.

We insist on the fact that the test has been run locally, and that the limitations they may have observed are probably related to the bandwidth of the connection they're using for benchmarking:

user@somemachine ~ $ ab -c 100 -n 1000 http://www.evax.fr/
(...)
Requests per second:    137.94 [#/sec] (mean)
(...)
Transfer rate:          2753.08 [Kbytes/sec] received

In the test above, indeed, the 2753.08 Kbytes/sec transfer rate observed is extremely near from the maximum rate of the connection:

plone@evax ~/iperf-3.0b4/src $ ./iperf3 -c somemachine
Connecting to host XXX.XXX.XXX.XXX, port 5201
[  5] local 92.243.9.66 port 46131 connected to XXX.XXX.XXX.XXX port 5201
[ ID] Interval       Transfer     Bandwidth
      Sent
[  5] 0.00-5.00 sec  17.4 MBytes  29.2 Mbits/sec
      Received
[  5] 0.00-5.00 sec  17.0 MBytes  28.5 Mbits/sec

iperf Done.