Professional Documents
Culture Documents
Polipo
Polipo
Juliusz Chroboczek
Polipo is a caching web proxy designed to be used as a personal cache or a cache shared among
a few users.
Copyright
c 2003 – 2010 by Juliusz Chroboczek.
i
Table of Contents
1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 The web and HTTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Proxies and caches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.3 Latency and throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.4 Network traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.4.1 Persistent connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4.2 Pipelining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4.3 Poor Man’s Multiplexing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.5 Caching partial instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.6 Other requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.7 Other HTTP proxies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.7.1 Harvest and Squid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.7.2 The Apache proxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.7.3 WWWOFFLE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.7.4 Junkbuster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.7.5 Privoxy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.7.6 Oops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Running Polipo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1 Starting Polipo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.2 Running as a daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.3 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Configuring your browser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3 Stopping Polipo and getting it to reload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.4 The local web server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.4.1 The web interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4 Caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.1 Cache transparency and validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.1.1 Tuning validation behaviour . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.1.2 Further tweaking of validation behaviour . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.2 The in-memory cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.3 The on-disk cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.3.1 Asynchronous writing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.3.2 Purging the on-disk cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.3.3 Format of the on-disk cache. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.3.4 Modifying the on-disk cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5 Memory usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
5.1 Chunk memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
5.2 Malloc allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
5.3 Limiting Polipo’s memory usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
5.3.1 Limiting chunk usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
5.3.2 Limiting object usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
5.3.3 OS usage limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Copying . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Variable index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Concept index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Chapter 1: Background 1
1 Background
was that a separate TCP connection (“virtual circuit” for them telecom people) was created for
every entity transferred.
Opening multiple TCP connections has significant performance implications. Obviously,
connection setup and teardown require additional packet exchanges which increase network
usage and, more importantly, latency.
Less obviously, TCP is not optimised for that sort of usage. TCP aims to avoid network
congestion, a situation in which the network becomes unusable due to overly aggressive traffic
patterns. A correct TCP implementation will very carefully probe the network at the beginning
of every connection, which means that a TCP connection is very slow during the first couple of
kilobytes transferred, and only gets up to speed later. Because most HTTP entities are small
(in the 1 to 10 kilobytes range), HTTP/0.9 uses TCP where it is most inefficient.
1.4.2 Pipelining
With persistent connections it becomes possible to pipeline or stream requests, i.e. to send
multiple requests on a single connection without waiting for the replies to come back. Because
this technique gets the requests to the server faster, it reduces latency. Additionally, because
multiple requests can often be sent in a single packet, pipelining reduces network traffic.
Pipelining is a fairly common technique1 , but it is not supported by HTTP/1.0. HTTP/1.1
makes pipelining support compulsory in every server implementation that can use persistent
connections, but there are a number of buggy servers that claim to implement HTTP/1.1 but
don’t support pipelining.
Polipo carefully probes for pipelining support in a server and uses pipelining if it believes
that it is reliable. Polipo also deeply enjoys being pipelined at by a client2 .
keeps all resource meta-data in memory, it requires a fair amount of RAM in order to manipulate
a reasonably sized cache.
Squid doesn’t cache partial instances, and has trouble with instances larger than available
memory4 . If a client connection is interrupted, Squid has to decide whether to continue fetching
the resource (and possibly waste bandwidth) or discard what it already has (and possibly waste
bandwidth).
Some versions of squid would, under some circumstances, pipeline up to two outgoing requests
on a single connection. At the time of writing, this feature appears to have been disabled in the
latest version.
Squid’s developers have decided to re-write it in C++.
1.7.3 WWWOFFLE
WWWOFFLE, an elegant personal proxy, is the primary model for Polipo.
WWWOFFLE has more features than can be described here. It will censor banner ads, clean
your HTML, decorate it with random colours, schedule fetches for off-peak hours.
Unfortunately, the HTTP traffic that WWWOFFLE generates is disgusting. It will open a
connection for every fetch, and forces the client to do the same.
WWWOFFLE only caches complete instances.
I used WWWOFFLE for many years, and frustration with WWWOFFLE’s limitations is the
main reason why I started Polipo in the first place.
1.7.4 Junkbuster
Junkbuster is a simple non-caching web proxy designed to remove banner ads and cookies. It was
the main model for WWWOFFLE’s (and therefore Polipo’s) header and ad-removing features.
Junkbuster’s HTTP support is very simple (some would say broken): it doesn’t do persistent
connections, and it breaks horribly if the client tries pipelining. Junkbuster is no longer being
maintained, and has evolved into Privoxy.
1.7.5 Privoxy
Privoxy is the current incarnation of Junkbuster. Privoxy has the ability to randomly modify
web pages before sending them to the browser — for example, remove ‘<blink>’ or ‘<img>’ tags.
Just like its parent, Privoxy cannot do persistent connections. Under some circumstances,
it will also buffer whole pages before sending them to the client, which significantly adds to its
latency. However, this is difficult to avoid given the kinds of rewriting it attempts to perform.
4
Recent versions of Squid support instances larger than available memory by using a hack that the authors
call a “sliding window algorithm”.
Chapter 1: Background 5
1.7.6 Oops
Oops is a caching web proxy that uses one thread (lightweight process) for every connection.
This technique does cost additional memory, but allows good concurrency of requests while
avoiding the need for complex non-blocking programming. Oops was apparently designed as a
wide-scale shared proxy.
Although Oops’ programming model makes it easy to implement persistent connections,
Oops insists on opening a separate connection to the server for every single resource fetch,
which disqualifies it from production usage.
Chapter 2: Running Polipo 6
2 Running Polipo
2.1.1 Configuration
There is a number of variables that you can tweak in order to configure Polipo, and they
should all be described in this manual (see [Variable index], page 24). You can display the
complete, most up-to-date list of configuration variables by using the ‘-v’ command line flag or
by accessing the “current configuration” page of Polipo’s web interface (see Section 2.4.1 [Web
interface], page 8). Configuration variables can be set either on the command line or else in the
configuration file given by the ‘-c’ command-line flag.
Configuration variables are typed, and ‘-v’ will display their types. The type can be of one
of the following:
• ‘integer’ or ‘float’: a numeric value;
• ‘boolean’: a truth value, one of ‘true’ or ‘false’;
• ‘tristate’: one of ‘false’, ‘maybe’ or ‘true’;
• ‘4-state’, one of ‘false’, ‘reluctantly’, ‘happily’ or ‘true’;
• ‘5-state’, one of ‘false’, ‘reluctantly’, ‘maybe’, ‘happily’ or ‘true’;
• ‘atom’, a string written within double quotes ‘"’);
• ‘list’, a comma-separated list of strings;
• ‘intlist’, a comma-separated list of integers and ranges of integers (of the form ‘n–m’).
The configuration file has a very simple syntax. All blank lines are ignored, as are lines
starting with a hash sign ‘#’. Other lines must be of the form
var = val
where var is a variable to set and val is the value to set it to.
It is possible to change the configuration of a running polipo by using the local configuration
interface (see Section 2.4.1 [Web interface], page 8).
2.1.3 Logging
When it encounters a difficulty, Polipo will print a friendly message. The location where these
messages go is controlled by the configuration variables logFile and logSyslog. If logSyslog
is true, error messages go to the system log facility given by logFacility. If logFile is set,
it is the name of a file where all output will accumulate. If logSyslog is false and logFile is
empty, messages go to the error output of the process (normally the terminal).
The variable logFile defaults to empty if daemonise is false, and to ‘/var/log/polipo’
otherwise. The variable logSyslog defaults to false, and logFacility defaults to ‘user’.
If logFile is set, then the variable logFilePermissions controls the Unix permissions with
which the log file will be created if it doesn’t exist. It defaults to 0640.
The amount of logging is controlled by the variable logLevel. Please see the file ‘log.h’ in
the Polipo sources for the possible values of logLevel.
Keeping extensive logs on your users browsing habits is probably a serere violation of their
privacy. If the variable scrubLogs is set, then Polipo will scrub most, if not all, private infor-
mation from its logs.
If you use polipo as a publicly accessible web server, you might want to set the variable
disableProxy, which will prevent it from acting as a web proxy. (You will also want to set
disableLocalInterface (see Section 2.4.1 [Web interface], page 8), and perhaps run Polipo in
a chroot jail.)
It is important to understand that these features merely make it slightly more difficult to
gather statistics about the user’s behaviour. While they do not actually prevent such statistics
from being collected, they might make it less cost-effective to do so.
The general mechanism is controlled by the variable censoredHeaders, the value of which
is a case-insensitive list of headers to unconditionally censor. By default, it is empty, but I
recommend that you set it to ‘From, Accept-Language’. Adding headers such as ‘Set-Cookie’,
‘Set-Cookie2’, ‘Cookie’, ‘Cookie2’ or ‘User-Agent’ to this list will probably break many web
sites.
The case of the ‘Referer’2 header is treated specially because many sites will refuse to serve
pages when it is not provided. If censorReferer is false (the default), ‘Referer’ headers are
passed unchanged to the server. If censorReferer is maybe, ‘Referer’ headers are passed to the
server only when they refer to the same host as the resource being fetched. If censorReferer
is true, all ‘Referer’ headers are censored. I recommend setting censorReferer to maybe.
Another header that can have privacy implications is the ‘Via’ header, which is used to
specify the chain of proxies through which a given request has passed. Polipo will generate ‘Via’
headers if the variable disableVia is false (it is true by default). If you choose to generate ‘Via’
headers, you may want to set the proxyName variable to some innocuous string (see Section 3.1
[Client connections], page 9).
2
HTTP contains many mistakes and even one spelling error.
Chapter 3: Polipo and the network 12
4
Or sequences of eight 16-bit integers if you are running IPv6.
5
The Unix interface to the resolver is provided by the gethostbyname(3) library call (getaddrinfo(3) on recent
systems), which was designed at a time when a host lookup consisted in searching for one of five hosts in a
‘HOSTS.TXT’ file. The gethostbyname call is blocking, meaning that all activity must cease while a host lookup
is in progress. When the call eventually returns, it doesn’t provide a time to live (TTL) value to indicate
how long the address may be cached. For these reasons, gethostbyname is hardly useful for programs that
need to contact more than a few hosts. (Recent systems replace gethostbyname(3) by getaddrinfo(3), which
is reentrant. While this removes one important problem that multi-threaded programs encounter, it doesn’t
solve any of the other issues with gethostbyname.)
6
While Polipo does its own caching of DNS data, I recommend that you run a local caching name server. I am
very happy with pdnsd, notwithstanding its somewhat bizarre handling of TCP connections.
Chapter 3: Polipo and the network 15
Polipo can use two protocols to speak to a parent proxy: HTTP and SOCKS. When config-
ured to use both HTTP and SOCKS proxying, Polipo will contact an HTTP proxy over SOCKS
— in other words, SOCKS is considered as being at a lower (sub)layer than HTTP.
4 Caching
It is useful when you are consulting a live web site but don’t want to be bothered with failed
revalidations.
If relaxTransparency is true, missing data is fetched from remote servers, but stale data
are unconditionally served with no validation. Client-side ‘Cache-Control’ directives are still
honoured, which means that you can force an end-to-end revalidation from the browser’s interface
(typically by shift-clicking on “reload”). This setting is only useful if you have very bad network
connectivity or are consulting a very slow web site or one that provides incorrect cache control
information2 and are willing to manually revalidate pages that you suspect are stale.
If mindlesslyCacheVary is true, the presence of a ‘Vary’ header (which indicates that
content-negotiation occurred, see Section 3.3.2.1 [Censor Accept-Language], page 11) is ignored,
and cached negotiated instances are mindlessly returned to the client. If it is false (the default),
negotiated instances are revalidated on every client request.
Unfortunately, a number of servers (most notably some versions of Apache’s mod_deflate
module) send objects with a ‘ETag’ header that will confuse Polipo in the presence of a ‘Vary’
header. Polipo will make a reasonable check for consistency if ‘dontTrustVaryETag’ is set to
‘maybe’ (the default); it will systematically ignore ‘ETag’ headers on objects with ‘Vary’ headers
if it is set to ‘true’.
A number of websites incorrectly mark variable resources as cachable; such issues can be
worked around in polipo by manually marking given categories of objects as uncachable. If
dontCacheCookies is true, all pages carrying HTTP cookies will be treated as uncachable. If
dontCacheRedirects is true, all redirects (301 and 302) will be treated as uncachable. Fi-
nally, if everything else fails, a list of uncachable URLs can be given in the file specified by
uncachableFile, which has the same format as the forbiddenFile (see Section 3.8.1 [Internal
forbidden list], page 13). If not specified, its location defaults to ‘~/.polipo-uncachable’ or
‘/etc/polipo/uncachable’, whichever exists.
The value diskCacheWriteoutOnClose (64 kB by default) is the amount of data that Polipo
will write out when closing a disk file. Writing out data when closing a file can avoid subsequently
reopening it, but causes unnecessary work if the instance is later superseded.
The integers diskCacheDirectoryPermissions and diskCacheFilePermissions are the
Unix filesystem permissions with which files and directories are created in the on-disk cache;
they default to ‘0700’ and ‘0600’ respectively.
The variable maxDiskCacheEntrySize specifies the maximum size, in bytes, of an instance
that is stored in the on-disk cache. If set to -1 (the default), all objects are stored in the on-disk
cache,
diskCacheTruncateSize specifies the size at which files are truncated after they have not been
accessed for diskCacheTruncateTime; it defaults to 1 MB.
Usually, Polipo uses a file’s modification time in order to determine whether it is old enough
to be expirable. This heuristic can be disabled by setting the variable preciseExpiry to true.
5 Memory usage
Polipo uses two distinct pools of memory, the chunk pool and the malloc pool.
The value objectHighMark is the absolute maximum of objects held in memory (including
resources and server addresses). When the number of in-memory objects that haven’t been
superseded yet falls below publicObjectLowMark, Polipo will stop writing out objects to disk
(superseded objects are discarded as soon as possible).
On 32-bit architectures, every object costs 108 bytes of memory, plus storage for every globally
unique header that is not handled specially (hopefully negligible), plus an overhead of one word
(4 bytes) for every chunk of data in the object.
You may also want to change objectHashTableSize. This is the size of the hash table used
for holding objects; it should be a power of two and defaults to eight times objectHighMark.
Increasing this value will reduce the number of objects being written out to disk due to hash
table collisions. Every hash table entry costs one word.
1
There are exactly three places in the code where Polipo will give up and exit if out of memory; all three are
extremely unlikely to happen in practice.
2
How I wish for a ‘SIGXMEM’ signal.
Copying 23
Copying
You are allowed to do anything you wish with Polipo as long as you don’t deny my right to be
recognised as its author and you don’t blame me if anything goes wrong.
More formally, Polipo is distributed under the following terms:
Copyright
c 2003–2010 by Juliusz Chroboczek
Copyright
c 2009–2010 by Christopher Davis
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in the
Software without restriction, including without limitation the rights to use, copy,
modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
and to permit persons to whom the Software is furnished to do so, subject to the
following conditions:
The above copyright notice and this permission notice shall be included in all copies
or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AU-
THORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAM-
AGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFT-
WARE.
The last sentence is what happens when you allow lawyers to have it their way with a
language.
Variable index 24
Variable index
A I
allowedClients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 idleTime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
allowedPorts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
alwaysAddNoTransform . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
authCredentials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
L
authRealm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 laxHttpParser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
localDocumentRoot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
logFacility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
B logFile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
logFilePermissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
bigBufferSize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
logLevel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
logSyslog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
C
cacheIsShared . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 M
censoredHeaders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 MALLOC_CHUNKS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
censorReferer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 maxAge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
CHUNK_SIZE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 maxAgeFraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
chunkCriticalMark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 maxDiskCacheEntrySize . . . . . . . . . . . . . . . . . . . . . . . . . 18
chunkHighMark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 maxDiskEntries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
chunkLowMark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 maxExpiresAge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
maxNoModifiedAge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
maxObjectsWhenIdle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
D maxPipelineTrain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
daemonise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 maxSideBuffering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
disableConfiguration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 maxWriteoutWhenIdle . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
disableLocalInterface. . . . . . . . . . . . . . . . . . . . . . . . . . . 8 mindlesslyCacheVary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
disableProxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
disableServersList . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
disableVia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 O
DISK_CACHE_BODY_OFFSET . . . . . . . . . . . . . . . . . . . . . . . . 20 objectHashTableSize . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
diskCacheDirectoryPermissions . . . . . . . . . . . . . . . . 18 objectHighMark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
diskCacheFilePermissions . . . . . . . . . . . . . . . . . . . . . . 18
diskCacheRoot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
diskCacheTruncateSize . . . . . . . . . . . . . . . . . . . . . . . . . 19 P
diskCacheTruncateTime . . . . . . . . . . . . . . . . . . . . . . . . . 19 parentAuthCredentials . . . . . . . . . . . . . . . . . . . . . . . . . 15
diskCacheUnlinkTime . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 parentProxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
diskCacheWriteoutOnClose . . . . . . . . . . . . . . . . . . . . . . 18 pidFile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
dnsGethostbynameTtl . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 pipelineAdditionalRequests . . . . . . . . . . . . . . . . . . . . 12
dnsMaxTimeout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 pmmFirstSize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
dnsNameServer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 pmmSize. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
dnsNegativeTtl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 preciseExpiry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
dnsQueryIPv6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 proxyAddress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9, 10
dnsUseGethostbyname . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 proxyName . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9, 10
dontCacheCookies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 proxyOffline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
dontCacheRedirects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 proxyPort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9, 10
dontIdentifyToClients . . . . . . . . . . . . . . . . . . . . . . . . . 10 publicObjectLowMark . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
dontTrustVaryETag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
R
E redirector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
redirectorRedirectCode . . . . . . . . . . . . . . . . . . . . . . . . 13
expectContinue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 replyUnpipelineSize . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
replyUnpipelineTime . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
F
forbiddenFile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 S
forbiddenRedirectCode . . . . . . . . . . . . . . . . . . . . . . . . . 13 scrubLogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
forbiddenUrl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 serverExpireTime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
serverMaxSlots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Variable index 25
serverSlots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 T
serverSlots1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 tunnelAllowedPorts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
smallRequestTime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
socksParentProxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
socksProxyType . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 U
socksUserName . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 uncachableFile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
useTemporarySourceAddress . . . . . . . . . . . . . . . . . . . . . 10
Concept index 26
Concept index
A headers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Accept-Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 HTTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1, 10
access control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 HTTP/SSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9, 10 https . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
advertisement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Adzapper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
age . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
I
Allowed ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 instance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
anonymity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 intermediate proxies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Apache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 invocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9, 15 IPv6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9, 10, 14
B J
banner ad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Junkbuster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
breaking pipelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
browser configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
browsing offline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 K
bug . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 keep-alive connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
C L
cache transparency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 large request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 latency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
chunk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 limiting memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
configuration file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 local server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
configuration variable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
CONNECT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 loop. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 loopback address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
content negotiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
cookies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
counter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 M
malloc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
D multiple addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 multiplexing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2, 13
DNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
E N
name server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
entity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 negotiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
expire . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 NFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
F O
filesystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 offline browsing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
firewall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 on-disk cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
forbidden . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 on-disk file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Forbidden ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 OOM killer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
fresh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Oops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
OPTIONS request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
out-of-date instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
G
GET request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
gethostbyname . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 P
parent proxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
partial instance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
H password . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Harvest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 persistent connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
HEAD request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 pid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Concept index 27
Pipelining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 stale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Poor Man’s Multiplexing . . . . . . . . . . . . . . . . . . . . . . 2, 13 stopping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9, 10
ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
POST request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3, 15 T
privacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 terminal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Privoxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 throughput. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
PROPFIND request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
proxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1, 3 transparent cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
proxy loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 tunnel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
proxy name . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9, 10 tunnelling proxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
purging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
PUT request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3, 15
U
ulimit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
R uncachable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
range request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 upstream proxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
redirect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 URL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
redirector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 usage limit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Referer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 user-agent configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
resolver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 username . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
revalidation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
round-trip time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 V
rsync . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
runtime configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . 6, 8 variable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
vary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
via . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
S
security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9, 10
server statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 W
shift-click . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 warning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
shutting down . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 web ad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 web bug . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
small request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 web counter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
SOCKS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 web interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Squid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 web server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Squid-style redirector . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 WWWOFFLE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4