Subject: Large scale URN testbed

From: Martin Hamilton <[email-masked]>
Date: Fri, 4 Jun 1999 17:03:33 +0100

-----BEGIN PGP SIGNED MESSAGE-----

[apologies if you saw this already - I couldn't grep for it in the
Bunyip archives for May or June so suspect it got squished en route]

Hi,

I help to run the JANET Web Cache Service [1], which uses the Squid
[2] proxy cache server to ship between 20%-25% of the incoming ac.uk
WWW traffic from the US - bandwidth to the US has been the most scarce
and most in demand resource until recently.  At the moment that's some
25 million proxy HTTP requests per day, and about 230GB of content
transferred through 28 Linux and FreeBSD boxes FWIW.

We've noticed that a significant fraction of the stuff requested
through our caches is actually mirrored at multiple sites, including
in many cases mirror sites in the UK, and managed to persuade TERENA
(the club of European national research networks - [3]) to give us a
bit of money to work on some technology to help everyone in the
caching community take advantage of the mirror sites that do exist.

A first cut at this is now available - [4].  Essentially we put up a
'fake' Web Cache which advertises itself as a 'hit' (using the
Internet Cache Protocol now, and the Cache Digest protocol Real Soon
Now) for all of the URL prefixes by which a widely replicated resource
may be fetched.  Or at least, all of the prefixes it knows about -
figuring those out is a separate part of the project.

For instance, for OpenLDAP's FTP site we have the following URL
prefixes: (derived from their list of mirror sites)

  ftp://ftp.net.lut.ac.uk/openldap/
  ftp://ftp.OpenLDAP.org/pub/OpenLDAP/
  ftp://gd.tuwien.ac.at/infosys/network/OpenLDAP/
  ftp://mirror.AARNet.edu.au/pub/OpenLDAP/
  ftp://ftp.ntua.gr/mirror/OpenLDAP/
  ftp://sunsite.cnlab-switch.ch/mirror/OpenLDAP/

Other caches which peer with the fake one will (modulo config tweaks
and network/server performance being up to scratch) now try to fetch
any URL prefixed with one of the above strings from the fake - rather
than (say) direct from the site in question.  The fake cache is
actually a combination of a custom ICP server (running our Perl ICP
code) and a real Squid cache running a custom redirector to rewrite
incoming URLs.

The redirector program will rewrite any of the above URLs to the URL
prefix of the local mirror site before Squid proceeds to fetch it.
For example, if you try to fetch:

  ftp://mirror.AARNet.edu.au/pub/OpenLDAP/openldap-release/openldap-1.2.1.tgz

the redirector will rewrite it to (say):

  ftp://ftp.net.lut.ac.uk/openldap/openldap-release/openldap-1.2.1.tgz

At the moment, the actual choice of preferred mirror site is something
that has to be configured by the person who sets this stuff up.  We're
thinking about an automated configurator which sucks in all of the
mirror sites and uses ping and traceroute to find a best match.

Phew!

So, an obvious next step would be for us to throw in URNs, which can
be included in this scheme trivially as another type of 'URL' prefix -
IFF we leave the file names in the URL suffix alone.  This isn't
immediately useful to our users like the 'redirection' to mirror sites,
but obviously it's important from a long term point of view.

We'd like some advice from this group as to the URN structure for this
type of thing (let's call them widely replicated archives), if people
can spare a couple of minutes to think about this...  Since we're
effectively doing the URN->URL resolution by sleight of hand, all of
the URN up to the suffix can actually be any string we like - but we'd
like to create URNs which are compatible with the current IETF
thinking :-)

Thanks in advance for any thoughts on this...

Martin

[1] <URL:http://wwwcache.ja.net/>
[2] <URL:http://squid.nlanr.net/Squid/>
[3] <URL:http://www.terena.nl/>
[4] <URL:http://wwwcache.ja.net/dev/perlmod/>

---

From: Martin Hamilton <[email-masked]>
Date: Tue, 8 Jun 1999 11:31:30 +0100

-----BEGIN PGP SIGNED MESSAGE-----

Just to add that the slides for my talk about this tomorrow at the
TERENA/NORDUnet networking conference are available at:

  <URL:http://www.net.lut.ac.uk/~martin/tracker-slides/>

Martin