Subject: Re: API Rate Limits and HTTP Code [#255]

From: Bryce Nesbitt <[email-masked]>
Date: Thu, 11 Nov 2010 14:41:29 -0800

Berkeley Electronic Press (http://www.bepress.com/) is using 503
plus retry_after to rate limit:

out $cgiutils->header(-status=>"503 Excessive OAI queries from IP address
($ip)", -retry_after => $throttle);
out $cgiutils->start_html(-title=>"OAI database busy");

But more importantly we're using (and this is an extension) a 200 error code
for success, but also including retry_after:

out $cgiutils->header(MIME_TYPE => 'text/xml', -retry_after => $throttle);

The goal is to say "you got this reply, but if you ask back in less than 5
seconds, we're going to block you.  It saves an entire server round trip.
 Going directly by the standard our clients would need to do:

    Request->200
    Request->503 retry after 10 seconds
    Wait ten seconds
    Request->200
    Request->503 retry after 10 seconds
    Wait ten seconds

What would it take to make retry_after an officially list for 2xx?  Then
you'd get:

    Request->200 retry after 10 seconds
    Wait ten seconds
    Request->200 retry after 10 seconds
    Wait ten seconds
    Request->200 retry after 10 seconds
    Wait five seconds
    Request->503 retry after 5 seconds
    Wait five seconds
    Request->200 retry after 10 seconds
    Wait ten seconds

We have this in production use, with a particular high volume client, who in
fact halved the number of transactions hitting our server.

We use 5xx because we want the client to come back later, not think the
resource is gone.
We added retry_after to 2xx codes reduce the number of server round trips,
for clients that are crawling
the heck out of us.

On Thu, Oct 28, 2010 at 5:33 PM, Mark Nottingham <[email-masked]> wrote:

> Hi Karl,
>
> That's a very timely question; I've had similar discussions privately with
> a number of folks, and have created an issue to track it:
>  http://trac.tools.ietf.org/wg/httpbis/trac/ticket/255
>
> My personal feeling is that we should clarify one of the existing status
> codes to include this use case, allowing space for someone to define an
> additional header or two giving more information. I'm happy to be argued
> into a new status code, but don't think it's necessary at this point.
>
> Based on the links below, it looks like common practice is to use either
> 403 or 503.
>
> The commonly cited case for 4xx codes (also explained in the blog you link
> to) is that it's a client error, not a server error. While I agree
> generally, it's important to keep in mind the effects on clients. 403 (and
> any 4xx error that a client doesn't recognise) will result in a conservative
> client believing that they're not allowed to resubmit the request, whereas
> 503 + Retry-After results in the correct client behaviour.
>
> That said, I'm more interested in coming to an agreement and assuring that
> existing software doesn't get broken than in coming up with "perfect"
> semantics. To that end, it looks like our options are:
>
> a) Clarify that 503 can be used for rate limiting, or
>
> b) Clarify that 403 can be used for rate limiting, and allow Retry-After to
> appear there, or
>
> c) Define a new status code (4xx or 5xx TBD).
>
>
>
> Some other folks doing similar things:
>
> Github uses 403:
>
> http://support.github.com/discussions/site/1151-api-returns-403-when-rate-limiting
>
> BaseCamp uses 503:
>  http://developer.37signals.com/basecamp/
>
> Google seems to use 404:
>
> http://groups.google.com/group/google-base-data-api/browse_thread/thread/cb5752b43b030fed
>
> Amazon appears to use 503:
>
> http://developer.amazonwebservices.com/connect/message.jspa?messageID=142896
>
> Apache mod_limitipconn uses 503:
>  http://dominia.org/djao/limit/contrib/sbarta/mod_limitipconn.c
>
> Lighttpd mod_evasive uses 403:
>
> http://redmine.lighttpd.net/projects/lighttpd/repository/entry/trunk/src/mod_evasive.c
>
> ... as does Apache mod_evasive:
>  http://www.zdziarski.com/blog/?page_id=442
>
> Nginx uses 503:
>  http://wiki.nginx.org/NginxHttpLimitZoneModule
>
> Yahoo uses 403:
>  http://developer.yahoo.com/search/rate.html
>
> I see references to Facebook using 341, e.g.,:
>  http://forum.developers.facebook.net/viewtopic.php?pid=245966
>
> Mozilla Weave doesn't specify a status code, but re-specifies Retry-After:
>  https://wiki.mozilla.org/Labs/Weave/Sync/1.0/API#X-Weave-Backoff

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 12 Nov 2010 14:01:08 +1100

Good to hear you're using 503 + Retry-After.

Instead of setting Retry-After on 200, why not just set appropriate Cache-Control: max-age?

Regards,

On 12/11/2010, at 9:41 AM, Bryce Nesbitt wrote:

> Berkeley Electronic Press (http://www.bepress.com/) is using 503 plus retry_after to rate limit:
> 
> out $cgiutils->header(-status=>"503 Excessive OAI queries from IP address ($ip)", -retry_after => $throttle);
> But more importantly we're using (and this is an extension) a 200 error code for success, but also including retry_after:
> 
> The goal is to say "you got this reply, but if you ask back in less than 5 seconds, we're going to block you.  It saves an entire server round trip.  Going directly by the standard our clients would need to do:
> 
>     Request->200 
> What would it take to make retry_after an officially list for 2xx?  Then you'd get:
> 
> We have this in production use, with a particular high volume client, who in fact halved the number of transactions hitting our server.
> 
> We use 5xx because we want the client to come back later, not think the resource is gone.
> We added retry_after to 2xx codes reduce the number of server round trips, for clients that are crawling
> the heck out of us.
> 
> 
> That's a very timely question; I've had similar discussions privately with a number of folks, and have created an issue to track it:
> My personal feeling is that we should clarify one of the existing status codes to include this use case, allowing space for someone to define an additional header or two giving more information. I'm happy to be argued into a new status code, but don't think it's necessary at this point.
> 
> Based on the links below, it looks like common practice is to use either 403 or 503.
> 
> The commonly cited case for 4xx codes (also explained in the blog you link to) is that it's a client error, not a server error. While I agree generally, it's important to keep in mind the effects on clients. 403 (and any 4xx error that a client doesn't recognise) will result in a conservative client believing that they're not allowed to resubmit the request, whereas 503 + Retry-After results in the correct client behaviour.
> 
> That said, I'm more interested in coming to an agreement and assuring that existing software doesn't get broken than in coming up with "perfect" semantics. To that end, it looks like our options are:
> 
> b) Clarify that 403 can be used for rate limiting, and allow Retry-After to appear there, or
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: Adrien de Croy <[email-masked]>
Date: Sat, 13 Nov 2010 14:04:19 +1300

I'm seeing something that seems to be usable also for this sort of thing.

the Refresh header.

Many ISPs here have a cache for youtube content, since NZ is relatively 
poorly-connected to the rest of the world.

I'm seeing things like the following

GET /vi/WX1ABUY6iO8/default.jpg HTTP/1.1
Host: i4.ytimg.com
Referer: http://www.youtube.com/watch?v=hCvofr11Mqo&feature=related
Accept: */*
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) 
AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.44 Safari/534.7
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Connection: Keep-Alive

HTTP/1.1 200 OK
Content-Type: text/html
Pragma: no-cache
Refresh: 1; URL=http://i4.ytimg.com/vi/WX1ABUY6iO8/default.jpg
Connection: Close

<HTML></HTML>

Chrome and FF barf on this and display a broken image.

In this case, I think the intent is to give the ISP youtube cache 1s to 
fetch the original so the re-request gets a local cached copy.  However, 
given that you can effectively use this to tell a browser to re-request 
after a delay, that's a possible option for rate limiting.  Except for 
the fact that it doesn't seem to work.

Is this Refresh header (which isn't in HTTP, so surely should be 
X-Refresh?) legit?  Is it supported?

As for why they do it this way rather than just acting as a caching 
proxy is beyond me.  I guess it makes concurrent same-requests easier to 
handle or something.

Regards

Adrien

On 12/11/2010 4:01 p.m., Mark Nottingham wrote:
>> On Thu, Oct 28, 2010 at 5:33 PM, Mark Nottingham<[email-masked]>  wrote:
>> Hi Karl,
>>

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 13 Nov 2010 08:26:22 +0100

> X-Refresh?) legit?  Is it supported?

Well, it used to be silently supported in many browsers for a long
time, maybe since Netscape 2 or 3. I think it started with the HTML
"meta http-equiv" tags where we used to see it. I think that by
extension they supported it in HTTP headers.

I've made use of it in the stats page of haproxy because it's not a
critical feature, and it happens to work with at least FF and Chrome.
I seem to remember that MSIE supports it too. You can try here if you
want to check, I have just enabled it : http://demo.1wt.eu/

The difference is that I did not add the ";url=" part to the header,
it only contains the delay.

> proxy is beyond me.  I guess it makes concurrent same-requests easier to 

It's also possible that different servers are used for such fetches and
that they program a download which could not be linked to your existing
connection.

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 13 Nov 2010 08:38:45 +0100

> extension they supported it in HTTP headers.

Interestingly, it's been discussed "here" a long time ago :

  http://www.hpl.hp.com/personal/ange/archives/archives-96/http-wg-archive/1656.html
  http://www.hpl.hp.com/personal/ange/archives/archives-96/http-wg-archive/1657.html

And apparently it was introduced in Netscape 1.1 :

  http://web.archive.org/web/20080412204810/http://wp.netscape.com/assist/net_sites/pushpull.html

Regards,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Sat, 13 Nov 2010 10:40:58 +0100

On 13.11.2010 02:04, Adrien de Croy wrote:
> ...
> ...

Well, it's widely implemented, just not registered. Prefixing with "X-" 
wouldn't help; "X-" isn't special for message headers.

As far as I can tell, the right thing to do is to write a short Internet 
Draft specifying the syntax and the semantics and registering it. Bonus 
points for discussing alternate approaches (Retry-After, Redirects), and 
the connection to HTML...

Best regards, Julian

PS: I'm glad to help with the formatting if an IETF "newbie" wants to 
give this a try...

---

From: Karl Dubost <[email-masked]>
Date: Sat, 13 Nov 2010 08:12:49 -0500

Le 13 nov. 2010 à 04:40, Julian Reschke a écrit :
> As far as I can tell, the right thing to do is to write a short Internet Draft specifying the syntax and the semantics and registering it. Bonus points for discussing alternate approaches (Retry-After, Redirects), and the connection to HTML...

Content for helping with the writing. There are implications for caches, accessibility, usability, plus I guess a few funky rules for parsing. 
The producer must be strict if defined, but we have to be careful with the parsing.

History of refresh techniques
http://www.securiteam.com/securityreviews/6Z00320HFQ.html

Accessibility Issues
http://www.w3.org/TR/WCAG10-HTML-TECHS/#meta-element
And the way to disable it in Firefoxv3
http://antivirus.about.com/od/securitytips/ht/ffmetarefresh.htm

On wikipedia
http://en.wikipedia.org/wiki/URL_redirection#Refresh_Meta_tag_and_HTTP_refresh_header

A W3C QA tip, recommending other practices
http://www.w3.org/QA/Tips/reback

Refresh in html5
http://dev.w3.org/html5/spec/semantics.html#attr-meta-http-equiv-refresh

It's available in IIS
http://support.microsoft.com/kb/240774

Permanent redirect and Refresh
http://www.somacon.com/p145.php

It seems Firefox and IE take " " in addition of ";" as a separator for the META to be tested on http headers too
http://code.google.com/p/chromium/issues/detail?id=20117

Mechanize, a python lib is implementing it
http://wwwsearch.sourceforge.net/mechanize/doc.html

Issues with Mobile Best Practices
http://www.w3.org/TR/mobile-bp/#d0e959
-- 
Karl Dubost
Montréal, QC, Canada
http://www.la-grange.net/karl/