Subject: xml2rfc down ?

From: Toerless Eckert <[email-masked]>
Date: Fri, 7 Jul 2017 00:19:42 +0200

Seems to be down or at least running extremely slowly, most times requests fail. Since about
30 minutes. Maybe something running amok stealing resources ?

Thanks
    Toerless

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 7 Jul 2017 02:57:37 +0200

Hi Toerless,

> 30 minutes. Maybe something running amok stealing resources ?

Indeed.  A chinese spider out of these addresses which does not
honour robots.txt, and hits Trac repository diff, log, and other
pages despite those being disallowed:

	106.11.159.200
	106.11.158.200
	106.11.154.196
	106.11.153.212
	106.11.157.212
	106.11.155.194

Henrik

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 7 Jul 2017 08:27:11 -0700

Hi Henrik,

This seems like the kind of thing CDNs are supposed to deal with. Are these
domains not behind Cloudflare?

-Ekr

---

From: Russ Housley <[email-masked]>
Date: Fri, 7 Jul 2017 11:35:33 -0400

Most of the IETF site is served by Cloudflare.  However, datatracker.ietf.org and tools.ietf.org are not.

Russ

> On Jul 7, 2017, at 11:27 AM, Eric Rescorla <[email-masked]> wrote:
> 
> Hi Henrik,
> 
> This seems like the kind of thing CDNs are supposed to deal with. Are these domains not behind Cloudflare?
> 
> -Ekr
> 
> 
> On Thu, Jul 6, 2017 at 5:57 PM, Henrik Levkowetz <[email-masked] <mailto:[email-masked]>> wrote:
> Hi Toerless,
>

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 7 Jul 2017 08:57:50 -0700

Is there any reason why they are also not behind Cloudflare?

-Ekr

On Fri, Jul 7, 2017 at 8:35 AM, Russ Housley <[email-masked]> wrote:

> Most of the IETF site is served by Cloudflare.  However,
> datatracker.ietf.org and tools.ietf.org are not.
>
> Russ
>
>

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 7 Jul 2017 17:59:02 +0200

Using a CDN to protect against attacks sounds a bit like using NAT as
a firewall. The CDN is primarily meant to distribute content to make it scale.
The fact that this does hide/protect the backend is mostly a side effect.

I guess that its difficult to move hand-knitted software on datatracker/tools
into a CDN (as opposed to simple static content or javascripts). And
AFAIK there is no need to scale.

Are there not web-firewalls or the like, maybe even cloud-based, that would
proxy all web-connections and recognize/protect against DoS ? 

Where is the IETF honeypot ? ;-)

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 7 Jul 2017 17:59:35 +0200

Hi Eric,

> domains not behind Cloudflare?

No, tools.ietf.org has always been running on donated hardware and
donated bandwidth, as a pro-bono activity I've been running.

Putting parts of tools.ietf.org behind a CDN has been discussed, but
whether that will work for the wiki-like Trac instances I don't know.

	Henrik

---

From: "Salz, Rich" <[email-masked]>
Date: Fri, 7 Jul 2017 16:03:05 +0000

Using a CDN, for mostly-dynamic sites, just as forms and posts and uploads and such, is hard if not nearly worthless.

---

From: "Salz, Rich" <[email-masked]>
Date: Fri, 7 Jul 2017 16:07:30 +0000

> donated bandwidth, as a pro-bono activity I've been running.

Ouch!  And thanks!

Should that be moved onto IETF machines soon?

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 7 Jul 2017 10:42:51 -0700

> The fact that this does hide/protect the backend is mostly a side effect.

Hmm... That's not how the CDN people think of it. Handling DoS is an
explicit
feature they provide.

I guess that its difficult to move hand-knitted software on
> datatracker/tools

The way this works is that you *front* your site with the CDN, so they don't
have to know what is in your content, they just cache. And they absolutely
do have ways for tools like ours (Django, etc.) to manage the cache

> proxy all web-connections and recognize/protect against DoS ?

That's what Cloudflare does.

-Ekr

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 7 Jul 2017 10:44:03 -0700

On Fri, Jul 7, 2017 at 9:03 AM, Salz, Rich <[email-masked]> wrote:

> Using a CDN, for mostly-dynamic sites, just as forms and posts and uploads
> and such, is hard if not nearly worthless.
>

Most of the stuff on tools is in fact largely static. And in particular all
of the references are.

-Ekr

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 7 Jul 2017 20:13:56 +0200

Hi Rich,

> Should that be moved onto IETF machines soon?

Many services initially provided on tools.ietf.org has already been
moved to ietf machines, and more are in progress.  However, since
I still use tools.ietf.org as a place to spin up experimental stuff,
it's going to be a while till there's nothing there.  And that's OK.

Best regards,

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 7 Jul 2017 20:17:23 +0200

> Most of the stuff on tools is in fact largely static. And in particular all

And there's never been a problem serving the static content.  It's
the non-static CPU-intensive pages which sometimes cause hiccups.

Anyway, I've added fail2ban rules which should catch the kind of thing
that caused the latest slowdown for the future.

	Henrik

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 7 Jul 2017 20:20:23 +0200

Inline

> do have ways for tools like ours (Django, etc.) to manage the cache

Ok.  Any pointer to those django cache management tools ?

> That's what Cloudflare does.

So lets take xml2rfc as an example, which is what i use mostly. I can see how
cloudfare could cache the static pages, but whenever you do a conversion, that
can not be cached. So how would cloudfare help against attacks that
brute force try to overload tools via those non-cacheable/dynamic URLs ?

Any type of overall rate limiters, automatic/admin driven creation of blacklists of
requesters, actively call the police, inject malware back into recognized intruder
requests, alert the site administrator via logs/emails, slow down recognized attack requests,
explicit whitelisting / prioritizing of known regulars, ...

Any of these i'd call a real FW, the caching i call NAT ;-)

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 7 Jul 2017 14:10:40 -0700

> brute force try to overload tools via those non-cacheable/dynamic URLs ?

Minimally it would keep the static pages working even when the other ones
are
under attack.

> Any of these i'd call a real FW, the caching i call NAT ;-)

I believe Cloudflare does some if not all of these.

-Ekr

> > > > > This seems like the kind of thing CDNs are supposed to deal with.
> Are
> > > these domains not behind Cloudflare?
> > > > >
> > > > > -Ekr
> > > > >
> > > > >
> > > > > On Thu, Jul 6, 2017 at 5:57 PM, Henrik Levkowetz <
> [email-masked]
> > > <mailto:[email-masked]>> wrote:
> > > > > Hi Toerless,
> > > > >