Subject: [Tools-discuss] datatracker perf issues

From: Eric Rescorla <[email-masked]>
Date: Thu, 21 Oct 2021 15:54:40 -0700

I just watched datatracker take ~20s to server:
https://datatracker.ietf.org/wg/priv/documents/

And then on reload it sent me a 500.

-Ekr

---

From: Bob Hinden <[email-masked]>
Date: Thu, 21 Oct 2021 17:22:41 -0700

Ekr,

I tried the link and got a response in about a second.

If I go to:

    https://datatracker.ietf.org/wg/6man/documents/

It takes about 5 seconds.

Bob

> On Oct 21, 2021, at 3:54 PM, Eric Rescorla <[email-masked]> wrote:
> 
> ___________________________________________________________
> Tools-discuss mailing list - [email-masked]
> This list is for discussion, not for action requests or bug reports.
> * Report datatracker and mailarchive bugs to: [email-masked]
> * Report tools.ietf.org bugs to: [email-masked]
> * Report all other bugs or issues to: [email-masked]
> List info (including how to Unsubscribe): https://www.ietf.org/mailman/listinfo/tools-discuss

---

From: Richard Barnes <[email-masked]>
Date: Thu, 21 Oct 2021 20:28:23 -0400

Given how cacheable these resources are, anything measured in units greater
than milliseconds seems like failure.

On Thu, Oct 21, 2021 at 20:23 Bob Hinden <[email-masked]> wrote:

> > List info (including how to Unsubscribe):
> https://www.ietf.org/mailman/listinfo/tools-discuss
>
> List info (including how to Unsubscribe):
> https://www.ietf.org/mailman/listinfo/tools-discuss
>

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 21 Oct 2021 18:07:51 -0700

On Thu, Oct 21, 2021 at 5:28 PM Richard Barnes <[email-masked]>
wrote:

> Given how cacheable these resources are, anything measured in units
> greater than milliseconds seems like failure.
>

While I agree with you in principle, it's probably worth distinguishing
"working as designed" from
"we think this should be designed differently". My understanding is that
for design reasons datatracker
is not very fast, so 1s, white undesirable, is expected. However, 5s, let
alone 20s or a 500 seems indicative of
some kind of failure.

-Ekr

---

From: Chris Morrow <[email-masked]>
Date: Fri, 22 Oct 2021 02:31:50 +0000

On Fri, 22 Oct 2021 00:28:23 +0000,
Richard Barnes <[email-masked]> wrote:
> 
> [1  <multipart/alternative (7bit)>]
> [1.1  <text/plain; UTF-8 (7bit)>]

it's not clear that there's a cache between you and the service... (or
not clear to ME that that is the case). (datatracker isn't served
behind CloudFlare, for instance)

That point aside, almost certainly it's unhelpful to say: "now things
are bad" because: "now" is .. not "now"... It'd be nice if there were
some ongoing performance monitoring, perhaps exposed to just the
sysadmin-type-folks, and the ietf contract-relevant folks?

It'd be nice if that were also exposed publicly, though I get that:
  "Eww! someone might see my bedroom before I clean it up!!!"

At least with some historical data about request/response we might
know how/if there are problems to chase. Ideally the whole process
of managing the ietf datatracker would include some SLO like:
  "uptime of http/https service is XXXX"
  "Page load times of example pages x, y, z are:
     XX percentile 200ms
     YY percentile 300ms
     ZZ percentile 500ms
   errors only served A in B times on a daily running average"
  etc, etc, etc. (I'm a chemical engineer not a contracting engineer)

-chris
(I kicked off some smokeping on my personal thingies to watch datatracker...
glenn or robert can send me a "yo, dude, stop" if that's their inclination,
I'm just personally curious about normal behavior for this service.)

> [1.2  <text/html; UTF-8 (quoted-printable)>]
> [2  <text/plain; us-ascii (7bit)>]

---

From: Robert Sparks <[email-masked]>
Date: Fri, 22 Oct 2021 11:55:16 -0500

I agree with EKR here.

Fwiw, we have a spring focusing on performance coming up next week 
(you've seen the announcements here).

My instinct, however, for _this_ particular instance is that there was 
something else going on that made things bad enough to get to a 500. 
We're working towards getting better visibility into what those 
something elses might be.

A response to Richard inline:

RjS

On 10/21/21 8:07 PM, Eric Rescorla wrote:
>
>
> On Thu, Oct 21, 2021 at 5:28 PM Richard Barnes 
> <[email-masked]> wrote:
>
>     Given how cacheable these resources are
>
Yes, we can make better use of caching than we are, but there is a _lot_ 
of data on pages like the one EKR was looking at that is not cachable, 
unless we change the expectations in the community on how quickly they 
will see a change when, say, an AD changes a position, a new version of 
a document is posted, etc. Long term, I plan to refactor things so that 
it will be easier to cache the pieces rather than the assembled result, 
and make invalidation simpler.

>     , anything measured in units greater than milliseconds seems like
>     failure.
>
>
> While I agree with you in principle, it's probably worth 
> distinguishing "working as designed" from
> "we think this should be designed differently". My understanding is 
> that for design reasons datatracker
> is not very fast, so 1s, white undesirable, is expected. However, 5s, 
> let alone 20s or a 500 seems indicative of
> some kind of failure.
>
> -Ekr
>
>
>         > * Report tools.ietf.org <http://tools.ietf.org> bugs to:
>         [email-masked]
>         * Report tools.ietf.org <http://tools.ietf.org> bugs to:
>         [email-masked]
> Tools-discuss mailing list [email-masked]
> * Report datatracker and mailarchive bugs to:[email-masked]
> * Report tools.ietf.org bugs to:[email-masked]
> * Report all other bugs or issues to:[email-masked]
> List info (including how to Unsubscribe):https://www.ietf.org/mailman/listinfo/tools-discuss

---

From: Robert Sparks <[email-masked]>
Date: Fri, 22 Oct 2021 12:03:06 -0500

On 10/21/21 9:31 PM, Chris Morrow wrote:
> Richard Barnes<[email-masked]>  wrote:
> it's not clear that there's a cache between you and the service...
Only in your browser
>   (or
Using Cloudflare for the datatracker is planned.
> sysadmin-type-folks, and the ietf contract-relevant folks?

Yes it would. We are building it.

This is not a permanent view (and we are talking about how to make it 
more useful) but:

https://scoutapm.com/shares/427d60eb-abcb-4d2b-9d2c-89abab1e72a5

I also have a significant amount of data collected using newrelic 
synthetics, but there isn't an easy way to expose that publicly that 
I've found.

> I'm just personally curious about normal behavior for this service.)
Let me know offline what your frequency and what pages you are watching 
please.
>
>> On Thu, Oct 21, 2021 at 20:23 Bob Hinden<[email-masked]>  wrote:
>>
>>> Ekr,
>>>
>>>> On Oct 21, 2021, at 3:54 PM, Eric Rescorla<[email-masked]>  wrote:
>>>>

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 22 Oct 2021 17:57:29 -0700

On Fri, Oct 22, 2021 at 9:56 AM Robert Sparks <[email-masked]> wrote:

> I agree with EKR here.
>
> Fwiw, we have a spring focusing on performance coming up next week (you've
> seen the announcements here).
>
> something else going on that made things bad enough to get to a 500. We're
> working towards getting better visibility into what those something elses
> might be.
>

That's what I assumed as well, which is why I reported it.

A response to Richard inline:
> unless we change the expectations in the community on how quickly they will
> see a change when, say, an AD changes a position, a new version of a
> document is posted, etc. Long term, I plan to refactor things so that it
> will be easier to cache the pieces rather than the assembled result, and
> make invalidation simpler.
>
Can you say more about why this data is not cacheable? While it's true that
we want changes to appear quickly, the change rate is not high.

-Ekr

> , anything measured in units greater than milliseconds seems like failure.
>>
>

---

From: Tim Wicinski <[email-masked]>
Date: Fri, 22 Oct 2021 21:05:12 -0400

On Fri, Oct 22, 2021 at 8:58 PM Eric Rescorla <[email-masked]> wrote:

> Can you say more about why this data is not cacheable? While it's true
> that we want changes to appear quickly, the change rate is not high.
>
>
I was thinking that for some period around meeting time there is much more
activity and perhaps it could not be cached.
But there is a period between meetings where the cache could be refreshed
nightly, or at some whim of a WGChair/AD/etc.

tim

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 22 Oct 2021 18:07:01 -0700

> nightly, or at some whim of a WGChair/AD/etc.

As I understand it, at least some CDNs have cache invalidation mechanisms
so that servers can tell them that caches are invalid.

Maybe mnot or caw can weigh in here?

-Ekr

---

From: Robert Sparks <[email-masked]>
Date: Fri, 22 Oct 2021 22:20:19 -0500

On 10/22/21 8:07 PM, Eric Rescorla wrote:
>             Yes, we can make better use of caching than we are, but
>             there is a _lot_ of data on pages like the one EKR was
>             looking at that is not cachable, unless we change the
>             expectations in the community on how quickly they will see
>             a change when, say, an AD changes a position, a new
>             version of a document is posted, etc. Long term, I plan to
>             refactor things so that it will be easier to cache the
>             pieces rather than the assembled result, and make
>             invalidation simpler.
>
>         Can you say more about why this data is not cacheable? While
>         it's true that we want changes to appear quickly, the change
>         rate is not high.
>
>
>     I was thinking that for some period around meeting time there is
>     much more activity and perhaps it could not be cached.
>     But there is a period between meetings where the cache could be
>     refreshed nightly, or at some whim of a WGChair/AD/etc.
>
>
> As I understand it, at least some CDNs have cache invalidation 
> mechanisms so that servers can tell them that caches are invalid.
Cloudflare does, at least, and we will take advantage of it when the 
datatracker is behind it, but at the moment there are pages, like all 
the variants of the search result pages (which may have arbitrary 
drafts), which would require a nigh impossible computation to figure out 
what to invalidate. Some subset of those (like the wg document pages) 
are more tractible, and when we get that far we can take advantage of that.

---

From: Robert Sparks <[email-masked]>
Date: Fri, 22 Oct 2021 22:55:24 -0500

Trimming, hopefully keeping enough of the relevant context

On 10/22/21 7:57 PM, Eric Rescorla wrote:
>
>     Yes, we can make better use of caching than we are, but there is a
>     _lot_ of data on pages like the one EKR was looking at that is not
>     cachable, unless we change the expectations in the community on
>     how quickly they will see a change when, say, an AD changes a
>     position, a new version of a document is posted, etc. Long term, I
>     plan to refactor things so that it will be easier to cache the
>     pieces rather than the assembled result, and make invalidation
>     simpler.
>

Be precise about "this data".

Something like an AD's particular position on a given document would be 
nice to cache beyond the server, but we don't serve an individual ADs 
position right now. We serve pages that combine that with a _very_ large 
amount of other bits of data that have been assembled into a blob. The 
things that are involved in changing one of the bits of data are not 
aware of the full set of assembled blobs that might contain that data.

We can (and will) evolve to where we are serving those small bits of 
data relatively independently and letting the browsers assemble what we 
serve now as a blob, but we've for quite a while been held to a 
requirement for things to work for people who didn't allow javascript to 
run in their browser. Requiring javascript is becoming more tolerated, 
so we can change the way the services are constructed to take advantage 
of that. Even without that ability, of course, there are things we may 
be able to decompose, but at the moment, the cacheable unit for, say, 
https://datatracker.ietf.org/group/tls/documents/, is the whole page.

The effect of the rate of change of the individual pieces adds up on 
these blocks of aggregated data, and the desire to have changes appear 
quickly overtakes how long you can let such a block live.

RjS

---

From: Eric Rescorla <[email-masked]>
Date: Sat, 23 Oct 2021 19:41:47 -0700

> Be precise about "this data".

Well in this case it was a WG page, but another example is drafts. These
change quite infrequently and so should be quite cacheable.

-Ekr

> We can (and will) evolve to where we are serving those small bits of data
> relatively independently and letting the browsers assemble what we serve
> now as a blob, but we've for quite a while been held to a requirement for
> things to work for people who didn't allow javascript to run in their
> browser. Requiring javascript is becoming more tolerated, so we can change
> the way the services are constructed to take advantage of that. Even
> without that ability, of course, there are things we may be able to
> decompose, but at the moment, the cacheable unit for, say,
OK, but how often does this change?

In any case the "no client JS" seems like something we should revisit. If
this were a single page app with a JSON API, the situation would be far
more flexible (and it would also make third party apps much easier)

-Ekr

---

From: Robert Sparks <[email-masked]>
Date: Mon, 25 Oct 2021 09:25:52 -0500

On 10/23/21 9:41 PM, Eric Rescorla wrote:
>>         Yes, we can make better use of caching than we are, but there
>>         is a _lot_ of data on pages like the one EKR was looking at
>>         that is not cachable, unless we change the expectations in
>>         the community on how quickly they will see a change when,
>>         say, an AD changes a position, a new version of a document is
>>         posted, etc. Long term, I plan to refactor things so that it
>>         will be easier to cache the pieces rather than the assembled
>>         result, and make invalidation simpler.
>>
>>     Can you say more about why this data is not cacheable? While it's
>>     true that we want changes to appear quickly, the change rate is
>>     not high.
>>
>>     -Ekr
>>
>
> Well in this case it was a WG page, but another example is drafts. 
> These change quite infrequently and so should be quite cacheable.
>
> -Ekr
>
>
>     We can (and will) evolve to where we are serving those small bits
>     of data relatively independently and letting the browsers assemble
>     what we serve now as a blob, but we've for quite a while been held
>     to a requirement for things to work for people who didn't allow
>     javascript to run in their browser. Requiring javascript is
>     becoming more tolerated, so we can change the way the services are
>     constructed to take advantage of that. Even without that ability,
>     of course, there are things we may be able to decompose, but at
>     the moment, the cacheable unit for, say,
> In any case the "no client JS" seems like something we should revisit. 
> If this were a single page app with a JSON API, the situation would be 
> far more flexible (and it would also make third party apps much easier)
Agreed, and we have effectively already made that decision - there are 
many important pages that require client javascript already.