Subject: Narrowing the slowdown down...

From: Glen <[email-masked]>
Date: Mon, 27 Jun 2011 07:44:04 -0700

All -

I have sent detailed data to the tools team and Henrik, but I wanted to alert
everyone to a pattern I've seen during my analysis:

This request:

POST /doc/draft-ietf-payload-rfc3016bis/edit/position/ HTTP/1.1" 302 - 
https://datatracker.ietf.org/doc/draft-ietf-payload-rfc3016bis/edit/position/

was seen in the logs at the start of both slowdowns, and I now suspect that
there may be database corruption and/or some problem with the code related
either to ballot positions generally, or this draft specifically.

It comes to my mind that, while I was gone, a request came in to clear the
ballot positions for a draft, which the secretariat did.  This may have been
the draft that was cleared - and clearing it may have caused some type of
problem for the datatracker.

Of course, the datatracker should not loop or fail even if data is bad, but
not all possibilities can be forseen.

It is my hope that we will both be able to correct a potential database
problem, and find and harden a potential datatracker bug, quickly.

In the meantime, until we hear from the tools team, it might be best to
at least refrain from voting on the above draft, if not all drafts.

If you do vote on a draft, and get a response, don't get too excited either
way.  The server actually survives for an hour or more once the bug starts
using resources (I'm actually proud of this - it's a HUGE server with lots
of resources - the old servers would have died much more quickly. ;-) so
things can appear okay for a while.

Now that we know what to look for, we can catch it earlier, but I'm still
hopeful for a quick fix and repair today.

Thanks,
Glen

---

From: Robert Sparks <[email-masked]>
Date: Mon, 27 Jun 2011 09:48:15 -0500

This was not the draft that was cleared.
On Jun 27, 2011, at 9:44 AM, Glen wrote:

> _______________________________________________
> TOOLS-DEVELOPMENT mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-development

---

From: Glen <[email-masked]>
Date: Mon, 27 Jun 2011 07:53:34 -0700

> This was not the draft that was cleared.

Ahh, thank you Robert.  That means that (hopefully) the secretariat's action
was not related to this.

But we still do have a problem either with this draft, the database, or the
datatracker.  I'm going to make the 20 minute run to the office in a bit, but
we will keep watching to make sure things are cool.  We'll try and do some
more testing later this morning (Pacific time), but I hope the tools team
beats me to it.  :-)

Glen

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 27 Jun 2011 17:19:12 +0200

Hi Glen,

This might help you.

This document is from AVT that was concluded and then split into 4 WGs
and the documents got re-assigned to the new WGs. I have earlier filed
several ticket on the WG tracker regarding documents that has not
correctly accessible despite it showing up as WG document. Apparently
there is more than one field that contains which WG the document belongs
to. Henrik should know more about this issue.

So it is not surprising if there are issues around AVTs old documents as
they have been re-assigned.

Cheers

Magnus

-- 

Magnus Westerlund

----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------

---

From: Robert Sparks <[email-masked]>
Date: Mon, 27 Jun 2011 10:10:54 -0500

I have reproduced the failure on a local instance and believe it to be related to the code that formats email messages
before we send them. Henrik is looking at that now.

The trigger was the URL Stephen entered (the spaces at the beginning of the line have to be there to trigger the bug)
   [1] http://tools.ietf.org/rfcdiff?url1=rfc3016&url2=draft-ietf-payload-rfc3016bis-01.txt

RjS

---

From: Robert Sparks <[email-masked]>
Date: Mon, 27 Jun 2011 10:22:56 -0500

Hi Magnus -

Thanks for helping brainstorm. In this case, I'm pretty confident that the history of the document has no impact on the bug.

RjS