Subject: Fwd: New Version Notification for draft-divilly-status-555-00.txt

From: Colm Divilly <[email-masked]>
Date: Tue, 24 Mar 2020 23:02:38 +0000

Hi All,
 for a product I help develop: Oracle REST Data Services (ORDS) [1], we provide a hosted environment where third party users can dynamically at runtime define their own REST APIs using SQL. Naturally there will be coding and runtime errors in these REST APIs. This creates a challenge for users and operators. When such a resource raises an error the only appropriate HTTP status code to use is 500 Internal Server Error. This causes confusion as it looks like there is something wrong with ORDS, where in fact there is only something wrong with the user supplied SQL. Despite explanatory text to clarify this, operators and users very often miss this distinction, and file support issues against ORDS. Further, automated tools that monitor the access log only see the 500 status code, and thus cannot differentiate between 'real' 500 errors in ORDS itself that need remediation versus 500 errors in the user supplied REST APIs that probably do not need any remediation. If a distinct status code for these kinds of errors appeared in the access log, it makes it immediately clear to both humans and tools that the error condition is caused by a user defined resource.

We believe assigning a new HTTP status code (we chose 555 just because it is memorable) to clearly identify errors arising in these dynamically defined user REST APIs is the cleanest way to resolve this confusion. TBH not sure if this use-case is common enough to be worth standardising, so have filed this as an 'Informational' draft. Perhaps 'serverless' platforms that have a similar paradigm of third party users publishing programmatic resources to their platforms might also find this approach useful.

Appreciate any feedback anyone may have on this approach.

Regards,
Colm Divilly

[1]: https://oracle.com/rest

> Begin forwarded message:
> 
> From: [email-masked]
> Subject: New Version Notification for draft-divilly-status-555-00.txt
> Date: 20 March 2020 at 11:35:55 GMT
> To: "Colm Divilly" <[email-masked]>
> 
> 
> A new version of I-D, draft-divilly-status-555-00.txt
> has been successfully submitted by Colm Divilly and posted to the
> IETF repository.
> 
> Name:		draft-divilly-status-555
> Revision:	00
> Title:		User Defined Resource Error HTTP Status Code
> Document date:	2020-03-20
> Group:		Individual Submission
> Pages:		5
> URL:            https://www.ietf.org/id/draft-divilly-status-555-00.txt
> Status:         https://datatracker.ietf.org/doc/draft-divilly-status-555/
> Htmlized:       https://tools.ietf.org/html/draft-divilly-status-555-00
> Htmlized:       https://datatracker.ietf.org/doc/html/draft-divilly-status-555
> 
> 
> Abstract:
>   This document specifies an additional HyperText Transfer Protocol
>   (HTTP) status code to indicate server error conditions arising during
>   evaluation of user defined resources hosted by the server.
> 
> Conventions and Terminology
> 
> 
> 
> Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
> 
> The IETF Secretariat

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 25 Mar 2020 09:13:59 +0000

--------
In message <[email-masked]>, Colm Divilly writes:

>We believe assigning a new HTTP status code (we chose 555 just because it is
>memorable)

IMO status code 555 should only be used for something which happens periodically :-)

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Colm Divilly <[email-masked]>
Date: Wed, 25 Mar 2020 09:30:40 +0000

:-), very good, I admit I had to google that [1]. Also apparently in Chinese Internet Slang, 555 means 'Crying' [2]. I kind of like the idea of the server being made to 'cry' by some user supplied code. 

[1]: https://en.wikipedia.org/wiki/555_timer_IC <https://en.wikipedia.org/wiki/555_timer_IC>
[2]: https://en.wikipedia.org/wiki/Chinese_Internet_slang#Numbers,_representing_Chinese_terms_(%E6%95%B0%E5%AD%97%E8%A1%A8%E7%A4%BA%E6%B1%89%E5%AD%97_sh%C3%B9z%C3%AC_bi%C7%8Eosh%C3%AC_h%C3%A0nz%C3%AC) <https://en.wikipedia.org/wiki/Chinese_Internet_slang#Numbers,_representing_Chinese_terms_(%E6%95%B0%E5%AD%97%E8%A1%A8%E7%A4%BA%E6%B1%89%E5%AD%97_sh%C3%B9z%C3%AC_bi%C7%8Eosh%C3%AC_h%C3%A0nz%C3%AC)>

> On 25 Mar 2020, at 09:13, Poul-Henning Kamp <[email-masked]> wrote:
> 
> --------

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 26 Mar 2020 01:42:28 +1300

On 25/03/20 12:02 pm, Colm Divilly wrote:
> Hi All,
>  for a product I help develop: Oracle REST Data Services (ORDS) [1], we
> provide a hosted environment where third party users can dynamically at
> runtime define their own REST APIs using SQL. Naturally there will be
> coding and runtime errors in these REST APIs.

> This creates a challenge
> for users and operators. When such a resource raises an error the only
> appropriate HTTP status code to use is 500 Internal Server Error. This
> causes confusion as it looks like there is something wrong with ORDS,
> where in fact there is only something wrong with the user supplied SQL.

I see no guidance details about when this status is expected to be used,
and the description given is lacking some important details:

* Is this status used to reply to requests containing bad SQL?

 If yes, then use of any 5xx is incorrect. The broken thing being part
of the client message means it should be a 4xx status of some type.

If no;

* Is the origin pre-configured with some SQL it failed to validate /
wrongly accepted, which is now breaking responses?

 If yes, then 500 is indeed correct status. The "Internal Error" just
happens to have been created by the SQL acceptance.

 Otherwise implies no SQL is known to the origin. So 404 would be
appropriate. No such resource exists.

 Or, ...

* Is the origin pre-configured with some SQL it has already accepted,
but which does not produce a resource instance?

 If yes, then the status should be 404 or 406. The origin is not broken,
it simply does not have any representation that meets the request.

At the very least it would be good to describe what alternatives (other
status, and other mechanisms) have been considered and why they are
unworkable.

PS. IIRC the number itself is supposed to be assigned by IANA *if* this
document reaches acceptance. Not chosen by draft authors.

Amos

---

From: Colm Divilly <[email-masked]>
Date: Wed, 25 Mar 2020 13:07:47 +0000

Thanks for your feedback, see my comments below:

> On 25 Mar 2020, at 12:42, Amos Jeffries <[email-masked]> wrote:
> 
No, this is not the model. At some earlier point user defines a REST endpoint powered by some program that they define (in our case SQL statements)
At a later point same/different user accesses the REST endpoint, the endpoint fails because of an error in the program.
> happens to have been created by the SQL acceptance.

Yes, understood, that is the current state of the art. The problem is that this 500 Internal Server is too vague. It confuses the user making them think there is a problem with the server engine, rather than the specific user defined resource They cannot distinguish that the problem lies with the particular REST endpoint. It confuses automatic monitoring tools as they cannot distinguish between serious 500 Internal Server errors that require remediation and those of this kind where the fact that a given user created REST endpoint is error-ring is typically a benign state. It is expected that users will create endpoints that have error conditions. Those error conditions do not affect the overall availability of the server, just that particular user defined endpoint. They need to be clearly marked as such and the resource author provided with tools to review logs associated with the error condition to enable them to take action to address the error.

> status, and other mechanisms) have been considered and why they are
We have done the following:

1. Added an additional proprietary header named Error-Reason, you can think of this as a sub-status code, that indicates the root cause of the internal server error lies in the user defined resource
2. Added explanatory text to the response document body for the 500 Internal Server Error status explaining that the root cause of the error is an error in the user defined resource

Problems with 1 are that it is not logged in things like access logs belonging to other intermediaries sitting in front of our product. Intermediary sees a 500 error in their access log, forms understanding that our software is broken, files support ticket, etc. 
Problems with 2 are that end users generally don't read the text carefully, they focus on the status code and the words 'Internal Server Error' and google that and form understanding that our software is broken, files support ticket etc.

> document reaches acceptance. Not chosen by draft authors.
Understood, we are not wedded to any particular status code number, we just chose one that was memorable and unallocated, it can change no problem.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 26 Mar 2020 14:49:49 +1100

Hi Colm,

For reference, the guidance for new status codes is here:
  https://httpwg.org/specs/rfc7231.html#considerations.for.new.status.codes

I agree this condition is potentially applicable to any resource, but it might fall afoul of another principle that we haven't written down as well -- that HTTP focuses on standardising the interface to the resource, not its implementation details (which this seems to be about). Roy might have some more thoughts here.

More generally, when someone needs to convey more refined information about what caused an error, the advice we give is to put that information in the response body or a response header. Did you consider that approach?

Cheers,

P.S. As an aside - we discourage people from "squatting" on status codes like this because it makes it harder to use them in standards later, and they are a relatively scarce resource. Please don't use it until it has been standardised, and in the future, it's best to make proposals along the lines of "a new 5xx status code" rather than a specific number. Thanks.

> On 25 Mar 2020, at 10:02 am, Colm Divilly <[email-masked]> wrote:
> 
> Hi All,

--
Mark Nottingham   https://www.mnot.net/

---

From: Colm Divilly <[email-masked]>
Date: Thu, 26 Mar 2020 12:27:53 +0000

Hi Mark,
 thanks for taking the time to review this, based on your & Amos' [1] feedback, I've submitted a new draft:

> More generally, when someone needs to convey more refined information about what caused an error, the advice we give is to put that information in the response body or a response header. Did you co…

Yes, we include information in both the response header and the response body, we have found it to be largely ineffective. Our experience is that tools and humans see 500, think 'I know what that means...' and proceed to filing a problem report/complaining to the wrong party far too often. Also the extra information in the response header and body may end up being lost in the journey back to the client/monitoring tools. For example the access log will only show the status code, custom error pages may hide extra information in the response body. 

https://datatracker.ietf.org/doc/draft-divilly-user-defined-resource-error/ <https://datatracker.ietf.org/doc/draft-divilly-user-defined-resource-error/>

Changes in this draft:

- Use 5NN instead of 555 through out
- Remove 555 from the draft name
- Explain use of 5NN: https://tools.ietf.org/html/draft-divilly-user-defined-resource-error-00#section-1 <https://tools.ietf.org/html/draft-divilly-user-defined-resource-error-00#section-1>
- Explain why 500 Internal Server Error does not suffice: https://tools.ietf.org/html/draft-divilly-user-defined-resource-error-00#section-2.1 <https://tools.ietf.org/html/draft-divilly-user-defined-resource-error-00#section-2.1>. TLDR, folks only see the status code in the response, they ignore anything else is my experience
- Do not indicate that the draft seeks to update RFC 7231

> I agree this condition is potentially applicable to any resource, but it might fall afoul of another principle that we haven't written down as well -- that HTTP focuses on standardising the interfa…

I see this proposal as enhancing the interface rather than clouding it with implementation details. When an error occurs, then very often some follow up action needs to be taken. Often that action is to file a problem report or examine some logs. The purpose of this status code is to help inform the client with whom to file the problem report or which logs to examine.

This proposal provides a clear classification for tools and humans, that can inform their decision about to how to deal with the error condition.

This falls outside what HTTP does or should specify. I see this proposal as a tiny layer atop the HTTP protocol to help with this pain point. 

I still feel this a pretty specialised use-case at this time and there likely is not enough consensus to standardise this approach, which is why I filed it as an informational draft. My hope with submitting the original draft was that others with similar requirements might come across the draft and choose to harmonise on using 555 as the status code for this condition, and that over time a consensus might build that it would be appropriate to try get it standardised.

I understand now that this approach is not in line with RFC 7231, so I've amended the draft to avoid preferring (squatting on) any particular status code.

Regards,
Colm

[1]: https://lists.w3.org/Archives/Public/ietf-http-wg/2020JanMar/0244.html <https://lists.w3.org/Archives/Public/ietf-http-wg/2020JanMar/0244.html>

> On 26 Mar 2020, at 03:49, Mark Nottingham <[email-masked]> wrote:
> 
> Hi Colm,
> 
>  https://httpwg.org/specs/rfc7231.html#considerations.for.new.status.codes <https://httpwg.org/specs/rfc7231.html#considerations.for.new.status.codes>

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 27 Mar 2020 14:54:49 +1100

Hi Colm,

> On 26 Mar 2020, at 11:27 pm, Colm Divilly <[email-masked]> wrote:
> 
> Hi Mark,

Great.

> Yes, we include information in both the response header and the response body, we have found it to be largely ineffective. Our experience is that tools and humans see 500, think 'I know what that m…

I understand and sympathise. However, if the bar for getting a new status code is "that's what people pay attention to", we're going to run out of status codes pretty quickly (which is the underlying concern with lots of these discussions). Also, the more status codes we mint, the more confusion there is about which one to use -- a phenomenon we already see a lot of.

That doesn't mean that we can't or won't standardise this -- just that we need to be thoughtful about it.

> https://datatracker.ietf.org/doc/draft-divilly-user-defined-resource-error/
> 
> Changes in this draft:
> 
> - Explain use of 5NN: https://tools.ietf.org/html/draft-divilly-user-defined-resource-error-00#section-1
> - Explain why 500 Internal Server Error does not suffice: https://tools.ietf.org/html/draft-divilly-user-defined-resource-error-00#section-2.1. TLDR, folks only see the status code in the response, they ignore anything else is my experience

Thanks, that's a very good start.

> I understand now that this approach is not in line with RFC 7231, so I've amended the draft to avoid preferring (squatting on) any particular status code.

Thanks again.

Regarding next steps (chair "hat" on) -- I'd love to hear what other WG participants (and people who don't participate here regularly, of course) think about this. One of the things we look for is level of interest, as well as level of agreement.

Normally, Tommy and I would probably ask you to present at the next HTTP Working Group meeting in Madrid. It's not clear what's going to happen with that meeting, but I suspect that one way or another we'll be having a WG meeting soon, if only virtually, and we can probably have a discussion there. Between now and then, hopefully we'll see more mailing list discussion.

Is that workable for you?  I'm not sensing a need to get a decision quickly on your part, but if that's not the case, please say so.

I've also added this to our Watch List so we remember to check in on it from time to time:
  https://github.com/httpwg/wiki/wiki/WatchList

Cheers,

> 
> Regards,
> Colm
> 
> [1]: https://lists.w3.org/Archives/Public/ietf-http-wg/2020JanMar/0244.html
>

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 27 Mar 2020 18:25:45 +1300

>From the Squid perspective; this is largely irrelevant.

The few use-cases we have where this code might apply are typically
better served by the existing generic 500 for privacy and security
protection of internal ICAP or CDN origin infrastructure.

AYJ

---

From: Colm Divilly <[email-masked]>
Date: Fri, 27 Mar 2020 08:48:14 +0000

Hi Mark,
 see my responses inline below:

> On 27 Mar 2020, at 03:54, Mark Nottingham <[email-masked]> wrote:
> 

Agreed, we feel this is a pain point for us, but then we'd expect it to be a pain point for others operating a similar model (where the party hosting the resource is not directly connected to the party authoring the resource), but we haven't seen any indication of that. My hunch is that others have more or less sidestepped this issue, by giving their tenants more control of the HTTP stack, effectively making them the operator of their HTTP resources. Thus the distinction between operator and author becomes moot.

> Is that workable for you?  I'm not sensing a need to get a decision quickly on your part, but if that's not the case, please say so.

No urgency, my gut feel is that for 99% of cases the author of the resource and operator of the server have a close relationship and this would provide zero value. I feel like our model is an edge case, we see a class of customer not interested in operating their own HTTP infrastructure, but still wanting to publish rich Web facing resources. It's only in that use case that this might become useful. Happy to present if there's any interest in the approach.

Regards,
Colm