Subject: HTTP Alternative Services Best Practices?

From: Lucas Pardue <[email-masked]>
Date: Mon, 16 Dec 2019 12:01:51 +0000

Hello QUIC and HTTP members,

HTTP Alternative Services (Alt-Svc) is specified in RFC 7838 which was
published in 2016 [1]. Many of us are starting to use Alt-Svc and I wonder
if its appearance of simplicity might cause some unintended effects on the
Internet. In the 3 or so years since it was published, have any best
practices emerged that might be useful to capture.

Major uses of Alt-Svc today in no particular order: switching to gQUIC
(typically on the same port), switching to HTTP/3, Opportunistic Encryption
(RFC 8164) [2], Opportunistic Onion (advertising .onion [3]), and traffic
management by advertising alternatives with different destination IPs or
network path characteristics.

Arguably, HTTP/3 will be the largest-scale deployed use case of Alt-Svc
both in terms of advertisements and clients that take them up. Alt-Svc for
this can be deceptively simple, which may lead to unexpected outcomes. For
example, the minimal expression:

Alt-Svc: h3-24=":443"

invokes default values for parameters, "ma" is fresh for 24 hours and
"persist" is false (i.e. clear alternative cache on network changes). One
could imagine how this could cause bursts of activity at regular periods,
or cascades due to end-user local conditions such as flocking or hopping.

Finally, the proposal for an HTTPSVC DNS record [4] might attract a
different population of operator that is less familiar with the expected
behaviour of Alt-Svc implementations.

Does anyone think it would be useful to share or document some guidance?

Cheers
Lucas

[1] https://tools.ietf.org/html/rfc7838
[2] https://tools.ietf.org/html/rfc8164
[3] https://tools.ietf.org/html/rfc7686
[4] https://tools.ietf.org/html/draft-ietf-dnsop-svcb-httpssvc-01

---

From: Ryan Hamilton <[email-masked]>
Date: Mon, 16 Dec 2019 12:23:34 -0800

Thanks for raising this issue! I think documenting best practices/guidance
would be a great idea. I think the Google servers advertise a 30 day
expiration (because 24 hours is *really* short). Further, it turns out that
Chrome implicitly assumes persist = true (because Chrome incorrectly
doesn't implement persist = false). We have a bunch of deployment
experience with this model which I think suggests that shorter ttls and
persist = false would be have a deleterious impact on performance.

---

From: Mike Bishop <[email-masked]>
Date: Tue, 17 Dec 2019 19:27:47 +0000

Saying that “persist=false would be deleterious” is a bit simplistic.  You wouldn’t use persist=false for your scenarios, true enough.  Persist=true is intended for alternatives that aren’t derived from the user’s network location, and therefore don’t need to be flushed when the network location changes.  (For example, this host also supports QUIC.)

However, there are other scenarios where it is location-dependent – a CDN routing you to a closer node because DNS or Anycast didn’t get you close enough, for example.  If you change networks, not only is the node you were directed to no longer your closest, it might actually be prohibited from serving users on other networks.

The right answer here is that you need to know which type of Alt-Svc you’re issuing.

From: QUIC <[email-masked]> On Behalf Of Ryan Hamilton
Sent: Monday, December 16, 2019 3:24 PM
To: Lucas Pardue <[email-masked]>
Cc: QUIC WG <[email-masked]>; HTTP Working Group <[email-masked]>
Subject: Re: HTTP Alternative Services Best Practices?

Thanks for raising this issue! I think documenting best practices/guidance would be a great idea. I think the Google servers advertise a 30 day expiration (because 24 hours is *really* short). Further, it turns out that Chrome implicitly assumes persist = true (because Chrome incorrectly doesn't implement persist = false). We have a bunch of deployment experience with this model which I think suggests that shorter ttls and persist = false would be have a deleterious impact on performance.

On Mon, Dec 16, 2019 at 4:02 AM Lucas Pardue <[email-masked]<mailto:[email-masked]>> wrote:
Hello QUIC and HTTP members,

HTTP Alternative Services (Alt-Svc) is specified in RFC 7838 which was published in 2016 [1].. Many of us are starting to use Alt-Svc and I wonder if its appearance of simplicity might cause some unintended effects on the Internet. In the 3 or so years since it was published, have any best practices emerged that might be useful to capture.

Major uses of Alt-Svc today in no particular order: switching to gQUIC (typically on the same port), switching to HTTP/3, Opportunistic Encryption (RFC 8164) [2], Opportunistic Onion (advertising .onion [3]), and traffic management by advertising alternatives with different destination IPs or network path characteristics..

Arguably, HTTP/3 will be the largest-scale deployed use case of Alt-Svc both in terms of advertisements and clients that take them up. Alt-Svc for this can be deceptively simple, which may lead to unexpected outcomes. For example, the minimal expression:

Alt-Svc: h3-24=":443"

invokes default values for parameters, "ma" is fresh for 24 hours and "persist" is false (i.e. clear alternative cache on network changes). One could imagine how this could cause bursts of activity at regular periods, or cascades due to end-user local conditions such as flocking or hopping.

Finally, the proposal for an HTTPSVC DNS record [4] might attract a different population of operator that is less familiar with the expected behaviour of Alt-Svc implementations.

Cheers
Lucas

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 17 Dec 2019 19:45:06 +0000

I agree with both of your points. IMHO there are a few patterns of
deployment that likely benefit from different approaches to Alt-Svc usage.
RFC7540 provides the toolkit but not everything is a hammer nor a nail.

In my original email I also neglected to mention the Alt-Used header or the
"clear" special value, which may have different considerations applicable
to different deployments. Is there much experience with these at Internet
scale?

Lucas

On Tue, 17 Dec 2019, 19:27 Mike Bishop, <[email-masked]> wrote:

> Saying that “persist=false would be deleterious” is a bit simplistic.  You
> wouldn’t use persist=false for your scenarios, true enough.  Persist=true
> is intended for alternatives that aren’t derived from the user’s network
> location, and therefore don’t need to be flushed when the network location
> changes.  (For example, this host also supports QUIC.)
>
>
>
> However, there are other scenarios where it is location-dependent – a CDN
> routing you to a closer node because DNS or Anycast didn’t get you close
> enough, for example.  If you change networks, not only is the node you were
> directed to no longer your closest, it might actually be prohibited from
> serving users on other networks.
>
>
>
> *From:* QUIC <[email-masked]> *On Behalf Of * Ryan Hamilton
> *Sent:* Monday, December 16, 2019 3:24 PM
> *To:* Lucas Pardue <[email-masked]>
> *Cc:* QUIC WG <[email-masked]>; HTTP Working Group <[email-masked]>
> *Subject:* Re: HTTP Alternative Services Best Practices?
>
>
>
> published in 2016 [1].. Many of us are starting to use Alt-Svc and I wonder
> network path characteristics..
>
>
>

---

From: Daniel Stenberg <[email-masked]>
Date: Tue, 17 Dec 2019 20:48:28 +0100 (CET)

On Mon, 16 Dec 2019, Lucas Pardue wrote:

> could imagine how this could cause bursts of activity at regular periods, or 
> cascades due to end-user local conditions such as flocking or hopping.

The "persist" thing gives me, as author of the alt-svc handling in curl, a 
small headache too. Let me elaborate:

curl is a command line tool that you typically fire off one or two requests 
with now and then again at a later point.

When curl saves/caches the alt-svc information because we want to take 
advantage of for example HTTP/3 at subsequent invokes, we just have no idea if 
the network changed or not in between the invokes. Figuring out if a network 
chanfes is hard enough when you catch it in run-time (I wrote a lot of such 
code for Firefox), but even worse when you do occasional runs with potentially 
long intervals.

I'm struggling to come up with a decent way to deal with this. Right now I 
just ignore the persistent parameter completely, but I realize that's just 
waiting for a case when it will hurt me. I'm leaning towards either always 
consider the network as changed, or to add some very rudimentary checks that 
for example consider it the same if the source IP is still the same or 
something...

(the curl alt-svc implementation is still marked "experimental" and as such 
isn't actually used very widely yet)

-- 

  / daniel.haxx.se

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 17 Dec 2019 23:07:18 +0000

Hi Cory,

I think your questions are probably good implementation-oriented ones but
less about BCP to my mind. If there is something worth documenting, my
instinct is a separate document that compliments RFC7838.

To answer:

On Tue, 17 Dec 2019, 22:46 Cory Nelson, <[email-masked]> wrote:

> Two questions I had while implementing Alt-Svc, and ended up looking at
> other implementations for guidance:
>
>
>
> Is it valid to have Alt-Svc: clear followed by, on another line, more
> Alt-Svc to replace the old services?
>
RFC7838 says about Alt-Svc:

"The field value consists either of a list of values, each of which
indicates one alternative service, or the keyword "clear"."

So I think it is clear that it is invalid to have both clear and an
alt-value. But what might be less clear is the expected failure mode if
this does actually occur.

> Is it valid for an authority to extend their lease by sending an Alt-Svc
> for themselves?
>
If you mean is it valid for a selected alternative to advertise itself and
therefore keep itself fresh, then yes, the alternative is fully
authoritative and I think that this is captured in the specification.

Cheers
Lucas

---

From: Ryan Hamilton <[email-masked]>
Date: Tue, 17 Dec 2019 16:01:08 -0800

On Tue, Dec 17, 2019 at 11:27 AM Mike Bishop <[email-masked]> wrote:

> Saying that “persist=false would be deleterious” is a bit simplistic.
>

Yes, totally fair.

[snip] The right answer here is that you need to know which type of Alt-Svc
> you’re issuing.

Completely agree! All of which suggests some sort of Alt-Svc best
practices/guidance doc would be helpful here.

---

From: Ryan Hamilton <[email-masked]>
Date: Tue, 17 Dec 2019 16:04:36 -0800

On Tue, Dec 17, 2019 at 11:45 AM Lucas Pardue <[email-masked]>
wrote:

> In my original email I also neglected to mention the Alt-Used header or
> the "clear" special value, which may have different considerations
> applicable to different deployments. Is there much experience with these at
> Internet scale?
>

I believe that Chrome does not set the Alt-Used header (missing Chrome
feature #2 in this thread :>) so I have no experience there. On the other
hand we do have experience with clear. We've used it, I believe, when
attempting to shut QUIC off because of security problems. (*sigh*) This
seemed to be relatively effective (as you would expect) in getting clients
to cease attempting to make QUIC handshakes. But I'm not sure we have much
to write home about beyond that...

---

From: Cory Nelson <[email-masked]>
Date: Tue, 17 Dec 2019 22:46:45 +0000

Two questions I had while implementing Alt-Svc, and ended up looking at other implementations for guidance:

Is it valid to have Alt-Svc: clear followed by, on another line, more Alt-Svc to replace the old services?
Is it valid for an authority to extend their lease by sending an Alt-Svc for themselves?

Would be nice if spec could get updated to clarify these and other best practices.

From: Lucas Pardue <[email-masked]>
Sent: Tuesday, December 17, 2019 11:45 AM
To: Mike Bishop <[email-masked]>
Cc: Ryan Hamilton <rch=[email-masked]>; QUIC WG <[email-masked]>; HTTP Working Group <[email-masked]>
Subject: [EXTERNAL] Re: HTTP Alternative Services Best Practices?

I agree with both of your points. IMHO there are a few patterns of deployment that likely benefit from different approaches to Alt-Svc usage. RFC7540 provides the toolkit but not everything is a hammer nor a nail.

In my original email I also neglected to mention the Alt-Used header or the "clear" special value, which may have different considerations applicable to different deployments. Is there much experience with these at Internet scale?

Lucas

On Tue, 17 Dec 2019, 19:27 Mike Bishop, <[email-masked]<mailto:[email-masked]>> wrote:
Saying that “persist=false would be deleterious” is a bit simplistic.  You wouldn’t use persist=false for your scenarios, true enough.  Persist=true is intended for alternatives that aren’t derived from the user’s network location, and therefore don’t need to be flushed when the network location changes.  (For example, this host also supports QUIC.)

From: QUIC <[email-masked]<mailto:[email-masked]>> On Behalf Of Ryan Hamilton
Sent: Monday, December 16, 2019 3:24 PM
To: Lucas Pardue <[email-masked]<mailto:[email-masked]>>
Cc: QUIC WG <[email-masked]<mailto:[email-masked]>>; HTTP Working Group <[email-masked]<mailto:[email-masked]>>
Subject: Re: HTTP Alternative Services Best Practices?

Alt-Svc: h3-24=":443"

Cheers
Lucas

[1] https://tools.ietf.org/html/rfc7838<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc7838&data=02%7C01%7CCory.Nelson%40microsoft.com%7C77afde374a174f49929608d7832a0332%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637122088801114286&sdata=fVy2O5a31U%2BT41Qnp4TtAGbDyJr%2B8RZzfGoPYTkctIE%3D&reserved=0>
[2] https://tools.ietf.org/html/rfc8164<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc8164&data=02%7C01%7CCory.Nelson%40microsoft.com%7C77afde374a174f49929608d7832a0332%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637122088801123984&sdata=pHhXebysCX7H8ywUVRFn6du%2BUL46MyBV64hInK%2BOcAk%3D&reserved=0>
[3] https://tools.ietf.org/html/rfc7686<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc7686&data=02%7C01%7CCory.Nelson%40microsoft.com%7C77afde374a174f49929608d7832a0332%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637122088801123984&sdata=vobQdfgEayQ3MK%2FXzNB%2FFqtWmun7JFyfozBkFgJjAvs%3D&reserved=0>
[4] https://tools.ietf.org/html/draft-ietf-dnsop-svcb-httpssvc-01<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-dnsop-svcb-httpssvc-01&data=02%7C01%7CCory.Nelson%40microsoft.com%7C77afde374a174f49929608d7832a0332%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637122088801133979&sdata=o8dZyNm8YjM7ybwxLZRF7xpZyLpB5%2BLXjVbXgW1dgxI%3D&reserved=0>