Subject: Re: [tsvwg] Why measure RTT passively?

From: Tom Herbert <[email-masked]>
Date: Tue, 18 Sep 2018 15:53:34 -0700

On Tue, Aug 21, 2018 at 5:42 AM, Brian Trammell (IETF) <[email-masked]> wrote:
> Greetings, TSVWG, QUIC, and IPPM,
>
> I've just submitted a new I-D, "Why do we need passive measurement of round trip time?" (https://tools.ietf.org/html/draft-trammell-why-measure-rtt-00), which may be of some interest to each of these WGs. :) It presents a set of use cases for on-path passive RTT measurement, both generally, and specific to the facility described in "A Transport-Independent Explicit Signal for Hybrid RTT Measurement" (https://tools.ietf.org/html/draft-trammell-tsvwg-spin-00), or the "spin bit".
>
Hi Brian,

I looked at that draft draft-trammell-tsvwg-spin-00. It's unclear to
me how this mechanism is transport-independent. The draft shows how to
do this in QUIC and TCP, but there's no support that I see for DCCP,
SCTP, IPsec, other encapsulations, etc. I would think that for
something to be truly transport-independent the information would need
to reside in the network layer.

Tom

> - For QUIC, this draft addresses the second half of the risk/utility question raised by the spin bit and spin+VEC latency signal -- much of the content comes from the original QUIC spin bit document, but there are new sections on using VEC for loss/reordering proxy metrics and exploiting correlations between RTT and other metrics to drive intraflow diagnostics from only RTT signals. In particular, I'll talk about ongoing experimentation along the lines of section 3 at the interim in New York.
>
> - For TSVWG and tsvarea, it addresses some of the "why design wire images" questions raised in the tsvarea meeting in Montreal, with respect to the specific tasks for which RTT data is useful.
>
> - And for IPPM, it gives a bit more background on the "passive RTT" metric in the initial registry document.
>
> Feedback is much appreciated. :)
>
> Thanks, cheers,
>
> Brian (as an individual contributor)
>
> P.S. apologies for the resend: the previous message was not an obscure joke about ubiquitous encryption: apparently the interaction among my MUA, its bolt-on GPG, and the somewhat underprovisioned network in SIGCOMM led to an encrypted draft being posted to the list. :/
>

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Tue, 18 Sep 2018 18:59:12 -0500

Speaking without a hat (heck, I don't even have hair) ...

On Tue, Sep 18, 2018 at 5:53 PM Tom Herbert <[email-masked]> wrote:

> > I've just submitted a new I-D, "Why do we need passive measurement of
> round trip time?" (
> https://tools.ietf.org/html/draft-trammell-why-measure-rtt-00), which may
> be of some interest to each of these WGs. :) It presents a set of use cases
> for on-path passive RTT measurement, both generally, and specific to the
> facility described in "A Transport-Independent Explicit Signal for Hybrid
> RTT Measurement" (https://tools.ietf.org/html/draft-trammell-tsvwg-spin-00),
> or the "spin bit".
> >
> Hi Brian,
>

Forget that I'm an AD, because I'm not wearing any hats, but I'm having a
really difficult time explaining to other people internally how a spin bit
that's not in the IP header, but is part of a mostly-encrypted transport
header, is better than putting something in an IP (extension? iOAM?) header
that would be transport-independent.

I recognize there are trade-offs, starting with the availability of bits in
headers at different layers of the protocol stack, and I recognize that I
probably don't know what all of them are, so it's likely that I'm missing
something critical here.

Me missing something critical would surprise no one, but as an individual
participant, I have the same question Tom has.

Thanks,

Spencer

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 19 Sep 2018 13:46:31 +0200

> On 19. Sep 2018, at 00:53, Tom Herbert <[email-masked]> wrote:
> 
I agree with Tom, right now this does not look like transport-independent.
One could add a chunk for SCTP to expose the 3 bits, but I was wondering
why not just let the endpoints willing to expose the RTT signal let them
send the measured RTT explicitly?

Best regards
Michael

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Wed, 19 Sep 2018 08:28:44 -0400

> On 18 Sep 2018, at 18:53, Tom Herbert <[email-masked]> wrote:
> 
> SCTP, IPsec, other encapsulations, etc.

Those bindings are future work -- and, indeed, the draft should be clearer about that. Mea culpa, usual excuses about -00, and all that.

> I would think that for

This is, architecturally speaking, my preference as well. The reasons we propose this as a layer 4 signal, as opposed to a layer 3 signal, are elaborated in section 5.1. The overwhelming one here is that ports are often used in combination with addresses to make forwarding decisions, such that a 2-tuple does not fully describe a "path" between two endpoints at any instance in time.

IOW, we're trying to solve the measurement problem it looks like we have, not the one we'd like to.

That said, the technique described here could also be applied to three bits (theoretically clawed back) from the network layer header, with the logic to be applied to the flow aggregate of all packets sent between two IP addresses. Since there's no two-tuple state structure that we could already hang this off (that I know of), this would require the design of new state structures in the IP stack.

Cheers,

Brian

---

From: Christian Huitema <[email-masked]>
Date: Wed, 19 Sep 2018 08:24:00 -0400

> On Sep 18, 2018, at 7:59 PM, Spencer Dawkins at IETF <[email-masked]> wrote:
> 
> Forget that I'm an AD, because I'm not wearing any hats, but I'm having a really difficult time explaining to other people internally how a spin bit that's not in the IP header, but is part of a mostly-encrypted transport header, is better than putting something in an IP (extension? iOAM?) header that would be transport-independent. 

The one advantage of the bit in transport header is that it is authenticated end to end. The other advantage is that it is automatically tied to a transport flow and its five tuple. Then there is the reality that IP header bits are commonly rewritten by a variety of middleboxes.

The downside is that different encrypted transports will most likely end up adopting different encodings and conventions.

But of course we could also repurpose one of the TOS bits or one of the flow header bits for the purpose, to mirror the transport level bit. Mirror instead of replace: that would alleviate the need of parsing transport headers, but still allow end to end detection of IP header rewriting.

-- Christian Huitema

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 19 Sep 2018 14:56:03 +0100

On 19/09/2018 13:24, Christian Huitema wrote:
> But of course we could also repurpose one of the TOS bits
Do you mean the DSCP field? - That would sound like a very odd proposal.
> or one of the flow header bits for the purpose, to mirror the transport level bit. Mirror instead of replace: that would alleviate the need of parsing transport headers, but still allow end to end detection of IP header rewriting.
>
> -- Christian Huitema

Gorry

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 19 Sep 2018 15:57:21 +0200

Hi Brian,

On Wed, Sep 19, 2018 at 08:28:44AM -0400, Brian Trammell (IETF) wrote:
> Those bindings are future work -- and, indeed, the draft should be clearer
> about that. Mea culpa, usual excuses about -00, and all that.
> 
> > I would think that for
> This is, architecturally speaking, my preference as well. The reasons we
> propose this as a layer 4 signal, as opposed to a layer 3 signal, are
> elaborated in section 5.1. The overwhelming one here is that ports are often
> used in combination with addresses to make forwarding decisions, such that a
> 2-tuple does not fully describe a "path" between two endpoints at any
> instance in time.

I think the mechanism described in the draft could be generalized to
any transport layer. You could even imagine stuffing this into Websocket
or H2 to provide measurements at *certain* layers if that made sense. I
tend to think that the document should focus more on the algorithm itself
and how to reliably implement it, and discuss application to existing
transport layers only as examples of what an implementation could look
like. It would address the reserves you've set regarding applicability
to TCP in the real world and would possibly help other implementations
adopt similar techniques.

By the way I'm seeing an minor problem there related to how you decide
how flips the bit : you need to know who's the connection initiator and
who's the target. In TCP you have a simultaneous connect feature which
allows two clients to connect to each other if they know their respective
source ports (try it using netcat if you want), thus maybe there are
other protocols where it's not that obvious who's the initiator. Maybe
you should take this as a counter-example to try to figure a more robust
mechanism to decide who flips the bit. Maybe it could be derived from
other information such as the packet/sequence numbers, maybe it could
be figured from the spin bit mechanism itself (spin after a certain
VEC sequence or so).

Regards,
Willy

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Wed, 19 Sep 2018 10:30:05 -0400

hi Willy, all,

> On 19 Sep 2018, at 09:57, Willy Tarreau <[email-masked]> wrote:
> 
> Hi Brian,

<snip>

> or H2 to provide measurements at *certain* layers if that made sense.

Yes, but. The signal is only useful for passive measurement if it's visible along the path, and anything beyond the encryption boundary is not visible. So H2/WS spinning would not appear to be useful here.

> I
> like.

This is one approach we could take, but at least for the -00 revision we wanted to show how this worked for both QUIC and TCP, because that's what we implemented (see also our preprint, to appear at IMC: https://mami-project.eu/wp-content/uploads/2018/09/spinbit.pdf).

> It would address the reserves you've set regarding applicability
> other protocols where it's not that obvious who's the initiator.

Yes, this is an issue we have not addressed yet, and is derived from the fact that QUIC is explicitly asymmetric, as it is basically a transport protocol wrapped around TLS, which is also explicitly asymmetric.

> Maybe
> VEC sequence or so)

Without thinking too hard about it, one could break an initiator/responder tie for simultaneous open by sorting two-tuples.

Thanks, cheers,

Brian

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 19 Sep 2018 16:47:01 +0200

Hi Brian,

On Wed, Sep 19, 2018 at 10:30:05AM -0400, Brian Trammell (IETF) wrote:
> Yes, but. The signal is only useful for passive measurement if it's visible
> along the path, and anything beyond the encryption boundary is not visible.
> So H2/WS spinning would not appear to be useful here.

It depends. Typically TLS is often stripped at the edge of an
infrastructure, and providing an easy means of measuring some end-to-end
performance can be useful. It could also be useful within the ends
themselves. Right now if a server needs to measure the RTT on an H2
connection, I guess it has to actively send PING frames and implement
its own mechanism. Maybe your mechanism could be used in a more natural
way so that both ends constantly have access to that signal and know
more accurate information in a passive way. These are just ideas, nothing
more, but I see some value in this work that could be used beyond its
initial design purpose.

> This is one approach we could take, but at least for the -00 revision we
> wanted to show how this worked for both QUIC and TCP, because that's what we
> implemented (see also our preprint, to appear at IMC:
> https://mami-project.eu/wp-content/uploads/2018/09/spinbit.pdf).

OK that makes sense of course.

(...)
> Yes, this is an issue we have not addressed yet, and is derived from the fact
> that QUIC is explicitly asymmetric, as it is basically a transport protocol
> wrapped around TLS, which is also explicitly asymmetric.
(...)
> Without thinking too hard about it, one could break an initiator/responder
> tie for simultaneous open by sorting two-tuples.

Yes very likely, I just think that for completeness it deserves being
studied a little bit if you want to see it adopted/adoptable anywhere.

Thanks,
Willy

---

From: Tom Herbert <[email-masked]>
Date: Wed, 19 Sep 2018 07:59:43 -0700

> send the measured RTT explicitly?
Yes! The application or at least network stack will have this
information and much more that might be of interest to the network--
they will also know RTT variance, packet loss on the flow, number of
out of order packets received, etc. If the network needs this sort of
information why not just ask the application for it instead of trying
deduce it from a limited view of the transport layer and a couple of
fundamentally insecure bits in the packet?

Tom

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 19 Sep 2018 11:14:13 -0400

Tom, it may be quite difficult to ask the application. An operator may have
many thousands ongoing connections with mobile phones and want to measure
the overall network quality. It would be quite impossible to discuss
latency with all phones in the network. Additionally it would require a
secure connection, trust, etc. etc.

Passive measurement is invasive in the sense that it steals private
information that could be abused without consent. So the balance is expose
as little as possible, but enough to make it useful for reasonable purposes.

Trammels paper shows that a very high quality can be achieved with 3 bits,
which is a lot simpler than asking an application, whether the option
existed or not. The question is more: would 8 bits, or a full sequence
number be better? The answer, according to the paper is no - only in
certain lossy conditions, and not by very much. On the other hand, more
information leaks more private information.

Furthermore, the simplicity of 3 bits make it possible to stuff the bits
almost anywhere, so the concept is universal.

There is almost no bad things to say about it, except the connection
asymmetri mentioned, and the fact it must happen in the layer 4 packet
rather than the IP packet. But then, it could be mirrored into a lower
layer protocol header since it is space efficient.

On 19 September 2018 at 17.00.33, Tom Herbert ([email-masked]) wrote:

round trip time?" (
> I agree with Tom, right now this does not look like
transport-independent..

Tom

> Best regards
> Michael
>>
>> Tom
>>
>>> - For QUIC, this draft addresses the second half of the risk/utility
question raised by the spin bit and spin+VEC latency signal -- much of the
content comes from the original QUIC spin bit document, but there are new
sections on using VEC for loss/reordering proxy metrics and exploiting
correlations between RTT and other metrics to drive intraflow diagnostics
from only RTT signals. In particular, I'll talk about ongoing
experimentation along the lines of section 3 at the interim in New York.
>>>
>>> - For TSVWG and tsvarea, it addresses some of the "why design wire
images" questions raised in the tsvarea meeting in Montreal, with respect
to the specific tasks for which RTT data is useful.
>>> P.S. apologies for the resend: the previous message was not an obscure
joke about ubiquitous encryption: apparently the interaction among my MUA,
its bolt-on GPG, and the somewhat underprovisioned network in SIGCOMM led
to an encrypted draft being posted to the list. :/

---

From: Tom Herbert <[email-masked]>
Date: Wed, 19 Sep 2018 10:13:06 -0700

Hi Brian,

The fact that every transport protocol can be changed to support this
feature really doesn't make it transport independent. Besides, even if
you did change a protocol like SCTP to support spin bit,  it's useless
until intermediate devices support the transport protocol-- it's
simply a fantasy to think that vendors are going to rush out to
support every possible transport protocol. At best, instead of the
Internet being ossified around one transport protocol it might be
ossified around two. IMO, QUIC encrypting all the transport layer was
a big step forward in the fight against protocol ossification.
Purposely leaking any transport layer information, like the spin bit,
is a step backward in that regard.

The obvious answer to make this transport independent is to put
information into a Hop-by-Hop option. Not only does HBH work with any
transport protocol, it also allows a rich set of information, and can
correctly be interpreted and optionally changed at intermediate
devices, and can be secured. This is why FAST uses it for instance.
The predictable counter argument is of course that IPv6 options don't
work on some paths in the Internet. But, I view that as a _current_
condition. I see nothing architecturally that prevents fixing the
situation or applying "happy eyeballs" like mechanisms to mitigate the
problem. In fact defining useful options could break deadlock where
vendors aren't going to support options before useful ones have been
defined. If given a choice between middlebox vendors trying to
transport support every transport or working on generic
transport-independent solutions, I'd much rather see the latter
happen.

Tom

---

From: Tom Herbert <[email-masked]>
Date: Wed, 19 Sep 2018 13:12:02 -0700

> The one advantage of the bit in transport header is that it is authenticated end to end. The other advantage is that it is automatically tied to a transport flow and its five tuple. Then there is t…
Christian,

That only helps to a degree. Intermediate nodes wouldn't be able to
authenticate those bits, so they'll just accept whatever is there
regardless of whether packets might be ultimately be dropped at the
endpoint. So it's still possible that if the bits are modified to
(adversely?) affect the network state maintained for a flow.

One disadvantage of QUIC spin bit compared to TCP is that the QUIC
lies within a UDP payload which would mean that an intermediate device
would need to parse the payload. Presumably, the only way an
intermediate node could detect a QUIC packet is by matching port
numbers to 80 or 443. But, as described in RFC7605 port numbers only
have meaning at endpoints, so it's possible that an application might
be using a QUIC assigned port for a completely different purpose and
interpreting the payload as QUIC might be completely wrong.  IP
protocol numbers don't suffer from this ambiguity; if a next header is
0x6 it is considered a TCP packet anywhere in the network. This
ambiguity should probably be mentioned in the QUIC spin bit draft.

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 20 Sep 2018 08:49:28 +1200

> Do you mean the DSCP field? - That would sound like a very odd proposal.

To be clear, there are no free bits in the DSCP field, there are only
free codepoints in the 6-bit space. And of course since DSCP values
explicitly act as a QoS request, they also implicitly change the
one-way latency and therefore the RTT. So the DSCP bits would be
remarkably bad as a tool for RTT measurement.

   Brian C

> .

---

From: Tom Herbert <[email-masked]>
Date: Wed, 19 Sep 2018 14:17:49 -0700

>>> or one of the flow header bits for the purpose, to mirror the transport level bit. Mirror instead of replace: that would alleviate the need of parsing transport headers, but still allow end to en…

Neither are there any flow label bits available. Re-purposing even a
single bit would adversely effect deployed use cases where flow label
is being used for packet steering or other purposes. Using the flow
label for packet marking was discussed in
https://tools.ietf.org/html/draft-fioccola-spring-flow-label-alt-mark-01

Tom

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 19 Sep 2018 17:30:35 -0400

On the actual experiments:

has there been any work on patterns on a single bit? At some point we
discussed various signals including square wave, PI, and fractal gray codes.

If privacy or space is a concern with 3 bits, perhaps a gray code could
solve the same problem trading space for time.

On 19 September 2018 at 23.18.25, Tom Herbert ([email-masked]) wrote:

On Wed, Sep 19, 2018 at 1:49 PM, Brian E Carpenter
<[email-masked]> wrote:
>>>> Forget that I'm an AD, because I'm not wearing any hats, but I'm
having a really difficult time explaining to other people internally how a
spin bit that's not in the IP header, but is part of a mostly-encrypted
transport header, is better than putting something in an IP (extension?
iOAM?) header that would be transport-independent.
>>> The one advantage of the bit in transport header is that it is
authenticated end to end. The other advantage is that it is automatically
tied to a transport flow and its five tuple. Then there is the reality that
IP header bits are commonly rewritten by a variety of middleboxes.
>>>
>>> The downside is that different encrypted transports will most likely
end up adopting different encodings and conventions.
>>> or one of the flow header bits for the purpose, to mirror the transport
level bit. Mirror instead of replace: that would alleviate the need of
parsing transport headers, but still allow end to end detection of IP
header rewriting.

Tom

---

From: <[email-masked]>
Date: Wed, 19 Sep 2018 23:35:56 +0200

> https://tools.ietf.org/html/draft-fioccola-spring-flow-label-alt-mark-01

One very hackish way would be to use the TTL:

Assuming that the number of L3 hops is constant long enough for a given 5-tuple, 
one can add a small varying offset to the TTL. The baseline can be learnt on the 
receiver by simple minimum extraction, assuming offset 0 is guaranteed to happen 
early in the connection's lifetime.

For example, adding a value in [0,7] to the normal TTL of the sender's OS
  (1) is doable as long as the value is less than 248
  (2) allows to encode 3 bits
  (3) will not upset the receiver nor any intervening node

But granted, that's disgusting.

-Alex

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Christian Huitema <[email-masked]>
Date: Wed, 19 Sep 2018 17:42:28 -0400

> On Sep 19, 2018, at 5:17 PM, Tom Herbert <[email-masked]> wrote:
> 

OK. So basically, all the bits are taken for both v4 and v6. Spencer's original question why, should we put this kind of tracing bit in the transport header or in the network header. In theory, there are some arguments for putting common functions in the network header rather than in a variety of transport headers. But in practice that would mean some kind of IP option, and that's both more complicated and more error prone than reserving a bit in a transport header.

-- Christian Huitema

---

From: <[email-masked]>
Date: Wed, 19 Sep 2018 23:46:19 +0200

We've spent some energy trying to get something from 1 bit indeed. The problem 
is twofold:
  (1) robustness to reordering => only commutative things like unary or average 
(like 1-bit DACs) work
  (2) scarce outgoing flow => as few as 2 ACKs per RTT happen => must spread the 
info over a looong time if you're the receiver

For an example with unary see the "E" bit in our 3-bit proposal (spin + 2 loss 
bits) at page 15 of

https://github.com/quicwg/wg-materials/blob/master/interim-18-09/Spin%20bit%20and%20beyond%20%40%20Orange%20v1.pdf

On 09/19/18 23:30, Mikkel Fahnøe Jørgensen wrote:
> If privacy or space is a concern with 3 bits, perhaps a gray code could solve 
> the same problem trading space for time.
>
> On 19 September 2018 at 23.18.25, Tom Herbert ([email-masked] 
> <mailto:[email-masked]>) wrote:
>
>> <[email-masked] <mailto:[email-masked]>> wrote:
>> >>>> On Sep 18, 2018, at 7:59 PM, Spencer Dawkins at IETF 
>> <[email-masked] <mailto:[email-masked]>> wrote:
>> >>>>
>> >>>> Forget that I'm an AD, because I'm not wearing any hats, but I'm having 
>> a really difficult time explaining to other people internally how a spin bit 
>> >>> The downside is that different encrypted transports will most likely end 
>> up adopting different encodings and conventions.
>> >>>
>> parsing transport headers, but still allow end to end detection of IP header 
>> rewriting.
>>

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 20 Sep 2018 10:35:33 +1200

> OK. So basically, all the bits are taken for both v4 and v6. Spencer's original question why, should we put this kind of tracing bit in the transport header or in the network header. In theory, the…

I agree. The main references we have for this sort of thing turn out to be
RFC3514 and RFC8136.

   Brian