Subject: What CSRs Do

From: "Border, John" <[email-masked]>
Date: Tue, 19 Nov 2019 03:34:08 +0000

I agree that it is too late to solve problems like troubleshooting packet loss for v1.  Plus, we want a good solution (which takes time).  But, we need to solve this problem or it will impact QUICv1 deployment...

     When a customer calls reporting that an application does not work, the Customer Service Representative is trained to do whatever it takes to fix the problem as quickly as possible.  Time on the phone equals money.  And, unhappy customers equal churn.  There is a lot of anecdotal learning involved with the troubleshooting process.  If the CSRs learn that disabling QUIC fixes the problem (because the application falls back to TCP), eventually this will become the first thing that is tried every time.  Eventually, their supervisors will push for disabling it across the board.  Once disabled, it is a lot of work to get it reenabled.

John

---

From: Frode Kileng <[email-masked]>
Date: Tue, 19 Nov 2019 04:51:14 +0100

On 19/11/2019 04:34, Border, John wrote:
>
>      I agree that it is too late to solve problems like 
> troubleshooting packet loss for v1.  Plus, we want a good solution 
> (which takes time).  But, we need to solve this problem or it will 
> impact QUICv1 deployment…
>
>      When a customer calls reporting that an application does not 
> work, the Customer Service Representative is trained to do whatever it 
> takes to fix the problem as quickly as possible.  Time on the phone 
> equals money.  And, unhappy customers equal churn.  There is a lot of 
> anecdotal learning involved with the troubleshooting process. If the 
> CSRs learn that disabling QUIC fixes the problem (because the 
> application falls back to TCP), eventually this will become the first 
> thing that is tried every time. Eventually, their supervisors will 
> push for disabling it across the board.  Once disabled, it is a lot of 
> work to get it reenabled.
>
I do think this exemplifies that some network operators have operational 
practices they want to preserve and requests "packet loss" signal 
support in v1. Other operators use alternative operational practices. 
Should the WG  make efforts to contribute to preserving current 
practices of some network operators?

Frode

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 19 Nov 2019 05:29:04 +0100

On Tue, Nov 19, 2019 at 04:51:14AM +0100, Frode Kileng wrote:
> >      I agree that it is too late to solve problems like troubleshooting
> > packet loss for v1.  Plus, we want a good solution (which takes time). 
> > But, we need to solve this problem or it will impact QUICv1
> > deployment...
> > 
> >      When a customer calls reporting that an application does not work,
> > the Customer Service Representative is trained to do whatever it takes
> > to fix the problem as quickly as possible.  Time on the phone equals
> > money.  And, unhappy customers equal churn.  There is a lot of anecdotal
> > learning involved with the troubleshooting process. If the CSRs learn
> > that disabling QUIC fixes the problem (because the application falls
> > back to TCP), eventually this will become the first thing that is tried
> > every time. Eventually, their supervisors will push for disabling it
> > across the board.  Once disabled, it is a lot of work to get it
> > reenabled.
> > 
> practices they want to preserve and requests "packet loss" signal support in
> v1. Other operators use alternative operational practices.

The only known alternative is to claim "we have no loss here" and point the
finger at the next step in the chain operated by someone else. Actually
this is already what incompetent operators do, hence the common need for
captures to know whether everything is OK at one point or not. Usually
this has to be done within the operator's infrastructure once others
around managed to prove that something is already missing on their side.

> Should the WG 
> make efforts to contribute to preserving current practices of some network
> operators?

It will depend how long some operators will want to troubleshoot the issues
before claiming the protocol doesn't work :-/  Worse, since the network stack
will be implemented in browsers and servers, they will simply point the
fingers at these components, proving that just changing them (hence the
protocol as well) solves the issue. My personal take on this has been
expressed many times, which is that network must be observable if you want
to fix it, otherwise it degrades by itself till a point of poor quality
that "works enough" for most users (e.g. public WiFi access points). Things
as simple as detecting packet reordering caused by multiple paths is trivial
at the TCP level, it's way harder when you cannot say what the packet is
about.

> Frode

Willy

---

From: Frode Kileng <[email-masked]>
Date: Tue, 19 Nov 2019 06:57:07 +0100

On 19/11/2019 05:29, Willy Tarreau wrote:
> finger at the next step in the chain operated by someone else.

What additional value can "loss bit(s)" provide beyond confirming that 
the source is within your network domain or somewhere else, i.e. 
information that is available from other "loss sources". How would 
network operators use this loss bit(s) signal when providing such 
"excellent customer support" when the problem source is external?

frodek

---

From: "Lubashev, Igor" <[email-masked]>
Date: Tue, 19 Nov 2019 06:25:51 +0000

On Tue, Nov 19, 2019 at 12:57 AM, Frode Kileng <[email-masked]> wrote:
> information that is available from other "loss sources".
> How would network operators use this loss bit(s) signal when providing such

This is assuming there is a robust set of "other 'loss source'" available. This is far from reality for the vast majority of cases.

Also, the loss bits(s) would allow one to determine whether the loss is upstream or downstream.  That would allow you to know "whom do you call".  I.e. is the loss in your transit or peer? Is it in the user's own home WiFi network?  Or is it really in your network?

In general, it is important to keep in mind that we live in the world as it "is" and not in a world as it "should be".  I know little of ISP tech support business, but if ISPs says they have no other tools, I believe that they have no other tools currently. We can to encourage change, but if you push too hard, the reality has a way of pushing back.

> frodek

- Igor

---

From: Frode Kileng <[email-masked]>
Date: Tue, 19 Nov 2019 07:42:08 +0100

> This is assuming there is a robust set of "other 'loss source'" available. This is far from reality for the vast majority of cases.

So then there's two alternatives: a) Standardize a "loss-signal" option 
in QUIC and hope this is implemented and supported. b) Network operators 
lacking a "robust set of other loss sources" obtain such sources.

frodek

---

From: "Lubashev, Igor" <[email-masked]>
Date: Tue, 19 Nov 2019 07:02:57 +0000

> On Tue, Nov 19, 2019 at 2:42 PM, Frode Kileng <[email-masked]> wrote:
> 
> > This is assuming there is a robust set of "other 'loss source'" available. This is
> far from reality for the vast majority of cases.
> 
> So then there's two alternatives: a) Standardize a "loss-signal" option in QUIC
> and hope this is implemented and supported. b) Network operators lacking a
> "robust set of other loss sources" obtain such sources.

I do not think these are conflicting alternatives.  I am sure vendors will try to invent new tools, which is great.

As for loss-signal (as well as delay signal), these are definitely "experimental" features. There are many features that IETF standardized and never got widespread adoption. The benefit of QUIC is that it looks poised for a quick protocol evolution. If a feature gains widespread support, it would remain; if not, it would not make a future QUIC version and the bits occupied by loss- and delay- bits would be reused for something else.

- Igor

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 19 Nov 2019 08:32:20 +0100

On Tue, Nov 19, 2019 at 06:57:07AM +0100, Frode Kileng wrote:
> What additional value can "loss bit(s)" provide beyond confirming that the
> source is within your network domain or somewhere else, i.e. information
> that is available from other "loss sources". How would network operators use
> this loss bit(s) signal when providing such "excellent customer support"
> when the problem source is external?

I find your condescending tone inappropriate to discuss technical matters.

But since you asked, when I'm invited to troubleshooting sessions when
people experience issues ranging from "seems slow" to "sometimes hangs"
on TCP, I look at IP+ports, sequence numbers, ACK numbers, flags, window
sizes, timestamps and SACK fields, to figure what's happening. It just
happens that in QUIC all of them except IP+ports are encrypted, which
basically means my only solution will be to ask people "does it still
happen when you block UDP port 443?". And that's really sad. Network
issues are never white-or-black. Large buffers in VM environments cause
retransmits without losses, resulting in apparent loss on one side and
duplicate on the other one. Packets can get corrupted on their way from
one side to the other one, and guess what ? Some devices even manage to
reconstruct a valid checksum, but from inspection you cannot tell whether
the packet is OK or not. Some packets can be dropped past a certain size
(typical problem of PPPoE whose encapsulation takes 8 bytes), resulting
in only full packets not to be delivered. On the side experiencing the
miss, usually you immediately spot this thanks to the sequence number as
you notice a hole of 1460 bytes. Here you still see packets coming but
you don't know anything about them so you can just guess. Did you know
that you can even lose packets between two processes on the loopback
under sustained load ?

Debuggability is a key to any protocol's health. The vast majority of
the fixes that go into any network stack come from observations in the
field resulting in captures exhibiting a bad behavior. Here such captures
will never exist so there will be nothing to fix. This is still what I
really do not like at all in this way of designing a protocol.

Just my two cents,
Willy

---

From: Frode Kileng <[email-masked]>
Date: Tue, 19 Nov 2019 08:44:17 +0100

> I find your condescending tone inappropriate to discuss technical matters.

I'm very surprised as this was questions I asked trying to understand 
the benefits of the loss bit(s). As a non-native speaker, and after 
having looked  up the word "condescending", I understand that I should 
avoid further discussion on this topic in fear of being misinterpreted.

frodek

---

From: <[email-masked]>
Date: Thu, 21 Nov 2019 14:59:10 +0100

On Tue, 2019-11-19 at 07:42 +0100, Frode Kileng wrote:
> > This is assuming there is a robust set of "other 'loss source'"
> > available. This is far from reality for the vast majority of cases.
> 
> So then there's two alternatives: a) Standardize a "loss-signal"
> option in QUIC and hope this is implemented and supported. b) Network
> operators lacking a "robust set of other loss sources" obtain such
> sources.
> 
> frodek

To *lack* these sources, one might be incompetent, as you are
suggesting many people are. Alternatively, one might simply care about
a long end-to-end path with several responsibility boundaries, which by
definition are opaque. In this case, as Igor said, the
upstream/downstream location is absolutely crucial.

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Frode Kileng <[email-masked]>
Date: Fri, 22 Nov 2019 04:07:46 +0100

> upstream/downstream location is absolutely crucial.

When in https://github.com/quicwg/base-drafts/issues/3189 stating that 
"Shipping v1 without loss observation support would undermine day-to-day 
network maintenance work in at least that part of the ecosystem", naming 
a list of "supporters" and "The list is growing daily", I do think in 
all fairness it is important to indicate that this view is not 
necessarily an universal truth and a view is supported by everyone in 
"this part of the ecosystem". And that it should be possible to indicate 
this, giving a rationale and without being accused for claiming someone 
is "incompetent".

frodek

---

From: "Lubashev, Igor" <[email-masked]>
Date: Fri, 22 Nov 2019 04:02:37 +0000

Frode, Alex,

The unhelpful discussion of tone or word choice aside, I think that both of you make valid points.  I've heard similar views on both sides from multiple people.

I would repeat what I said yesterday in MOPS.  QUIC header encryption is a significant change in the way transport works, and we have no experience operating it at a scale where it is more than a single-digit percentage of Internet traffic.  Some people believe that they need new tools; others believe that they do not.  A lot of effort went into QUIC to ensure it will continue to evolve.  So the best thing we can do is to give the tool to people who claim they need it and see if it gets used.  If the tools remains largely unused, it will naturally disappear from QUIC (it will be a negotiated extension after all!). 

- Igor

> From: Frode Kileng <[email-masked]>
> 
> > a long end-to-end path with several responsibility boundaries, which
> > by definition are opaque. In this case, as Igor said, the
> network maintenance work in at least that part of the ecosystem", naming a
> list of "supporters" and "The list is growing daily", I do think in all fairness it is
> important to indicate that this view is not necessarily an universal truth and a
> view is supported by everyone in "this part of the ecosystem". And that it
> should be possible to indicate this, giving a rationale and without being
> accused for claiming someone is "incompetent".
> 
> frodek