Subject: QUIC trace tooling

From: Victor Vasiliev <[email-masked]>
Date: Thu, 20 Sep 2018 12:00:32 -0400

QUIC can't use the regular TCP debugging tools like tcptrace since
~everything is encrypted, however, there's still a need for tooling to
debug issues related to congestion control.  As a solution to that problem,
over the past few years Google has created tooling to record the relevant
state on the sender, together with additional metadata that allows to
understand the connection state in more depth that it would be possible
from the wire (e.g. it allows recording the CWND, which is never sent
explicitly).

We've open-sourced our tooling, and it's available at
https://github.com/google/quic-trace.  It includes:

   - A schema
   <https://github.com/google/quic-trace/blob/master/lib/quic_trace.proto>
   for recording traces (in protobuf)
   - An OpenGL-based tool that renders the trace and allows to inspect the
   state for each packet.
   - An example trace
   <https://github.com/google/quic-trace/blob/master/examples/example.qtr>
   (generated from gQUIC unit tests).

I know some people in the working group have expressed interest in this
before, so I'm eager to hear any feedback, both on the list and in GitHub
issues.

Cheers,
  Victor.

---

From: "Salz, Rich" <[email-masked]>
Date: Thu, 20 Sep 2018 16:28:31 +0000

How about developing a wireshark plugin?

---

From: Robin MARX <[email-masked]>
Date: Thu, 20 Sep 2018 19:19:32 +0200

Very nice to see this.
We have been working on something similar and hope to present our ideas at
the EPIQ workshop in December.
Intermediate results can be found at quic.edm.uhasselt.be (less "production
ready" than quic-trace). We try to go further than tcptrace-alike plots
with for example a timeline.

Will take a deeper look into quic-trace after my vacation, thanks for
sharing!

-- 

Robin Marx
PhD researcher - web performance
Expertise centre for Digital Media

T +32(0)11 26 84 79 - GSM +32(0)497 72 86 94

www.uhasselt.be
Universiteit Hasselt - Campus Diepenbeek
Agoralaan Gebouw D - B-3590 Diepenbeek
Kantoor EDM-2.05

---

From: <[email-masked]>
Date: Thu, 20 Sep 2018 22:19:06 +0200

Thanks. But please note that the availability of such advanced endpoint 
instrumentation doesn't weaken the need for on-path measurements: we still need 
them to debug long paths by dichotomy.

And also note that the spin bit not only gives RTT contributions of either side 
of the capture point, but also the full BIF (which mostly follows the CWIN), 
allowing to analyse congestion control as well.

On 09/20/18 18:00, Victor Vasiliev wrote:
> QUIC can't use the regular TCP debugging tools like tcptrace since ~everything 
> is encrypted, however, there's still a need for tooling to debug issues 
> related to congestion control.  As a solution to that problem, over the past 
> few years Google has created tooling to record the relevant state on the 
> sender, together with additional metadata that allows to understand the 
> connection state in more depth that it would be possible from the wire (e.g. 
> it allows recording the CWND, which is never sent explicitly).
>
>   * A schema
>   * An OpenGL-based tool that renders the trace and allows to inspect the
>     state for each packet.
>   * An example trace
> before, so I'm eager to hear any feedback, both on the list and in GitHub issues.
>
> Cheers,
>   Victor.

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 21 Sep 2018 09:56:16 +0000

Hey Victor,

Thanks for sharing this work. I think there will be a lot of value in visualisation of such data, particularly for newer aspects of the transport like multiplexing. Is the scope of this work purely for the transport layer, or do you think there is room for application layer focused views?

One comment made during the discussion at the New York interim was around data capture and file format with protobuf. I think there is a lot of opportunity for the community to build visualisation tools for different needs if we can come together to define common formats and transformations.

I'm only vaguely familiar with protobuf, are there resources that describe how this is intended to work (either in general, or specifically for this project)? Coming from a more application-layer oriented background, I was sort of expecting something in a similar format to HAR (HTTP archive). I presume protobuf is a more efficient serialization, but how does it compare in terms of ease of proces, conversion and interchange?

Kind regards
Lucas
________________________________
From: QUIC [[email-masked]] on behalf of Victor Vasiliev [vasilvv=[email-masked]]
Sent: 20 September 2018 17:00
To: IETF QUIC WG
Subject: QUIC trace tooling

QUIC can't use the regular TCP debugging tools like tcptrace since ~everything is encrypted, however, there's still a need for tooling to debug issues related to congestion control.  As a solution to that problem, over the past few years Google has created tooling to record the relevant state on the sender, together with additional metadata that allows to understand the connection state in more depth that it would be possible from the wire (e.g. it allows recording the CWND, which is never sent explicitly).

We've open-sourced our tooling, and it's available at https://github.com/google/quic-trace.  It includes:

  *   A schema<https://github.com/google/quic-trace/blob/master/lib/quic_trace.proto> for recording traces (in protobuf)
  *   An OpenGL-based tool that renders the trace and allows to inspect the state for each packet.
  *   An example trace<https://github.com/google/quic-trace/blob/master/examples/example.qtr> (generated from gQUIC unit tests).

I know some people in the working group have expressed interest in this before, so I'm eager to hear any feedback, both on the list and in GitHub issues.

Cheers,
  Victor.

----------------------------

http://www.bbc.co.uk
This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.

---------------------

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 21 Sep 2018 03:55:37 -0700

Protocol buffers are best when small message size is a priority or when you
cannot trust the message source (buffer overrun).

If you need to read data fast from disk and trust the data, FlatBuffers is
a better format because it allows for zero copy and memory mapping.
FlatBuffers can also validate untrusted data but works best when trusted.

If you have massive volumes of similar data, Arrow might be better format,
but it is still work in progress and not as widely supported.

Disclaimer, I work on FlatBuffers for C.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 21 September 2018 at 11.56.42, Lucas Pardue ([email-masked])
wrote:

Hey Victor,

Thanks for sharing this work. I think there will be a lot of value in
visualisation of such data, particularly for newer aspects of the transport
like multiplexing. Is the scope of this work purely for the transport
layer, or do you think there is room for application layer focused views?

One comment made during the discussion at the New York interim was around
data capture and file format with protobuf. I think there is a lot of
opportunity for the community to build visualisation tools for different
needs if we can come together to define common formats and transformations.

I'm only vaguely familiar with protobuf, are there resources that describe
how this is intended to work (either in general, or specifically for this
project)? Coming from a more application-layer oriented background, I was
sort of expecting something in a similar format to HAR (HTTP archive). I
presume protobuf is a more efficient serialization, but how does it compare
in terms of ease of proces, conversion and interchange?

Kind regards
Lucas
------------------------------
*From:* QUIC [[email-masked]] on behalf of Victor Vasiliev [vasilvv=
[email-masked]]
*Sent:* 20 September 2018 17:00
*To:* IETF QUIC WG
*Subject:* QUIC trace tooling

   - A schema

Cheers,
  Victor.

http://www.bbc.co.uk
This e-mail (and any attachments) is confidential and may contain personal
views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance
on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.

---------------------

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 21 Sep 2018 11:26:43 +0000

Hi Mikkel,

Thanks for the overview. This reminds me of a point I forgot to make. In one of the links explaining qtrace, it highlights performant visualisation even up to a sample size of 80k packets. To contrast to my HAR example, that is generally used for a single page load. It will be interesting to see how long the sample periods are that people want to visualise. I wonder if there is merit in subsampling for bulk processing e.g. a HTTP/QUIC server or reverse proxy that can record QUIC state in each completed transaction in an access logs (like TCP info can be used today).

Regards
Lucas
________________________________________
From: Mikkel Fahnøe Jørgensen [[email-masked]]
Sent: 21 September 2018 11:55
To: Lucas Pardue; IETF QUIC WG; Victor Vasiliev
Subject: RE: QUIC trace tooling

Protocol buffers are best when small message size is a priority or when you cannot trust the message source (buffer overrun).

If you need to read data fast from disk and trust the data, FlatBuffers is a better format because it allows for zero copy and memory mapping. FlatBuffers can also validate untrusted data but works best when trusted.

If you have massive volumes of similar data, Arrow might be better format, but it is still work in progress and not as widely supported.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 21 September 2018 at 11.56.42, Lucas Pardue ([email-masked]<mailto:[email-masked]>) wrote:

Hey Victor,

Kind regards
Lucas
________________________________
From: QUIC [[email-masked]<mailto:[email-masked]>] on behalf of Victor Vasiliev [vasilvv=[email-masked]<mailto:[email-masked]>]
Sent: 20 September 2018 17:00
To: IETF QUIC WG
Subject: QUIC trace tooling

Cheers,
  Victor.

http://www.bbc.co.uk

---------------------

---

From: Victor Vasiliev <[email-masked]>
Date: Fri, 21 Sep 2018 17:00:22 -0400

Hi Lucas,

Thank you for your feedback!

> layer, or do you think there is room for application layer focused views?

I have considered adding some application-level metadata, in particular,
annotating what kind of data is on which stream.  I am not entirely sure
what other things there would be to add that are interesting to inspect on
packet-level.

> needs if we can come together to define common formats and transformations.

Well.  My focus was on making something that both worked for Google's use
case but was easily adoptable by third-parties, since there was a gap in
tooling that could potentially slow down QUIC deployment.  If people want
something interoperable using a more widely-used serialization format, I
could add a JSON-based format.

> in terms of ease of proces, conversion and interchange?

It's a combination of a schema language
<https://developers.google.com/protocol-buffers/docs/proto>, a binary
serialization
format <https://developers.google.com/protocol-buffers/docs/encoding>, and
schema compilers for a bunch of languages.  It's more space-efficient, but
I mostly chose it because it was easier to deal with from C++.  It also has
a built-in JSON serialization, so, for instance, the example trace in the
GitHub repo would start out like this:

{
 "protocolVersion": "UTA0NA==",
 "destinationConnectionId": "AAAAAAAAACo=",
 "events": [
  {
   "timeUs": "0",
   "eventType": "PACKET_SENT",
   "packetNumber": "1",
   "frames": [
    {
     "frameType": "STREAM",
     "streamFrameInfo": {
      "streamId": "3",
      "fin": false,
      "length": "1326",
      "offset": "0"
     }
    }
   ],
   "packetSize": "1350",
   "encryptionLevel": "ENCRYPTION_1RTT"
  },

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sat, 22 Sep 2018 05:26:37 +0000

And FlatBuffers can read and write the same JSON format as long as unions are avoided.

________________________________
From: QUIC <[email-masked]> on behalf of Victor Vasiliev <vasilvv=[email-masked]>
Sent: Friday, September 21, 2018 11:00:22 PM
To: [email-masked]
Cc: IETF QUIC WG
Subject: Re: QUIC trace tooling

Hi Lucas,

On Fri, Sep 21, 2018 at 5:56 AM Lucas Pardue <[email-masked]<mailto:[email-masked]>> wrote:

Hey Victor,

I have considered adding some application-level metadata, in particular, annotating what kind of data is on which stream.  I am not entirely sure what other things there would be to add that are interesting to inspect on packet-level.

Well.  My focus was on making something that both worked for Google's use case but was easily adoptable by third-parties, since there was a gap in tooling that could potentially slow down QUIC deployment.  If people want something interoperable using a more widely-used serialization format, I could add a JSON-based format.

It's a combination of a schema language<https://developers.google.com/protocol-buffers/docs/proto>, a binary serialization format<https://developers.google.com/protocol-buffers/docs/encoding>, and schema compilers for a bunch of languages.  It's more space-efficient, but I mostly chose it because it was easier to deal with from C++.  It also has a built-in JSON serialization, so, for instance, the example trace in the GitHub repo would start out like this:

{

---

From: Roberto Peon <[email-masked]>
Date: Mon, 24 Sep 2018 17:42:08 +0000

Protobufs are really great for interop, and *much* less-so for efficiency (which is why we were hearing about Flatbufs).
I’m pretty excited about having these tools!

-=R

From: QUIC <[email-masked]> on behalf of Victor Vasiliev <vasilvv=[email-masked]>
Date: Friday, September 21, 2018 at 2:01 PM
To: "[email-masked]" <[email-masked]>
Cc: IETF QUIC WG <[email-masked]>
Subject: Re: QUIC trace tooling

Hi Lucas,

It's a combination of a schema language<https://urldefense.proofpoint.com/v2/url?u=https-3A__developers.google.com_protocol-2Dbuffers_docs_proto&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=djPsS1HxmpTMtcSXNADJ6VMWxuoJG1N3f86n7rTge6o&s=lABNgqFRgS7iEARUosnrH7Hgnw11-RcAeVJMt9e25To&e=>, a binary serialization format<https://urldefense.proofpoint.com/v2/url?u=https-3A__developers.google.com_protocol-2Dbuffers_docs_encoding&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=djPsS1HxmpTMtcSXNADJ6VMWxuoJG1N3f86n7rTge6o&s=A3SSaAbvWE_o27QDnyrmCTzWdKFEHafiOtKpfcVyDUs&e=>, and schema compilers for a bunch of languages.  It's more space-efficient, but I mostly chose it because it was easier to deal with from C++.  It also has a built-in JSON serialization, so, for instance, the example trace in the GitHub repo would start out like this:

{