Subject: Unit test proposal

From: Kenneth McMillan <[email-masked]>
Date: Wed, 20 Feb 2019 01:05:31 +0000

I have a proposal to make regarding unit testing of QUIC implementations. I would like to ask if there is any interest in the group and if anything along these lines has been considered or is in the works.

Currently I'm working on unit testing of QUIC implementations, but the effort is somewhat hampered by the fact that they all have different API's. That is, they present different interfaces to applications, and also use different system service API's. As a result, I can only test the implementations through the network (since this interface is standardized). This means that there are many behaviors that cannot be tested, since they can only stimulated through the implementation's client or system API's.

What I would like to propose is a common unit test interface for QUIC. This would be a value-based, language-independent API that provides QUIC services to applications and system services to QUIC. This interface would have several advantages:

1) Since it is generic, it would allow re-use of unit tests between implementations
2) Since it is value-based, it would allow logging and replay of runs
3) It could be made to support deterministic replay even for threaded implementations, by simulating cooperative multitasking.
4) It could also be used in interoperability testing to provide local replay.

In my experience, logging and replay of an interface is also enormously useful for bug reports, since there is no need even to build the user's application to diagnose the problem.

The investment required would be that each implementation would need a wrapper that presents the common unit test interface. My sense is that this effort would be moderate, since QUIC's interface to applications is relatively simple, and the implementations seem to require few system services (mainly file I/O, network I/O, locks, timers and random number generation). I think the advantages mentioned above would more than pay back this effort.

I'm not thinking of this as an official standard. It would just be a de facto standard that would be used optionally by developers who want to take advantage of a common base of unit tests.

Please let me know if you think this seems worthwhile, or if it's redundant with any existing work.

Thanks -- Ken

---

From: Dirkjan Ochtman <[email-masked]>
Date: Thu, 21 Feb 2019 09:21:46 +0100

> redundant with any existing work.

 It sounds useful to me. On the other hand, I'm not sure a
language-independent API is the best form here? It seems that languages are
sufficiently different that a single API might be hard to implement
sanely/simply across different languages. Maybe it would make more sense to
define a file format, similar to what's already been done for QPACK (I
believe it's called QIF)? It could probably be based on how Wireshark pcaps
are defined.

Regards,

Dirkjan

---

From: Ian Swett <[email-masked]>
Date: Wed, 27 Feb 2019 11:21:54 -0500

This would be interesting, but I'm a bit unclear on exactly what the unit
test interface would be.  And are you thinking of interfacing at the QUIC
transport layer, the HTTP layer, or both?

Agreed that logging and replay is extremely important in debugging.

The closest work I can think of is QUIC-Tracker, which I believe is
somewhat different than what you have in mind:
https://github.com/QUIC-Tracker/quic-tracker

---

From: Robin MARX <[email-masked]>
Date: Wed, 27 Feb 2019 17:45:13 +0100

Hello Ken,

We are working on trying to get a standardized logging format for QUIC set
up (see https://quic.edm.uhasselt.be/).
The idea would be that this allows implementations to log endpoint-specific
information (that would not be visible on the wire) in a machine
processable way.

One of the envisioned use cases would be to have a testing client execute
various scenarios (e.g., what QUIC-tracker is doing) and then validate the
expected behavior based on the server-side log output.
I have also talked about this to the people from RWTH Aachen who are doing
symbolic execution analysis (this paper https://dl.acm.org/authorize?N664187
).
He thought the logs might be used to obtain what I think is called "belief
states" (I'm not at all into symbolic execution) and use that in their
analysis as well.

This is certainly far from a full-fledged unit testing setup and doesn't
offer all the features you mentioned (though I think replay would be
interesting/possible to add if we optionally log full packet dumps as well).
However, I do see some synergies. If you agree, feel free to contact us.

*switches hats*
I am also working on a JavaScript based QUIC implementation (yes, I know)
that has only limited testing atm.
>From that viewpoint, generic unit testing would certainly be interesting,
though I have a hard time imagining how that might look/work in practice.

With best regards,
Robin

-- 

Robin Marx
PhD researcher - web performance
Expertise centre for Digital Media

T +32(0)11 26 84 79 - GSM +32(0)497 72 86 94

www.uhasselt.be
Universiteit Hasselt - Campus Diepenbeek
Agoralaan Gebouw D - B-3590 Diepenbeek
Kantoor EDM-2.05

---

From: Kenneth McMillan <[email-masked]>
Date: Wed, 27 Feb 2019 19:10:27 +0000

Hi Robin,

There's definitely some synergy. A real-time log in a standard format would be very helpful for on-line test generation, since the tester could make decisions based on internal server events that aren't visible on the wire. Of course, what I want to do is actually *drive* the interfaces of the server or client.

I do have a generic tester for QUIC servers, based on a formal specification. If you're interested in using it, perhaps for generating logs for your visualization tools, let me know.

Best--Ken

Sent from Outlook<http://aka.ms/weboutlook>

________________________________
From: Robin MARX <[email-masked]>
Sent: Wednesday, February 27, 2019 8:45:13 AM
To: Ian Swett
Cc: Dirkjan Ochtman; QUIC WG; Kenneth McMillan
Subject: Re: Unit test proposal

Hello Ken,

We are working on trying to get a standardized logging format for QUIC set up (see https://quic.edm.uhasselt.be/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fquic.edm.uhasselt.be%2F&data=02%7C01%7Ckenmcmil%40exchange.microsoft.com%7C778aa70ebcb74b4e53b308d69cd30640%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C636868827495740951&sdata=vrzqdAchhF%2Ff874Q82vvLfFFq9uEssJIR9YftF4860g%3D&reserved=0>).
The idea would be that this allows implementations to log endpoint-specific information (that would not be visible on the wire) in a machine processable way.

One of the envisioned use cases would be to have a testing client execute various scenarios (e.g., what QUIC-tracker is doing) and then validate the expected behavior based on the server-side log output.
I have also talked about this to the people from RWTH Aachen who are doing symbolic execution analysis (this paper https://dl.acm.org/authorize?N664187<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdl.acm.org%2Fauthorize%3FN664187&data=02%7C01%7Ckenmcmil%40exchange.microsoft.com%7C778aa70ebcb74b4e53b308d69cd30640%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C636868827495750964&sdata=3powwCDGQhdaat9H1vvl1WMFK%2FCeqfD0FrSXoNKHk5I%3D&reserved=0>).
He thought the logs might be used to obtain what I think is called "belief states" (I'm not at all into symbolic execution) and use that in their analysis as well.

This is certainly far from a full-fledged unit testing setup and doesn't offer all the features you mentioned (though I think replay would be interesting/possible to add if we optionally log full packet dumps as well).
However, I do see some synergies. If you agree, feel free to contact us.

*switches hats*
I am also working on a JavaScript based QUIC implementation (yes, I know) that has only limited testing atm.
>From that viewpoint, generic unit testing would certainly be interesting, though I have a hard time imagining how that might look/work in practice.

With best regards,
Robin

On Wed, 27 Feb 2019 at 17:22, Ian Swett <ianswett=[email-masked]<mailto:[email-masked]>> wrote:
This would be interesting, but I'm a bit unclear on exactly what the unit test interface would be.  And are you thinking of interfacing at the QUIC transport layer, the HTTP layer, or both?

The closest work I can think of is QUIC-Tracker, which I believe is somewhat different than what you have in mind: https://github.com/QUIC-Tracker/quic-tracker<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FQUIC-Tracker%2Fquic-tracker&data=02%7C01%7Ckenmcmil%40exchange.microsoft.com%7C778aa70ebcb74b4e53b308d69cd30640%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C636868827495750964&sdata=Lr%2Fl0NUbpaDimBjwkMAvlLEykZibapSp5A8lbFgkPt4%3D&reserved=0>

On Thu, Feb 21, 2019 at 3:22 AM Dirkjan Ochtman <[email-masked]<mailto:[email-masked]>> wrote:
On Wed, Feb 20, 2019 at 2:05 AM Kenneth McMillan <kenmcmil=[email-masked]<mailto:[email-masked]>> wrote:
Please let me know if you think this seems worthwhile, or if it's redundant with any existing work.

 It sounds useful to me. On the other hand, I'm not sure a language-independent API is the best form here? It seems that languages are sufficiently different that a single API might be hard to implement sanely/simply across different languages. Maybe it would make more sense to define a file format, similar to what's already been done for QPACK (I believe it's called QIF)? It could probably be based on how Wireshark pcaps are defined.

Regards,

Dirkjan

--

Robin Marx

www.uhasselt.be<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.uhasselt.be&data=02%7C01%7Ckenmcmil%40exchange.microsoft.com%7C778aa70ebcb74b4e53b308d69cd30640%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C636868827495760977&sdata=VwdaBfS1JY6qUaxSv1wCuDuH82eJKt0uvHul4KzQ18Q%3D&reserved=0>
Universiteit Hasselt - Campus Diepenbeek
Agoralaan Gebouw D - B-3590 Diepenbeek
Kantoor EDM-2.05

[https://www.uhasselt.be/images/nieuwsbrief/dcm/UHasselt-liggend.jpg]

---

From: Kenneth McMillan <[email-masked]>
Date: Wed, 27 Feb 2019 19:57:44 +0000

Hi Ian,

I'm thinking of doing just QUIC for now and not HTTP/3.  So the upper interface would have operations like "listen", "connect", "open stream", "send/receive" and so on. The lower interface would have operations like "datagram send/receive", "sleep", "random bytes" and so on. I'm also thinking of an interface for cooperative multitasking. This would have two calls: "block" would be called by a thread when it is about to do an operation that might block, like taking a lock, and "unblock" would be called when the operation completes. This would give the test harness the ability to schedule threads, which would allow deterministic replay, as well as exploring thread schedules.

The interface would be defined in terms of simple C-style calls and callbacks, with basic serializable data structures and opaque pointers or ID's. This makes logging and replay possible. One option is to use an IDL (for example, gRPC) to make the interface language-independent and also generate a lot of boilerplate code automatically.

Dirkjan makes a good point that it may be difficult to capture all the functionality  of the various

implementations in different languages. On the other hand, a lot of the complexity of interfaces is for

efficiency purposes, and we are not concerned with efficiency here. We can make the interface very simple

and  the adapters as inefficient as we like, since this code is only for functional testing purposes.

Bets -- Ken

________________________________
From: Ian Swett <ianswett=[email-masked]>
Sent: Wednesday, February 27, 2019 8:21 AM
To: Dirkjan Ochtman
Cc: Kenneth McMillan; QUIC WG
Subject: Re: Unit test proposal

The closest work I can think of is QUIC-Tracker, which I believe is somewhat different than what you have in mind: https://github.com/QUIC-Tracker/quic-tracker<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FQUIC-Tracker%2Fquic-tracker&data=02%7C01%7Ckenmcmil%40exchange.microsoft.com%7Ce441287ef6474c7bb7eb08d69ccfbad1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636868813340085808&sdata=RWtaXX39RKM0YnGrLFSGHjKLTp6%2FeVUiOVRp0fNsjQ4%3D&reserved=0>

Regards,

Dirkjan