Subject: Re: [tsvwg] Fwd: New Version Notification for draft-heist-tsvwg-ecn-deployment-observations-00.txt

From: Dave Taht <[email-masked]>
Date: Mon, 1 Mar 2021 07:54:24 -0800

I had pinpointed a whole bunch of mismarked ecn traffic over
bittorrent to an ISP in argentina last year. In fact, all their
traffic was mismarked. I can find the relevant IPs...

On Fri, Feb 26, 2021 at 3:58 PM Jonathan Morton <[email-masked]> wrote:
>
> > On 27 Feb, 2021, at 1:31 am, Martin Duke <[email-masked]> wrote:
> >
> > I wonder if the non-TCP ECN traffic is QUIC? I don't think the main implementations are doing it, but there are a few ECN-capable implementations in production.
>
> This would be a reasonable interpretation of traffic on ports typically used with HTTP or similar L5 protocols.  We have, similarly, interpreted non-TCP traffic using ports commonly associated with tunnel protocols as actually being (at least partly) TCP through a tunnel.
>
> However, most of what we saw was on random high ports, and some that are most often used by BitTorrent clients.  So we think it's probably due to BT clients trying to do "clever" things with TOS/DSCP but misunderstanding it.  I believe the default in libtorrent is to set CS1, correctly, so I don't know which particular clients might be so affected, or whether it is due to end users setting manual firewall mangle rules incorrectly.
>
> Since the data collection tool we used doesn't have a way to unambiguously discriminate between QUIC and any other UDP-supported protocol, it's difficult for us to say more about how much QUIC-ECN was seen.  If you have a criterion we could use for that, we might be able to build it into a future run.
>
>  - Jonathan Morton

-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

[email-masked] <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

---

From: Pete Heist <[email-masked]>
Date: Tue, 02 Mar 2021 09:33:07 +0100

> traffic was mismarked. I can find the relevant IPs...

Just checking- meaning all their BT traffic was mismarked?

I only looked at one non-libtorrent client, Transmission, which seemed
to not set any bits in the DS field.

> > > I wonder if the non-TCP ECN traffic is QUIC? I don't think the
> > > main implementations are doing it, but there are a few ECN-
> > > capable implementations in production.
> > 
> > This would be a reasonable interpretation of traffic on ports
> > typically used with HTTP or similar L5 protocols.  We have,
> > similarly, interpreted non-TCP traffic using ports commonly
> > associated with tunnel protocols as actually being (at least
> > partly) TCP through a tunnel.
> > 
> > However, most of what we saw was on random high ports, and some
> > that are most often used by BitTorrent clients.  So we think it's
> > probably due to BT clients trying to do "clever" things with
> > TOS/DSCP but misunderstanding it.  I believe the default in
> > libtorrent is to set CS1, correctly, so I don't know which
> > particular clients might be so affected, or whether it is due to
> > end users setting manual firewall mangle rules incorrectly.
> > 
> > Since the data collection tool we used doesn't have a way to
> > unambiguously discriminate between QUIC and any other UDP-supported
> > protocol, it's difficult for us to say more about how much QUIC-ECN
> > was seen.  If you have a criterion we could use for that, we might
> > be able to build it into a future run.
> > 
> >  - Jonathan Morton
> 
> 
>

---

From: Pete Heist <[email-masked]>
Date: Sun, 07 Mar 2021 10:01:38 +0100

Hi Martin, circling back to this, we noticed one line in our non-TCP
data among the packet counts by user IPs to dst ports:

                       ECT(0)  CE      ECT(1)  ECT(0) CE   ECT(1)
                       from    from    from    from   from from
    IP/Port            WAN     WAN     WAN     LAN    LAN  LAN
    -------            ---     ---     ---     ---    ---  ---
    udp:443 (https)    4603    0       0       1882   0    0

As Jon mentioned we can't say for sure, but it's not outside of the
realm of possibility that this is QUIC-ECN, so I would add this to our
draft and mention it at MAPRG.

Pete

On Fri, 2021-02-26 at 15:31 -0800, Martin Duke wrote:
> Thanks Pete!
> 
> I wonder if the non-TCP ECN traffic is QUIC? I don't think the main
> implementations are doing it, but there are a few ECN-capable
> implementations in production.
> 
> On Sun, Feb 21, 2021 at 4:34 PM Bob Briscoe <[email-masked]>
> wrote:
> > Pete,
> > 
> > Thx for doing this. I'm afraid I won't be able to now analyse the
> > data 
> > you've provided for a couple of days, 'cos I have to focus on a
> > couple 
> > of deadlines tomorrow, one being the IETF draft deadline.
> > 
> > But whatever, thx v much.
> > 
> > Bob
> > 
> > On 20/02/2021 13:39, Pete Heist wrote:
> > > New revision posted and some comments below...
> > > 
> > > On Sat, 2021-02-20 at 09:36 +0000, Bob Briscoe wrote:
> > > > Pete,
> > > > 
> > > > If you plan to post a revision, in the TCP table (Sec.5.3) would
> > you
> > > > pls
> > > > consider including the counts of ECT packets, as in the non-TCP
> > table
> > > > in
> > > > Sec.5.3? Otherwise we don't know what percentage of the total ECN
> > > > traffic each count represent.
> > > That's a useful addition. An ECT(0) column is now added, and the
> > TCP
> > > table's column order is now similar to the non-TCP table. ECE for
> > the
> > > opposite direction is still next to CE as that seems most readable.
> > I
> > > left out ECT(1) for TCP as there's so little of that, and then it
> > gets
> > > harder with the line length limit.
> > > 
> > > > I'm trying to separate out the IPs known to be behind the FQ-
> > > > CoDel
> > > > nodes
> > > > in the backhaul. I.e. separating known from unknown causes.
> > > That's mentioned in Section 3.2.
> > > 
> > > > Separately, a count of total packets, or equivalently a count of
> > > > not-ECT, would be v useful too.
> > > I didn't capture Not-ECT by IP address, mainly because some testing
> > > made me question how at least one ipset lookup per-packet would
> > affect
> > > performance on their production gateway. I added that to the
> > > Limitations section as it could be useful if available.
> > > 
> > > Pete
> > > 
> > > > If you can do this without losing the ECE counts, even better.
> > Having
> > > > CE
> > > > and ECE together enables a rough estimate of the average
> > window.{Note
> > > > 1}
> > > > 
> > > > Specifically,
> > > >        avgWindow = ECE * delAckRatio / CE
> > > > 
> > > > Thank you (again)
> > > > 
> > > > 
> > > > Bob
> > > > 
> > > > {Note 1} And lots of other assumptions, such as:
> > > > - delayed Ack Ratio including stretch ACKs has to be guessed
> > > > (e.g.
> > > > 2???)
> > > > - minimal additional ack coalescing in the network
> > > > - minimal reordering
> > > > - traffic volume dominated by elephants
> > > > - the TCP protocol is working as it should
> > > > - and so on.
> > > > 
> > > > 
> > > > On 19/02/2021 13:06, Pete Heist wrote:
> > > > > Yes, that was somewhat unexpected to us at first too, but
> > > > > misuse
> > of
> > > > > the
> > > > > ECN field esp. on non-TCP packets seems to explain it in cases
> > > > > where
> > > > > there ratios don't look like AQM signaling. The data for TCP is
> > > > > overall
> > > > > easier to attribute to ECN when you can see feedback via ECE.
> > > > > 
> > > > > I didn't editorialize much on the results, but I'll use this as
> > > > > a
> > > > > chance to add what struck me:
> > > > > 
> > > > > * While ECN negotiations were relatively small at around 1.44%,
> > > > > they
> > > > > were spread across 45% of the IPs, so the proportion of paths
> > using
> > > > > it
> > > > > seems significant.
> > > > > 
> > > > > * The data seem to show significant 3168 marking AQM
> > > > > deployment,
> > > > > when
> > > > > 24% of LAN IPs that negotiated ECN saw CE or ECE. The draft
> > > > > mentions
> > > > > how some of it is from known AQM instances and some not.
> > Congestion
> > > > > overall doesn't seem excessive, but that could be quantified
> > > > > better.
> > > > > 
> > > > > * A wider survey might help on the non-TCP data.
> > > > > 
> > > > > Anyway thanks for taking a look...
> > > > > 
> > > > > Pete
> > > > > 
> > > > > On Fri, 2021-02-19 at 11:44 +0000, Mirja Kuehlewind wrote:
> > > > > > Hi Pete,
> > > > > > 
> > > > > > thanks for putting this together and sharing!
> > > > > > 
> > > > > > I have one question on the data. Maybe I'm not readying this
> > > > > > correctly but if I look at the big table at the end, then I
> > > > > > see
> > a
> > > > > > lot
> > > > > > of cases where there are much more CE marks than ECT(0)
> > > > > > marks.
> > > > > > That's
> > > > > > a bit unexpected as usually an AQM should only mark a small
> > > > > > portion
> > > > > > of the packets. Or do I interpret the data there incorrectly?
> > > > > > 
> > > > > > Mirja
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > On 18.02.21, 17:38, "tsvwg on behalf of Pete Heist"
> > > > > > <[email-masked] on behalf of [email-masked]> wrote:
> > > > > > 
> > > > > > 
> > > > > >        > A new version of I-D, draft-heist-tsvwg-ecn-
> > > > > > deployment-
> > > > > > observations-00.txt
> > > > > >        > has been successfully submitted by Peter G. Heist
> > > > > > and
> > > > > > posted to
> > > > > > the
> > > > > >        > IETF repository.
> > > > > >        >
> > > > > >        > Name:             draft-heist-tsvwg-ecn-deployment-
> > > > > > observations
> > > > > >        > Revision: 00
> > > > > >        > Title:            Explicit Congestion Notification
> > > > > > (ECN)
> > > > > > Deployment Observations
> > > > > >        > Document date:    2021-02-18
> > > > > >        > Group:            Individual Submission
> > > > > >        > Pages:            27
> > > > > >        > URL:
> > > > > >      
> > > > > > 
> > https://www.ietf.org/archive/id/draft-heist-tsvwg-ecn-deployment-observations-00.txt
> > > > > >        > Status:
> > > > > >      
> > > > > > 
> > https://datatracker.ietf.org/doc/draft-heist-tsvwg-ecn-deployment-observations/
> > > > > >        > Html:
> > > > > >      
> > > > > > 
> > https://www.ietf.org/archive/id/draft-heist-tsvwg-ecn-deployment-observations-00.html
> > > > > >        > Htmlized:
> > > > > >      
> > > > > > 
> > https://tools.ietf.org/html/draft-heist-tsvwg-ecn-deployment-observations-00
> > > > > >        >
> > > > > >        > Abstract:
> > > > > >        >   This note presents data gathered at an Internet
> > Service
> > > > > > Provider's
> > > > > >        >   gateway on the observed deployment and usage of
> > > > > > ECN.
> > > > > > Relevant IP
> > > > > >        >   counter and flow tracking data was collected and
> > > > > > analyzed for
> > > > > > TCP and
> > > > > >        >   other protocols.
> > > > > > 
> > > > > >        This draft adds some data on the current usage of ECN.
> > > > > > It
> > > > > > was
> > > > > > gathered over several weeks at a cooperative ISP with around
> > > > > > 660
> > > > > > members, and looks at ECN endpoint activity, AQM deployment
> > > > > > and
> > > > > > ECN
> > > > > > usage on non-TCP protocols. While this study is still
> > > > > > relatively
> > > > > > small, it’s hopefully at least a little more useful than the
> > > > > > stateless counter data I posted late last year, which should
> > > > > > set
> > > > > > the
> > > > > > bar suitably low… :)
> > > > > > 
> > > > > >        Pete
> > > > > > 
> > > > > > 
> > > 
> >