Subject: [OPSAWG] draft-gharris-opsawg-pcap.txt --- FCS length description

From: Michael Richardson <[email-masked]>
Date: Mon, 21 Dec 2020 19:31:42 -0500

Hi, I have reworked the document that Guy put into XML describing the *PCAP*
(not NG) format.   I found the text for LinkType to be confusing, and
frankly, I think wrong.

   *  LinkType (32 bits): an unsigned value that defines, in the lower
      16 bits, the link layer type of packets in the file, and
      optionally indicates the length of the Frame Check Sequence (FCS)
      of packets in the upper 16 bits.  The list of Standardized Link
      Layer Type codes is available in [LINKTYPES].  If bit 5 is set,
      bits 0 through 3 contain the length of the FCS field at the end of
      all packets; if bit 5 is not set, the length of the FCS field at
      the end of all packets is unknown.  Bit 4, and bits 6 through 15,
      SHOULD be filled with 0 by pcap file writers, and MUST be ignored
      by pcap file readers.

The numbering is driven by how the IETF likes to number bits, which has
always been a bit bizarre.

You can see the document at:
    https://datatracker.ietf.org/doc/draft-gharris-opsawg-pcap/
and from the links at:
    https://github.com/pcapng/pcapng
    which are fixed and updated thanks to circleci actions, I hope.

Looking at libpcap's pcap/pcap.h:
   https://github.com/the-tcpdump-group/libpcap/blob/master/pcap/pcap.h#L217

we see:
/*
 * Macros for the value returned by pcap_datalink_ext().
 *
 * If LT_FCS_LENGTH_PRESENT(x) is true, the LT_FCS_LENGTH(x) macro
 * gives the FCS length of packets in the capture.
 */
#define LT_FCS_LENGTH_PRESENT(x)	((x) & 0x04000000)
#define LT_FCS_LENGTH(x)		(((x) & 0xF0000000) >> 28)
#define LT_FCS_DATALINK_EXT(x)		((((x) & 0xF) << 28) | 0x04000000)

this suggests that the FCS length is really only 3 bits (maximum FCS size of
7 bytes?  Or does 0 indicate 8 bytes?  Ethernet is 4...).

I see, however:
   pcap-dag.c:
        p->linktype_ext = LT_FCS_DATALINK_EXT(pd->dag_fcs_bits/16);

I can find no other references.  So I guess Ethernet gets a value of 2 (*16 bits).
I can't find any other uses.
pcap_datalink_ext() is in pcap.c, but no the man page.

The code does not ignore bits 28:16 of the linktype field (the bits numbered
6:15 in the diagram).
Since we are nowhere close to 64K link types, from looking at the pcap
document only, we could make it 28-bits:
         BUT: pcapng has a 16-bit LinkType only, so we really need to limit this to
         16-bits.... OOPS.  I'll fix this in -01.

What I'm looking for in this email is:
1) confirmation that Linktype is 16-bits.
2) some explanation of valid FCS values. Seems to be a multiple of 16-bits.
   Is 0 valid?  Or would that be indicated by LENGTH_PRESENT(x)==0?
   Or is 0 ==> 8 * 16-bits => 128 bits of FCS.

I'm going to propose IANA considerations in a followup email and in -01.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: Michael Richardson <[email-masked]>
Date: Mon, 21 Dec 2020 20:39:27 -0500

{resend due to problems with PGP signatures vs mailman}

we see:
/*

I see, however:
   pcap-dag.c:

---

From: Michael Richardson <[email-masked]>
Date: Mon, 21 Dec 2020 20:51:10 -0500

The short of it is:

1) reserve bits 16:28 of linktype as zero.
2) lower 32K Specification Required (any document),
   upper 32K First Come First Served

Details:
  The Registry has three sections according to {{RFC8126}}:
  * values from 0 to 32767 are marked as Specification Required.
  *   except that values 147 to 162 are reserved for Private Use
  * values from 32768 to 65000 are marked as First-Come First-Served.
  * values from 65000 to 65536 are marked as Private Use.

3) I have included the tcpdump.org linktype database that we maintain.
   It might be a week old, but I don't think we've allocated anything
   in the past month.
   It is at: http://www.tcpdump.org/linktypes.html
   Some entries have further links under /linktypes/... which we could
   enter as the correct reference.
   There are 112 entries.
   It may be that we should do something different for initializing IANA.

I did some editing of the description field to shorten in a lot, but I got
tired about 30% through the list, not sure if we should even include that
column.
There are many entries like:
   LINKTYPE_PPP_ETHER                  |   51   |PPPoE; per RFC 2516

where I think that maybe this should point directly at RFC2516.
I'm not entirely sure about this.

Where this gets more complex is:
   LINKTYPE_PPP_WITH_DIR	204	DLT_PPP_WITH_DIR	PPP, as per
   RFC 1661 and RFC 1662, preceded with a one-byte pseudo-header with a zero
   value meaning "received by this host" and a non-zero value meaning "sent
   by this host"; if the first 2 bytes are 0xff and 0x03, it's PPP in
   HDLC-like framing, with the PPP header following those two bytes,
   otherwise it's PPP without framing, and the packet begins with the PPP
   header. The data in the frame is not octet-stuffed or bit-stuffed.

in this case, the text in linktypes.html is probably most appropriate.

Name:		draft-gharris-opsawg-pcap
Revision:	01
Title:		PCAP Capture File Format
Document date:	2020-12-22
Group:		Individual Submission
Pages:		29
URL:            https://www.ietf.org/archive/id/draft-gharris-opsawg-pcap-01.txt
Status:         https://datatracker.ietf.org/doc/draft-gharris-opsawg-pcap/
Html:           https://www.ietf.org/archive/id/draft-gharris-opsawg-pcap-01.html
Htmlized:       https://tools.ietf.org/html/draft-gharris-opsawg-pcap-01
Diff:           https://www.ietf.org/rfcdiff?url2=draft-gharris-opsawg-pcap-01

---

From: Guy Harris <[email-masked]>
Date: Mon, 21 Dec 2020 18:23:34 -0800

> 1) reserve bits 16:28 of linktype as zero.

In pcap files, presumably; you have only bits 0:15 in pcapng IDBs.

Note that the registry is for both pcap and pcapng, so the specs should say that.

>  * values from 65000 to 65536 are marked as Private Use.

Presumably "to 65535" - 65536 doesn't fit in the 16-bit pcapng field.

So, for FCFS, does that mean anybody who wants a linktype can just grab one?

And, as per my idea of using 65535 to mean "custom linktype", similar to pcapng custom blocks and options, with either:

	the pcap file header/pcanng IDB option containing a Private Enterprise Number and private linktype number;

	the pcap file header/pcanng IDB option containing a Private Enterprise Number, with any linktype specifier being in the link-level header;

	the Private Enterprise Number and anything else being in the link-level header;

should we reserver 65535?

>   LINKTYPE_PPP_ETHER                  |   51   |PPPoE; per RFC 2516

That one's there for NetBSD; I *think* the packet contains just a PPPoE header and payload.  I may have to dig into the NetBSD code to see what they do.

---

From: Guy Harris <[email-masked]>
Date: Mon, 21 Dec 2020 18:39:24 -0800

(Resent, from the correct address this time.)

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 22 Dec 2020 04:03:44 +0100

On 2020-12-22, at 01:31, Michael Richardson <[email-masked]> wrote:
> this suggests that the FCS length is really only 3 bits

4 bits, really (assuming 32-bit unsigned integers).

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Mon, 21 Dec 2020 23:13:32 -0500

> 4 bits, really (assuming 32-bit unsigned integers).

The LT_FCS_LENGTH(x) (bit 28) is treated as a flag to say that the thing is there.
The DATALINK_EXT(x) might be borked here, only accepting odd values, which
would really be a problem.  I suspect there are bugs here.

- --

---

From: Guy Harris <[email-masked]>
Date: Tue, 22 Dec 2020 01:01:01 -0800

>      SHOULD be filled with 0 by pcap file writers, and MUST be ignored

Perhaps that field should be called "LinkTypeandInfo", or something such as that, to indicate that only the lower 16 bits are the link type.  (Link-layer header types are shared by pcap and pcapng, and the link-layer header type in a pcapng Interface Description Block is 16 bits.)

> 7 bytes?  Or does 0 indicate 8 bytes?  Ethernet is 4...).

0 indicates "no FCS present".

And, yes, the spec should indicate that.

> I can find no other references.  So I guess Ethernet gets a value of 2 (*16 bits).

Yes, the length of the FCS is in 16-bit units.

And, yes, the spec should indicate that.

> The code does not ignore bits 28:16 of the linktype field (the bits numbered

They were originally intended for use with some stuff NetBSD was doing (I'd have to look into the history of the NetBSD code), but I think NetBSD stopped doing that.

> 1) confirmation that Linktype is 16-bits.

Yes.

>   Is 0 valid?

Yes - it means "packets do not contain an FCS".

>  Or would that be indicated by LENGTH_PRESENT(x)==0?

*That* means "the FCS length, or whether there is an FCS, is unknown"; Wireshark does some heuristics to try to guess whether Ethernet packets have an FCS (I added those because, a long time ago, in a galaxy far far away, some Macs delivered Ethernet FCSes when capturing over BPF, and that messed up packet dissection in some cases).

---

From: Guy Harris <[email-masked]>
Date: Tue, 22 Dec 2020 01:23:48 -0800

> They were originally intended for use with some stuff NetBSD was doing (I'd have to look into the history of the NetBSD code), but I think NetBSD stopped doing that.

The commit message for the change that added the macros was:

commit afbb1ce7227dc5edb291f242ed8d95cd3762fc51
Author: Guy Harris <[email-masked]>
Date:   Sat Sep 29 19:33:29 2007 +0000

    Based on work from Florent Drouin, split the 32-bit link-layer type
    field in a capture file into:
    
            a 16-bit link-layer type field (it's 16 bits in pcap-NG, and
            that'll probably be enough for the foreseeable future);
    
            a 10-bit "class" field, indicating the group of link-layer type
            values to which the link-layer type belongs - class 0 is for
            regular DLT_ values, and class 0x224 grandfathers in the NetBSD
            "raw address family" link-layer types;
    
            a 6-bit "extension" field, storing information about the
            capture, such an indication of whether the packets include an
            FCS and, if so, how many bytes of FCS are present.

So what NetBSD had was a convention where a capture file could have a link-layer type that combined an AF_ value with some additional bits to distinguish the value from a regular LINKTYPE_ value; I don't know what AF_ values they supported for that, or where that code was, or whether it's still supported.

---

From: Michael Richardson <[email-masked]>
Date: Tue, 22 Dec 2020 11:36:02 -0500

<#secure method=pgp mode=sign>

    > In pcap files, presumably; you have only bits 0:15 in pcapng IDBs.

Yes.  That's why I changed the illustration of the packet so that only 16
bits were in the LinkType.

    > Presumably "to 65535" - 65536 doesn't fit in the 16-bit pcapng field.

Oops.

    > So, for FCFS, does that mean anybody who wants a linktype can just grab one?

Yes, but it's not quite the free-for-all one might imagine.
They have to email IANA and there are some records kept.
IANA does do some amount of abuse control.

I've re-read RFC8726, in case we want to send pcap via the ISE, as OPSAWG is
very slow to adopt documents.   The short of it is that we *CAN* create
Registries on the Independent Stream Editor Q.
  https://www.rfc-editor.org/about/independent/

But, we can not create Specification Required registries as that requires
assignment of Designated Experts.  FCFS streams are fine.
Options are there:
  1) get OPSAWG to adopt this document.
  2) ask an AD to sponsor the document.
  3) not split the Registry as per above, use FCFS for it all.
  4) move the LinkType Registry to pcapng, send pcap via ISE as
     Informational (might get hung up waiting for pcapng, however, depending
     upon how we write the text, and whether or not it results in a MISREF)

I have no particular preference.

    > And, as per my idea of using 65535 to mean "custom linktype", similar
    > to pcapng custom blocks and options, with either:

I'm happy with this proposal, but isn't it pcapng specific?
We can reserve 65535 itself for that use.

    > That one's there for NetBSD; I *think* the packet contains just a PPPoE
    > header and payload.  I may have to dig into the NetBSD code to see what
    > they do.

okay, but we don't have to get that perfect in the document.
What matters is that it points to /linktypes.html in the IANA registry.

---

From: Michael Richardson <[email-masked]>
Date: Tue, 22 Dec 2020 11:40:47 -0500

> So what NetBSD had was a convention where a capture file could have a
    > link-layer type that combined an AF_ value with some additional bits to
    > distinguish the value from a regular LINKTYPE_ value; I don't know what
    > AF_ values they supported for that, or where that code was, or whether
    > it's still supported.

Wow, lots of ill-defined complexity here.

I think that we should just regard this as water under the bridge.
If NetBSD wants to propose a use for those empty bits, then a new
specification could update that use case.

---

From: <[email-masked]>
Date: Tue, 22 Dec 2020 18:53:11 +0000

Hi Michael,

One comment inline. 

Cheers,
Med

> -----Message d'origine-----
> De : OPSAWG [mailto:[email-masked]] De la part de Michael
> Richardson
> Envoyé : mardi 22 décembre 2020 17:36
> À : Guy Harris <[email-masked]>
> Cc : Pcap-ng file format <[email-masked]>;
> [email-masked]; tcpdump-workers <[email-masked]>
> Objet : Re: [OPSAWG] [pcap-ng-format] draft-gharris-opsawg-pcap.txt
> --- IANA considerations
> 
> But, we can not create Specification Required registries as that
> requires assignment of Designated Experts.  

[Med] I'm not sure if the rules changed since then, but experts can be designated for registries created by ISE documents. See for example: https://www.iana.org/assignments/iax-parameters/iax-parameters.xhtml#iax-parameters-7 

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Michael Richardson <[email-masked]>
Date: Tue, 22 Dec 2020 14:36:31 -0500

> [Med] I'm not sure if the rules changed since then, but experts can be
    > designated for registries created by ISE documents. See for example:
    > https://www.iana.org/assignments/iax-parameters/iax-parameters.xhtml#iax-parameters-7

Well, this document is ten years ago, and RFC8726 is Nov. 2020, and I think
that the confusion on this question is what generated the need for RFC8726.

Adrian: wrt:
  https://mailarchive.ietf.org/arch/msg/opsawg/m4I4vMrYyAxhv18k3LcUk-_Te94/

---

From: Guy Harris <[email-masked]>
Date: Tue, 22 Dec 2020 12:04:57 -0800

> I'm happy with this proposal, but isn't it pcapng specific?

No - it's *cleaner* to implement in pcapng, as you can use Interface Description Block (IDB) options to provide the Private Enterprise Number (PEN) and an enterprise-specific encapsulation type, but:

	if we go with the PEN and and enterprise-specific encapsulation type with IDB options, for pcap we can steal the former time stamp offset (Reserved1) and time stamp accuracy (Reserved2) fields, interpreting them as the PEN and enterprise-specific encapsulation type, respectively, if the link type is 65535;

	if we go with the PEN as an IDB option, and say that if an enterprise wants more than one encapsulation type, they'd have to put a encapsulation type at the beginning of the payload, so, for pcap, we'd steal the former time stamp offset (Reserved1), interpreting it as the PEN if the link type is 65535;

	if we go with putting the PEN and encapsulation type at the beginning of the payload, that would work the same way for pcap as it does for pcapng.

---

From: "Adrian Farrel" <[email-masked]>
Date: Tue, 22 Dec 2020 20:44:46 -0000

Hi,

I'm pitching in here as Independent Submissions Editor (ISE). I really should use the proper email for this ([email-masked]) but I'm sticking with this address as it is already subscribed to the OPSAWG list.

Responding to this particular email because (I think) it is the first one that mentioned me by name.

Let's take a few things:

1. Find out more about the Independent Submissions Stream by looking at https://www.rfc-editor.org/about/independent/ and reading the referenced RFCs.

2. Michael is right that RFC 8726 will govern how I handle drafts that make requests for IANA action. Precedent will not carry any weight: 8726 describes the process in place at the moment.

3. While it *is* true that 8726 describes situations under which IANA registries can be created by Independent Submission Stream documents, be aware that the conditions are very limited. Also be aware that the assignment policies that you can set for such registries are also very limited.

4. The Independent Submissions stream is not a short-cut to RFC publication. If there is a proper home within the IETF then that path must be followed. If the IETF process is not functioning to your satisfaction, then you need to work with the IETF processes to resolve that issue.

A previous email on this thread said...

> IANA does do some amount of abuse control.

FWIW, you can impose a little bit of control by specifying a form that must be used to make a FCFS request to your registry.

>     upon how we write the text, and whether or not it results in a MISREF)

I strongly suggest that you pursue options 1 and 2 because the work appears to be in scope for the OPS Area.
If the OPSAWG declines to adopt the work, and if you can find no AD to sponsor the draft, then it is worth coming to the ISE (me). Additionally, in the (very unlikely) event that progress is procedurally blocked and appeals are not producing results, I can also consider publication.

Obviously, as already noted, if the document does wind up with the ISE:
- IANA actions will need to conform to 8726
- normative references will need to be cleared before an RFC can be published

Cheers,
Adrian
--
Independent Submissions Editor (ISE)
[email-masked]

Adrian: wrt:

---

From: Michael Richardson <[email-masked]>
Date: Tue, 22 Dec 2020 20:14:07 -0500

> No - it's *cleaner* to implement in pcapng, as you can use Interface
    > Description Block (IDB) options to provide the Private Enterprise
    > Number (PEN) and an enterprise-specific encapsulation type, but:

Ah. I'd like to say that we document the PEN based extension it in pcapng,
and not worry about pcap.

---

From: tom petch <[email-masked]>
Date: Wed, 23 Dec 2020 10:53:02 +0000

From: OPSAWG <[email-masked]> on behalf of Adrian Farrel <[email-masked]>
Sent: 22 December 2020 20:44

Hi,

Let's take a few things:

<tp>
And  Designated Experts are appointed by the IESG and if no WG can be found to take on the I-D then the IESG might consider that the IETF does not have the expertise to provide Designated Experts - well, I would.

As Adrian says below, options 1 and 2 look the right way to go.

Tom Petch

Cheers,
Adrian

Adrian: wrt:

_______________________________________________
OPSAWG mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/opsawg