Subject: [homenet] DNCP/HNCP Revisited

From: "Ray Hunter (v6ops)" <[email-masked]>
Date: Wed, 18 Sep 2019 09:36:05 +0200

Hi,

I've been experimenting with Homenet before looking at enhancing HNCP 
for extended naming functionality (the current implementation only 
covers resolver configuration and not name server configuration).

During my testing I managed to break HNCP, so that it got stuck in a 
state where it didn't converge.

Looking at the specification in detail this shouldn't have been a 
surprise to me or anyone else due to the design choices that were made.

First observation: Running HNCP over L2TPv3 breaks HCNP because L2TPv3 
breaks UDP fragmentation (works as designed).

The L2TPv3 tunnel has a lower MTU than the local LAN, and does not 
report ICMP PTB, so HNCP packets in one direction get through, but 
replies get dropped.

Early drafts of HNCP stated that UDP fragmentation would not be broken 
in the Homenet for the foreseeable future. Well I managed to break that ;)

So this is a known situation.

Changing the MTU on the LAN interfaces of my routers to 1280 brought 
everything back to normal, as expected.

Question: If Homenets are moving to flat L2 meshes over foo, as some 
have said, will this impact HNCP?

The Linux kernel (in Openwrt) can/could easily support PMTUD.

But since the L2 tunnels don't report drops, it makes PMTUD potentially 
challenging for a simple protocol like HNCP.

Question: As a simple mitigation, is there any way of manually 
signalling to the kernel that ALL UDP packets on port 8231 should assume 
an PMTU of 1280 octets?

That wouldn't require any specification change and would allow HNCP to 
work reliably in the presence of tunnels and varying MTU's that don't 
match the local interface MTU.

Second Observation: HNCP packets grow quickly in size, which may become 
a concern for scaling.

Again, by design, DNCP is limited to TLV content of 64K (RFC7787 intro).

AFAICS that is 64K per DNCP system, not per node.

Question: Is that correct?

For my 4 node Homenet, HNCP reached 3140 octets, without any 
modifications or additions or long names.

Question: Does the amount of state scale linearly with the number of 
routers?

Question: Is this starting to become a concern?

The limitation for 64K seems to come from the following:

RFC7788
Request Network State TLV (Section 7.1.1): The receiver MUST reply
       with a Network State TLV (Section 7.2.2) and a Node State TLV
       (Section 7.2.3) for each node data used to calculate the network
       state hash.

In the Openwrt implementation, I observe that the requester is 
requesting the node state for all 4 devices in one packet.

I also observe that the reply from the peer contains the entire state of 
the entire Homenet in a single UDP packet (network TLV and node state 
TLV for all 4 routers + optional payload data).

Question: Is that as designed/intended? Is this where the 64K data limit 
comes from in RFC7787?

If this was a design goal to reduce chatiness, and improve convergence 
time, is it possible to implement this differently to reduce packet size 
and/or increase overall TLV data capacity?

Could the requester in step1 request the network status TLV, and the 
peer reply with just the network status TLV and the node TLV's, but 
without the underlying optional HNCP payload data TLV's.

Maybe include a new "more-HNCP-data-to-come" TLV to distinguish this 
from blank node data?

That would give enough information for both nodes to know exactly which 
payload data needed synchronising.

Then as step 2, the requester could individually request the Node TLV 
for each node one per packet, and the peer would reply with the Node TLV 
but this time including the optional data TLV's.

The receiver and peer would both know that HNCP hadn't converged whilst 
waiting for the additional data, and could mark it internally as being 
stale whilst waiting for the additional detailed update.

Question: What would break?

Question: What would need to be amended in the standard RFC? The 
more-HNCP-data-to-come TLV in RFC7788?

Question: Would this tweak increase the ±64K limit of TLV data from 
being per network to being 64K per node? [max UDP packet size for a 
single node TLV + associated payload data]

Question: Is this raw fragmentation by separating DNCP essentials from 
HNCP payload something that the WG would support?

regards,

-- 
regards,
RayH
<https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach>

---

From: Mark Andrews <[email-masked]>
Date: Wed, 18 Sep 2019 20:00:48 +1000

> On 18 Sep 2019, at 5:36 pm, Ray Hunter (v6ops) <[email-masked]> wrote:
> 
> Hi,
> 
> I've been experimenting with Homenet before looking at enhancing HNCP for extended naming functionality (the current implementation only covers resolver configuration and not name server configuration).
> 
> During my testing I managed to break HNCP, so that it got stuck in a state where it didn't converge.
> 
> Looking at the specification in detail this shouldn't have been a surprise to me or anyone else due to the design choices that were made.
> 
> 
> 
> First observation: Running HNCP over L2TPv3 breaks HCNP because L2TPv3 breaks UDP fragmentation (works as designed).
> 
> The L2TPv3 tunnel has a lower MTU than the local LAN, and does not report ICMP PTB, so HNCP packets in one direction get through, but replies get dropped.
> 
> Early drafts of HNCP stated that UDP fragmentation would not be broken in the Homenet for the foreseeable future. Well I managed to break that ;)
> 
> Changing the MTU on the LAN interfaces of my routers to 1280 brought everything back to normal, as expected.
> 
> Question: If Homenets are moving to flat L2 meshes over foo, as some have said, will this impact HNCP?
> 
> But since the L2 tunnels don't report drops, it makes PMTUD potentially challenging for a simple protocol like HNCP.
> 
> Question: As a simple mitigation, is there any way of manually signalling to the kernel that ALL UDP packets on port 8231 should assume an PMTU of 1280 octets?

setsockopt(IPV6_USE_MIN_MTU=1); from RFC 3542 works provided the OS has implemented it.  It’s a really pity that POSIX hasn’t picked up RFC 3542 in the 16 years since it was published like they picked up RFC 3493.  This is a epic fail of SDOs.

> That wouldn't require any specification change and would allow HNCP to work reliably in the presence of tunnels and varying MTU's that don't match the local interface MTU.
> 
> 
> 
> 
> Second Observation: HNCP packets grow quickly in size, which may become a concern for scaling.
> 
> For my 4 node Homenet, HNCP reached 3140 octets, without any modifications or additions or long names.
> 
> Question: Does the amount of state scale linearly with the number of routers?
> 
> In the Openwrt implementation, I observe that the requester is requesting the node state for all 4 devices in one packet.
> 
> I also observe that the reply from the peer contains the entire state of the entire Homenet in a single UDP packet (network TLV and node state TLV for all 4 routers + optional payload data).
> 
> Question: Is that as designed/intended? Is this where the 64K data limit comes from in RFC7787?
> 
> If this was a design goal to reduce chatiness, and improve convergence time, is it possible to implement this differently to reduce packet size and/or increase overall TLV data capacity?
> 
> Could the requester in step1 request the network status TLV, and the peer reply with just the network status TLV and the node TLV's, but without the underlying optional HNCP payload data TLV's.
> 
> Maybe include a new "more-HNCP-data-to-come" TLV to distinguish this from blank node data?
> 
> That would give enough information for both nodes to know exactly which payload data needed synchronising.
> 
> Then as step 2, the requester could individually request the Node TLV for each node one per packet, and the peer would reply with the Node TLV but this time including the optional data TLV's. 
> 
> The receiver and peer would both know that HNCP hadn't converged whilst waiting for the additional data, and could mark it internally as being stale whilst waiting for the additional detailed update.
> 
> Question: What would need to be amended in the standard RFC? The more-HNCP-data-to-come TLV in RFC7788?
> 
> Question: Would this tweak increase the ±64K limit of TLV data from being per network to being 64K per node? [max UDP packet size for a single node TLV + associated payload data]
> 
> Question: Is this raw fragmentation by separating DNCP essentials from HNCP payload something that the WG would support?
> 
> regards,
> 
> -- 
> regards,
> RayH
> _______________________________________________
> homenet mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/homenet

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: [email-masked]

---

From: "Ray Hunter (v6ops)" <[email-masked]>
Date: Wed, 18 Sep 2019 16:25:21 +0200

> setsockopt(IPV6_USE_MIN_MTU=1); from RFC 3542 works provided the OS has implemented it.  It’s a really pity that POSIX hasn’t picked up RFC 3542 in the 16 years since it was published like they pic…

Thanks for the tip.

Unfortunately checking <linux/in6.h> in the current openwrt build tree.
.
#if 0   /* not yet */
#define IPV6_USE_MIN_MTU        63
#endif

:( not yet implemented

---

From: Michael Richardson <[email-masked]>
Date: Wed, 18 Sep 2019 15:32:33 -0400

> The L2TPv3 tunnel has a lower MTU than the local LAN, and does not report
    > ICMP PTB, so HNCP packets in one direction get through, but replies get
    > dropped.

    > Early drafts of HNCP stated that UDP fragmentation would not be broken in the
    > Homenet for the foreseeable future. Well I managed to break that ;)

(Why are we using a L2 tunnel?  So that a few of us, including Ray, can hack on stuff together)

    > everything back to normal, as expected.

I think that the right answer is for the software to assume a 1280 MTU.
Someone could write an HNCP extension to do PLPMTUD inside afterwards to
increase the size if desired.

    > said, will this impact HNCP?

Unlikely, in my opinion, because "good" L2 meshes will preserve the 1500 byte MTU.

    > Question: As a simple mitigation, is there any way of manually signalling to
    > the kernel that ALL UDP packets on port 8231 should assume an PMTU of 1280
    > octets?

    > That wouldn't require any specification change and would allow HNCP to work
    > reliably in the presence of tunnels and varying MTU's that don't match the
    > local interface MTU.

I think that the HNCP code can do this.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Juliusz Chroboczek <[email-masked]>
Date: Wed, 18 Sep 2019 21:39:41 +0200

> The L2TPv3 tunnel has a lower MTU than the local LAN, and does not report ICMP
> PTB, so HNCP packets in one direction get through, but replies get dropped.

Is that not a bug?

-- Juliusz

---

From: Ted Lemon <[email-masked]>
Date: Wed, 18 Sep 2019 16:05:39 -0400

> Is that not a bug?

The problem is, how’d the packet get so big that it was fragmented?

---

From: Gert Doering <[email-masked]>
Date: Wed, 18 Sep 2019 22:57:32 +0200

Hi,

On Wed, Sep 18, 2019 at 04:05:39PM -0400, Ted Lemon wrote:
> The problem is, how???d the packet get so big that it was fragmented?

If you have a discontinuous L2 MTU, you do not need fragmented packets
to see packets disappear.

Host A has an ethernet MTU of 1500.  Sends a packet with 1400 bytes, no
fragmentation needed.

Somewhere in between is an ethernet bridge that only handles 1350 payload
("L2TPv3 with an outer max packet size capped to something too small to
transport 1518 byte packets inside, and not doing outer fragmentation")

Packet gone.

No fragmentation of any sort involved, just incorrectly set up L2 segments.

(Rule #1 for real world operation: ensure that end system L3 MTU is 
always <= the smallest L2 MTU that the packet might encounter in your 
L2 fabric)

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                      Vorstand: Sebastian v. Bomhard, Michael Emmer
Joseph-Dollinger-Bogen 14        Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                 HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444         USt-IdNr.: DE813185279

---

From: Ted Lemon <[email-masked]>
Date: Wed, 18 Sep 2019 17:12:23 -0400

> to see packets disappear.

That’s kind of a non-sequitur.  The packet would need to be fragmented in this case.  The question is, how’d it get that big?  If we’re seeing packets large enough to fragment in a normal-sized network, that’s a really big problem (pardon the pun).

---

From: Juliusz Chroboczek <[email-masked]>
Date: Thu, 19 Sep 2019 01:02:33 +0200

> The problem is, how’d the packet get so big that it was fragmented?

HNCP relies on network-layer fragmentation: it uses UDP and has no
application-layer mechanism for fragmenting large TLVs.  See Section 4.2
and Appendix B.2 of RFC 7787.

(I seem to recall that an earlier version of DNCP included provisions for
application-layer fragmentation of TLVs, but that was removed at some
point.  I don't remember why.)

Let's please come back to my question.  RFC 4443 paragraph 3.2 says

   A Packet Too Big MUST be sent by a router in response to a packet
   that it cannot forward because the packet is larger than the MTU of
   the outgoing link.

If your tunnelling software violates this, how is it not buggy?

-- Juliusz

---

From: Juliusz Chroboczek <[email-masked]>
Date: Thu, 19 Sep 2019 01:09:42 +0200

> to see packets disappear.

Ah, I see.

> No fragmentation of any sort involved, just incorrectly set up L2 segments.

Right.  It's an incorrect network setup.

To fix that, it should be enough to point the tunnel directly at a Homenet
router rather than relying on bridging.  (A good idea in any case, since
Babel is able to make better routing decisions if the tunnel is directly
visible to it.)

-- Juliusz

---

From: Ted Lemon <[email-masked]>
Date: Wed, 18 Sep 2019 19:30:40 -0400

This still doesn’t address the problem that the HNCP packet needs to be fragmented. Fragmented Multicast doesn’t scale well. 

> On Sep 18, 2019, at 19:09, Juliusz Chroboczek <[email-masked]> wrote:
> 
> 
>>

---

From: Juliusz Chroboczek <[email-masked]>
Date: Thu, 19 Sep 2019 10:56:35 +0200

> This still doesn’t address the problem that the HNCP packet needs to be
> fragmented.  Fragmented Multicast doesn’t scale well.

HNCP doesn't fragment multicast, it only uses fragmentation for (link-local)
unicast.  This is way less severe than what you incorrectly claim.

At any rate, the right time to discuss that was 2015, not now.  HNCP is
a standards track protocol, and there's nobody left who's willing and
competent to work on a new revision.

-- Juliusz

---

From: Ted Lemon <[email-masked]>
Date: Thu, 19 Sep 2019 09:11:48 -0400

Yes, of course. We can never change a standards track protocol. That would be wrong. :)

What I’m trying to understand is how bad a problem this is. 

> On Sep 19, 2019, at 04:56, Juliusz Chroboczek <[email-masked]> wrote:
> 
> 
>>

---

From: "Ray Hunter (v6ops)" <[email-masked]>
Date: Thu, 19 Sep 2019 15:30:06 +0200

> and Appendix B.2 of RFC 7787.
Agreed. I'm aware it was a design decision.
> point.  I don't remember why.)
Yes. I've read earlier versions of the draft and it was an open point 
whether to do L7 fragmentation as part of HNCP or not.

I can't recall the exact discussion on list but it changed between 
version 4 and 5
https://tools.ietf.org/html/draft-ietf-homenet-hncp-04 (Appendix A)
https://tools.ietf.org/html/draft-ietf-homenet-hncp-05 (rely on UDP 
fragmentation)

> If your tunnelling software violates this, how is it not buggy?
Yes, this is a very poor network set up that could be avoided.

Question is: is it worth addressing?

I'm still working on the basis of a simple workaround to address the L2 MTU mismatch (limit IPv6 UDP fragment size to 1280 octets).

The separate problem is that if the amount of HNCP state increases (e.g. due to electing a central name server or mud server or whatever), the apparent limit of having to transmit the entire superset of all node TLV states in a single UDP packet may become problematic.

I notice in the current Openwrt code that the max UDP packet size is set at 9000 octets with the comment:

/* Very arbitrary. On some implementations, I have seen some issues
  * with 10+kb frames so we use this for now. It MUST be significantly
  * more than 4k, due to how code is written at the moment. */
#define HNCP_MAXIMUM_UNICAST_SIZE 9000

With current code (without expanding the TLV data set), and my sample test routers, that sets a current maximum network size of approx. 12 nodes.

---

From: RayH <[email-masked]>
Date: Thu, 19 Sep 2019 17:19:35 +0200

On 19 Sep 2019 15:11, Ted Lemon <[email-masked]> wrote:

That's what I'm still trying to understand.

The reason for the 64k limit is clear (due to the design choice of relying on UDP fragmentation).

What's still unclear to me is why the packet contains all the nodes' TLV's.

That is whether the limit is per node or per network.

It's not clear to me if that's an implementation issue (requesting four node TLV status requests in one packet, and ditto replies) or a standards issue (all node TLV's must be transmitted when the network TLV changes).

Regards,

RayH

homenet mailing list

[email-masked]

---

From: Juliusz Chroboczek <[email-masked]>
Date: Fri, 20 Sep 2019 01:06:40 +0200

> With current code (without expanding the TLV data set), and my sample
> test routers, that sets a current maximum network size of approx. 12
> nodes.

I don't think that the limit applies to the total network size, it applies to
the maximum size of a single NODE-STATE TLV.  If memory serves, HNCP is
able to send each NODE-STATE in a separate datagram, so the network can
grow without bound; it's the amount of data published by a single node
that is limited to slightly less than 9kB.

However, I agree that this could prove problematic if you build a very
dense network (a single node with a lot of neighbours) or if you have
large numbers of external connections terminating at a single node.

-- Juliusz

---

From: Juliusz Chroboczek <[email-masked]>
Date: Fri, 20 Sep 2019 01:29:46 +0200

>> HNCP is a standards track protocol, and there's nobody left who's
>> willing and competent to work on a new revision.

> Yes, of course. We can never change a standards track protocol. That

My wording was perhaps badly chosen.  Sorry for that.

I meant to say that I don't currently see anyone who would be both willing
and able to (1) change the HNCP spec to add application-layer
fragmentation, (2) update the hnetd implementation to obey the new
protocol, and (3) go through the somewhat time- and energy-consuming
process required to publish a new Standards Track protocol.

(To be clear -- as far as I'm concerned, I declare myself incompetent on
all three points above.  The most I could conceivably do would be to review
a new spec and update shncpd so it interoperates with a new revision of hnetd.)

> What I’m trying to understand is how bad a problem this is.

My understanding is that while HNCP should have no trouble scaling to
large networks, it is not designed to carry large amounts of per-node data.

This could cause trouble in the following cases:

  - if a single node has hundreds of HNCP neighbours (e.g. because it is
    connected to a large switch or serves as a tunnel server);

  - if a single node announces large numbers (dozens) of external
    connections; or

  - if a protocol extension dumps large binary blobs into HNCP.
  
-- Juliusz

---

From: Juliusz Chroboczek <[email-masked]>
Date: Fri, 20 Sep 2019 01:51:41 +0200

> What's still unclear to me is why the packet contains all the nodes' TLV's.

Because HNCP allows aggregating multiple TLVs in a single packet, to the
implementation's discretion.  Section 4.2 of RFC 7787.

-- Juliusz

---

From: Ted Lemon <[email-masked]>
Date: Thu, 19 Sep 2019 20:06:22 -0400

> implementation's discretion.  Section 4.2 of RFC 7787.

Thanks, Juliusz.  This and your previous message were quite helpful.

I think there does need to be some hacking done on HNCP, but I agree with you that it’s a heavy lift.

---

From: RayH <[email-masked]>
Date: Fri, 20 Sep 2019 12:20:39 +0200

Hi,

I've done some more investigation and the "problem" packet size growth looks to be the result of a single UDP reply packet containing multiple node state TLV's.

This reply is triggered by a query from a new node joining an existing network, where the new node requests node TLV updates for all of the existing routers in a single request packet.

It would seem to me that there needs to be some clarification for implementations to avoid excessive packet growth, and potentially exceeding the limits of UDP fragmentation, with unpredictable results.

1) DNCP allows an option of whether a network state TLV contains optional nested payload (HNCP) TLV's or not.

There is no TLV defined in DNCP to indicate whether optional data has been included or excluded, or whether the node TLV optional payload is simply empty.

If one or more (optional) node data TLV's are included nested in the network status TLV reply, the reply could grow unbounded.

Humbly suggest: In order to avoid excessive growth of network status TLV's, and to allow trickle to work effectively, the HNCP profile should specify that optional data SHOULD NOT be sent with a network status TLV reply.

Node status updates SHOULD be explicitly requested where necessary to obtain updates of optional data, separately from network status TLV updates.

2) The node requesting a node status TLV doesn't know in advance how big a reply packet will be generated.

DNCP states that nodes MUST reply to all node status TLV queries.

So requesting multiple node status TLV's in one packet could lead to an oversized UDP reply packet.

Humbly suggest: In order to avoid potential overflow of the size limits of UDP reply packets, an HNCP node SHOULD NOT request more than one node status TLV update per request packet.

Thoughts?

-- Juliusz

homenet mailing list

[email-masked]

---

From: Juliusz Chroboczek <[email-masked]>
Date: Fri, 20 Sep 2019 12:40:38 +0200

> nested payload (HNCP) TLV's or not.

I'm pretty sure that's not the case.  RFC 7787 Section 7.2.2.

The Network-State TLV only contains the network state hash, short
Node-State TLVs are separate top-level TLVs.  An implementation may choose
to send them in the same packet, but they're independent TLVs and can be
sent in different packets.

> DNCP states that nodes MUST reply to all node status TLV queries.

The replying node MUST reply to all node state queries.  However, it is up
to the replying node whether these replies are sent in a single packet or
split into multiple packets.

In other words, the only constraint is that every single node state TLV
must be sent in its entirety within a single packet.  As described in
a previous mail, this does bound the amount of data that a single node can
publish, but no bounds on the total size of the network.

> oversized UDP reply packet.

The replying node's behaviour has nothing to do with whether the requests
are aggregated in a single packet or not.  The replying node processes
each request independently, whether it finds them in a single packet or in
multiple packets.

-- Juliusz

---

From: "Ray Hunter (v6ops)" <[email-masked]>
Date: Fri, 20 Sep 2019 13:37:34 +0200

Thanks for your response.

> sent in different packets.
A OK so you're saying this is already covered in (Section 4.4 of) 7787

   Upon receipt of:

    o  Request Network State TLV (Section 7.1.1 <https://tools.ietf.org/html/rfc7787#section-7.1.1>): The receiver MUST reply
       with a Network State TLV (Section 7.2.2 <https://tools.ietf.org/html/rfc7787#section-7.2.2>) and a Node State TLV
       (Section 7.2.3 <https://tools.ietf.org/html/rfc7787#section-7.2.3>) for each node data used to calculate the network
       state hash.  The Node State TLVs SHOULD NOT contain the optional
       node data part to avoid redundant transmission of node data,
       unless explicitly specified in the DNCP profile.

So what I was suggesting was merely additional clarification of that.

> each request independently, whether it finds them in a single packet or in

So if I understand you correctly, you're saying this is the problem of 
the sender of the response to ensure UDP fragmentation doesn't break, 
and that multiple long UDP replies can be generated from a single query 
packet (potential amplification).

If there's multiple UDP replies required for a single query, would you 
expect sending of these individual packets to also be rate limited by 
trickle?

What behavior would we expect from the requester during this time?
Wait for all outstanding replies to arrive?
Re-transmit a node TLV request for missing / dropped replies?

Thanks!

---

From: Juliusz Chroboczek <[email-masked]>
Date: Fri, 20 Sep 2019 14:23:42 +0200

> A OK so you're saying this is already covered in (Section 4.4 of) 7787

[...]

> So what I was suggesting was merely additional clarification of that.

Careful here.  There is no optional part in the *network*-state TLV - this
TLV's payload is just the network state hash.  The optional payload is for
the *node*-state TLVs.

> oversized UDP reply packet.

Yes, this was discussed back in July 2015.  Here's what I said back then:

    It should also say that a node MUST NOT send a datagram with a larger
    payload, and that it MUST silently drop any larger datagrams (optionally
    log to system management, etc.).  If this is not done, persistent state
    desynchronisation may occur.

This is what Markus replied:

    I am not sure I want to cripple use in non-crippled networks, just provide
    guaranteed base value which works everywhere and *RED ALERT* light should
    light up on crippledbox if it encounters this

Since nobody supported my position at the time, I had to agree to the
following:

  - Section 3 of RFC 778 says that "Each node MUST be able to receive (and
    potentially reassemble) UDP datagrams with a payload of at least
    4000 bytes."

  - there is no limit on the size of the packets that a node may send.

> So if I understand you correctly, you're saying this is the problem of the
> sender of the response to ensure UDP fragmentation doesn't break,

I'm not sure what you mean.  We certainly assume that the network obeys
RFCs 2460 and 4443.

> packet (potential amplification).

Let's please discuss security at some other time, I'd like the current
discussion to come to a conclusion first.

> If there's multiple UDP replies required for a single query, would you expect
> sending of these individual packets to also be rate limited by trickle?

Let's please discuss congestion control at some other time, I'd like the
current discussion to come to a conclusion first.

> What behavior would we expect from the requester during this time?

The requester parses each top-level TLV in turn, and acts upon it.

> Wait for all outstanding replies to arrive?

No.  The requester parses each top-level TLV and acts upon it as soon as
it arrives.

> Re-transmit a node TLV request for missing / dropped replies? 

How would the receiving node detect missing replies?

If some node-state TLVs are missing, then the receiving node's state might
not get updated correctly, which will cause the network hash to mismatch,
which will be detected when it receives a new network-state TLV
(trickle-driven, worst-case 17s).

-- Juliusz

---

From: "Ray Hunter (v6ops)" <[email-masked]>
Date: Sun, 22 Sep 2019 12:49:12 +0200

> TLV's payload is just the network state hash.  The optional payload is for
I see now.

The network state TLV's and node state TLV's are completely separate 
TLV's (not nested in any way) that can be both generated as the result 
of one inbound request (request network state TLV), and they may be sent 
in separate UDP packets.
> The requester parses each top-level TLV in turn, and acts upon it.
This is very illuminating for me, and removes a false assumption on my part.
> (trickle-driven, worst-case 17s).

Thanks very much for your patience and replies to my questions that have 
certainly filled gaps in my understanding of the protocol that I had 
from reading the standards and the code alone.

AFAICS there's certainly enough room for me to experiment with patches to
i) reduce MTU to avoid problems arising from L2 MTU mismatches and
ii) to reduce the amount of fragments (at the expense of more UDP packets)
without any tweaking of the standards.

My reason for investigating ii) is to potentially reduce the impact of 
the loss of an individual frame on a lossy link (we would lose 1 node 
status TLV from 1 device rather than multiple TLV's related to multiple 
devices).

---

From: Juliusz Chroboczek <[email-masked]>
Date: Sun, 22 Sep 2019 14:10:39 +0200

> without any tweaking of the standards.

In case you're interested, here's the code that decides whether to
aggregate or to ship out in shncpd:

  https://github.com/jech/shncpd/blob/master/send.c#L99

In short -- we try to limit our payload to 1412 bytes whenever possible,
within the constraints of not being able to fragment large TLVs.  (This
should depend on the outgoing interface's MTU, as in Babel, but I decided
I'm lazy and just hardwired the constant.)

> status TLV from 1 device rather than multiple TLV's related to multiple

Recall, however, that 802.11 has absolutely horrific per-frame overhead.
Recall further that HNCP does flooding, so that if there are parallel
lossy and lossless links, packet loss on the lossy link doesn't matter as
long as the data gets correctly flooded over the lossless link.

In short, there is a non-trivial tradeoff here.

-- Juliusz