Subject: Re: [Sidrops] I-D Action: draft-ietf-sidrops-prefer-rrdp-01.txt

From: Jeroen Massar <[email-masked]>
Date: Wed, 1 Dec 2021 10:59:15 +0100

TLDR: Fix problems, instead of putting everything into HTTP.... the Internet is not just the Web

Hi Koen,

> On 20211130, at 23:17, Koen van Hove <[email-masked]> wrote:
> 
> Dear Tim, all,
> 
> I have looked at your motivation, particularly:
>>   *  rsync is CPU and memory heavy on the server side, and easy to DoS
> 
> I decided to test your hypothesis, as RFC 6480 says otherwise, and created a
> small program that aims to DoS an rsync daemon using many simultaneous
> connections from different IP addresses (but the same machine). My
> conclusion is the same as yours - with the old computer (4 cores, 16 GB of
> RAM, home fiber connection) underneath my desk I can currently take down my
> own test rsync server (8 cores, 16 GB RAM, 1 Gbps connection, different
> network).

8 cores? 16 GB? My phone has better CPU/mem and connectivity than that... that is literally puny.

Heck, your university of utwente has way more connectivity than that, and they where DoSed easily off-net too:

 https://status.snt.utwente.io/incidents/42   (the answer is always 42)
 "Major outage effecting (nearly) all systems of SNT"

Please realize that ANYTHING can be (D)DoSsed given enough resources.

And as we have a LOT of people on the Internet and many that do not properly maintain their connected devices (IoT but also for instance Wordpress, or ManageEngine ADSelfService or soooo many problems), botnets in the order of 100.000's times 100mbit are common today and you go to any "Booter" or "Stresser" service and you get 50Gbit/s for a few dollars.
But, heck for 666CHF/year I can even have 25GBit/s at home from init7.net as a single link...

Of course, it would require you to sustain that DoS for a week, possibly two, to actually do any real damage. But centralisation is bad (even though one always have to go through the RIR) thus many orgs will want to host themselves (also makes some processes easier). [and the init7 example they will have complained and kick you off net by then ;) ]

As such, any infrastructure that does not want to be (D)DoSsed should be redundant, likely anycasted, and appropriately scaled as per the RFC.

It is also why many people opt for Hosted RPKI at the RIRs and why that setup will cost them a dear cent.

Fortunately both rsync and RRDP require TCP connections. Most UDP can be filtered at the border, that helps quite a bit.

But then as noted a 100k botnet can just do legit connections.... for a bit, as ISPs do band together and contact each other and try to resolve at least swaths of those botnets.
Hence... the longer the DDoS, the more annoyance, more fixes (or blocks) will be applied and at least the size of the botnet and the attack reduces.

I personally also advise doing the rpki-client part from disjunct networks, heck do it from random ASNs, just in case an operator wants to specifically target a specific ASN.

> Discussions with operators from RIRs seem to show that this would
> also work for their rsync servers.  I believe this would even allow one to
> take down all rsync servers currently running in the RPKI with a handful of
> computers, but I have not tested that as I do not want to disrupt actual
> operations.

Just in case you did not know yet: Please realize that it would be illegal to do it under the Dutch law you fall under (and other countries have bilateral agreements and will find you too).
I bet there is a law professor at your university that can explain to you if it is criminal or civil law that it will fall under and what the penalties would be.

[..]
> Rsync was not created to be used with a possibly malicious server.
> Therefore, there are currently simple things everyone could do that would
> disrupt a relying party over rsync, which are difficult to solve:
> 1. One can set a motd file that is several gigabytes in size. Rsync will
> wait until this is received (the --no-motd switch only hides it on the
> client) and the timeout set in rsync does not affect the maximum time this
> may take.

I can only suggest to fix the code.

Every piece of code in the world will have problems somewhere: please report and people will fix.
(Though as rpki.exposed shows, some of the problems (J+K) you have reported are still not resolved in some clients, things take time).

Also remember the original motto of the IETF: running code and rough consensus...

The running code so that one has experience in operating something, as one learns a lot from it.

Easy fix: if a server sends an overlong MOTD (eg >1MiB can already be considered that) put it on a blacklist and ignore it.
The only resources DDoSed that way are their own. You'll have a 'delay' once, but after that your client is happy.

> 2. One can create an rsync repository with many folders (I tested it with 27
> million folders). This fills up the maximum amount of inodes on most Linux
> and *BSD filesystems within minutes whilst bypassing any size limits or
> --includes and --excludes, as a folder has a size of 0, and folders are used
> in non-malicious repositories. 

I don't know about you, but filesystem quotas are things that have existed for well over 4 decades by now.

Thus that is really a non-issue.

Simpler version: have a filesystem specifically where you retrieve things.
Cooler version: have a filesystem per CA one fetches -- could use loopback FS or LVM-alike.

Heck, the latter one could even nicely have versioning on the filesystem and lots of other cool features allowing rollback etc etc with minimal overhead in storage.

Note that code has already solved that specific problem too:

https://manpages.debian.org/buster/rsync/rsync.1.en.html

8<-----------------
--prune-empty-dirs, -m

              This option tells the receiving rsync to get rid of empty
              directories from the file-list, including nested
              directories that have no non-directory children.  This is
              useful for avoiding the creation of a bunch of useless
              directories when the sending rsync is recursively scanning
              a hierarchy of files using include/exclude/filter rules.
------------------>8

Indeed, you are right, a malicious operator will be able to cause some kind of problems.
Blacklisting that operator is very easy and would only affect the resources for that operator.

And all other things, when discovered, report them, patch them and keep on running.

It is extremely easy to be on the "red team", and so many people think they are cool playing attacker. It is a lot more fun to play the blue team and defend against things. But as it is a game, both sides will change tactics, red will find something, blue will fix it... that is the way of life on the Internet.

Btw, for RRDP, you do not want to know how much fun I can have with HTTP/1.0... let alone with HTTP/2 or QUIC... so many evil server possibilities... the RFCs have comments on many of these techniques and the clients and servers address many too; can't stop them all though, at one point "that server is malicious" and you need to disconnect.

Greets,
 Jeroen

---

From: Ties de Kock <[email-masked]>
Date: Wed, 1 Dec 2021 18:06:37 +0100

Hi Jeroen,

> On 1 Dec 2021, at 10:59, Jeroen Massar <jeroen=[email-masked]> wrote:
> 

Remember that rsyncd is not a “modern” evented server. Real rsync deployments
have an extremely limited number of connections which is trivial to exceed.
Volumetric attacks are not needed.

During a short discussion on the risk posed by rsync in February 2020 at the
RIPE NCC, we listed at least the following risks with rsync:
  a: rsync - big repo
  b: rsync - inode exhaustion
  c: rsync - slowloris
  d: rsync - exhausting client their connection pool

(this is a partial list and resource exhaustion attacks on the server are much
easier than for modern http servers - even without considering CDN usage)

I think that Koen's earlier responsible disclosure/CVD (managed by NCSC) covered
a and d.
c hit the RIPE NCC [0]. And practically you can only monitor against this and
play whack-a-mole.
b was just mentioned in the thread.

> 
> https://status.snt.utwente.io/incidents/42 <https://status.snt.utwente.io/incidents/42>   (the answer is always 42)
> But, heck for 666CHF/year I can even have 25GBit/s at home from init7.net <http://init7.net/> as a single link...

> Of course, it would require you to sustain that DoS for a week, possibly two, to actually do any real damage. But centralisation is bad (even though one always have to go through the RIR) thus many…

Eight (? - 16 for the RIPE NCC)  hours before manifests expire. Not an big
issue at the moment, but a material risk if (a) have a standard that forces
compliant RPs to downgrade and open their RP up to trivial attacks and (b) you
deploy a protocol based on RPKI certificates that fails closed.

But that material risk would likely not exist in practice - I would expect
operators to opt out of deploying a security protocol with such a hole.

> I personally also advise doing the rpki-client part from disjunct networks, heck do it from random ASNs, just in case an operator wants to specifically target a specific ASN.

You can test what ASN an RP is used by. This actually is an interesting
experiment to see how much centralisation and re-use of RPs is happens :).

> I bet there is a law professor at your university that can explain to you if it is criminal or civil law that it will fall under and what the penalties would be.

My attacker model includes _malicious_ attackers (i.e. "this is illegal" does
not prevent the attack). Requiring signed objects is a road-bump (furthermore,
possession of private key does not prove the created hosted CA is legitimate).

> The only resources DDoSed that way are their own. You'll have a 'delay' once, but after that your client is happy.

I strongly discourage that tactic, that is way for an on-path attacker to
persistently blackhole a repository.

I think rsync clients should have a flag that:
  * aborts after a set number of objects received.
  * aborts after a set amount of bytes transferred by the connection.

And after aborting, the initial state should be restored, a partial transfer
breaks a repo. Here the behaviour of rsync implementations also differs.
Transfering files alphabetically instead of per directory majorly increases the
time window for a failed fetch.

> Simpler version: have a filesystem specifically where you retrieve things.

If you run an RP in a container I recommend running the storage on tmpfs,
especially if you have more memory than IOPs.

> https://manpages.debian.org/buster/rsync/rsync.1.en.html <https://manpages.debian.org/buster/rsync/rsync.1.en.html>
> 
> 8<-----------------
> --prune-empty-dirs, -m
> 

You can transfer empty files instead.
> Btw, for RRDP, you do not want to know how much fun I can have with HTTP/1.0... let alone with HTTP/2 or QUIC... so many evil server possibilities... the RFCs have comments on many of these techniq…

And both attackers and defenders can only spend the energy once. I think that
HTTP(S) is where we should spend the energy, and also where a lot of research
has been performed already.

And to all the readers: if I missed a "modern" evented rsyncd implementation
please let me know. I would be extremely happy to evaluate that.

Kind regards,
Ties

[0]: https://www.ripe.net/ripe/mail/archives/routing-wg/2020-April/004084.html
--

> 
> Greets,
> Jeroen
> 
> _______________________________________________
> Sidrops mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/sidrops <https://www.ietf.org/mailman/listinfo/sidrops>

---

From: "Koen van Hove" <[email-masked]>
Date: Fri, 3 Dec 2021 02:35:51 +0100

Dear Jeroen, all,

I am not sure I fully follow you - could you clarify your point?

This is not about DDoS - this is about a single ordinary machine exhausting
all connections of multiple rsync servers that should be "highly available"
according to the RFC. If it were unclear before: the actual problem is, as
Ties already pointed out, that rsyncd cannot handle many simultaneous
connections - high link speeds are not necessary (unlike most DDoS attacks).
I am pretty sure that attack would not work against most HTTP servers, let
alone large CDN providers.

> As such, any infrastructure that does not want to be (D)DoSsed should be
redundant, likely anycasted, and appropriately scaled as per the RFC. It is
also why many people opt for Hosted RPKI at the RIRs and why that setup will
cost them a dear cent.

I agree. Could you explain how you would do that, and how anycast would
protect against the DoS attack I described? What I described would already
take down all rsyncd for all RIRs. 

> Every piece of code in the world will have problems somewhere: please
report and people will fix. (Though as rpki.exposed shows, some of the
problems (J+K) you have reported are still not resolved in some clients,
things take time).

Reporting issues is in part what the CVD was for. Regarding J+K: they are
indeed unsolved, and to my knowledge all implementations fundamentally
suffer from this problem - the Tree Hints draft was written to address that.
I am not sure who maintains the spreadsheet from rpki.exposed, but I
severely question the validity of some of the "not vulnerable" statements.
For example, K seems to still be possible over rsync for rpki-client 7.5.

> I don't know about you, but filesystem quotas are things that have existed
for well over 4 decades by now. Thus that is really a non-issue.

Could you explain why it is a non-issue? It likely breaks all RPs currently
- that strikes me as an issue. RRDP can directly be written to object
storage to avoid this issue, and implementing a maximum total size/file
number limits is easier as well.

> Btw, for RRDP, you do not want to know how much fun I can have with
HTTP/1.0... let alone with HTTP/2 or QUIC... so many evil server
possibilities... the RFCs have comments on many of these techniques and the
clients and servers address many too; can't stop them all though, at one
point "that server is malicious" and you need to disconnect.

Actually, I do - please share.

Koen van Hove

-----Oorspronkelijk bericht-----
Van: Jeroen Massar <[email-masked]> 
Verzonden: woensdag 1 december 2021 10:59
Aan: Koen van Hove <[email-masked]>
CC: SIDR Operations WG <[email-masked]>
Onderwerp: Re: [Sidrops] I-D Action: draft-ietf-sidrops-prefer-rrdp-01.txt

TLDR: Fix problems, instead of putting everything into HTTP.... the Internet
is not just the Web

Hi Koen,

> I decided to test your hypothesis, as RFC 6480 says otherwise, and 
> created a small program that aims to DoS an rsync daemon using many 
> simultaneous connections from different IP addresses (but the same 
> machine). My conclusion is the same as yours - with the old computer 
> (4 cores, 16 GB of RAM, home fiber connection) underneath my desk I 
> can currently take down my own test rsync server (8 cores, 16 GB RAM, 
> 1 Gbps connection, different network).

8 cores? 16 GB? My phone has better CPU/mem and connectivity than that...
that is literally puny.

Heck, your university of utwente has way more connectivity than that, and
they where DoSed easily off-net too:

And as we have a LOT of people on the Internet and many that do not properly
maintain their connected devices (IoT but also for instance Wordpress, or
ManageEngine ADSelfService or soooo many problems), botnets in the order of
100.000's times 100mbit are common today and you go to any "Booter" or
"Stresser" service and you get 50Gbit/s for a few dollars.
But, heck for 666CHF/year I can even have 25GBit/s at home from init7.net as
a single link...

Of course, it would require you to sustain that DoS for a week, possibly
two, to actually do any real damage. But centralisation is bad (even though
one always have to go through the RIR) thus many orgs will want to host
themselves (also makes some processes easier). [and the init7 example they
will have complained and kick you off net by then ;) ]

As such, any infrastructure that does not want to be (D)DoSsed should be
redundant, likely anycasted, and appropriately scaled as per the RFC.

It is also why many people opt for Hosted RPKI at the RIRs and why that
setup will cost them a dear cent.

Fortunately both rsync and RRDP require TCP connections. Most UDP can be
filtered at the border, that helps quite a bit.

But then as noted a 100k botnet can just do legit connections.... for a bit,
as ISPs do band together and contact each other and try to resolve at least
swaths of those botnets.
Hence... the longer the DDoS, the more annoyance, more fixes (or blocks)
will be applied and at least the size of the botnet and the attack reduces.

I personally also advise doing the rpki-client part from disjunct networks,
heck do it from random ASNs, just in case an operator wants to specifically
target a specific ASN.

> Discussions with operators from RIRs seem to show that this would also 
> work for their rsync servers.  I believe this would even allow one to 
> take down all rsync servers currently running in the RPKI with a 
> handful of computers, but I have not tested that as I do not want to 
> disrupt actual operations.

Just in case you did not know yet: Please realize that it would be illegal
to do it under the Dutch law you fall under (and other countries have
bilateral agreements and will find you too).
I bet there is a law professor at your university that can explain to you if
it is criminal or civil law that it will fall under and what the penalties
would be.

[..]
> Therefore, there are currently simple things everyone could do that 
> would disrupt a relying party over rsync, which are difficult to solve:
> 1. One can set a motd file that is several gigabytes in size. Rsync 
> will wait until this is received (the --no-motd switch only hides it 
> on the
> client) and the timeout set in rsync does not affect the maximum time 
> this may take.

Every piece of code in the world will have problems somewhere: please report
and people will fix.
(Though as rpki.exposed shows, some of the problems (J+K) you have reported
are still not resolved in some clients, things take time).

Also remember the original motto of the IETF: running code and rough
consensus...

The running code so that one has experience in operating something, as one
learns a lot from it.

Easy fix: if a server sends an overlong MOTD (eg >1MiB can already be
considered that) put it on a blacklist and ignore it.
The only resources DDoSed that way are their own. You'll have a 'delay'
once, but after that your client is happy.

> 2. One can create an rsync repository with many folders (I tested it 
> with 27 million folders). This fills up the maximum amount of inodes 
> on most Linux and *BSD filesystems within minutes whilst bypassing any 
> size limits or --includes and --excludes, as a folder has a size of 0, 
> and folders are used in non-malicious repositories.

I don't know about you, but filesystem quotas are things that have existed
for well over 4 decades by now.

Simpler version: have a filesystem specifically where you retrieve things.
Cooler version: have a filesystem per CA one fetches -- could use loopback
FS or LVM-alike.

Heck, the latter one could even nicely have versioning on the filesystem and
lots of other cool features allowing rollback etc etc with minimal overhead
in storage.

8<-----------------
--prune-empty-dirs, -m

Indeed, you are right, a malicious operator will be able to cause some kind
of problems.
Blacklisting that operator is very easy and would only affect the resources
for that operator.

And all other things, when discovered, report them, patch them and keep on
running.

It is extremely easy to be on the "red team", and so many people think they
are cool playing attacker. It is a lot more fun to play the blue team and
defend against things. But as it is a game, both sides will change tactics,
red will find something, blue will fix it... that is the way of life on the
Internet.

Btw, for RRDP, you do not want to know how much fun I can have with
HTTP/1.0... let alone with HTTP/2 or QUIC... so many evil server
possibilities... the RFCs have comments on many of these techniques and the
clients and servers address many too; can't stop them all though, at one
point "that server is malicious" and you need to disconnect.

Greets,
 Jeroen

---

From: Job Snijders <[email-masked]>
Date: Fri, 3 Dec 2021 08:48:29 +0100

> severely question the validity of some of the "not vulnerable"
> statements. For example, K seems to still be possible over rsync for
> rpki-client 7.5.

I suspect the 'not vulnerable' fields are filled based on test results
obtained using the testbed at https://rpki.koenvh.nl/

control run: 144665 lines of validated output under RIPE TA
             https://sobornost.net/~job/control.txt 

RIPE TA + J: 144671 lines of output (within error margin)
             https://sobornost.net/~job/j1.txt

RIPE TA + K: 144665 lines of output (within error margin)
             https://sobornost.net/~job/k1.txt

The testbed does not offer rsync access, so your claim about rsync
cannot be verified. If you are creating variants of those tests, it
would be helpful if you use distinct labels (K.1, K.2, etc) for each
variant, and clearly document each permutation.

Another consideration would be to make your testbed and source code
publicly available so that others can replicate/confirm/inspect whatever
claims are made.

Regards,

Job

---

From: Claudio Jeker <[email-masked]>
Date: Fri, 3 Dec 2021 10:52:26 +0100

There is no difference between rsync or RRDP. Both protocols use a
filesize limit in rpki-client so if the file makes it to disk then it is
below the maximum allowed file size.  The .roa file in K parses just fine
in rpki-client 7.5:

rpki-client -nrv -t koenvh-K.tal
rpki-client: ta/koenvh-K: using cache
rpki-client: https://k--XXXXXX-YYYY.rpki.koenvh.nl/notification.xml: using cache
rpki-client: rrdp/117fe1e3bacf74a79db3d1d015378bb38430d491dbc3c9f50df8aba0cb666cf2/k--XXXXXX-YYYY.rpki.koenvh.nl/repository/koenvh.crl: File too large
rpki-client: all files parsed: generating output
rpki-client: Processing time 1 seconds (0 seconds user, 0 seconds system)
rpki-client: Route Origin Authorizations: 1 (0 failed parse, 0 invalid)
rpki-client: BGPsec Router Certificates: 0
rpki-client: Certificates: 1 (0 invalid)
rpki-client: Trust Anchor Locators: 1 (0 invalid)
rpki-client: Manifests: 1 (0 failed parse, 0 stale)
rpki-client: Certificate revocation lists: 1
rpki-client: Ghostbuster records: 0
rpki-client: Repositories: 2
rpki-client: Cleanup: removed 0 files, 0 directories
rpki-client: VRP Entries: 262145 (262145 unique)
        1.36 real         1.04 user         0.30 sys

It took a total of 1.36 seconds to process this .roa file with 262145 VRP
entries. So please tell me how this result is considered "vulnerable" for
you?

-- 
:wq Claudio

---

From: Job Snijders <[email-masked]>
Date: Fri, 3 Dec 2021 12:13:34 +0100

On Fri, Dec 03, 2021 at 02:35:51AM +0100, Koen van Hove wrote:
> Could you explain why it is a non-issue? It likely breaks all RPs
> currently - that strikes me as an issue.

Is it really that likely? I have yet to see it happen.

> RRDP can directly be written to object storage to avoid this issue,
> and implementing a maximum total size/file number limits is easier as
> well.

Earlier on I mentioned 'rsync-os':
https://mailarchive.ietf.org/arch/msg/sidrops/RQTi6TbyRQtYpfaigA5OA0JJNvg/

github.com/kaiakz/rsync-os
"A port of rsync written in Go, with object storage support"
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^

The concept of 'object storage' doesn't seem RRDP-specific.

Regards,

Job

---

From: Mikhail Puzanov <[email-masked]>
Date: Fri, 3 Dec 2021 12:48:59 +0100

> You can transfer empty files instead.

You can set --min-size ;)

Which for RPKI objects should be more than 100 bytes (vague hand-waving).
Actually, it would be nice to have a consensus about the minimal object 
size, we can have a good number here contrary to the max size.

So, lets say, we have timeboxed "rsync --max-size 32Mb --min-size 100 --prune-empty-dirs”.
How bad it can still be?

--
Misha

---

From: Ties de Kock <[email-masked]>
Date: Fri, 3 Dec 2021 15:09:06 +0100

> On 3 Dec 2021, at 12:48, Mikhail Puzanov <[email-masked]> wrote:
> 
> 
>>> 

Superficially this may look like an improvement, but in reality, it does not not
help against a realistic attacker. For any reasonable lower limit on object size,
many filesystems will have much more free space than free inodes.

For example, what I see on a host with small disks:
[user@host ~]$ df -i
Filesystem                                                         Inodes   IUsed    IFree IUse% Mounted on
...
/dev/mapper/vg01-lv_root                                          5242880   66960  5175920    2% /
[user@host ~]$ df -h
...
/dev/mapper/vg01-lv_root                                           10G  7.3G  2.7G  74% /

2.7GB of data until the disk is full, but if you send 100b files the disk is full
much sooner (remember that the inodes themselves will use a non-trivial amount
of space). You can also exhaust both space and size efficiently: transfer a
collection of sparse files and rsync transfers if efficiently and materialises
it non-sparsely for the receiver.

And for many RPs, the server could just transfer symlinks instead.

> size, we can have a good number here contrary to the max size.

Can we? Do we know what future object the RPKI will contain?

> So, lets say, we have timeboxed "rsync --max-size 32Mb --min-size 100 --prune-empty-dirs”.

Just as bad.

I'll reiterate my point from earlier. An rsync client that does not:
  * abort after a set number of objects received, or
  * abort after a set amount of bytes transferred by the connection, or
  * get killed by the parent process after a certain time

It is inherently vulnerable to resource exhaustion attacks when reading from an
untrusted source.

Kind regards,
Ties

---

From: Job Snijders <[email-masked]>
Date: Fri, 3 Dec 2021 15:22:09 +0100

>   * get killed by the parent process after a certain time

Ties - Mikhail says "timeboxed",  you say "get killed after X time". 

I think you are saying the same thing?

OpenBSD rpki-client kills syncing after 15 minutes (or 1/4th of the
timeout specified via '-s'), what do other RPs do?

Please leave a note at https://trac.ietf.org/trac/sidrops/wiki/FencingLimits

Kind regards,

Job

---

From: "Koen van Hove" <[email-masked]>
Date: Sat, 4 Dec 2021 03:25:31 +0100

Dear Job, all,

The idea of K is that one can create a lot of entries in the ROV table by
splitting, for example, a /48 into 2^80 /128s (or for an actual attack, a
random subset), and subsequently overload it. To my knowledge, the only
hindrances so far posed by implementations are a maximum file size, a
maximum size of an RRDP repository, a maximum number of entries on a
manifest, a maximum number of repositories, and a maximum number of prefixes
per file. K hit, if I recall correctly, the maximum file size limits and the
maximum repository size limits. The first is simply solved by splitting a
ROA into multiple ROAs, the latter is easily solved by using rsync does not
have a maximum repository size limitation, and a maximum files per MFT is
easily bypassed by using multiple CAs in the same repository. I just tried
this out by decreasing the number of prefixes in one ROA to 2^17, but
creating 2^8 different ROAs. This works, and I do not believe I was nearing
any limits yet. For context, this is roughly two orders of magnitudes larger
than all current ROV entries from the five RIRs combined.

The testbed does support rsync for the tests that do not require RRDP to
function (i.e., tests that do not rely on XML vulnerabilities and the
likes). Just like for RRDP, the rsync repositories are generated on-demand
at the first request, thus your initial rsync request may take some time.
(This is, by the way, the entire reason the testbed requires IPv6 to use, as
rsync does not support SNI.)

However, my point is not so much whether this specific test succeeds or
fails. The question I asked myself was: "Can an attacker disrupt RPKI RP
software by introducing a malicious CA/repository into the tree?". The
answer to that question currently is: yes. This threat of many entries is
made easier by rsync, hence I support moving away from rsync towards RRDP. 

Koen van Hove

PS: Making it open source is something that is under consideration, but it
is not a decision I can make on my own.

-----Oorspronkelijk bericht-----
Van: Job Snijders <[email-masked]> 
Verzonden: vrijdag 3 december 2021 08:48
Aan: Koen van Hove <[email-masked]>
CC: 'Jeroen Massar' <[email-masked]>; 'SIDR Operations WG'
<[email-masked]>

The testbed does not offer rsync access, so your claim about rsync cannot be
verified. If you are creating variants of those tests, it would be helpful
if you use distinct labels (K.1, K.2, etc) for each variant, and clearly
document each permutation.

Another consideration would be to make your testbed and source code publicly
available so that others can replicate/confirm/inspect whatever claims are
made.

Regards,

Job

---

From: "Koen van Hove" <[email-masked]>
Date: Sat, 4 Dec 2021 03:32:06 +0100

Dear Job, all,

BGP hijacks were once not likely - until the first one happened. Similarly,
attempts to sabotage the RPKI are not likely. We can wait until it happens,
or proactively resolve the issue to prevent it from happening in the first
place. The latter has my preference.

Koen van Hove

-----Oorspronkelijk bericht-----
Van: Job Snijders <[email-masked]> 
Verzonden: vrijdag 3 december 2021 12:14
Aan: Koen van Hove <[email-masked]>
CC: 'Jeroen Massar' <[email-masked]>; 'SIDR Operations WG'
<[email-masked]>

Regards,

Job

---

From: Mikhail Puzanov <[email-masked]>
Date: Mon, 6 Dec 2021 20:00:41 +0100

> And for many RPs, the server could just transfer symlinks instead.

True. And apparently there’s no flag to avoid it. 

> Can we? Do we know what future object the RPKI will contain?

I think we can and probably should since there is an effort to define all sort of common 
limits for RPs. As a quick estimate example -- every certificate has, at least, one RSA key 
of 256 bytes, a couple of dates - 24 bytes, SKI + AKI - 46(I guess?) bytes, serial, other 
3 extensions, distribution points and ASN.1 machinery, which at the very least is 350 bytes 
or so. The same calculation can be done for signed objects, all of them have an EE and 
a SHA256 somewhere, so probably these 350 byte would also be ok.

All in all, I would suggest `--min-size 300` for RPs to use consistently.

>  * get killed by the parent process after a certain time

I have a hunch that “get killed by the parent process after a certain time” may 
cover most of the cases. Or it may not but it's pretty hard to say without a 
realistic test.

We also need to keep in mind that almost any rsync-based attack on RPs would require
proportional amount of resources on the attacker's side -- feeding every single RP
in the world with zillions of tiny files is quite a drag, so it may happen that an 
attacker would DDoS themselves before causing any serious damage to anyone.

Koen did a very convincing job on RRDP-related vulnerabilities, so probably it would 
make sense to have something the same systematic for rsync, similar to the test 
with directories.

Having said that, I entirely support the initial direction of the draft and would 
want rsync to become optional/deprecated, but first and foremost because rsync is 
a headache for repository operators (RIPE NCC, specifically), rather less so (from 
what I can see now) for RP users. It is way too easy to bring practically any rsync 
repository to its knees, the only reason it has not happen yet is that nobody tried.

--
Misha

---

From: Claudio Jeker <[email-masked]>
Date: Tue, 7 Dec 2021 09:18:36 +0100

> True. And apparently there’s no flag to avoid it. 

Just don't add -l or -a to your rsync command. Symlinks are skipped by
default.

> All in all, I would suggest `--min-size 300` for RPs to use consistently.

I doubt that --min-size will result in RPs not being vulnerable.
300 bytes is smaller than most disk block sizes and with that you can
actually run a FS out of space as well.

> repository to its knees, the only reason it has not happen yet is that nobody tried.

There are some much larger rsync repositories out there then the one from
RIPE NCC. I think the main issue with rsync is that there is only one real
implementation of the server and there is no official specification of the
protocol.

rsync has a few benefits over RRDP so I would be very careful about
removing rsync support without fixing or adding an RRDP replacement.

-- 
:wq Claudio

---

From: George Michaelson <[email-masked]>
Date: Tue, 7 Dec 2021 18:29:33 +1000

> implementation of the server and there is no official specification of the

Aside from the 5 RIR who do not directly produce roa, although they do
publish roa for hosted services in distinct paths what are these much
larger repositories? Do you mean APNIC or another RIR? Large to object size
or large to count or ... ?

> removing rsync support without fixing or adding an RRDP replacement.

It pays to be explicit. What are these benefits? The only one I have seen
discussed is the cost shifting to compute the object tree from deltas
(shifted cost to client side in RRDP) vs the complete structure in rsync at
a cost of per fetch difference calculation, server side.

There are also disadvantages. Rsync is inherently serialised. Http has at
least implicit parallelism in its transport protocol (HTTPS). And is CDN
friendly. Rsync CDN support is tcp anycast and that's about it. And the
ones you stated (lack of implementations, lack of ietf standard)

Cheers

G

---

From: Job Snijders <[email-masked]>
Date: Tue, 7 Dec 2021 10:30:39 +0100

On Tue, Dec 07, 2021 at 06:29:33PM +1000, George Michaelson wrote:
> larger repositories? Do you mean APNIC or another RIR? Large to object
> size or large to count or ... ?

One example that comes to mind is: rsync://mirror.leaseweb.com/ (25T,
millions of files, anycasted)

When an RSYNC repository is DoSsed, perhaps it is unreachable from some
parts of the world for a brief period of time; but when the RSYNC
operator decides to no longer do their job (by permanently turning off
the RSYNC server); the rsync service goes offline, completing the DoS.

> It pays to be explicit. What are these benefits? The only one I have
> seen discussed is the cost shifting to compute the object tree from
> deltas (shifted cost to client side in RRDP) vs the complete structure
> in rsync at a cost of per fetch difference calculation, server side.

0/ Via rsync all Signed Objects (and all other x509v3 extension
   references to related objects) are available as distinct downloadable
   resources. This is a wealth of data for debuggers: 'CA Issuers',
   'Signed Object', CRLDP, etc.

1/ An RP has to replay the enitre RRDP session to figure out if they are
   desynchronized. RRDP doesn't offer an index table (other than
   downloading the entire snapshot).

2/ The RRDP <withdraw/> element is unauthenticated and unsigned.

3/ RRDP clearly was a hackjob: all URIs in the XML are rsync:// URIs.

4/ The RRDP spec is awfully light on how rsync/rrdp should co-exist.

5/ RRDP did not need XML as wrapper, should've/could've been DER.

6/ The B in base64 stands for Bloat.

7/ RRDP has a hard dependency on HTTP Keep-Alive: without Keep-Alive you
   might fetch a Notification file from server A, and attempt to fetch
   the delta or snapshot from B. Rsync is a single TCP stream, less
   sensitive to repository operator mistakes in this regard.

> ones you stated (lack of implementations, lack of ietf standard)

I find it interesting how CDNs are SIDROPS' deus ex-machina :-)

Why is the 'lack of implementations' meme being repeated, even though by
now (anno 2021) more rsync implementations seem to exist than RRDP
implementations? Times have changed.

Kind regards,

Job

---

From: George Michaelson <[email-masked]>
Date: Tue, 7 Dec 2021 19:45:15 +1000

> millions of files, anycasted)

So not an RPKI repository. Context is everything.

> the RSYNC server); the rsync service goes offline, completing the DoS.

I don't understand the moral force of this sentence in the context of RPKI.

>    'Signed Object', CRLDP, etc.

OK. A good point. RRDP could do with stable pointers to objects, noting the
D has meaning, but the point stands. I think in an http only world the URI
below would stand, no?

>    downloading the entire snapshot).

Seems to me it's fixable. Rsync also does not offer an index that zsync
would (IIRC) and incurs index and size and checksum passes server side.
Byron and I explored this in our "attack on rsync" presentation to ietf. A
hand rolled server can do a lot with in memory state and also walk over the
per file checks and just send the stream. There's a lot of optimisation
potential in rsync even though I dislike it.

> 2/ The RRDP <withdraw/> element is unauthenticated and unsigned.

Likewise.

> 3/ RRDP clearly was a hackjob: all URIs in the XML are rsync:// URIs.

This is a point of contention. Personally I would have preferred to remove
this obligation which dates from prior dependency on the URI. I think this
needs to change but I was taken as a co author to "not now"

I really don't think hackjob is helpful Job.

What relevance does this have to relative merits? Not that it's untrue but
it's a non sequester.

> 5/ RRDP did not need XML as wrapper, should've/could've been DER.

You want to do raw DER over https? Is there any precedent for that? I agree
in principle because..

> 6/ The B in base64 stands for Bloat.

This is of course true. The encoding is a cost. But it's also well
understood.

>    sensitive to repository operator mistakes in this regard.

Cache coherency in the local node. Hmm. This is something I hadn't thought
about. Thank you.  That's informative.

>
> > There are also disadvantages. Rsync is inherently serialised. Http has
> atDE/

I repeated a mem the OP responded to. Ask of the OP.

> Kind regards,

Kind, Hackjob aside :-) the keep alive thing is well worth noting.

Cheers

George

---

From: Tim Bruijnzeels <[email-masked]>
Date: Tue, 7 Dec 2021 10:51:22 +0100

> On 7 Dec 2021, at 10:30, Job Snijders <job=[email-masked]> wrote:
> 

Let me remind you that the proposal is to prefer RRDP, but leave rsync
in place as a secondary option. So you can still download files for
debugging.

Of course all files are also b64 encoded in the snapshot. I understand
that is more work - but it's not impossible to get. Also RPs can save
the decoded files and many do.

>   downloading the entire snapshot).

This is why there is a session and serial. If you have reason to
believe you are out of sync then get a new snapshot.

You are free to disagree with the choice, but the objective was
to shift the load onto RPs where it can scale in paralel, rather
than leaving it to the server.

> 2/ The RRDP <withdraw/> element is unauthenticated and unsigned.

Indeed, so is rsync's --delete.

> 4/ The RRDP spec is awfully light on how rsync/rrdp should co-exist.

The idea at the time was, and in prefer-rrdp is, that the protocols
can co-exist and the rsync URIs are used as identifiers.

> 6/ The B in base64 stands for Bloat.

You are welcome to disagree now, but at the time it was decided that
rather than inventing a new encoding RRDP would leverage existing
code and definitions from the publication protocol (RFC8181).

>   sensitive to repository operator mistakes in this regard.

Server implementation dependent..

Many CDNs will fetch from source on cache misses. In a simple setup with
multiple local http proxies one can implement the same. If local copies
are used then one can sync the snapshot and deltas first in synchrony,
and add a small delay before making notification files available.

---

From: Claudio Jeker <[email-masked]>
Date: Tue, 7 Dec 2021 11:36:05 +0100

> or large to count or ... ?
 
rsync is still used by various software distributions to sync large
repositories. For example the OpenBSD CVS repository is made available via
multiple rsync servers and at 7.7GB and 500'000 files it is a fair amount
larger than any RIR repo.

I noticed that many RIR repos like to add unnecessary load on their
servers by changing file timestamps in their repositories. This breaks
rsync fast path.
 
> a cost of per fetch difference calculation, server side.

RRDP lacks integrity checks. If a file is for whatever resason missing on
the RP then it will not be refetched for a very long time. Rsync has the
benefit that missing files are refetched automatically. After a rsync run
your repository is the same as the on the remote side and any local
modification is detected and fixed. RRDP has nothing like that. It only
carries a hash for delete and update operations but there is nothing that
ensures that other files are actually what their supposed to be.
So every file served via RRDP needs to be kept on the RP system even if it
has nothing todo with the RPKI.
 
> ones you stated (lack of implementations, lack of ietf standard)

RRDP is inherently serialised. You can not parallel process it. You can
run many rsync servers on different IPs. There is no need for tcp anycast.
I could run rsync as a CDN but there is probably not that much interest in
doing that from the big CDN players.

-- 
:wq Claudio

---

From: George Michaelson <[email-masked]>
Date: Tue, 7 Dec 2021 20:58:03 +1000

> larger than any RIR repo.

So as job pointed out this is a generality about the applicability of rsync
to massive filestore synchronisation which is what Tridge designed it for.

> rsync fast path.

To be clear, you must mean when there's been no change to filestate because
reissuance of files demands that be resigned and retimestamped.

Because of partial update problems a lot of people server side are now
doing republication by the near-atomic operation on the root dir of the
repository so that its an integral collection. Because of validation
failure states for partially updated state. It's possible the mechanistic
cheap ways of doing this accidentally change file and dir timestamps, worth
noting. We could try harder to avoid this.

> has nothing todo with the RPKI.

I think you're conflating issues here. RRDP is a delta protocol but
includes pointers, URI to every object. All rpki objects being signed I
don't understand the point about files not being what they are supposed to
be. If you mean and adversarial attack, remember the crypto is there.

But it is probably fair that neither rsync nor rrdp were designed for bad
actor risks. I think people are driving hard to attack RRDP but remember
we're here because of the trivially low bar to attacking rsync.

> run many rsync servers on different IPs. There is no need for tcp anycast.

I think we're talking at cross purposes. Per rsync connection there is no
internal threading it's a single linear sequence of fetches. If you run two
rsyncs in parallel between some A and B you stomp on each others fetches.

But in principle (if not in practice) you can thread and multifetch the
stream of an RRDP. In any case snapshot aside you fetch less and you do not
do per file checking, the delta sync is a smaller total fetch size.

I don't think anyone has coded RRDP to exploit parallelism. So this is only
in hypothesis. But, unless you move to zsync or other modern takes on rsync
this is pretty hard to do.

> doing that from the big CDN players.

As an RIR employee I can only agree. There is no first class support for
rsync as a clouded service. We asked. You have to build your own. You can
now embed rsync into services which do closest TCP provider mapping, but
there's no meaningful edge cache. APNICs AS0 is in cloudflare and there is
a significant RTT advantage in second fetcher because of the caching, which
came along for free in http cloud services.

Cheers

G

---

From: Alex Band <[email-masked]>
Date: Tue, 7 Dec 2021 12:51:50 +0100

An aspect that appears to be overlooked is the operational experience a typical ISP or cloud provider will have with running a highly available rsync service vs an HTTP service. While it may be technically possible to offer rsync at scale, the barrier to having an IT department commit to operating a globally load balanced HTTP service is much lower than doing the same for rsync, purely from an experience standpoint. 

I know for a fact that the rsync requirement has been the reason for several operators to abandon their plans for running Delegated RPKI. 

Be that as it may, with the prospect of more RIRs offering RFC8181 publication services this barrier is likely taken away. It’ll be interesting to see if the ~10 Delegated RPKI instances in the ARIN region now will start rising to the likes of the ~1000 in Brazil at the moment.

-Alex

> On 7 Dec 2021, at 11:36, Claudio Jeker <[email-masked]> wrote:
> 
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidrops

---

From: Claudio Jeker <[email-masked]>
Date: Tue, 7 Dec 2021 12:57:08 +0100

> noting. We could try harder to avoid this.

I just noticed that rsync refetches the hole repository more often than it
has to. This is because timestamps change but the file content is still
the same. So this adds extra load to the rsync servers.
 
> be. If you mean and adversarial attack, remember the crypto is there.

I can send you a delta that holds files that are not in the snapshot. You
will store those files in your cache until you fetch a new snapshot. This
way the disk on RP systems can be filled with very little effort from a
publication point.

RRDP is a weak delta protocol there is no way to know if all files are
present because there is no index which can be compared. In this regard
the consistency of the database is just presumed.
 
> we're here because of the trivially low bar to attacking rsync.
 
The problem here is that RPKI is setup in a way that makes it easy to
DoS RP software. It does not matter if RRDP or rsync are used.
 
> rsyncs in parallel between some A and B you stomp on each others fetches.

How is this different to a HTTPS connection? There is not internal
threading there either. If you run two RRDP processes in parallel you also
stomp on each other fetches.
 
> this is pretty hard to do.
 
You must apply all RRDP deltas in order. You can not apply them in
parallel or you may end up with a corrupted repository. In this regard I
think RRDP and rsync are behaving the same way.

>From a security standpoint neither RRDP nor rsync are good. In RPKI all
risks are on the RP side and very little on the side of publication points.
This is a fundamental issue. 

-- 
:wq Claudio