Subject: Re: mboned: multicast attack using SAP group

From: Hank Nussbacher <[email-masked]>
Date: Fri, 13 Jan 2006 12:37:41 +0200

We too got hit on Wed and didn't know what it was either, but guessed it
was multicast related.  Our RP router (running bleeding-edge 12.4(4)T1)
logged the following:

Jan 11 13:38:41 IST: %MROUTE-3-RECUR_ROUTE_DEPTH_ERR: Recursive route
lookup has reached its max depth for: 128.139.220.91
-Process= "PIM Process", ipl= 0, pid= 211
Jan 11 13:38:41 IST: %MROUTE-4-RPF_LOOKUP_LOOP: RPF route lookup loop
for 128.139.13.237, route 128.139.0.0/16
Jan 11 13:39:45 IST: %MROUTE-3-RECUR_ROUTE_DEPTH_ERR: Recursive route
lookup has reached its max depth for: 128.139.220.91
-Process= "PIM Process", ipl= 0, pid= 211
Jan 11 13:39:45 IST: %MROUTE-4-RPF_LOOKUP_LOOP: RPF route lookup loop
for 128.139.13.237, route 128.139.0.0/16
Jan 11 13:39:54 IST: %MSDP-5-PEER_UPDOWN: Session to peer 62.40.102.51
going down
Jan 11 13:39:58 IST: %OSPF-5-ADJCHG: Process 378, Nbr 128.139.220.90 on
GigabitEthernet0/1 from LOADING to FULL, Loading Done
Jan 11 13:40:25 IST: %MSDP-5-PEER_UPDOWN: Session to peer 62.40.102.51
going up

Note that our main router:
http://noc.ilan.net.il/stats/ILAN-CPU/tel-aviv-gp-cpu.html
is a 7600 with a WS-SUP720-3BXL. It is no easy task to push it to close
to 93% CPU.

It rippled through our network and caused our international traffic to
plummet:
http://noc.ilan.net.il/stats/TEL-AVIV-GP/stm1up-gp1.ilan.net.il.html

-Hank
AS378

_______________________________________________________________
user interface: http://darkwing.uoregon.edu/~llynch/mboned.html
web archive:  http://darkwing.uoregon.edu/~llynch/mboned/

---

From: Simon Leinen <[email-masked]>
Date: Sun, 22 Jan 2006 12:24:36 +0100

[I sent this on Sunday, but somehow the message hasn't made it to the
 list, so I'm resending.  Sorry if anybody receives this twice.]

Thanks for sharing your observations! Here's another (late) data point
on the January 11 SAP packet flood.  We were also hit by this, because
we had a few routers that had "ip sdr listen" configured on an
interface - we no longer do.  I also heard from a few other research
networks that they had problems.  It would be interesting to look at
route-views or RIPE RIS data from that day to find out how much BGP
instability was induced by this.

Somehow I don't think this was an attack in the sense of being
deliberate.  The source, sgemaster.csp.it [194.116.6.50], seems to be
a legitimate source of SAP announcements in normal times, at a rate of
slightly less than 1pps.  "Never attribute to malice..."
(But of course "multicast attack" sounds much more exciting. :-)

Normalized to UTC, here are the flows that we (AS559) captured on our
border router to GEANT2, our multicast upstream in this case.  All
flows are UDP flows from 194.116.6.50 to the SAP address
(224.2.127.254 port 9875).  The flows were collected using NFDUMP
(http://nfdump.sourceforge.net/).

flow start  flow end          pps      bps  packets    bytes  src port
[...]                                                         
11:26:43.31-11:38:02.01      0.97  2208.12      655   187330  34571    
11:38:04.48-11:40:41.58  15277.28  34.87 M    2.4 M  684.8 M  34585  
11:41:45.62-11:44:41.18  15948.60  35.90 M    2.8 M  787.9 M  34585  
11:46:01.44-11:47:35.81  16954.18  37.74 M    1.6 M  445.2 M  34585  
11:49:48.00-11:51:09.78  15895.53  36.03 M    1.3 M  368.3 M  34585  
13:19:33.77-13:34:33.74      0.98  2239.76      881   251966  34660    
[...]

Until 11:38:02, there were normal announcements at ~1pps.

At 11:38:04, the source port changes and the rate changes
dramatically.  The flood goes on for about twelve minutes, but with
three intermittent 1-2 minute periods of silence.  These periods of
silence might have been due to breakage in our own network, or to
instabilities further upstream.  The high-rate source falls silent at
11:51:09.  At 13:34:33 (after lunch break; this is 14:34 in Italy :-)
it resumes sending normally at 1pps.

The fact that the high-rate flows came from a different source port
could indicate that the sending program was restarted with different
parameters or with a code change.

Your timestamps may be different.  Although the source is physically
close to us (about 300km driving distance :-), their packets travel
through Shep's farm^H^H^H^Hrouter complex on the US West coast and
then back to us through Abilene and GEANT/GEANT2.  I'm quite impressed
that the upper part of the path (probably a GRE tunnel between CSP and
Shepfarm) supports 37 Mbps of traffic (at least for a few minutes :-).

Mtrace from 194.116.6.50 to 62.40.124.22 via group 224.2.127.254
>From source (sgemaster.csp.it) to destination (swice2-10ge-1-1.switch.ch)
Querying full reverse path... * switching to hop-by-hop:
 0  swice2-10ge-1-1.switch.ch (62.40.124.22)
-1  swice2-10ge-1-1.switch.ch (62.40.124.22) [AS 20965] PIM/MBGP Reached RP/Core [194.116.0.0/18]
-2  switch.rt1.gen.ch.geant2.net (62.40.124.21) [AS 20965] PIM  [194.116.0.0/18]
-3  so-7-2-0.rt1.fra.de.geant2.net (62.40.112.22) [AS 20965] PIM  [194.116.0.0/18]
-4  abilene-gw.rt1.fra.de.geant2.net (62.40.125.6) [AS 20965] PIM  [194.116.0.0/18]
-5  atlang-washng.abilene.ucaid.edu (198.32.8.65) [AS 11537] PIM  [194.116.0.0/18]
-6  hstnng-atlang.abilene.ucaid.edu (198.32.8.33) [AS 11537] PIM  [194.116.0.0/18]
-7  losang-hstnng.abilene.ucaid.edu (198.32.8.21) [AS 11537] PIM  [194.116.0.0/18]
-8  snvang-losang.abilene.ucaid.edu (198.32.8.95) [AS 11537] PIM  [194.116.0.0/18]
-9  198.9.201.180 [AS 24] PIM  [194.116.0.0/18]
-10 topix-paix.shepfarm.com (198.58.3.221) [AS 3948] PIM/MBGP Reached RP/Core [194.116.0.0/18]
-11 194.116.97.26 [AS 25309] PIM/Static Reached RP/Core [194.116.6.0/24]
-12 * * * 
-- 
Simon.