Subject: Re: [Idr] draft-scholl-idr-advisory

From: Rob Shakir <[email-masked]>
Date: Wed, 1 Jul 2009 09:19:28 +0100

On Tue, Jun 30, 2009 at 01:58:14PM -0400, Ron Bonica wrote:
> Folks,
> 
> Last month, Tom Scholl presented draft-scholl-idr-advisory to NANOG's
> general meeting in Philadelphia. The operator community demonstrated
> strong support for the draft.

I'd like to add my support for this draft.

In addition, I presented at the last LINX meeting relating to the the problems
observed with AS4_PATH earlier in the year, mentioning this draft.The feedback
that I got from the operators that I spoke to at the meeting was overwhelmingly
positive towards both draft-ietf-idr-optional-transitive and Tom's draft.

Kind regards,
Rob

-- 
Rob Shakir                      <[email-masked]>
Network Development Engineer    GX Networks/Vialtus Solutions
ddi: +44208 587 6077            mob: +44797 155 4098
pgp: 0xc07e6deb                 nic-hdl: RJS-RIPE

This email is subject to: http//www.vialtus.com/disclaimer.html

---

From: Joe Provo <[email-masked]>
Date: Tue, 30 Jun 2009 17:14:12 -0400

On Tue, Jun 30, 2009 at 07:41:48PM +0100, Andy Davidson wrote:
> 
> On 30 Jun 2009, at 18:58, Ron Bonica wrote:
> 
> >In light of strong operator support, I would like to encourage the  
> >WG to
> >consider adopting this draft as a WG item.
> 
> Ron, all --
> 
> I support the advisory proposal as is.
> 
> I was present when the straw poll was taken at nanog and agree that  
> the operator community show a huge volume of support for this proposal.

As stated last go-round, yes yes and yes. The sticky, et al ideas
make for goo -01 update fodder, but the concept is sound and as 
long as it is a) negotiated and b) any message-type lists have 
accomodations for free-text* then it will be strongly supported
by the operator community at large.

Cheers,

Joe

* most network operators are humble enough to realize we can't 
  possibly foresee all conditions needed in human interprovider
  signalling

-- 
             RSUC / GweepNet / Spunk / FnB / Usenix / SAGE

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 1 Jul 2009 10:57:31 -0400

> long as it is a) negotiated and b) any message-type lists have

+1 for a)

the overall concept would be helpful.

-Chris

> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
>

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 01 Jul 2009 23:48:18 -0400

In message <[email-masked]>
Ron Bonica writes:
> In light of strong operator support, I would like to encourage the WG to
>                                     Ron Bonica

Ron,

No objections, just a few questions.

Has NOC to NOC communication really degraded to the point where we
need to send messages through the BGP peering to reach someone with a
clue on the other side?  Didn't NOCs used to exchange PGP keys a long
while back and prioritize PGP signed email from peers?  Have we
forgotten how to do that or is there some reason that it is no longer
workable?  [perhaps clueless peers who can't get PGP into outlook?  I
doubt its because we're paralyzed over the PGP vs. S/MIME dilema.]

The peering has an IP address.  What about putting these IP addresses
in the email to the peer to identify where the problem is rather than
put the message in the session itself?

Would it be wise to explicitly limit the scope of these messages to
the immediate peer and its logging so that the global internet isn't
bloated with advisory messages that are only of interest to a pair of
providers somewhere?  I didn't see where that limit was recommended.

If this really is needed, its a sad commentary on the operator
community.  But if so, go for it.

Curtis

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 2 Jul 2009 00:16:05 -0400

> clue on the other side?  Didn't NOCs used to exchange PGP keys a long

I think that, and was poked in the eye by a former collegue about
this, this is an attempt to solve a 'social' problem with technology.
This isn't exactly right, because if this were  implemented well it
feeds into the NOC ticketing/alerting system in a manner that lets the
NOC auto-ignore a flapped session if the note came across prior to the
flap. Implemented well means from the router-vendor/user up to the OSS
that feeds alerts/alarms/triggers to the 'NOC'.

At larger ISP/NSP/Operators the current state is that there are 'lots
of customers' and basically no linkage between 'customer' (or 'peer'
in this context) and the 'NOC'. Getting through the phone-queue/hell
to someone who'd be able to tell the 'NOC' that "hey, customer X's
session is going to bounce ... yea, 2 hrs ago:(" isn't realistic.

'used to' share PGP keys, I've not seen anyone at a large 'ISP' use
PGP (hell most of their parent companies forbid it's use on corporate
equipment) in a very long time. For anything going to the 'NOC'
there's a standard method to get a ticket created/started, only
infrequently would email actually work out, and almost never would PGP
actually be involved in that process.

> forgotten how to do that or is there some reason that it is no longer

it's not about forgetting, it's about email stopped being useful for
'noc@' at any large ISP, at least  email from 'public' places.

> doubt its because we're paralyzed over the PGP vs. S/MIME dilema.]

Nope, it's because email's not used and you can't rely on email in a
timely fashion or reaching the 'right' place in the remote org. You
CAN say that in X period of time after message-via-bgp arrives on
routers you can automate log-scraping and alert de-priotitization in
your netcool-like system.

> put the message in the session itself?

Keep in mind that email is also not timely, If I need to shutdown a
peer 'now' and I don't want/need help I'd rather trigger a message/log
on their side that makes the event just not show up on their OSS/Alarm
system.

I'd also note that Tom does cover almost all of these reasons in his
summary/abstract...

> the immediate peer and its logging so that the global internet isn't

yes... though I thought Tom's idea was 'capability' which is a
peer-to-peer negotiated element, not something sent along beyond the
immediate peer, correct?

> community.  But if so, go for it.

thanks for your vote? I actually thought it was a GOOD sign for
operators, we've grown up and decided that some technologies could
help us to make things better/faster/smoother for all parties.

-Chris

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 2 Jul 2009 00:18:38 -0400

> 'used to' share PGP keys, I've not seen anyone at a large 'ISP' use

I'd note that when you left ANS/UUNET there was very likely almost no
one using PGP for email communications between peers (aside from
personal emails)... certainly the NOC ticketting system at UUNET
didn't understand PGP email, nor prioritizing that over other email it
received.

-Chris

---

From: Joe Provo <[email-masked]>
Date: Thu, 2 Jul 2009 06:31:43 -0400

> > consider adopting this draft as a WG item.
[snip]
> clue on the other side?

Yes.  Large companies apply factory-management to ops centers and just 
as with call centers this drives initiative and care out of the front 
lines, in the interest of ticket closure and call stats.  See also,
official communication channels, and game of telephone when trying to 
send reasonable noticies around simple things like prefix filters,
max-pfx adjutsments, etc.  

This smells like a stelar customer-facing tool as well, to standardize 
some message-passing WRT needing prefix filters adjusted, etc.  To 
look at the first item [sorry, first cuppa], today there are a mishmosh 
of ascii mail, word docs[ugh], web forms, IRR-based austomation and
'customer portals' that add friction to the provisioning process. Is
it no wonder that we-collectively have a problem filtering customers 
when a customer has to use a pile of different methods? I would see
this capability being used at all AS-external edges (peer, customer, 
and provider) and provide utility across each path.

>  Didn't NOCs used to exchange PGP keys a long
> while back and prioritize PGP signed email from peers?  
[snip]

Never in my ~13 years of doing a nontrivial amount of peering & ISP
operations.  

> put the message in the session itself?

This is already done, and is operationally insufficeint and inefficient.
Piggybacking messages on a known, good session gives a strong vouch for 
the veracity of any assertions as well; email or relayed phone messages
burn staff time of usually senior people to ferret out if that a human
contact change, prefix growth, etc is true.

> providers somewhere?  I didn't see where that limit was recommended.

Negotiated compatibility implies a limit to neighbors, but I would
imagine this to be clarified as non-transitive in the *-01. 

> community.  But if so, go for it.   

I think you mispelled "commoditization of Internet service".  Existing
methods fail to DTRT at scale, therefore seeking a new tool on the 
workbench is IMO a Good Thing.

Cheers,

Joe

---

From: "Gill, James E (gill)" <[email-masked]>
Date: Thu, 02 Jul 2009 13:59:35 +0000

Hello folks,

I support adopting this as a WG item.

However, and I'm sure we will get into this as the WG proceeds, I think
there are fundamental elements of this which are bad.  Notably, I don't
think that bloating BGP with operator-to-operator messages is a good
move to maintain a stable, secure (fwiw) protocol.

I agree that the problem of op-2-op communication needs more attention,
and I hope with this starting point the WG can forge a reasonable
solution that doesn't increase the fragility and risk of operating an
Internetwork.

--gill

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Ron Bonica
Sent: Tuesday, June 30, 2009 1:58 PM
To: [email-masked]
Subject: [Idr] draft-scholl-idr-advisory

Folks,

                                    Ron Bonica

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 02 Jul 2009 08:06:42 -0700

Hi Jim,

I very much share your point of view. I think the current version of the 
draft may be a good starting point for the discussion as apparently 
there is operator's community requirement for improvement in the space 
of inter-NOC communication.

I think automation is great and if we can automate anything I am always 
in support. If this regardless of comments already made in the other 
thread is to happen in BGP so be it.

Would we have a way to keep this new communication channel to BGP 
related messages - perhaps no.

Example: If EBGP peers agree to run BFD would we use the same new BGP 
msg type channel to pass BFD related warnings too ?

But in spite of couple of people's immediate voting to accept it "as is" 
or "exactly as written" which interestingly enough Ron's email did not 
even contained in the question I would like to state that "as is" or 
"exactly as written" may be a bit too strong of requirement.

Besides which version of the draft are we talking of "as is" ? Those in 
version -00 or those on Nanog slides including "good ideas" from pages 
11 & 12 ?

http://www.nanog.org/meetings/nanog46/presentations/Tuesday/Scholl_bgp_adv_light_N46.pdf

Example:

Concept of sticky messages
•An idea where you can transmit an advisory message to “stick” in the 
BGP neighbor state. This way, upon viewing “show bgp nei x.x.x.x”, you 
can see the latest sticky message transmitted by the peer.
•Would not be sent to the log.

Question: How can something with random length (without max limit) be 
part of cli show command and yet still make this show command nicely 
readable and parsable by scripts ?

So I would suggest both for sticky msg and syslog msg to have max length 
defined.

Also I think we should enforce the free form language to be ASCII 
English only.

/* And yes I can already see a request coming for a spell checker to be 
added to router's OS ... After all it is in the cell phone why not in 
the router :-) */

Cheers,
R.

---

From: Jay Borkenhagen <[email-masked]>
Date: Thu, 2 Jul 2009 11:06:20 -0400

Curtis Villamizar writes:
 > need to send messages through the BGP peering to reach someone with
 > a clue on the other side?  [...]

Hi Curtis,

The advisory capability is meant to supplement other NOC to NOC
communications. 

Here are a couple key (IMHO) passages from the draft:

   While [email/phone] have been used for many years, it can be
   troublesome for an operator to correlate a BGP- related event in
   the network with a notice that was transmitted in email.

and:

   This capability can speed up operator reaction and resolution time
   by providing a direct correlation between a BGP event and the root
   cause.

In my day job I'm often asked to look through available information
including router syslogs to help figure out what had happened in the
network.  I'm not normally a recipient of NOC to NOC emails, so it
would be a nice help to me to find a log message like "we're about to
begin the maintenance we notified [email-masked] of on <date> under our
ticket <number>", or "the session flap we just experienced was caused
by <bar>".

In my view, it's a way to put operationally useful information in
another of the places someone might be looking.

Thanks.

						Jay B.

PS: I support adopting the draft as a WG item.

-- 
  Jay Borkenhagen     [email-masked]

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 2 Jul 2009 11:20:41 -0400

> Hi Jim,

(James, normally)

> version -00 or those on Nanog slides including "good ideas" from pages 11 &
> 12 ?
>
> •An idea where you can transmit an advisory message to “stick” in the BGP
> neighbor state. This way, upon viewing “show bgp nei x.x.x.x”, you can see
> the latest sticky message transmitted by the peer.
> Question: How can something with random length (without max limit) be part
> of cli show command and yet still make this show command nicely readable and
> parsable by scripts ?
>
> Also I think we should enforce the free form language to be ASCII English
> only.
>
> added to router's OS ... After all it is in the cell phone why not in the
> router :-) */

and format string checking and.... That's all well past the 'protocol
now carries this new message' though, which I think is IDR's purview,
yes? Every vendor (Doing bgp) will then have figure out how to
shoehorn this into their implementation. First though:

1) is IDR willing to accept the work
2) is this something that the routing protocol should be used for?
3) are the details of the protocol change spec proper/valid/good?

I share the 'overloading bgp yet again is a bad plan' but also
'getting ops to run yet another thing on their hardware and arranging
that to work properly amongst their peers' is not trivial. (so this
goes into bgp, please keep it very simple, small payload + limited
ascii-set)

-Chris

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 02 Jul 2009 08:52:10 -0700

Hi,

> goes into bgp, please keep it very simple, small payload + limited

Well I am not in the camp of folks who get high blood pressure if one 
wants to add new thing to BGP. I don't really think we are any close to 
overloading BGP or it's commercial implementations.

But whatever we add for one should be useful for the operation and in 
the same time should not provide any means for negative impact on the 
existing mechanism.

I am glad you share the small payload (hopefully with limited size) and 
limited character set to basic ASCII.

Would the latter be acceptable for vendors making domestic equipment in 
various parts of the world ?

But those are rather minor details which I think Tom would be willing to 
consider ... For me addition of free form text to fill your peer's 
syslog into BGP is questionable from the point of view that router will 
not be able to do any action on it. It is step into opposite direction 
as intelligent networks. It is forcing human to log in, search, read, 
parse, make a phone call (yes one of the example lists phone number) etc 
...

Sometimes I do wonder why we are so behind in networking that so much 
human intervention is required to operate it. As one of million 
examples: we have planes which can fly, start and land on an auto-pilot 
even in complete fog .. yet here we are adding more stuff which will 
force more human labor and human operational dependability.

Cheers,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 02 Jul 2009 09:04:52 -0700

Hi Dorian,

> Given the actual practices of operators today, and the realities of poor 
> coordination in most cases, this proposal will provide a net benefit
> to day to day operations of the Internet.

It will - I agree. But it will by a sort of patch and accept the actual 
practices and realities of poor coordination isn't it ?

I think some people at last IETF proposed to look into INFORM draft ... 
It can be extended with as many pre-defined messages as required. I have 
not see much discussion comparing both.

I think we here are essentially making the call if free form text to 
syslog/show command is something we should endorse in the BGP protocol 
or not.

If we do let's proceed with the advisory draft making it just a bit more 
robust, if not let's resubmit INFORM and make sure it will address all 
BGP messaging requirements in a well pre-defined form.

Or perhaps both should be done ?

> Whether the benefit is worth the cost in abstract is unclear to me, but it 
> certainly is far more useful than any number of other things people have 
> shoehorned into BGP of late. 

You got a point :)

Cheers,
R.

---

From: David Freedman <[email-masked]>
Date: Thu, 02 Jul 2009 17:08:26 +0100

> parse, make a phone call (yes one of the example lists phone number) etc

My view is that we should have a minimal number of standardised message
types (admin, operational etc..) and then allow these to provide
freetext payloads, you could provide a "vendor-specific" type to cover
where routers from the same vendor which to exchange structured payloads
(which will end up displaying structured notices / flags in their CLI)

Am careful to make it "Vendor Specific" rather than not so (i.e generic
"Machine-Machine") as I feel this would encourage us to try and
standardise the payload of this type which starts to go against the
spirit of the draft.

We can leave that to the vendors to implement if they wish to further
enrich their products.

Dave.

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 2 Jul 2009 12:43:13 -0400

On Thu, Jul 2, 2009 at 11:52 AM, Robert Raszuk<[email-masked]> wrote:

> consider ... For me addition of free form text to fill your peer's syslog
> into BGP is questionable from the point of view that router will not be able
> to do any action on it. It is step into opposite direction as intelligent
> networks. It is forcing human to log in, search, read, parse, make a phone
> call (yes one of the example lists phone number) etc ...

I did say something about netcool, or other syslog aggragators. I
don't expect (though it's likely going to happen) people to login to
routers to see 'oh yes session X is down with message Y', but I do
expect my syslog aggragator to get this message and pref/depref some
other alerting mechanism so my 'NOC' doesn't have to go chasing some
bouncing peer.

Again though, this is sort of in the weeds of the 'should this be here
in the first place' discussion.

So far there have been ~10-15 'yes please have IDR do this work'
messages, I didn't see any 'no idr for you!' messages, did I miss
them?

-chris

---

From: Dorian Kim <[email-masked]>
Date: Thu, 2 Jul 2009 11:51:39 -0400

> I share the 'overloading bgp yet again is a bad plan' but also

This ship has sailed long ago unfortunately, and at this point
I think the "overloading bgp yet again is a bad plan" cry is reaching
the point of "save the children". 

I support the adoption of this as a WG item. 

-dorian

---

From: Andy Davidson <[email-masked]>
Date: Thu, 2 Jul 2009 17:48:28 +0100

Hi,

I do NOT share your view.  We have seen evidence of real disruption on  
the internet caused by the lack of a soft-notify feature in BGP (e.g.  
ASN32 error causing hard notify and session drop), that a soft-notify  
(send message and discard update message) would solve.

Furthermore I can not use the advisory feature to disrupt BGP  
stability on the internet because my messages to peers are not  
transited to their peers.  Unlike, for example, bgp attributes which  
are transited, and *can* lead to routing instability, in the event  
that malicious activity is attempted, or accidents occur.

This is a safe proposal, and a useful feature.  Please support it's  
adoption as a WG item and eventually an internet standard.

Andy

---

From: Dorian Kim <[email-masked]>
Date: Thu, 2 Jul 2009 12:58:49 -0400

> practices and realities of poor coordination isn't it ?

I think too much discussions down this path will lead us into varying
degress of organisational pathologies and not very useful. IDR will not
and cannot change the way ISPs operate nor will it be able to  address the 
root cause of the disfunction in inter-ISP and even intra-ISP communications. 

That's almost entirely in the realm of social engineering and we've been down
that road many times before to no fruitful end.

Leaving that aside, people who run networks are asking for this because it solves
a real world problem they encounter. That by itself should in my opinion be enough
for the working group to decide on "should this be discussed and worked on here"
question.
 
> I think some people at last IETF proposed to look into INFORM draft ... It 
> can be extended with as many pre-defined messages as required. I have not 
> see much discussion comparing both.
> 
> syslog/show command is something we should endorse in the BGP protocol or 
> not.
> 
> robust, if not let's resubmit INFORM and make sure it will address all BGP 
> messaging requirements in a well pre-defined form.
> 

I'm most likely not the best advocate for this draft since I'm somewhat lukewarm
about its benefit given the cost. However, allow me to bring up and analogy
with all the usual caveats inherent in anaologies. CLNS host info doesn't by 
itself amount to much, but has shown itself to be of many time saving operational
uses. So to that extent, some facilities for having a limited free form text
to syslog command tied to BGP may be of more use than we can think of at the moment.

-dorian

---

From: vijay gill <[email-masked]>
Date: Thu, 2 Jul 2009 11:08:22 -0700

> to syslog command tied to BGP may be of more use than we can think of at the moment.

I disagree strongly with the operational reality of this draft. It may
be useful for some folks who have the time to actually sync up and
talk with another person on the far side of an _already_ operational
BGP session. That corner case is not common enough to actually make
this useful in my opinion. The real utility might be if the session
was not established and in that case we fall back to OOB communication
anyway.

/vijay

---

From: Richard A Steenbergen <[email-masked]>
Date: Thu, 2 Jul 2009 13:19:06 -0500

> was not established and in that case we fall back to OOB communication

Funny, I've heard from many operations folks at Google who said they
really wanted this feature, and saw immediate uses for it. Clearly this
isn't the most elegant feature in the world, it certainly isn't
impressing anyone up in the ivory tower, but for those of us who
actually operate networks (and no, working for a company who has a
network is not the same thing as being a network operator), the benefits
are clear and compelling.

-- 
Richard A Steenbergen <[email-masked]>       http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

---

From: "John G. Scudder" <[email-masked]>
Date: Thu, 2 Jul 2009 14:09:44 -0400

> providers somewhere?  I didn't see where that limit was recommended.

It's a new message type and has no semantics defined other than being  
logged by the receiving party, so it's implicitly scoped.  For it NOT  
to be locally scoped, some sort of propagation semantics would have to  
be defined.

--John (with co-author hat on and co-chair hat off)

---

From: vijay gill <[email-masked]>
Date: Thu, 2 Jul 2009 11:31:39 -0700

> are clear and compelling.

Let me know who these are, oob, and we'll go from there

/vijay

---

From: Joe Abley <[email-masked]>
Date: Thu, 2 Jul 2009 15:43:34 -0400

> was not established and in that case we fall back to OOB communication

I have spent some time over the past few years (and continue to spend  
time) with small, regional ISPs whose network topology is not  
sufficiently complicated for BGP specialists to be employed, or for  
elaborate network management tools with hooks into ticket systems to  
exist.

For these people, the reality is that the guy who is most familiar on  
the day with the router in question dives in and takes a look, trawls  
through logs, and types a bunch of show commands. Out-of-band  
communication with staff at other organisations will follow, but that  
communication is typically far more targeted and useful if there is a  
readable indication of the context for an operational state than if  
people have to guess.

It has been my observation that amongst people who aren't full-time  
router jockeys, the initial three or four e-mail exchanges are pretty  
wide of the mark. If (for example) they could quote a ticket number or  
some other meaningful text along with their initial communication,  
chances are good that streamlining could occur.

I am not familiar in any current, hands-on sense with large hosting  
organisations, and no doubt the state of the art has moved on since I  
last had my fingers in the router of a default-free backbone operator.  
However, I would suggest that in the circles I described a simple,  
textual capability along the lines of that described would have  
substantial value.

I support this draft's adoption by idr and its further refinement by  
the working group.

Joe

---

From: "Gill, James E (gill)" <[email-masked]>
Date: Thu, 02 Jul 2009 19:40:42 +0000

With some trepidation  I wade into the discussion on the opposite side
of many minds wiser than mine, but here goes...

The benefits are clear and compelling, but the method isn't (to me).

The argument that BGP is already overloaded isn't enough to convince me
that this is an appropriate feature to add into a routing protocol.  Mom
always said two wrongs don't make a right.

A separate protocol might be defined for a daemon might run alongside
BGPd and do essentially what this document suggests  (Perhaps the draft
R. Raszuk mentioned earlier, INFORM, is worth a look here).  Separating
this function from the BGPd means that it could be turned off if it was
problematic or when it wasn't necessary any more, or re-used later when
something better than BGP comes along*. To some extent, this mindset is
left over from the unix-y idea that tools/code should "do one thing and
do it well."

The work to get your peers and customers to upgrade their code and turn
on such a feature would be approximately equal to the work of getting
them to upgrade their code and configure these new features that are
being proposed as additions to BGP.  In either case, someone has to
actually read and react to these notices or it does nothing.  That isn't
garunteed, but it would presumably put these messages into the syslog
along with the BGP entries and make troubleshooting/postmortem easier,
even if it won't get the message to the right humans before the event
occurrs.

Let me cut and paste a paragraph here:

IMO the need for this kind of communication exists independent of any
EGP and should be defined separately.  We are all in "business" which
means we all have suppliers (upstreams/peers) and customers
(downstreams), and a key component of any business is communicating with
your suppliers and customers.  Just because we all happen to speak BGP
does not mean that BGP needs to be the protocol through which we make
our important communication.

--gill

* Heretical?  Possibly.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Richard A Steenbergen
Sent: Thursday, July 02, 2009 2:19 PM
To: [email-masked]
Subject: Re: [Idr] draft-scholl-idr-advisory

-- 
Richard A Steenbergen <[email-masked]>
http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1
2CBC) _______________________________________________
Idr mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/idr

---

From: vijay gill <[email-masked]>
Date: Thu, 2 Jul 2009 12:59:49 -0700

On Thu, Jul 2, 2009 at 12:43 PM, Joe Abley<[email-masked]> wrote:
> I have spent some time over the past few years (and continue to spend time)
> with small, regional ISPs whose network topology is not sufficiently
> complicated for BGP specialists to be employed, or for elaborate network
> management tools with hooks into ticket systems to exist.
>
> For these people, the reality is that the guy who is most familiar on the
> day with the router in question dives in and takes a look, trawls through
> logs, and types a bunch of show commands. Out-of-band communication with
> staff at other organisations will follow, but that communication is
> typically far more targeted and useful if there is a readable indication of
> the context for an operational state than if people have to guess.
>
> It has been my observation that amongst people who aren't full-time router
> jockeys, the initial three or four e-mail exchanges are pretty wide of the
> mark. If (for example) they could quote a ticket number or some other
> meaningful text along with their initial communication, chances are good
> that streamlining could occur.

> organisations, and no doubt the state of the art has moved on since I last
> had my fingers in the router of a default-free backbone operator. However, I
> would suggest that in the circles I described a simple, textual capability
> along the lines of that described would have substantial value.
>
> I support this draft's adoption by idr and its further refinement by the
> working group.

I still disagree, but after conversation with ras et al, I withdraw my
objection and let the cards fall as they may.

/vijay

---

From: Richard A Steenbergen <[email-masked]>
Date: Thu, 2 Jul 2009 15:21:02 -0500

> left over from the unix-y idea that tools/code should "do one thing and

Last I looked most existing BGP implementations already dealt with
things like receiving notifications from peers and generating syslog or
snmp traps based on those events without requiring a separate process. 
The draft proposal here is simply adding another type of event to log
on, using the exact same mechanisms already in place to log other
events.

I don't see the benefit in creating another protocol for this specific
message type, only unnecessary additional complexity and the destruction
of the implied relationship (authentication, etc) provided by in-band
signaling over the existing established BGP sessions. Besides, since the
message is specifically for and about one particular BGP session
anyways, why would you want to handle it any other way?

---

From: Joe Provo <[email-masked]>
Date: Thu, 2 Jul 2009 16:28:19 -0400

> being proposed as additions to BGP.  

Your suggestion posits creation (and worse, maintenance) of a separate
adjacency.  The proposed draft does not. This simplification alone is
a dramatic win.  In future, should the code be part of mainstream bgp
releases, new neighbors would just have this capability from the get-go, 
rather than need to set up the separate adjacency; therefore over time 
the net workload is reduced.  Your suggested  parallel adjacency creates 
a constant additional workload without future optimization as it is 
mainstreamed. Again, win for advisory.

Cheers,

Joe

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 02 Jul 2009 14:08:08 -0700

Hi Joe,

Very true email.

But something here is not synchronizing for me well.

* Do you think that those big providers would even care to tell the 
small few router guys over new advisory msg that they are going to be 
down if they don't tell them today via any other means ? What is so 
magic about this new way of communication ?

* We have number of proposals in IETF for graceful BGP shutdown. Great 
work, but why bother if we can just inject advisory msg saying "You guys 
are going to be down soon .. be aware and prepared."

* I really wish someone would come with this idea years ago ... It would 
save huge resources of development spend for features like 
Non-Stop-Routing to make any ASBR/PE any potentially disruptive 
operational activity 100% transparent to the peer. It would be so much 
easier to implement this draft then just shoot an advisory message to 
them and be done.

Cheers,
R.

> elaborate network management tools with hooks into ticket systems to exist.
>

---

From: "SCHOLL, TOM (ATTLABS)" <[email-masked]>
Date: Thu, 2 Jul 2009 17:18:48 -0400

> easier to implement this draft then just shoot an advisory message to 

What does NSR have to do with links going down due to maintenance? Or
cards on a router having to be replaced?

---

From: Terry Manderson <[email-masked]>
Date: Fri, 3 Jul 2009 10:45:33 +1000

On 03/07/2009, at 6:28 AM, Joe Provo wrote:
>>
>   The proposed draft does not. This simplification alone is
> a dramatic win.

I'm not averse to the idea. Conceptually I'm struggling with the  
position that a _routing protocol_ in its adjacency relationship is  
used for administrative Inter-AS(?presumably) NOC-to-NOC  
communications or alerts. That might be just me, and I may have  
silence the academic pedant in me, get over it and move on.

I have a second concern that relates to the Charter of IDR. While IDR  
has a task of working on BGP, does NOC-to-NOC specifics fall in this  
mandate? If the chairs believe this is allowable then I'll agree that  
IDR should task itself with this draft as a WG document.

Terry

---

From: Greg Skinner <[email-masked]>
Date: Fri, 3 Jul 2009 01:02:58 +0000

> IDR should task itself with this draft as a WG document.

Is it possible that this should be in the O&M area, perhaps in the
GROW WG?

--gregbo

---

From: Joe Provo <[email-masked]>
Date: Thu, 2 Jul 2009 21:51:42 -0400

> Very true email.

Thanks.  I like to think that we want all these computers to actually 
reduce aggregate workload rather than increase it.

> magic about this new way of communication ?

If manual, once it is part of the lowest-common-denominator NOC's 
flowchart, it will just happen.  But an additional win of this proposal
is leaving it up to implementation to allow automagic inform messages
based on sane triggers, eg:
- send advisory when peer hits x% of maxprefix
- send a (limited) countdown to planned maintenance a la wall(1) when
  one is doing a "shutdown at <timestamp>" or its equivalent
- when a neighbor is sending you so much trash that you're discarding
  X% of the prefixes
- when additional address families are supported by a neighbor

> easier to implement this draft then just shoot an advisory message to 

This is orthogonal to an graceful or nonstop mechanisms.  First and 
most simply, as negotiated optional attributes, local policy may 
dictate what set of attributes are desirable to be supported.  Those
for whome GS, GR or NSF/NSR is scary might be more inclined to use
advisory.  Secondarily, automated methods will be better for those 
that opt to use them, so dismissing automated methods for human
messaging ones seems to be a step backwards.

Cheers,

Joe

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 03 Jul 2009 00:40:52 -0700

Currently we have an option for either human or machine generated 
message to be saved in syslog or "stick" to the bgp show command.

How about if we add one more message type of having this pop-up on the 
console window ?

During troubleshooting it may be very valuable to have this type of 
interactive IM chat with the peer I suppose to pass brief live msg.

Cheers,
R.

---

From: "Neil J. McRae" <[email-masked]>
Date: Fri, 3 Jul 2009 10:21:52 +0100 (BST)

I'm strongly not in favour of this draft.

I never thought I'd see the the day where BGP had an MTA baked into it!

Does anyone who runs a decent sized network really think this is a good
idea? With hundreds of peering sessions between peers and customers?!

How are you going to check all of these?

Then there is the potential for abuse

"Call 1-800-ISP for our latest offer"

anyone who thinks that this messaging system won't be open to any more
abuse than all the other messaging system is kidding themselves on, I can
see my marketing droids rubbing their hands in glee already.

Its another thing for already overworked Network Operations Centres to
monitor where the functionailty exists elsewhere.

The draft would be better if it had a fixed number of error numbers and
these were defined as part of the deaft  but its still a pretty terrible
direction in my view.

I can see contracts already:

Section 7 - Network Monitoring
------------------------------
$customer will send messages to $supplier using the BGP-Scholl draft.
$supplier is required to act on these messages within
_UNREASONABLE_TIMEFRAME_  $supplier is not permitted to use BGP-Scholl
draft for any _USEFUL_MESSAGE_ to $customer.

hmm, wonder when my transit contract is up for renewal!

Its a steep and slipery slope in my view, I cringe thinking what vendors
might end up doing with this.

Regards,
Neil.
--
Neil J. McRae -- Alive and Kicking.
[email-masked]

---

From: David Freedman <[email-masked]>
Date: Fri, 03 Jul 2009 11:03:08 +0100

> magic about this new way of communication ?

Nothing new or magical, new $tool in $belt.

> are going to be down soon .. be aware and prepared."

See above.

> easier to implement this draft then just shoot an advisory message to

See above.

The stark reality we have to face is INFORM and SOFT NOTIFY being shot
down whilst real operators are crying out for something actually useful
and time-saving, especially when having to deal with
customer/peer/provider oddness
(which as we know these days can be time consuming enough).

Rather than waste time arguing over if this is right or not (even for
this WG) I suggest we refer to the charter in this case:

"The Inter-Domain Routing Working Group is chartered to standardize
and promote the Border Gateway Protocol Version 4 (BGP-4) [RFC
1771] capable of supporting policy based routing for TCP/IP
internets."

Since we are all in agreement at the very least that something needs to
be done about the effects of NOTIFICATION messages ( I challenge
somebody to refute this )  then I suggest we either :

- - Support making the draft a WG item and work through its problems such
to make an internet standard

or

- - Propose and agree an alternative which deals with our core concerns.

So far we have done nothing but allow personal judgements and positions
to cloud the discussion as can be seen historically on this issue, I
suggest we get off our backsides.

Dave.

---

From: Will Hargrave <[email-masked]>
Date: Fri, 03 Jul 2009 11:07:41 +0100

> idea? With hundreds of peering sessions between peers and customers?!

I would think so, yes. Look at the list of supporters, many of whom run
(have enable on) larger multi-country and multi-continent networks.
Saying a proposal is invalid because proposer's network is tiny and he
doesn't know what he's talking about is a poor way of making an argument.

> see my marketing droids rubbing their hands in glee already.

This would be rather a pointless task. The fact you have a set of BGP
sessions with someone means you have a _pre-existing_ contractual
relationship with them; what is the point of sending short advertising
messages to some NOC staff who aren't even likely to have purchasing
authority?

> Section 7 - Network Monitoring

You oppose a technology because your contract drafting and negotiation
skills are poor? They could just as easily require you to send smoke
signals. Or send it over ipsec-encrypted router-to-router syslog. Or
anything else unreasonable.

---

From: Paul Wall <[email-masked]>
Date: Fri, 3 Jul 2009 06:21:37 -0400

On Fri, Jul 3, 2009 at 3:40 AM, Robert Raszuk<[email-masked]> wrote:
>
> Currently we have an option for either human or machine generated message to
> be saved in syslog or "stick" to the bgp show command.
>

How would that pop-up look in a VT100 window exactly? Would it be
ncurses based? Given how many operators disable logging to the console
as they type, this would probably be useless. Since you don't run an
actual production network, I can see why you might not know this.

With the mocking attitude shown in your email, its no wonder you got
let go from Juniper.

Hows Cisco working out this second time around Robert? Are you going
to quit again if they don't make you a Distinguished Engineer?

Cheers

Drive Slow

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 03 Jul 2009 03:59:48 -0700

The proposal was actually serious and not mocking.

But if someone has no concept on how to open two windows while 
performing troubleshooting with the peer instead of one I can see why 
the suggestion made may not have been understood.

R.

PS. Btw ... People asking personal questions are usually bright enough 
to do it offline and I am happy to answer any of them without spamming 
an IDR list.

---

From: "Neil J. McRae" <[email-masked]>
Date: Fri, 3 Jul 2009 13:15:57 +0100 (BST)

I thought this was IDR but in anycase:

> doesn't know what he's talking about is a poor way of making an argument.

Will if thats what you've implied them I take it you don't feel you know
what you are talking about, you've clearly demonstrated it that's for
sure.

In my view this revision to the standard adds a tremendous amount of work
for anyone who runs a large network with a large number of peers, both
customers and suppliers, one would imagine that smaller organisations who
run smaller networks with less resources would be equally burdened. I
currently run a few networks (have enable as you put it!) for different
companies of varied size from two elements in one cupboard to over twenty
thousand elements on every continent on the planet, and in my view this
adds a significant additional overhead of monitoring such messages, more
so than the status quo, for a very limited and quite debatable benefit.

Then there is the overhead of what happens if someone were to put a
message that you couldn't understand or was confusing "LINK DOWN" - 'what
does that mean?'

> messages to some NOC staff who aren't even likely to have purchasing

I have pre-existing contractual relationships with many people and they
still send me junk that I don't want- why would this be any different?

> anything else unreasonable.

!?!

This isn't technology though is it? Its a fudge of a routing protocol to
do something that already exists in other messaging protocols that bad
social and management capabilities force us to consider. If we want to
have interdomain router to router operator communication as a technology
then lets design something that does that but please lets not stuff this
into technolgy that is used for network reachability. I'd call it an
anti-technology.

Whilst it won't change my view that this is a bad idea, I think that at
the very least there should be no free form of text and that a list of
"message/error codes" and their meaning be adopted a revised draft, here
are a couple of ideas:

CALLNOC (Call the NOC)
PLANNEDWORKS (We have planned works)
FAB or A-OK (Generic status for normal operating environment)
etc...

I welcome any proposal that makes operating networks simpler and enhances
the service we provide to our customers but in this instance it occurs to
me, reviewing the thread so far, that we've all jumped on the positive
bandwagon without really thinking about all the causal side effects.

Regards,
Neil.

---

From: Florian Weimer <[email-masked]>
Date: Fri, 03 Jul 2009 12:26:13 +0000

> so than the status quo, for a very limited and quite debatable benefit.

Why would you have to monitor this messages?  Wouldn't the interface
be more like this:

# show ip bgp neighbor 192.0.2.1
[...]
   Last advisory message: "maximum prefix limit exceeded, retry in 10 minutes"
[...]
# show ip bgp neighbor 192.0.2.1 messages
Fri Jul  3 12:20:12 UTC 2009   approaching configured prefix limit
Fri Jul  3 12:24:52 UTC 2009   maximum prefix limit exceeded, retry in 10 minutes
#

There could be an SNMP trap for this, but would be totally optional.

This is not intended as a replacement for SMTP or IRC, I think. 8-)

-- 
Florian Weimer                <[email-masked]>
BFK edv-consulting GmbH       http://www.bfk.de/
Kriegsstraße 100              tel: +49-721-96201-1
D-76133 Karlsruhe             fax: +49-721-96201-99

---

From: Andy Davidson <[email-masked]>
Date: Fri, 3 Jul 2009 13:27:02 +0100

On 3 Jul 2009, at 13:15, Neil J. McRae wrote:

> for anyone who runs a large network with a large number of peers,

It's capability based, so (for example) ....

router bgp 1234
  no bgp advisory-message
^Z

Nothing changes.

However, remember the incidents of Dec 10th last year which sparked  
much of this debate originally.  Networks were disconnecting from the  
internet because all notifies had to be hard-notifies and cause a bgp  
session drop.  http://www.merit.edu/mail.archives/nanog/ 
msg14345.html .  A soft-notify message & discard would have led to a  
more stable internet that day (and when similar problems crop up in  
the future.)

However, your network, your rules !

Andy

---

From: David Freedman <[email-masked]>
Date: Fri, 03 Jul 2009 13:33:51 +0100

> It's capability based, so (for example) ....

and even

(config-router)#no bgp advisory-message ?

  administrative        disable administrative advisory
  operational           disable operational advisory
  <insert types here>
  <cr>

(this should be per neighbor as well btw)

Dave.

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 03 Jul 2009 05:35:10 -0700

Hi Andy,

 > much
 > of this debate originally.  Networks were disconnecting from the
 > session drop.  http://www.merit.edu/mail.archives/nanog/msg14345.html

How would draft-scholl-idr-advisory help in this case ?

 > A soft-notify message & discard would have led to a more stable
 > internet that day (and when similar problems crop up in the future.)

I think we all do support Soft-Notify or Inform draft like solution with 
pre-defined error codes. So why would we not adopt them first and see 
how much improvement this brings and what is missing ? Then after 
getting operation experience we could revisit the need and the best way 
to exchange free form text messages.

Thx,
R.

> However, remember the incidents of Dec 10th last year which sparked much 
> of this debate originally.  Networks were disconnecting from the 
> session drop.  http://www.merit.edu/mail.archives/nanog/msg14345.html .  
> A soft-notify message & discard would have led to a more stable internet 
> that day (and when similar problems crop up in the future.)
>

---

From: "Neil J. McRae" <[email-masked]>
Date: Fri, 3 Jul 2009 13:37:26 +0100 (BST)

Florian,

> Why would you have to monitor this messages?  Wouldn't the interface

Why would you have them if you didn't?

...
> This is not intended as a replacement for SMTP or IRC, I think. 8-)

So if nobody is reading it or monitoring it - what value does it add? Why
not change the max-prefix functionality to do all that stuff for you? ( I
think it does on some vendor implementations now ).

Please don't mis-understand me, I'm sure there are some use-cases for this
that for some will be positive, from my point of view there are a number
of side effects that don't make this worth while - but I re-iterate if you
need router to router operator communication then I'd argue that we should
do that outside of BGP (and IDR!). You could add more info into it for
MPLS IPVPN NNI or other NNI situations.

---

From: Andy Davidson <[email-masked]>
Date: Fri, 3 Jul 2009 13:43:32 +0100

On 3 Jul 2009, at 13:35, Robert Raszuk wrote:

> msg14345.html

Our recommendation was to discard the update, but in tandem an  
automatic message could be generated to the upstream network informing  
them that a prefix with unclean attributes was transited.

> I think we all do support Soft-Notify or Inform draft like solution  
> with pre-defined error codes. So why would we not adopt them first  
> and see how much improvement this brings and what is missing ? Then  
> after getting operation experience we could revisit the need and the  
> best way to exchange free form text messages.

In my opinion, any proposal which does not incorporate the freetext  
ability of the Scholl draft is incomplete.

Human derived annotations in my sh log will help debug inter-network  
communication issues more quickly and improve internet stability.  Fact.

Andy

---

From: Florian Weimer <[email-masked]>
Date: Fri, 03 Jul 2009 12:44:33 +0000

> Why would you have them if you didn't?

To avoid phoning your peer when the session doesn't come up after some
hick-up.  Due to the way BGP session handling works, you have to
reject incoming BGP connections for some time after you've received
some update you didn't like.  If you don't that, you potentially
receive hundreds of thousands of prefixes again and again, in each
session, until the problematic update is hit, the session is torn
down, and the whole thing starts over again.

> So if nobody is reading it or monitoring it - what value does it add?

If your peer sends an RST to your TCP segment, how do you know today
it's caused by previously exceeding the maximum prefix count limit?

I think this is really an area where BGP could be improved.

> Why not change the max-prefix functionality to do all that stuff for
> you? ( I think it does on some vendor implementations now ).

Certainly not, how would they do that?  You'd have to generate an SMNP
trap and send mail to your peer, but this seems unlikely.

A concern could be that some operators won't like the information leak
this functionality entails, but I would assume they are in the
minority.  If your peer can diagnose more BGP issues without real-time
coordination, it's a win for both sides.

---

From: "Neil J. McRae" <[email-masked]>
Date: Fri, 3 Jul 2009 13:52:48 +0100 (BST)

Andy,

> Nothing changes.

Other than the side effects of this. But I appreciate that you don't have
to use this tool.

> more stable internet that day (and when similar problems crop up in

I'm reading this: http://tools.ietf.org/html/draft-scholl-idr-advisory-00

Andy, there is nothing in the standard that I read that talks about this
or how it would help, or the exact same event that happened 1996 when an
AS leaked a long AS Path. I am all for fixing that problem, but this
leaves it open to implementation as to how it is used - please correct me
if I'm mis-reading this.

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 03 Jul 2009 05:57:17 -0700

Hi Florian,

> down, and the whole thing starts over again.

Let's examine this use case perhaps a bit closer ...

In the malformed update case you are right that the BGP behaviour is to 
reset the session. While doing such reset number of implementations I am 
familiar with log the malformed part of the update so you can know 
exactly what was the reason for the reset and perform corresponding action.

I am not clear at what point you will expect or inject an advisory msg 
and what would such message contain ?

Also advisory message as currently defined will not survive on it's own 
nor be sufficient to keep a session up. At least a pair of matching 
AFI/SAFI must be negotiated along with it. So can you just for better 
understanding of this use case describe sequence of events showing the 
place where advisory msg will be send by any peer for the malformed 
update session reset case ?

Many thx,
R.

---

From: "Neil J. McRae" <[email-masked]>
Date: Fri, 3 Jul 2009 13:58:50 +0100 (BST)

> ability of the Scholl draft is incomplete.

Why?

> communication issues more quickly and improve internet stability.  Fact.

Andy - can you show me the data that backs this claim of fact up? :-)
Because I don't think you've considered the side effects of how bad
communication within this protocol may hinder Internet stability, and the
previous poster says we won't be monitoring them so the network will have
fallen over but we'll have a load of historical messages that nobody has
read telling us what happened. Will be useful for network incident reports
if nothing else!

---

From: "Neil J. McRae" <[email-masked]>
Date: Fri, 3 Jul 2009 14:04:43 +0100 (BST)

> down, and the whole thing starts over again.

So the outage has already happened, the session is down? I'd have thought
it would be better to have monitored that this was an issue and deal with
it accordingly.

> it's caused by previously exceeding the maximum prefix count limit?

Usually because they've sent an email to our NOC telling us about it.

> I think this is really an area where BGP could be improved.

Its not BGP that you are improving! Its implementation around it.

>  If your peer can diagnose more BGP issues without real-time

I agree that is a win-win and so far you've almost convinced me for one
use case atleast after the fact you've had a failure. If it was some bogus
NOTIFICATION, its already too late.

---

From: Florian Weimer <[email-masked]>
Date: Fri, 03 Jul 2009 13:06:44 +0000

* Robert Raszuk:

> In the malformed update case you are right that the BGP behaviour is
> to reset the session. While doing such reset number of implementations
> I am familiar with log the malformed part of the update so you can
> know exactly what was the reason for the reset and perform
> corresponding action.

This log entry is only visible to the peer which dropped the
connection.

> and what would such message contain ?

In case of malformed updates, it's generally a shared issue, i.e. both
parties are potentially at fault.

For the maximum-prefix case, it's usually the fault of the peer who
sends to many updates.  In this case, the log messsage which explains
things is logged by one side only.  The guilty party has to guess the
cause, or obtain the log message by phoning/emailing the other
operator.  This process can be improved, and draft-scholl-idr-advisory
is one way to achieve that.

> Also advisory message as currently defined will not survive on it's
> own nor be sufficient to keep a session up. At least a pair of
> matching AFI/SAFI must be negotiated along with it. So can you just
> for better understanding of this use case describe sequence of events
> showing the place where advisory msg will be send by any peer for the
> malformed update session reset case ?

I didn't bring up the malformed update case, and it's not in the
draft.  I personally think the maximum-prefix use case is
justification enough for this draft.

In general, malformed updates are shared issue, with blame difficult
to attribute.  It's also difficult to guess how to rectify the issue
(in general, there's no "no send broken-updates" command), so a shared
debugging session is probably in order anyway.  A malformed update
should be extremely rare.  If the RFCs were correct and correctly
implemented, it's basically a "can't happen" scenario.  It's not
reasonable to expect that one party is to debug such an issue
unilaterally.

---

From: Florian Weimer <[email-masked]>
Date: Fri, 03 Jul 2009 13:14:31 +0000

> it would be better to have monitored that this was an issue and deal with

No, it's a free-text field.  You can't really use it for this purpose.

It's also not intended for pre-announcing maintenance windows.  You
could perhaps send an advisory message immediately before taking down
a session.  But as a said, this shouldn't be used as an IRC
replacement.

> Usually because they've sent an email to our NOC telling us about it.

Ideally, yes, but often it's pull and not push.  The proposed protocol
change gives you more direct access to this information.

> Its not BGP that you are improving! Its implementation around it.

I don't think it's sense to make this distinction.  After all, it is a
protocol change we're talking about.

---

From: Joe Provo <[email-masked]>
Date: Fri, 3 Jul 2009 10:34:19 -0400

On Fri, Jul 03, 2009 at 01:52:48PM +0100, Neil J. McRae wrote:
[snip, attribution stripped by poster, was andyd]
> leaves it open to implementation as to how it is used - please correct me

Please use context of handling in conjunction with the existing WG item,   
draft-ietf-idr-optional-transitive.  Given optional-transitive allowing
a peer to drop the garbage instead of reset the session, notify allows
one to automagically tell the garbage-emitter "prefix FOO was malformed 
and ignored: <detail of malformed crud>".  

Cheer,

Joe

PS - to those on IDR without fingers in routers, a reminder that 
     one needn't go back to 1996 or even the 2008 incident Andy 
     referenced.  There are horribly broken speakers emitting 
     refuse on a regular basis, offhand I think I blinked and
     missed whoever did one in June but May had at least the 
     Cairo AfNOG meeting revisiting the AS32 issue...

---

From: Rob Shakir <[email-masked]>
Date: Fri, 3 Jul 2009 15:35:24 +0100

>From reading this thread, I think what Dave says above really needs to be noted.
We have no mechanism at the moment to send a NOTIFICATION message that is
non-fatal - whilst we can argue about the fact that there are other mechanisms
of sending human-generated messages, I believe there is value in having some
means by which I can receive a message from my peer that is akin to a soft
notification.

I believe Dave already sent a message in previous discussions of this draft that
gave a summary of the differences, and per-case advantages of either the SOFT
NOTIFY draft, or ADVISORY, and their similarities. With a TLV (or similar) based
ADVISORY, then we could achieve both requirements.

>From my own (operational) point of view, I believe there is definite value in
the free-text message proposed in this draft.

Kind regards,
Rob

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 03 Jul 2009 07:45:19 -0700

Hi Florian,

 > I personally think the maximum-prefix use case is

I don't agree.

Max-prefix warning and error can be very easy addressed programmatically 
by either of those two below drafts:

http://tools.ietf.org/html/draft-nalawade-bgp-inform-02
http://tools.ietf.org/html/draft-nalawade-bgp-soft-notify-00

With draft-scholl used to address this case you will be required to 
write your own event script, deploy on each router and the probability 
that each ISP will come with different syntax to convey such information 
to the peer is guaranteed.

Many thx,
R.

---

From: David Freedman <[email-masked]>
Date: Fri, 03 Jul 2009 15:54:30 +0100

> by either of those two below drafts:

Yes, the key word being "drafts" here, since we don't have either of
these two being suggested for WG agenda I recommend we stick with what is.

---

From: John Leslie <[email-masked]>
Date: Fri, 3 Jul 2009 12:39:38 -0400

Andy Davidson <[email-masked]> wrote:
> session drop.
> http://www.merit.edu/mail.archives/nanog/msg14345.html
" 
" Strict RFC 4893 (4-byte ASN support) BGP4 implementations are vulnerable
" to a session reset by distant (not directly connected) ASes.

   It would certainly be nice to have better communication about the
details leading to a notification. I don't think the current version of
draft-scholl-idr-advisory gets us there.

   I don't really have a horse in this race -- I can certainly deal with
megabytes of logged information if it comes to that -- but I'd much
rather do idr-advisory in a manner which _can_ pass useful information
when NOTIFY is happening.

   That would require (at a minimum) being able to queue the ADVISORY
before the NOTIFY and (while refusing any further UPDATEs) keeping the
TCP connection up "long enough" to give it a reasonable chance of
being received.

" This vulnerability is a feature of the standard, and unless immediate
" action is taken an increasingly significant number of networks will be
" open to attack.

   (I'm curious what "immediate actions" _have_ been taken...)

http://www.merit.edu/mail.archives/nanog/msg14454.html recommends
] 
] There are two cases to consider when an invalid AS4_PATH is received:
]   (1) A path to the prefix is not already known from that neighbour. 
]   (2) A path to the prefix has already been learnt from that neighbour; 
]    
] In case (1) we recommend that the BGP speaker should discard the UPDATE
] and log the fact. The log entry should include the received AS_PATH and
] AS4_PATH to aid in debugging.
] 
] In case (2) we recommend that the BGP speaker should treat the UPDATE as
] a withdrawal of existing path to the prefix. As per case (1) a log entry
] should be raised to indicate that this has occurred.

   These sound to me like programming-level issues, not something that
can be done "immediately" via configuration changes.

> that day (and when similar problems crop up in the future.)

   +1 on the "when" !!!

   In fact, I'd be _very_ surprised if we don't have similar problems
crop up over other details of 4-byte ASNs.

--
John Leslie <[email-masked]>

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 03 Jul 2009 13:18:25 -0400

In message <[email-masked]>
> didn't understand PGP email, nor prioritizing that over other email it

At that time there was an operator's security list that only accepted
PGP signed mail within a specific keyring.

You could email the NOC or a clueful person at another ISP if you knew
one.  Most of the clueful people used procmail or equiv and many
prioritized PGP signed email that was authenticated in a specific
keyring.  I'm not sure if the ANS or UUNET NOC also prioritized PGP
signed email.

Spam definitely hurt and maybe ended the use of a noc@ email as a way
to report trouble.  The phone certainly doesn't replace it, and the
web mazes aren't much better.  There were also the NOCs or "support
centers" that told their peers that "you are not in our customer list
so go away".  As you mentioned, that is a social problem.

Anyway, I agreed - if its needed, go for it.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 03 Jul 2009 13:33:05 -0400

In message <[email-masked]>
Joe Provo writes:
> I think you mispelled "commoditization of Internet service".  [...]
>  
> Cheers,
>  
> Joe 

Very careless spelling error on my part.

Failure to distinguish between consumer mass market customer, small
businnes customer, and peer may be the problem.  Large customer seems
to get prioritization that sometimes peer might not, since large
business customer (or government) brings in larger dollars and some
organizations fail to see the importance of coordination with peers.

This probably isn't the venue to discuss the social problems that lead
to the draft's requirement.  I buy the argument.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 03 Jul 2009 13:39:56 -0400

In message <[email-masked]>
> --John (with co-author hat on and co-chair hat off)

New message type that just gets logged.  Nothing in the update.  Like
CEASE but with message and has no effect on the session.  OK I get it.
Thanks.

Curtis

---

From: Richard A Steenbergen <[email-masked]>
Date: Fri, 3 Jul 2009 12:48:15 -0500

> a session.  But as a said, this shouldn't be used as an IRC

Actually, that is precisely what this draft was intended to do (well, 
one primary use case among many at any rate).

Let me spell out a typical day to day NOC scenerio, since there seems to 
be a fair amount of confusion of how and why this draft would be 
beneficial.

Event: I'm going to be reloading one of my peering edge routers with 60 
BGP sessions, connected to 60 different networks, to upgrade my router 
code.

1. Days before the event, I send out a maintenance notice to my peers.
2. Someone's NOC receives this notice, reads it, parses it, and then 
   either puts the ticket in some kind of pending queue or notes the 
   event on some kind of internal maintenance calendar.
3. Days later, it's finally time to do the maintenance. I clear the BGP 
   sessions, and reload my router.
4. I start to find out how badly step 2 failed. It turns out that of the 
   60 NOCs who received this notification, no fewer than 10% (and 
   usually more) either failed to correctly parse the e-mail, failed to 
   record it properly internally, couldn't find it when they searched 
   their ticketing system for what ever reason, etc.
5. These NOCs (now with completely different shifts staffed by 
   completely different people from when that first e-mail came in) now 
   have a big red PEER DOWN popping up on their monitoring system, and 
   so they do their jobs and either start sending e-mails asking about 
   the down session, or they pick up the phone and call my NOC.
6. Now my noc has to ask my eng staff whats going on, and start telling 
   everyone who is calling that it will be back up shortly, start 
   closing tickets, etc.

Note all of this was for a PLANNED event where an e-mail was sent. Now 
why don't we further extend this scenerio to what is an even more 
realistic event, my $routeros just blew up in some weird and unpleasant 
way and I was forced to do an emergency upgrade or reload. Now the 
number of calls and e-mails doubles or triples. And I can't fault these 
peers for calling when they see an event, because honestly at least 25% 
of our peers (even some big international carriers) are so disorganized 
that they wouldn't know about a legitimate failure on their own if 
nobody told them.

Now lets try that again, but with the ADVISORY draft.

3. Days later, it's time to do the maintenance. I send out an advisory 
   message to my peers indicating that I am performing a maintenance 
   activity under ticket #ABC12345, and that I expect the session to be 
   back up within 10 minutes.
4. The advisory message POPs up in syslog and in the monitoring systems 
   of my peers, followed a few seconds later by the peer down message.
5. Those 60 NOCs now have immediate access to the information about 
   this session, why it went down, what ticket # I'm tracking this under 
   on my side, and information about how long they should wait before 
   assuming there is a problem. Even if they are a small network without 
   fancy monitoring systems, someone is going to login to the router and 
   type show log | match x.x.x.x and see the message.
6. The router comes back up, and a dozen confused calls and e-mails are 
   avoided. If not, they know a) what happened, b) that we know about 
   it, and c) what ticket its referenced under if they do have to call.

Why does this work? Think about the problems with e-mail only 
notifications:

* First you have to maintain up to date contact information for your 
  peers, which in and of itself is not an easy task. Some networks out 
  there have over 1000 peers (easily obtained if you have an open 
  peering policy at a few european IXs).
* The possession of someone's noc e-mail address implies nothing about 
  a relationship between the sender and the recepient. That is why 
  50% of the noc tickets that come in are spam, 45% are things which 
  are not appropriate for the NOC queue, and only 5% are valid tickets.
* Humans (especially noc monkeys :P) are very fallible when it comes to 
  process. They will misparse the e-mail, misplace the ticket, mis-
  record the event, mistype the search key, etc. The odds of being able 
  to successfully correlate an event at 2am with an e-mail 3 days ago 
  are not good in a busy noc with a lot of shift changes and a lot of 
  tickets.
* E-mail isn't real time. In the example of the emergency reload, there 
  is no way we would even have time to draft an e-mail before reloading 
  let alone have it reach the right people on the other side.

Now think about the advantages of the ADVISORY draft.

* No external contact database or other information is required. I'm not 
  saying that this should replace your well maintained contact database, 
  but it does provide for a way to deliver a message about this BGP 
  session with no external information required. I still receive e-mails 
  from the NOCs of tier 1 networks sent to companies that went away 3 
  acquisitions and 5 years ago, despite numerous notifications that they 
  update their contact information, so don't tell me this is never a 
  problem.
* The existance of the BGP session itself has inherent authentication 
  and message routing properties. I immediately know for every message 
  that a) it is coming from someone who I am directly connected to and 
  thus have a relationship with and b) which session it is about, 
  without any human parsing. This is already lightyears ahead of an 
  e-mail or phone call.
* Because of the inherent authentication mechanism, I don't have to 
  worry about swapping PGP keys or any other nonsense to maintain this 
  avenue of communication. Ask any network operator how much grief the 
  MD5 scare was, how many e-mails were sent trying to coordinate keys, 
  how many peers lost their keys, etc. It turns out that even an 
  activity as trivial as swapping a password is not trivial when you 
  have to do it 1000 times between completely different organizations 
  across the world who may not even speak the same languages.
* This draft allows for real-time delivery of messages, which means 
  things like emergency reloads can now be communicated with sufficient 
  speed.

Without devolving this into bikeshed painting territory any more than it
already is, I think it is clear that there are legitimate needs and uses
for a free-form text message, and that this message is just as much a
part of the legitimate signaling mechanisms of BGP as the fixed-form
notification messages. This draft doesn't replace the need for a soft
notification mechanism or network-to-network communication via e-mail,
but it is one more tool that operators can take advantage of to help run
their networks. If the overwhelming support we've seen from network
operators so far doesn't prove the value they see in this proposal, I
don't know what will.

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 03 Jul 2009 14:24:06 -0400

In message <[email-masked]>
vijay gill writes:
[trimmed]
> was not established and in that case we fall back to OOB communication

Vijay,

If you can get past the open exchange then a message can be sent and
logged before the first update and with a "btw- I think the reason this
session is flapping all day is ...".  If the IP address is wrong or
authentication fails or the session does not establish at all for some
other reason, then its a no go.

It does sound to me to be useful.  Since it can't pollute anything but
the syslogs, it also seems harmless.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 03 Jul 2009 15:06:36 -0400

In message <[email-masked]>
Robert Raszuk writes:
[trim]
> easier to implement this draft then just shoot an advisory message to 

Robert,

The message would more likely be of the form

  At [some time] we plan to do [X] which will inherently test the
  non-stop routing capability of our router and since we have less
  than 100% confidence in our router vendor's implementation of [some
  feature, maybe related to X, maybe NSR], we are notifying you of our
  intended action, just in case...

:-)

That would not remove the requirement for graceful restart, etc.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 03 Jul 2009 15:42:50 -0400

In message <[email-masked]>
> interactive IM chat with the peer I suppose to pass brief live msg.

Routers have consoles?  How about we sent it to the card punch too?
:-)

Seriously the issue is that for a BGP peering to exist there is
generally an agreement that the peer by reachable in a timely fashion,
but this agreement is not enforceable.  The phone or email contact
information is often useless.  The one thing that can be guarenteed is
that there is a BGP session and if that is the only means to get a
message across it might be better than nothing.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 03 Jul 2009 15:45:12 -0400

In message <[email-masked]>
> How are you going to check all of these?

Hint:  automate!

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 03 Jul 2009 15:53:09 -0400

In message <[email-masked]>
Paul Wall writes:
[deleted, see archives]

I believe that personal attacks are prohibited on IETF mailing lists.
[but I'm too lazy to reread the Tao or other docs to try to find it.]

Curtis

---

From: Richard A Steenbergen <[email-masked]>
Date: Fri, 3 Jul 2009 15:06:50 -0500

> > How are you going to check all of these?

I don't think anyone would recommend you use an advisory message to send
communication you definitely want the other side to read or respond to,
e-mail is far more appropriate for this task. The point of an advisory
message is to convey some information which would be useful to figuring
out a problem if and when the other side goes to look, the same way
someone might go look at what NOTIFICATIONs were received when a peer
goes down.

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 03 Jul 2009 16:43:59 -0400

In message <[email-masked]>
> someone might go look at what NOTIFICATIONs were received when a peer

Maybe a one word hint was too terse?  :-)

In a past life we automated collection of information from every
conceivable source and ran it through a small set of (mostly perl)
programs at the NOC.  The automated part of the analysis ranged from
looking for patterns in access to ports that we didn't use (to find
network security scans), to finding near full disks or machines that
had started paging (yes, our routers had disks and would page to disk
and get slow rather than outright fail).  The outcome would be a note
to the NOC that a security scan was occurring, that disk needed
cleanup, that a machine might need a RAM upgrade, etc.  Today, the
patterns to look for and actions to take would differ.  Adding a
message to the NOC that some subset of routers received a recent
ADVISORY would not be hard.

Curtis

---

From: Kristian Larsson <[email-masked]>
Date: Sat, 4 Jul 2009 00:24:49 +0200

> idea? With hundreds of peering sessions between peers and customers?!

I represent AS1257 in this matter. We run a
network covering the better parts of Europe as
well as the US. We have in the hundreds of peers
and we support this proposal.

> How are you going to check all of these?

Automation. I imagine that when we receive a peer
down alarm, we would collect the last ADVISORY
message and display this together with the peer
down alarm in the NOCs alarm system so that they
may get a better view of what happened.
 
> "Call 1-800-ISP for our latest offer"

If that's the message our peers wants to send us
before their peer go down, then so be it, the NOC
would probably laugh at it.

> see my marketing droids rubbing their hands in glee already.

Your marketing droids have enable on your routers?
 
People have scraped peeringdb.com in the past to
sell transit, it usually ends up with some sales
dude getting beaten up by the engineers at the
same company. I don't see this very different -
you could do it, but you would get ignored.
Receiving such a message, I would probably
consider depeering ;)

> monitor where the functionailty exists elsewhere.

At least in our environment, we would ignore the
messages most of the time and really only pay
attention to them when another event has occured
(typically a peer down).

> these were defined as part of the deaft  but its still a pretty terrible

No it wouldn't, free-text for the win :)

> Section 7 - Network Monitoring

If you let customers dictate your contracts you
have another problem.

Kind regards,
   Kristian. 

-- 
Kristian Larsson                                        KLL-RIPE
+46 704 910401			              [email-masked]

---

From: "Neil J. McRae" <[email-masked]>
Date: Sat, 4 Jul 2009 10:15:31 +0100 (BST)

> Hint:  automate!

Reply: More cost!

And how do you automate something usefully based on a free text field?

---

From: Curtis Villamizar <[email-masked]>
Date: Sat, 04 Jul 2009 15:11:44 -0400

In message <[email-masked]>
> On Fri, July 3, 2009 20:45, Curtis Villamizar wrote:

Run your network any way you want.  We found it cheaper to have
programs do the first level probing and problem detection.

> And how do you automate something usefully based on a free text field?

You just notify the NOC that a message has been received on a
particular router (just once until the notice is cleared indicating
someone looked at it, or periodically if no one clears it).  Then you
let a person read the actual message.  Otherwise a person has to poll
the routers and that IMO would be a lot less effective and a lot more
expensive.

I'm not suggesting that you launch an AI research project.

Curtis

---

From: john heasley <[email-masked]>
Date: Tue, 7 Jul 2009 15:31:32 -0700

> to syslog command tied to BGP may be of more use than we can think of at the moment.

this would be a far more useful capability if
a) it were generated automatically by the router when possible; eg:
	send advisory 'over prefix limit'
	send cease
   OR
	send advisory 'reboot'
	send cease
   etc.
b) it had a non-freeform prefix defined by the draft that can be interpretted
   programmatically, like smtp, and the remainder freeform left for humans.
c) were included in the bgp state change or a advisory-specific snmp trap
   and mandated by the draft

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 07 Jul 2009 23:46:06 -0400

>    and mandated by the draft

I'm not sure why this could not be done with the existing
NOTIFICATION.  The Cease code has no subcodes, but 'over prefix limit'
and 'reboot' could certainly be added and there is already a data
field which is unused for Cease and could be defined as human readable
free form, zero padded.

As long as we can live with 256 Cease subcodes this should be OK.  The
catch is where we *don't* want to drop the session.

The original question was "should this be a WG draft".  The answer
seems to be "yes" and we are already debating the details.

Curtis

---

From: john heasley <[email-masked]>
Date: Wed, 8 Jul 2009 14:49:09 -0700

> field which is unused for Cease and could be defined as human readable

I believe thats been suggested in past discussion about this draft.

> catch is where we *don't* want to drop the session.

unless soft-notify is added, which has been suggested in this thread.

> seems to be "yes" and we are already debating the details.

I suggest that notification/soft-notify is a better solution, which
implies a code/subcode that can be meaningful to a program.