Subject: NEW AMS AT&T IPV6 CIRCUIT READY - TEST REQUEST

From: Glen <[email-masked]>
Date: Sat, 30 Jul 2011 17:47:44 -0700

Dear IETF Leadership and IPV6 Experts:

As you know, for some time AMS has been planning an upgraded primary internet
connection for the IETF and our other clients (including the RFC Editor.)  I
am pleased to announce that, as of earlier today, the installation and initial
testing of that new connection has been completed, and the new circuit is ready
for us to use.

Our old/current circuit is a 10MPBS fiber connection, which had workable IPV6
service that has served us well.  However, although it was presented natively
to us at the router, that IPV6 service did have some tunnelling components in
it, and that was not the optimal configuration.

The new circuit is a 45MBPS fiber connection, so there is a huge speed increase
right from the start.  In recent days, especially during the last meeting, I
and others saw some slowness during peak meeting times... this should return 
us to full speed and alleviate all of those issues.  (We are also maintaining
additional infrastructure to further calm traffic behind the scenes.)

But more importantly, and the reason I'm emailing each of you, is that this
new circuit has true native dual-stack IPV6 service, complete with BGP, and 
the ability for us to add and support additional blocks, connections, AS 
numbers, and so forth.

I am prepared to move the IETF over to the new circuit now, but before I do,
I thought it best to contact each of you and let you know what's happening.
It was actually Iljitsch who reminded me that I wanted to do this... as I was
looking for a specific bit of data about BGP today, Google led me to 
	http://www.bgpexpert.com/
and when I clicked on the link "My BGP Book", I laughed out loud with pleasure
to see Iljitsch's name on the cover.

Like many of you, Iljitsch was instrumental in helping with the IETF website
redesign, and I am asking all of you for your help now.  Before we cut the
IETF over, I'm asking those of you who care to do this type of thing to have
"a look" at our new circuit, run your various tests and examinations that you
are all so good at, and let me know if there are any immediate issues, or
concerns, that I should take back to AT&T before we cut over.

Here are the relevant details:

	Internal IPV4 LAN IP address: 12.22.58.1/24
	Internal IPV6 LAN IP address: 2001:1890:123a::1/56

I know that many of you are busy, and returning home from the IETF, but since
this is so important to you and the IETF generally, it is my hope that those
of you who wish to can have a quick look to make sure that this circuit is
working appropriately, and give me a "thumbs up" or a problem report as 
appropriate.  (To cut down on traffic, I'll ask each of you to reply directly
to me where possible and appropriate, so we don't flood each other.  I know
some of you were duplicated on this email, but I wanted to get the word out
to everyone as quickly as possible.)

Everyone at AT&T has worked extremely hard over the past months and weeks to
bring this circuit to us, and to make it as perfect as possible.  I've told
them that the eyes of the IETF are on this circuit; likewise the eyes of all
of AT&T are now waiting for any report either way.  I would like to cut the
IETF over to this new circuit now, but in order to give everyone time to
respond I am going to wait a few days to do the move.  Please do take whatever
chance you have to look things over, and make sure things are operating to
your collective expectations.  An ounce of prevention now could prevent TONS
of headaches later.

Thank you for your assistance, and I look forward to bringing us online 
quickly.

Glen
Glen Barney
IT Director
AMS (IETF Secretariat)

---

From: Fred Baker <[email-masked]>
Date: Sat, 30 Jul 2011 18:34:40 -0700

> your collective expectations.  An ounce of prevention now could prevent TONS

Not entirely sure what you're asking me to do. Sounds like "generate a bunch of traffic and verify that it works." Close?

I should be home around midnight PDT tonight. I have a script (developed for IPv6 day) that spiders a set of sites. If it is helpful to you, I can run that tonight with a view to finding all the web pages I can find and kicking up some dust. It will be limited to about 2 MBPS in the nature of the case, and as I have an HE tunnel it won't really test PMTU. But if there's something to kick loose there is some probability that it will. Is that what you're looking for?

---

From: Glen <[email-masked]>
Date: Sat, 30 Jul 2011 18:41:38 -0700

On Sat, Jul 30, 2011 at 06:34:40PM -0700, Fred Baker wrote:
> Not entirely sure what you're asking me to do. Sounds like
> "generate a bunch of traffic and verify that it works." Close?
> I should be home around midnight PDT tonight. I have a script
> (developed for IPv6 day) that spiders a set of sites. If it is
> helpful to you, I can run that tonight with a view to finding all
> the web pages I can find and kicking up some dust. It will be
> limited to about 2 MBPS in the nature of the case, and as I have an
> HE tunnel it won't really test PMTU. But if there's something to
> kick loose there is some probability that it will. Is that what
> you're looking for?

Hi Fred -

Thanks for your email!  In response to your question, I guess I would say,
"Well, not exactly."  The IETF is not hooked up to this new circuit yet, so
there is really nothing on it yet.   There is only the router itself, and
our internal default gateway IP address pair, against which people can test.

So my email was targeted more at the IPV6 junkies ... I mean, EXPERTS ... who 
like to do traceroute reports, check path MTU, and do other circuit-specific 
testing things.  

I'm already getting reports from some, for example, who are reporting much 
better rtt and transit times compared to our existing circuit...  a positive 
trend that bodes well for us all.  But I'm equally interested in finding any
negative issues, problems, or red flags pertaining to the new circuit and its
setup and configuration... *before* we cut over.  More of a "due dilgence"
kind of thing.

I personally think the circuit is top-grade, and is going to be just perfect
for us.  But there are many in the IETF who know more about the internal
details of IPV6 than I do.   Those are they who come to me from time to time
with all kinds of less-well-known details about our circuit that they'd like
to see changed.  This new circuit is my response to all those requests.  I
just want to make sure, as much as I can, that everyone is happy with what
they see before I cut us over... or at least that there are no red flags at
this stage.

Thanks!
Glen

---

From: Glen <[email-masked]>
Date: Sun, 31 Jul 2011 09:13:59 -0700

On Sun, Jul 31, 2011 at 02:42:49PM +0200, Iljitsch van Beijnum wrote:
> I'm glad I was able to make you laugh. If your BGP question wasn't
> answered, let me know.

I think I'm good for now...  :-)

> Possible path MTU discovery issues that Fred mentioned are
> probably the most important issue to test for. However, we can only
> really do that if there is a live host running some kind of TCP
> service available behind the circuit. Would it be possible to park a
> temporary HTTP server there and serve up a few files of different
> sizes? Including a few large ones so we can test download bandwidth.
> Iljitsch

I'm moving our other servers over to the new circuit today, so there will
be live hosts later.  I don't have anything temporary I can do, but once I
get, for example, amsl.com moved over I can put some files there you can 
download.

I'm not too concerned about bandwidth - I did extensive testing of that
prior to setting anything up (so I could get a true picture) and the bandwidth
and performance seem perfect.  I'm more concerned about IPV6 issues - anything
that might make us say "Wait" with respect to cutting over.

So far the only negative I've gotten is that from some routes, AT&T still
does not expose their internal routing.  I have raised that issue with them
in the past, and will do so again; however, that's not technically related
to our circuit.  It was more of an issue when we were tunnelling (our old/
current circuit), but with the new circuit it seems like routes just go
directly to us from many different locations, so this is less of an issue
for us.   I don't pretend to understand all the details about "why" they
are doing this - I get that it's undesireable and I know that they're aware
of it - but my leverage to fight that particular battle is limited.

Once the dust settles I will raise it with them again; right now,
I'm just trying to get at whether this circuit is "good" and we want to 
move to it, or whether there is a problem that I need to go back and get
fixed.

Glen

---

From: Fred Baker <[email-masked]>
Date: Sun, 31 Jul 2011 15:19:02 -0700

On Jul 31, 2011, at 5:42 AM, Iljitsch van Beijnum wrote:
> Possible path MTU discovery issues that Fred mentioned are probably the most important issue to test for. However, we can only really do that if there is a live host running some kind of TCP service available behind the circuit. Would it be possible to park a temporary HTTP server there and serve up a few files of different sizes? Including a few large ones so we can test download bandwidth.

Another way of asking the same question I did. Note that I'm on an HE tunnel, and therefore will have PMTU questions anyway - and never had any to complain about.

Begin forwarded message:

> From: Fred Baker <[email-masked]>
> Date: July 31, 2011 3:16:10 PM PDT
> To: Glen <[email-masked]>
> Subject: Re: NEW AMS AT&T IPV6 CIRCUIT READY - TEST REQUEST
> 
> 
> On Jul 30, 2011, at 9:15 PM, Glen wrote:
> 
>> On Sat, Jul 30, 2011 at 08:57:25PM -0700, Fred Baker wrote:
>>> OK. From my home, I'm one of those junkies. What's the address of
>>> the router I should test?
>> 
>> The IPV4 address is 12.22.58.1/24, and IPV6 is 2001:1890:123a::1/56.
> 
> I ran an 8 hour ping study starting this morning - I had an outage when I got home last night.
> 
> --- 12.22.58.1 ping statistics ---
> 28800 packets transmitted, 28800 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 28.817/41.651/1556.058/18.499 ms
> 
> --- 2001:1890:123a::1 ping6 statistics ---
> 28800 packets transmitted, 28738 packets received, 0.2% packet loss
> round-trip min/avg/max/std-dev = 63.073/75.310/1131.349/17.028 ms
> 
> I can't say I see a lot to complain about. Next question will be how they behave under load. When are you planning to put the circuit into play?

The one thing I would complain about in this is the packet loss rate - a ping once a second shouldn't trigger loss. That said, I don't know what link or links the loss was coming from. So I hesitate to lay it at any given given door.

---

From: Glen <[email-masked]>
Date: Sun, 31 Jul 2011 16:04:37 -0700

On Sun, Jul 31, 2011 at 03:19:02PM -0700, Fred Baker wrote:
> Another way of asking the same question I did. Note that I'm on an
> HE tunnel, and therefore will have PMTU questions anyway - and never
> had any to complain about.

I am working now on moving all AMS servers - except for the IETF/ISOC/RFCP
servers - to the new circuit now.  They should be online and good to within
four hours from the time of this message.  After that, you should all be able 
to test against www.amsl.com for both IPV4 and IPV6.  DNS will be up to date,
but it will be 12.22.58.20 and 2001:1890:123a::1:14 when it's live.

Glen