Subject: Re: [Mtgvenue] limiting our set of cities

From: Alissa Cooper <[email-masked]>
Date: Thu, 20 Feb 2020 15:42:46 -0500

Hi Chris, all,

As you have noted, we have a working group, MTGVENUE, that is chartered as "the forum where the IETF community can discuss and agree on what should go into the policies, the selection process, and the detailed criteria” for meeting venue selection. Please direct discussion of these topics to [email-masked] <mailto:[email-masked]>.

Thanks,
Alissa

> On Feb 20, 2020, at 3:19 PM, Christian Hopps <[email-masked]> wrote:
> 
> 
> 
>> On Feb 20, 2020, at 12:57 PM, Keith Moore <[email-masked]> wrote:
>> 
>> On 2/20/20 9:13 AM, Christian Hopps wrote:
>> 
>>>> On Feb 20, 2020, at 8:17 AM, Keith Moore <[email-masked]> wrote:
>>>> 
>>>> On 2/20/20 7:46 AM, Christian Hopps wrote:
>>>> 
>>>>> I think that we should pick the top 12-16 locations that participants are from, then for each destination prior to it being considered we calculate the travel PAIN (cost + time) for that set of participants.
>>>> Why favor participants from large cities?   It's not like they're representative of the whole group.
>>> By trying to make it easier for the most people, of course it's going to be helping areas with the most people. The point is to obtain a list of sites to measure travel cost and time from.
>> I think that's statistically incorrect.   The likely effect of what you propose would be to artificially discourage participation by people not living in large cities.
> 
> It would only be statistically incorrect if most people didn't live in large metropolitan areas/cities. I seriously doubt the majority of IETF participants are rural.
> 
> FWIW *I* am rurally located. :)
> 
> I'm a plane ride away from (DTW/MSP/ORD) airports and I'm perfectly fine with the Detroit(AA)/Chicago/Minneapolis representing me with their pain. I'll always have that extra plane ride, and the rest of the trip will be minimized along with those "nearby" large city folks.
> 
>>>> Also, different participants have different ideas of pain.
>>> I think it's reasonable to equate "painful" with travel time and cost.
>> 
>> For me, pain includes those things but also includes things like the availability of food that is compatible with my diet.   And not just travel time but discomfort, which is nonlinearly related to travel time.
> 
> But what I've been talking to is the travel effort that is talked about in the mtgvenue documents/policy. Dietary restrictions are not a part of travel effort/pain. I think we need to avoid getting too abstract here until we figure out how to at least get the easy metrics optimized.
> 
>> 
>>> Do we really want people who love to travel and couldn't care less where we go to be diluting the "pain pool" for the measurement?
>> I have yet to meet anyone who doesn't have a preference about where they travel.   But I think that letting everyone decide for themselves what is painful and whether the meeting location is an effective enough place to work to justify the trip, is much better than having one person unilaterally declare what the criteria should be.
>>>> A fairer method would be to poll every participant about their preferences for future meeting cities, then for each meeting, pick N polled participants at random from those who have attended the last M meetings (locally or remotely), and select from the cities show up in their preference lists.
>>> Ok as long as N is large and M is reasonable. But making N large is just going to skew to large metro areas being heavily represented anyway. *shrug*
>> 
>> The difference is that the chance of considering a small city as a location would be in proportion to the number of attendees from small cities.
> 
> I can think of many reasons we don't want to meet in a small city, even if (and again I seriously doubt this) a majority of participants were from small cities. The point is many people to gather in one place. Large cities have already been optimized for exactly that condition.
> 
>> But I also think that minimizing pain (however defined) is not the right overall criterion - it should instead be to maximize effective participation (of which pain is certainly a factor but not the only one).   Granted, that quantity is even harder to define than pain.   But several people have observed that the meeting venue has a lot of effect on the ability to get work done.
> 
> Correct, and there was a WG mtgvenue that put out documents on this. Travel pain is one of the factors in deciding a venue. That's all I've been commenting on, but not b/c I think the other criteria should be ignored.
> 
> Thanks,
> Chris.
> 
>> 
>> Keith
>> 
>> 
>

---

From: John C Klensin <[email-masked]>
Date: Fri, 21 Feb 2020 09:27:59 -0500

Jay,

(moving this to mtgvenue, per Alissa's suggestion, because the
dead horse has been adequately kicked on the IETF list, and
because I agree with Andrew that it would be good to let the
recent model run for a bit before we start second-guessing it)

--On Friday, February 21, 2020 07:37 +1300 Jay Daley
<[email-masked]> wrote:

>> I also admit to being a little confused by there being
>> several cities that were just added to the list of those
>> assessed as suitable yet already have meetings scheduled or
>> very recently hosted meetings (Philly, San Fran, Vancouver,
>> Singapore). Can you shed some light on this seeming
>> discrepancy? 
> 
> This was an administrative oversight on our part - basically
> it fell through the cracks in the transition from the IAOC to
> the LLC.

I was surprised by another omission/ apparent discrepancy.  We
met many times, IMO very successfully, in Minneapolis, often in
the winter.  Many of us didn't like the cold, others did, others
were not bothered.  There is an international airport with
direct flights to multiple cities in Europe and Asia as well as
many North American cities and plausible connections to Latin
America and the South Pacific.  It is definitely not a tourist
destination, especially in the winter so, assuming we can avoid
other meetings in the same hotel at overlapping times (should be
easier given our present scale), meetings there are fairly free
of distractions.

I'm not necessarily advocating going back there, but why isn't
it on the list as, at least, being evaluated?  More generally,
if there are other cities where we have successfully met
multiple times that are not on the list (there may not be), why
are they not listed?   I know that one of your predecessors
developed an intense personal dislike for the place (I've never
understood why), but one person's dislike, no matter what
position he or she holds, does not seem like a very good reason
to exclude a proven location from the list.

thanks,
   john

---

From: "Andrew G. Malis" <[email-masked]>
Date: Fri, 21 Feb 2020 10:13:32 -0500

Jay,

+1 to everything John said.

As you clean up the list and get to the point where cities can again be
nominated, it would be instructive to look back at past IETF cities that
aren't on the current list and re-evaluate them. Minneapolis was a GREAT
city for the IETF. In addition to what John said, the hotel was well-laid
out with wide hallways, and many lunch and dinner restaurants can be
reached without ever stepping outdoors thanks to the network of skywalks
and tunnels interconnecting city buildings. And we had some pleasant
socials there as well.

(I have to admit being a bit biased - I co-hosted one of the Minneapolis
meetings.)

Cheers,
Andy

On Fri, Feb 21, 2020 at 9:28 AM John C Klensin <[email-masked]> wrote:

> _______________________________________________
> Mtgvenue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/mtgvenue
>

---

From: John C Klensin <[email-masked]>
Date: Fri, 21 Feb 2020 10:38:00 -0500

Andrew,

I would actually advocate going a step further.  If we have held
multiple successful meetings with O(1000) attendees (not just a
few hundred or fewer) in a particular city, I think that city
should be treated as nominated and its omission as an error
because actual success should be considered at least a jump
start on the evaluation process.   Put differently, we should
learn from our successes, not just our failures or a desire to
explore new places and opportunities.   We don't _need_ to go
back there, but its omission from the assessment list seems
inappropriate.

Jay, another questions or suggestion: The introductory paragraph
at https://ietf.org/how/meetings/admin/meeting-location-input/
says 

	"This is the list of cities that have been assessed for
	suitability as a location for an IETF meeting or are
	still in the process of being assessed."

There are no cities listed as "not suitable".  Does that imply
that every city that has been evaluated has turned out to be
suitable (a result I'd find a bit odd but not impossible) or
that places deemed unsuitable are simply being dropped from the
web page?   I think the latter would be a bad idea if only to
save renominations.  Listing them might lead to some
second-guessing debates but they are probably inevitable and, in
the interest of transparency...

   best,
    john

--On Friday, February 21, 2020 10:13 -0500 "Andrew G. Malis"
<[email-masked]> wrote:

> As you clean up the list and get to the point where cities can
> again be nominated, it would be instructive to look back at
> past IETF cities that aren't on the current list and
> re-evaluate them. Minneapolis was a GREAT city for the IETF.
> In addition to what John said, the hotel was well-laid out
> with wide hallways, and many lunch and dinner restaurants can
> be reached without ever stepping outdoors thanks to the
> network of skywalks and tunnels interconnecting city
> buildings. And we had some pleasant socials there as well.
> 
> (I have to admit being a bit biased - I co-hosted one of the
> Minneapolis meetings.)
> 
> Cheers,
> Andy
> 
> 
>> > it fell through the cracks in the transition from the IAOC
>> > to the LLC.
>> 
>> met many times, IMO very successfully, in Minneapolis, often
>> in the winter.  Many of us didn't like the cold, others did,
>> others were not bothered.  There is an international airport
>> with direct flights to multiple cities in Europe and Asia as
>> well as many North American cities and plausible connections
>> to Latin America and the South Pacific.  It is definitely not
>> a tourist destination, especially in the winter so, assuming
>> we can avoid other meetings in the same hotel at overlapping
>> times (should be easier given our present scale), meetings
>> there are fairly free of distractions.
>> 
>> multiple times that are not on the list (there may not be),
>> why are they not listed?   I know that one of your
>> predecessors developed an intense personal dislike for the
>> place (I've never understood why), but one person's dislike,
>> no matter what position he or she holds, does not seem like a
>> very good reason to exclude a proven location from the list.
>> 
>> thanks,
>>    john
>> 
>> 
>>

---

From: Bob Hinden <[email-masked]>
Date: Fri, 21 Feb 2020 07:42:00 -0800

Please, do we have to talk about Minneapolis again.   There were lots of reason why we stopped going to MPLS.  We outgrew the hotel, the hotel got tired of us, there are (as far as I can tell) no direct flights out side of North America.   It was only luck we never got snowed at the times we meet.

It seem to me that when the old IAOC decided it no longer needed a meetings committee and the LLC replaced the IAOC a lot of institutional knowledge was lost.

Bob

> On Feb 21, 2020, at 7:13 AM, Andrew G. Malis <[email-masked]> wrote:
> 
> Jay,
> 
> As you clean up the list and get to the point where cities can again be nominated, it would be instructive to look back at past IETF cities that aren't on the current list and re-evaluate them. Minneapolis was a GREAT city for the IETF. In addition to what John said, the hotel was well-laid out with wide hallways, and many lunch and dinner restaurants can be reached without ever stepping outdoors thanks to the network of skywalks and tunnels interconnecting city buildings. And we had some pleasant socials there as well.
> 
> (I have to admit being a bit biased - I co-hosted one of the Minneapolis meetings.)
> 
> Cheers,
> Andy
> 
>

---

From: Victor Kuarsingh <[email-masked]>
Date: Fri, 21 Feb 2020 10:54:53 -0500

I would agree with Bob here and that's in line with what Jason had noted on
the first thread.  Cities and venues can change over time vs. our needs.
In this case, I think it appears to be true.   Also, as Bob noted, there
are not a lot of non-stop flights from outside of NA to that
destination which is a larger consideration now that we have more attendees
from outside of NA.

Being from Canada, snow does not bother me, but I would say that we should
not forget as we increase our base of attendees, many of those folks may
not be accustomed to the cold or the snow.  This includes making them
needlessly uncomfortable and making folks buy clothing they may not have.
 People tend to be better prepared for warm and mild weather vs. very cold
weather.   I think avoiding weather related issues should be a
factor/consideration (just like holding a meeting on the south east coast
of the US may not be a good idea in Hurricane season such as July).

regards,

Victor K

On Fri, Feb 21, 2020 at 10:42 AM Bob Hinden <[email-masked]> wrote:

> Please, do we have to talk about Minneapolis again.   There were lots of
> reason why we stopped going to MPLS.  We outgrew the hotel, the hotel got
> tired of us, there are (as far as I can tell) no direct flights out side of
> North America.   It was only luck we never got snowed at the times we meet.
>
> It seem to me that when the old IAOC decided it no longer needed a
> meetings committee and the LLC replaced the IAOC a lot of institutional
> knowledge was lost.
>
> Bob
>
>

---

From: Mary Barnes <[email-masked]>
Date: Fri, 21 Feb 2020 09:57:41 -0600

I just checked and Minneapolis does have direct flights from Europe (maybe
they didn't previously).  At least about the same number of locations as
does Austin, TX, which only has direct flights to 4 European cities and no
Asian.  And, suggesting that we can guess bad weather and arrange meetings
to totally avoid isn't realistic.  We had that flood last time in Dallas.
  And, there was the bad Tornados in Dallas in October 2019:
https://www.weather.gov/fwd/tornadoes-october2019   And, remember the
disaster in Dallas with the snow/ice storm around the super bowl time.  At
least Minneapolis has the infrastructure to deal with snow.  And, Austin
had an issue last year due to lots of rain and contamination of the
drinking water.    So, you can have problems with weather and
infrastructure anywhere.

Mary.

---

From: Mary Barnes <[email-masked]>
Date: Fri, 21 Feb 2020 09:59:44 -0600

Well, October isn't tornado season in Texas. So, again, supposing we can
work around weather and other conditions over which we have no control
whatsoever shouldn't be a key factor.  And, I don't it is in the agreed
document.

Regards,
Mary.

On Fri, Feb 21, 2020 at 9:55 AM Victor Kuarsingh <[email-masked]> wrote:

> I would agree with Bob here and that's in line with what Jason had noted
> on the first thread.  Cities and venues can change over time vs. our
> needs.  In this case, I think it appears to be true.   Also, as Bob noted,
> there are not a lot of non-stop flights from outside of NA to that

---

From: Victor Kuarsingh <[email-masked]>
Date: Fri, 21 Feb 2020 11:49:18 -0500

Mary,

It's more of a suggestion and only represents an opinion.  But I think that
somewhat predictable weather issues should be taken into consideration
since they are somewhat predictable.    I agree Tornado season for Texas is
hit and miss.  That said, it's quite likely, in July, the US East Coast
would get a Hurricane so a meeting there could be impacted.  Also, Snow in
Northern US and Canada is also very likely in winter (Nov).    Again, I say
this not due to my personal concerns, but considering others.

I am happy to drop this topic as well since it's likely bearing no fruit.

regards,

Victor K

On Fri, Feb 21, 2020 at 10:59 AM Mary Barnes <[email-masked]>
wrote:

>> Being from Canada, snow does not bother me, but I would say that we
>> should not forget as we increase our base of attendees, many of those folks
>> may not be accustomed to the cold or the snow.  This includes making them
>>> > As you clean up the list and get to the point where cities can again
>>> be nominated, it would be instructive to look back at past IETF cities that

---

From: Ted Lemon <[email-masked]>
Date: Fri, 21 Feb 2020 11:49:36 -0500

On Feb 21, 2020, at 10:57 AM, Mary Barnes <[email-masked]> wrote:
> I just checked and Minneapolis does have direct flights from Europe (maybe they didn't previously).  At least about the same number of locations as does Austin, TX, which only has direct flights to 4 European cities and no Asian.  And, suggesting that we can guess bad weather and arrange meetings to totally avoid isn't realistic.  We had that flood last time in Dallas.    And, there was the bad Tornados in Dallas in October 2019: https://www.weather.gov/fwd/tornadoes-october2019 <https://www.weather.gov/fwd/tornadoes-october2019>   And, remember the disaster in Dallas with the snow/ice storm around the super bowl time.  At least Minneapolis has the infrastructure to deal with snow.  And, Austin had an issue last year due to lots of rain and contamination of the drinking water.    So, you can have problems with weather and infrastructure anywhere.  

Minneapolis is a hub for Delta, so it’s not too surprising that this is the case.   If you are flying on United (/partners) or American (/partners) then you will have to change planes in Dallas, Denver or Chicago, depending on where you are coming from.   So this is perhaps a bit of an inconvenience.   That said, Vancouver is insanely inconvenient from the east coast.  I have in the past actually flown into Seattle and taken the train to avoid the hassle.   But of course this just creates a different hassle.   I like Vancouver and am not suggesting avoiding it, but I do not think it’s any more convenient than Minneapolis if you are willing to be flexible about airlines.

Like Andrew, I lament the absence of Minneapolis from our list, mostly for the utterly silly reason that I have friends who live there whom I have not seen f2f in quite some time because it’s a bit of a schlep just to catch up over lunch.

---

From: John C Klensin <[email-masked]>
Date: Fri, 21 Feb 2020 12:06:37 -0500

--On Friday, February 21, 2020 07:42 -0800 Bob Hinden
<[email-masked]> wrote:

> Please, do we have to talk about Minneapolis again.   There
> were lots of reason why we stopped going to MPLS.  We outgrew
> the hotel, the hotel got tired of us, there are (as far as I
> can tell) no direct flights out side of North America.   It
> was only luck we never got snowed at the times we meet.

Bob,

I have tried, carefully, to not argue for going back to
Minneapolis.  Speaking personally, I really don't care and there
are lots of other cities on the list.  More important, as long
as we are systematically avoiding venues in the US because of
concerns about visa policies, all of the US cities on the list
are irrelevant, "suitable" or not, and adding another irrelevant
one would change nothing.

I am arguing for a process that is open, transparent, and
predictable.  When we moved away from the oft-secretive meetings
committee, I thought part of the goal was increase openness,
transparency, and predictability of the process, with the public
list of cities being assessed as an important part of that
process.  

One of the other advantages is that dubious assertions or
reasons to identify a location as not suitable can be
cross-checked by the community.  Taking your last two sentences
are examples:

* A very quick search yields the information that there are
non-stop flights from MSP to Amsterdam, London, Paris,
Reykjavik, and Tokyo, in addition to multiple locations in the
Caribbean, Mexico, and Belize (North American on my map but
maybe not in all definitions).  That is several more than "no
direct flights outside..." and several more than several North
American cities listed as "suitable".  Or perhaps Minneapolis is
being confused with, e.g., Austin or Ottawa, neither of which
has non-North American flights to anywhere but London but both
of which are "suitable".

* The risk of getting snowed in is indeed, a reason to avoid
Minneapolis.  It would also be a good reason to avoid Calgary
and Denver, both listed as Suitable, and probably other cities
on the list I didn't scan down far enough notice. Anecdotally,
I've been snowed in and delayed for more than 24 hours at
airports in Denver and Chicago (another place we've met
successfully more than once that is not on the list).
Similarly, the risk of floods or hurricanes would be reasons to
avoid Houston, New Orleans, and a number of Caribbean locations,
most of which are identified as "suitable".   Those problems are
also seasonal: to the best of my knowledge, no one has ever been
snowed in or out of Minneapolis in July but the suitability
criteria don't seem to reflect that.

I'll skip more examples in the interest of not making this
longer.  But, again, what I'm after is a rational, consistent,
open, and transparent process.   Had Minneapolis,  Chicago, and
(in principle) Orlando been listed and listed as "unsuitable",
we might be having a discussion, but it certainly wouldn't be
_this_ discussion.

best,
    john

(And, Jay, my apologies -- there are cities on the list marked
"not suitable"; I just had not spotted them before.)

---

From: Adam Roach <[email-masked]>
Date: Fri, 21 Feb 2020 11:19:21 -0600

On 2/21/2020 11:06 AM, John C Klensin wrote:
>   Or perhaps Minneapolis is

Just for the record, there are nonstops to Austin Bergstrom from 
London/LHR (British Airways), London/LGW (Norwegian Air), Amsterdam 
(KLM), Frankfurt (Lufthansa), and Paris (Norwegian Air).

/a

---

From: Fred Baker <[email-masked]>
Date: Fri, 21 Feb 2020 09:41:18 -0800

Minneapolis fell off the list due to the difficulty of the Chinese getting visas for a string of meetings - a US State Department issue that affected us. I personally like Minneapolis in December (January could be quite a bit different) - the city has facilities to enable people to get around easily without walking outdoors, and it works.

I tend to think that the original reason should either be applied consistently across US locations, or abandoned as not specifically about Minneapolis.

My two yen, yours may vary.

> On Feb 21, 2020, at 6:27 AM, John C Klensin <[email-masked]> wrote:
> 
> Jay,
>

---

From: Fred Baker <[email-masked]>
Date: Fri, 21 Feb 2020 09:54:02 -0800

> On Feb 21, 2020, at 7:38 AM, John C Klensin <[email-masked]> wrote:
> 

My guess is that cities deemed "unsuitable" are quite a few more than the ones deemed "suitable", but exclusion might be for any number of reasons and may have happened simply in the meetings committee before anybody boarded an airplane. Recovering that data would, I suspect, be reasonably difficult. Also, we considered having meetings in a number of places that turned out to not work for reasons other than in the venue-selection RFC; for example, we looked pretty hard at possible meetings in Sydney and Auckland, which were set aside because of cost - not because there was an actual issue with the city (I have never been to Auckland, but consider Sydney a very viable place).

From my perspective, I would rather see people nominate cities they would like investigated than try to resurrect what is now old data.

I also suspect that we will want to think in terms of a budget for exploring places. We could pretty quickly nominate any city that has an international airport, and task an intern with determining whether each city also has conference facilities of a certain size. That sounds simple and possible, but it would also mean that we suddenly have a whole lot of make-work that we don't know will go anywhere. I would suggest that we confine research to places potential hosts want to hold meetings and have reasonable transportation and conference facilities, and explore at most N options per unit time.

---

From: John C Klensin <[email-masked]>
Date: Fri, 21 Feb 2020 13:39:19 -0500

--On Friday, February 21, 2020 11:19 -0600 Adam Roach
<[email-masked]> wrote:

> Just for the record, there are nonstops to Austin Bergstrom
> from London/LHR (British Airways), London/LGW (Norwegian Air),
> Amsterdam (KLM), Frankfurt (Lufthansa), and Paris (Norwegian
> Air).

Thanks and apologies.  My quick search did not turn those up
although I found an article indicating that Austin was offering
incentives to try to get nonstops from additional European
cities.

   john

---

From: John C Klensin <[email-masked]>
Date: Fri, 21 Feb 2020 14:22:31 -0500

--On Friday, February 21, 2020 09:54 -0800 Fred Baker
<[email-masked]> wrote:

>...
> From my perspective, I would rather see people nominate cities
> they would like investigated than try to resurrect what is now
> old data.

Partially for the reason you give below, I don't think we should
be casually discarding cities where we have had multiple
successful experiences.  The list of those is not very long but
we know far more about them from actual meeting experience than
we are going to find out from any amount of investment spend on
site visits.  Whether the data are old enough to be irrelevant
is, of course, another issue.  I'd say "too old" for meetings
before about 2000 or 2005 but at least "maybe not" for most
recent ones; YMMD.

Minneapolis is probably not a good example because there are
many people who really liked it as a location and many who
really hate it.  It seems clear to me that there are enough
people in each group that trying to assess their relative sizes
would not be productive.  Oddly, if we pretended we had never
been there before and assessed it as a new location, I think it
would come out as "suitable", at least with adjustments for
season that would affect some other "suitable" places at least
as strongly.  

San Diego might be another interesting example: we have met
there several times (if one goes back before 2000, maybe as many
or more times than anywhere else).  There are direct
international air connections to both Europe and East Asia
(although not a huge number of destinations in either), and the
odds of getting snowed in are rather low.  But it isn't on the
list either. 

And, as you pointed out in your other note, if we are going to
rate Minneapolis as "unsuitable" because of issues with US visa
policies, we need to take every other city in the US off the
"suitable" list until those policies change or it is clear that
they are changing for the better.

> I also suspect that we will want to think in terms of a budget
> for exploring places. We could pretty quickly nominate any
> city that has an international airport, and task an intern
> with determining whether each city also has conference
> facilities of a certain size. That sounds simple and possible,
> but it would also mean that we suddenly have a whole lot of
> make-work that we don't know will go anywhere. I would suggest
> that we confine research to places potential hosts want to
> hold meetings and have reasonable transportation and
> conference facilities, and explore at most N options per unit
> time.

I generally concur, at least to the extent of believing it would
be useful to prioritize cities for exploration based on some
rational and consistent criterion so as to minimize the cost of
evaluations that are likely to be pointless.  I do think that
"we already know a lot because we've met there at least a couple
of times, especially if that is 'several times'" is a good
criterion for prioritization because a high-information
evaluation should be inexpensive whether it produces
"conclusively 'no'" (Orlando comes to mind but maybe it has its
fans too), "conclusively 'yes'", or "strong subjective feelings
that are too contradictory to easily reach a conclusion" (for
which Minneapolis is almost certainly an example). 

best.
   john

---

From: Jay Daley <[email-masked]>
Date: Sat, 22 Feb 2020 10:44:54 +1300

Replying to a whole bunch of emails:

I have no idea how cities originally got onto this list, or why some cities are not there or why some cities we have been too are no longer there.  I’m also not sure what is assessed at what stage (which I will go away and find out).  The current state is that I’ve inherited an opaque, historical process where the secretariat had done their job some time ago but the list had not been updated and the community not informed, and so I corrected that.

My future focus is on defining an overall assessment pipeline which implements the consensus policy. My initial thoughts about that are that this definition should include:

- how things enter the pipeline.  I expect this will have multiple entry points depending on who is asking and probably different requirements for each, such as:
— sponsors, who offer to pay for specific cities/regions and who will probably have the lowest burden of all
— venues who send us bids with special pricing who will probably have the highest burden and be asked to do substantial work if they want their bid considered
— local community members, who will probably be expected to do some initial research
— community members who have preferred cities for an endless set of reasons
- what stages there are in the assessment and what is assessed at each stage
- what community engagement takes place and when

The detailed assessment process is already in place and ably managed by the secretariat so this will be more an exercise in clear definitions than changing the underlying details.

One exception to that and something you are going to see from me a lot, is that I prefer to use independent empirical data for complex decisions instead of human judgement.  So when it comes to airport links I will be looking to see if there is a public index of airport connectivity that we can refer to.  Similarly, comments have been made about some cities being unsuitable due to air-pollution and I will be looking at the standard air-quality indices to guide us there.  Obviously there are nuances and ranges for which human judgement must be used, but the index should be the starting point not an email thread about the number of flights in/out.

Finally, there are actually two cities listed as unsuitable - Nassau and Macau - for the reasons given in my announcement.

Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]
+64 21 678840

---

From: John C Klensin <[email-masked]>
Date: Fri, 21 Feb 2020 17:15:48 -0500

Jay,

I see your response below as very encouraging.  I agree with you
about striving for objective measurements as well, but hope we
can avoid the common trap of confusing "hard to measure" with
"unimportant".  

Let me give threeexamples from recent discussions.  (1) Jordi's
passionate argument against meeting in Barcelona because of
potential serious disruption (and the parallels it might have in
Hong Kong) are not easily reduced to objective and quantitative
measures and I'm sure we could find people on the other side of
either position who would be equally passionate.  Both are not
only hard to quantify or scale objectively but they require
judging risks that cannot be known exactly unless one can
accurately predict the future (one could say the same thing
about the odds of getting snowed in during a given week in
Minneapolis (or Chicago, or Calgary, or Stockholm or Helsinki)).
However, that does not diminish the importance of those issues.
(2) There are several cities on the list that have extensive
international and intercontinental direct air connections
--something that, as you point out, can be evaluated objectively
-- but only if one includes carriers that a significant number
of people may find problematic and/or that some companies have
on "do not use" lists.   The latter is going to be much harder
to get good, objective, data about about, but it can be almost
as real an issue in deterring travel as places that require
itineraries with multiple connections for many participants.
(3) Travel or other departments push back aggressively on what
they see as "tourist destinations", at least requiring extra
justification.  I'd guess we pretty much know what those
destinations are (although we might quibble about edge cases)
but I can't think of a way to either make that categorization
objective or to quantify the number of people who would be
effected (especially if we ignore whatever historical
information we have because things change).   Again, that does
not make the issues less important.

In any even, the general strategy seems sound.  Good luck with
it.   And, with this note, I'm going to stop posting to this
thread in the hope that it will help you get on with it.

regards,
    john

--On Saturday, February 22, 2020 10:44 +1300 Jay Daley
<[email-masked]> wrote:

> I have no idea how cities originally got onto this list, or
> why some cities are not there or why some cities we have been
> too are no longer there.  I'm also not sure what is assessed
> at what stage (which I will go away and find out).  The
> current state is that I've inherited an opaque, historical
> process where the secretariat had done their job some time ago
> but the list had not been updated and the community not
> informed, and so I corrected that.
> 
> My future focus is on defining an overall assessment pipeline
> which implements the consensus policy. My initial thoughts
> about that are that this definition should include:
> 
> - how things enter the pipeline.  I expect this will have
> multiple entry points depending on who is asking and probably
> different requirements for each, such as: — sponsors, who
> offer to pay for specific cities/regions and who will probably
> have the lowest burden of all — venues who send us bids with
> special pricing who will probably have the highest burden and
> be asked to do substantial work if they want their bid
> considered — local community members, who will probably be
> expected to do some initial research — community members who
> have preferred cities for an endless set of reasons - what
> stages there are in the assessment and what is assessed at
> each stage - what community engagement takes place and when
> 
> The detailed assessment process is already in place and ably
> managed by the secretariat so this will be more an exercise in
> clear definitions than changing the underlying details.
> 
> One exception to that and something you are going to see from
> me a lot, is that I prefer to use independent empirical data
> for complex decisions instead of human judgement.  So when it
> comes to airport links I will be looking to see if there is a
> public index of airport connectivity that we can refer to.
> Similarly, comments have been made about some cities being
> unsuitable due to air-pollution and I will be looking at the
> standard air-quality indices to guide us there.  Obviously
> there are nuances and ranges for which human judgement must be
> used, but the index should be the starting point not an email
> thread about the number of flights in/out.
> 
> Finally, there are actually two cities listed as unsuitable -
> Nassau and Macau - for the reasons given in my announcement.
> 
> Jay

---

From: Michael Richardson <[email-masked]>
Date: Sun, 23 Feb 2020 17:22:43 -0500

> has non-North American flights to anywhere but London but both

I was very confused by the statement about no-direct flights to MSP.
(None from Ottawa, for me, but that's because Ottawa is definitely not
suitable... in a dozen different ways. Frankly, Ottawa shouldn't even have
it's own airport.)

    > Minneapolis.

Actually, if you are worried about getting snowed in (**specifically**), a
place that gets snow regularly (like Minneapolis or Ottawa or Calgary) is far
less likely to blink than places like Toronto or NYC, where they don't deal
with it every week :-)
Of course, we could go to MSP in July.

I am for what John said: we have been to at least 50 different cities since
IETF30, which I'll arbitrarily say was when we outgrew university campuses,
(IETF36 was my first, in LA).  And while I'm sure that we can name 10 meeting
locations (not, cities, but specific locations) which we can all agree were
terrible, that leaves around  40 locations for which we have non-trivial
amount of experience with.

Jim Martin was my manager for about 15 months, and I observed how much
new-location scouting he did for the IETF.  Does it need to be done?
Of course.   Do we need to check out 3-4 new locations/year?  I don't know.
Should we attempt to amortize all the planning by returning two or three
times over 5-6 years?  I think so, provided we can do the conditional parts
right.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-