Subject: [vmeet] IETF83 and draft-ietf-genarea-rps-reqs-03

From: SM <[email-masked]>
Date: Mon, 26 Mar 2012 11:23:14 -0700

Hi Paul,

I have a few commenst about IETF 83 and draft-ietf-genarea-rps-reqs-03.
In Section 3.1:

   "At recent meetings, delays of around 30 seconds have been recorded, with
    minimum delays typically being five seconds.  This delay is caused by
    buffering at the hop-by-hop ISPs and in the remote attendee's
    computer."

The delay was around 15 seconds today.  The audio stream stopped 
every 302 seconds (VLC error ES_OUT_RESET_PCR).  I consider it as a 
problem at my end as nobody else reported the issue.

The information at 
http://www.ietf.org/meeting/83/remote-participation.html is not kept 
up to date.  Although EAI was cancelled, it is still listed on that 
web page.  Given the TBD, it seems that nobody pays attention to the 
information provided after the initial publication.

The audio did not work for URNBIS and during the initial 45 minutes 
of the technical plenary.  FWIW, audio for the plenary is generally a 
problem.  Would it be too much to ask for a web page which shows the 
status of the audio streams?

There are a few people (Andrew Sullivan, Dan York for the plenary) 
who helped remote participants follow what was going on.

In Section 4.11:

   "**Requirement 03-58**: Remote attendees SHOULD be able to make
    comments at the mic approximately as well as if they were local
    attendees.  This means that either remote audio to the plenary room
    speakers be available, or that IM-to-room relay be available."

Was anyone designated to channel questions to the microphone for the 
IETF 83 plenary?

Regards,
-sm

---

From: John C Klensin <[email-masked]>
Date: Mon, 26 Mar 2012 15:42:21 -0400

--On Monday, March 26, 2012 11:23 -0700 SM <[email-masked]>
wrote:

> The delay was around 15 seconds today.  The audio stream
> stopped every 302 seconds (VLC error ES_OUT_RESET_PCR).  I
> consider it as a problem at my end as nobody else reported the
> issue.

For comparison purposes, I was seeing/hearing 15-20 seconds of
delay as compared to the transcript (I have no way to know how
close to real-time that was), but audio was smooth and
uninterrupted.  So there are presumably some differences in ISPs
and/or connectivity and/or client equipment.  Figuring which of
those is involved might be important; if it is, the requirements
doc is lacking provisions for remote instrumentation and
sampling.

That said, I think one of the comments made in the plenary about
mobile devices is going to turn out to apply to remote
participation as well: this is going to put stress on shortcuts
and sloppy behavior.  As a nearly-trivial example, if real-time
captioning/transcripts are only for the benefit of those in the
room, it is really nice to have identification of speakers, but
it may not be essential.  If one is remote --and especially if
the audio is broken or substandard-- transcripts without speaker
identification are pretty close to useless.

> The information at
> http://www.ietf.org/meeting/83/remote-participation.html is
> not kept up to date.  Although EAI was cancelled, it is still
> listed on that web page.  Given the TBD, it seems that nobody
> pays attention to the information provided after the initial
> publication.

Already noted to them and to the Meetecho site (which contained
the same difficulty).  But my guess is that having that page at
all is a symptom of a problem (and of the old database principle
that having to maintain the same information in different form
in multiple places is just about a guarantee of problems).  My
guess, after a few days of the meeting (the first session I did
remotely was on Saturday) and remembering the last time I went
through it, is that remote participation is going to be a Fail
for all but the most experienced and determined of users until
we can collapse all of the information on the remote
participation page, plus all of the info that is now in the
tools agenda (or should be), onto a single page and make
maintaining/updating that page a Secretariat responsibility so
that everything stays synchronized, including cancellations and
room changes.

>...
> There are a few people (Andrew Sullivan, Dan York for the
> plenary) who helped remote participants follow what was going
> on.

Hugely helpful.  It would have been much worse without them.  I
don't know how to build them, or the job they were doing, into a
requirements spec but I believe it/they belong there.

> Was anyone designated to channel questions to the microphone
> for the IETF 83 plenary?

Nope.  Dan did manage to channel two questions for me.  But that
raises two other issues:  (1) The person who answered either
didn't understand the main point of the question or choose to
deal with it dismissively (no way to know from here and
certainly not Dan's fault).  Had I been standing at the mic, I
would have immediately responded with "you didn't answer my
question" and followed up as needed.  While I wouldn't expect it
to happen often, it suggests that a requirement that permits
IM-to-audio (and probably one that permits remote audio input)
needs to explicitly allow for follow-up, perhaps by requiring
chairs/ moderators to ask whether the question was answered.
Tedious, but possibly necessary if we are really going to act as
if remote participants have something to offer in their
questions or comments.   (2) If a remote participant submits two
questions, some minutes apart, and it takes whomever is doing
the Jabber-to-audio mapping some time to get to the front of the
line, is a Chair response of "you can't ask the second question
now because the remote person shouldn't monopolize the mic" the
one we want?  Perhaps it is, but the question would seem to need
some careful examination, especially if the person who makes it
to the mic actually has multiple questions from each of multiple
people and when we remember that remote people may be more
likely to have questions to resolve confusion about the
presentation than those who were in the room.

regards,
   john

---

From: SM <[email-masked]>
Date: Mon, 26 Mar 2012 14:23:16 -0700

Hi John,
>close to real-time that was), but audio was smooth and

The transcription delay might be one or two seconds at most.

>doc is lacking provisions for remote instrumentation and

The requirements doc actually have an item (03-63) for testing.  I'll 
test with another client to see whether the same issue occurs.

>and sloppy behavior.  As a nearly-trivial example, if real-time

Yes.

>identification are pretty close to useless.

The transcripts may be useful for non-English speakers.

>in multiple places is just about a guarantee of problems).  My

Yes.

>that everything stays synchronized, including cancellations and

It may help remote participants if small incremental changes could be 
considered in the interim instead of looking at a full-scale solution only.

>requirements spec but I believe it/they belong there.

I don't think you can build them into a requirements specification. 
:-)  There has been a non-IAOC effort which could be realigned to do 
some of that.

>Nope.  Dan did manage to channel two questions for me.  But that

  "**Requirement 03-08**: WG chairs MUST be able to test whether or not
   the tools for their session are working at least 30 minutes before
   the meeting begins (unless, of course, there is already another
   meeting occurring in the room during that time)."

WG could be replaced with "session".

>question" and followed up as needed.  While I wouldn't expect it

The point here is that you were not able to send a follow-up 
question.  It may be different in a WG session.

>presentation than those who were in the room.

BTW, 03-16 discusses about relaying from IM to the mic.  There is 
some discussion about floor control in Section 4.2.2.  In an earlier 
WG session, the ADs picked some (remote) comments which they probably 
considered pertinent to the discussion.  The in-room angle is to 
avoid monopolizing the mic whereas remote comments do not always have 
to be FIFO.

Regards,
-sm

---

From: John C Klensin <[email-masked]>
Date: Mon, 26 Mar 2012 19:15:59 -0400

Hi,

--On Monday, March 26, 2012 14:23 -0700 SM <[email-masked]>
wrote:

>...
> The requirements doc actually have an item (03-63) for
> testing.  I'll test with another client to see whether the
> same issue occurs.

Yes, I know.  But audio was "tested".  After all, it worked in
Appsawg in the morning.   It will be interesting to hear about
what actually happened from Verilan (I hope someone will ask at
Wednesday's plenary, since we usually don't get more details
than "fixed") but, unless there is evidence that this was a
truly unique case, "testing" needs to include someone or
something remote verifying reachability and audio delivery on
all channels at, say, 15 minute intervals from 0730 until after
the last session each day (0730 and not 0855 to give them a
chance to fix things if there is an alarm).

>...
>> captioning/transcripts are only for the benefit of those in
>> the room, it is really nice to have identification of
>> speakers, but it may not be essential.  If one is remote
>> --and especially if the audio is broken or substandard--
>> transcripts without speaker identification are pretty close
>> to useless.
> 

I know that.  And non-English participants (and hearing-impaired
ones) whether they are in the room or not.  But the transcripts
are useful _remotely_ only if they identify speakers -- or we
have video as well as audio and the video covers the floor mics
as well as the podium.

>...
>> guess, after a few days of the meeting (the first session I
>> did remotely was on Saturday) and remembering the last time I
>> went through it, is that remote participation is going to be
>> a Fail for all but the most experienced and determined of
>> users until we can collapse all of the information on the
>> remote participation page, plus all of the info that is now
>> in the tools agenda (or should be), onto a single page and
>> make maintaining/updating that page a Secretariat
>> responsibility so that everything stays synchronized,
>> including cancellations and room changes.
> 
> It may help remote participants if small incremental changes
> could be considered in the interim instead of looking at a
> full-scale solution only.

Indeed.  And a consolidated agenda (superset of the current
"tools" one) is one of those things that could be done
relatively quickly although not this week.  As something that
might, in principle, be done even sooner, I'd like to see the
transcribers equipped with a pushbutton and a large red light
(ideally rotating and/or flashing) -- if they can't get a name
well enough to record it, they turn on the light and things
simply stop until the get that information.  I'd lay good odds
that someone could purchase such a device in Paris or build it
from parts readily available there and have it in service before
the Wednesday plenary :-)

>> Hugely helpful.  It would have been much worse without them.
>> I don't know how to build them, or the job they were doing,
>> into a requirements spec but I believe it/they belong there.
> 
> I don't think you can build them into a requirements
> specification. :-)  There has been a non-IAOC effort which
> could be realigned to do some of that.

Right.  But the practical alternative to that sort of human
intervention is more discipline and better tools.

> WG could be replaced with "session".

We won't know whether that would have helped with whatever
happened before or during the first afternoon session Monday
until and unless we get a post mortem on what caused the
problem.  For the second afternoon slot (URNBIS, etc.) and the
Plenary, we shouldn't have had to rely on test tools because the
problem was already in "working on it" state.  And that brings
us back to a suggestion you or someone else made earlier: a web
page that reports, in real time, whether the NOC thinks things
are working, what they think isn't, and what the status and
expectations are about the latter.  That would have the added
advantage of cutting down the "wtf, is foo really broken"
traffic to the IETF and meeting lists.

> question.  It may be different in a WG session.

<rant>
IMO, shouldn't be.  If it is, then the plenaries have ceased to
be useful dialog between the IAB/IESG/presenters/ etc. and the
community could save a lot of time and free up WG slots by
preparing the plenary sessions in advance and making them
available as prerecorded webcasts that people could pick up at
their convenience.  The elimination of the real-time requirement
would even permit video.  I am _not_ suggesting that: I much
prefer that plenaries actually permit dialogue and encourage
thinking by the presenters as well as the audience.  But...
</rant>

>> questions or comments.   (2) If a remote participant submits
>> two questions, some minutes apart, and it takes whomever is
>> doing the Jabber-to-audio mapping some time to get to the
>> front of the line, is a Chair response of "you can't ask the
>> second question now because the remote person shouldn't
>> monopolize the mic" the one we want?  Perhaps it is, but the
>> question would seem to need some careful examination,
>> especially if the person who makes it to the mic actually has
>> multiple questions from each of multiple people and when we
>> remember that remote people may be more likely to have
>> questions to resolve confusion about the presentation than
>> those who were in the room.
> 
> BTW, 03-16 discusses about relaying from IM to the mic.  There
> is some discussion about floor control in Section 4.2.2.  In
> an earlier WG session, the ADs picked some (remote) comments
> which they probably considered pertinent to the discussion.
> The in-room angle is to avoid monopolizing the mic whereas
> remote comments do not always have to be FIFO.

Yes, I understand the principles.  I'm only questioning whether
we understand how to optimize them... and whether or not that
has actually been thought about.

regards,
   john

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 27 Mar 2012 01:21:01 +0200

> chance to fix things if there is an alarm).

Of the 3 WG sessions I was in today, audio was broken in two (WebSec WG
in 242AB and URNBIS WG in 243). Both times, we had to contact the NOC
and someone came into the room to fix things. Once someone showed up,
audio worked. These were not the first sessions of the day. I don't know
if audio had been working earlier in the day in those room but then
broke. I'll follow up about that tomorrow.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: John C Klensin <[email-masked]>
Date: Mon, 26 Mar 2012 19:33:59 -0400

--On Tuesday, March 27, 2012 01:21 +0200 Peter Saint-Andre
<[email-masked]> wrote:
 
>>> ...
> Of the 3 WG sessions I was in today, audio was broken in two
> (WebSec WG in 242AB and URNBIS WG in 243). Both times, we had
> to contact the NOC and someone came into the room to fix
> things. Once someone showed up, audio worked. These were not
> the first sessions of the day. I don't know if audio had been
> working earlier in the day in those room but then broke. I'll
> follow up about that tomorrow.

Peter, 

I wasn't aware that audio ever came up in URNBIS.  That might
have been because it was down for the entire WG session (which I
had assumed) or it might have been that it came up and no one
managed to tell the remote participants to try again.  The
message I (and I think others) were getting at the beginning of
the session was approximately "player can't find file", which,
as I understand the technology, is not a sign of a local audio
equipment failure.  With the plenary, the necessary file(s) were
found, there was just silence, following by unpleasant noises
for the first part of the session, then we were told it was up,
but had to restart player software in order to get it.  I don't
have any way to know whether that was just hardware or not.

Follow up appreciated.

best,
   john

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 27 Mar 2012 01:46:35 +0200

On 3/27/12 1:33 AM, John C Klensin wrote:
> I wasn't aware that audio ever came up in URNBIS.

By the time someone showed up to fix the problem, the WG session was
nearly finished.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: John C Klensin <[email-masked]>
Date: Mon, 26 Mar 2012 20:00:24 -0400

--On Tuesday, March 27, 2012 01:46 +0200 Peter Saint-Andre
<[email-masked]> wrote:

> By the time someone showed up to fix the problem, the WG
> session was nearly finished.

So does that mean that Paul's requirements spec needs to have an
SLA item on how rapidly someone shakes a leg if there is an
outage?  I.e., if, in spite of all the testing and other issues,
a problem is discovered that blocks and has a severe negative
impact on remote participation, how long does it take someone
effective to show up after the problem is reported?

   john

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 27 Mar 2012 02:05:25 +0200

> effective to show up after the problem is reported?

Yes, we might need that. In my limited experience, local A/V issues at
the physical meeting can have a large impact on our ability to deliver
vmeet services to remote participants.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Joel Jaeggli <[email-masked]>
Date: Tue, 27 Mar 2012 08:41:58 +0200

> vmeet services to remote participants.

Note that some attempts at debugging are more disruptive then others...

As with the skype episode in sidr on monday, if you consider the remote
participation integral to the local meeting you're going to need to stop
the meeting in order to restore service. that may expensive given the
fixed size of time slots.

I expect that a rigorously defined contracted service would include
specifing an SLA. The volunteers never offered anything other than best
effort.

---

From: Dave Crocker <[email-masked]>
Date: Tue, 27 Mar 2012 09:13:13 +0200

>> outage?
...
> vmeet services to remote participants.

As I recall, Paul's mandate is to produce a functional specification, not a 
contract/requirements service document.

That is, what should be the functional capabilities of remote participation 
service, not what are the critical operational parameters written into contracts.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: SM <[email-masked]>
Date: Tue, 27 Mar 2012 01:07:05 -0700

Hi John,

The problem I mentioned yesterday about the stream stopping is a client issue.

>than "fixed") but, unless there is evidence that this was a

As that is mid-meeting, it may be more informative to have a report 
at the end of the meeting.

>intervention is more discipline and better tools.

Yes.  The message you sent this morning highlights the human side.

>effective to show up after the problem is reported?

Is there any data about this?

Regards,
-sm

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 28 Mar 2012 11:20:14 +0900

On 2012/03/27 8:15, John C Klensin wrote:
> Hi,
>
> --On Monday, March 26, 2012 14:23 -0700 SM<[email-masked]>
> wrote:

> have video as well as audio and the video covers the floor mics

And identification of the speaker is helpful for blind participants, 
too. And also for attendees who are not regulars and therefore don't 
know all the regulars that speak up often. And also for the occasional 
person at the mike who may be a (relative) newcomer. And all this 
applies both remote as well as very much also locally.

So, everybody, when you start speaking, please say your name. And 
chairs, please insist on this.

> _______________________________________________
> NOTE WELL: This list operates according to
> http://mipassoc.org/dkim/ietf-list-rules.html.

Why do we have to point to an off-site web page for IETF mailing list 
rules? If there's something wrong with what we have now at the IETF 
site, let's fix that.

Regards,   Martin.

---

From: John C Klensin <[email-masked]>
Date: Wed, 28 Mar 2012 05:04:54 -0400

--On Wednesday, March 28, 2012 11:20 +0900 "\"Martin J.
Dürst\"" <[email-masked]> wrote:

>> I know that.  And non-English participants (and
>> hearing-impaired ones) whether they are in the room or not.
>> But the transcripts are useful _remotely_ only if they
>> identify speakers -- or we have video as well as audio and
>> the video covers the floor mics as well as the podium.
> 
> And identification of the speaker is helpful for blind
> participants, too. And also for attendees who are not regulars
> and therefore don't know all the regulars that speak up often.
> And also for the occasional person at the mike who may be a
> (relative) newcomer. And all this applies both remote as well
> as very much also locally.
> 
> So, everybody, when you start speaking, please say your name.
> And chairs, please insist on this.

Martin (and others),

I believe that, with the exception of specific tools, like audio
transmission, that physically enable remote users and have
almost no direct effect on the typical person sitting in the
room, virtually anything that make remote participation more
plausible will be helpful to local participants as well.  

Arguably, much of that is "operational".  That category probably
includes whether speaker names are made clear, Jabber is
monitored, and people remember to turn Webex or Meetecho on when
they've said that they would as well as whether people use the
microphones even in small rooms and whether the super-fancy
audio gear is actually powered up and connected. 

>From my point of view, that operational and procedural stuff,
and whatever training, reminders, and even contractual
requirements are needed to make it work, are a key part of
remote participation requirements, if only for the obvious
reason that, if the technology (no matter how good) is not
actually usable, we haven't accomplished anything.

I infer from his note that Dave Crocker disagrees and doesn't
want those issues considered on this list.  If Paul or Russ
could clarify what they do want and do consider relevant, it
might be helpful to many of us.

    john