Subject: [iccrg] VR Problem Statement: draft-han-iccrg-arvr-transport-problem-00

From: Bob Briscoe <[email-masked]>
Date: Fri, 7 Apr 2017 18:33:35 +0100

Matt, Christian,

A couple of comments on this that I wanted to make during the meeting in 
response to your points at the mic, but I was having problems with the 
remote attendance stuff (ironically, given my points below).

*Christian, **
*Re: Your point about how it would be preferable to use model-based 
local video rendering.

The draft already says its purpose is to determine how much of the 
problem is believed to be solvable at higher layers (see S.1.1 Scope), 
so that the remaining problem for the transport layer can be better 
quantified.

Do you really think it is likely that model-based video will become a 
sufficiently mature reality before access capacity is routinely high 
enough to do high quality VR by sending straightforward encoded video?

Bear in mind that we can do low enough latency now (L4S). And for the 
last few decades, access bw has been doubling every 1.6 years (though 
not smoothly of course). So if we say typical access bw in the developed 
world is 10-100 Mb/s now, if it maintains that trend, on avg it should be

  * 20-200 Mb/s in under 2 years (enough for the 'Entry-Level' 4K VR in
    the draft);
  * 0.4-4 Gb/s in 8 years (enough for the 'Advanced' 12K VR in the
    draft) and
  * 3-30 Gb/s in roughly 12 years (enough the 'Ultimate' 24K VR in the
    draft).

{Note: assuming the low end of each bw range will be sufficient for VR 
that avoids I-frames (just like the real world doesn't have sudden scene 
changes), while the high-end could accommodate I-frames without 
buffering - but see my next point to Matt.}

Put the question another way. In 8 years time, how sure can you be that 
model-based video will be able to faithfully detect, transmit and 
recreate, the subtle alterations in someone's facial muscles that the 
human eye can detect at a distance, and that the 'Advanced' VR should be 
able to faithfully encode and transmit by then? And how likely would you 
be able to build the local processing necessary for model-based video 
into a lightweight pair of VR glasses?

My point is merely that there is more than one way to the future, and 
the elegant way doesn't always trump the "more grunt" way.

*Matt,*

You said (paraphrasing) that CC has been steadily improving but this 
problem seems to document the impossible. I can't see why you're being 
so curmudgeonly.

There are clearly challenges with acceleration and deceleration to 
handle the I-frame bursts (if they are necessary). That would certainly 
be challenging for multiplexing this traffic with other traffic, whether 
or not it was in a priority class. There could be other bursty traffic 
(e.g. other VR sessions) in the priority class too - not everything can 
have priority. So, if the CC of competing flows had to deal with these 
burst without buffering, that would be a mighty challenge. But solutions 
are not beyond the bounds of imagination.

Removing the I-frames is of course one possibility. But also (as the 
draft suggests), a rethink of medium access scheduling in layer-2 access 
networks, in order to multiplex a lot more users together on a 
per-packet basis, rather than using naive time-division-multiplexing as 
is currently done on PoNs, cable and wireless technologies. This could 
leave the remainder of the problem tractable at L4.

Put it another way, at MMSys in May last year L4S was demonstrated with 
all the following apps over the same 40Mb/s broadband access link (7ms 
base delay emulated to make the demo portable):

  * VR goggles sending head movements to control the scene streamed from
    a 360deg camera in a racing car rendered in the cloud and returned
    over the broadband access;
  * The demo of L4S you saw at the IETF in Jul 2016 (finger-gesture
    control of a panoramic video of a football match rendered in the cloud)
  * An http adaptive steamed video
  * A network gaming benchmark
  * Synthetic web downloads at 10 item/s (see paper
    <http://www.bobbriscoe.net/pubs.html#uld4all-demo> for details)

The queueing delay was no more than 1 ms for any packet, even with all 
these apps on the same line. And the VR felt natural, although let down 
by the frame-rate of the goggles.

Impossible is a provocative word.

Cheers

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Marie-Jose Montpetit <[email-masked]>
Date: Fri, 7 Apr 2017 18:22:02 -0400

Actually VR - well networked VR with interactivity like what is planned for gaming and interactive movies - is one of these interesting problems. If you move beyond video and add haptics and other sensors to the mix you get the infamous multi-source multi-destination problems. It’s not a question of CC. One of the few ones that could be resolved by combinations of overlay/caching architectures (like in some ICN implementations) to pre-stage some content, some P2P with completion times, of course CC and also FEC to reduce the impact of both delays and packet loss. 

Nice system problem.

mjm
> On Apr 7, 2017, at 1:33 PM, Bob Briscoe <[email-masked]> wrote:
> 
> Matt, Christian,
> 
> A couple of comments on this that I wanted to make during the meeting in response to your points at the mic, but I was having problems with the remote attendance stuff (ironically, given my points below).
> 
> Christian, 
> Re: Your point about how it would be preferable to use model-based local video rendering.
> 
> The draft already says its purpose is to determine how much of the problem is believed to be solvable at higher layers (see S.1.1 Scope), so that the remaining problem for the transport layer can be better quantified. 
> 
> Do you really think it is likely that model-based video will become a sufficiently mature reality before access capacity is routinely high enough to do high quality VR by sending straightforward encoded video?
> 
> Bear in mind that we can do low enough latency now (L4S). And for the last few decades, access bw has been doubling every 1.6 years (though not smoothly of course). So if we say typical access bw in the developed world is 10-100 Mb/s now, if it maintains that trend, on avg it should be 
> 20-200 Mb/s in under 2 years (enough for the 'Entry-Level' 4K VR in the draft);
> 0.4-4 Gb/s in 8 years (enough for the 'Advanced' 12K VR in the draft) and
> 3-30 Gb/s in roughly 12 years (enough the 'Ultimate' 24K VR in the draft).
> 
> {Note: assuming the low end of each bw range will be sufficient for VR that avoids I-frames (just like the real world doesn't have sudden scene changes), while the high-end could accommodate I-frames without buffering - but see my next point to Matt.}
> 
> Put the question another way. In 8 years time, how sure can you be that model-based video will be able to faithfully detect, transmit and recreate, the subtle alterations in someone's facial muscles that the human eye can detect at a distance, and that the 'Advanced' VR should be able to faithfully encode and transmit by then? And how likely would you be able to build the local processing necessary for model-based video into a lightweight pair of VR glasses?
> 
> My point is merely that there is more than one way to the future, and the elegant way doesn't always trump the "more grunt" way.
> 
> Matt,
> 
> You said (paraphrasing) that CC has been steadily improving but this problem seems to document the impossible. I can't see why you're being so curmudgeonly.
> 
> There are clearly challenges with acceleration and deceleration to handle the I-frame bursts (if they are necessary). That would certainly be challenging for multiplexing this traffic with other traffic, whether or not it was in a priority class. There could be other bursty traffic (e.g. other VR sessions) in the priority class too - not everything can have priority. So, if the CC of competing flows had to deal with these burst without buffering, that would be a mighty challenge. But solutions are not beyond the bounds of imagination. 
> 
> Removing the I-frames is of course one possibility. But also (as the draft suggests), a rethink of medium access scheduling in layer-2 access networks, in order to multiplex a lot more users together on a per-packet basis, rather than using naive time-division-multiplexing as is currently done on PoNs, cable and wireless technologies. This could leave the remainder of the problem tractable at L4.
> 
> Put it another way, at MMSys in May last year L4S was demonstrated with all the following apps over the same 40Mb/s broadband access link (7ms base delay emulated to make the demo portable):
> VR goggles sending head movements to control the scene streamed from a 360deg camera in a racing car rendered in the cloud and returned over the broadband access;
> The demo of L4S you saw at the IETF in Jul 2016 (finger-gesture control of a panoramic video of a football match rendered in the cloud)
> An http adaptive steamed video
> A network gaming benchmark
> Synthetic web downloads at 10 item/s (see paper <http://www.bobbriscoe.net/pubs.html#uld4all-demo> for details)
> The queueing delay was no more than 1 ms for any packet, even with all these apps on the same line. And the VR felt natural, although let down by the frame-rate of the goggles.
> 
> Bob Briscoe                               http://bobbriscoe.net/ <http://bobbriscoe.net/>_______________________________________________
> iccrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/iccrg

---

From: Matt Mathis <[email-masked]>
Date: Mon, 10 Apr 2017 14:10:17 -0700

Many years ago some PSC users addressed a similar problem using the
techniques described by Christien - transmitting models (including velocity
and acceleration), prerendering multiple scenarios (who shot first?) and
doing the final POV rendering in a tight loop with the headset.

If I had spoken after Christien, I would have said this can always be made
to fail by moving too much computation back to the central location.
Perhaps I missed the point of the paper: it is useful to explore some of
the boundaries of this trade off and to speculate how network evolution
might move they boundry.  It is less useful to imagine a particular fixed
trade off (which was preselected to be slightly too hard), and then prove
that the network (or what ever) fails.

Again perhaps I missed the point of the paper, but it could have been
asking "What small number of (network) improvements would enable some new
choices in this trade of?"

I guess my point was that the narrow version of the problem statement (the
precise performance goals, etc) was somewhat arbitrary, and were presumably
tuned to make the the network "fail" in a way that suggested a particular
evolutionary goal.  Understanding the tradeoffs is interesting, studying
one data point in depth is less interesting.

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

Privacy matters!  We know from recent events that people are using our
services to speak in defiance of unjust governments.   We treat privacy and
security as matters of life and death, because for some users, they are.

On Fri, Apr 7, 2017 at 10:33 AM, Bob Briscoe <[email-masked]> wrote:

> *Christian, *
> The draft already says its purpose is to determine how much of the problem
> is believed to be solvable at higher layers (see S.1.1 Scope), so that the
> remaining problem for the transport layer can be better quantified.
>
> sufficiently mature reality before access capacity is routinely high enough
> to do high quality VR by sending straightforward encoded video?
>
> Bear in mind that we can do low enough latency now (L4S). And for the last
> few decades, access bw has been doubling every 1.6 years (though not
> smoothly of course). So if we say typical access bw in the developed world
> is 10-100 Mb/s now, if it maintains that trend, on avg it should be
>
>    - 20-200 Mb/s in under 2 years (enough for the 'Entry-Level' 4K VR in
>    the draft);
>    - 0.4-4 Gb/s in 8 years (enough for the 'Advanced' 12K VR in the
>    draft) and
>    - 3-30 Gb/s in roughly 12 years (enough the 'Ultimate' 24K VR in the
>    draft).
>
>
> changes), while the high-end could accommodate I-frames without buffering -
> but see my next point to Matt.}
>
> model-based video will be able to faithfully detect, transmit and recreate,
> the subtle alterations in someone's facial muscles that the human eye can
> detect at a distance, and that the 'Advanced' VR should be able to
> faithfully encode and transmit by then? And how likely would you be able to
> build the local processing necessary for model-based video into a
> lightweight pair of VR glasses?
>
> My point is merely that there is more than one way to the future, and the
> elegant way doesn't always trump the "more grunt" way.
>
> *Matt,*
>
> problem seems to document the impossible. I can't see why you're being so
> curmudgeonly.
>
> There are clearly challenges with acceleration and deceleration to handle
> the I-frame bursts (if they are necessary). That would certainly be
> challenging for multiplexing this traffic with other traffic, whether or
> not it was in a priority class. There could be other bursty traffic (e.g.
> other VR sessions) in the priority class too - not everything can have
> priority. So, if the CC of competing flows had to deal with these burst
> without buffering, that would be a mighty challenge. But solutions are not
> beyond the bounds of imagination.
>
> Removing the I-frames is of course one possibility. But also (as the draft
> suggests), a rethink of medium access scheduling in layer-2 access
> networks, in order to multiplex a lot more users together on a per-packet
> basis, rather than using naive time-division-multiplexing as is currently
> done on PoNs, cable and wireless technologies. This could leave the
> remainder of the problem tractable at L4.
>
> all the following apps over the same 40Mb/s broadband access link (7ms base
> delay emulated to make the demo portable):
>
>    - VR goggles sending head movements to control the scene streamed from
>    a 360deg camera in a racing car rendered in the cloud and returned over the
>    broadband access;
>    - The demo of L4S you saw at the IETF in Jul 2016 (finger-gesture
>    - An http adaptive steamed video
>    - A network gaming benchmark
>    - Synthetic web downloads at 10 item/s (see paper
> these apps on the same line. And the VR felt natural, although let down by
> the frame-rate of the goggles.
>

---

From: Lin Han <[email-masked]>
Date: Wed, 12 Apr 2017 00:54:07 +0000

Hi, Matt

The problem is clearly stated. Let’s use an average data from the paper, can the current network and transport technology support application requiring 500Mbps bandwidth (with big burst) and/or  single low digit ms round trip latency? This requirement is not out of imagine in 10 years, and I don’t think this is to make the network “fail”.
By current technology, I would think the requirement is hard to be satisfied in general, but definitely not impossible. For example, if we don’t consider the SLA negotiation time, if the network is well-planed (router latency, link bandwidth, client-server distance, network topology, limited hops, etc), the user number sharing the bottleneck link is managed, each user’s traffic is also controlled. Using leased line or L4S or other new CC may be able to reach above target.
The difficulty to support such extreme application is for a network without extra planning or designing, for example, the current BNG is normally a bottleneck router for home fixed line access, can we support AR&VR? Even we can assume the AR&VR cloud server is directly connected to the BNG. It is doubtful.
As I analyzed in the paper, current fairness based CC cannot give an application extra resource than another, this makes the AR&VR application and other non-bandwidth-and-delay-sensitive applications share the resource equally. This may cause completely different User Experience for different type of applications.
More importantly, if the user of AR&VR likes to pay more, can network supports it with certain degree of assurance? This may not be a requirement far away from operator if the FCC changes the net-neutrality regulations. Even the current Open Internet Order of 2015 has already opened a door for operator to provide such service.
We would like to see if current CC can solve this problem, and if there is any improvement space. We don’t want to hint any particular evolutionary goal for CC. I believe the current CC algorithm and technologies works very well for most of scenarios. Just for the extreme app or corner case like AR&VR, it may need some improvement.

Thanks

Lin

From: iccrg [mailto:[email-masked]] On Behalf Of Matt Mathis
Sent: Monday, April 10, 2017 2:10 PM
To: Bob Briscoe <[email-masked]>
Cc: Christian Huitema <[email-masked]>; iccrg IRTF list <[email-masked]>
Subject: Re: [iccrg] VR Problem Statement: draft-han-iccrg-arvr-transport-problem-00

Many years ago some PSC users addressed a similar problem using the techniques described by Christien - transmitting models (including velocity and acceleration), prerendering multiple scenarios (who shot first?) and doing the final POV rendering in a tight loop with the headset.

If I had spoken after Christien, I would have said this can always be made to fail by moving too much computation back to the central location.   Perhaps I missed the point of the paper: it is useful to explore some of the boundaries of this trade off and to speculate how network evolution might move they boundry.  It is less useful to imagine a particular fixed trade off (which was preselected to be slightly too hard), and then prove that the network (or what ever) fails.

Again perhaps I missed the point of the paper, but it could have been asking "What small number of (network) improvements would enable some new choices in this trade of?"

I guess my point was that the narrow version of the problem statement (the precise performance goals, etc) was somewhat arbitrary, and were presumably tuned to make the the network "fail" in a way that suggested a particular evolutionary goal.  Understanding the tradeoffs is interesting, studying one data point in depth is less interesting.

Thanks,
--MM--

Privacy matters!  We know from recent events that people are using our services to speak in defiance of unjust governments.   We treat privacy and security as matters of life and death, because for some users, they are.

On Fri, Apr 7, 2017 at 10:33 AM, Bob Briscoe <[email-masked]<mailto:[email-masked]>> wrote:
Matt, Christian,

Christian,

  *   20-200 Mb/s in under 2 years (enough for the 'Entry-Level' 4K VR in the draft);
  *   0.4-4 Gb/s in 8 years (enough for the 'Advanced' 12K VR in the draft) and
  *   3-30 Gb/s in roughly 12 years (enough the 'Ultimate' 24K VR in the draft).

Matt,

  *   VR goggles sending head movements to control the scene streamed from a 360deg camera in a racing car rendered in the cloud and returned over the broadband access;
  *   The demo of L4S you saw at the IETF in Jul 2016 (finger-gesture control of a panoramic video of a football match rendered in the cloud)
  *   An http adaptive steamed video
  *   A network gaming benchmark
  *   Synthetic web downloads at 10 item/s (see paper<http://www.bobbriscoe.net/pubs.html#uld4all-demo> for details)
The queueing delay was no more than 1 ms for any packet, even with all these apps on the same line. And the VR felt natural, although let down by the frame-rate of the goggles.

Cheers

Bob

--

---

From: Marie-Jose Montpetit <[email-masked]>
Date: Wed, 12 Apr 2017 06:00:13 -0400

Hi Lin:

I think what you describe below in terms of bit rates is the video aspect of one-way VR (AR can be significantly lower rate). And like I said in a previous post true VR (and advanced gaming) is multisensor with different rates. You do not know if current CC works well with VR: they work relatively well with one-way high speed video, a component of VR. But add the other sensors and interactivity which is not extreme but part of the experience and yes you may need new approaches.

mjm

> On Apr 11, 2017, at 8:54 PM, Lin Han <[email-masked]> wrote:
> 
> Hi, Matt
>  
> From: iccrg [mailto:[email-masked] <mailto:[email-masked]>] On Behalf Of Matt Mathis
> To: Bob Briscoe <[email-masked] <mailto:[email-masked]>>
> Cc: Christian Huitema <[email-masked] <mailto:[email-masked]>>; iccrg IRTF list <[email-masked] <mailto:[email-masked]>>
> On Fri, Apr 7, 2017 at 10:33 AM, Bob Briscoe <[email-masked] <mailto:[email-masked]>> wrote:
> Matt, Christian,
> 
> Bob Briscoe                               http://bobbriscoe.net/ <http://bobbriscoe.net/>
>  
> _______________________________________________
> iccrg mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.irtf.org/mailman/listinfo/iccrg <https://www.irtf.org/mailman/listinfo/iccrg>

---

From: Lin Han <[email-masked]>
Date: Wed, 12 Apr 2017 17:26:28 +0000

Hi, Marie

Yes, the paper is based on the simplest case of unicast point-to-point VR scenario and does not consider multiple flows, which can be easily analyzed quantitatively.
What you mentioned multi-source multi-destination problems is more complicated. I’m not clear how the network model would be for that scenario, so, cannot judge if the current CC can support them.
But my feeling is it likes the MP-to-MP multicast use cases in IP network. If this is the true, the current CC does not apply, since the multicast application does not used TCP. Maybe UDP based RMCAT can answer. However, as I mentioned in the paper, I think any UDP variation without congestion control cannot beat TCP with CC in terms of 4 targets (efficiency, convergence, fairness, latency); UDP with congestion control may behave same as TCP with CC unless its CC algorithm is completely different as TCP CC. The only benefit of using UDP is saving bandwidth and gain performance if some data is loss tolerant. But no doubt that UDP based RTP and QUIC have many advantages over TCP is other areas.

Thanks

Lin

From: Marie-Jose Montpetit [mailto:[email-masked]]
Sent: Wednesday, April 12, 2017 3:00 AM
To: Lin Han <[email-masked]>
Cc: Matt Mathis <[email-masked]>; Bob Briscoe <[email-masked]>; Christian Huitema <[email-masked]>; iccrg IRTF list <[email-masked]>
Subject: Re: [iccrg] VR Problem Statement: draft-han-iccrg-arvr-transport-problem-00

Hi Lin:

mjm

On Apr 11, 2017, at 8:54 PM, Lin Han <[email-masked]<mailto:[email-masked]>> wrote:

Hi, Matt

Thanks

Lin

From: iccrg [mailto:[email-masked]] On Behalf Of Matt Mathis
Sent: Monday, April 10, 2017 2:10 PM
To: Bob Briscoe <[email-masked]<mailto:[email-masked]>>
Cc: Christian Huitema <[email-masked]<mailto:[email-masked]>>; iccrg IRTF list <[email-masked]<mailto:[email-masked]>>
Subject: Re: [iccrg] VR Problem Statement: draft-han-iccrg-arvr-transport-problem-00

Thanks,
--MM--

Christian,

Matt,

Cheers

Bob

--

_______________________________________________
iccrg mailing list
[email-masked]<mailto:[email-masked]>
https://www.irtf.org/mailman/listinfo/iccrg

---

From: Christian Huitema <[email-masked]>
Date: Fri, 14 Apr 2017 21:38:55 -0700

On 4/10/2017 2:10 PM, Matt Mathis wrote:
> techniques described by Christien - transmitting models (including
> velocity and acceleration), prerendering multiple scenarios (who shot
> first?) and doing the final POV rendering in a tight loop with the
> headset.

My comment was based on VR system, and on user perception studies. The
problem is that we have two different interaction loops: a "remote"
loop, which requires that changes in the scene be reflected "quickly";
and a "local" loop, which requires that changes head position and
orientation be reflected "very quickly".

For the remote loop, the time scale is of the order of 100 milliseconds
-- check for example the Bell Labs studies describing the effects of
voice delays, and other studies on voice/video synchronization. The
target delay actually depends on the application. It tends to be shorter
for interactive games, such as those mentioned by Matt. As Matt points
out, model based systems and dead reckoning are not going to help there.
If the remote loop latency is not met, the video game does not work.

For the close loop, the time scale is very short. You can check for
example the work of Jeremy Bailenson and the Virtual Human Interaction
Lab at Stanford University. If the head moves and the view does not
change soon enough, people can get physically ill. On the other hand,
the closest we get to "really short", the better the perception of
virtual reality. And really short probably means "a few milliseconds".

You can get a "guesstimate" of the problem by looking at the propagation
of signals along nerves. The "remote loop" of the video games has to
contend with hand/eye coordination, and it takes about 10 ms for the
nerve signals to travel from brain to hand. But for the "close loop"
coordination, you have to contend with the eyes and the inner ear, and
the nerve circuits there are much shorter, only a few centimeters. I
don't know how long it takes for these signals to propagate, but I would
assume something of the order of 1 ms.

...
>     ...
>     *Re: Your point about how it would be preferable to use
>     model-based local video rendering.
>     ...
>     Do you really think it is likely that model-based video will
>     become a sufficiently mature reality before access capacity is
>     routinely high enough to do high quality VR by sending
>     straightforward encoded video?
>

Model-based video is routine in video games, so, yes, it could probably
be done. Notice that even with the "straightforward encoded video" you
do need a form of model based rendering at the center. The center must
obtain the head position from sensors in the VR system, estimate the
point of view of the viewer in the virtual world, and render the stereo
image accordingly, before transmitting the "straight video". So you do
have the model information somewhere.

Once you have a model, the VR system can perform real time adjustments
based on instant motions. You will still need a tight "remote loop", but
it won't need to be quite as tight.

-- Christian Huitema