Subject: [MMUSIC] The 3D drafts

From: Bert Greevenbosch <[email-masked]>
Date: Mon, 06 Feb 2012 00:31:37 +0000

Hi all,

I was wondering, if everybody is happy with the current versions of the 3D drafts, or are there still open issues?
I look forward to your comments.

Best regards,
Bert

http://datatracker.ietf.org/doc/draft-ietf-mmusic-parallax-attribute/
http://datatracker.ietf.org/doc/draft-ietf-mmusic-signal-3d-format/

---

From: "Pedro Capelastegui" <[email-masked]>
Date: Fri, 10 Feb 2012 13:32:55 +0100

Hi Bert,

After taking another look at the “3d format” draft, I have found a few
additional issues. They are summarized below:

1)      Encoding of auxiliary stream: In the “video plus auxiliary stream”
schemes using a single stream (i.e. “LD”, “LP”,  “CD”, and  “CP”), no
encoding information is provided for the auxiliary stream. 

2)      Ambiguous negotiation for scenarios with multiple 3D streams: There
is no clear way for an offerer to indicate when multiple 3D streams (e.g. to
show in multiple displays) are offered, as opposed to a single 3D stream
with several possible configurations.

3)      Support for H264/MVC streams: The H264/MVC codec is very well suited
for 3d video applications. It uses some encoding formats that are not
supported by the current draft but could be added without much effort.

I’ll cover each topic in a separate mail. Here is some discussion on point
1), auxiliary stream encoding.

Auxiliary video streams such as depth maps or parallax can be encoded, just
like any other video stream. In the “Signal 3d format” draft, the encoding
for these streams can be described and negotiated normally when they are
transmitted as separate RTP streams. However, this is no longer true when
depth or parallax maps are encapsulated with a regular view as a single RTP
stream, as defined in ISO 23002-3. This corresponds to the formats “2DA CD”,
“2DA CP” , “2DA LD” and “2DA LP” in the draft.

It could be assumed that, since no encoding information is provided for
these streams, they can use the same encoding used for the video view they
are encapsulated with. As an example, consider the following stream:

m=video 49170 RTP/AVP 99

a=rtpmap:99 H264/90000

a=3dFormat:2DA CD

The above lines describe a video stream representing a central view, with a
depth map included in the same RTP stream as auxiliary data. The central
view is encoded in H264, but the depth map encoding is undefined. We could
solve this by stating in the draft that, when no encoding is specified for
an auxiliary stream, it uses the same configuration as the same view. Thus,
the depth map in the example would also use H264.

I think this works well as a default assumption. However, it does not cover
all possible scenarios, as the auxiliary stream CAN use a different format
than the base view. In fact, that will likely be the best performing
configuration, since depth maps have different properties than regular video
streams, and codecs like H264 don’t perform particularly well with them. 

To address this, we would need a way to express the media format for
auxiliary streams. At the very least, this would involve including the
information normally expressed in “a=rtpmap” and “a=fmtp” attributes.

One possible way to do this would be to add the values of these attributes
(for the auxiliary map) as optional parameters in the “a=3dformat”
attribute. In the previous example, this could look as follows: 

m=video 49170 RTP/AVP 99

a=rtpmap:99 H264/90000

a=3dFormat:2DA CD aux-map:H264/90000 aux-fmtp:(…)

A limitation of this approach is that it just describes a single encoding
for the auxiliary map. Ideally, a solution should allow the offerer to
suggest multiple configurations from which the answerer can choose one, as
with any media stream in the offer/answer model. For this, we could define a
new attribute, like “a=3daux”, and include in it a payload type number, the
value of “a=rtpmap” for the auxiliary stream, and the value of “a=fmtp” for
that stream. It could look as follows:

m=video 49170 RTP/AVP 99 100

a=3dFormat:2DA CD 

a=rtpmap:99 H264/90000

a=3daux: 99 H263/90000 aux-fmtp:(…)

a=rtpmap:100 H264/90000

a=3daux: 100 H264/90000 aux-fmtp:(…)

In this example, the offerer is providing two possible configurations:

-          PT 99:  Central view encoded in H.264, depth map encoded in H.263

-          PT 100: Central view and depth map encoded in H.264

In addition, there may be other SDP fields or attributes that have to be
taken into account while signalling these auxiliary streams. 

Best regards,

Pedro

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Bert Greevenbosch
Sent: Monday, February 06, 2012 1:32 AM
To: IETF - MMUSIC
Subject: [MMUSIC] The 3D drafts

Hi all,

I was wondering, if everybody is happy with the current versions of the 3D
drafts, or are there still open issues?

Best regards,

Bert

  _____  

No se encontraron virus en este mensaje.
Comprobado por AVG - www.avg.com
Versión: 10.0.1424 / Base de datos de virus: 2112/4791 - Fecha de
publicación: 02/05/12

---

From: "Pedro Capelastegui" <[email-masked]>
Date: Fri, 10 Feb 2012 17:37:47 +0100

Hi Bert,

Another open issue for the “3d format” draft was the negotiation of sessions
with more than one 3D stream. I discuss it in this mail.

In the current version of the draft, it is possible to set up a session with
multiple 3D streams. An offerer can include several 3D streams in its SDP,
and the answerer can select any number of them. There are rules that prevent
the answerer from selecting more streams than it can process. However,
nothing stops an answerer from selecting more streams than _the offerer_ can
send or receive – in fact, the answerer has no way to know how many streams
can be handled by the offerer.

This can come up even in very simple scenarios. Consider the following
offer:

m=video 1111 RTP/AVP 99

a=rtpmap:99 H264/90000

a=3dFormat:FP SbS

m=video 1112 RTP/AVP 99

a=rtpmap:99 H264/90000

a=3dFormat:FP TaB

This SDP is intended as an offer of a single 3D stream with 2 possible
configurations: frame-packed side-by-side, and top-bottom. The UA generating
the offer expects the answerer to select only one 3D format. But the
answerer doesn’t know that, and it’s perfectly legal for it to respond by
selecting both 3D formats: 

m=video 2222 RTP/AVP 99

a=rtpmap:99 H264/90000

a=3dFormat:FP SbS

m=video 2223 RTP/AVP 99

a=rtpmap:99 H264/90000

a=3dFormat:FP TaB

At this point, the offerer is receiving media that it’s not prepared to
handle (which goes against the offer/answer model), and should either
re-invite, or close the session.

How to avoid this scenario?

The straightforward solution is to forbid answerers from selecting more than
one 3D format altogether. This is certainly effective, but too limiting, as
it rules out the possibility of multi-stream 3D sessions.

A better solution would be to create a new  session level attribute
(“a=max3dstreams”) for the offerer to indicate the maximum number of 3D
streams it wants for this session. The default value for this attribute
would be ‘1’, so that the attribute would only be needed for sessions with
multiple streams. 

This should cover most scenarios, but there are a few corner cases that are
left unsupported. Consider a user agent that wants to initiate a session
with 2 3D streams, with the constraint that one of the streams needs to have
2 views, and the other needs to have a view and a depth map. This is a rare
scenario, but it can happen if the UA has different 3D screens, one taking
left and right views as input, and the other using video+depth map:

a=max3dstreams:2

m=video 1111 RTP/AVP 99

a=rtpmap:99 H264/90000

a=3dFormat:FP SbS

m=video 1112 RTP/AVP 99

a=rtpmap:99 H264/90000

a=3dFormat:FP TaB

m=video 1113 RTP/AVP 99

a=rtpmap:99 H264/90000

a=3dFormat:2DA CD

For this UA, an answer selecting “FP SbS” and “2DA CD” as 3d formats, or “FP
TaB” and “2DA CD” would be acceptable, but one with “FP SbS” and “FP  TaB”
would not.

We could extend the signalling to cover this kind of scenario (e.g. by
introducing a new type of grouping), but I think this would be too complex,
and not worth the effort. In these cases, the offerer can start by
configuring the session with a single 3D stream, and add any additional 3D
streams in later offer/answer exchanges.

Best regards,

Pedro

Hi all,

Best regards,

Bert

  _____

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 10 Feb 2012 13:24:55 -0500

Pedro,

The issue you raise here is not unique to 3D.
It shows up in many forms:

- an offer of multiple codecs on an m-line
   theoretically means the offerer can use them concurrently.
   But some implementations can only support one at a time.

- offering seccure and insecure versions of a stream

- offering ipv4 and ipv6 versions of a stream

It is not desirable to come up with a 3D-specific solution to this 
problem. Rather you should look to what is available as a generic solution.

The most obvious approach that comes to mind for me is to use the 
capability framework, to offer one form and describe the others as 
capabilities.

	Thanks,
	Paul

On 2/10/12 11:37 AM, Pedro Capelastegui wrote:
> Hi Bert,
>
> Another open issue for the “3d format” draft was the negotiation of
> sessions with more than one 3D stream. I discuss it in this mail.
>
> In the current version of the draft, it is possible to set up a session
> with multiple 3D streams. An offerer can include several 3D streams in
> its SDP, and the answerer can select any number of them. There are rules
> that prevent the answerer from selecting more streams than it can
> process. However, nothing stops an answerer from selecting more streams
> than _the offerer_ can send or receive – in fact, the answerer has no
> way to know how many streams can be handled by the offerer.
>
> configurations: frame-packed side-by-side, and top-bottom. The UA
> generating the offer expects the answerer to select only one 3D format.
> But the answerer doesn’t know that, and it’s perfectly legal for it to
> respond by selecting both 3D formats:
>
> m=video 2222 RTP/AVP 99
>
> a=rtpmap:99 H264/90000
>
> a=3dFormat:FP SbS
>
> m=video 2223 RTP/AVP 99
>
> a=rtpmap:99 H264/90000
>
> a=3dFormat:FP TaB
>
> The straightforward solution is to forbid answerers from selecting more
> than one 3D format altogether. This is certainly effective, but too
> limiting, as it rules out the possibility of multi-stream 3D sessions.
>
> would be ‘1’, so that the attribute would only be needed for sessions
> with multiple streams.
>
> This should cover most scenarios, but there are a few corner cases that
> are left unsupported. Consider a user agent that wants to initiate a
> session with 2 3D streams, with the constraint that one of the streams
> needs to have 2 views, and the other needs to have a view and a depth
> map. This is a rare scenario, but it can happen if the UA has different
> 3D screens, one taking left and right views as input, and the other
> using video+depth map:
>
> a=max3dstreams:2
>
> m=video 1111 RTP/AVP 99
>
> a=rtpmap:99 H264/90000
>
> a=3dFormat:FP SbS
>
> m=video 1112 RTP/AVP 99
>
> a=rtpmap:99 H264/90000
>
> a=3dFormat:FP TaB
>
> m=video 1113 RTP/AVP 99
>
> a=rtpmap:99 H264/90000
>
> a=3dFormat:2DA CD
>
> For this UA, an answer selecting “FP SbS” and “2DA CD” as 3d formats, or
> “FP  TaB” and “2DA CD” would be acceptable, but one with “FP SbS” and
> “FP  TaB” would not.
>
> introducing a new type of grouping), but I think this would be too
> complex, and not worth the effort. In these cases, the offerer can start
> by configuring the session with a single 3D stream, and add any
> additional 3D streams in later offer/answer exchanges.
>
> Best regards,
>
> Pedro
>
> *From:[email-masked] [mailto:[email-masked]] *On
> Behalf Of *Bert Greevenbosch
> *Sent:* Monday, February 06, 2012 1:32 AM
> *To:* IETF - MMUSIC
> *Subject:* [MMUSIC] The 3D drafts
>
> Hi all,
>
> I was wondering, if everybody is happy with the current versions of the
> 3D drafts, or are there still open issues?
>
> ------------------------------------------------------------------------
>
> Comprobado por AVG - www.avg.com <http://www.avg.com>
> _______________________________________________
> mmusic mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/mmusic

---

From: Bert Greevenbosch <[email-masked]>
Date: Mon, 13 Feb 2012 03:11:49 +0000

Hi Pedro,

Thank you for your feedback. I will soon reply to your other e-mail, about the simulcast case, too.

As to the issue with multiple offered streams, this has been considered in the draft. Especially, section 7 contains the following text:

"
The following statements apply for the answerer:
...
   o  If the answerer selects multiple 3D formats, it MUST be prepared
      to send/receive (depending on whether it is a streaming server or
      client or both) associated streams simultaneously.

The following statements apply for the offerer:
...
   o  When multiple 3D formats are selected, the offerer MAY initiate
      all associated streams.  Alternatively, it MAY update its offer
      with a reduced number of 3D formats.
"

As you wrote below, indeed the draft proposes re-negotiation if the offerer doesn't like the choice of the answerer.

Why do you think this should be avoided?

Best regards,
Bert

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Paul Kyzivat
Sent: 11 February 2012 02:25
To: [email-masked]
Subject: Re: [MMUSIC] The 3D drafts

Pedro,

	Thanks,
	Paul

On 2/10/12 11:37 AM, Pedro Capelastegui wrote:
> Hi Bert,
>
> Another open issue for the "3d format" draft was the negotiation of
> than _the offerer_ can send or receive - in fact, the answerer has no
> But the answerer doesn't know that, and it's perfectly legal for it to
> At this point, the offerer is receiving media that it's not prepared to
> ("a=max3dstreams") for the offerer to indicate the maximum number of 3D
> would be '1', so that the attribute would only be needed for sessions
> with multiple streams.
>
> For this UA, an answer selecting "FP SbS" and "2DA CD" as 3d formats, or
> "FP  TaB" and "2DA CD" would be acceptable, but one with "FP SbS" and
> "FP  TaB" would not.
>

---

From: "Pedro Capelastegui" <[email-masked]>
Date: Mon, 13 Feb 2012 18:01:57 +0100

Hi Bert,

My main concern is that the offerer may end up receiving more video streams
than it wants and, more importantly, more than it can process. Although the
draft does allow for re-negotiating the session to reduce the number of
streams, this still leaves a window where an excess of streams is
transmitted, leading to wasted bandwidth, and potentially overloading the
offerer’s network or terminal.

As a side effect, when QoS systems are used, this may result in excessive
bandwidth reservations,  or in rejected sessions due to insufficient network
resources.

Note that, while I also discussed re-negotiating sessions in my message, my
proposal in that case was to start out with a single stream and add more
later, ruling out the possibility of transmitting unwanted streams.

That said, there is one point that I hadn’t really taken into consideration
until Paul brought it up: the potential for an offerer to receive lots of
unwanted streams already exists in other SDP offer/answer scenarios.
Notably, this includes offers with multiple media formats for a m= line,
(which describes almost every SDP offer in existence). So it’s possible that
the problem is less severe than I originally thought, but also more general
than this specific 3D video scenario. I’ll need to give this issue some
thought.

Best regards,
Pedro

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Bert Greevenbosch
> Sent: Monday, February 13, 2012 4:12 AM
> To: Paul Kyzivat; [email-masked]
> Thank you for your feedback. I will soon reply to your other e-mail, about
the
> simulcast case, too.
> 
> As to the issue with multiple offered streams, this has been considered in
> the draft. Especially, section 7 contains the following text:
> 
> "
> As you wrote below, indeed the draft proposes re-negotiation if the
offerer
> doesn't like the choice of the answerer.
> 
problem.
> Rather you should look to what is available as a generic solution.
> 
capability
> framework, to offer one form and describe the others as capabilities.
> 
> 	Thanks,
> 	Paul
> 
> > In the current version of the draft, it is possible to set up a
> > session with multiple 3D streams. An offerer can include several 3D
> > streams in its SDP, and the answerer can select any number of them.
> > There are rules that prevent the answerer from selecting more streams
> > than it can process. However, nothing stops an answerer from selecting
> > more streams than _the offerer_ can send or receive - in fact, the
> > answerer has no way to know how many streams can be handled by the
> offerer.
> >
> > At this point, the offerer is receiving media that it's not prepared
> > to handle (which goes against the offer/answer model), and should
> > either re-invite, or close the session.
> >
> > The straightforward solution is to forbid answerers from selecting
> > more than one 3D format altogether. This is certainly effective, but
> > too limiting, as it rules out the possibility of multi-stream 3D
sessions.
> > ("a=max3dstreams") for the offerer to indicate the maximum number of
> > 3D streams it wants for this session. The default value for this
> > attribute would be '1', so that the attribute would only be needed for
> > sessions with multiple streams.
> >
> > This should cover most scenarios, but there are a few corner cases
> > that are left unsupported. Consider a user agent that wants to
> > initiate a session with 2 3D streams, with the constraint that one of
> > the streams needs to have 2 views, and the other needs to have a view
> > and a depth map. This is a rare scenario, but it can happen if the UA
> > has different 3D screens, one taking left and right views as input,
> > and the other using video+depth map:
> >
> > a=max3dstreams:2
> >
> > m=video 1111 RTP/AVP 99
> >
> > a=rtpmap:99 H264/90000
> >
> > a=3dFormat:FP SbS
> >
> > m=video 1112 RTP/AVP 99
> >
> > a=rtpmap:99 H264/90000
> >
> > a=3dFormat:FP TaB
> >
> > m=video 1113 RTP/AVP 99
> >
> > a=rtpmap:99 H264/90000
> >
> > a=3dFormat:2DA CD
> >
> > For this UA, an answer selecting "FP SbS" and "2DA CD" as 3d formats,
> > or "FP  TaB" and "2DA CD" would be acceptable, but one with "FP SbS"
> > and "FP  TaB" would not.
> >
> > complex, and not worth the effort. In these cases, the offerer can
> > start by configuring the session with a single 3D stream, and add any
> > I was wondering, if everybody is happy with the current versions of
> > the 3D drafts, or are there still open issues?
> >
> -----
> Versión: 10.0.1424 / Base de datos de virus: 2112/4800 - Fecha de
publicación:
> 02/09/12

---

From: Bert Greevenbosch <[email-masked]>
Date: Sat, 25 Feb 2012 00:32:40 +0000

Hi Pedro,

As promised, here the answer to your second e-mail.

The current simulcast solution in the draft indeed assumes a single codec, which internally contains both a 2D and an auxiliary stream.

But my understanding is that your main concern is the simulcast of two streams (2D and aux), with different codecs. Your proposed solution would indeed resolve that problem.

Now that we have the discussions on RTP multiplexing, especially the "BUNDLE" draft (http://datatracker.ietf.org/doc/draft-holmberg-mmusic-sdp-bundle-negotiation), I was wondering whether we could somehow use that mechanism to resolve the problem. In that case, your original example

             m=video 49170 RTP/AVP 99 100
             a=3dFormat:2DA CD
             a=rtpmap:99 H264/90000
             a=3daux: 99 H263/90000 aux-fmtp:(...)
             a=rtpmap:100 H264/90000
             a=3daux: 100 H264/90000 aux-fmtp:(...)

would become as follows:

             a=group:3DS 1 2
             a=group:BUNDLE 1 2
             m=video 49170 RTP/AVP 99
             a=rtpmap:99 H264/90000
             a=3dFormat:SC C
             a=mid:1
             m=video 49170 RTP/AVP 100
             a=rtpmap:100 H263/90000
             a=3dFormat:SC D
             a=mid:2
             a=group:3DS 3 4
             a=group:BUNDLE 3 4
             m=video 49170 RTP/AVP 99
             a=rtpmap:99 H264/90000
             a=3dFormat:SC C
             a=mid:3
             m=video 49170 RTP/AVP 100
             a=rtpmap:100 H264/90000
             a=3dFormat:SC D
             a=mid:4

This means, that we use the separate streams mechanism + the "BUNDLE" mechanism to achieve our goal. It this is maybe a bit premature, as the "BUNDLE" draft is only a personal draft, but it seems to work and we would not need an extra mechanism in the 3D format draft.

Admittedly, this solution also looks complex, especially because of the double grouping mechanism. However, it would not require to define a new "3daux" attribute and "aux-fmtp" semantics.

What do you think?

Best regards,
Bert

From: Pedro Capelastegui [mailto:[email-masked]]
Sent: 10 February 2012 20:33
To: Bert Greevenbosch; 'IETF - MMUSIC'
Subject: RE: [MMUSIC] The 3D drafts

Hi Bert,

After taking another look at the "3d format" draft, I have found a few additional issues. They are summarized below:

1)      Encoding of auxiliary stream: In the "video plus auxiliary stream" schemes using a single stream (i.e. "LD", "LP",  "CD", and  "CP"), no encoding information is provided for the auxiliary stream.

2)      Ambiguous negotiation for scenarios with multiple 3D streams: There is no clear way for an offerer to indicate when multiple 3D streams (e.g. to show in multiple displays) are offered, as opposed to a single 3D stream with several possible configurations.

3)      Support for H264/MVC streams: The H264/MVC codec is very well suited for 3d video applications. It uses some encoding formats that are not supported by the current draft but could be added without much effort.
I'll cover each topic in a separate mail. Here is some discussion on point 1), auxiliary stream encoding.

Auxiliary video streams such as depth maps or parallax can be encoded, just like any other video stream. In the "Signal 3d format" draft, the encoding for these streams can be described and negotiated normally when they are transmitted as separate RTP streams. However, this is no longer true when depth or parallax maps are encapsulated with a regular view as a single RTP stream, as defined in ISO 23002-3. This corresponds to the formats "2DA CD", "2DA CP" , "2DA LD" and "2DA LP" in the draft.

It could be assumed that, since no encoding information is provided for these streams, they can use the same encoding used for the video view they are encapsulated with. As an example, consider the following stream:

m=video 49170 RTP/AVP 99
a=rtpmap:99 H264/90000
a=3dFormat:2DA CD

The above lines describe a video stream representing a central view, with a depth map included in the same RTP stream as auxiliary data. The central view is encoded in H264, but the depth map encoding is undefined. We could solve this by stating in the draft that, when no encoding is specified for an auxiliary stream, it uses the same configuration as the same view. Thus, the depth map in the example would also use H264.

I think this works well as a default assumption. However, it does not cover all possible scenarios, as the auxiliary stream CAN use a different format than the base view. In fact, that will likely be the best performing configuration, since depth maps have different properties than regular video streams, and codecs like H264 don't perform particularly well with them.

To address this, we would need a way to express the media format for auxiliary streams. At the very least, this would involve including the information normally expressed in "a=rtpmap" and "a=fmtp" attributes.

One possible way to do this would be to add the values of these attributes (for the auxiliary map) as optional parameters in the "a=3dformat" attribute. In the previous example, this could look as follows:

m=video 49170 RTP/AVP 99
a=rtpmap:99 H264/90000
a=3dFormat:2DA CD aux-map:H264/90000 aux-fmtp:(...)

A limitation of this approach is that it just describes a single encoding for the auxiliary map. Ideally, a solution should allow the offerer to suggest multiple configurations from which the answerer can choose one, as with any media stream in the offer/answer model. For this, we could define a new attribute, like "a=3daux", and include in it a payload type number, the value of "a=rtpmap" for the auxiliary stream, and the value of "a=fmtp" for that stream. It could look as follows:

m=video 49170 RTP/AVP 99 100
a=3dFormat:2DA CD
a=rtpmap:99 H264/90000
a=3daux: 99 H263/90000 aux-fmtp:(...)
a=rtpmap:100 H264/90000
a=3daux: 100 H264/90000 aux-fmtp:(...)

In addition, there may be other SDP fields or attributes that have to be taken into account while signalling these auxiliary streams.

Best regards,
Pedro

From: [email-masked] [mailto:[email-masked]] On Behalf Of Bert Greevenbosch
Sent: Monday, February 06, 2012 1:32 AM
To: IETF - MMUSIC
Subject: [MMUSIC] The 3D drafts

Hi all,

Best regards,
Bert

http://datatracker.ietf.org/doc/draft-ietf-mmusic-parallax-attribute/
http://datatracker.ietf.org/doc/draft-ietf-mmusic-signal-3d-format/
________________________________

No se encontraron virus en este mensaje.
Comprobado por AVG - www.avg.com<http://www.avg.com>
Versión: 10.0.1424 / Base de datos de virus: 2112/4791 - Fecha de publicación: 02/05/12