Subject: [MMUSIC] 3D format negotiation in draft-greevenbosch-mmusic-signal-3d-format

From: "Pedro Capelastegui" <[email-masked]>
Date: Thu, 3 Nov 2011 17:27:00 +0100

Hi,

Here are some further comments on
draft-greevenbosch-mmusic-signal-3d-format. These are focused on the 3d
format negotiation process, and specifically how the extension behaves when
multiple 3d configuration options are offered.

My main concern is that, in order to offer alternate 3d format
configurations, the offerer has to include additional 'm=' lines in the SDP.
Currently, each media descriptor can have a single 3d format, and each 3d
format is associated to 1 or 2 media descriptors. This is not an issue in
offers with a single 3d format (see examples 9.1, 9.2 and 9.3), but when
several 3d formats are offered, the number of media descriptors multiplies.
This can be observed in example 9.4, where 4 media descriptors are used in
the offer, though the answerer is expected to reject all but two. In
general, in order to provide N 3D format options for a 3D video
stream,between N and 2N media descriptors will be required in the offered
SDP, of which only one or two will be used.

I think this proliferation of media descriptions has several drawbacks. For
one, it goes against the general philosophy of SDP negotiation. Normally,
when different possible configurations exist for a media stream, they are
shown as different media formats within a single media descriptor. Using
multiple media descriptors that actually refer to the same media stream (the
3D video stream, or a subset of it), and having the answerer reject all but
the ones with the desired configuration breaks this model. In addition, this
excess of 'm=' lines may have a negative effect when combined with
techniques such as ICE, and could be confusing for 3d-unaware nodes.

On the other hand, this is only a problem if user agents are expected to
select between many 3d formats on an average session. Unfortunately, I think
this will be the case. At the very least, a UA starting a 3D session should
provide a single 3D format and the option not to use 3D. Apart from that, a
3D user agent intended to be interoperable with other UAs should support
several 3D formats, including video+depth and transmission of 2 views. 

The solution, in my opinion, would be to shift away from the current model
and instead associate each 3D format with a payload type within a media
description. I will discuss how this can be done in a separate mail.

Regards,

Pedro

---

From: "Pedro Capelastegui" <[email-masked]>
Date: Thu, 3 Nov 2011 17:34:31 +0100

Hi,

In the previous post, I argued against the 3D format negotiation mechanism
currently used in draft-greevenbosch-mmusic-signal-3d-format, and suggested
associating 3D formats with payload types instead. Here I explain how this
could be achieved.

The process could be summarized as follows: 3d format configuration
parameters are described in media level attributes (like in the current
draft), but these attributes have a payload type as an additional parameter:

	a=3dformat:<Payload_Type> <3dformat parameters>

Each 3D media stream will be composed of one or two media descriptions, each
with one or more 3d format attributes. For each media description, an
answerer would select one of the available media formats, with the preferred
combination of codec and 3d format. In order to keep complexity manageable,
the answerer would only be allowed to choose a single media format when 3d
is enabled.

This still leaves us with the problem of declaring dependencies between
media descriptions within a 3D media stream (e.g. stream B is the depth map
of stream A, or stream B is the right view and stream A is the left one).
This could be signaled with the ‘a=depend’ attribute (see RFC 5583).

As an example, here is how this system would work in the scenario shown in
section 9.4 of draft-greevenbosch-mmusic-signal-3d-format. The offerer is
trying to initiate a session with a 3D video stream which can have 2
possible configurations: 

	(1)	1 stream for video and one stream for a parallax map
	(2)	1 stream for the left view and one for the right view.

This is what the offer SDP would look like
	v=0
	o=Alice 2890844526 2890842807 IN IP4 131.163.72.4
	s=The technology of 3D-TV
	c=IN IP4 131.164.74.2
	t=0 0
	a=group:3DS 1 2
	m=video 49170 RTP/AVP 99 100
	a=rtpmap:99 H264/90000
	a=3dFormat:99 2DA C
	a=rtpmap:100 H264/90000
	a=3dFormat:100 SC L
	a=mid:1
	m=video 49172 RTP/AVP 99 100
	a=rtpmap:99 H264/90000
	a=3dFormat:99 2DA P
	a=rtpmap:100 H264/90000
	a=3dFormat:100 SC R
	a=mid:2
	a=depend 99 3dv 1:99; 100 3dv 1:100
	m=audio 52890 RTP/AVP 10
	a=rtpmap:10 L16/16000/2

‘3dv’ would be a new dependency-type defined for use with 3d format. The
‘depend’ attribute is tipically used in combination with “DDP” (decoding
dependency) groups, though I think using it with “3DS” groups in this
scenario wouldn’t go against the specification.

This offer SDP has 2 video streams in a 3DS group, each with 2 possible
configurations. The first one can be configured as a ‘center’ view in a
video+auxiliary map scheme, or as a left view in a simulcast scheme. The
second media stream can be configured as a parallax map in a video+aux
configuration, or as a right view in a simulcast configuration. Only certain
payload type combinations are viable, which is signaled in the ‘a=depend’
line: video stream 2 can only be configured as a parallax map if video
stream 1 is the center view, and it can only be configured as a right view
if stream 1 is the left view.

The SDP answer would look like this, if the answerer chooses the "center
view plus parallax map" option: 

	v=0
	o=Bob 2890844528 2890842809 IN IP4 131.163.72.5
	s=The technology of 3D-TV
	c=IN IP4 131.164.74.3
	t=0 0
	a=group:3DS 1 2
	m=video 48170 RTP/AVP 99
	a=rtpmap:99 H264/90000
	a=3dFormat:99 2DA C
	a=mid:1
	m=video 48172 RTP/AVP 99 
	a=rtpmap:99 H264/90000
	a=3dFormat:99 2DA P
	a=mid:2
	a=depend 99 3dv 1:99; 100 3dv 1:100
	m=audio 52890 RTP/AVP 10
	a=rtpmap:10 L16/16000/2

Compared to the SDPs used in draft-greevenbosch-mmusic-signal-3d-format,
this offer/answer exchange only uses 3 media descriptions instead of 5.
Also, including additional 3D formats would be more simple using this
mechanism.

Regards,
Pedro

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Pedro Capelastegui
Sent: Thursday, November 03, 2011 5:27 PM
To: IETF - MMUSIC; [email-masked]
Subject: [MMUSIC] 3D format negotiation in
draft-greevenbosch-mmusic-signal-3d-format

Hi,

Regards,

Pedro

_______________________________________________
mmusic mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/mmusic
-----
No se encontraron virus en este mensaje.
Comprobado por AVG - www.avg.com
Versión: 10.0.1411 / Base de datos de virus: 2092/3992 - Fecha de
publicación: 11/02/11

---

From: Bert Greevenbosch <[email-masked]>
Date: Tue, 15 Nov 2011 07:59:50 +0000

Hi Pedro,

Sorry for my late response; I have been quite busy lately.

Thanks for your e-mail, and for thinking with us about how to move forward with the 3D drafts.

To summarise your proposal:
(1) Point at multiple possible video streams in a single media "m=" line (i.e. a group of video streams per one media line; I'll call that a "media group").
(2) Stipulate that per media group, only one video stream can be chosen.
(3) In one of the media groups, use the "depend" attribute to indicate which video streams are associated with which video streams from the other media group(s).
(4) Use "3DS" grouping + "mid" attribute to label the media groups; needed for the "depend" attribute.
(5) Use the payload type to differentiate the streams within a media group.

I have thought about your proposal, and the following issues arose:
(A) The usage of the payload type requires a different payload type for each video stream. This is OK for streams that use a dynamic payload type. However, certain RTP streams have fixed payload types. For example, MPEG-2 video over MPEG Transport Stream over RTP has fixed payload type 33. With your proposal, it is not possible to do 3D simulcast of L- and R-streams in MPEG-2.
(B) It is true, that the number of "m=" lines in your proposal is less than in the current draft. But this is only a virtual reduction, as now multiple video streams are advertised in a single media line. In principle, the same number of video streams is advertised.
(C) With your proposal, in addition to the "3DS" grouping and the "3dFormat" attribute, the "depend" attribute needs also be supported, with the particular meaning as you have defined it.
(D) Does having less "m=" lines justify the additional complexity of using the "depend" attribute?
(E) What if different video streams in the same media group require different SDP attributes?

What do you think?

Best regards,
Bert

________________________________________
From: Pedro Capelastegui [[email-masked]]
Sent: 04 November 2011 00:34
To: 'IETF - MMUSIC'; Bert Greevenbosch
Subject: RE: [MMUSIC] 3D format negotiation in  draft-greevenbosch-mmusic-signal-3d-format

Hi,

        v=0

Regards,
Pedro

-----Original Message-----

Hi,

Regards,

Pedro

---

From: "Pedro Capelastegui" <[email-masked]>
Date: Wed, 16 Nov 2011 15:03:24 +0100

Hi Bert,

Thanks for answering. Answers inline.

Regards,
Pedro

> From: Bert Greevenbosch [mailto:[email-masked]]
> Sent: Tuesday, November 15, 2011 9:00 AM
> To: Pedro Capelastegui; 'IETF - MMUSIC'
> (1) Point at multiple possible video streams in a single media "m=" line
(i.e. a
> group of video streams per one media line; I'll call that a "media
group").
> (3) In one of the media groups, use the "depend" attribute to indicate
which
> video streams are associated with which video streams from the other media
> group(s).

Your summary is correct. However, I'd like to comment something about the
terminology. I think it's best to refer to the "m=" lines as "media
descriptions", and to the configurations associated with each payload type
number as "media format", since this matches the vocabulary used in other
RFCs.

 Your use of "video streams" and "media groups" could be helpful in drawing
parallels between both proposals, but it wouldn't be entirely accurate, for
the following reasons: 
	- A media descriptor ("media group") in my proposal is not the
equivalent to a "3DS" group of media streams in the current draft. 
	- A media format ("video stream") in my proposal may actually
include multiple video streams, either through frame packing, or by using
2D+auxiliary data formats.

> (A) The usage of the payload type requires a different payload type for
each
> video stream. This is OK for streams that use a dynamic payload type.
> However, certain RTP streams have fixed payload types. For example,
> MPEG-2 video over MPEG Transport Stream over RTP has fixed payload type
> 33. With your proposal, it is not possible to do 3D simulcast of L- and R-
> streams in MPEG-2.

This is something I hadn't taken into account. However, I'm not sure if it
could be a problem. As far as I can tell, all a static payload type implies
is that an encoding (such as MPEG-2) is mapped by default to a given number
(33, in this case). But I haven't been able to find a rule that prevents the
same encoding from being mapped to a dynamic payload type number, after
checking several RFCs about SDP and RTP. Is there any such rule that I may
have missed? Would this remapping of encodings with static payload types
break anything in existing implementations, otherwise?

Assuming that I've missed something, and static payload types cannot be
remapped, it's an issue that needs to be addressed in my proposal. On the
bright side, it doesn't prevent signalling 3D formats while using codecs
with static payload types - following your example, it WOULD be possible to
do 3D simulcast of L and R streams in MPEG-2, such as in the following SDP:

	v=0
	m=video 49170 RTP/AVP 33
	a=rtpmap: 33 MP2T/90000
	a=3dFormat:100 SC L
	a=mid:1
	m=video 49172 RTP/AVP 33
	a=rtpmap:33 MP2T/90000
	a=3dFormat:33 SC R
	a=mid:2
	a=depend 33 3dv 1:33
	m=audio 52890 RTP/AVP 10
	a=rtpmap:10 L16/16000/2

Where static payload types would have a negative impact would be in using
multiple 3d formats in the offer. In the example above, the offerer could
only add another 3d format (say, simulcast 2D+parallax) by adding a new
payload type - but we wouldn't be able to use MP2T encoding for the
alternative 3d format.

> (B) It is true, that the number of "m=" lines in your proposal is less
than in the
> current draft. But this is only a virtual reduction, as now multiple video
> streams are advertised in a single media line. In principle, the same
number
> of video streams is advertised.

This is one case where I think your choice of terminology can be confusing,
so I'll refer to what you call "video streams" as "media formats".

It is true that the amount of media formats offered would remain the same
across proposals, so the overall complexity of the sdp would not change.
However, I believe reducing the number of media descriptors (media lines)
makes the negotiation process more efficient, for the following reasons:
	- NAT traversal. The complexity of NAT traversal usually increases
with the number of media descriptors in an SDP. As an example, ICE needs to
gather candidates for each RTP media stream (i.e. each media descriptor) -
it is unaffected by the number of media formats in any descriptor.
	- Firewalling. Likewise, each media descriptor in the SDP offer
usually requires a unique listening port (plus another for RTCP), so adding
more descriptors may complicate firewall configuration.
	- Subsequent offers. Not a huge issue, but under your proposal, if a
user agent re-INVITES (e.g. to disable a stream), all discarded media lines
from the original offer must still be included, cluttering the SDP (though
their attributes may be omitted).

> (C) With your proposal, in addition to the "3DS" grouping and the
"3dFormat"
> attribute, the "depend" attribute needs also be supported, with the
> particular meaning as you have defined it.

Yes

> (D) Does having less "m=" lines justify the additional complexity of using
the
> "depend" attribute?

I think so, but I wouldn't mind some feedback on the matter. Regarding the
costs, I think the complexity introduced by "a=depend" is relatively low,
though I admit I don't have direct experience using it in implementations.
On the other hand, if an implementation was already using "a=depend" for
other purposes, this cost would be negated. Current applications requiring
the  attribute include H.264/SVC coding, for example. Also, H.264/MVC, when
it's finished, is highly likely to also use "a=depend" - and I'd expect that
format to be widely adopted by 3D-capable UAs.

As for the benefits, I've discussed the merits of reducing "m=" lines above.
On top of that, a benefit of my overall approach which I had't mentioned yet
is that it's better suited for scenarios with multiple 3D streams (e.g.
multiple displays, or 3D telepresence).In the current draft, though it is
possible for an answerer to accept more than one 3d format at once, there is
no clear way for the offerer to indicate whether it actually intends to use
more than one. An offer could have multiple 3d formats as alternatives to
each other, or as actual 3d streams that could be reproduced simultaneously.
By contrast, in my proposal, each "3DS" group would always correspond to a
separate 3d stream. Of course, this can be addressed through other kind of
update to the current draft, but that may have its own complexity cost.
Either way, this may be worth discussing as a separate topic.

> different SDP attributes?

Of all media-level attributes, the most likely to change from one 3d media
format to the other is "a=fmtp", which is defined for each payload type.
That said, there are a few potentially problematic interactions.

"a=recvonly", "a=sendonly", "a=sendrecv", "a=inactive": It would be rare,
but not impossible, for an offerer to want to specify different stream
directions depending on the 3D media format (maybe has display capabilities,
but cannot capture, for a certain format). In this specific case, a simple
re-invite once the offer/answer process has narrowed down the 3d formats to
a sigle option should work.

"b=" : clearly, different 3d formats may have different bandwidth
requirements, so having many formats under a single media descriptor makes
the "b=" attribute less accurate than under your proposal. That said, it is
already possible to have a single media descriptor with multiple 2D formats
with very different bandwidth requirements, so this is hardly a new problem.

I haven't found other media level attributes that could cause trouble. Did
you have any in  mind?