Subject: Re: [MMUSIC] New Version Notification for draft-abhishek-mmusic-superimposition-grouping-00.txt

From: Christer Holmberg <[email-masked]>
Date: Thu, 14 Jan 2021 09:54:11 +0000

Hi,

>One more thing, regarding transparency:
>
>I'm fuzzy on how this is intended to work. I would think that each superposition layer would have some parts that are fully transparent. 
>(Transparent pixels.) And then the transparency attribute would apply to the parts that aren't fully transparent. But that is only a guess. I think you need to say more about how the superposition is supposed to work.

Isn't "transparent background" information carried within the video stream (codec or container) itself?

Regards,

Christer

On 12/14/20 11:21 AM, Paul Kyzivat wrote:
> Rohit,
> 
> Here are a few more points I forgot:
> 
> 1) syntax
> 
> In section 3 you repeat the syntax for the 'mid' attribute from RFC5888. 
> I guess you don't intend it to be normative, but it gave me that 
> impression.
> 
> I don't think you need to repeat the syntax here. It should be 
> sufficient to mention that you follow RFC5888 in use of the 'mid'
> attribute to identify the media sections that are to be included in 
> the superposition.
> 
> Similarly in section 5, I suggest that you only provide ABNF for the 
> additions to syntax required for your extension. Specifically:
> 
>    semantics =/ "superposition" <semantics as defined in RFC5888>
> 
> (Everything else is inherited from RFC5888.)
> 
> 2) applicability
> 
> Regarding the following in section 3:
> 
>     ... An application that receives a session
>     description that contains "m" lines grouped together using "S"
>     semantics MUST superimpose the corresponding media streams on top 
> of
>     the background media stream.
> 
> You need to recognize that because this feature is an extension its 
> requirements can only apply to those that implement the extension. Its 
> best to call that out specifically.
> 
> Also, please give some thought to what will happen a device supporting 
> this negotiates a session with a device that doesn't. The 
> non-supporting device will treat these as independent media streams. 
> If that is a problem for you then we need to discuss strategies for 
> avoiding that situation.
> 
> 3) differing size/resolution
> 
> How the overlaying will work seems well defined if the image width and 
> height in pixels and pixel shape is the same for all the media streams 
> in the superposition group. But what if they aren't?
> 
> I think the document should specify what should happen in such cases.
> 
>      Thanks,
>      Paul
> 
> On 12/13/20 3:54 PM, Paul Kyzivat wrote:
>> On 12/12/20 2:06 PM, Christer Holmberg wrote:
>>> Hi,
>>>
>>> A few comments (some of which I gave already previously):
>>
>> Christer covered most of what I wanted to say. I'll just add a few 
>> tweaks below.
>>
>>> Q1:
>>>
>>> In Section 1, where you talk about the limitations of the mechanism, 
>>> you should mention that CLUE must be used something more "fancy" is 
>>> needed.
>>>
>>> ---
>>>
>>> Q2:
>>>
>>> In Section 3, you require a specific order of the m- lines in the 
>>> SDP. I think SDP explicitly says that the order of m- lines is not 
>>> relevant. Also, I know that parsers may change the orders of the m- 
>>> lines.
>>>
>>> (I also think that any counting should start from 1, not 0)
>>>
>>> My suggestion would be to have an explicit "order" attribute instead.
>>
>> I think it would be acceptable to use the order of the tokens in the 
>> a:group line to define the ordering of the "layers". The "lowest"
>> layer would then be the background on which the others are overlain. 
>> (Whether the lowest layer is the first or last in the list is TBD.)
>>
>>> ---
>>>
>>> Q3:
>>>
>>> The example in Section 6 talks about "a background video". But, this 
>>> background video is not described in the SDP.
>>>
>>> Related to that, if the SDP describes multiple "background videos", 
>>> you need to describe how you map superimposed videos to a specific 
>>> background video.
>>>
>>> Based on the text in Section 3 (see Q2), I am not sure whether the 
>>> 0th m- line is supposed to be the background video, but the text in 
>>> Section 6 says that both m- lines are for superimposed video streams.
>>
>> See my comment above.
>>
>> The background layer would presumably have a transparency of zero.
>>
>> (It would also be good to specify that if the transparency attribute 
>> is omitted it defaults to zero.)
>>
>>> ---
>>>
>>> Q4:
>>>
>>> What happens to the superimposed videos if I disable/remove/mute the 
>>> background video?
>>>
>>> ---
>>>
>>> Q5:
>>>
>>> This is editorial, but please call the group attribute something 
>>> else than just "S". "supim", "spim" or something :)
>>
>> Yes please. "S" is way to cryptic.
>>
>>      Thanks,
>>      Paul
>>
>>> Regards,
>>>
>>> Christer
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: mmusic <[email-masked]> On Behalf Of Rohit Abhishek
>>> Sent: lauantai 12. joulukuuta 2020 0.28
>>> To: [email-masked]
>>> Subject: [MMUSIC] FW: New Version Notification for 
>>> draft-abhishek-mmusic-superimposition-grouping-00.txt
>>>
>>> Dear All,
>>>
>>> A new version of the draft “SDP Superimposition Grouping framework” 
>>> has been uploaded.
>>> We have changed the name from  “SDP Overlay Grouping framework for 
>>> immersive telepresence media streams” to  “SDP Superimposition 
>>> Grouping framework” as the previous name wasn’t an accurate 
>>> description of the content. Therefore, we had to start from version 00.
>>>
>>> Thank you all for the comments/suggestions on the previous draft.
>>> We have tried to address them in this version. Specifically, a 
>>> transparency attribute and an informative Section on relationship 
>>> with CLUE has been added.
>>>
>>> Comments/Questions/Feedbacks are welcome.
>>>
>>> Best Regards,
>>> Rohit
>>>
>>> On 12/11/20, 12:41 PM, "[email-masked]" 
>>> <[email-masked]> wrote:
>>>
>>>
>>>      A new version of I-D,
>>> draft-abhishek-mmusic-superimposition-grouping-00.txt
>>>      has been successfully submitted by Rohit Abhishek and posted to 
>>> the
>>>      IETF repository.
>>>
>>>      Name:        draft-abhishek-mmusic-superimposition-grouping
>>>      Revision:    00
>>>      Title:        SDP Superimposition Grouping framework
>>>      Document date:    2020-12-11
>>>      Group:        Individual Submission
>>>      Pages:        8
>>>      URL: 
>>> https://www.ietf.org/archive/id/draft-abhishek-mmusic-superimpositio
>>> n-grouping-00.txt
>>>
>>>      Status: 
>>> https://datatracker.ietf.org/doc/draft-abhishek-mmusic-superimpositi
>>> on-grouping/
>>>
>>>      Htmlized: 
>>> https://datatracker.ietf.org/doc/html/draft-abhishek-mmusic-superimp
>>> osition-grouping
>>>
>>>      Htmlized: 
>>> https://tools.ietf.org/html/draft-abhishek-mmusic-superimposition-gr
>>> ouping-00
>>>
>>>
>>>
>>>      Abstract:
>>>         This document defines semantics that allow for signaling a 
>>> new SDP
>>>         group "S" for superimposed media in an SDP session.  The "S"
>>>         attribute can be used by the application to relate all the
>>>         superimposed media streams enabling them to be added as an 
>>> overlay on
>>>         top of any media stream.  The superimposition grouping 
>>> semantics is
>>>         required, if the media data is separate and transported via 
>>> different
>>>         sessions.
>>>
>>>
>>>
>>>
>>>      Please note that it may take a couple of minutes from the time 
>>> of submission
>>>      until the htmlized version and diff are available at 
>>> tools.ietf.org.
>>>
>>>      The IETF Secretariat
>>>
>>>
>>> _______________________________________________
>>> mmusic mailing list
>>> [email-masked]
>>> https://www.ietf.org/mailman/listinfo/mmusic
>>> _______________________________________________
>>> mmusic mailing list
>>> [email-masked]
>>> https://www.ietf.org/mailman/listinfo/mmusic
>>>
>>
>> _______________________________________________
>> mmusic mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/mmusic
> 
> _______________________________________________
> mmusic mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/mmusic

_______________________________________________
mmusic mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/mmusic

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 14 Jan 2021 12:25:04 -0500

> Isn't "transparent background" information carried within the video stream (codec or container) itself?

That is my impression, but I don't really know. But the transparency 
attribute must do *something*.

The draft needs to explain better.

	Thanks,
	Paul

---

From: "rabhishek(RohitAbhishek)" <[email-masked]>
Date: Sun, 17 Jan 2021 22:29:48 +0000

Hi Paul, Christer,

My understanding is the transparent attribute is same as alpha compositing which would control the linear interpolation between the foreground and the background pixels. Here’s an example wrt figure 1 in the draft.

X= Background Image
A, B, C= foreground image (Media C partially over B)
Transparency(to be applied each pixel) =alpha(A), beta (B), gamma (C)
Background media which shows through A= (1-alpha)X
Background media which shows through B and C = (1-beta)( 1-gamma)X
Overall composed image= gamma.C+ (1-gamma)[beta.B+ (1- beta.B)X]

Best Regards,
Rohit 

On 1/14/21, 9:25 AM, "mmusic on behalf of Paul Kyzivat" <[email-masked] on behalf of [email-masked]> wrote:

    	Thanks,
    	Paul

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 18 Jan 2021 10:32:16 -0500

> My understanding is the transparent attribute is same as alpha compositing which would control the linear interpolation between the foreground and the background pixels. Here’s an example wrt figur…

First, I'm concerned with you starting your reply with "my understanding 
is ...". This is your draft! The goal is to ensure that all readers come 
to the same understanding. There should be no vagueness.

Another thing: Section 5 says:

        transparency-attribute =
                     "a=transparency:" transparency-tag
        transparency-tag =tranparency-value *("," tranparency-value) CRLF
        transparency-value= alpha

    Alpha describes the transparency for the foreground media stream.  It
    is identified by its transparency-tag values in the transparency-
    attribute.  It could be an integer with values between 0 and 100.
    This is an informative value.  Details of interpretion to be left
    open to the renderer, expect that a value of 0 means foreground media
    is opaque and value of 100 means that it is transparent.

 From an ABNF perspective "alpha" is undefined. The text says it *could 
be" an integer between 0 and 100, but "details of interpretation are 
open to the renderer". That negates the value of creating a standard. 
The whole point of a standard is to ensure that interoperating parties 
handle things the same way.

> Overall composed image= gamma.C+ (1-gamma)[beta.B+ (1- beta.B)X]

I don't know much about this technology, so perhaps this would be well 
known to implementers. But it shouldn't just be assumed. If this is then 
intended mechanism then it should be specified in the document, either 
directly or by reference to some public document.

Finally, ISTM that you don't need to treat the background document as a 
special case. The grouping mechanism can be used to identify all the 
media streams participating in this superposition mechanism, including 
the one you are considering to be the background.

IIUC the scheme you describe defines the result of superposition based 
on the transparency values of the individual streams, so that no other 
ordering mechanism is needed. (Do I have that right?) If so, isn't the 
background just the media stream with tranparency = 100?

	Thanks,
	Paul

---

From: "rabhishek(RohitAbhishek)" <[email-masked]>
Date: Tue, 19 Jan 2021 07:57:56 +0000

Hi Paul,

Thanks again for your feedback.

My comments below

On 1/18/21, 7:32 AM, "Paul Kyzivat" <[email-masked]> wrote:

Rohit: I will make sure to phrase it correctly moving forward 😃

            transparency-attribute =

Rohit: The value can be between 0-1/0-10/0-100 with lowest value indicating opaque and highest value indicates that it is transparent.  Do you mean I should be specific with the values or will changing the text to " expect that lowest value means foreground media is opaque and highest value means that it is transparent" be acceptable?

Rohit: Will include the reference in the update

Rohit: The ordering mechanism will be needed. Suppose we have a background X with transparency=100, and a superimposed media A with transparency=50. A over X would mean that X would visible partially through A, however X over A would mean that A wouldn't show through X.

Best Regards,

Rohit

                Thanks,

                Paul

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 19 Jan 2021 12:16:53 -0500

Rohit,

On 1/19/21 2:57 AM, rabhishek(RohitAbhishek) wrote:
>      > Hi Paul, Christer,
>      >
>      > My understanding is the transparent attribute is same as alpha 
> compositing which would control the linear interpolation between the 
> foreground and the background pixels. Here’s an example wrt figure 1 in 
> the draft.
> 
> *Rohit: I will make sure to phrase it correctly moving forward **😃***
> 
> *Rohit: The value can be between 0-1/0-10/0-100 with lowest value 
> indicating opaque and highest value indicates that it is transparent. 
>   Do you mean I should be specific with the values or will changing the 
> text to " expect that lowest value means foreground media is opaque and 
> highest value means that it is transparent" be acceptable?*

You have to imagine that the offerer and the answerer know nothing about 
each other. So the offerer needs to understand: is the value to be from 
0-1, 0-10, or 0-100? And is it an integer or a real.

So you need to decide this and put it in the spec. And define the syntax 
and associated normative text to make this clear.

And yes, also the significance of lower and higher values.

>      > X= Background Image
> 
>      > A, B, C= foreground image (Media C partially over B)
> 
>      > Transparency(to be applied each pixel) =alpha(A), beta (B), gamma (C)
> 
>      > Background media which shows through A= (1-alpha)X
> 
>      > Background media which shows through B and C = (1-beta)( 1-gamma)X
> 
>      > Overall composed image= gamma.C+ (1-gamma)[beta.B+ (1- beta.B)X]
> 
> **
> *Rohit: Will include the reference in the update*

That will help.

> *Rohit: The ordering mechanism will be needed. Suppose we have a 
> background X with transparency=100, and a superimposed media A with 
> transparency=50. A over X would mean that X would visible partially 
> through A, however X over A would mean that A wouldn't show through X.*

If the ordering is needed, then you need to specify how it is 
determined. IMO it is permissible to use the order of values in the 
a=group: line to specify this ordering. (RFC5888 doesn't ascribe any 
semantics to this ordering of values, but neither does it forbid 
assigning meaning for particular grouping semantics.) I am not certain 
if Christer agrees with me on this point. The alternative is to use a 
new attribute in the grouped media descriptions for this purpose.)

And then you really need to specify the algorithm by which the layering 
plus the transparency is applied to the media streams to yield the 
composed media stream.

	Thanks,
	Paul

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 19 Jan 2021 13:44:34 -0500

Rohit,

I had another thought. See below.

> On 1/19/21 2:57 AM, rabhishek(RohitAbhishek) wrote:

[snip]

> new attribute in the grouped media descriptions for this purpose.)

Perhaps a clean way to do this would be to replace the "transparency" 
attribute with one that specifies both the transparency and the order. E.g.,

    a=superposition:transparency=80,layer=3

	Thanks,
	Paul

---

From: Christer Holmberg <[email-masked]>
Date: Tue, 19 Jan 2021 18:45:00 +0000

Hi,

...
 
> If the ordering is needed, then you need to specify how it is determined. IMO it is permissible to use the order of values in the
> a=group: line to specify this ordering. (RFC5888 doesn't ascribe any semantics to this ordering of values, but neither does it forbid assigning meaning for particular grouping semantics.)
> I am not certain if Christer agrees with me on this point. 

In BUNDLE we define semantics for the ordering of the values, by saying that the first value in the attribute line represents the so called "tagged m- line". The order of the other values have no semantics, though.

> The alternative is to use a new attribute in the grouped media descriptions for this purpose. 

Correct. If we anyway are going to need at least two values (order and value), one option is to put them in the same attribute.

Regards,

Christer

---

From: Christer Holmberg <[email-masked]>
Date: Tue, 19 Jan 2021 18:49:15 +0000

Hi,

...

>    a=superposition:transparency=80,layer=3

And let's make that the ABNF allows new values, to avoid some of the pain if additional value(s) would be needed at some point :)

Regards,

Christer