Subject: Re: [AVT] Clarification on Offer Answer usage of H.264 payload format (RFC 3984)

From: Randell Jesup <[email-masked]>
Date: Tue, 04 Mar 2008 19:11:23 -0500

This is a response to a VERY old message (and there's been one long
discussion of this since then (last summer)).  

Magnus Westerlund <[email-masked]> writes:
>We authors was made aware of an issue in the Offer/Answer usage for the
>H.264 RTP payload format by Tang Yongliang.
>
>The issue is that section 8.2.2 says:
>
>    o  The parameters identifying a media format configuration for H.264
>       are "profile-level-id", "packetization-mode", and, if required by
>       "packetization-mode", "sprop-deint-buf-req".  These three
>       parameters MUST be used symmetrically; i.e., the answerer MUST
>       either maintain all configuration parameters or remove the media
>       format (payload type) completely, if one or more of the parameter
>       values are not supported.
>
>And later
>
>"  o  Parameters declaring a configuration point are not downgradable,
>       with the exception of the level part of the "profile-level-id"
>       parameter."
>
>That is contradicting for the "profile-level-id" parameter. The intention
>of the authors was that the profile part (profile_idc and profile_iop)
>would be fixed but the level (level_idc) can be downgraded. That way you as
>an receiver state the profiles and highest level for that profile you are
>willing to accept. The answerer can then respond with the same profile but
>with a possibly lower level.
>
>So for clarification: An answerer MUST only maintain the profile_idc and
>profile_iop bytes of the profile-level-id value and MAY downgrade the level
>part.
>
>Please note: This may require the offering party to make a new offer to
>provide its "sprop" parameters correctly due to the reduction in level.
>
>However without this clarification the only way of getting a successful
>offer/answer for H.264 when not fully aware of the counter-parts
>capabilities would be to write one payload type for each profile-level
>combination that one could downgrade to.
>
>Any comments on this clarification?

Yes:

There is another side-effect on the text of RFC 3984 of this clarification.

In particular, this text (in description of Offer/Answer):

  o  The "sprop-parameter-sets" parameter is used as described above.
      In addition, an answerer MUST maintain all parameter sets received
      in the offer in its answer.  Depending on the value of the
      "parameter-add" parameter, different rules apply: If "parameter-
      add" is false (0), the answer MUST NOT add any additional
      parameter sets.  If "parameter-add" is true (1), the answerer, in
      its answer, MAY add additional parameter sets to the "sprop-
      parameter-sets" parameter.  The answerer MUST also, independent of
      the value of "parameter-add", accept to receive a video stream
      using the sprop-parameter-sets it declared in the answer.

         Informative note: care must be taken when parameter sets are
         added not to cause overwriting of already transmitted parameter
         sets by using conflicting parameter set identifiers.

is a real problem.

If you offer level 3.0, and include a level 3.0 sprop-paramater-set (which
you should), then per the above, I MUST maintain the 3.0 parameter-set in
my response.  If I want to respond with level 1.0, Magnus says above
that I can, but I'm going to have to include a level 3.0 parameter set.
I can also (if allowed by parameter-add) add a level 1.0 set.  When you add
this:

 o  The parameters "sprop-parameter-sets", "sprop-deint-buf-req",
      "sprop-interleaving-depth", "sprop-max-don-diff", and "sprop-
      init-buf-time" describe the properties of the NAL unit stream that
      the offerer or answerer is sending for this media format
      configuration.  This differs from the normal usage of the
      Offer/Answer parameters: normally such parameters declare the
      properties of the stream that the offerer or the answerer is able
      to receive.  When dealing with H.264, the offerer assumes that the
      answerer will be able to receive media encoded using the
      configuration being offered.

That says the offerer should assume the answerer can receive it, and the
previous quote says that the answerer MUST be willing to receive any
parameter-set in the answer, and it also says the answerer MUST maintain
the original parameter sets in the answer.  Since in most cases, the reason
for downgrading the level in a response is the inability to handle the
level of the offer, the answerer won't be able to handle video encoded
using the parameter sets from the offer.  

Thus, I see no way an answerer can validly answer with a downgrade unless
it internally decides to lie, and drop all video packets until the offerer
renegotiates to remove the unusable parameter sets.

Also, to verify: does this mean that offer-matching is done using the first
4 digits of profile-level-id (and packetization-mode and
sprop-deint-buf-req as needed)?  Should "is the same" (when applied to
profile-level-id) always mean "first 4 digits the same, and last two digits
be the same or lower"?

This really, really, really needs a rewrite.  It's also so complex, it
needs more examples, and if possible an informational(?) algorithm on
how to respond. Right now, I'm not certain *anyone* has come even close to
100% compliance.  (Maybe one or two have, but...)

Existing examples of H.264 traces I've seen show huge variations:

*) Many respond without a matching packetization-mode (i.e. respond to an
   offer of packetization-mode=1 with no packetization-mode)
*) Some respond with a level *higher* than the offer.
*) None I've seen include the sprop-parameter-sets from the offerer.
*) I haven't seen any that respect parameter-add, though given the previous
   point it may not matter.
*) Some don't even include profile-level-id at all.
*) Some accept packetization-mode=1 in the SDP, but send packetization-mode=0.
and there are more problems, that's just the start.  :-(

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
[email-masked]
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www.ietf.org/mailman/listinfo/avt

---

From: Randell Jesup <[email-masked]>
Date: Tue, 11 Mar 2008 23:46:04 -0400

Someone replied privately; I'd like to make my response public but don't
have feedback from the sender to disclose who it was (though it's not
controversial at all, just politeness)

Note: I wrote this before the AVT meeting today.

Someone wrote (in private email):
>Ow.  Much of this concerns the complexity of the interleaved modes; does
>anyone actually implement them at all, let alone in offer/answer?

Actually, the real problem here is the mixture of the clarification which I
replied to, that level downgrading is allowed, with sprop-parameter-sets.
(And the general complexity, and that some are "I will send" parameters,
and the "match by level/packetization/deint" bit, etc.)

The examples I gave of negotiation problems don't even include any of the
interleaved modes (packetization-mode=2).  They just involve modes 0
(single NAL) and 1 (non-interleaved, with STAPs and FU-A's, etc).

>>and there are more problems, that's just the start.  :-(

So, how do we avoid specifying every possible level (and constraint
setting!)?  Plus all the parameter sets for each one.

How important is compatibility with existing implementations?  What's the
most-common actually-implemented-and-works subset?

We could use re-INVITE, but that adds additional layers of exchanges.

It would help to relax the requirement to include the parameter-sets from
the offer in the response, effectively allowing the two directions to be
more different - but that means that if the answer dropped the level, the
offerer would have to send one of the parameter sets from the response,
even though it had offered a higher level and parameter-set.

Perhaps allow level downgrading only if either a) the offerer included an
parameter-set for that level in the reply, or b) the offerer agrees to use
the parameter-set from the response, and if for some reason it can't be
used, the offerer will re-INVITE at the lower level with their own
parameter-set for that level, or c) they agree to in-band paramsets.

Does the parameter-set from the offer *need* to be in the response, if the
levels match?  Parameter sets are what the provider plans to send; if they
were what you plan to receive, then it would make sense.  Just because you
expect/agree to receive a specific parameter set doesn't mean you should
tell the offerer that you might send it.  I'm not sure it hurts, except
that if you want to send a different parameter set you need to make sure
they don't conflict.  This is doable, but requires much more interaction
with the codec and the signalling layer.

---

From: Randell Jesup <[email-masked]>
Date: Wed, 12 Mar 2008 00:36:06 -0400

To summarize the discussion today at the AVT meeting on RFC 3984bis and
open issue #7 - the interaction between "level downgrading" and
sprop-parameter-sets, and my suggestions:

1) The intent originally was to allow level-downgrading (per the 2-year-old
   clarification).  Note: this assumes that any decoder that can decode
   level N can decode levels 1.0 - N.  (Also note that level 1b is a
   confusion factor here.)  Also note this assumes that level-downgrading
   is due to inability (in some way) or unwillingness to handle levels
   higher than the response indicates.

2) Doing so interacts badly with how sprop-parameter-sets is handled in the
   spec, since a level downgrade may leave you with unusable parameter
   sets.  I'd contend it's close to impractical/impossible to implement as
   written.

3) Many existing implementations do not seem to have implemented the spec
   as written, especially when it comes to packetization-mode and 
   sprop-parameter-sets (and parameter-add).  (MASSIVE UNDERSTATEMENT)

If anyone disagrees with those assertions, please speak up.

Possible solutions/resolutions discussed:

1) Disallow level downgrading.  This was not the intent; existing
   implementations downgrade, and it would mean allowing downgrading
   requires a large number of payload types be added as lower-level
   alternatives.

2) Specify parameter-sets that include parameters for each level below the
   one offered.  This is smaller and requires less explosion of SDP lines
   than option 1, but still may require a LOT of parameter-sets, and some
   of them can be large.  The answerer can chose any lower level for which
   there's a parameter set.

3) Find some way that the answerer can modify the offered parameter-set to
   lower the level.  Since modifying a paramset you didn't generate is
   unlikely to be a workable idea (H.264 specialists, feel free to correct
   this impression!), perhaps the idea would be something like: "on a level
   downgrade, the parameter-sets offered in the response shall be used by
   the offerer if acceptable, otherwise the offerer shall re-negotiate the
   session using the lower level from the response".

   Downside: requires a possible re-INVITE/etc.

4) On level downgrade (or on downgrade when there was no parameter-set
   for the new level), re-INVITE/etc (re-negotiate).  (See previous answer
   for a way to avoid this in some cases.)

(Not really discussed in detail in the meeting):
5) Allow in-band parameter sets explicitly and allow a way to signal their
   use in SDP, either in general or for this case.

   This allows the actual parameter sets to be in the stream, so the
   offerer can choose the sets after seeing the level-downgrade.

   Downside: parameter-sets are now contained in unreliable transmission,
   and so can be lost.  An IDR request (FIR/etc) could be defined to (in
   this case at least) regenerate the parameter-sets in the stream, but
   this won't help cases with a non-existant or non-functional return-path.

   This could be something like:

      parameters-in-media=1

   or implicitly signalled by a lack of sprop-parameter-sets.  I'm not sure
   which one would be more compatible with existing implementations -
   comments?

   That also could be in an offer (or a lack of sprop-parameter-sets
   implicitly already does this).

   This has the (MAJOR) advantage of trying to standardize existing
   (mis)-use of RFC 3984.  This is done all the time; many implementations
   totally ignore sprop-parameter-sets, let alone deal with the subtleties
   of parameter-add, or the offer-answer requirement to include the offer's
   parameter-set, and make sure ones added don't conflict with it.

   This current incompatibility between implementations that try to follow
   the RFC as written and others that kinda-follow it is a REAL problem.  I
   know (confirmed by Roni) that most of the conferencing/H.323-derived
   H.264 devices implement in-media parameter-sets.

   Note that using in-media parameter sets may limit the ability to
   change them reliably in mid-stream, if that's an issue, and it may not be.

Some additional issues for 3984bis:

3984bis should include the clarification about how level 1b is signalled
(reserved bit in the middle byte is used for constraint_3, per the email
on AVT).

How does level downgrade to level 1b work?  Is it allowed?  (Note that
level 1b is signalled as "level 1.1 with constraint_3 set".)  What about
level downgrade *from* level 1b? (to 1.1 or 1.0)?  This was discussed, but
the answer/consensus wasn't clear (to me), and time was an issue at
the meeting due to SVC taking a lot....

How does level downgrade interact with the text on offer-matching, since
it's matched on profile-level-id, packetization-mode, and (for mode 2)
sprop-deint-buf-req?

How does all this interact with MCU/conferencing servers?

We *really* need to get H.264 users on at least the same "chapter" when it
comes to SDP negotiation; right now they're barely in the same book, let
alone chapter or page.  And really, unless we want to majorly rewrite the
3984 rules, everyone *needs* to support mode 1.  It's not that hard, guys,
and it cuts the overhead of in-media parameter-sets because you can use a
STAP-A packet.  I suspect it also helps compression (a little? more? at
all?), especially in smaller-MTU transports like cellphones, since
NAL/slice size isn't as constrained.

Right now, it seems like barely a majority (if that) even know mode 1
exists.

For reference, I'm including my previous email, though I'm not really
replying to it directly.

---

From: Randell Jesup <[email-masked]>
Date: Thu, 13 Mar 2008 00:58:44 -0400

>sprop-parameter-sets, and my suggestions:

With a mind to not only solving the level downgrade issue, but also to
finding some way to standardize current practice, which often doesn't

>   offerer can choose the sets after seeing the level-downgrade.

The way this might work:

1) Offer (only) in-band parameter sets:
  * Generate an offer with no sprop-parameter-sets
  * Add a=parameters-in-media=1

  *   If response has parameters-in-media, you know both sides agree to
      in-media params.  Note: if sprop-parameter-sets is there, those
      parameters must be installed by the offerer before decoding.
      Additional or duplicate parameter sets may occur in-media.
  *   If the response doesn't have parameters-in-media and doesn't have
      sprop-parameter-sets, then the answerer is older but wants params
      in-media (it has to).
  *   If the response doesn't have parameters-in-media, but does have
      sprop-parameter-sets, then the offerer should assume that it will
      receive a stream using those parameters.  It should send parameter
      sets in-media, but not assume it will receive them.
  
2) Offer both in-band parameter sets but allow answers to use them
  * Generate an offer with sprop-parameter-sets and parameters-in-media=1

  *   If the answer has only sprop-parameter-sets, then both sides
      user the parameter sets, and SHOULD re-negotiate to add/change
      parameter sets.
  *   If the answer has only parameters-in-media, then the offerer
      can assume the answerer will install the parameter sets in the offer,
      but the offerer can add/change parameter sets in-media.  The offerer
      will assume that all received parameter sets will be in-media
  *   If the answer has both, then both sides will install the parameter
      sets from the other side, and allow changes/additions in-media.

Or, in another, simpler form:
-----------------------------
* If the other side provides parameter sets, you must install them
  (Note: see corollaries below)
* If the other side doesn't provide parameter sets, assume in-band sets
* If the other side gives PIM, you can expect them to (possibly)
  send/update parameter sets in-media even if they also provided parameter
  sets.

Corollaries:
-----------
* If you want to allow downgrading levels, you should include
  parameters-in-media=1 (so you can send in-band parameter sets on a
  downgrade)
* An answerer can downgrade if the offer has parameters-in-media=1 or
  does not have sprop-parameter-sets.
* The answerer need only install parameter sets from the level accepted (or
  lower?)

Optimizations:
--------------
* To avoid sending (potentially largish?) parameters sets on every IDR,
  especially for high-loss uses, an additional CCM message should be
  defined (separate draft? I'd prefer to include in this one).  The CCM
  message should ask for parameter sets.

  A lost parameter set at startup would result in a CCM for parameter sets
  AND a FIR or NACK CCM request, unless by signalling we know that the
  other side supports the parameter-set CCM (and we probably would...).
  Once parameter sets are received (note: how do we know we got all of
  them?), later packet losses can generate just a FIR or NACK.

* Alternatively, we could define an alternative to FIR that requests
  an IDR without parameter sets, and (re)define the current FIR to require
  them.  Many implementations send parameter sets on every IDR now.

* We don't *need* to use special CCMs if a NACK is used, since we're
  telling the sender which packets were lost, though this imposes a
  requirement on the sender to remember which packets are parameter sets.
  Also, it could cause problems with lost parameter sets at startup - the
  NACK might not realize N packets were lost at the start of the stream.
  I'd recommend if this approach is take, a FIR should be used instead
  of NACK until (all?) parameter sets are received.

* If sending parameter sets with every IDR, packetization-mode 1 is
  more efficient (lets you use STAP-A).

Question: can we somehow infer or know if "all" in-band parameter sets have
been received, especially if there are more than 1 each of sequence and
picture sets?  Or we could signal the number, or use errors from the
decoder to know if we get a NAL that references an un-seen parameter set?

To those who work in the videoconferencing world, how would this affect
you?  How much closer to interoperating with your existing equipment would
this come?  (A compliant 3984 implementation would seem to have a lot of
problems with existing videoconferencing equipment.)

Phew!!!

---

From: Randell Jesup <[email-masked]>
Date: Fri, 14 Mar 2008 17:52:57 -0400

>come close to the current RFC 3984:

And replying to myself for a second time....  :-)

Another couple of points I should note which are part of the discussion on
parameter sets:

In an offer-answer negotiation when relying on sprop-parameter-sets,
pre-answer-media (i.e. media that arrives before the 200 OK or 183 Call
Progress, etc) will NOT be decodable.  It's only decodable if there are
in-media parameter sets (and they don't get lost!), or if the answer has
been processed.  When the answer arrives, the parameter sets need to get
installed into the decoder at that point.

For in-media parameter sets, you can't change them in any incompatible
manner during a stream unless you have some (out-of-band, reliable?) method
of telling the other side that they need to stop decoding until they get
the new, incompatible paramsets.  (And to send requests for IDRs or
paramsets as needed to handle loss.)  Compatible changes (using different
paramset indexes, etc) are ok, though again you need an IDR request or a
paramset request to handle loss.

*None* of these should be news to anyone, but I thought it's important
to include in the discussion of pros and cons of paramset negotiation.

---

From: Randell Jesup <[email-masked]>
Date: Thu, 20 Mar 2008 17:57:09 -0400

So, just reminding people, now that they're home from IETF - this is a
real issue with RFC 3984 that needs to be worked on so we can get some sort
of reasonable level of interoperability.  Currently, both spec-compliance
and interoperability suck industry-wide for H.264 RTP, and as written
(including the current 3984bis) there are areas that effectively are
un-implementable (like level downgrade).

---

From: "Even, Roni" <[email-masked]>
Date: Fri, 21 Mar 2008 00:40:37 +0200

Randell,
Since most video conferencing implementations did H.323 they understood
that level downgrade was OK. I also assume that others understood it to
be OK based on understanding the profile and level concept. I think this
is the common practice.
Roni

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Randell Jesup
> Sent: Thursday, March 20, 2008 11:57 PM
> To: IETF AVT WG
> Cc: [email-masked]; Magnus Westerlund; Stephan Wenger; Dave
Singer;
> [email-masked]
> Subject: Re: [AVT] H.264/RFC 3984bis and SDP - levels, parameter-sets,
etc.
> Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga
OS
> team
> [email-masked]
> "The fetters imposed on liberty at home have ever been forged out of
the
> weapons
> provided for defence against real, pretended, or imaginary dangers
from
> abroad."

---

From: Randell Jesup <[email-masked]>
Date: Thu, 20 Mar 2008 23:18:23 -0400

>be OK based on understanding the profile and level concept. I think this

I understand it's the desired behavior, but in 3984 it's not mentioned
(except for the on-list clarification by Magnus), and 3984bis (which
attempts to correct that oversight) shows that the idea was not properly
thought out during design of 3984.  This isn't too surprising, since it
wasn't actually mentioned, and if people had been considering it several
parts (like the offer/answer matching stuff) would have been written
differently. 

I say that because a level downgrade simply doesn't work with the
recommended negotiation based on sprop-parameter-sets (or if you prefer,
requires some very painful, large, complex and effectively unworkable
SDPs).  Level downgrade without sprop-parameter-sets is probably
not inherently problematic, though the text proposed is insufficient
to describe the behavior even if you limit it to the "no
sprop-parameter-sets" case.

Now there are solutions (as discussed), though there are downsides,
limitations or side-effects to all of them.  If we're going to do a
3984bis, we need to figure out some solution, or remove it (again).

There are related issues to address in 3984bis, such as whether we can do
anything to increase the chances of people supporting the spec as written
(and not a subset), and if they do, increasing the chances of them actually
succeeding in this effort.  The sampling of soft and hard clients I've seen
shows that few if any come close to a conforming implementation (even
ignoring the level downgrade stuff).

If you read the earlier messages in this thread, I'm proposing (as a
possible solution and codification of existing(?) practice) to suggest
in-band parameter sets for at some uses.  As for packetization-mode, we can
either increase pressure on people to support it (it's not that hard to
minimally support it, though it is some work), or try to codify how to
handle people who ignore it (which may have some compatibility risks, but
probably less than the alternative).

Since we discussed it a bit after the AVT meeting at IETF, here are 
a few (sanitized) examples of responses to (roughly):

m=video 4321 RTP/AVP 99
a=rtpmap:99 H264/90000
a=fmtp:99 profile-level-id=428014;packetization-mode=1;parameter-add=0;sprop-parameter-sets=<base-64 data>

I've seen worse than these, and I've seen one or two that come close to correct.
Note: at least two or more of these 4 clients I mention are from major
videoconferencing makers.

Client 1:
  The SDP response is incorrect: the ftmp line for H264 includes a
  level-id, but does not include an sprop-parameter-sets or
  packetization-mode.  They do have the profile-level-id correct in the
  response, but not the packetization-mode.
  Two minor points: The initial sequence and timestamp numbers are not
  properly randomized. It uses the same SSRC for audio and video, which is
  allowed but I would not suggest, and if there's a collision, it's required to
  change SSRCs for that stream.

Client 2:
  The SDP response is incorrect: we offered profile-level-id of 428014,
  they responded with 42801f (higher!), and there is no
  sprop-parameter-sets or packetization-mode.  There is no match on media
  unless profile-level-id and packetization-mode match.

Client 3:
  When we call them, their SDP response is significantly invalid.  It
  accepts video (apparently H264), but includes no rtpmap line(!!) and no fmtp
  lines for video, so it's unclear what they meant to do.

  When they call us, their SDP has no profile-level-id specified.  We
  respond using ours, but I'm fairly certain the two sides aren't agreeing
  on the profile-level to use.  We do properly respect the request that
  they prefer packetization-mode=0, and include that in the response.

Client 4:
  The SDP response is incorrect; they do have sprop-parameter-sets, but
  they change the values of both profile-level-id and sprop-parameter-sets.
  Changing sprop-parameter-sets is not allowed with parameter-add=0, and
  even without it you're not allowed to remove a parameter set.  They do
  correctly include packetization-mode=1.
  Minor: None of the H264 packets from them have the 'Mark' bit set to indicate
  end-of-frame (per RFC 3984).

---

From: "Jaehwan Kim" <[email-masked]>
Date: Fri, 21 Mar 2008 12:24:04 +0900

Hi all,
Recently we found some common mistake regarding width-height signaling
in SDP. Although we can parse it from sprop-parameter-sets, it looks not
easy and not human readable. What's more, it is encoded Base64.
Therefore, still some vendors depend on a=framsize which was defined in
3GPP or proprietary ways. 

If it does not create big confusion and really simple add, why don't we
add width-height to 3984bis?

I appreciate your opinion in advance on this issue.

Regards,
Jaehwan
Vidiator

---

From: <[email-masked]>
Date: Fri, 21 Mar 2008 13:41:07 +0200

Such a parameter has been currently included in
draft-ietf-avt-rtp-svc-08, copied below for convenience. 

sprop-spatial-resolution: This parameter MAY be used to indicate the
property of a stream or the capability of a receiver or sender
implementation.  The value is a base16 of the width and height of the
spatial resolution, in pixels, separated by a comma. 

However, as can be seen from the associated editing notes in
draft-ietf-avt-rtp-svc-08, this parameter should go to a draft for
generic SDP extensions, as it is not specific to H.264 or SVC, it
applies to any video codec. 

BR, YK

>From: [email-masked] [mailto:[email-masked]] On 
>Behalf Of ext Jaehwan Kim
>Sent: Friday, March 21, 2008 5:24 AM
>To: IETF AVT WG
>Subject: [AVT] H.264/RFC 3984bis and SDP - width & height
>
>Hi all,
>Recently we found some common mistake regarding width-height 
>signaling in SDP. Although we can parse it from 
>sprop-parameter-sets, it looks not easy and not human 
>readable. What's more, it is encoded Base64.
>Therefore, still some vendors depend on a=framsize which was 
>defined in 3GPP or proprietary ways. 
>
>If it does not create big confusion and really simple add, why 
>don't we add width-height to 3984bis?
>

---

From: Dave Singer <[email-masked]>
Date: Fri, 21 Mar 2008 11:24:39 -0700

>spatial resolution, in pixels, separated by a comma.

what do you mean by 'base 16'?  It's expressed as a hexadecimal 
number, or the number must be a multiple of 16, or it is the width/16 
and the height/16?

Generally, I'd prefer a decimal representation of the 'natural' width 
and height of the video (like framesize).

>applies to any video codec.

Indeed, for many years we (and others) would indicate the 'usable' 
picture from an H.261 or H.263 bitstream with a cliprect parameter, 
indicating top,left,bottom,right of the useful pixels in e.g. a CIF 
image.  (It was all 4 numbers as VIC, bless it, used to center the 
video in the frame.  Most other encoders put it top-left).

-- 
David Singer
Apple/QuickTime

---

From: "Even, Roni" <[email-masked]>
Date: Sat, 22 Mar 2008 01:27:38 +0200

Hi,
The parameter set give the information about the size, note that you
also need to look at the pixel aspect ratio information in the VUI to
scale to display.
H.264 encoder does not define the frames size but uses NAL units, it is
not required to support scaling.  The level parameter defines the mbps
and fs parameters that gives potential picture size look in annex A of
H.264. H.241 for H.323 systems and RFC 3984 defines the max-mbps and
max-fs that enable a receiver to show that it can support bigger picture
size than the levels in annex A. This for example enable a receiver to
support bigger frames at lower frame-rate.
It is up to the decoder to scale the received data to the display it
wants.
Look also at
http://tools.ietf.org/html/draft-ietf-avt-rtp-h264-params-01 that will
be merge to RFC3984bis that adds the picture aspect ratio information.

There is work going on at the ITU that will enable a receiver to request
from a sender (if it can scale) for a specific frame size and aspect
ratio. Note that today the screens can be 4:3 or 16:9 and thus clipping
may be required, this is why the receiver will get better picture if it
can signal its display aspect ratio  
Roni Even  

> Jaehwan Kim

---

From: Randell Jesup <[email-masked]>
Date: Fri, 21 Mar 2008 19:57:19 -0400

>and height of the video (like framesize).

I'd agree, though base16 (hex) of the actual width/height is ok (they do
say "in pixels", so it's not pixels/16, etc). I'd slightly rewrite it as:

   sprop-spatial-resolution: This parameter MAY be used to indicate the
   property of a stream or the capability of a receiver or sender
   implementation.  The value is a base16 [6] (hexadecimal) representation
   of the width and height of the spatial resolution, in pixels, separated
   by a comma.

(See profile-level-id, which is also in hex).

NOTE: In offer-answer, unless otherwise specified this would be (I assume)
the maximum capability of the receiver.  The text from SVC is overly vague
as to which direction it applies to.  I'd assume it's a receiver capability
in offer-answer, and a sender capability in declarative.  I should also
note (as people know) that currently (in RFC 3984 and H.264 in general)
resolution is specified in total macroblocks and this is associated with
the level, modified by max-fs.

When you combine this with level (or max-fs), there's a question about
what it means, and how the sender should use this information.  What does
it mean if it implies less macroblocks than the level/max-fs indicate?
Is it a preference, a maximum, a minimum, a list (of "good" resolutions),
etc?   What if it implies more macroblocks than the level/max-fs
indicate?  More fundamentally, what is the goal of this parameter?

Since maximum resolution is implicitly indicated with max-fs (though note
you can't reduce the fs with max-fs, only increase it), I'd suggest
this be a preferred resolution.  (Like "I can decode level N (which allows
1920x1080), but prefer 800x480 because that's the resolution of my display".)
In declarative mode, it should be the resolution to expect (though the
video stream would not be required to adhere to that; or we could say that
the stream would be that resolution or less).  Also, how does this interact
with the draft we're going to merge into 3984bis on aspect ratio?

None of these questions mean it's a bad idea, but they mean the current
definition above would most likely *increase* problems for interoperation
due to an ambiguous/missing definition of what to do with it.

Anyone working on the SVC draft should take note!!

>>applies to any video codec.

Agreed, though specific RFCs should talk about how this interacts (where
needed).  They may not need to if it's a preference, depending on what's
said about it in this theoretical generic spec.  Note that the SVC people
(per the discussion at IETF 71) may not be willing to wait for a new
draft to go through the whole process.  We may be able to define it in
SVC and 3984bis such that we can generalize it with a separate draft that
works with the SVC/3984bis definition.

>video in the frame.  Most other encoders put it top-left).

Example?  Does anyone still use this?

---

From: Randell Jesup <[email-masked]>
Date: Fri, 21 Mar 2008 20:10:39 -0400

>also need to look at the pixel aspect ratio information in the VUI to

However, there may be multiple parameter sets with different resolutions in
the SDP, and they may be absent from the SDP.  Also, in-media parameter sets
might be sent with different resolutions than the ones in the SDP.  So I
would not recommend parsing the parameter-set values in the SDP.  (Plus
you'd need to be able to decode a picture-parameter-set NAL.)

>can signal its display aspect ratio  

This would tie into the questions I raised in the email I just sent in
reply to David Singer.  If you or someone could tell us what they're
planning, we might be able to do something complimentary in 3984bis or
whatever.