Subject: [AVT] Submission and request for feedback on draft-valin-celt-rtp-profile-00.txt

From: Gregory Maxwell <[email-masked]>
Date: Sat, 7 Mar 2009 17:54:29 -0800

I'd like to solicit feedback on the CELT RTP profile proposal.

CELT is an audio codec which can provide high-quality and low-latency. Its
loss robustness, rate adaptability, and IPR-free status make it a good
fit for the Internet community. CELT is suitable for both speech and
music in VoIP/telepresence applications and, with codec delays as low
as 2ms, new applications such networked musical collaboration are possible.

More information on CELT can be found at http://www.celt-codec.org/

Due to some confusion with respect to the cut-off dates we've missed
the formal submission window for -00 drafts. Would it be possible to
submit this draft anyway? I am planning on attending IETF 74. 

In the interim the proposed profile is available at:

http://people.xiph.org/~jm/draft-valin-celt-rtp-profile-00.txt

I'm especially interested in feedback from application developers with
respect to difficulties in negotiating/handling multiple sample rates and
frame sizes as well as the use of the SDP ptime parameter, considering
that many desirable packetization intervals do not fall on integer values.

We've had a number of early adopters producing their own non-interoperable
and not deeply considered packetization formats, so we consider the
matter of creating a recommended profile to be important even though
the CELT bitstream itself is not yet finalized.

--
Gregory Maxwell

---

From: Randell Jesup <[email-masked]>
Date: Sat, 07 Mar 2009 23:53:43 -0500

> 3.2.  CELT payload
>    An RTP packet MAY contain CELT frames of the same bit rate or of
>    varying bit rates, since the bitrate for the frames is explicitly
>    conveyed in band with the signal.  The encoding and decoding
>    algorithm can change the bit rate at any frame boundary, with the bit
>    rate change notification provided in-band.  No out-of-band
>    notification is required for the decoder to process changes in the
>    bit rate sent by the encoder.

I assume that the in-band signalling of the bitrate isn't affected by
packet loss (i.e. each frame includes implicitly or explicitly the
whatever bitrate information is needed to decode it.)

>    It is RECOMMENDED that sampling rates 32000, 44100, or 48000 Hz be
>    used for most applications, unless a specific reason exists -- such
>    as requirements for a very specific packetization time.  For example,
>    51200 Hz sampling may be useful to obtain a 5 ms packetization time
>    with 256-sample frames.  For compatibility reasons, the sender and
>    receiver MUST support 48000 Hz sampling rate.
> 
>    The CELT codec always produces an integer number of bytes and can
>    produce any integer number of bytes, so no padding is ever required.
>    Bitrate adjustment SHOULD be used instead of padding.

Not really clear why this is here, and not sure what exactly is being
suggested in order to pad (though padding appears not be needed).

> 3.3.  Multiple CELT frames in a RTP packet
> 
>    The bitrate used by CELT is implicitly determined by the size of the
>    compressed data.  When more than one frame is encoded in the same
>    packet, it is not possible to determine the size of each encoded
>    frame, so the information MUST be explicitly encoded.  If N frames
>    are present in a packet, N compressed frame sizes need to be encoded
>    at the beginning of the packet.  Each size that is less than 255
>    bytes is encoded in one byte (unsigned 8-bit integer).  For sizes
>    greater or equal to 255, a 0xff byte is encoded, followed by the
>    size-255.  Multiple 0xff bytes are allowed if there are more than 510
>    bytes transmitted.  The length is always the size of the CELT frame
>    excluding the length byte itself.  The payload MUST NOT be padded,
>    except in accordance with the padding bit definition in the RTP
>    header.

The description of how >= 255 byte packets is slightly confusing; it could
use a re-write.  No technical problem, though.

> 5.  SDP usage of CELT
> 
>    When conveying information by SDP [rfc2327], the encoding name MUST
>    be set to "CELT".  The sampling frequency is typically between 32000
>    and 48000 Hz.  Implementations SHOULD support both 44100 Hz and 48000
>    Hz.  The maximum bandwidth permitted for the CELT audio is encoded
>    using the "b=AS:" header, as explained in SDP [rfc2327].

Above it said 48000 is a MUST, so it should say that here.  

You should not be using b=AS: here for codec bandwidth settings.  b=AS
applies to ALL payloads negotiated for a session; the bandwidth for
the CELT payload may be different (and in fact you could offer several
payload values, each with a different bandwidth specified in the a=fmtp
line for each payload.  So, bandwidth moves to fmtp.

>    The SDP parameters have the following interpretation with respect to
>    CELT:
> 
>       b=AS: The maximum bandwidth (in kbit/s) allowed for CELT,
>       excluding the header overhead.  The default is 64 kbit/s.

See above.

>       ptime: The desired packetization time.  The sender SHOULD choose a
>       number of frames per packet that corresponds to the smallest
>       packetization time greater or equal to the specified ptime for the
>       selected frame size.  The default is 20 ms as specified in
>       [rfc3551]

Ok, though realize again that ptime is shared by all payloads.  You may
want to consider negotiating it directly much as iLBC does via fmtp.
 
>       maxptime: The maximum packetization time desired.  If the maximum
>       is lower than the smallest packetization time determined from the
>       chosen frame size (as described above), then that packtization
>       time SHOULD be used despite the maxptime value.  The default is
>       "no maximum".

Ok, but this really isn't part of the codec.  Doesn't hurt to tell people.
(I'm assuming that the SDP spec for maxptime allows ignoring it (somewhat)
- if it doesn't, then you have to reject that payload.)

>    CELT-specific parameters can be given via the "a=fmtp:" directive.
>    Several parameters can be given in a single a=fmtp line provided that
>    they are separated by a semi-colon.  The following parameters are
>    defined for use in this way:
> 
>       frame-size: The frame size is the duration of each frame in
>       samples.  If more than one frame size is supported, a comma-
>       separated list can be used.  It is possible to use "any" to denote
>       that all even frame sizes are supported.  The default is 480.

You should give an example.  Also make sure there's a BNF. (I haven't read
ahead yet.)

>       mapping: Optional string describing the multi-channel mapping.
> 
>    Because the frame-size is not transmitted in-band, an SDP answer MUST
>    contain only one frame-size, even if multiple frame sizes were
>    offered.

Why?  I thought I knew, but then I thought about it.  Framesize is an fmtp
entry that defines the framesizes you expect to receive.  

You also should define how the set of allowable framesizes should be
determined, and if the answered frame size MUST be one of the offered ones
or not.  If the answer does lock-down the framesize in both directions,
then you need to remind the offerer that they must be ready to receive any
of the framesizes they offered, without any answer, since the answer can
be delayed or lost, and media has already started flowing.

>    The selected frame-size values MUST be even.  They SHOULD be
>    divisible by 8 and have a prime factorization which consists only of
>    2, 3, or 5 factors.  For example, powers-of-two and values such as
>    160, 320, 240, and 480 are recommended.  Implementations MUST support
>    receiving and sending the default value of 480, and if the size 480
>    is supported it MUST be offered.  Implementations SHOULD also support
>    frame sizes of 256 and 512 since these are the ones that lead to the
>    lowest complexity.  When frame sizes that are powers of two are
>    supported, they SHOULD be listed first in the offer and chosen over
>    non powers of two in the answer.

(editorial) Needs hyphenation.

Why 2, 3 or 5?  Need to speak to why I think.
 
>    Care must be taken when setting the value of ptime: and b=AS: so that
>    the RTP packet size does not exceed the path MTU.
> 
>    An example of the media representation in SDP for offering a single
>    channel of CELT at 48000 samples per second might be:
> 
> 
>       m=audio 8088 RTP/AVP 97
> 
>       a=rtpmap:97 CELT/48000
> 
>    Note that the RTP payload type code of 97 is defined in this media
>    definition to be 'mapped' to the CELT codec at a 48kHz sampling
>    frequency using the 'a=rtpmap' line.  Any number from 96 to 127 could
>    have been chosen (the allowed range for dynamic types).  If there is
>    more than one channel being encoded the rtpmap MUST specify the
>    channel count.

You shouldn't re-explain SDP payload value selection.  You should mention
the channel (and the default to 1 channel), and give an example of that.

>    The following example illustrates the case where the offerer cannot
>    receive more than 64 kbit/s.
> 
>       m=audio 8088 RTP/AVP 97
> 
>       b=AS:64
> 
>       a=rtmap:97 CELT/48000
>
>    In this case, if the remote party agrees, it should configure its
>    CELT encoder so that it does not use modes that produce more than 64
>    kbit/s.  Note that the "b=" constraint also applies on all payload
>    types that may be proposed in the media line ("m=").

Yes, which is a major problem.  You cannot do this in this manner; it
should become something like 

       a=fmtp:97 max-bandwidth=64
(or maybe to save SDP size, max-bw=64)

Note I set it up as max-bandwidth/bw, because that's what you said it was.
I can see the possibility that there might be a minimum, or a preferred
that's less than the maximum, so by specifying it now as max-bw we leave
options more easily open.
 
>    The following example demonstrates the use of the a=fmtp: parameters:
> 
> 
>       m=audio 8008 RTP/AVP 97
> 
>       a=ptime: 21
> 
>       a=rtpmap:97 CELT/44100
> 
>       a=fmtp:97 frame-size=512;

There should be no semicolon if there's no following parameter, though most
parsers won't care.

Also again note that ptime is a shared value, and most other codecs use
multiples of 10ms.  That said, it's merely a recommendation/preference, but
if so it should generally not be specific to CELT.  In this case, it
probably should be 20, and CELT would round up.  If this is an important
parameter that may want to be different from the values other payloads
might want to use, you may want to move it to fmtp.

>    This examples illustrate an offerer that wishes to receive a CELT
>    stream at 44100 Hz, by packing two 512-sample frames in each packet.

It's not clear why 2 frames fit into each packet.  You should explain that.

> 5.1.  Multichannel Mapping
> 
>    When more than two channels are used, a mapping parameter MUST be
>    provided.  The mapping parameter is defined as comma separated list
>    of integers which specify the number of channels contained in each
>    CELT stream, OPTIONALLY followed by a '/' and a comma separated list
>    of channel identifiers, then OPTIONALLY another '/' and a string
>    which provides an application specific elaboration on any speaker-
>    feed definitions.  The channels per stream entries MUST be either 1
>    or 2.  The total number of channels is indicated by the sum of the
>    channels per stream entries.  The sum of the channel counts MUST be
>    equal to the total number of channels.
> 
>    Channel identifiers are short alphanumeric strings.  Each identifier
>    MUST begin with a letter indicating the type of channel.  'A' MUST be
>    used to indicate an ambisonic channel, 'S' to indicate a speaker-feed
>    channel, or 'O' indicating other usage.
> 
>    A channel identifier MAY be repeated, but the meaning of such
>    repetition is application specific.  Applications SHOULD attempt to
>    utilize channel identifiers such that mixing all identical
>    identifiers would produce a reasonable result.
> 
>    Non-surround usage such as individual performer tracks, effect send,
>    "order wire", or other administrative channels may be given
>    application specific identifiers which MUST not conflict with the
>    identifiers defined in this draft.  These identifiers SHOULD begin
>    with S if it would be sensible to include them in a mono-downmix, or
>    O if it would be most sensible to exclude them from a mono-downmix.
>    An example usage might be mapping=2,1,2,1,1/
>    SLguitar,SRguitar,OheadsetG,SLkeyboard,SRkeyboard,OheadsetK,SMbass,Oh
>    eadsetB"
> 
>    Ambisonic channels MUST follow the Furse-Malham naming and weighing
>    conventions for up to third order spherical[Ambisonic].  Higher order
>    ambisonic support is application defined but MUST NOT reuse any of
>    WXYZRSTUVKLMNOPQ for higher order components.  For example, second
>    order spherical ambisonics SHOULD use the mapping
>    "mapping=1,1,1,1,1,1,1,1,1/AW,AX,AY,AZ,AR,AS,AT,AU,AV".  Any set of
>    Ambisonic channels MUST contain at least one "AW" channel.
> 
>    Speaker-feed identifiers are named based on the intended speaker
>    locations.  "L", "R" for the left and right speakers, respectively,
>    in conventional stereo or the front left and right in 4, 5, 5.1, or
>    7.1 channel surround.  "LR", "RR" for the left and right rear
>    speakers in 4,5 or 5.1 channel surround.  C" is used for a center
>    channel, "MLFE" for a low frequency extension channel.  "LS", "RS"
>    for the side channels in 7.1 channel surround.  Additional speaker-
>    feeds are application specific but should not reuse the prior
>    identifiers.  For 5.1 surround in non-ambisonic form the mapping
>    SHOULD be "mapping=2,2,1,1/L,R,LR,RR,C,MLFE/ITU-RBS.775-1".  When
>    only one or two channels are used, the mapping parameter MAY be
>    omitted, in which case the default mapping is used.  For one channel,
>    the default is "mapping=1/C", while for two channels, the default is
>    "mapping=2/L,R".

Should there be defaults for anything beyond 2?  Not saying you need to,
just asking.
 
>    For example a stereo configuration might signal:
> 
> 
>       m=audio 8008 RTP/AVP 97
> 
>       a=ptime: 5
> 
>       a=rtpmap:97 CELT/44100/2
> 
>       a=fmtp:97 frame-size=256;
> 
>    Which specifies a single two-channel CELT stream according to the
>    default mapping.

Examples of a couple of the more complex ones are needed.

> 5.2.  Low-Overhead Mode
> 
>    A low-overhead mode is defined to make more efficient use of
>    bandwidth when transmitting CELT frames.  In that mode none of the
>    length values need to be transmitted.  One the a=fmtp: parameter low-
                                            ^^^ remove
>    overhead: is defined and contains a single frame size, followed by a
>    '/', followed by the number of frames (per channel) per packet,
>    followed by a '/', followed by a comma-separated list of the number
>    of bytes per frame for each stream defined in the channel mapping.
>    The frame-size: parameter MUST not be specified and SHOULD be ignored
>    if encountered in an SDP offer or answer.  The ptime:, maxptime: and
>    b=AS: parameters SHOULD also be ignored since the low-overhead:
>    parameter makes them redundant.  When the low-overhead: parameter is
>    specified, the length of each frame MUST NOT be encoded in the
>    payload and the bit-rate MUST NOT be changed during the session.

So the packet layout in the initial payloiad diagram is not really *the*
layout; there's an alternative.

Overall, I'm a little leary of how this is defined.  I understand the wish
to save bandwidth, but the mandating of frames/packet is somewhat unusual
for audio codecs.  Can't you just eliminate that, since you've locked down
the frame size and size for each channel, and so the decoder can know where
the next frame starts.  Also, how does this work with Offer/Answer?  If the
answerer starts sending immediately, and the answer itself is delayed or
lost, then the offerer might start receiving media without being able to
know if it has length bytes.  You probably should make it clear it's a 
*receive*-only parameter - that removes the issue with delayed/lost OK's.
You're saying "I support receiving low-bandwidth data on this payload".
It says nothing about what you're sending.

>    For example a low-overhead surround configuration could be signaled
>    as:
> 
>       m=audio 8008 RTP/AVP 97
>
>       a=ptime: 5
> 
>       a=rtpmap:97 CELT/48000/6
> 
>       a=fmtp:97 low-overhead=256/1/86,86,43,30;mapping=2,2,1,1/
>       L,R,LR,RR,C,MLFE/ITU-RBS.775-1
> 
>    In this example, 4 bytes per packet would be saved.  This corresponds
>    to a 6 kbit/s reduction in the overhead, although the 60 kbit/s
>    overhead of the IP, UDP and RTP headers is still present.
> 
> 6.  Congestion Control
> 
>    CELT allows for bitrate adjustment in one byte per frame increments
>    without any signaling requirement or overhead.  Applications SHOULD
>    utilize congestion control to regulate the transmitted bitrate.  In
>    some applications it may make sense to increase the packetization
>    interval rather than decreasing the codec bitrate.  Congestion
>    control implementations should consider the users differential
>    tolerance for high latency and low quality.

That would be (I assume) really 1 byte per channel per frame, not 1 byte
per frame.
 
> 7.  Security Considerations
> 
>    RTP packets using the payload format defined in this specification
>    are subject to the security considerations discussed in the RTP
>    specification [rfc3550], and in any applicable RTP profile.  The main
>    security considerations for the RTP packet carrying the RTP payload
>    format defined within this memo are confidentiality, integrity and
>    source authenticity.  Confidentiality is achieved by encryption of
>    the RTP payload.  Integrity of the RTP packets through suitable
>    cryptographic integrity protection mechanism.  Cryptographic system
                                                   ^The

>    may also allow the authentication of the source of the payload.  A
>    suitable security mechanism for this RTP payload format should
>    provide confidentiality, integrity protection and at least source
>    authentication capable of determining if an RTP packet is from a
>    member of the RTP session or not.
> 
>    Note that the appropriate mechanism to provide security to RTP and
>    payloads following this memo may vary.  It is dependent on the
>    application, the transport, and the signalling protocol employed.
>    Therefore a single mechanism is not sufficient, although if suitable
>    the usage of SRTP [rfc3711] is recommended.  Other mechanism that may
>    be used are IPsec [rfc4301] and TLS [rfc5246] (RTP over TCP), but
>    also other alternatives may exist.

DTLS?

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
[email-masked]
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 09 Mar 2009 10:34:46 -0400

Hi Randall,

Thanks very much for taking time to carefully read our draft and comment 
it. Some more comments/answers below.
> whatever bitrate information is needed to decode it.)
Yes, the bit-rate information is actually the compressed frame length 
itself.
> suggested in order to pad (though padding appears not be needed).
What we mean is that 1) there's no padding required and 2) if someone 
ever felt like padding, it's best to just increase the bitrate such as 
to actually use the extra bytes. I guess we'll need to make that a bit 
clearer.

> use a re-write.  No technical problem, though.
Will fix.

> Above it said 48000 is a MUST, so it should say that here.  
Noted. We missed that when changing some SHOULDs for MUSTs. We hesitated 
a lot in deciding what's a SHOULD and what's a MUST. Basically, CELT's 
advantage is that it can operate with almost any sampling rate, frame 
size or bit-rate. The disadvantage is that unless we specify "baseline 
requirements", we might end up with several implementations that are 
unable to inter-operate. Also, feel free to suggest a better baseline if 
you think we didn't select the right one.
> line for each payload.  So, bandwidth moves to fmtp.
I was under the impression that b=CT was applied to all payloads while 
b=AS: was just for one media:

"AS gives a bandwidth figure for a single media at a single site, although there may be many sites sending simultaneously."
Isn't that what we want? 

> want to consider negotiating it directly much as iLBC does via fmtp.
Well, I didn't see that to be a problem considering that one would 
probably want the same packetization time for any codec. Do you see a 
case where you wouldn't want that? I'm not not quite sure how people use 
the ptime in practice and how much it is followed.

> - if it doesn't, then you have to reject that payload.)
As far as I understand, maxptime is a SHOULD in the rfc, so I thought 
we'd just mention its interpretation wrt CELT.
> You should give an example.  Also make sure there's a BNF. (I haven't read
OK, so maybe we should add a BNF. As for the examples, I think they're 
adequate, but let me know if that's not the case.
> be delayed or lost, and media has already started flowing.
The fundamental issue here is that one needs to know the frame size to 
be able to initialise the decoder. Just like a codec like iLBC had two 
modes (for 20 ms and 30 ms frame), CELT has a *very large* number of 
modes: one for each combination of frame size and sampling rate. So the 
idea was that one side offers a list of frame sizes and the other side 
responds with the one it likes best and both sides use that. There is no 
way to decode media without knowing the frame size. Any idea what's the 
best way to handle that?

We're also considering having a "configuration" packet to be sent at the 
beginning of any stream and that includes even more mode-specific data 
to increase flexibility. However, we haven't found a good way of doing 
that yet (wrt loss of the configuration packet). Any thought on that?

> Why 2, 3 or 5?  Need to speak to why I think.
OK, that's mainly because of the CELT uses an FFT, which is faster for 
small radices. We'll explain that better. In the end, I don't think many 
people will really want frame sizes like 238 samples.

> the channel (and the default to 1 channel), and give an example of that.
Noted. Will fix that.

> options more easily open.
We had something like that originally, we we thought it was better to 
use the existing b= parameter. Why is that a major problem?

> There should be no semicolon if there's no following parameter, though most
Will fix.
> might want to use, you may want to move it to fmtp.
I don't see a reason to make it different from other codecs considering 
that CELT can handle about any ptime. I wouldn't mind doing it though if 
there's a use case for it.
> It's not clear why 2 frames fit into each packet.  You should explain that.
Actually, I think I made a mistake in that one, though the general idea 
was that 2 frames of 512 samples at 48 kHz (not 44.1) gives a 21.33 ms 
packet.

> Should there be defaults for anything beyond 2?  Not saying you need to,
I'm open to suggestions, but I didn't think there was anything that 
could be considered as a default for >2.
> Examples of a couple of the more complex ones are needed.
OK.

> layout; there's an alternative.
Actually, we're not yet sure the reduced overhead is worth the trouble 
of adding another layout. Any thought?
> the next frame starts.  
You're right. By specifying the frame size and the bit-rate, we don't 
need to specify the number of frames per packet.
> Also, how does this work with Offer/Answer?  If the
Well, we have the same fundamental issue everywhere. If we don't know 
what frame size was selected, we cannot decode anything. So we always 
need the answer. The only way I can see to go around that would be to 
use a different payload type for every parameter combination, but I 
think that would be ugly.
> That would be (I assume) really 1 byte per channel per frame, not 1 byte
Well, technically you can increase the bit-rate by one byte for only one 
channel...

Cheers,

    Jean-Marc

---

From: Gregory Maxwell <[email-masked]>
Date: Mon, 9 Mar 2009 10:26:44 -0700

Jean-Marc Valin wrote:
> Randell Jesup wrote:
[snip]

> Yes, the bit-rate information is actually the compressed frame length

To clarify: CELT is designed assuming a length-preserving transport.
If a CELT frame of N bytes is received, the receiver assumes the
encoder encoded that frame at a rate of sample_rate/frame_samples*N
bytes per second.

For naturally length preserving transports (such as raw UDP, RTP, or
most other transports) no signalling of bitrate is required at all,
internally to the codec or externally, beyond the length preservation.

However, since we needed to support multiple frames per packet for RTP
(for multichannel audio, and to allow flexible trade-off of latency vs
overhead), we needed to provide an in-channel length encoding that the
decoder can use differentiate the bitrates used by the component frames
of the packet.

This is accomplished by explicitly enumerating the lengths of the packed
frames at the start of the packet payload.  The transport still must be
length preserving since the total packet length must be known to determine
when to stop reading lengths.  This is perfectly robust against packet
loss, but should be protected against bit-error.  I suppose it might
be worthwhile to make a recommendation regarding CRC coverage (i.e.
for UDP light).

We also considered an alternative of encoding the number of frames
per packet in a byte, followed by N-1 lengths, but decided that simply
encoding N lengths would likely be slightly easier to implement and would
likely have somewhat better bit error robustness properties in practice.

(Of if someone has reason to believe that another scheme is more likely
to be correctly implemented…)

This length encoding process must be made sufficiently clear in the
draft. If it isn't currently then that needs to be resolved.

> Well, technically you can increase the bit-rate by one byte for only one 

Perhaps the clarity of the draft would be improved with some standalone
definitions:

* Packet  — A unit of network transmission, containing IP/UDP/RTP
headers and one or more CELT frames.

* Frame — A unit of codec operation, representing frame_samples samples
for one or two channels of audio.

* Channel — A stream of audio, transported in groups of one or two in a
single frame by CELT

The codec can adjust its instantaneous bitrate with one byte per frame
granularity. This means that for congestion control purposes one byte
per packet granularity is available regardless of the number of channels
encoded or frames per packet, so long as the encoder is willing to use
slightly unequal bitrates per frame.

The logic the sender uses in distributing the available capacity to frames
(and channels) isn't relevant for interoperability, so I don't see cause
to make a recommendation. However, if it would make the draft more clear
it would be possible to do so.

>> DTLS?

Quite reasonable.  Most of that section is from the boiler-plate RTP
profile security considerations in draft-ietf-avt-rtp-howto-06.txt. We
can add that to draft-valin-celt-rtp-profile-00.txt but the changes should
also be made to the rtp-howto unless there is a reason not to do so.

Our enhancements to that section consist of recommendations regarding
timing and length-information-leakage attacks, the latter of which is
somewhat CELT specific since CELT can fill any target byte-size.

---

From: Randell Jesup <[email-masked]>
Date: Mon, 09 Mar 2009 15:45:05 -0400

>it. Some more comments/answers below.

No problem.  Easier to fix problems up front.

>Noted. We missed that when changing some SHOULDs for MUSTs. We hesitated a
>lot in deciding what's a SHOULD and what's a MUST. Basically, CELT's
>advantage is that it can operate with almost any sampling rate, frame size
>or bit-rate. The disadvantage is that unless we specify "baseline
>requirements", we might end up with several implementations that are unable
>to inter-operate. Also, feel free to suggest a better baseline if you think
>we didn't select the right one.

Is this intended to be used as a "speech" codec at all; as an alternative
to iLBC/G.729/G.722.x/etc?  If so, then support for 8KHz and/or 16KHz may
be important to mandate for interoperability reasons.  You DON'T want to be
mixing sample rates if multiple codecs are accepted.  (In theory it can be
done, but in practice it would be risky, especially in the face of packet
loss.)  For example, if you have this:

Random example (I probably have the G722 media type wrong):
m=audio 4321 RTP/AVP 0 97 98
a=rtpmap:0 PCMU/8000
a=rtpmap:97 G722/16000
a=rtpmap:98 CELT/48000

To quote from an earlier AVT email I wrote on this subject on 3 Dec 2007:

    Subject: Re: [AVT] I-D ACTION:draft-ietf-avt-rtcpxr-audio-01.txt
    [big SNIP]
    This means the timestamp rate can change at any point, on a
    packet-by-packet basis.  It's even theoretically allowable to alternate
    G711 and G722 packets.  Totally odd and non-useful, but it illustrates
    the point.  More realistic is a change from one to the other half-way
    through an RTCP monitoring period.

There were earlier discussions about this; no work has been done on finding
ways to deal with the issues raised by that sort of SDP, especially in the
face of packet loss.  Such a shift in timestamp rate can mess up
NTP<->timestamp conversion, and mess up things like RFC 2833 DTMF packets
that might overlap a change (or be near one).

>"AS gives a bandwidth figure for a single media at a single site, although
>there may be many sites sending simultaneously." 
>
>Isn't that what we want?

The wording there is very lawyer-ese, and really addressing how it's
affected by a multicast or conference setting. 

If the b=AS is at the m= level of the SDP (not above all the m='s), then it
only applies to that one media stream.  However, exactly what b=AS *means*
is very fuzzy.  b=AS is not a codec parameter; it's a stream parameter, and
it's also a reception parameter, not a "I plan to send" parameter.
There's been a lot of discussion about b=TIAS (RFC 3890) as a better way to
specify bandwidth.  Note that b=AS INCLUDES RTP/UDP/IP overhead, and thus
implicitly is dependent on packet rate.

More to the point, b=AS is just one way to specify bandwidth.  

Another huge blocker for using b=AS (or b=anything) in this way: what if
another codec also offered in this stream wanted to re-use b=AS as well?
And what if the preferred bitrates (or max-bitrate) for each codec was
different?

How much do you *need* to specify bandwidth here?  Realize that most
devices don't have a good idea what receive bandwidth is even theoretically
available, let alone practically.  Most configuration is done at the sender
end, or by explicit choice of codec and bitrate.

I suggest reviewing how other multiple-bitrate codecs like G.722.x handle
this (AMR-WB, etc).

Also, isn't CELT true variable-bitrate?  If so, the bitrate to use
(initially?) might be very different than the "maximum" bitrate.

>Well, I didn't see that to be a problem considering that one would probably
>want the same packetization time for any codec. Do you see a case where you
>wouldn't want that? I'm not not quite sure how people use the ptime in
>practice and how much it is followed.

Sure.  You might prefer 10ms, which G.711 and some others support, but iLBC
only supports 20 and 30 ms frames, and thus only multiples of those for the
actual packetization time. You can still specify a ptime of 10 when using
iLBC. 

ptime is merely a "I would prefer to receive" parameter.  Do not rely on it
for anything.  The actual packetization time does NOT have to be the same
in each direction, or even the same from one RTP packet to the next.
(packet 1 could have 1 frame and packet 2 could have 10).  

iLBC is really negotiating the framesize, not the packetization time.

>As far as I understand, maxptime is a SHOULD in the rfc, so I thought we'd
>just mention its interpretation wrt CELT.

Ok, then say something like "per [RFC 4566], if the maximum is lower..."

>adequate, but let me know if that's not the case.

BNF may not be *needed*, but if there's anything at all complex it's handy to
avoiding mistakes.

>The fundamental issue here is that one needs to know the frame size to be
>able to initialise the decoder. Just like a codec like iLBC had two modes
>(for 20 ms and 30 ms frame), CELT has a *very large* number of modes: one
>for each combination of frame size and sampling rate. So the idea was that
>one side offers a list of frame sizes and the other side responds with the
>one it likes best and both sides use that. There is no way to decode media
>without knowing the frame size. Any idea what's the best way to handle
>that?

a) use different payloads.  Clear, easy, wastes space in SDP
b) include framesize in the bitstream in *every* packet.  Clear, easy,
   no-fuss, wastes some bandwidth all the time.  And isn't this required
   anyways if there's more than one channel?
c) ignore media until you get an answer with a clear selection.  Clear,
   easy, could be a major loss of media on a delayed answer.
d) require media sent (at least until acknowledgment of an answer) be
   clearly distinguishable - i.e. do not vary framesize from the offered
   values until an ACK, and do not allow packet sizes that are common
   multiples of offered framesizes.
   For example, if you offer (say) 20 and 30, do not send a packet 
   containing 60, 120, etc.  You can send 20, 30, 40, 80, 90, 100, since
   they can't be mis-understood.
   Complex, doesn't waste bits, artificial constraints, no adaptation to
   congestion/etc until ACK.

>beginning of any stream and that includes even more mode-specific data to
>increase flexibility. However, we haven't found a good way of doing that
>yet (wrt loss of the configuration packet). Any thought on that?

Yes: you can't assume 0-loss.  Also, if possible, the bitstream should be
decodable without the out-of-band channel information.  Video people have
struggled with this with sprop-parameter-sets in H.264 (RFC 3984).
Downsize will be bandwidth used.  You can amortize the overhead by sending
the config packets only periodically, but you'll probably need to send
them reasonably often to allow mid-stream join (think conferences).

>there's a use case for it.

Then specify a value, and let CELT round up (or down) as needed.
It might be nice to show how this might interact with alternative payloads
that might want multiples of 10.

>Actually, we're not yet sure the reduced overhead is worth the trouble of
>adding another layout. Any thought?

Probably not.  How big a saving is it?  This mandates (effectively) CBR.
Too bad there's no way in-stream to know when you've finished decoding
the channel (stop token or the like).  Then you don't need channel
sizes at all (though perhaps you might want them).

>Well, we have the same fundamental issue everywhere. If we don't know what
>frame size was selected, we cannot decode anything. So we always need the
>answer. The only way I can see to go around that would be to use a
>different payload type for every parameter combination, but I think that
>would be ugly.

You really *need* to handle the media-before-answer case somehow, even if
it's separate payloads.

>Well, technically you can increase the bit-rate by one byte for only one

I meant that the change rate is one byte per channel per frame (max).  This
implies the frame-size rate of change could be anywhere in the range of
plus/minus the number of channels (in bytes).

I assume the decoder would handle large frame gaps with corresponding large
changes in framesize?

---

From: Jean-Marc Valin <[email-masked]>
Date: Mon, 09 Mar 2009 17:41:50 -0400

Randell Jesup wrote:
> be important to mandate for interoperability reasons.  
CELT is (so far at least) not intended for lower sampling rates like 8 
kHz or 16 kHz and doesn't operate in the same space as the codecs above 
or Speex. It's closer to codecs such as AAC-LD, G.722.1C, G.719, and 
ULD, though only ULD has a delay as short as CELT's.
> You DON'T want to be
So you mean that you'd need to maintain a coherent timestamp despite 
changing between codecs that have different sampling rates? I wasn't 
aware of that, so I guess it's something that needs to be addressed. 
Maybe just by saying that different sampling rates SHOULD NOT be used 
with the same m=

> implicitly is dependent on packet rate.
I've seen many G.711 implementations using b=AS:64, so I thought we 
could use it in a way that excludes the overhead.
> (initially?) might be very different than the "maximum" bitrate.
CELT can change bit-rate at any time, but so far it only changes based 
on what the senter wants to use, i.e. to adapt to congestion. Even if 
used with b=AS:, I was thinking that it would be more like a max 
bit-rate anyway.

> iLBC is really negotiating the framesize, not the packetization time.
Well, I was thinking of using the ptime just for a preference. You 
specify the frame size and ptime helps decide how many frames get sent 
per packet.
> Ok, then say something like "per [RFC 4566], if the maximum is lower..."
OK, will do that.
> BNF may not be *needed*, but if there's anything at all complex it's handy to
Noted.
>    congestion/etc until ACK.
OK, I'll need to give this a bit more thought.

> them reasonably often to allow mid-stream join (think conferences).
Yes, I'm aware of the loss problem and I'm not sure how to handle that. 
I'll have a look at the RFC you mention.
> that might want multiples of 10.
I guess the rounding would just be a bit more. WIll add an example.
> sizes at all (though perhaps you might want them).
Well, the stop token would waste about the same space as the size value...

> You really *need* to handle the media-before-answer case somehow, even if
Hadn't realised it was an issue. Need to give more thought into this. 
Other suggestions welcome.
> changes in framesize
Basically, any channel of any frame can have a different size and change 
arbitrarily from one frame to the next. You can have one channel jumping 
from 64 to 128 kbps while the other goes from 96 to 48 kbps. That's why 
there's one byte per channel per frame used for the size.

Cheers,

    Jean-Marc

---

From: Randell Jesup <[email-masked]>
Date: Mon, 09 Mar 2009 23:43:56 -0400

Jean-Marc Valin <[email-masked]> writes:

>CELT is (so far at least) not intended for lower sampling rates like 8 kHz
>or 16 kHz and doesn't operate in the same space as the codecs above or
>Speex. It's closer to codecs such as AAC-LD, G.722.1C, G.719, and ULD,
>though only ULD has a delay as short as CELT's.

Ok, though the issue still remains.  And since the offerer might not know who
it's calling, it may need to offer both low-bandwidth/sampling codecs and
higher-quality/bandwidth codecs.

Even among higher quality codecs like AAC-LD (which does go down to 16K I
think), you still have the same issue - offering mixes of sampling rates is
potentially a boatload of fun - and with these codecs, it's almost
impossible to avoid unless the offerer insists on a sampling rate, and then
you have problems if the answerer wants something different.

I'm not expecting you to solve the general issue all codecs have here, but
I do want to sensitize you to it, since your codec is much more flexible in
sampling rate and bandwidth than most others.  You may want to take it into
consideration in choosing required rates - choosing rates that other codecs
it might be combined with would use (like 48000, or 44100).  The downside
is that if the HW codec doesn't match the negotiated codec you have to do a
resampling after decode and before encode.

>changing between codecs that have different sampling rates? I wasn't aware
>of that, so I guess it's something that needs to be addressed. Maybe just
>by saying that different sampling rates SHOULD NOT be used with the same m=

As written, when you switch payloads to one with a different sample
rate/timestamp rate, the timestamp rate would change at that point as well.
This causes considerable confusion in other related specs, since most
writers don't consider this case.  And it really causes confusion when a
packet loss occurs at the point of timestamp change (and to a slightly
lesser extent, when timestamp changes occur between RTCP SR/RR's, or during
the playout of an RFC 2833 packet).

I would NOT try to outlaw this within a codec; it's an issue for general
AVT resolution.

>I've seen many G.711 implementations using b=AS:64, so I thought we could
>use it in a way that excludes the overhead.

Those implementations are wrong... :-)  Yet another reason not to rely on 
it.  Also, why specify the bandwidth for a fixed-bandwidth codec? 

>CELT can change bit-rate at any time, but so far it only changes based on
>what the senter wants to use, i.e. to adapt to congestion. Even if used
>with b=AS:, I was thinking that it would be more like a max bit-rate anyway.

If you're using bandwidth parameters as specified in RFC 4566, then you
probably don't need to address them here at all (and then you don't have
to worry about newer ways to specify bandwidth, like b=TIAS - it's the
problem of a higher level).

>Well, I was thinking of using the ptime just for a preference. You specify
>the frame size and ptime helps decide how many frames get sent per packet.

Sure.  That's the intention of ptime.  It's a preference.  Sounds like
you're using it as intended, so you may not need to spend much time
discussing it, except to mention the (unusual) flexibility of CELT.

>OK, I'll need to give this a bit more thought.

My suggestion is (a), with a second choice of (b).  Or delete it entirely.

>Yes, I'm aware of the loss problem and I'm not sure how to handle
>that. I'll have a look at the RFC you mention.

It may not be obvious.  You'll want to look at the discussions in the last
year leading up to RFC 3489bis, now at WGLC or so.  You won't have the
complications that led to sprop-level-parameter-sets, but you have the
choice to put the data out-of-band (like sprop-parameter-sets), or in-band
(H.264 Picture and Sequence Parameter Set NAL units).

>Well, the stop token would waste about the same space as the size value...

Well, if the frame size is fixed, then (depending on how things are
encoded) you may be able to have an implicit length (stop when you generate
enough bytes) - but the codec might be more frequency-domain and this idea
wouldn't fly.

Also, I assumed it might be a variable bitstring encoding, and so a "STOP"
symbol might only cost a few bits.

>Hadn't realised it was an issue. Need to give more thought into this. Other
>suggestions welcome.

Anyone else?  Basically, you need to be able to reasonably decode when you
have the offer and received data, but no answer.  This might (or might not)
be related to how you'd handle people joining a conference or multicast.
See also the RTP topologies RFC (forget the number).

>there's one byte per channel per frame used for the size.

So what then was the "one-byte change in bitrate per frame" then?

I was referring to what happens if 10 packets in a row are lost - can the
receiver decode a packet that might be 10 bytes smaller or larger?  Does it
need as input the number of packets lost (if any) since the last decoded
packet?

---

From: Gregory Maxwell <[email-masked]>
Date: Mon, 9 Mar 2009 22:07:41 -0700

Randell Jesup writes:
[snip]
> need as input the number of packets lost (if any) since the last decoded

I'll let JM respond to the rest of your ongoing conversation but I
wanted to take a crack at clarifying this as I probably caused the
misunderstanding on this point.

The text "CELT allows for bitrate adjustment in one byte per frame
increments without any signalling requirement or overhead." is referring
to the *resolution* of the instantaneous bitrate, not the rate of
change. 

Stated differently: At the moment a frame of audio is compressed you can
ask the encoder to produce anywhere from zero to hundreds of bytes (well,
the current encoder may fail to produce useful output for lengths <~5
bytes, but that should change). This is in contract to most communication
codecs which can only encode to a fairly limited number of sizes. The next
frame may be encoded with an entirely different number of bytes without
regard to what any of the prior frames were encoded with or what future
frames may be encoded with.

Whatever space is available will be well used which is why we recommend
that you never pad CELT: If you want a larger size you can ask the codec
for it and the extra space will be used to improve quality.

A million packets could be lost or otherwise mutilated and the decoder
will happily decode whatever comes in when the stream resumes. The codec
may take 40 milliseconds or so to reach full quality after a loss event
but it is guaranteed to recover quickly and it doesn't matter what the
bitrate did during the loss, nor does the decoder need to know how
much was lost or damaged.

So, for example, the encoder could be encoding at 64kbit/sec and receive
notice from the rate control part of an application (or operating system)
and on the next frame be producing 24kbit/sec with no coordination with
the decoder.  Alternatively, it could change the bitrate as little as
((sample_rate / frame_samples) * 1 byte * 8bits/byte),  1.38kbit/sec for
the 44100 sample_rate and 256 frame_samples case, or by any other amount
which results in an whole number of bytes per frame of encoded audio.

If a system were to provide a loss-free method of congestion detection,
such as ECN or some kind of queuing delay estimation, the codec could
gracefully and gradually rate adapt to provide the best supportable
quality without any glitches at all. (If congestion is detected via
loss then there will be a quality hit from lost frames, of course)

The codec design goal here was to efficiently provide the maximum amount of
rate-adaptability possible for a byte oriented transport. If not for the
need to place multiple CELT frames in a packet, there would be no need
to have any overhead for length coding at all as the RTP transport will
preserve payload length.

If the text were changed to say "CELT can encode to any whole number of
bytes per frame at any time without signalling a change to the decoder"
would this avoid the potential for misunderstanding?

-- 
Greg Maxwell

---

From: Randell Jesup <[email-masked]>
Date: Tue, 10 Mar 2009 16:07:42 -0400

>to the *resolution* of the instantaneous bitrate, not the rate of

Aha.  It's "one byte increments" then, not an "adjustment [of] one byte per
frame".  Very different.  Thanks.

>would this avoid the potential for misunderstanding?

Yes.