Subject: [MMUSIC] maxptime

From: lazzaro <[email-masked]>
Date: Tue, 19 Oct 2004 17:54:21 -0700

Magnus writes:
> In the work on updating RFC 3267 (AMR and AMR-WB RTP payload format) 
> which originally defines "a=maxptime" I have recieved a request for 
> clarification that I think make sense and which do not create any 
> backwards compatibility issues:
>
> In Section 6: under attribue "a=maxptime:<maximum packet time>"
>
> I would like to rephrase the sentence:
> "The time SHOULD be a multiple of the frame size."
>
> To:
> "The time SHOULD be an integer multiple of the frame size.

If maxptime is now considered a parameter for use in
all codecs (like the "ptime" it was derived from), shouldn't
its definition admit that not all codecs are frame based?

The RTP MIDI I-Ds use ptime and maxptime, and RTP MIDI
is not frame based ... indeed, a ptime and maxptime of 0 are
perfectly valid and useful for RTP MIDI.  Zero signals a
one-command-per-packet sending style. A MIDI command
has no temporal extent -- a packet needs two commands
to have non-zero time, as a rule (that has exceptions ...).

---
John Lazzaro
http://www.cs.berkeley.edu/~lazzaro
lazzaro [at] cs [dot] berkeley [dot] edu
---

_______________________________________________
mmusic mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/mmusic

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 20 Oct 2004 08:51:32 +0200

Hi John,

The attribute is not intended to exclude non-frame based codecs. However 
it does in some degree assume that audio content has some type of duration:

       a=maxptime:<maximum packet time>

          The maximum amount of media which can be encapsulated in each
          packet, expressed as time in milliseconds.  The time SHALL be
          calculated as the sum of the time the media present in the
          packet represents.  The time SHOULD be a multiple of the frame
          size.  This attribute is probably only meaningful for audio
          data, but may be used with other media types if it makes
          sense.  It is a media attribute, and is not dependent on
          charset.  Note that this attribute was introduced after RFC
          2327, and non updated implementations will ignore this
          attribute.

We could clarify this so it want be misunderstood. I also received a 
comment from Dave Singer that "frame size" is not really correct. Thus I 
think one can modify the sentence to be more clear without affecting the 
intended semantics:

For any frame-based media, the maximum packet time SHOULD be an integer 
multiple of the frame duration.

Is that better?

Cheers

Magnus

> lazzaro [at] cs [dot] berkeley [dot] edu

Magnus Westerlund

Multimedia Technologies, Ericsson Research EAB/TVA/A
----------------------------------------------------------------------
Ericsson AB                | Phone +46 8 4048287
Torshamsgatan 23           | Fax   +46 8 7575550
S-164 80 Stockholm, Sweden | mailto: [email-masked]

---

From: lazzaro <[email-masked]>
Date: Wed, 20 Oct 2004 10:42:49 -0700

On Oct 19, 2004, at 11:51 PM, Magnus Westerlund wrote:
> The attribute is not intended to exclude non-frame based codecs.
> However it does in some degree assume that audio content has some type 
> of duration:

The RTP MIDI payload's duration is always has a defined value.
It just so happens that 0 ms is an acceptable value.

So, as long as 0 ms is an acceptable maxptime duration its OK
by me. 0 ms needn't be explicitly permitted in the text, but it should 
not
be explicitly forbidden.

> For any frame-based media, the maximum packet time SHOULD be an 
> integer multiple of the frame duration.
>
> Is that better?

Yes, sounds good to me.

---
John Lazzaro

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 21 Oct 2004 09:45:35 +0200

Hi John,

> by me. 0 ms needn't be explicitly permitted in the text, but it should

I think the usage of maxptime=0 may be a bit problematic. A 
interpretation is that no audio can be included in the packet, as no 
audio duration is allowed.

I see that you have made another interpretation in the MIDI draft 
(draft-ietf-avt-rtp-midi-format-06):

"0 ms is a reasonable media time value for MIDI packets.  In a packet
with a 0 ms media time, all commands execute at the instant coded by the
packet timestamp.  Prohibitions in [13] against 0 ms ptime values are
not relevant for MIDI streams, and may be ignored."

An RTP timestamp tick is strictly not an instant, it has a duration 
equal to 1/(RTP timestamp rate). However these is my interpretation. 
maxptime was never intended for this usage, where you would have 
interest to limit the duration to smaller intervals than a 1 ms. We 
where mostly thinking of the smallest units as something like 5 ms or 
so. Is really your intention of using maxptime=0 that for each timestamp 
tick there is a command(s), a new packet should be sent? I can 
understand using ptime=0 to recommend that you do this behaviour but 
using maxptime to enforce the behaviour?

Another question is how well parsers out there will handle 0. Hopefully 
well, but I am not certain. Especially any that is primarily to handle 
Offer/Answer, where a normative MUST be larger than 0 is present. I 
think you should consider the impact on interoperability.

So lets try to sort this out by answering some questions:

- How big is the need to be able to specify ptime=0 and maxptime=0, and 
what is the effect is the smallest one can use is ptime=1?

- Do any implementers of generic SDP parsers have any problems with 
ptime or maxptime equals 0?

- Why does RFC 3264 have a normative MUST against using ptime=0?

- Any further considerations?

If the answer to the above questions and any other input indicates that 
this is reasonable to do this, then I think a clarification is in order 
to the parameters about the interpretation of 0.

Cheers

Magnus Westerlund

---

From: lazzaro <[email-masked]>
Date: Thu, 21 Oct 2004 15:15:38 -0700

> Hi John,

Hi Magnus!

	There are a few separable issues here, let's take
them one by one ... CC'd to AVT since these issues cross
boundaries ...

> However these is my interpretation.

Here's the underlying motivation for the choice I made.

Imagine playing the piano.

Play three notes simultaneously. In MIDI, this corresponds
to three separate NoteOn commands.  In RTP MIDI, each
has a timestamp, expressed in RTP timestamp units.

It's really important to the sonic integrity of the piece to code that
the notes happen simultaneously -- even one RTP tick matters.
The reason is, phase effects as the waveforms beat against each
other change the timbre.  Hand drummers on the list (conga, etc)
can attest to this -- the art of creating certain timbres on those
types of percussion instruments is based on fine timing of strokes
on the drum with the two hands.  Standard MIDI Files do in fact code
true simultaneity, which is why they can be used in studio work.
Musicians go crazy shifting around MIDI notes by tiny amounts inside
the DAW to get this sort of stuff right, and SMFs preserve that hard 
work
on export.

Pull back from the nanosecond level to the few-millisecond
level, and we move from timbre distortion to timing sloppiness.
MIDI as transmitted on a MIDI 1.0 DIN cable suffers from this:
at 300 microseconds/command, playing full chords with both
hands creates timing slews you can hear.  Part of the motivation
to move off of MIDI DIN cable technology is to make instrument
controllers that don't suffer this timing problem.

You might ask -- why can't just put all simultaneous notes
in the same packet, and use something other than RTP timestamps
to denote intra-packet timing?

To understand the answer, you need to understand that a core
use of RTP MIDI is on "media LANs" -- a stage or studio situation
that may consist of two nodes -- piano controller and laptop --
connected by an Ethernet cable as the Layer 2 for RTP MIDI.
These folks want absolute lowest latency -- so, they don't want
their piano controller to wait around after the first MIDI command
to see if a second command that is "simultaneous" is happening soon.
Instead, they want the piano to take each command, put it in its own RTP
packet, and send it off.  So, they want the ability to send two RTP 
packets
in a row, each coding one MIDI command, each with the same
RTP timestamp.  To do this, they need the RTP timestamp semantics
that say an RTP MIDI packet can have 0 ms media time.

Some instrument makers use Ethernet today.  They use their own UDP
solutions (some use OSC on top of UDP) and have their own code.  They
want to add support for RTP MIDI, but RTP MIDI needs to work with their
model, which is as I describe it above.

I would also not be surprised if the recovery journal semantics
failed if we moved to your RTP timestamp interpretation, although
I'd have to think about to say for certain.  Other things might break
too.  This is a 4-year-old project now, and the timestamp model is
a fundamental assumption that underlies RTP MIDI in subtle ways.

So, that's the reply to issue #1.

Issue #2:

> Is really your intention of using maxptime=0 that for each timestamp 

Yes. Once again, we're talking about apps in the "media network"
world, a laptop and piano keyboard with a point-to-point 2-node
Ethernet as the Layer 2 for RTP MIDI.  The maxptime=0 parameter
says: all packets must either code no MIDI commands, or code one
or more MIDI commands with the same timestamp -- and thus, the
media time for all packets must be 0 ms.

This accommodates receivers (such as, currently, sfront!) that do not
implement a receiver buffer of any sort.  Instead, sfront grabs a 
packet,
and executes all MIDI commands in the packet simultaneously.
It relies on the constant nominal latency of the point-to-point Layer 2,
and the constant nominal latency of the keyboard sender, for timing
accuracy -- not RTP timestamps, which are ignored.

Thus, these receivers can't handle a packet that has a non-zero media
time -- all they know is "now".

If one is designing receivers exclusively for use on media LANs
as live performance instruments (in other words -- a musical
instrument synthesizers intended to only be played locally), this
"no buffer" model makes perfect sense -- it's how soft synths that
are driven by MIDI 1.0 cables work today.  Eventually, it would be
better if these synths incorporated small configurable buffers to
compensate for sub-ms jitter, but the conceptual hurdle to do so
would be too great for many synth authors to do.  In practice,
OS APIs will probably implement those buffers if/when they choose
to support RTP MIDI as a media LAN MIDI source.

So, we need a way for those soft synths to negotiation (via 
Offer/Answer)
to say "I can only handle 0 ms packets, do not send me any other
kind".

As an aside, RTP MIDI works fairly well Berkeley/Stanford/Berkeley using
the no-buffer sfront  -- it doesn't get down to the "hand percussion 
jitter"
I mention above, but with a few coding heuristics it produces a playable
instrument when I do loop-back (keyboard in Berkeley, RTP MIDI to
Stanford and back, which sfront then turns into audio).  Chris Chafe
sees jitter under the audio sample rate on his DAT-quality long-distance
streaming sessions on occassion!

Issue #3

> Especially any that is primarily to handle Offer/Answer, where a 
> normative
> MUST be larger than 0 is present. I think you should consider the 
> impact on interoperability.

Well, the easiest thing to do is invent my own SDP parameter as an
RTP MIDI replacement for maxptime -- this skirts the compatibility
issue.  But as I explained above, I think we have a legitimate need
for the negotiating functionality.  And I think the most elegant thing
to do is for maxptime to bend its definition to include 0 ms, just for
RTP MIDI, which is why I did it that way ... a compromise would be
to put interoperability warning language into the RTP MIDI I-D, but
keep the normative text as is.

> - How big is the need to be able to specify ptime=0 and maxptime=0, 
> and what is the effect is the smallest one can use is ptime=1?

Hopefully this is answered above.

> - Any further considerations?

These questions seem aimed at others on the list ... however, I
should note that I didn't take the route I did on maxptime
lightly -- after several years of consensus building with the music
technology community, the approach to RTP timing I outline above
is the best way I can see to solve their problems.

---
John Lazzaro

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 22 Oct 2004 14:55:38 +0200

Hi John,

Please see below. I also have to point out that I haven't read your 
drafts during the last year or so. I have only looked at certain 
specific things in them.

>  > However these is my interpretation.

I probably should clarify myself as I have received comments that it 
seems like I am reinterpreting the timestamp model. The only point I was 
trying to make is that in a digital domain, compared to the analog an 
instant is not strictly a precise moment without any duration. Due to 
the analog to digital conversion the smallest point in time we can 
express is a single sample (RTP TS tick). That sample represent some 
duration, equal to 1/(sampling frequency), thus it is not truly 
instantaneous. To connect to John's example below, about the 
keyboardist. Even if he strikes the keys simultaneously on a timescale 
of 1/10000 part of second it is most likely not true when you go to a 
timescale of 1/10^9 of a second.

I will return to this later down.

> controllers that don't suffer this timing problem.

I think I understand this, however the RTP timestamp rate as I 
understand it for MIDI would be equal to the sample rate for the audio 
produced? That would mean that it is reasonable to expect that we are 
working with a sample frequency of for example 48kHz. Or do you needed 
even higher resolution?

> to denote intra-packet timing?

No, I am not questioning this timing model at all. It seems appropriate 
for RTP based on your description. However one needs to be aware what it 
means. A MIDI command can't be correctly synchronized with a finer scale 
then one RTP TS tick. Thus the precision is depending on the timestamp 
rate.

> that say an RTP MIDI packet can have 0 ms media time.

First, I can understand that due to the desire to keep the latency to a 
minimal even a short buffering is not desirable, however RTP payloads 
are normal gathered using some type of buffering, either at the encoding 
step, due to the need to gather sufficient amount of samples to perform 
the encoding, or at the packetization step to aggregate together 
sufficient amount of data to get the right efficiency in the 
transmission. Thus MIDI commands are a bit special.

Secondly I don't think we are talking media time really here. The 
command does not truly have a duration as you say. They takes affect the 
instant (on a specific sample) indicated by the RTP timestamp. The 
problem is really about indicating packetization strategies.

> a fundamental assumption that underlies RTP MIDI in subtle ways.

I am not redefining anything. I think we simply have misunderstood each 
other.

> accuracy -- not RTP timestamps, which are ignored.

Then it is not really RTP you are using. You are ignoring the 
functionality the timestamp provides to place the different packets 
correctly on a timescale to combat any transport jitter. As it seems the 
local stage applications are relying on a consistent transport behavior.

> time -- all they know is "now".

Then they are not RTP aware, a RTP receiver needs to have knowledge 
about how time flows.

> to say "I can only handle 0 ms packets, do not send me any other

The problem is really to indicate the desired behavior. Which is not 
really what ptime and maxptime is about. They give the recommend and 
maximal amount of media duration for a packet.

> keep the normative text as is.

The basic problem is that you are using ptime and maxptime way beyond 
what was ever intended by the authors of these parameters. I am certain 
of this when it comes to maxptime, as I am the one that defined that one 
with some help.

Therefore I think one needs to look at the different use cases for RTP 
midi. I think there are several.

1. Local play, where most of RTP's timing model seems to be ignored, as 
it can't be excepted to buffer in either end of the chain.

2. Transmitting a live session to another point, but in which cases it 
is acceptable to introduce some small delay. This is done over a network 
that do not have the jitter behavior necessary to ignore the timestamp. 
Which means a receiver must follow the timestamp to give the right inter 
command time spacing.

3. Streaming of midi content.

In 1, I think you need something new to indicate the intended behavior 
of minimal transmission delay and local live play. However I do hope 
that a sender at least correctly timestamp the RTP packets sent. Thus 
the RTP receiver can in the future start using the timing model correctly.

In 2, I think the usage of ptime and maxptime could be reasonable to 
instruct a sender to gather 10, 50 or 100 samples or 1, 5, 20 ms of 
commands into each packet. However if you like to have finer resolution 
than 1 ms then it is necessary to have a new parameter.

In 3, theses parameters are not used at all. The amount of commands in 
the packets are pre determined for best performance at the receiver.

In conclusion I would suggest that you use another MIME parameter than 
ptime and maxptime to indicate the desired behavior.

I also think that the format drafts needs to be more explicit on 
considerations for the RTP timestamp. What rates are suitable, what are 
the impact on playback timing. There seems to be a bit on this in the 
guidelines, but I haven't had time to read it fully.

Cheers

Magnus Westerlund

---

From: lazzaro <[email-masked]>
Date: Fri, 22 Oct 2004 14:17:06 -0700

On Oct 22, 2004, at 5:55 AM, Magnus Westerlund wrote:
> I think I understand this, however the RTP timestamp rate
> as I understand it for MIDI would be equal to the sample rate for the 
> audio produced?

Yes -- for best results, the MIDI timestamp resolution should
equal the audio sampling rate (44.1, 48, 96, or 192 KHz).

> However I do hope that a sender at least correctly timestamp the RTP 
> packets sent.
> Thus the RTP receiver can in the future start using the timing model 
> correctly.

Sender time stamping is a MUST.

> ptime and maxptime to indicate the desired behavior.

OK, I'll redo this part of the I-D, and create new MIME parameters
to partially convey what the I-D previously coded using ptime and 
maxptime.

Unless a miracle happens, I won't make the Monday deadline with
a new I-D pair ... so it will have to wait post-D.C.  But I really hope 
we can
make it to Last Call for the two I-Ds by the holidays ...

> considerations for the RTP timestamp. What rates are suitable, what 
> are the impact on playback timing. There seems to be a bit on this in 
> the guidelines, but I haven't had time to read it fully.

See C.6.3 in the normative I-D.  Part of Appendix C.6, the
"SDP Interoperability Section", it might be easier to read all of C.6.

Also, in the guide I-D, Section 4 discusses sender timing,
Section 6 discusses receiver timing.

> for RTP based on your description.

Great -- thinking it over last night, I'm pretty sure that if I was 
forced
to define the media time for a packet with a single MIDI command
whose command timestamp equals the RTP timestamp to be 1 RTP
timestamp tick long (and not 0 ticks as it is now), the recovery journal
would break.

---
John Lazzaro