Subject: [AVT] RE: RTP MIDI I-D updates ...

From: lazzaro <[email-masked]>
Date: Thu, 12 May 2005 11:16:50 -0700

On Sun, 24 Apr 2005 21:40:18 -0700 John Lazzaro wrote:
>         Just sent off RTP MIDI I-D updates
> to internet-drafts, you can download them
> now from the Berkeley mirror site:
>
> http://www.cs.berkeley.edu/~lazzaro/sa/pubs/txt/current-rtp-midi.txt
> http://www.cs.berkeley.edu/~lazzaro/sa/pubs/txt/current-guide.txt
>
>         Until May 15th, I'll be busy wrapping up
> the course I'm teaching this semester [...]

Teaching responsibilities ended a few days early.
So, if anyone has comments they've been holding
back sending to the list, now is a good time to send.

A good first step might be to classify which of the
responses for the 43 issues are considered resolved
and which ones are contentious.  I could then release
a document with a more manageable change log, that
listed the still-open issues.  This could also let me slip
in some errata that implementors have passed along
in the past few weeks ...

Also, one note from my earlier posting:

>         C.6.2 is interesting because I did my best
> to push RTSP as hard as I could to handle the
> full range of stage and studio products. In the process,
> I basically had to bend RTSP until it (almost) broke,
> and I still didn't quite get all the semantics I need
> to get the functionality pro audio folks will expect.
> See C.6.2 for details; for those short on time, read the
> preamble, then skip to the examples in C.6.2.3.
>
>         There are many paths possible on this issue.
> Ideally, there's a way to keep the stage and studio
> material in the RTP MIDI I-Ds, use RTSP as is, and
> fully solve the problem.  But maybe that's not possible.
> If so, we'll have to think outside the box.

I'm starting to believe that the right thing to do is to
accept that the IETF has mature session management
protocols and frameworks that are a good match to the
VoIP-like Network Musical Performance application
for RTP MIDI (namely, SIP) and for content-streaming
RTP MIDI work (namely, RTSP).

But, getting a good fit for the "stage and studio" applications
from an IETF session management protocol (RTSP, as I
tried to do in C.6.2, or SIP, which I have sketched out
informally in the past) is going to take real work that
is more appropriately done somewhere else than AVT.

So, it may be wise to limit the scope of the I-Ds to cover
the two mature applications, and limit references to
"stage and studio" to the introduction, where we would
note that the protocol was designed to be useful in
those domains, but that the I-D focuses on applications
for which the upper layers of the IETF stack is ready to go.

I think this particularly makes sense because many
stage and studio devices (particularly video devices)
won't be using MIDI at all, but yet will face the same
issues that the examples in C.6.2 do.

One could imagine reworking C.6.2 to be less MIDI specific,
and spinning it off into an MMUSIC document ... or, one could
imagine leaving session management for stage and studio to
a standards organization that specializes in the pro world,
rather than leaving it in the IETF ...

---
John Lazzaro
http://www.cs.berkeley.edu/~lazzaro
lazzaro [at] cs [dot] berkeley [dot] edu
---

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 20 May 2005 14:04:25 +0200

Hi,

Some comments on the updated draft (08) regarding the MIDI command 
spaces and how they are multiplexed.

1. Section 2.1:
"All RTP streams from all parties in a multimedia session whose payload
types (coded by the PT header field) are mapped to the rtp-midi media
type share a single RTP session, and thus a common SSRC payload field
space (as defined in [2]).  Likewise, all RTP streams from all parties
in a multimedia session whose payload types are mapped to the
mpeg4-generic media type in mode rtp-midi share an (independent) single
RTP session."

I don't like this definition at all. RTP has a session context and that 
context should be indicated by the signalling, rather than being 
overridden by the RTP payload format. This limits the possibilities for 
usage. The signalling should tell the participant in a session if there 
is a single session or multiple sessions. So please don't change the RTP 
session usage. I would also like to point out that RTP payload types 
only have RTP session scope. They are not carried over to other 
sessions. Each RTP session can have its own PT=96 with what ever 
configuration.

Instead to support the use case you are talking about in the below 
paragraph:
"So, for example, a party may split a single 16-channel + Systems MIDI
name space into two RTP MIDI streams, each containing a subset of MIDI
commands.  One stream may be sent over UDP transport (perhaps this
stream contains real-time Note commands), the other stream may over TCP
(perhaps this stream contains only bulk-data System Exclusive commands,
unsuitable for UDP).  Each stream uses the same SSRC.  The session
description maps a different payload type onto each stream, and via this
payload type describes the nature of the MIDI command split, and perhaps
the rendering method for the stream."

This combination of different media lines in SDP should be done 
explicitly using a grouping of media lines RFC 3366.

2. Section 2.1:
"If a media line contains an RTP MIDI payload type, the media line
payload type list MUST consist entirely of payload types mapped to the
rtp-midi media type, or entirely of payload types mapped to the
mpeg4-generic media type in mode rtp-midi."

This restriction is also not good. It prevents usage of any of the 
standardized RTP robustness mechanism that operates on RTP paylaods. See 
RFC 2198, RFC 2733 and the onging work on RTP retransmission. I don't 
understand the reasoning behind this. Are there any motives for this.

My understanding what you tried to accomplish was that you may have 
several MIDI command spaces, each identified by the RTP PT. Each midi 
command space may have multiple entities sending (kind of shared cable 
or multiplexor box), and any number of entities receiving it. In 
addition you like to have the capability to split certain commands or 
data over different transports for reliabilty reasons.

To allow this I would propose to modify your proposal to the following:

Different MIDI spaces are identified by use of payload types as each 
MIDI space needs its configuration data. An SSRC SHALL only be sending 
for a single MIDI space. Any number of receivers MAY be used. One or 
more senders with unique SSRCs MAY exist within a MIDI space, depending 
on the MIDI space usage. Several MIDI spaces may share a single RTP 
session.

If one desires to share MIDI space between more than one transport 
option. Several m= lines are declared. These are grouped together using 
Grouping of media lines.

Comments, opinions?

Cheers

Magnus Westerlund

Multimedia Technologies, Ericsson Research EAB/TVA/A
----------------------------------------------------------------------
Ericsson AB                | Phone +46 8 4048287
Torshamsgatan 23           | Fax   +46 8 7575550
S-164 80 Stockholm, Sweden | mailto: [email-masked]

---

From: lazzaro <[email-masked]>
Date: Fri, 20 May 2005 19:08:11 -0700

On May 20, 2005, at 5:04 AM, Magnus Westerlund wrote:
> Some comments on the updated draft (08) regarding the MIDI
> command spaces and how they are multiplexed.

OK, these fit in to where -09.txt is going ... namely, to only
cover in the I-D session management issues for the application
domains where mature IETF session management protocols exist
for the domains (RTSP content streaming, and network musical
performance using SIP sessions) and to remove SDP-related
tools that are specific to stage and studio.  The issues you address
below have their roots in stage and studio applications.

> I don't like this definition at all. RTP has a session context and  
> that context should be indicated by the signalling, rather than  
> being overridden by the RTP payload format. This limits the  
> possibilities for usage. The signalling should tell the participant  
> in a session if there is a single session or multiple sessions. So  
> please don't change the RTP session usage. I would also like to  
> point out that RTP payload types only have RTP session scope. They  
> are not carried over to other sessions. Each RTP session can have  
> its own PT=96 with what ever configuration.

OK, this will go back to the normal way.

> explicitly
> using a grouping of media lines RFC 3366.

We weren't using PTs for grouping here -- RTP MIDI uses the musicport  
parameter
for MIDI-savvy grouping.  Musicport, defined in Appendix C.4, is a  
better tool
for RTP MIDI grouping than RFC 3388, because musicport encodes several
types of information about a set of MIDI namespaces -- identity and  
orderedness,
as defined in Appendix C.4 -- into a single integer value.

What we were using PTs for is to get two streams whose timestamp
field can be used for interleaving in a feed-forward way -- two streams
whose timestamp fields have the same random offset.  This property
greatly simplifies the task of reliably splitting up a MIDI source  
and putting
it back together.

But, this example becomes much less interesting now that -09.txt will
remove stage and studio applications from the I-D -- so, I'll just  
remove
examples of this sort, and accept that this problem is hard presently.

> standardized RTP robustness mechanism that operates on RTP  
> paylaods. See RFC 2198, RFC 2733 and the onging work on RTP  
> retransmission. I don't understand the reasoning behind this. Are  
> there any motives for this.

The restriction was written overly broad -- it only was intended to
prohibit mpeg4-generic RTP MIDI and native RTP MIDI appearing
on the same line.  So I'll rewrite it to remove your concerns.

Also, I'll take a second look to see if even this narrow native/mpeg4- 
generic
restriction is really needed.  The underlying reasons were complex,  
and may
have changed now that stage and studio is slated for removal.

> several MIDI command spaces, each identified by the RTP PT. Each  
> midi command space may have multiple entities sending (kind of  
> shared cable or multiplexor box), and any number of entities  
> receiving it. In addition you like to have the capability to split  
> certain commands or data over different transports for reliabilty  
> reasons.
>

As I described above, musicport handles grouping, not PT,
and its very finely tuned to handle what RTP MIDI needs.  So,
when we remove the PT functionality from -09.txt, we will still
have all the expressive power we need, and the changes you
propose are not needed.

---
John Lazzaro

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 23 May 2005 16:53:56 +0200

Hi John,

See comments inline:

> as defined in Appendix C.4 -- into a single integer value.

I reread section C.4. I don't find the section particular clear. 
Especially I reacted to this first normative statement:

"If any payload types on an RTP MIDI media line define a musicport
parameter, all payload types on the media line MUST define a musicport
parameter that is assigned the same value."

This requires that different midi spaces use separate RTP session. Was 
that really intended?

My concern is that you do a grouping of the media lines from payload 
type level, without anything at the top level like RFC 3388 to indicate 
that this is happening. Grouping of media lines are intended for exactly 
the purpose of identifying that two media lines has a relation that goes 
beyond the normal we are streams in a synchronized multi-media session.

Thus I don't see a problem with musicport to provide the ordered 
relation. However I think that this special relation should be indicated 
on a higher level than payload types, using RFC 3388.

So as long as the music port is linked to the PT you are using the PT 
for this type of multiplexing.

> greatly simplifies the task of reliably splitting up a MIDI source  and 
> putting
> it back together.

Okay, this mechanism I have missed. To my understanding what you tried 
to accomplish what that two different transport destination would form 
part of the same RTP session. Thus the same SSRC on both transports 
would be the same part so that the following RTP sequence could be sent:

TCP  1      4 5
UDP    2  3     6 7

However the above can't be used, because the packet loss detection 
doesn't work. It is impossible to determine what the loss rate actually 
is over a specific transport link.

So what can work is to say that in the case of split transport. For a 
split stream the same SSRC should be used in both sessions. This 
requirement has some implication on collision detection and resolution. 
RTCP Cname would need to be used to confirm that two different sources 
really is the same.

Then in addition to simplify things the same RTP TS offset can be 
recommended to be used. However the synch information in RTCP must 
confirm this view that they are running the same time frame.

> remove stage and studio applications from the I-D -- so, I'll just  remove
>> standardized RTP robustness mechanism that operates on RTP  paylaods. 
>> See RFC 2198, RFC 2733 and the onging work on RTP  retransmission. I 
>> don't understand the reasoning behind this. Are  there any motives for 
>> this.
> 
> 

Good.

> restriction is really needed.  The underlying reasons were complex,  and 
> may
>> command space may have multiple entities sending (kind of  shared 
>> cable or multiplexor box), and any number of entities  receiving it. 
>> In addition you like to have the capability to split  certain commands 
>> or data over different transports for reliabilty  reasons.
>>

I am not fully agreeing that what I as suggesting is not needed. I think 
there are still concerns on how the multiplexing is done an signaled in 
these more advanced case.

Cheers

Magnus Westerlund

---

From: lazzaro <[email-masked]>
Date: Mon, 23 May 2005 12:41:35 -0700

> I reread section C.4. I don't find the section particular clear.

I'll rewrite it to make it clearer ...

[points below are reordered to help the narrative flow ...]

> My concern is that you do a grouping of the media lines from  
> payload type level, without anything at the top level like RFC 3388  
> to indicate that this is happening. Grouping of media lines are  
> intended for exactly the purpose of identifying that two media  
> lines has a relation that goes beyond the normal we are streams in  
> a synchronized multi-media session.
>
> relation. However I think that this special relation should be  
> indicated on a higher level than payload types, using RFC 3388.

OK, that's fine -- I'll add extra RFC 3388 stuff to signal the  
multiplexing
is happening, and musicport will stay to signal the MIDI semantics of
the multiplexing.

> This requires that different midi spaces use separate RTP session.  
> Was that really intended?

My answer is forward-looking to -09.txt, where RFC 3388 will signal  
multiplexing, not PTs.

In -09.txt, the answer to your question will be "yes".   Each RTP  
session
is dedicated to a single MIDI name space.   If a particular party  
wishes to send
data on two name spaces, it needs two RTP sessions.

To do anything else is to use the PT for MIDI multiplexing.  If PT1
for an RTP session handles MIDI channels 0-15, and PT2 for the
same RTP session handles MIDI channels 16-31, and if a sending party
takes turns switching between PT1 and PT2 on the same RTP session, well,
this seems like the very definition of using the PT for multiplexing  
to me.

Also, recall that two parties sending MIDI streams in the same
RTP session are NOT sending to the same namespace -- these name
spaces have an association of some sort (for example, handshaking
commands in MIDI consider the two cables as a paired entity), but
that is different from "sharing the same name space".  For the
association to make sense, all parties need to choose a PT whose
MIDI name space matches the PT chosen by all other parties.
The only way to effectively do this is to have one name space for all  
PTs.

> So what can work is to say that in the case of split transport. For  
> a split stream the same SSRC should be used in both sessions. This  
> requirement has some implication on collision detection and  
> resolution. RTCP Cname would need to be used to confirm that two  
> different sources really is the same.
>

My preference now is to use the traditional RTP tools: split  
transport is done
with 2 RTP sessions, the grouping is signalled by RFC 3388 signalling,
the MIDI semantics of the grouping is signalled by musicport, the two
streams are independent synchronization sources with independent RTP
timestamp offsets, and synchronization is done via RTCP NTP/RTP
timestamp pairs.

The responsibility lies with the creator of the session descriptions to
split streams in a way where the temporal rearrangement of
commands on the two streams does not change MIDI semantics.

Note that packet loss is not an issue here -- the UDP session has
a recovery journal to handle loss, the TCP session has no loss.
The issue is that, due to lost or late packets, commands can be
executed out of order from the standpoint of the original unsplit
MIDI stream.

However ...

Consider that the only MIDI commands that are truly "too big"
for UDP are commands like "transfer this large block of data".
These are all System Exclusive commands.  I believe that
it is acceptable to segregate these commands to a TCP
stream, and the relative timing between the large SysEx
commands and real-time commands will be fine with the
temporal sync that RTCP NTP/RTP TS pairs provides.

Smaller real-time oriented SysEx commands belong on
the UDP RTP session, because real-time response is important.
I believe the journalling tools are sufficient for protecting
these sorts of commands, and if not, the answer lies in
designing better journalling systems, this is why there is
a parameter for signalling the journal system presently in use.

If one is making a new product from scratch using RTP MIDI
(as opposed to transcoding a MIDI 1.0 cable from an existing
product), one simply uses two MIDI streams, each with an
independent name space, one to do file transfer and one to do
real-time.  Or alternatively, one adds a web server into their
product to transfer files, and uses MIDI only for real-time work.

> I am not fully agreeing that what I as suggesting is not needed. I  
> think there are still concerns on how the multiplexing is done an  
> signaled in these more advanced case.

Hopefully the adoption of RFC 3388 described above handles
these concerns.  Please let me know, I'm writing up -09.txt
as we speak ... ideally, we can get your input in time so that
the -09.txt can handle all of your remaining concerns.

---
John Lazzaro

---

From: lazzaro <[email-masked]>
Date: Mon, 23 May 2005 14:46:52 -0700

> Was that really intended?

One addendum to my earlier posting (which as you
recall, answered this question "yes" -- this answer
is unchanged, below provides addition info on it ...).

---

 From RFC 3550:

Synchronization source (SSRC):

[... start of definition deleted ...]

All packets from a synchronization source form part of the same
timing and sequence number space, so a receiver groups packets by
synchronization source for playback.

[...]

A synchronization source may change its data format,
e.g., audio encoding, over time.

---

I don't think switching between PTs that specify different MIDI
name spaces during a session is in the spirit of the final line
of text from the definition above.

Two MIDI name spaces are not different "audio encodings" of the
same  "microphone".  Instead, channels 0-15 might be the output of
a MIDI keyboard controller player by musician A, channels
16-31 might be the output of a MIDI drum pad played by
musician B, were A and B are two players in a network musical
performance who happen to both be located in the same physical
place sharing the same computer for session management.

Using PT values to switch back and forth between musician A
and musician B isn't "changing the audio encoding" of a single source --
it's using PTs to multiplex two sources into one RTP session.

In addition, the recovery journal was not designed to support
this mode of operation -- any sequence number break triggers
recovery journal actions to mend the command stream, but
using PTs in this way creates lots of innocuous sequence
breaks.

The sensible way to use PTs here is to select different
"audio encodings" of the same MIDI performance.  For example,
different renderers mapped to different PTs (although there is an
alternative way to choose different renderers within a single PT,
using the RTP MIDI multimode parameter, which is more flexible;
most apps will use that instead of PTs).

---
John Lazzaro

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 24 May 2005 10:11:41 +0200

Hi John,

I agree with the below interpretation regarding what PT is intended to 
be used. However I would suggest that one requires the use of different 
SSRCs for different midi spaces but have the PT to indicate which space 
it uses. This should resolve all your concerns below regarding packet 
losses as each SSRC has its own RTP sequence number space.

This I think is to correct trade off for using RTP functions. One uses 
different RTP sessions when the intended usage of the MIDI is different. 
For example separating light control and instruments. Which to me if I 
interpret your language right would be different classes of renderers.

The SSRC is used to indicate different sources/receivers being connected 
to the virtual "midi cable". Which midi cable it is, is indicated 
through the Payload type together with all the other parameters around 
this cable.

The alternative I see is to move the also the "cable" specification to 
RTP session level as you suggest. However I don't see a real gain in 
this. It cost RTP sessions, which people often complain are expensive in 
ports etc, it doesn't really have that much clearer multiplexing. The 
existing midi spaces are clearly signaled upfront through the RTP 
payload types that may occur in the RTP session. Thus a participants now 
what renders etc that are needed.

Others input on this would be appreciated.

Cheers

Magnus

-- 

Magnus Westerlund

---

From: lazzaro <[email-masked]>
Date: Tue, 24 May 2005 16:57:53 -0700

On May 24, 2005, at 1:11 AM, Magnus Westerlund wrote:
> I agree with the below interpretation regarding what PT is intended  
> to be used. However I would suggest that one requires the use of  
> different SSRCs for different midi spaces but have the PT to  
> indicate which space it uses. This should resolve all your concerns  
> below regarding packet losses as each SSRC has its own RTP sequence  
> number space.

OK, if we start down this path then it probably also makes

> confirm this view that they are running the same time frame.

So that -09.txt will represent all of the extra features needed
to do more complicated session things.  If later on it turns out that  
there
isn't consensus on these features, I'll pull them back out and do a  
-10.txt ...

---
John Lazzaro