Subject: RE: [Rmt] Should FLUTE File Aggregation I-D be a working group item?

From: [email-masked]
Date: Tue, 22 Nov 2005 11:43:19 +0200

Hi All

If you haven't read the document yet there are two things as play which
may be confused:
Group of files into:
- logical groups OR physically one transmission object
Extened-FLUTE-FDT signalling of files in physically aggregated objects:
- using FLUTE normally (e.g. TOI 0) OR embedding the aggregated files'
FDT in the object

Magnus, correct me if I misunderstood - you're talking about the second
issue. (i.e. you are questioning the placement of the file transport
data - but are happy with the files going into multipart MIME?)

Yes, unfortunately it will not be backwards compatible 
>with the normal FDT, but it could have a number of benefit:

Well you could give the extended FDT a new MIME type and put it on
"other than TOI=0" and then it's backwards compatible. Also, you can
extend the FDT with "aggregated file" elements (instead of using the
"file" element and so you'd be able to change semantics for aggregated
files without gamaging "not-enhnaced" FLUTE receiveres. Is there
something I missed about backward compatability onthis?

>- The possibility to externally before receiving the object 
>determine if all files already has been received or if some 
>has. This is quite important to minimize the connection time 
>when doing repeated transmission to repair/complete unfinished 
>sessions.

This seems to confirm it's about where to signal the file details of
objects. Doing this as an "index FDT" in the multipart MIME assumes that
the kind of use case mentioned is not significant. i.e. that it's highly
unlikely that clients will have all the data of an object before it gets
sent, and it's highly unlikely that objects with different fileURIs but
the same files will get sent. If these are not good assumptions,
extending the FDT is indeed a better method: but not without cost as
bloated FDTs start to require their own FEC and reliability techniques
and (unlike embedding in the aggregated object) there is no garuntee
that all FDT instances will be received even though all files of an
aggregated object are.

So do we need to allow both methods, or is there a way to limit to just
one? (To me it seems both, but with advisory text as to when each is
better and some recommendations).

>- Possibility to extract data from an partially decoded block. 
>With small files and large FEC blocks this becomes highly 
>interesting. Thus the data offset and length needs to be 
>available from external source. 
>Special data integrity mechanisms to make this reliable may 
>also need to be considered.

There are two partically reveived deocding aspects:
 - partially received AOs, but some fully received blocks which enable
extraction of some complete files (relatively simple)
 - partially revieced blocks with some fancy maths to revover parts of
them (i.e. some files) (this is what Magnus said - but except for well
ordered FEC, or null FEC, codes seems improbable)

For well ordered FEC codes, indeed prior knowledge of byte ranges of
files can be mapped to source symbols and the FEC parameters can tell
the client whether it can restore the relevant source symbols for all or
part of certain files (generally only all is useful, but for some
applications even files with corruption may be partially renderable -
e.g. corrupt images on a web page).

For the extreme-FEC-dudes, putting into nomral FLUTE TOIs, instead of
embedding into the multipart MIME, is much more robust for partial
recovery from incomplete received blocks.

Cheers, Rod.

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On 
>Behalf Of ext Magnus Westerlund
>Sent: 21 November, 2005 12:24
>To: Christoph Neumann
>Cc: Vincent Roca; [email-masked]
>Subject: Re: [Rmt] Should FLUTE File Aggregation I-D be a 
>working group item?
>
>Hi Christoph,
>
>See inline.
>
>Christoph Neumann wrote:
>> Dear all,
>>  To start the discussion on the object aggregation draft, I 
>wanted to 
>> point out some issues that I think are in favour of having 
>this as an WG item.
>>  Physical aggregation scheme is of great advantage to avoid 
>the coupon 
>> collector problem. In the case of a large number of small files, 
>> physical aggregation combined with FEC is the only way to 
>effeciently 
>> transport the content.
>> Therefore I think it is necessary to have some standardized solution 
>> for that, which could possibly be reused in several use cases with 
>> different FEC schemes...
>> The draft exaclty addresses this problem.
>
>But in the process it looses a number of properties that 
>individual files. I think there would be a number of benefit 
>to NOT use multi-part mime and instead use an FDT based 
>scheme. Yes, unfortunately it will not be backwards compatible 
>with the normal FDT, but it could have a number of benefit:
>
>- The possibility to externally before receiving the object 
>determine if all files already has been received or if some 
>has. This is quite important to minimize the connection time 
>when doing repeated transmission to repair/complete unfinished 
>sessions.
>
>- Possibility to extract data from an partially decoded block. 
>With small files and large FEC blocks this becomes highly 
>interesting. Thus the data offset and length needs to be 
>available from external source. 
>Special data integrity mechanisms to make this reliable may 
>also need to be considered.
>
>I think these comments becomes extra important when 
>considering battery powered wireless devices.
>
>
>>  I heard that some physical aggragation scheme has also been 
>defined in 3GPP
>> MBMS, but I'm not really following 3GPP activities and are 
>therefore not
>> aware of that. Maybe someone has an input here?
>>  Beside the coupon collector issue, some FEC schemes perform 
>badly with
>> small files, and aggregating them, seems the only solution 
>in this context
>> (as it has been mentioned on the slides during the rmt session).
>
>The aggregation mechanism defined in 3GPP is to aggregate several 
>different UDP datagram streams used in streaming. Please see
>http://www.ietf.org/internet-drafts/draft-watson-tsvwg-fec-sf-00.txt
>for the outline of that solution and which is an initial proposal for 
>the most likely to be FECFrame WG. The actual 3GPP 
>specification of this 
>is TS 26.346:
>http://www.3gpp.org/ftp/Specs/html-info/26346.htm
>Select the latest version.
>
>Cheers
>
>Magnus Westerlund
>
>Multimedia Technologies, Ericsson Research EAB/TVA/A
>----------------------------------------------------------------------
>Ericsson AB                | Phone +46 8 4048287
>Torshamsgatan 23           | Fax   +46 8 7575550
>S-164 80 Stockholm, Sweden | mailto: [email-masked]
>
>_______________________________________________
>Rmt mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/rmt
>

_______________________________________________
Rmt mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rmt

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 28 Nov 2005 16:16:53 +0100

Hi,

To clarify my position and say that I don't care about specific solutions.

Yes I am talking about the physically aggregated objects in previous 
emails. I think I share the same concerns that Joerg Ott raised. There 
is no meaning to reinvent an exact duplication of tar or zip at FLUTE 
level. If one should do this in flute rather than simply write a 
recommendation that applications that have many small files should use a 
aggregated file container, then I think we need to provide some clear 
benefits, for example extraction of files from partially recovered 
aggregates. Simplified or more efficient recovery and handling of 
updates of the files part of the aggregate.

Simply using multi-party MIME does not really gain anything compared to 
tar or zip, then perhaps somewhat clear rules on how URI for files are 
constructed. For example multi-party MIME requires the capability to 
search the aggregate from start to end to find delimiters. This 
operation could be seriously hampered or even produce errors if the file 
is partial. In addition it is missing any integrity mechanism that I 
think is important if one tries to use partially repaired aggregates.

My proposal is to define a physical aggregation mechanism that primarily 
defines how one builds the source block from multiple files and then 
allows the receiver to perform the reverse operation to extract them 
from that aggregate also in cases when decode has not been completely 
successful. For large burst errors the loss for an aggregate may result 
in loss of one or several objects within the aggregate but still have 
parts of the aggregate for which all source symbols has been received.

So if one can find a way around the issues with backwards compatible 
FDT, sure no problem. However my view on this is that if we introduce a 
aggregation mechanism it will any way result in all receiver to support 
it, some further required extensions of the FDT will not cost us any 
interoperability.

Cheers

Magnus Westerlund