Subject: RE: [Rmt] Should FLUTE File Aggregation I-D be a working group item?

From: [email-masked]
Date: Tue, 22 Nov 2005 11:58:33 +0200

The thread reminded me of the RMT minutes which said something about
applications for object aggregation - I think "web sites" was questioned
in the meeting?

As for using object aggregation for websites - it's a good idea
(providing you have a way to pay for the process of doing it :)

Obviously for static content there are no big update issues. For more
dynamic content (regularly changing web sites), then it is likely that
some "complete site" update is less frequent than a "delta" to update
the changed parts. There is no requirement to use a physically
aggregated object for the delta. In such a case it would make sense to
logically group the large aggregated object and the small update file
(or files) - as the semantics of "all belong together" are preserved.

Also, we have the logical aggregation semantics. This groups files to
the same logical object without physically transporting them as one. For
large files (multiples of the source block size), the FEC benefits of
physical aggregation drop off so it is simpler to use logical
aggregation: although optimal configurations of FEC schemes differ, HD
movies >1GB are likely to fall into the "easily big enough to skip
physical aggregation" category.

There are other times when logical aggregation is desirable. 3GPP uses
the technique to allow (for multicast service announcements) a metadata
envelope (version etc.) to be grouped with its corresponding metadata
fragment (e.g. a media description) this way so the client would know
that these belong together and the content/MIME type indicates the
behaviour. Multipart MIME is also allowed to combine these two related
objects, but the release 6 specification does not detail exactly how all
metadata gets to the client and so this technique allows, e.g., repair
of the metadata over HTTP without a GET for a correctly received
envelope.

Otherwise, the selection between logical and physical aggregation is
pretty much implementation specific (i.e. application specific) - where
the preference for FEC performance and SW complexity is balanced
according to the use cases and environment.

Hope that helps (or at least helps more than it confuses :)

Cheers, Rod.

_______________________________________________
Rmt mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rmt

---

From: [email-masked]
Date: Tue, 22 Nov 2005 12:00:42 +0200

I think this should be a work item for RMT. It is very FLUTE transport
specific and fits the role of being useful to the general Internet
community.

Cheers, Rod.

PS as a co-author I guess this could be inferred anyway :)

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On 
>Behalf Of ext Vincent Roca
>Sent: 09 November, 2005 21:01
>To: [email-masked]
>Subject: [Rmt] Should FLUTE File Aggregation I-D be a working 
>group item?
>
>Dear colleagues,
>
>Following the recommendation of Lorenzo on Monday, during the 
>RMT WG session, we are asking you whether you see any good 
>reason to make the FLUTE File Aggregation I-D 
>(draft-neumann-rmt-flute-file-aggregation-02.txt)*** *a 
>working group item or not.
>
>Please do not hesitate to comment.
>
>Cheers,
>
>   Vincent.
>
>

---

From: "Mark Watson" <[email-masked]>
Date: Thu, 24 Nov 2005 20:50:05 -0500

Rod, all,

I support this being a WG item.

Further on the comments below, I think there are some advantages if we
can keep some commonality between the old and new FDT syntax for
physical aggregation as well as for logical aggregation. What I would
suggest is the following:

- in both cases the files are described in normal <File> elements, with
their own TOIs and a possible the 'Group' attribute
- the new <AggregateObject> aggregates a set of _objects_ (identified by
their TOIs) into a new aggregate object (which is given a TOI of its
own)

The <AggregateObject> element simply needs to define the list of TOIs
and their offsets into the aggregate object (their lengths and MIME
types etc. will already be in their corresponding <File> elements). We
don't really need multi-part MIME encoding of the aggregate object and
I'm not sure what value it adds.

This approach has a number of advantages:
- less new XML syntax needed!
- Receivers that do not support the <AggregateFile> element will at
least understand the <File> elements and could receive the files
individually if the sender chose to send them that way (i.e. if the
sender explicitly wanted to provide backwards compatibility for such
receivers, perhaps at some lower service level than for new receivers)
- We already have a mechanism for sending updated versions of a file: we
signal the same URI but a different TOI. The approach above preserves
this mechanism: The receiver can tell when a new aggregate object
contains some of the same objects as a previously received aggregate
object (same TOIs) - it can use the stored copies of these files to fill
in parts of the new object in advance (i.e. extract FEC source symbols
from the previously stored objects) and thus reduce the reception time
for the new object.

In fact, in cases where it is expected that many but not all users will
already have some of the files being aggregated, it may make sense for
the sender to send only FEC repair symbols for the aggregated file -
these can be combined at receivers with the source symbols obtained from
previously received files which have not changed in the new aggregated
object.

Some other minor comments:
- I don't think we need to worry about backwards compatibility with
FLUTE receivers compliant to the experimental version which did not have
element extensibility in the FDT XML - this was a mistake and the
'experimental' phase is exactly supposed to allow us to find and correct
such mistakes.
- the draft includes some text attaching semantics to TOI ordering -
something we removed from FLUTE in favour of FDT Instance IDs. We should
stick with the FLUTE principle that later version of a file is signaled
by using a different TOI in an FDT with a larger FDT Instance ID - it
doesn't matter whether the TOI value is greater or less than the
previous one.
- the draft mentions that one might only consider file aggregation when
the aggregate is less than a single source block - I think it is simpler
and has no disadvantages if you decouple the source blocking (which is a
feature of the FEC scheme chosen) from the file aggregation. If files
belong together in an aggregate then they should be put together and
then if the FEC code needs to split that data into blocks this is ok and
does not result in any worse performance so long as it correctly
interleaves the packets from the different blocks.

Regards,

Mark

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
[email-masked]
Sent: Tuesday, November 22, 2005 1:59 AM
To: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: RE: [Rmt] Should FLUTE File Aggregation I-D be a working group
item?

Cheers, Rod.

---

From: "Mark Watson" <[email-masked]>
Date: Wed, 30 Nov 2005 19:39:55 -0500

Magnus, all,

I agree with your comments below that a new aggregation scheme must
offer advantages over simply zipping the files. I think such advantages
do exist.

Some of the requirements that should be met by an aggregation scheme -
which illustrate these advantages - are as follows:

- backwards-compatibility - at least source symbols sent for the files
being aggregated should be able to be interpreted by receivers which do
not support the aggregation scheme
- incremental update - as I mentioned in my other email, it should be
possible for repair symbols for the aggregate object to be combined with
source symbols which are already known (perhaps from receipt of an
earlier version of the aggregate which contained some of the same files)
- partial reception - if sufficient data has been received to recover
part of the aggregate, then it should be possible to extract the files
which are contained within the recovered part
- single file transmission - it should still be possible to send the
files individually, in particular for updates

So I think there is a question for the draft authors on whether they are
happy to take on these requirements in making the draft a working group
item - it may result in the solution being rather different from that
presently proposed.

...Mark

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Magnus Westerlund
Sent: Tuesday, November 29, 2005 12:17 AM
To: [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [Rmt] Should FLUTE File Aggregation I-D be a working group
item?

Hi,

To clarify my position and say that I don't care about specific
solutions.

Yes I am talking about the physically aggregated objects in previous 
emails. I think I share the same concerns that Joerg Ott raised. There 
is no meaning to reinvent an exact duplication of tar or zip at FLUTE 
level. If one should do this in flute rather than simply write a 
recommendation that applications that have many small files should use a

aggregated file container, then I think we need to provide some clear 
benefits, for example extraction of files from partially recovered 
aggregates. Simplified or more efficient recovery and handling of 
updates of the files part of the aggregate.

Simply using multi-party MIME does not really gain anything compared to 
tar or zip, then perhaps somewhat clear rules on how URI for files are 
constructed. For example multi-party MIME requires the capability to 
search the aggregate from start to end to find delimiters. This 
operation could be seriously hampered or even produce errors if the file

is partial. In addition it is missing any integrity mechanism that I 
think is important if one tries to use partially repaired aggregates.

My proposal is to define a physical aggregation mechanism that primarily

defines how one builds the source block from multiple files and then 
allows the receiver to perform the reverse operation to extract them 
from that aggregate also in cases when decode has not been completely 
successful. For large burst errors the loss for an aggregate may result 
in loss of one or several objects within the aggregate but still have 
parts of the aggregate for which all source symbols has been received.

So if one can find a way around the issues with backwards compatible 
FDT, sure no problem. However my view on this is that if we introduce a 
aggregation mechanism it will any way result in all receiver to support 
it, some further required extensions of the FDT will not cost us any 
interoperability.

Cheers

Magnus Westerlund

Multimedia Technologies, Ericsson Research EAB/TVA/A
----------------------------------------------------------------------
Ericsson AB                | Phone +46 8 4048287
Torshamsgatan 23           | Fax   +46 8 7575550
S-164 80 Stockholm, Sweden | mailto: [email-masked]