Subject: RE: [Rmt] Should FLUTE File Aggregation I-D be a working group item?

From: [email-masked]
Date: Fri, 25 Nov 2005 12:18:28 +0200

Hi Mark et al.

Thanks for the support

<I'll take time to consider the AO semantic suggestions - an email for
another day> 

>Some other minor comments:

>- I don't think we need to worry about backwards compatibility 
>with FLUTE receivers compliant to the experimental version 
>which did not have element extensibility in the FDT XML - this 
>was a mistake and the 'experimental' phase is exactly supposed 
>to allow us to find and correct such mistakes.

Now the update FLUTE is practically ready (anticipate one last I-D
update), I agree.

The AO I-D previously the use of non-TOI=0 FDTs to help  with Exp
backwards compatibility. Non-TOI=0 FDTs are allowed anyway, but we need
not explain that this is useful in handling the experimental FDT
extensibility issue.

>- the draft includes some text attaching semantics to TOI 
>ordering - something we removed from FLUTE in favour of FDT 
>Instance IDs. We should stick with the FLUTE principle that 
>later version of a file is signalled by using a different TOI 
>in an FDT with a larger FDT Instance ID - it doesn't matter 
>whether the TOI value is greater or less than the previous one.

Note, for FLUTE there is no such semantic. The client has no gauntees
about FDT Instance numbers, no garuntees about TOI numbers.

Generally, using TOI and 'FDT Instance number' values for this has
similar issues (assuming wrap-around is handled smoothly, higher/later
values represent later versions, gaps in the range of number are
feasible, FDT Instances may never be received). TOI keeps semantics with
the file declaration; FDT requires inheritence from FDT Instance to
files described for that Instance. Implemantation-wise there's little
difference between the two: probably the server will be incrementing FDT
Instance numbers and not describing out-of-date TOIs on new Instances;
probably the server will use higher TOI values for later versions; the
client must store TOI with fileURI. There is no requierment that the
client stores FDT Instance numbers.

Quick FLUTE quote:

   A receiver of the file delivery session keeps an FDT database for
   received file description entries.  The receiver maintains the
   database, for example, upon reception of FDT Instances.  Thus, at any
   given time the contents of the FDT database represent the receiver's
   current view of the FDT of the file delivery session.  Since each
   receiver behaves independently of other receivers, it SHOULD NOT be
   assumed that the contents of the FDT database are the same for all
   the receivers of a given file delivery session.

   Since FDT database is an abstract concept, the structure and the
   maintaining of the FDT database are left to individual
   implementations and are thus out of scope of this specification.
...
   *  The receiver SHOULD NOT use a received FDT Instance to interpret
      packets received beyond the expiration time of the FDT Instance.

i.e. the expiry time of file description should be kept with other file
data, but FDT Instance ID is up to the implementation.

Forcing clients to additional store this value is not much trouble - but
it is _not_ "the FLUTE principle that later version of a file is
signaled by using a different TOI in an FDT with a larger FDT Instance
ID". Which is exactly why the AO draft offers the simplest solution to
this issue. If we make the assumptions of "well-behaved simpple FLUTE
servers" above, there really is nothing between FDT Instance ID and TOI
to be used for this purpose. The reason I give all this background is
that this doesn't hold for some obvious AO usage...

For a physically aggregated object (e.g. very large) with some small
updates (e.g. 2 file versions are new from 10,000 files). The server
will use TOI (i) for the initial full object files descriptions. It is
highly likely that the big aggregated object will continue to be
transmitted for some time and with only minimal updates it is probably
more bandwidth efficient to send the symbols for the original object and
the few (e.g. 2) new file versions seperates (more efficient due to,
e.g. fountain, symbols of earlier reception still being useful to all
receivers regarless of complete, partial or not-started receiption
status). So the server sends FDT (ii) with the new TOIs for the files
which have been updated. <At this point using either TOI or FDT Instance
ID would work>. However, some objects of FDT Insatnce (i) are now
changed/removed so that the server need to describe the slighlty old
large aggregated object on TOI (iii). All recievers throw away the new
files (even though the TOI is higher) as the old files are described in
a later FDT Instance.

FLUTE is specifically designed to allow this ebhaviour and make no
assumption the receivers FDT/file database state and no assumption on
reception of individual FDT Instances.

Using TOI for this is simple and works. FDT Instance ID fails. Hence,
the mention of TOI for this purpose in the I-D. (Originally it was start
with TOI=1 and increment by 1, but that's overkill as just 'higher
value' is needed for subseqent versions of a file (fileURI) instance)

(I've been trying to explain this for a while - but it's not important
in 3GPP rel6, and I'm not up-to-date with DVB or OMA to see if
something's fractured there - so maybe this is the first time I've
explained by mail.)

>- the draft mentions that one might only consider file 
>aggregation when the aggregate is less than a single source 
>block - I think it is simpler and has no disadvantages if you 
>decouple the source blocking (which is a feature of the FEC 
>scheme chosen) from the file aggregation. If files belong 
>together in an aggregate then they should be put together and 
>then if the FEC code needs to split that data into blocks this 
>is ok and does not result in any worse performance so long as 
>it correctly interleaves the packets from the different blocks.

Exactly.

The I-D does not mean to bind AO and FEC blocking. The discussion is for
implementation guidence: for files much greater than the block size
there's no FEC benefit in physical aggregation; for the composite files
being less than the block size it always makes sense to aggregate for
robustness; inbetween it's a compromise; the I-D absolutely does not
impose any FEC instance retraints and so it's implementation specific to
optomise complexity and performance in this aspect. The guidence is
valuable. If we inadvertantly made normative specification on this, it
was an error to be corrected.

Also, the logical grouping between objects can be used to get the
"belongs together" semantics for multiple AOs - e.g. where partial
recovery is undesirable, or different levels of robustness are
desirable).

Cheers, Rod.

_______________________________________________
Rmt mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rmt

---

From: "Michael Luby" <[email-masked]>
Date: Fri, 25 Nov 2005 09:41:53 -0800

Hi Rod,
A couple of quick comments below. I've deleted most of your response and
just kept that parts needed to understand my response.
Regards, Mike

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> [email-masked]
> Sent: Friday, November 25, 2005 2:18 AM
> To: [email-masked]
> Cc: [email-masked]; [email-masked];
> [email-masked]
> Subject: RE: [Rmt] Should FLUTE File Aggregation I-D be a working group
> item?
>
>
> Hi Mark et al.
>
...
> about FDT Instance numbers, no garuntees about TOI numbers.

I think Mark's main point was not about the ordering with respect to FDTs
(whether it is there or not is kind of irrelevant) but instead that that
TOIs can be used by a receiver to uniquely identify which version of a file
it has, which can be useful for incremental file updates using an AO, and
that TOIs do not have any restrictions (like ordering) on their values,
which is all good.

> was an error to be corrected.

There is a lot of FEC benefit to having the FEC apply over the entire AO
when the AO is much larger than a single source block, and so the guidance I
believe is misguided and steers implementors in the wrong direction.  We can
talk more about this as the work progresses, but this is one of several
parts of the file aggregation draft that I think needs significant
rethinking and revision.

---

From: Joerg Ott <[email-masked]>
Date: Mon, 28 Nov 2005 10:09:00 +0200

Hi Rod,

I think fioe aggregation is basically a useful thing to consider as
the motivation clearly points out.  In Vancouver, I raised the point
about re-inventing gzip.  The following statement in the motivation
section makes me a bit nervous:

    Another motivation for having an object aggregation scheme compared
    to a basic archive based solution (e.g. tarball), is that no extra
    transformation (i.e. archive creation or extraction) is required at
    either the FLUTE sender or receiver.  Everything is managed
    automatically by the transport mechanism according to transport-
    specific optimizations and can be transparent to upper applications
    (i.e. built on top of FLUTE), or enhanced by application hints on
    file relationships, without breaking the basic semantics of FLUTE
    sessions.

Yes, you can always push functionality down a layer (which still means
that you have to perform the function but you just abstract from its
details for the application and limit its amount of work and the
flexibility in applying this function).  But you should carefully
ask yourself how far you want to take this.  Will we see
semantically more complex relationships tomorrow?  Compression
of data?  ...?  How do you deal with partially received objects?
How complex does the API to your transport get?

There is a certain number of things that only the application can do
right by itself.  I would like to see a clear borderline drawn
of what shall go into the FLUTE transport and what should stay up
at the application layer and then we should make sure that the
extensions we define now are going to be orthogonal to other
features we may want to add to the transport later and that they
are easily usable by the application for what it may want to do.

(Remember that we are not building "systems" in the Transport Area
and that we do not necessarily have to build application libraries
either.  So, how much is needed for interoperability in FLUTE?)

Cheers,
Joerg