Subject: Re: [Cellar] matroska and side data vs timecode

From: Steve Lhomme <[email-masked]>
Date: Sun, 3 Nov 2019 11:29:34 +0100

Le mar. 29 oct. 2019 à 17:04, Dave Rice <[email-masked]> a écrit :
> I drafted a pull request at https://github.com/cellar-wg/matroska-specification/pull/348 that builds upon the new side data structure defined above. This pull request registers a BlockAddIDType for a 64 bit binary expression of timecode as defined by SMPTE ST12-1:2014 (which is mostly hours, minutes, seconds, frames, a drop frame flag, and other a few other flags and binary groups). This would allow timecode to be associated with Matroska frames such as:
>
> <Segment>
>  <!-- skipped elements here -->
>  <Tracks>
>   <!-- lots of other track data here -->
>   <TrackEntry>
>    <BlockAdditionMapping>
>     <BlockAddIDValue>2</BlockAddIDValue>
>     <BlockAddIDName>SMPTE ST 12-1 timecode</BlockAddIDName>
>     <BlockAddIDType>121</BlockAddIDType>
>    </BlockAdditionMapping>
>  </Tracks>
>  <!-- then within the Cluster which stores audiovisual data within frames in Blocks, there is -->
>  <Cluster>
>   <BlockGroup>
>    <Block> { a video frame } </Block>
>    <BlockAdditions>
>     <BlockMore>
>      <BlockAddID>2</BlockAddID>
>      <BlockAdditional> { SMPTE ST12-1:2014 64-bit binary representation of a 07:32:54;18 } </BlockAdditional>
>     </BlockMore>
>    </BlockAdditions>
>   </BlockGroup>
>   <BlockGroup>
>    <Block> { the next video frame } </Block>
>    <BlockAdditions>
>     <BlockMore>
>      <BlockAddID>2</BlockAddID>
>      <BlockAdditional> { SMPTE ST12-1:2014 64-bit binary representation of a 07:32:54;19 } </BlockAdditional>
>     </BlockMore>
>    </BlockAdditions>
>   </BlockGroup>
>  </Cluster>
> </Segment>
>
> With the BlockAdditions, BlockMore, BlockAddID, and BlockAdditional Elements, I think this would add 18 bytes per timecode expression. Adding timecode to one hour of PAL video (25 frames * 3600 seconds/hour) would add about 1.5 MB to the file.

That seems like a big overhead. Is the 64 bits value necessary ? It
seems 32 bits might be sufficient (saving 4 bytes). Maybe 40 if you
need extra flags.

> Implementation Considerations:
>
> Thinking ahead of this work, I think it would be worthwhile to add a dedicated section to the Matroska specification on timecode and describe two methods for adding a reference timecode to a Matroska Segment.
>
> 1. Using tags. Make an official tag “TIMECODE” to store a string of the timecode expression that correlates to the Matroska timestamp of 0. This approach is limited to storing timecode that is incremental and continuous. Defaults for the incrementation and the behavior of the timecode could be described as part of the tag definitions and feasibly another tag or tags could be reserved for timecode flags when the timecode behavior is not a default. Defining a use of a “TIMECODE” tag would provide the benefit of standardizing an existing practice in FFmpeg’s Matroska muxer, where rewrapping from a timecode-supportive format to Matroska carries the timecode value over as a tag.

In general I'm not in favor of using tags for data that may be needed
when remuxing, especially tied to a timestamp (yes, I mean timestamp).
If the first frame is damaged, does the timecode apply to the second
frame ? If the first frame timestamp wasn't 0, how do you know the
difference you have to apply ?

If you want something that will survive this scenario you have to tie
this value to the first timestamp. When remuxing, if timestamps are
altered of portions at the beginning of the file are removed, you need
to edit the first timecode. It may be better to store in the TrackInfo
with both the first timestamp and timecode values. It may not work
when stripping the audio (as Tobias pointed out). So it may even be
put in the SegmentInfo instead of the TrackInfo.

You may also set the value in Chapters (which are not tied to a
track). There's already a start timestamp. You could add the timecode
for this timestamp. That works for NLE sources as well, each
non-linear part would be a chapter with its timestamp translation. A
good NLE system would also tag this chapter with all kinds of data
about the source. (Matroska was also designed for this case in mind).

> 2. Using BlockAdditions as described above. This adds more overhead as each timecode value would be written, but would support non-continuous timecode. In FFmpeg, timecode is sometimes carried as side-data such as from the decklink device via https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/0946c0ec177dc48ef0677f890aa42d95e667c417. I’d be interested if the Block Addition Mapping described above could be used to carry the timecode side data from the decklink device (or other incoming stream with timecode side data) and store the result within the written Matroska BlockGroups.

That's the no-brainer option. It just works. It's just less optimal
that one value (or a few for NLE sources).

> Comments welcome.
>
> Thanks much,
> Dave Rice
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

-- 
Steve Lhomme
Matroska association Chairman

---

From: Jerome Martinez <[email-masked]>
Date: Sun, 3 Nov 2019 11:59:41 +0100

> seems 32 bits might be sufficient (saving 4 bytes). Maybe 40 if you

Advantage here is that we use a standardized and a lot used (in some 
domains) time code format, which can also be a direct dump from some 
sources, and it includes other data (color flag, binary group flags 
which can contain extra info etc), and could be exported to a SMPTE ST 
12 aware equipment without adaptation.

Other formats of time code, e.g. configuration (start time code, drop 
frame, frame rate...) in track header then a 32-bit number for each 
frame, could be implemented, but IMO it is just another format, 
independent. Both SMPTE ST 12 and a single number could be implemented, 
just different purposes/goals. Also sometimes the conversion from SMPTE 
ST 12 to other time code formats may be not lossless if you use just a 
number ("buggy" input with different drop frame info etc, differences 
would be lost).

In summary, both SMPTE ST 12 and just a number can work for time codes, 
but they have different goals, and one would not totally replace the 
other, just 2 "competing" formats.

> difference you have to apply ?

Advantage here is the simplicity of the implementation, and also 
informing bout what is already done in tools e.g. FFmpeg.
Not bullet proof but often good enough.

[...]

Jérôme

---

From: Jerome Martinez <[email-masked]>
Date: Sun, 3 Nov 2019 12:07:17 +0100

On 30/10/2019 17:28, Tobias Rapp wrote:
> [...] Thus I think it would be better to store timecode data as a 
> separate track, similar to how it could be done in MOV.

Both methods have pro and cons.
Lot of MOV time codes are linked to a video track, so side data may make 
sense here, as something we could transmux easily (with slightly less 
container overhead than a dedicated track).

So maybe we should permit both (SMPTE ST 12 in either side data or 
dedicated track).

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 3 Nov 2019 14:44:27 +0100

> other, just 2 "competing" formats.

Yes, IMO they are different timecode "codecs" (COder/DECoder). There
could also be string versions, I don't know if that's common.

That's also why I'd rather have a separate document for such "codecs".
It's the same with video/audio/spu codecs and chapter codecs should do
the same.

It all depends on what solution(s) we end up with. If there's
something in the SegmentInfo or TrackInfo then we have to define it in
the main document.

---

From: Tobias Rapp <[email-masked]>
Date: Tue, 5 Nov 2019 08:51:30 +0100

> put in the SegmentInfo instead of the TrackInfo.

It looks tempting to put timecode information inside the video track as 
they share the framerate as a common property. But in my opinion it 
would be cleaner to just copy that framerate info into the timecode 
value structure. For storing a single TC offset using an element in 
SegmentInfo sounds good.

> about the source. (Matroska was also designed for this case in mind).

Setting up a timestamp/timecode mapping via Chapters for non-continuous 
timecode situations looks like an even better solution than using a 
dedicated timecode track as this structure should represent the timeline 
for players.

Best regards,
Tobias

---

From: Dave Rice <[email-masked]>
Date: Thu, 21 Nov 2019 10:03:20 -0500

Hi all,

> On Nov 5, 2019, at 2:51 AM, Tobias Rapp <[email-masked]> wrote:
> 
> It looks tempting to put timecode information inside the video track as they share the framerate as a common property. But in my opinion it would be cleaner to just copy that framerate info into the timecode value structure. For storing a single TC offset using an element in SegmentInfo sounds good.

From the discussion, there seems to be three approaches for timecode in Matroska:

1. a single TC offset
I proposed having this as a metadata tag, but you’re suggestion to have the value in Segment Info is interesting. I think the advantage of having it as a metadata tag is that ffmpeg’s current handling of (ffmpeg -i timecode.mov output.mkv) would be retroactively valid, whereas a single offset in SegmentInfo would require implementation work.

2. frame side-data
This allow the timecode (and any other side data) to be stored with the frame in the Block structure.

3. a timecode track
I imagine this would be a track of Clusters where each block contains an encoded timecode value where the block structure associates the timecode to a timestamp.

Each of these have advantages and disadvantages in terms of simplicity, implementation, seekability, and resilience; however we do not necessarily have to select only one. All three approaches could be defined if we document how these options supersede one another.

I’m curious if other would consider it important to favor limiting the number of approaches listed above. I would prefer to proceed with refinement to options 1 and 2 concurrently. I’m less interested in option 3 since timecode is typically used to describe the contents of another track rather than serve as an independent piece of data.

> Setting up a timestamp/timecode mapping via Chapters for non-continuous timecode situations looks like an even better solution than using a dedicated timecode track as this structure should represent the timeline for players.

I’m uncertain how this would work, do you mean to use ordered chapters to place the content along a timeline according to the source timecode? This may not work if multiple frames in a track all share the same timecode value, such as when a tape is shuttled during a duplication process or contains resets within the timecode track.

[…]

Dave Rice

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 24 Nov 2019 17:43:44 +0100

> I proposed having this as a metadata tag, but you’re suggestion to have the value in Segment Info is interesting. I think the advantage of having it as a metadata tag is that ffmpeg’s current handl…

But usually tags are not considered/parse in most playback systems. If
you need information to playback things correctly (ie display the
proper TC) it should be either in Segment Info, Track Info or
Clusters. The Timecode may fall in the metadata category and may not
be included in there. Also using tags would only work if we adopt
exactly the solution already in use by ffmpeg. What tag does it write
exactly ? (name, kind of value, target for the tag)

> I’m curious if other would consider it important to favor limiting the number of approaches listed above. I would prefer to proceed with refinement to options 1 and 2 concurrently. I’m less interes…

That sounds like something we can discuss a No Time To Wait.

> I’m uncertain how this would work, do you mean to use ordered chapters to place the content along a timeline according to the source timecode? This may not work if multiple frames in a track all sh…

Not necessarily ordered-chapters. A chapter already has a mandatory
start time, it could have one Timecode to define what that start time
corresponds to (or more variants of Timecode for the same start time).
Technically that means you could have on Chapter per frame to match
it's time exactly with a timecode, that would be a dirty hack.

As for reset in tapes, After a reset you should use a different
Segment. If there are just gaps of time between continuous takes, you
can use on Chapter start/timecode after each discontinuity.

---

From: Dave Rice <[email-masked]>
Date: Mon, 25 Nov 2019 15:25:56 -0500

> On Nov 24, 2019, at 11:43 AM, Steve Lhomme <[email-masked]> wrote:
> 
> Clusters.

There aren’t that many players I can think of that display timecode on playback. QuickTime 7 used to but is deprecated. QuickTime X doesn’t show timecode nor VLC. ffplay can do it by some filter work that pulls from the metadata.

> The Timecode may fall in the metadata category and may not

The name is TIMECODE and it writes the timecode value as a string. Like TIMECODE=01:23:45;12
You can make an example of this via:
ffmpeg -f lavfi -i testsrc -metadata timecode='01:23:45;12' -t 1 ffmpeg_timecode.mkv

> That sounds like something we can discuss a No Time To Wait.

See you next week :)

> it's time exactly with a timecode, that would be a dirty hack.

Please no. :-O

> can use on Chapter start/timecode after each discontinuity.

I think this is hacky as well. It seems like a lot more to ask a recording tool to reset to a new Segment because the timecode skipped.
Dave

---

From: Tobias Rapp <[email-masked]>
Date: Tue, 26 Nov 2019 09:30:33 +0100

> exactly ? (name, kind of value, target for the tag)

Adding a tag for storing timecode offset is a simple solution and 
satisfies the most basic requirements. The current list [1] seems to 
already include other technical tags like "FPS" and "REPLAYGAIN_PEAK" so 
it should be fine.

I would not use a framecount for storing sub-seconds, though, but a 
HH:MM:SS.MSS pattern like used for date-time (without the date part). 
This would be valid for audio and video files. An application can render 
the value in HH:MM:SS:FF syntax for user display, if desired.

> can use on Chapter start/timecode after each discontinuity.

The latter option seems the most practical. As far as I understand from 
[2] there can be multiple chapters applied to one segment by using 
multiple EditionEntry elements. That might be useful because we can have 
one chapter structure for track labeling "scene one", "scene two", ... 
and another chapter structure for timecodes (if necessary).

I assume timecodes would get an own "ChapterTimecodeOffset" element, 
similar to existing ChapterTimeStart/-End. Or would you define a 
"ChapProcessCodecID" for timecodes explicitly? Also maybe we want to 
store some more information together with the plain time value. Possibly 
frame-rate and drop-frame flag (for rendering the time in HH:MM:SS[:;]FF 
syntax) should be put either as optional elements on EditionEntry level 
or into ChapProcessPrivate.

In theory there could be multiple timecode sources (VITC, LTC, etc) but 
in practice it seems hardware players just output the most precise one 
currently available so storing the source name in Matroska would not be 
considered relevant by me.

Best regards,
Tobias

Links:
[1] https://www.ietf.org/id/draft-ietf-cellar-tags-03.html
[2] 
https://datatracker.ietf.org/doc/html/draft-ietf-cellar-matroska-04#section-9.3.7

---

From: Tobias Rapp <[email-masked]>
Date: Tue, 26 Nov 2019 09:59:42 +0100

On 25.11.2019 21:25, Dave Rice wrote:
> 
>> On Nov 24, 2019, at 11:43 AM, Steve Lhomme <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
> I think this is hacky as well. It seems like a lot more to ask a 
> recording tool to reset to a new Segment because the timecode skipped.
> Dave

Just out of curiosity: How many timecode jumps (excluding jumps due to 
corrupt signal) do you experience when recording? From my knowledge 
there is usually one timecode at the start of a program, and about 1-4 
different programs on one carrier tape.

Best regards,
Tobias

---

From: Dave Rice <[email-masked]>
Date: Tue, 26 Nov 2019 09:02:27 -0500

> On Nov 26, 2019, at 3:59 AM, Tobias Rapp <[email-masked]> wrote:
> 
>>> On Nov 24, 2019, at 11:43 AM, Steve Lhomme <[email-masked] <mailto:[email-masked]>> wrote:
>>> 
> Just out of curiosity: How many timecode jumps (excluding jumps due to corrupt signal) do you experience when recording?

A non-sequential timecode could happen on every frame. For instance a tape to tape duplication where the source tape was left in pause mode while the other tape was recording would leave a long series of frames with identical timecode values.

> From my knowledge there is usually one timecode at the start of a program, and about 1-4 different programs on one carrier tape.
> 
> Best regards,
> Tobias
>