Subject: [Cellar] Any use for FLAC decoding testfiles?

From: Martijn van Beurden <[email-masked]>
Date: Tue, 5 Oct 2021 10:05:47 +0200

Hi all,

A few months ago I made a FLAC decoder testbench: a set of FLAC files
specifically designed to test certain aspects of the decoding process,
including rice partition escape codes, non-standard blocksizes, a
variable blocksize, very small and very large predictor coefficient
precision, wasted bits and extremely large metadata blocks.

Currently, it is a zip file with 59 FLAC files and a README. All FLAC
files contain audio that is licensed under some Creative Commons
license. More information can be found here:
https://wiki.hydrogenaud.io/index.php?title=FLAC_decoder_testbench

Would such a testbench be of any use to this working group?

Kind regards,

Martijn van Beurden

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 10 Oct 2021 08:23:51 +0200

> Would such a testbench be of any use to this working group?

I think so. In particular all the files that cause 
crash/freeze/reboot(!) should help define the security issues that need 
to be looked at when handling a FLAC bitstream.

BTW, what did you use to create those bitstream ? How do you check they 
are actually valid ?

> Kind regards,
> 
> Martijn van Beurden
> 
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar
>

---

From: Martijn van Beurden <[email-masked]>
Date: Sun, 10 Oct 2021 09:44:54 +0200

> BTW, what did you use to create those bitstream ? How do you check they

Files 1-12, 14-15, 17-23, 27-31, 33-44 and 47-59 were made with an
unmodified flac compile simply using specific command line switches.
For file 13, ffmpeg with a specific command line switch unavailable in
libFLAC was used. For file 16 and 32, a modified flac compile was used
where a deprecated encoder setting was reenabled. For file 24-27
various versions of Flake were used as described in the README. Files
45 and 46 had the total number of samples c.q. min-max framesize set
to zero with a hex editor. The extreme metadata of files 48-55 was
added with unmodified metaflac.

All files decode without problem with libFLAC, ffmpeg returns errors
on file 27 (which is made according to an old version of the FLAC
specification) and an error on 59 as it doesn't know image/avif, so
they seem valid to me.

Kind regards,

Martijn van Beurden

---

From: Jerome Martinez <[email-masked]>
Date: Sun, 10 Oct 2021 22:34:15 +0200

> Would such a testbench be of any use to this working group?

In my opinion it is actually part of the work about specs (for Matroska 
we have https://github.com/preforma/groundtruth/tree/master/matroska and 
moving these files on a CELLAR repo is in my todo-list; we should do 
same with FFV1).

Would it be fine for everyone if we copy these files to a repo e.g. 
"flac-testfiles" (and for Matroska a "matroska-testfiles" repo) at 
https://github.com/ietf-wg-cellar/ ? So we have a central place for them 
and also history management.

Jérôme

---

From: Martijn van Beurden <[email-masked]>
Date: Wed, 13 Oct 2021 17:23:42 +0200

> and also history management.

I have no objections. I hope the fact that these files contain
Creative Commons licensed music (most of them BY-NC-SA) from a few
different artists is not a problem? Some files in the testbench cannot
be recreated with test tones, as some tested features really need a
varying input to be used.

---

From: Martijn van Beurden <[email-masked]>
Date: Wed, 13 Oct 2021 20:35:26 +0200

> and also history management.

Something I forgot to mention: this testbench specifically consists of
FLAC subset-compliant files. So, these files are not only valid, but
also comply with the additional subset limitations as mentioned in the
flac specification. If you put these in a repository, it is perhaps a
good idea to place it in a 'subset' subdirectory, to differentiate
them from security specific test files that might be interesting to
place there as well.

---

From: Jerome Martinez <[email-masked]>
Date: Wed, 13 Oct 2021 22:34:15 +0200

On 13/10/2021 17:23, Martijn van Beurden wrote:
> I have no objections.

So https://github.com/ietf-wg-cellar/flac-testfiles !

I'll send some PRs for README, LICENSE etc as well as other tips you 
indicated e.g. FFmpeg compatibility.

The test results page would also be great to move there because it says 
a lot about compatibility, but GitHub Markdown is not so good 
(colors...), I'll check how we can do there, but if you have ideas let 
me know (putting the html page?).

A remark: version of players (HW or SW) are missing and it would be good 
to have them.

>   I hope the fact that these files contain

IMO free licenses (input based on free content) would have been better 
but I don't think that it is worth it to change that now, it should not 
be blocking as it is "only" test files. What you created is already 
great and tests must have taken a lot of time! But maybe for the next 
files your input could be some free content.

Jérôme

---

From: Florian Weimer <[email-masked]>
Date: Sat, 16 Oct 2021 08:14:38 +0200

* Martijn van Beurden:

> different artists is not a problem?

BY-NC-SA licensing precludes use of these files in test suites for
most users, I think.  Content creators have wildly differing opinions
on the meaning of NC licensing, so commercial users tend to avoid
BY-NC-*-licensed content.

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 17 Oct 2021 09:04:20 +0200

> and also history management.

OK for the Matroska repo as well. We may merge it with 
https://github.com/Matroska-Org/matroska-test-files

---

From: Michael Richardson <[email-masked]>
Date: Sun, 17 Oct 2021 16:34:15 -0400

> In my opinion it is actually part of the work about specs (for Matroska we
    > have https://github.com/preforma/groundtruth/tree/master/matroska and
    > moving these files on a CELLAR repo is in my todo-list; we should do same
    > with FFV1).
    >
    > Would it be fine for everyone if we copy these files to a repo
    > e.g. "flac-testfiles" (and for Matroska a "matroska-testfiles" repo) at

That would be entirely fine!

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: Steve Lhomme <[email-masked]>
Date: Tue, 26 Oct 2021 09:10:23 +0200

>      > https://github.com/Matroska-Org/matroska-test-files

Moved to https://github.com/ietf-wg-cellar/matroska-test-files