[Idea] The importance of checksum

[spoRv]

I never thought about the importance of the checksum, until today... it is not only because one can check if the downloaded file is indeed error-free, but also to know if a given project is untouched; I mean, I'm aware that some of my projects are remuxed, discarding audio and/or subs tracks, with other ones remuxed in their place, but, would the video be left untouched?

At the end, the hash string is like a signature of your work; it would help to know if a given file is the original, untouched one, or at least if one or more muxed tracks are.

So, from now on, I will include hash strings for the main m2ts file, as well as the h264 demuxed stream (and eventually audio and/or subs tracks, if they are restored/fixed as well); I have decided to use MD-5, SHA-1 and SHA3-512, to have more checksum options; also file size and mediainfo would be included as well.

Example of a info.txt file that would be included:

Code:

CHECKSUM

complete project (muxed using tsMuxeR GUI 2.6.12)

    File: Harry_Potter_and_the_Half-Blood_Prince_[Fundamental_Collection]\BDMV\STREAM\00000.m2ts
    Size: 43.779.772.416 bytes
     MD5: c57f7effaabba5fbf581f84ad90a4aa1
   SHA-1: 19d1e0b92ac425b0270a260890b7f4caaa2ff1ab
SHA3-512: e95e1e78861cf22f2cf746dfd2a289c728c31867e67c4a3b956ad1a028551fac6ef4d7bcec23003b9538af885876249eee5abb5923b822301854be96b296bf49


raw h264 stream (demuxed using tsMuxeR GUI 2.6.12, all options unchecked)

    File: Harry_Potter_and_the_Half-Blood_Prince_[Fundamental_Collection]\BDMV\STREAM\00000.track_4113.264
    Size: 34.613.808.135 bytes
     MD5: 13d4d90b865295b16bdb7bb277585fdc
   SHA-1: 6fd33098ed8a364ffc31ad2142b0ce81b3e3f8fd
SHA3-512: 68b1de89ce182f59f749e1ad2971c61d8efe316e41eaa8f18eb17b621656df0d162d8df257d78fd51d959adc707b3e0681ba6bb981b62306f6cdf50bc4ec29aa

################################################################################################

MEDIAINFO

General
ID                                       : 1 (0x1)
Complete name                            : Harry_Potter_and_the_Half-Blood_Prince_[Fundamental_Collection]\BDMV\STREAM\00000.m2ts
Format                                   : BDAV
Format/Info                              : Blu-ray Video
File size                                : 40.8 GiB
Duration                                 : 2 h 33 min
Overall bit rate mode                    : Variable
Overall bit rate                         : 38.0 Mb/s
Maximum Overall bit rate                 : 35.5 Mb/s

Video
ID                                       : 4113 (0x1011)
Menu ID                                  : 1 (0x1)
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4.1
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 4 frames
Codec ID                                 : 27
Duration                                 : 2 h 33 min
Bit rate mode                            : Variable
Maximum bit rate                         : 40.0 Mb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio #1
ID                                       : 4352 (0x1100)
Menu ID                                  : 1 (0x1)
Format                                   : DTS
Format/Info                              : Digital Theater Systems
Format profile                           : MA / Core
Mode                                     : 16
Format settings, Endianness              : Big
Muxing mode                              : Stream extension
Codec ID                                 : 134
Duration                                 : 2 h 33 min
Bit rate mode                            : Variable / Constant
Bit rate                                 : Unknown / 1 509 kb/s
Channel(s)                               : 6 channels
Channel positions                        : Front: L C R, Side: L R, LFE
Sampling rate                            : 48.0 kHz
Frame rate                               : 93.750 FPS (512 spf)
Bit depth                                : 16 bits
Compression mode                         : Lossless / Lossy

Audio #2
ID                                       : 4353 (0x1101)
Menu ID                                  : 1 (0x1)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Mode extension                           : CM (complete main)
Format settings, Endianness              : Big
Codec ID                                 : 129
Duration                                 : 2 h 33 min
Bit rate mode                            : Constant
Bit rate                                 : 640 kb/s
Channel(s)                               : 6 channels
Channel positions                        : Front: L C R, Side: L R, LFE
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 spf)
Bit depth                                : 16 bits
Compression mode                         : Lossy
Stream size                              : 703 MiB (2%)

Audio #3
ID                                       : 4354 (0x1102)
Menu ID                                  : 1 (0x1)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Mode extension                           : CM (complete main)
Format settings, Endianness              : Big
Codec ID                                 : 129
Duration                                 : 2 h 33 min
Bit rate mode                            : Constant
Bit rate                                 : 448 kb/s
Channel(s)                               : 6 channels
Channel positions                        : Front: L C R, Side: L R, LFE
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 spf)
Bit depth                                : 16 bits
Compression mode                         : Lossy
Stream size                              : 492 MiB (1%)

Audio #4
ID                                       : 4355 (0x1103)
Menu ID                                  : 1 (0x1)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Mode extension                           : CM (complete main)
Format settings, Endianness              : Big
Codec ID                                 : 129
Duration                                 : 2 h 33 min
Bit rate mode                            : Constant
Bit rate                                 : 448 kb/s
Channel(s)                               : 6 channels
Channel positions                        : Front: L C R, Side: L R, LFE
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 spf)
Bit depth                                : 16 bits
Compression mode                         : Lossy
Stream size                              : 492 MiB (1%)

Audio #5
ID                                       : 4356 (0x1104)
Menu ID                                  : 1 (0x1)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Mode extension                           : CM (complete main)
Format settings, Endianness              : Big
Codec ID                                 : 129
Duration                                 : 2 h 33 min
Bit rate mode                            : Constant
Bit rate                                 : 448 kb/s
Channel(s)                               : 6 channels
Channel positions                        : Front: L C R, Side: L R, LFE
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 spf)
Bit depth                                : 16 bits
Compression mode                         : Lossy
Stream size                              : 492 MiB (1%)

Audio #6
ID                                       : 4357 (0x1105)
Menu ID                                  : 1 (0x1)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Mode extension                           : CM (complete main)
Format settings, Endianness              : Big
Codec ID                                 : 129
Duration                                 : 2 h 33 min
Bit rate mode                            : Constant
Bit rate                                 : 448 kb/s
Channel(s)                               : 6 channels
Channel positions                        : Front: L C R, Side: L R, LFE
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 spf)
Bit depth                                : 16 bits
Compression mode                         : Lossy
Stream size                              : 493 MiB (1%)

Audio #7
ID                                       : 4358 (0x1106)
Menu ID                                  : 1 (0x1)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Mode extension                           : CM (complete main)
Format settings, Endianness              : Big
Codec ID                                 : 129
Duration                                 : 2 h 33 min
Bit rate mode                            : Constant
Bit rate                                 : 448 kb/s
Channel(s)                               : 6 channels
Channel positions                        : Front: L C R, Side: L R, LFE
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 spf)
Bit depth                                : 16 bits
Compression mode                         : Lossy
Stream size                              : 492 MiB (1%)

Audio #8
ID                                       : 4359 (0x1107)
Menu ID                                  : 1 (0x1)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Mode extension                           : CM (complete main)
Format settings, Endianness              : Big
Codec ID                                 : 129
Duration                                 : 2 h 33 min
Bit rate mode                            : Constant
Bit rate                                 : 448 kb/s
Channel(s)                               : 6 channels
Channel positions                        : Front: L C R, Side: L R, LFE
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 spf)
Bit depth                                : 16 bits
Compression mode                         : Lossy
Stream size                              : 492 MiB (1%)

Audio #9
ID                                       : 4360 (0x1108)
Menu ID                                  : 1 (0x1)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Format profile                           : Dolby Digital
Mode extension                           : CM (complete main)
Format settings, Endianness              : Big
Codec ID                                 : 129
Duration                                 : 2 h 33 min
Bit rate mode                            : Constant
Bit rate                                 : 192 kb/s
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 spf)
Bit depth                                : 16 bits
Compression mode                         : Lossy
Stream size                              : 211 MiB (1%)

Text #1
ID                                       : 4608 (0x1200)
Menu ID                                  : 1 (0x1)
Format                                   : PGS
Codec ID                                 : 144

Text #2
ID                                       : 4609 (0x1201)
Menu ID                                  : 1 (0x1)
Format                                   : PGS
Codec ID                                 : 144

Text #3
ID                                       : 4610 (0x1202)
Menu ID                                  : 1 (0x1)
Format                                   : PGS
Codec ID                                 : 144

Text #4
ID                                       : 4611 (0x1203)
Menu ID                                  : 1 (0x1)
Format                                   : PGS
Codec ID                                 : 144

Text #5
ID                                       : 4612 (0x1204)
Menu ID                                  : 1 (0x1)
Format                                   : PGS
Codec ID                                 : 144

Text #6
ID                                       : 4613 (0x1205)
Menu ID                                  : 1 (0x1)
Format                                   : PGS
Codec ID                                 : 144

Text #7
ID                                       : 4614 (0x1206)
Menu ID                                  : 1 (0x1)
Format                                   : PGS
Codec ID                                 : 144

Text #8
ID                                       : 4615 (0x1207)
Menu ID                                  : 1 (0x1)
Format                                   : PGS
Codec ID                                 : 144

Text #9
ID                                       : 4616 (0x1208)
Menu ID                                  : 1 (0x1)
Format                                   : PGS
Codec ID                                 : 144

Opinions?

Edit: to test its effectiveness, first I demuxed the m2ts file, checked the raw h264, remux this raw stream with the other tracks found in the m2ts, then demuxed it again from this second m2ts, and the raw h264 stream is identical (as expected).

[CSchmidlapp]

I know very little about this subject.
Should and can these be produced when compressing the final project for delivery as multiple .rars?

[Feallan]

RAR format has its own checksum control, but it wouldn't hurt to include checksums of the main video file

[deleted user]

Good idea for the main files and this is common practice for games (as errors there tend to be much more significant).

Though I doubt it's perfect for the elementary streams such as the h264 stream you suggested, because the slightest change in the most miniscule metadata will completely change the hash, including these SEI/whatever values that seem to be a kind of frame-based metadata. Programs like tsmuxerGUI allow you to rewrite those and stuff like that. So, in short, you may get false negatives. In those cases I think a comparison of bitrates and such should give the right answer most of the time. Also, most people seem to properly credit their source and what they did with it (aside from YIFI style stuff).

P.S. slightly unrelated but have you heard of so-called PAR files? That stuff is witchcraft, check it out.

[Chewtobacca]

Opinions?

I agree. For sizable uploads, I tend to create par2 files, which are capable of repairing archives.  For quite a while now, I have kept the archives that I download and created par2 files for them as soon I have ensured that they extract without error.  This also helps when transferring files from one hard drive to another, because it's possible for them to become corrupted.

The first time that I realized that one of my projects was corrupt was not fun.  Fortunately, a back-up on another drive remained error free.  Since then, I have archived and created recovery files for everything. It's not an absolute guarantee -- nothing is.  But it provides a measure of reassurance.

Also useful is eac3to's -check option.

Code:

eac3to input -check

It goes through the process of demuxing and flags up any errors without actually writing any files to the drive. I customarily use it before raring and uploading a video, to ensure its integrity.

[spoRv]

Well, the fact to add checksum to fan projects is more related to know if someone has actually "touched" the video somehow; of course, a download check is always a welcome benefit. PAR2 is important, but useless when we talk about torrents... indeed, if I can choose, I'd go with RARed version accompanied by PAR2 files, but it seems that torrent is the best way to go - that would take us into a completely different topic about project release modes; if someone would start it...

@TomArrow: you are right, but I did the test, as written in the first post; I repeat myself here:

• encoding the final project into a .264 raw stream (it can be HEVC, VC-1, MPEG-2, whatever of course)
  
• mux this stream (let's call it FIRST), to the final m2ts file (it can be mkv, but I stick with it to make BD); I flag out "Continually insert SPS/PPS" and choose "Do not change SEI and VUI data"
  
• demux the .264 stream (let's call it SECOND) - flag out "Continually insert SPS/PPS" and choose "Do not change SEI and VUI data"
  
• if I compare checksum for FIRST and SECOND, they are identical
  

so, let's say I want to apply the SPS/PPS and/or SEI/VUI... let's start again:

• encoding the final project into a .264 raw stream
  
• mux this stream (let's call it FIRST), to the final m2ts file; I flag in "Continually insert SPS/PPS" and/or choose "Always rebuild SEI and VUI data" (it can be "Insert SEI and VUI data if absent", but I want to be sure they are rebuilt)
  
• demux the .264 stream (let's call it SECOND) - flag out "Continually insert SPS/PPS" and choose "Do not change SEI and VUI data"
  
• if I compare checksum for FIRST and SECOND, they are different
  
• then, I mux the SECOND .264 stream in a .m2ts file, using the same setting of the previous mux
  
• demux the .264 stream (let's call it THIRD) - flag out "Continually insert SPS/PPS" and choose "Do not change SEI and VUI data"
  
• if I compare checksum for SECOND and THIRD, they are identical
  

Long story short; if no SPS/PPS and/or SAI/VUI are chosen, the .264 raw stream is left untouched.

AFAIR, "Insert SEI and VUI data if absent" and "Continually insert SPS/PPS" are reccomended for blu-ray compliance.

[deleted user]

But from how I understood you, you wanted to make sure that if somebody remuxes your thing, it's still possible to check it's the real stream. And that would be compromised of the remuxer changed anything about the metadata at all, these SEI/VUI/SPS/PPS or whatever are just one example. He could also decide to ditch the encoding parameters or all kinds of other stuff. Presume he doesn't know 100% what he's doing, that's more than likely to happen.

Either way, I feel like the best way to check identical streams usually is to compare encoding parameters, stream sizes, standard meta data stuff.

[spoRv]

I *think* that, if somebody takes the .264 stream from my .m2ts released file, and remux it using tsmuxer, then if demuxed it *should* have one identical checksum - I mean, one without SEI/VUI/SPS/PPS and one with them; the same *should* be also if muxed within .mkv

But, at the end, my goal is this: if one republish my project wherever (torrent, direct download etc.) claiming the files are untouched, the checksum MUST be the same; if one remux it, without touching the video, the checksum *SHOULD* be the same (according to different values due to SEI/VUI/SPS/PPS using)...

Oh, well, I have an headache now!

[deleted user]

I understand for republishing the whole project, it's a nice way of verifying. For the .264 I think it's pointless because there's too many variables and, frankly, who will go through all this pain (creating different .h264 versions from some downloaded 30GB file) just to find out whether it's the same bit-identical stream? Hell, I'm crazy and even I wouldn't go through such a pain, because it's a lot of time/effort without obvious benefit.

Besides, I wanna add another aspect: Torrent files.

You distribute your files via torrents. Now guess how torrent clients make sure you have the right file? That's right, with a checksum. Your project torrents (assuming they weren't created from already corrupted files) are perfectly suitable for verifying the bit-perfect identity of your project files. Even someone who doesn't want to download it where you uploaded it could just take the torrent file, load it into his client, choose the right folder, click "Force recheck", wait a bit, and if it checks out to 100%, the files are bit-perfect. Someone who downloads a file from you via torrent pretty much already has a guaranteed perfect copy with no need for further checks. If he wants to be extra diligent, he can click "Force Recheck" another time after downloading.

[spoRv]

I know about the fact (almost) nobody would take time and effort to modify a project, but you know, I'm a paranoid dinosaur, so...

said that, I'm aware of torrent checksum - how can it work without it?

Back to h264 checksum: I wonder if, after various instances of mux/demux with mixed settings of SEI/VUI and/or SPS/PPS, at the end there would be just two versions (one with, and one without) or a plethora of versions... if the former is the case, then my idea is not that bad - just check if checksum is either equal to A or B. But too lazy to make these tests now; would you mind to do them, Tom?