Deriving a method for aligning footage

[Synnove]

For projects that have limited sources available, this method could be invaluable.

Say you've a few different transfers of a film (this is all assuming they are frame to frame identical/have exactly the same editing):

-Source 1 has a lot of detail but also has crushed shadows and clipped highlights
-Source 2 has somewhat less detail but has the shadow and highlight information intact
-Source 3 has the proper color timing

If we could take elements from all three sources, we'd have an ideal restoration! The only problem is that since these are film scans, the images won't exactly line up, and the geometric differences will likely be non linear and variable due to gate weave, different scanners used, different reels of film, etc...

So the hard part is aligning, or registering, the footage. I'm trying to develop a method to accomplish this, but it's proving difficult. Hopefully in this thread we can work on establishing a work-flow, as I feel this could be an essential restoration tool.

So the methods I'm aware of are:

-Manual alignment via grid deformation
For example in Nuke; take two clips, merge them, blend in difference mode, and try your best to align them via the grid. Unfortunately not only is this long and tedium, but it doesn't account for temporally variable frame geometry (your alignment might only hold for a couple seconds).

-Derive motion vectors from two frames and warp
https://www.reddit.com/r/vfx/comments/3f...t_in_nuke/
Another Nuke example; "set-up of VectorGenerator where you make it think your two different plates are a pair of consecutive frames. You can then pull the motion vectors to get a uv warp between the plates, and the IDistort node to reapply the warp if needed." I tried to attempt this method but it seemed to fail.

-F_Align in Nuke
This will take two clips and align them frame by frame. Unfortunately, the results can be very unpredictable at times, and the alignment is limited to essentially corner pin transforms; it can't deform the interior of the frame.

-Motion Compensation via AVISYNTH
https://forum.doom9.org/showthread.php?t=154739
It seems, per the thread above, it could be done a couple ways via interleaving your clips and applying MVTools plugin. People in that thread have reported success but I've not been able to replicate (unfortunately I don't know my way around avisynth very well, and am receiving a "not a clip" error). Once I'm able to post on Doom9 I'll inquire further.

-Image Registration
This technique seems to be most used in the medical field for aligning scans. The most popular software seems to be elastix, but I've also seen stuff written in Matlab that does similar things. It's been said this technique can generate high quality results. Unfortunately, processing time is a concern; the transforms themselves are quick but the generation of them takes a bit of time. This means that you'd have to either use the same transform for an entire transfer, or you'd have to change between several at different points in the footage. It also requires a bit of technical know-how as the parameter file needs to be setup correctly for the thing to work.

To show the possibilities with this work, here are the results of a Nuke composition that was done using grid warp.

https://image.ibb.co/m2Zv9b/1_TMP_BLU_REVEAL.png
https://image.ibb.co/na30aG/2_TMP_HDTV_REVEAL.png
https://image.ibb.co/eyLDvG/3_TMP_COMP_REVEAL.png

The highlights and shadows of the HDTV capture were denoised and then added to the midtones of the Blu-Ray release. The result was then put through a MatchGrade node which brought it in line with the HDTV cap's color grade, and then a layer of film grain was added over the top. It's important to keep in mind that the alignment was far from perfect (especially near the edges); if the alignment were automated and done to a high enough quality, the result would look even clearer, crisper than what I've posted above.

[PDB]

Thanks for writing this up. I'm very interested in getting a solution also having tried similar techniques without much success.

[spoRv]

Sadly I deleted the project which used both warpresize and reformer, but the result was very good indeed.

Here I want to show what these avisynth plugins are able to do:

warpresize (http://avisynth.nl/index.php/SimpleResize)

original


result

(the numbers across the bottom have been added for illustration purposes)

---

reformer (http://www.avisynth.nl/users/vcmohan/Ref...ormer.html)

original


result

(here the used plugin is reform; principle is the same, but reformer is better)

[Synnove]

That does indeed seem to provide good results but it's basically the same thing as grid deform in Nuke; it entails a lot of manual work. Plus, if the deformations aren't constant through the footage, it will require even more manual attention.

For those who know their way around AVISYNTH, do you've any idea why this script keeps returning a "not a clip" error? This is a method that someone has said worked for them, so I feel it's worth investigation.

clip2=AVISOURCE("C:\Users\Synnove\Videos\TMP_INTERLEAVE_REVEAL.avi")
clip1=ConvertToYV16(clip2)
est = depanestimate(clip1, range=1,pixaspect=1.0,zoommax=1,trust=0)
dep = depaninterleave(clip1, est,pixaspect=1.0,prev=0,next=1,subpixel=2,mirror=0)

The source is 1920x802 RGB avi.

[spoRv]

For those who know their way around AVISYNTH, do you've any idea why this script keeps returning a "not a clip" error? This is a method that someone has said worked for them, so I feel it's worth investigation.

clip2=AVISOURCE("C:\Users\Synnove\Videos\TMP_INTERLEAVE_REVEAL.avi")
clip1=ConvertToYV16(clip2)
est = depanestimate(clip1, range=1,pixaspect=1.0,zoommax=1,trust=0)
dep = depaninterleave(clip1, est,pixaspect=1.0,prev=0,next=1,subpixel=2,mirror=0)

Because the result is not a video clip; just add

return dep

at the end, to call the depaninterleave - even if it shows only one frame in my system, and the rest are blank, don't know why... I'd ask in the doom9 forum, there are real avisynth gurus there, that helped me a lot!

[Synnove]

Ahh I see... always the little things . So doing that results an alignment comparable to what you get with Nuke's F_Align, though with the addition of some odd artifacts.

I'm definitely going to post on Doom9 when my account is allowed to (they've a five day waiting period for new registrations); I've a feeling improper setup of mvtools on my part is causing the artifacts with that method.

[spoRv]

That does indeed seem to provide good results but it's basically the same thing as grid deform in Nuke; it entails a lot of manual work. Plus, if the deformations aren't constant through the footage, it will require even more manual attention.

I'm an old dinosaur, and still using avisynth for everything, video related, and don't know Nuke.

What I can say is once I used those plugins to adjust a camrip, and I used only two settings for the whole movie, so *I guess* a professional made film scan *should* be constant, I hope!

Keep us updated with your progresses.

[Chewtobacca]

I'm an old dinosaur, and still using avisynth for everything, video related...