AutoOverlay (PKA AutoAlign) discussion

[deleted user]

Here's my code:

Code:

config=OverlayConfig(subpixel=2,minOverlayArea=80,minSourceArea=80,scaleBase=1.1,minSampleArea=200,requiredSampleArea=500,acceptableDiff=10)
OverlayEngine(srcdehalo2.ConvertToY8(), src2.ConvertToY8(), configs=config,editor=false,maxDiff=10,backwardFrames=0,forwardFrames=0)
OverlayRender(srcdehalo2.ConvertToYV24(), src2.ConvertToYV24(), debug=true, noise=0,\
upsize="Spline64Resize", mode=3, gradient=10,coloradjust=2,width=1920,height=1080,opacity=0.5) 
ConvertToRGB24()

scaleBase=1.1 Such value produces a very long sequence of downsized sample images. Default is good in most cases. In all that I had.
minSampleArea=200,requiredSampleArea=500 - too small values. To understand - resize your clips to the such area (width*height) and see that all details are lost.
acceptableDiff=10 - this parameter is used by plugin only if you have a sequence of OverlayConfigs: OverlayConfig(...) + OverlayConfig(...). With one config it takes no effect.
With backwardFrames=0 other prediction params such as maxDiff, forwardFrames take no effect.

To prevent misaligment try OverlayConfig params as follows:
minSampleArea=2000,requiredSampleArea=5000, correction=2, branches=3, downsize="Spline36Resize", upsize="Spline36Resize"
With correct minSourceArea, minOverlayArea, aspecRatios and angles of course. If you have unstablized different transfers then they have different in-frame aspect ratio in most cases. Check it in difference mode.

Thank you! I will try those tips asap. I just have one question. Can you explain what scaleBase does exactly? This 1.1 value once gave me good results, that's why I kept it. I believed that it decided how much one source can be scaled to fit another. Can you tell me what it does exactly?

[deleted user]

Means that the masters are slightly different and you can't reuse alignment frame-to-frame, because each master has different micro-movement from frame to frame, probably due to the way it's been stabilized (or not stabilized) or simply due to a different transfer where the film wobbled slightly differently. Make sense?

Makes sense.

Do you want to make an AAP-AR version, or just use both to improve overall quality?

The latter. From the code I've used, it would result in an AAP-AR version, but the sources are so very similar that effectively it simply results in both improving each other.

[spoRv]

So I'd use mode=1 or mode=2, as mode=3 would scale both... stuck with the best source, and let the other be scaled; you should get a better quality.

[deleted user]

Quality is already pretty fine. Plus, in a handful of situations I seem to be getting a little bit more image this way, mostly horizontally. I prefer it like this. But good point in general.

[random.next]

Thank you! I will try those tips asap. I just have one question. Can you explain what scaleBase does exactly? This 1.1 value once gave me good results, that's why I kept it. I believed that it decided how much one source can be scaled to fit another. Can you tell me what it does exactly?

Will try, it's difficult for me to explain complicated things in english 
From desc:

scaleBase (default 1.5) – base to calculate downscale coefficient by formula coef=scaleBase^(1 - (maxStep - currentStep))
Auto-align process for each configuration includes multiple scaling stages. At the first stage all possible combinations of align values are tested in the lowest resolution. The count of best align value sets passed to the next stage is given by OverlayConfig.branches parameter. At the next stages best sets from previous are corrected. Note that, simplified, one pixel in low resolution includes four from high. OverlayConfig contains correction parameter to set correction limit for all directions and sizes in pixels.

Look at the formula above. ^ is the pow. If you have large scaleBase then it will produce larger coef. Coef is the divider for width and height of input images to downscale them.
First the engine determines how much scaling steps we need to reach minSampleArea at source image. There is requiredSampleArea also. The last one is used if input frame have a large number of small details that may be lost on downscaling. 

maxSampleDifference (default 5) – max allowed DIFF between downscaled source clip at previous and current scale iteration. If it is too high then previous iteration will not be processed.

So larger scaleBase is used to reduce scale step count, lower is to increase. The large number of scaling steps may provide better alignment but decrease perfomance. Instead of scaleBase you may use correction and branches params.

[deleted user]

Thanks, random.next, now it makes a little more sense in my head.

By the way guys, I have come up with an idea to align scenes that are a little tough otherwise:

Code:

src2 = src2.BilinearResize(1920,1080)
src2edge=src2.converttoyv12().mt_edge(mode="min/max").converttoyv24()
srcdehalo2edge = srcdehalo2.converttoyv12().mt_edge(mode="min/max").converttoyv24()

config=OverlayConfig(subpixel=2,minOverlayArea=95,minSourceArea=95,scaleBase=1.1,minSampleArea=100,requiredSampleArea=200,acceptableDiff=15)
OverlayEngine(srcdehalo2edge.ConvertToY8(), src2edge.ConvertToY8(), configs=config,editor=false,maxDiff=20,backwardFrames=0)
OverlayRender(srcdehalo2.ConvertToYV24(), src2.ConvertToYV24(), debug=false, noise=0,\
upsize="Spline64Resize", mode=3, gradient=10,coloradjust=2,width=1920,height=1080,opacity=0.5)
ConvertToRGB24()

The trick is, you feed the OverlayEngine not with the video itself, but with an edge-detection run on the original video. From what I've tried so far, it works surprisingly well. Sadly the mt_edge filter is a bit ... problematic and produces weird artifacts, including some green cast over the entire image ... buggy thing.

That said ... once there is some amount of sharp detail in the image, it very quickly "snaps" on and then doesn't let go anymore.

You can see that this method can be tweaked and improved a lot. Currently the mt_edge is the best I could find and it only does a kind of "tresholding". If you find a more conservative filter that doesn't show "on or off" but instead a continuum of edge "intensity", then this could work very well even for less sharp details. In either case, mt_edge needs to go, it's ridiculously bugged up.

Maybe it can even be combined ... making an Overlay of the original image and the edge detection ... or maybe putting them in different channels, if OverlayEngine ever supports that. Or maybe using the detected edges as a "multiply" overlay or simply an alpha mask to mask away all unimportant parts, making overlay based purely on significant details in the image, not on color areas.

Etc. etc.

[spoRv]

Using this plugin since few weeks, and slowly started to know it better; it's almost perfect - if you use the right settings, of course; the only shots that (sometimes) fail are the ones totally white or black (or a shot with even colors/features), because it has no reference points, so it's not a real fault, as it would be impossible to align two of those kinds.

So, as I wrote before, I give it 99% - and the missing 1% to be perfect is not that previous problem - I can live with it - but a missing feature... yes, you know, custom border color!

I know I could get the mask running twice the plugin - one for the video, one for the mask - but it takes twice the time... so, getting a new optional custom border color would help. More, the best thing would be to have the chance of setting corners and borders colors separately - and the perfect thing to have top left and right and bottom left and right corners colors, as well as top/bottom and left/right borders.

Why? Well, sometimes you get just one "empty" corner, and would like to fill it using inpaint; having just that corner with a given color would help a lot - I know that inpaint is far from perfection, but sometimes would be possible to get good result... sometimes!

An example of what I mean:


As now, the plugin would give as result the image with black corners, and black borders on left and right; if it would be possible to set corner colors, inpaint could be easily used.

OK, I asked it enough. Still hope random.next would add this little feature - maybe if someone else will ask for it, he would be more motivated... anyone? Tom?

[deleted user]

I would prefer some implementation with alpha channel. Having a specific color is problematic because it it would mix with the color at the border and not make a perfect mask.

That said, the code is open source, so it doesn't have to be random.next implementing these wishes in principle, just someone who can program or is willing to learn it. spoRv, you might like programming yourself, you already know scripting, so you kind of understand the basics.

[spoRv]

Well, if you have custom colored corners and/or border, this would not interfere with the video itself - I mean, when a film image contains full reds, blues, greens, magentas, yellows, cyans? I guess never... I usually use red ($ff0000).

Programming... I wish I could be able to put my hands into that code, but I'm not... and I don't intend to study the programming language (C++ or what else) - to me, avisynth, HTML and Basic are enough, with a splash of Javascript and PHP here and there!

Maybe one of our members with the right skill would do - it should be not that difficult, if you know what to do... I have in mind at least one of them that *could probably* do that.

[deleted user]

spoRv, I'm just a web developer myself (PHP, Javascript ..), but I've dabbled in C/C++ and it's not that much of a step up. I would delve deeper into it myself if not for some reasons stopping me, but I do think you'd love it. Would open so many doors for someone who likes to fuck around with the signal like you do, if you don't mind me saying.