AviSynth AiUpscale

[Chewtobacca]

This could sure come in useful:

https://github.com/Alexkral/AviSynthAiUpscale

Some cuts of films might never make it to (official) UHD, or even HD, so decent upscaling is always useful.

Doom9 thread:

https://forum.doom9.org/showthread.php?t=181665

Just a heads-up!

[PDB]

Thanks for the heads up Chew. Looks like it is semi-based on MysteryX's AVIsynthshader which is what I still use for SR upscaling. Gigapixel is better for fine details (when it works) but seems to be producing odd artifacts for me. Especially in the dark details. So I've been looking for something in-between. Better scaling via AI but with the faster speed of SR. I'm talking 1 or 2fps versus .5 to .8fps.

I'm intrigued and will give it a try.

[Chewtobacca]

Great! Let us know your thoughts. You've done way more upscaling than I have, and the results have been impressive.

[zoidberg]

I have to say I am interested in AI upscaling but literally know nothing about it. The dream for me would be to be able to pair a HD and SD master, using the existing HD footage to 'train' the AI in order to create more accurate upscaling of the SD footage.

I'm mainly thinking of 1.66:1 Robocop (aren't we all) but I think it could also be really helpful when it comes to regrading new masters with older/better colour timed sources

[PDB]

I have to say I am interested in AI upscaling but literally know nothing about it. The dream for me would be to be able to pair a HD and SD master, using the existing HD footage to 'train' the AI in order to create more accurate upscaling of the SD footage.

I'm mainly thinking of 1.66:1 Robocop (aren't we all) but I think it could also be really helpful when it comes to regrading new masters with older/better colour timed sources

TomArrow was doing a bit of experimentation with training AI in a way you are talking about. Perhaps he will chime in.

[spoRv]

Curious to see some comparison between this and Gigapixel... any volunteer?

[deleted user]

I have to say I am interested in AI upscaling but literally know nothing about it. The dream for me would be to be able to pair a HD and SD master, using the existing HD footage to 'train' the AI in order to create more accurate upscaling of the SD footage.

I'm mainly thinking of 1.66:1 Robocop (aren't we all) but I think it could also be really helpful when it comes to regrading new masters with older/better colour timed sources

That's possible. There's two codebases I know of that can do this. One is for single images and one is for sequences. I havent tried the sequence one yet. Believe it needs a really beefy GPU setup, so it might not even work for me. But I did the one for the images and got pretty nice results sometimes.

The single image one is/was called ESRGAN (upscaling) and BasicSR (training).

The sequence one is/was called EDVR.

I believe they are now kinda all bundled together in something called "MMSR" or something similar. You can find all that info on Github.

You need a CUDA GPU and Python and a few packages, but the readme tells you how to install it basically.

Edit: For a sample, check out that Dark Knight thread of PDBs somewhere on this forum, I think I posted about it there. I also made a sample in motion, maybe PDB still has the link floating around.

[PDB]

Here are the experiments Tom was referencing:

https://fanrestore.com/thread-2807-post-...l#pid56276

I haven't tested this upscaling yet, having trouble getting it working but I will eventually. Just need time to work it out. Regardless, I used the THX Halloween DVD as an upscale test since I had it handy. I used this particular pic because fine hair is a great test for upscaling. Please don't take this as truth from on high, its just my little rough test. A lot of experience users here will find these results trivial. So here for baseline reference:

Original DVD (IVTC'd in AVIsynth, Converted to 709)


From here out everything is IVTC'd and converted from 601 to 709 in AVIsynth and then upscaled. No additional filters unless noted.

Bicubic/x200


Spline36/x200


Nnedi3_rpow2/x200


SuperRes x2 with Nnnedi3 upscale (AVIshader Version)/x200


SuperResXBR (AVIshader Version)/x200


SuperRes x2 with Nnnedi3 upscale (AVIshader Version)/x200 (this is with filters for deblocking, noise, halo removal, shaprening, blurring and re-graining)


Gigapixel Photo/x200 (50% sharpen, 0% noise reduction, upscale to 4K and down to 1080p)


Gigapixel Photo/x200 (50% sharpen, 0% noise reduction, upscale to 4K and down to 1080p plus a little blur and grain)


So the summary, if it is difficult to see, is that Superres is better than most AVIsynth filters. SRXBR is about the same level. Giga still wins hands down. With Giga you get a lot more detail in the hair.

The MPEG-2 of the DVD naturally has macroblocking. Running a deblock filter in AVI takes care of the problem whereas Giga naturally removes it. Adding grain covers a lot of the problems.

Macroblocking (Superes, SR processed with deblock/grain, Gigapixel, Gigal processed with grain)


So the next steps is that I need to add examples of AIupscale, SR from info and Gigapixel video.

[spoRv]

You need a CUDA GPU and Python and a few packages, but the readme tells you how to install it basically.

What a python... ehr, pity, I have an AMD GPU!

Here are the experiments Tom was referencing:

https://fanrestore.com/thread-2807-post-...l#pid56276

Thanks to you for the link, and Tom to have produced it; even if with too much noise, I like very much the third image! I wonder how many time it takes to make a single frame, though...

So the next steps is that I need to add examples of AIupscale, SR from info and Gigapixel video.

Thanks for the comparison!
Eagerly waiting for the next steps!

[Hitcher]

As a gamer I've been watching a lot about DLSS (Deep Learning Super Sampling) recently and how it can upscale low res source. I wonder if this could ever be used on videos instead.