Dr. Strangelove or: How I learned to Stop Worrying and Love Digital Noise Reduction

[apzlsoxk]

Project Info:

This project is more of a proof of concept of advancements in Digital Noise Reduction algorithms, and that DNR no longer needs to be considered a wholly destructive process. The algorithm I used for this can by found at https://arxiv.org/pdf/1811.12758.pdf which is a GPU-accelerated neural-network based non-local denoising algorithm.

I chose this movie for a couple reasons:

1. It was given a 4K HDR release relatively recently, meaning the film grain would be essentially losslessly preserved from the scan.
   
2. It's black and white, which reduces the processing power required by 2/3s relative to color film.
   
3. It's my favorite movie.
   

I've attached some screenshots below of results so far. The plane is pretty easy to denoise given its lack of fine textures: 




But check out the results of the DNR algorithm in this shot: Group Captain Lionel Mandrake's uniform goes from a noisy, homogeneous uniform, to a heather wool pattern. And the logo resting on top of the computer goes from a blurry mess to a clearly legible "IBM 7090" text.



This will probably take a couple weeks given the denoising processing power required (each frame takes a couple minutes) but I'll keep this updated on features of note, or ask any questions if there are any stylistic choices I'm stuck choosing between.

[deleted user]

I would never touch denoising for restorations/preservations, not even AI ones, but I do have to say, it is pretty impressive what AI can do. I could certainly picture good uses for it as a creative instrument for movie makers, but not necessarily for preservations ... maybe some day tho?

Are you implementing this algorithm yourself or are you using a repository by someone else? I'm kinda dreaming about acquiring some deeper understanding about such topics myself some day and am always curious to hear what people think a good starting point might be.

[apzlsoxk]

I would never touch denoising for restorations/preservations, not even AI ones, but I do have to say, it is pretty impressive what AI can do. I could certainly picture good uses for it as a creative instrument for movie makers, but not necessarily for preservations ... maybe some day tho?

Are you implementing this algorithm yourself or are you using a repository by someone else? I'm kinda dreaming about acquiring some deeper understanding about such topics myself some day and am always curious to hear what people think a good starting point might be.

That's actually a great question. So the results you see are from a pretrained model in this repository, but essentially what I hope to gain from this is a set of training data to generate a much smaller neural network for the purpose of exclusively removing film grain. VNLNet is made for a general purpose denoising, and the way that it works is that it finds features hidden beneath the noise, and then can extract the noisy bits. However, my goal is to identify film grain rather than features. Even though the film grain is random, it still has consistent characteristics.

The overall goal for this is to help with the 4K77 restoration project. There's been a lot of trouble trying to HDR it with the film as dinged up as it is without blowing up the grain to unwatchable levels. Plus, DNRing helps a lot with filtering out noise from film damage that needs to be filtered out. And then once the cleaning is done and the HDR is applied, the film grain is re-added back into the luminance channel, and all the film nerds like me can poo-poo DNR and marvel at the how 'unadulterated' the final product is.

[BDgeek]

apzlsoxk,

I'm with TomArrow in regards to preservation, but nonetheless these appear to be some very interesting results.

Would you care to post the screenshots in native 4k resolution for a proper evaluation?

My biggest gripe with DNR is the texture. Even worst than the possible loss in detail is the usual waxy resulting texture.

[apzlsoxk]

apzlsoxk,

I'm with TomArrow in regards to preservation, but nonetheless these appear to be some very interesting results.

Would you care to post the screenshots in native 4k resolution for a proper evaluation?

My biggest gripe with DNR is the texture. Even worst than the possible loss in detail is the usual waxy resulting texture.

Unfortunately, I had to downscale the images to 1080p in order to load them into my GPU. I've added the description of the neural network from the paper:

Our network can be divided in two stages: a non-local stage and a local stage. The non-local stage consists of four 1 × 1 convolution layers with 32 kernels. The rationale for these layers is to allow the network to compute pixel-wise features out of the raw non-local features f nl at the input. The second stage receives the features computed by the first stage. It consists of 14 layers with 64 3 × 3 convolution kernels, followed by batch normalization and ReLU activations. The output layer is a 3 × 3 convolution...

Then considering that the model uses 7 past frames and 7 future frames for each frame degrained, denoising each 1080p frames requires ~1400 MB of VRAM, but due to some IO decisions in the code, we spike to ~3 GB VRAM per frame. If I were to render at 4K, that'd mean a peak 12 GB VRAM per frame which I just don't have. I've tried slicing each frame into tiles in order to preserve the resolution, but I was getting some annoying tiling artifacts.

I've attached some 1080p frames below, I can't tell if this is exhibiting the 'waxy' textures you're talking about. I ran some initial denoising tests at much stronger settings, and it looked like the characters were wearing clown makeup. The settings I had were slightly below total noise removal. Some of the cockpit scenes still had visible noise in them.

Before DNR:


After DNR:


Before DNR:


After DNR:


I've compiled the first 20 minutes into a video, PM me and I can send you a link. The DNR software isn't perfect, and it produces interesting flickering that isn't visible in still frames.

[Samourai]

Interesting, but it definitely has that waxy look.

[CaveDoctor]

I would never touch denoising for restorations/preservations, not even AI ones, but I do have to say, it is pretty impressive what AI can do. I could certainly picture good uses for it as a creative instrument for movie makers, but not necessarily for preservations ... maybe some day tho?

Are you implementing this algorithm yourself or are you using a repository by someone else? I'm kinda dreaming about acquiring some deeper understanding about such topics myself some day and am always curious to hear what people think a good starting point might be.

That's actually a great question. So the results you see are from a pretrained model in this repository, but essentially what I hope to gain from this is a set of training data to generate a much smaller neural network for the purpose of exclusively removing film grain. VNLNet is made for a general purpose denoising, and the way that it works is that it finds features hidden beneath the noise, and then can extract the noisy bits. However, my goal is to identify film grain rather than features. Even though the film grain is random, it still has consistent characteristics.

The overall goal for this is to help with the 4K77 restoration project. There's been a lot of trouble trying to HDR it with the film as dinged up as it is without blowing up the grain to unwatchable levels. Plus, DNRing helps a lot with filtering out noise from film damage that needs to be filtered out. And then once the cleaning is done and the HDR is applied, the film grain is re-added back into the luminance channel, and all the film nerds like me can poo-poo DNR and marvel at the how 'unadulterated' the final product is.

I'm interested in participating, eventually.
mostly for Television series on bluray that are overcompressed due to film grain.
I hate removing grain and noise, but overcompression is something that has really pissed me off recently, with regards to "The Shield".
Overcompression led to extremely noticeable macroblocking, and I know you can't add detail, but I'm curious about something, I just can't put my finger on it.
I know Mill Creek has a terrible reputation when it comes to releases, and at least two series have come off terrible in their releases.
30 rock had several episodes that turned out horribly.
So, I'd like to use your denoising to see what I can repair.