Avisynth parallel encoding

[spoRv]

Studing a good way to improve speed with avisynth, I tested some different path...

• New CPU: obvious... but take in account the complete cpubenchmark score, not only the single thread - if, for example, your old dual core CPU has a single thread score of 500 and complete score of 1000, and the new quad core has a single thread score of 1000 and complete score of 4000, it will go four time faster, more or less!
  
• Faster HDD (or better SSD): haven't tried an SSD yet, but I'm pretty sure this will not improve the overall speed.
  
• Avisynth 64bit: could indeed improve speed, but it has quite few compatible plugins...
  
• MT version and other multithread plugins: could work, but sometimes MT crashes, and other plugins didn't work - probably my fault
  
• Parallel encoding: it works well! Follow me...
  

I tried this path in the latest days; I didn't dare to try it with my poor old dual core, dual threaded CPU, but with the new quad core, eight treaded one, I wanted to give it a try... and, oh boys, it works, very well!

How it works?

Well, just take your script, and "cut" it in several pieces - it seems that four pieces works well here, they run the CPU at around 80%, leaving some breath for surfing and other little things; probably it could work with more pieces, but your mileage can vary.

Before

project1.avs

Code:

avisource("your source.avi")
filter1
filter2
filter3

After

project1_part1.avs

Code:

avisource("your source.avi")
filter1
filter2
filter3
trim(000000,040020)

project1_part2.avs

Code:

avisource("your source.avi")
filter1
filter2
filter3
trim(040000,080020)

project1_part3.avs

Code:

avisource("your source.avi")
filter1
filter2
filter3
trim(080000,120020)

project1_part4.avs

Code:

avisource("your source.avi")
filter1
filter2
filter3
trim(120000,000000)

project1_final.avs

Code:

a=avisource("project1_part1.avi").trim(000000,039999)
b=avisource("project1_part2.avi").trim(000000,039999)
c=avisource("project1_part3.avi").trim(000000,039999)
d=avisource("project1_part4.avi")

a+b+c+d

Encode them at the same time; speed improvement: from 250% up to 400%, depending on how "heavy" are the filters/plugins...

You know, instead of waiting, let's say, two days, it's better to wait "just" 12 hours, don't you think?

Hope this would be of some help.

[althor1138]

I've recently jumped over to avisynth_mt as well now that I have 8 cores. I probably should have done this while on 4 cores too to be honest but I always heard that avisynth_mt was unstable. However, the speed gains are worth it as long as your workflow remains simple enough. With a really complicated script I'd recommend staying with single thread. This is how I initiate my multi-threaded scripts:

Code:

setmemorymax(2048)
setmtmode(3)
avisource(blah.avi)
trim()
setmtmode(2)
temporaldenoiser()
resizer()

The script setup I use for the gout syncing of laserdiscs looks similar to this. It usually encodes at less than 5fps on my old amd and my new ryzen. However, once I enable multi-threading I get 30+ fps during encode. It's like a gamechanger when it comes to rendering out stuff. Instead of a 15 hour encode it's under 2 hours for an entire film.

EDIT: I should mention that this will max out every core and thread to 100% for the duration of the encode. I have a 360mm radiator water-cooled setup now and it struggles to keep the cpu under 60C.

[zoidberg]

project1_final.avs

Code:

a=avisource("project1_part1.avi").trim(000000,039999)
b=avisource("project1_part1.avi").trim(000000,039999)
c=avisource("project1_part1.avi").trim(000000,039999)
d=avisource("project1_part1.avi")

a+b+c+d

Won't this give you the same clip repeated 4 times? I take it all of the scripts run simultaneously in virtualdub, might have to try this one day

[spoRv]

I remember I used MT some years ago, and *probably* it was faster than non-MT version... still can't remember why I have not used it anymore - probably problems with some of my scripts...

But I would give it a try, sooner or later... AFAIK it works only with certain filters/plugins, though... can you confirm this?

Ryzen: happy to know it works well; you know what? A great review about it would be surely appreciated by many - certainly by me!

[Colek]

Told you it's great way to boost your script

[spoRv]

[althor1138]

About Ryzen I can just say that I'm happy with it's capabilities however I'd recommend waiting a few months before pulling the trigger due to the ram instability and constant bios updates. With that being said, I've reached some impressive benchmark numbers with that cinegy app that poita posted. Here is a link to my post with the results:

https://fanrestore.com/thread-1351-post-...l#pid26928

Ryzen: happy to know it works well; you know what? A great review about it would be surely appreciated by many - certainly by me!

I will continue to update that post and give any relevant information that I come across. Like I said, I recommend waiting a few months for things to stabilize before buying. I couldn't even get POST for the first few days after putting everything together. The last couple bios updates have ironed out a few issues with the ram speeds but I still can't reach 3000 mhz with my ram that is rated at that speed so they have a ways to go imo.

[althor1138]

So I figured I'd try out avisynth+ with multi-threading and from my experience the last few hours this is a terrible idea. No matter what it only uses 4 cores at about 10% and averages 3.5 fps. It's really a mess. Uninstalling and getting back to avisynth 2.6mt was also a nightmare. Had to delete registry keys, hidden folders, etc. just to get it to autoload the plugin directory again since avs+ moves all of the pointers in the registry and uninstalling it didn't remove them.

What I'm trying to say here is avoid avs+ especially the multi-threaded variant. If somebody has had success please elaborate lol. I think I might try Vapoursynth here soon just to see if I can figure it out.

[spoRv]

Sad to hear that... at least your experience would help someone - me included...

[spoRv]

project1_final.avs

Code:

a=avisource("project1_part1.avi").trim(000000,039999)
b=avisource("project1_part1.avi").trim(000000,039999)
c=avisource("project1_part1.avi").trim(000000,039999)
d=avisource("project1_part1.avi")

a+b+c+d

Won't this give you the same clip repeated 4 times? I take it all of the scripts run simultaneously in virtualdub, might have to try this one day

I copied and pasted without changing part numbers... now fixed, thanks!

---

BlackMirror suggested to use mp_pipeline; I'm trying it right now, and speed increase is around 80%! Not bad at all... I should try it with parallel encoding, it may multiply speed up to 6x, if not more (if I did my math right).
Waiting for him to write a small guide to it!

Edit: without parallel encoding, using one of my slower filter, mp_pipeline increased speed more than 3x!!!