Avisynth parallel encoding

[spoRv]

Further tests showed mp_pipeline is faster (usually) than running a script without it; the problem is, it takes the CPU at 100% with just one instance; if I run, for example, four parallel avisynth script at once, CPU is more or less at the same workload, but summing single performance, speed is about doubled in comparison to single mp_pipeline; I should admit that sometimes (often?) one (or more) of these parallel encoding crashes, but if I have to re-run one or few of them, it's a small price to pay, in exchange of taking "only" two days instead four, for example...

Now, I'm curious to know if a CPU with more than 4 cores takes advantage of the mp_pipeline, or, as I suspect, running eight (instead four) concurrent instances would always take half time...

EDIT: forgot that, when I did the last parallel encoding, I used a lighter script... with the new one, with added filters, parallel encoding crashed almost immediately - even if still a bit faster, and CPU around 80%... so, mp_pipeline is still faster than not using it, AND it does not crash. I guess I should wait...

[CSchmidlapp]

So I figured I'd try out avisynth+ with multi-threading and from my experience the last few hours this is a terrible idea. No matter what it only uses 4 cores at about 10% and averages 3.5 fps. It's really a mess. Uninstalling and getting back to avisynth 2.6mt was also a nightmare.  Had to delete registry keys, hidden folders, etc. just to get it to autoload the plugin directory again since avs+ moves all of the pointers in the registry and uninstalling it didn't remove them.

What I'm trying to say here is avoid avs+ especially the multi-threaded variant. If somebody has had success please elaborate lol.  I think I might try Vapoursynth here soon just to see if I can figure it out.

Ive just had a similar experience!

I pipe my scripts through TMPGenc 6 which is 64bit. The first test was fantastic and used 100% of my processer then after that became ustable and would just crash the programe. (I had realtime playback in my scripts with MPC still, but I had that with AVS 2.6.0MT).

Returning back to AVS 2.6.0MT, while it works, is now is buggy as hell!!! AVISource even has problems

Any advice on how you corrected your system would be appreciated

[spoRv]

Since my catastrofic hard disk crash of about two months ago, I use Avisynth+ with no big problems, apart sometimes it run out of memory where before Avisynth 2.6 did not - dunno if it's due to the fact I used W10PRO before and W10HOME now, or about Intel Vs AMD...

Well, I discovered that run script through MP_pipeline usually solves the problem.

Apart that, it speed up encoding quite some time; here an example:

Code:

MP_Pipeline("""
source
filter 1
### prefetch: 32, 24
### branch: 8
filter 2
### prefetch: 32, 24
### branch: 8
filter 3
### prefetch: 32, 24
### branch: 8
### ###
""")

Of course, replace source with your source(2) and filter with your own ones; prefetch and branch values should be adjusted by the number of your CPU cores/thread; with this, and an AMD 16core, I jumped from 0.8fps and around 30% CPU usage to more than 2.2fps and more than 90% CPU usage... not bad!

WARNING: if you use filters that work in parallel, result could be corrupted, so just make a test clip before let it encode for days and discover later it didn't work properly!

[deleted user]

Since my catastrofic hard disk crash of about two months ago, I use Avisynth+ with no big problems, apart sometimes it run out of memory where before Avisynth 2.6 did not - dunno if it's due to the fact I used W10PRO before and W10HOME now, or about Intel Vs AMD...

Well, I discovered that run script through MP_pipeline usually solves the problem.

Apart that, it speed up encoding quite some time; here an example:

Code:

MP_Pipeline("""
source
filter 1
### prefetch: 32, 24
### branch: 8
filter 2
### prefetch: 32, 24
### branch: 8
filter 3
### prefetch: 32, 24
### branch: 8
### ###
""")

Of course, replace source with your source(2) and filter with your own ones; prefetch and branch values should be adjusted by the number of your CPU cores/thread; with this, and an AMD 16core, I jumped from 0.8fps and around 30% CPU usage to more than 2.2fps and more than 90% CPU usage... not bad!

WARNING: if you use filters that work in parallel, result could be corrupted, so just make a test clip before let it encode for days and discover later it didn't work properly!

Glad to see you saw the light!

You are most likely running your scripts in 32 bit VirtualDub/32 bit x264. AviSynth+ works in both 32 bit and 64 bit and has separate plugin folders (64-bit ones have a 64 in the folder name). If you use 64 bit, it's almost impossible to run out of memory, but you can't use any 32 bit plugins.  If your particular script doesn't rely on exclusively 32 bit plugins, I highly recommend using 64 bit for that script, it should solve any running out of memory issues.

[spoRv]

Yep, I know... sadly I guess most plugins I use are 32bit...

[althor1138]

I've switched to avs+ and decided to entirely do away with everything 32bit. This hasn't affected me as badly as I thought. Pretty much every filter I want to use so far in the last 6 months, I've been able to find the 64 bit equivalent and virtualdub2 and x264 also have 64 bit executables. Also, the latest versions of avs+ auto set each filter to the correct mt mode so you only need to set prefetch at the end of your script with the number of threads you would like to use. Life is way easier than it was a year ago in the avisynth world.

http://avisynth.nl/index.php/AviSynth%2B_x64_plugins

This page is a good jumping off point. Also, anything done by pinterf is 64 bit.

https://github.com/pinterf?tab=repositories

[Chewtobacca]

^ Yes, I've found the same: 64-bit x264 and VirtualDub2 have certainly made switching easier.  Only once (recently) has AviSynth+ caused me problems: repeated calls to Trim() and Dissolve() led to unexpected results, which were not present in non-MT AviSynth.  But apart from that, using AviSynth+ has been a stable experience for quite a while now.

[spoRv]

Ooohhhffff.... it seems I have to switch as well...

A lot of 64bit iversions, I must admit, but still some missing...

[althor1138]

That page is not definitive nor updated frequently. If you are looking for a 64bit version of an old 32 bit filter post it here and I'll let you know if I've run across it. There are a few that aren't on that page but listed on doom9 in some random thread.

[spoRv]

That's good, thanks!

On the top of my mind, two are median and colourlike.