Avisynth parallel encoding

[althor1138]

not sure about colourlike but there seems to be x64 versions of median available.

http://ajk.kapsi.fi/projects/avisynth/median-plugin/

[deleted user]

not sure about colourlike but there seems to be x64 versions of median available.

http://ajk.kapsi.fi/projects/avisynth/median-plugin/

From that folder, take the Median-0.6-RGB64-32+64bit.zip one. It's my modification that's both 32 bit and 64 bit, updated to the newer AviSynth+ headers and also supports RGB64 colorspace for high bit depth Median. Source code is also in the folder in a separate zip.

I actually mentioned this here a while back, but ofc spoRv already forgot.

[spoRv]

Of course...

[spoRv]

Encoding Mad Max right now... using MP_Pipeline as a single script, I got 0.9fps... wow... CPU at 5/7%

Code:

MP_Pipeline("""
source1
### ###
source2
filter1
filter2
filter3 #scriptclip
### ###
""")

then, I split the script in seven parts - didn't want to overload the CPU - each 20000 frames long

Code:

MP_Pipeline("""
source1
### ###
source2
filter1
filter2
filter3_1 #scriptclip
trim(000000,019999)
### ###
""")

and so on... I was used to make different scripts (_1, _2 etc) when using scriptclip, as it share the same variables, hence every one has its own variables (var1_1, var2_2 etc) - but I suspect this is not the case with avisynth+, still did not want to risk!

Conclusion: now 7 scripts running each at mostly the same speed of single script, for a total of around 6.5x speed - or, it will took 6.5 less time than running a single script - and CPU is running around 35% with "only" 23GB, but I still have A LOT of browser windows opened taking more than 5GB... so, 32bit still working well, let's see if some of them will crush over the night (there is always more than few chances...) - I know that I HAVE to use 64bit, but transition to avisynth+, albeit almost painless, happened few weeks ago, so please gimme a break!

All in all, a good compromise - probably I could run more scripts at the same time, but even if CPU usage is quite low, there are around 20 threads active, and I prepared my parallel scriptclip up to 8 versions - you know, before I got "only" a 4C/8T CPU, so...

[deleted user]

Just a hint, the problem with 32 bit isn't that you run out of RAM, it's t hat 32 bit applications can only address a maximum of I think 4 GB (or even only 2?). So as long as each single process uses less than 2 GB of RAM, you can run 6 of them simultaneously no problem. If you run out of RAM, Windows will use a pagefile by default, so you're safe in that regard.

[althor1138]

Would you be interested in telling us which filters and settings you are using? I'd like to try running this and see what kind of fps I can get on the 1800x.

EDIT: I've been playing around with mp_pipeline today and can't seem to get any kind of performance gain out of it vs running single-threaded. Maybe I'm missing something? I never get more than 10-15% usage out of the cpu while running qtgmc. However, I can switch to avs+ multi-threading and use all physical cpu cores and get it to jump up to 100% cpu usage if I want to.

[spoRv]

I continue to use MP_pipeline for any avisynth script since a lot of time, and I have discovered something in the meanwhile. Should add that upgrading to Avisynth+ was a breeze!

Do note that if you use ScriptClip/GScriptClip (more than once, usually), sometimes it goes awry (read: continue to encode, but in wrong ways) - don't know if it's my fault, though...
And sometimes, with very heavy scripts (the ones I use, even when I split them in successive steps), it crashes.
Said that, it works very well let's say 95% of the times, with A LOT of time spared!

I usually use these kind of scripts:

Code:

MP_Pipeline("""
yoursource
###
filter1
###
filter2
###
filter3

### prefetch: 32, 0
### branch: 32
### ###
""")

I have a CPU with 16 core and 32 threads, so I use 32 for branch; not tested if different numbers for prefetch would increase speed, but I can say that with branch equal to my CPU threads, script goes A LOT faster - and I mean it, like 10 times more...

WARNING: after every ### you lose any variable and get only the last version of the clip, so for example if you work with two clips you can use this:

Code:

MP_Pipeline("""
a=yoursource1
b=yoursource2

a.trim(x,y).filter1+\
b.trim(z,w).filter2+\
a.trim(k,j).filter3
###
filter4 # that will apply to the whole previous clip

### prefetch: 32, 0
### branch: 32
### ###
""")

I invite everyone to test it, firt without using prefetch and branch lines, then replacing them with various numbers, and report your experience here; this would be beneficial for the whole community!

Last thought: if it's 10x faster with a "mere" Threadripper 1950x 16core, think what could do with the 3990x that has 64core and higher IPC!!!

[spoRv]

Important note: even if I set the branch to 32, it works "only" at 24 - how can I know it? Well, the avisynth status windows shows actual frame number, and the increase step is 24 (while when I set branch at 12 or 16 reflected this number).

Now, don't know if the max branch is: CPU threads/4*3 or CPU threads-12