Hello guest, if you like this forum, why don't you register? https://fanrestore.com/member.php?action=register (December 14, 2021) x

Fanrestore - Fan Restoration Forum › Technical info › Converting, encoding, authoring v
Avisynth parallel encoding

Thread Rating:
  • 2 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Threaded Mode
Avisynth parallel encoding
althor1138 Offline
Power user
****
Posts: 139
Threads: 8
Joined: 2015 Mar
Thanks: 37
Given 70 thank(s) in 34 post(s)

Country: Sweden
#21
2019-05-11, 08:14 PM
not sure about colourlike but there seems to be x64 versions of median available.

http://ajk.kapsi.fi/projects/avisynth/median-plugin/
https://github.com/althor1138
Reply
Thanks given by:
deleted user
Unregistered
 
Thanks:
Given thank(s) in post(s)
#22
2019-05-11, 09:18 PM
(2019-05-11, 08:14 PM)althor1138 Wrote: not sure about colourlike but there seems to be x64 versions of median available.

http://ajk.kapsi.fi/projects/avisynth/median-plugin/

From that folder, take the Median-0.6-RGB64-32+64bit.zip one. It's my modification that's both 32 bit and 64 bit, updated to the newer AviSynth+ headers and also supports RGB64 colorspace for high bit depth Median. Source code is also in the folder in a separate zip.

I actually mentioned this here a while back, but ofc spoRv already forgot. Big Grin
Reply
Thanks given by: althor1138 , spoRv
spoRv Offline
Project maker _,,,^..^,,,_
Posts: 7,153
Threads: 601
Joined: 2015 Jan
Thanks: 1081
Given 1466 thank(s) in 963 post(s)

Country: Italy
#23
2019-05-12, 01:16 PM
Of course... Happy
Reply
Thanks given by:
spoRv Offline
Project maker _,,,^..^,,,_
Posts: 7,153
Threads: 601
Joined: 2015 Jan
Thanks: 1081
Given 1466 thank(s) in 963 post(s)

Country: Italy
#24
2019-05-14, 12:57 AM
Encoding Mad Max right now... using MP_Pipeline as a single script, I got 0.9fps... wow... CPU at 5/7%
Code:
MP_Pipeline("""
source1
### ###
source2
filter1
filter2
filter3 #scriptclip
### ###
""")

then, I split the script in seven parts - didn't want to overload the CPU - each 20000 frames long
Code:
MP_Pipeline("""
source1
### ###
source2
filter1
filter2
filter3_1 #scriptclip
trim(000000,019999)
### ###
""")
and so on... I was used to make different scripts (_1, _2 etc) when using scriptclip, as it share the same variables, hence every one has its own variables (var1_1, var2_2 etc) - but I suspect this is not the case with avisynth+, still did not want to risk! Big Grin

Conclusion: now 7 scripts running each at mostly the same speed of single script, for a total of around 6.5x speed - or, it will took 6.5 less time than running a single script - and CPU is running around 35% with "only" 23GB, but I still have A LOT of browser windows opened taking more than 5GB... so, 32bit still working well, let's see if some of them will crush over the night (there is always more than few chances...) - I know that I HAVE to use 64bit, but transition to avisynth+, albeit almost painless, happened few weeks ago, so please gimme a break! Tongue

All in all, a good compromise - probably I could run more scripts at the same time, but even if CPU usage is quite low, there are around 20 threads active, and I prepared my parallel scriptclip up to 8 versions - you know, before I got "only" a 4C/8T CPU, so... Big Grin
Reply
Thanks given by:
deleted user
Unregistered
 
Thanks:
Given thank(s) in post(s)
#25
2019-05-14, 01:16 AM
Just a hint, the problem with 32 bit isn't that you run out of RAM, it's t hat 32 bit applications can only address a maximum of I think 4 GB (or even only 2?). So as long as each single process uses less than 2 GB of RAM, you can run 6 of them simultaneously no problem. If you run out of RAM, Windows will use a pagefile by default, so you're safe in that regard.
Reply
Thanks given by:
althor1138 Offline
Power user
****
Posts: 139
Threads: 8
Joined: 2015 Mar
Thanks: 37
Given 70 thank(s) in 34 post(s)

Country: Sweden
#26
2019-05-14, 10:31 AM (This post was last modified: 2019-05-14, 02:22 PM by althor1138.)
Would you be interested in telling us which filters and settings you are using? I'd like to try running this and see what kind of fps I can get on the 1800x.

EDIT: I've been playing around with mp_pipeline today and can't seem to get any kind of performance gain out of it vs running single-threaded. Maybe I'm missing something? I never get more than 10-15% usage out of the cpu while running qtgmc. However, I can switch to avs+ multi-threading and use all physical cpu cores and get it to jump up to 100% cpu usage if I want to.
https://github.com/althor1138
Reply
Thanks given by:
spoRv Offline
Project maker _,,,^..^,,,_
Posts: 7,153
Threads: 601
Joined: 2015 Jan
Thanks: 1081
Given 1466 thank(s) in 963 post(s)

Country: Italy
#27
2020-04-02, 08:07 PM
I continue to use MP_pipeline for any avisynth script since a lot of time, and I have discovered something in the meanwhile. Should add that upgrading to Avisynth+ was a breeze!

Do note that if you use ScriptClip/GScriptClip (more than once, usually), sometimes it goes awry (read: continue to encode, but in wrong ways) - don't know if it's my fault, though...
And sometimes, with very heavy scripts (the ones I use, even when I split them in successive steps), it crashes.
Said that, it works very well let's say 95% of the times, with A LOT of time spared!

I usually use these kind of scripts:

Code:
MP_Pipeline("""
yoursource
###
filter1
###
filter2
###
filter3

### prefetch: 32, 0
### branch: 32
### ###
""")

I have a CPU with 16 core and 32 threads, so I use 32 for branch; not tested if different numbers for prefetch would increase speed, but I can say that with branch equal to my CPU threads, script goes A LOT faster - and I mean it, like 10 times more...

WARNING: after every ### you lose any variable and get only the last version of the clip, so for example if you work with two clips you can use this:

Code:
MP_Pipeline("""
a=yoursource1
b=yoursource2

a.trim(x,y).filter1+\
b.trim(z,w).filter2+\
a.trim(k,j).filter3
###
filter4 # that will apply to the whole previous clip

### prefetch: 32, 0
### branch: 32
### ###
""")

I invite everyone to test it, firt without using prefetch and branch lines, then replacing them with various numbers, and report your experience here; this would be beneficial for the whole community!

Last thought: if it's 10x faster with a "mere" Threadripper 1950x 16core, think what could do with the 3990x that has 64core and higher IPC!!! Eek
Reply
Thanks given by:
spoRv Offline
Project maker _,,,^..^,,,_
Posts: 7,153
Threads: 601
Joined: 2015 Jan
Thanks: 1081
Given 1466 thank(s) in 963 post(s)

Country: Italy
#28
2020-04-03, 06:10 AM
Important note: even if I set the branch to 32, it works "only" at 24 - how can I know it? Well, the avisynth status windows shows actual frame number, and the increase step is 24 (while when I set branch at 12 or 16 reflected this number).

Now, don't know if the max branch is: CPU threads/4*3 or CPU threads-12
Reply
Thanks given by:


Possibly Related Threads…
Thread Author Replies Views Last Post
  [Help] Encoding audio with LFE2? LucasGodzilla 9 3,756 2024-06-05, 12:43 AM
Last Post: LucasGodzilla
  Flagging 1080p25 as 1080i25 without re-encoding Dr. Cooper 3 2,170 2021-12-07, 01:14 PM
Last Post: FreaQ
  FFV1 encoding for archival pipefan413 0 1,806 2021-03-11, 07:14 PM
Last Post: pipefan413
  [Help] Encoding HDR to SDR awsmguy145 3 2,953 2020-09-28, 12:01 PM
Last Post: Chewtobacca
  x264 encoding from Adobe Premiere DoomBot 26 20,699 2019-03-25, 08:26 PM
Last Post: jaminmc
  [Help] Encoding issue with Xvid4psp v5.0.37.8 Beber 7 5,939 2018-12-12, 12:40 AM
Last Post: deleted user
  Best Anti-Aliasing AVIsynth filter? PDB 3 5,955 2017-08-10, 03:12 AM
Last Post: IcePrick
  Bitrate and video encoding spoRv 0 3,129 2017-05-01, 11:19 AM
Last Post: spoRv
  Banding With Avisynth PDB 5 7,822 2017-03-16, 03:56 PM
Last Post: PDB
  Avisynth: multithreading, 64bit and more spoRv 2 4,722 2017-03-11, 03:31 PM
Last Post: spoRv

Forum Jump:


Users browsing this thread: 3 Guest(s)