From 04a65a183bb4d30d2504bc2ec04cc6802ed7cd54 Mon Sep 17 00:00:00 2001 From: Eugene Syromyatnikov Date: Sat, 16 Sep 2017 02:30:40 +0200 Subject: [PATCH] xlat/gen.sh: speedup xlat generation This commit implements the following changes: - increases the number of concurrent jobs to twice the CPU count; - creates a circular buffer, so instead of running multiple jobs at once, the generator tries to keep about the same number of jobs being run concurrently; - runs gen_git and gen_make concurrently in order to squeeze in one more bit of concurrency. With my deeply scientific measurements, this approach achieves up to 15% speedup with dash and about 30-40% with bash as /bin/sh on a 4-core machine. * xlat/gen.sh (main): Declare pids local variable, append pid of every run job to it; increase the limit of jobs to ncpus * 2; when the limit is reached, wait for the first pid in pids instead of resetting jobs to zero and waiting for all the run jobs; put gen_git and gen_make into background. --- xlat/gen.sh | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/xlat/gen.sh b/xlat/gen.sh index 06c1b17a..d5286d10 100755 --- a/xlat/gen.sh +++ b/xlat/gen.sh @@ -290,6 +290,7 @@ main() local name local jobs=0 local ncpus="$(getconf _NPROCESSORS_ONLN)" + local pids= [ "${ncpus}" -ge 1 ] || ncpus=1 @@ -300,15 +301,20 @@ main() name=${f##*/} name=${name%.in} gen_header "${f}" "${output}/${name}.h" "${name}" & + pids="$pids $!" names="${names} ${name}" : $(( jobs += 1 )) - if [ ${jobs} -ge ${ncpus} ]; then - jobs=0 - wait - fi + if [ "${jobs}" -gt "$(( ncpus * 2 ))" ]; then + read wait_pid rest + pids="$rest" + wait -n 2>/dev/null || wait "$wait_pid" + : $(( jobs -= 1 )) + fi <<- EOF + $pids + EOF done - gen_git "${output}/.gitignore" ${names} - gen_make "${output}/Makemodule.am" ${names} + gen_git "${output}/.gitignore" ${names} & + gen_make "${output}/Makemodule.am" ${names} & wait else name=${input##*/} -- 2.40.0