Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
519 views
in Technique[技术] by (71.8m points)

python - subprocess.Popen taking too long on WSL Linux

I have this subprocess.Popen() context manager:

with Popen(
    args=command, shell=False, stdout=PIPE, bufsize=1, universal_newlines=True
) as process:

    # TIMING
    start = timer()
    lines = list(process.stdout)
    end = timer()
    print('Time taken:', end - start) # 53.662078000000065 seconds -> Linux

    for _ in tqdm(iterable=lines, total=len(lines)):
        sleep(0.1)

if process.returncode != 0:
    raise CalledProcessError(returncode=process.returncode, cmd=process.args)

And it seems to take 53 seconds to process list(process.stdout) when running in a WSL Linux enviorment. However, when I run it in a Windows enviorment, it only takes 0.6 seconds. I'm finding it strange to see why the timings are so different.

I've tried using subprocess.run() and subprocess.check_output() instead, but they still lead to the same long lag before processing the tqdm() loop.

Am I missing something here? I've tried looking at the docs to see what are the differences using subprocess.Popen() in a Windows vs WSL Linux enviorment, but I'm still unsure what the issue is. Perhaps list(process.stdout) is unnecessary here, and there is a better way to store the lines from stdout.

Any sort of guidance would be very helpful here.

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

You will need to reevaluate that performance issue in Q3 2019, with WSL2.

See "Announcing WSL 2" from Craig Loewen

Changes in this new architecture will allow for: dramatic file system performance increases, and full system call compatibility, meaning you can run more Linux apps in WSL 2 such as Docker.

File intensive operations like git clone, npm install, apt update, apt upgrade, and more will all be noticeably faster.
The actual speed increase will depend on which app you’re running and how it is interacting with the file system.
Initial tests that we’ve run have WSL 2 running up to 20x faster compared to WSL 1 when unpacking a zipped tarball, and around 2-5x faster when using git clone, npm install and cmake on various projects.

Linux binaries use system calls to perform many functions such as accessing files, requesting memory, creating processes, and more.
In WSL 1 we created a translation layer that interprets many of these system calls and allows them to work on the Windows NT kernel. However, it’s challenging to implement all of these system calls, resulting in some apps being unable to run in WSL 1.
Now that WSL 2 includes its own Linux kernel it has full system call compatibility.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...