python - What is the subprocess.Popen max length of the args parameter?

Question

I am using Popen function from the subprocess module to execute a command line tool:

subprocess.Popen(args, bufsize=0, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=False, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0)

The tool I am using takes a list of files that it then processes. In some cases, this list of files can be very long. Is there a way to find the max length that the args parameter can be? With a large number of files being passed to the tool, I am getting the following error:

Traceback (most recent call last):
  File "dump_output_sopuids.py", line 68, in <module>
    uid_map = create_sopuid_to_path_dict_dcmdump(dicom_files)
  File "dump_output_sopuids.py", line 41, in create_sopuid_to_path_dict_dcmdump
    dcmdump_output = subprocess.Popen(cmd,stdout=subprocess.PIPE).communicate(0)[0]
  File "c:python26libsubprocess.py", line 621, in __init__
    errread, errwrite)
  File "c:python26libsubprocess.py", line 830, in _execute_child
    startupinfo)
WindowsError: [Error 206] The filename or extension is too long

Is there a general way to find this max length? I found the following article on msdn: Command prompt (Cmd. exe) command-line string limitation but I don't want to hard code in the value. I would rather get the value at run time to break up the command into multiple calls.

I am using Python 2.6 on Windows XP 64.

Edit: adding code example

paths = ['file1.dat','file2.dat',...,'fileX.dat']
cmd = ['process_file.exe','+p'] + paths
cmd_output = subprocess.Popen(cmd,stdout=subprocess.PIPE).communicate(0)[0]

The problem occurs because each actual entry in the paths list is usually a very long file path AND there are several thousand of them.

I don't mind breaking up the command into multiple calls to process_file.exe. I am looking for a general way to get the max length that args can be so I know how many paths to send in for each run.

Answer 1

If you're passing shell=False, then Cmd.exe does not come into play.

On windows, subprocess will use the CreateProcess function from Win32 API to create the new process. The documentation for this function states that the second argument (which is build by subprocess.list2cmdline) has a max length of 32,768 characters, including the Unicode terminating null character. If lpApplicationName is NULL, the module name portion of lpCommandLine is limited to MAX_PATH characters.

Given your example, I suggest providing a value for executable (args[0]) and using args for the first parameter. If my reading of the CreateProcess documentation and of the subprocess module source code is correct, this should solve your problem.

[edit: removed the args[1:] bit after getting my hands on a windows machine and testing]