shell - How to execute '<(cat fileA fileB)' using python?

Question

I am writing a python program that uses other software. I was able to pass the command using subprocess.popen. I am facing a new problem: I need to concatenate multiples files as two files and use them as the input for the external program. The command line looks like this:

extersoftware --fq --f <(cat fileA_1 fileB_1) <(cat fileA_2 fileB_2)

I cannot use shell=True because there are other commands I need to pass by variables, such as --fq.(They are not limited to --fq, here is just an example)

One possible solution is to generate middle file. This is what I have tried:

file_1 = ['cat', 'fileA_1', 'fileB_1']
p1 = Popen(file_1, stdout=PIPE)
p2 = Popen(['>', 'output_file'], stdin=p1.stdout, stdout=PIPE)

p1.stdout.close()
output = p2.communicate()
print output

I got error message: OSError: [Errno 2] No such file or directory Which part did I do wrong?

It would be better if there is no middle file. For this reason, I am looking at named pipe. I do not quiet understand it.

I have looked at multiple questions that have been answered here. To me they are all some how different from my question here. Thanks ahead for all your help.

Answer 1

The way bash handles <(..) is to:

1. Create a pipe
2. Fork a command that writes to the write end
3. Substitute the <(..) for /dev/fd/N where N is the input end file descriptor of the pipe (try echo <(true)).
4. Run the command

The command will then open /dev/fd/N, and the OS will cause that to duplicate the inherited read end of the pipe.

We can do the same thing in Python:

import subprocess                                                            
import os                                                                    

# Open a pipe and run a command that writes to the write end                 
input_fd, output_fd = os.pipe()                                              
subprocess.Popen(["cat", "foo.txt", "bar.txt"], shell=False, stdout=output_fd)
os.close(output_fd);                                                         

# Run a command that uses /dev/fd/* to read from the read end                
proc = subprocess.Popen(["wc", "/dev/fd/" + str(input_fd)],                  
                        shell=False, stdout = subprocess.PIPE)               

# Read that command's output                                                 
print proc.communicate()[0]   

For example:

$ cat foo.txt 
Hello

$ cat bar.txt 
World

$ wc <(cat foo.txt bar.txt)
      2       2      12 /dev/fd/63

$ python test.py
      2       2      12 /dev/fd/4