pandas - arranging text files side by side using python

Question

I have 3000 text files in a directory and each .txt file contain single column data. i want to arrange them side by side to make it a mxn matrix file.

For example: paste 1.txt 2.txt 3.txt 4.txt .............3000.txt in linux

For this i tried

printf "%s
" *.txt | sort -n | xargs -d '
' paste

However it gives error paste: filename.txt: Too many open files

please suggest a better solution for the same using python.

Answer 1

You don't need python for this; if you first increase the number of open files a process can have using ulimit, it becomes easy to get columns in the right order in bash, zsh, or ksh93 shells, using paste and brace expansion to generate the filenames in the desired order instead of having to sort the results of filename expansion:

% ulimit -n 4096
% paste {1..3000}.txt > matrix.txt

(I tested this in all three shells I mentioned on a Linux box, and it works with all of them with no errors about the command line being too long or anything else.)

You could also arrange to have the original files use a different naming scheme that sorts naturally, like 0001.txt, 0002.txt, ..., 3000.txt and then just paste [0-9]*.txt > matrix.txt.