python - Traversing FTP listing

Question

I am trying to to get all directories' name from an FTP server and store them in hierarchical order in a multidimensional list or dict

So for example, a server that contains the following structure:

/www/
    mysite.com
        images
            png
            jpg

at the end of the script, would give me a list such as

['/www/'
  ['mysite.com'
    ['images'
      ['png'],
      ['jpg']
    ]
  ]
]

I have tried using a recursive function like so: def traverse(dir): FTP.dir(dir, traverse)

FTP.dir returns lines in this format:

drwxr-xr-x    5 leavesc1 leavesc1     4096 Nov 29 20:52 mysite.com

so doing line[56:] will give me just the directory name(mysite.com). I use this in the recursive function.

But i cannot get it to work. I've tried many different approaches and can't get it to work. Lots of FTP errors as well (either can't find the directory - which is a logical issue, and sometimes unexpected errors returned by the server, which leaves no log and i can't debug)

bottom line question: How to get a hierarchical directory listing from an FTP server?

Answer 1

Here is a naive and slow implementation. It is slow because it tries to CWD to each directory entry to determine if it is a directory or a file, but this works. One could optimize it by parsing LIST command output, but this is strongly server-implementation dependent.

import ftplib

def traverse(ftp, depth=0):
    """
    return a recursive listing of an ftp server contents (starting
    from the current directory)

    listing is returned as a recursive dictionary, where each key
    contains a contents of the subdirectory or None if it corresponds
    to a file.

    @param ftp: ftplib.FTP object
    """
    if depth > 10:
        return ['depth > 10']
    level = {}
    for entry in (path for path in ftp.nlst() if path not in ('.', '..')):
        try:
            ftp.cwd(entry)
            level[entry] = traverse(ftp, depth+1)
            ftp.cwd('..')
        except ftplib.error_perm:
            level[entry] = None
    return level

def main():
    ftp = ftplib.FTP("localhost")
    ftp.connect()
    ftp.login()
    ftp.set_pasv(True)

    print traverse(ftp)

if __name__ == '__main__':
    main()