automation - Downloading files from an http server in python

Question

Using urllib2, we can get the http response from a web server. If that server simply holds a list of files, we could parse through the files and download each individually. However, I'm not sure what the easiest, most pythonic way to parse through the files would be.

When you get a whole http response of the generic file server list, through urllib2's urlopen() method, how can we neatly download each file?

Answer 1

Urllib2 might be OK to retrieve the list of files. For downloading large amounts of binary files PycURL http://pycurl.sourceforge.net/ is a better choice. This works for my IIS based file server:

import re
import urllib2
import pycurl

url = "http://server.domain/"
path = "path/"
pattern = '<A HREF="/%s.*?">(.*?)</A>' % path

response = urllib2.urlopen(url+path).read()

for filename in re.findall(pattern, response):
    with open(filename, "wb") as fp:
        curl = pycurl.Curl()
        curl.setopt(pycurl.URL, url+path+filename)
        curl.setopt(pycurl.WRITEDATA, fp)
        curl.perform()
        curl.close()