Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
928 views
in Technique[技术] by (71.8m points)

python - How to use Paramiko getfo to download file from SFTP server to memory to process it

I am trying to download a CSV file (in-memory) from SFTP using Paramiko and import it into a pandas dataframe.

transport = paramiko.Transport((server, 22))
transport.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(transport)

with open(file_name, 'wb') as fl:
    sftp.getfo(file_name, fl, callback=printTotals)
    df = pd.read_csv(fl, sep=' ')

The code below fails, telling me:

OSError: File is not open for reading

I assume that I need some kind of buffer or file like object for fl instead, since open needs a file. I am relatively new to all of this, so I would be happy it if someone could help.

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

A simple solution that still allows you to use progress callback is:

  • Use BytesIO file-like object to store a downloaded file to memory;

  • You have to seek file pointer back to file start after downloading it, before you start reading it.

    with io.BytesIO() as fl:
        sftp.getfo(file_name, fl, callback=printTotals)
        fl.seek(0)
        df = pd.read_csv(fl, sep=' ')
    

Though with this solution, you will end up having the file loaded to memory twice.


Better solution is to implement a custom file-like object. It will even allow you to download and parse the file at the same time.

class FileWithProgress:

    def __init__(self, fl):
        self.fl = fl
        self.size = fl.stat().st_size
        self.p = 0

    def read(self, blocksize):
        r = self.fl.read(blocksize)
        self.p += len(r)
        print(str(self.p) + " of " + str(self.size)) 
        return r

And use it like:

with sftp.open(file_name, "rb") as fl:
    fl.prefetch()
    df = pd.read_csv(FileWithProgress(fl), sep=' ') 

For the SFTPFile.prefetch call, refer to:
Reading file opened with Python Paramiko SFTPClient.open method is slow
.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

1.4m articles

1.4m replys

5 comments

57.0k users

...