python - How to extract the content of `*.tar.gz` dynamically

Question

I have quite big *.tar.gz file (10Gb) that contains individuals files (no sub-folders). In Jupyter Notebook it takes several hours to untar this archive. Once all files are extracted, I need to upload them into a storage location.

This is what I currently have:

untar = tarfile.TarFile(tarfilename)
untar.extractall()
untar.close()

Is it possible to extract the content of *.tar.gz dynamically (i.e. continuously)? Something like this:

with open(tarfilename, "r") as tararchive:
   for eachfile in tararchive:
       save_to_storage_location(eachfile)

So, instead of waiting until the tar archive is untarred, I just want to "open" it and move all the content one by one into the storage location.

question from:https://stackoverflow.com/questions/65598833/how-to-extract-the-content-of-tar-gz-dynamically

Answer 1

A little of tinkering with the package has shown me you can list all the files under the tar file and you could extract them individually. Without more information on what you want to do with the files afterwards (i.e. where or how you want to upload them) I cannot help on that end.

You can cycle through your files like so:

import tarfile

with tarfile.open("path/to/file.tar.gz", "r") as file:
    for each in file.getnames():
        print(each)
        file.extract(each)

At the last stage the individual file has been extracted and will be sitting in your current working directory, you can hence do something with it