Python - How to efficiently write/read a JSON file inside a compressed archive in one go?

Question

I am using Python 3.8 and would like to save a dictionary of data into a JSON file that is compressed in an archive in one go, ideally using only the Python Standard Library. For example, this means my data is saved in a file data.json that is contained in an archive compressed_data.zip.

Right now, this is as far as I got:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# SPDX-License-Identifier: GPL-3.0-or-later

# Python Standard Library imports

import json
import zlib

# Prepare some data
data: dict = {
    "common_name": "Brongersma's short-tailed python",
    "scientific_name": "Python brongersmai",
    "length": 290
}

# Save data to a JSON file
with open("data.json", "w", encoding="utf-8") as output_JSON_file: 
    json.dump(data, output_JSON_file, ensure_ascii=False, indent=4)

# Open saved JSON then compress it
with open ("data.json", "r", encoding="utf-8") as input_JSON_file: 
    data: dict = json.load(input_JSON_file)
    # Data needs to be saved as bytes to be compressed
    data_bytes: bytes = json.dumps(data, indent=4).encode("utf-8")
    compressed_data = zlib.compress(data_bytes, level=zlib.Z_BEST_COMPRESSION)
    with open ("compressed_data.zip" , "wb") as output_zlib_file: 
        output_zlib_file.write(compressed_data)

This does not produce my desired results because (a) it saves the JSON file first, opens it, then saves the data into a compressed file ending up with two files on disk; and (b) the compressed file is the compressed data but not a JSON file in a ZIP file that can be opened in any generic GUI compression/decompression program.

So my questions are:

1. Is there a way to achieve my goal in one go without saving a JSON file first then compressing that into an archive? (i.e. the .json file never touches the disk, only the .zip file does)

2. How do I do the reverse and decompress from an archive directly into a dictionary in Python?

3. If there isn't a way to to 1. and 2. in one go, what would be a relatively efficient way to do it anyway?

Note: Ideally I'd like a solution that only uses the Python 3.8 Standard Library and the compressed archive doesn't have to use the zlib library or be a ZIP file. Other high compression ratio methods are fine, too.

Thank you!

question from:https://stackoverflow.com/questions/65834655/python-how-to-efficiently-write-read-a-json-file-inside-a-compressed-archive-i

Answer 1

Finally figured it out. Recording it here for future reference.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# SPDX-License-Identifier: GPL-3.0-or-later

# Python's internal `zipfile` module
import json
import zipfile

# Prepare some data
data: dict = {
    "common_name": "Brongersma's short-tailed python",
    "scientific_name": "Python brongersmai",
    "length": 290
}

# Use the `zipfile` module
# `compresslevel` was added in Python 3.7
with zipfile.ZipFile("compressed_data.zip", mode="w", compression=zipfile.ZIP_DEFLATED, compresslevel=9) as zip_file: 
    # Dump JSON data
    dumped_JSON: str = json.dumps(data, ensure_ascii=False, indent=4)
    # Write the JSON data into `data.json` *inside* the ZIP file
    zip_file.writestr("data.json", data=dumped_JSON)
    # Test integrity of compressed archive
    zip_file.testzip()

This solution uses the Python Standard Library's internal zipfile module. Key is zip_file.writestr() which allows you to essentially write to a file inside the ZIP file.

If there are other solutions, please do share them!