Rotating file handler for JSON logs in Python

Question

I am working on saving the json logs using python. Below is the code:

log_file = 'app_log.json'

log_json = dict()
log_json["Data"] = {}

log_json['Data']['Key1'] = "value1"
log_json['Data']['alert'] = False
log_json['Data']['Key2'] = "N/A"

log_json['Created'] = datetime.datetime.utcnow().isoformat()

with open(log_file, "a") as f:
    json.dump(log_json, f)#, ensure_ascii=False)
    f.write("
")

Now above code is generating the log file. But I have noticed that the file size is increasing a lot and in future, I might face disk space issue. I was wondering if there is any pre built rotating file handler available for json in which we can mention fixed size lets say 100mb and upon reaching this size it will delete and recreate the new file.

I have previously used from logging.handlers import RotatingFileHandler to do this in case of .log files but also want to do this for .json files. Please help. Thanks

question from:https://stackoverflow.com/questions/65841619/rotating-file-handler-for-json-logs-in-python

Answer 1

Python does not care about the log file name.

You can use the rotating handler which you used for .log file for .json file also.

See sample example below

# logging_example.py

import logging
import logging.handlers
import os
import time

logfile = os.path.join("/tmp", "demo_logging.json")

logger = logging.getLogger(__name__)

fh = logging.handlers.RotatingFileHandler(logfile, mode='a', maxBytes=1000, backupCount=5)  # noqa:E501

fh.setLevel(logging.DEBUG)

formatter = logging.Formatter("%(asctime)s - %(levelname)s - %(message)s")

fh.setFormatter(formatter)

logger.addHandler(fh)
logger.setLevel(logging.DEBUG)

while 1:
    time.sleep(1)
    logger.info("Long string to increase the file size")

You can also look at logrotate if you are working in Unix environment. It is a great and simple tool with good documentation to do just exactly what you need.