Last updated: Oct 13, 2023

Logging in Python to a File

Logging is used to record and monitor the behavior of your code, troubleshoot issues, and gain insights into how your application is performing. An event is a descriptive message which can optionally contain variable data.

Python offers a built-in logging module that provides a comprehensive framework for implementing a structured and organized logging strategy. This module supports various log levels, including DEBUG, INFO, WARNING, ERROR, and CRITICAL.

Logging within Python is made straightforward with a set of user-friendly functions, each tailored to a specific level of event severity. These functions are named as debug(), info(), warning(), error(), and critical(), aligning with the seriousness of the events they are designed to capture. The usage of these functions is elaborated below:

Level	Usage
DEBUG	To track detailed information when diagnosing problems.
INFO	To confirm things are working as expected.
WARNING	To indicate that everything is working as expected but something unexpected happened and may cause problems in the future. For Example - Low disk space.
ERROR	To indicate some serious problem occurred due to which some functions were failed to run.
CRITICAL	To indicate a serious problem due to which the program itself may not continue to work.

Importing the Logging Module

To use logging in Python, you must import the logging module first:

import logging

Simple Logging

Here's an example that demonstrates how to configure and use simple logging in Python:

import logging

logging.basicConfig(
    filename='myapp.log',
    level=logging.DEBUG,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    datefmt='%m/%d/%Y %I:%M:%S %p')

logger = logging.getLogger(__name__)

logger.debug("This is a debug message.")
logger.info("This is an informational message.")
logger.warning("This is a warning message.")
logger.error("This is an error message.")
logger.critical("This is a critical message.")

Here's an explanation of each part of this example code:

• import logging: This line imports the built-in logging module in Python, which is used for implementing logging functionality.
• logging.basicConfig(): This is a configuration step where you are setting up the behavior of the logging module. The basicConfig() function is used to set the basic configuration parameters, such as the log file name, logging level, log format, and date format. Here, filename='myapp.log' specifies the name of the log file where log messages will be written (in this case, 'myapp.log' in the current directory). level=logging.DEBUG sets the minimum log level to DEBUG, meaning that all log messages with a level of DEBUG or higher will be recorded. format specifies the format of each log message, which includes the timestamp, logger name, log level, and the log message itself, while datefmt defines the date format to be used in the log messages.
• logging.getLogger(__name__): This line creates a logger object. The logger is named after the current module (__name__), allowing you to categorize log entries by different parts of your application.
• logger.debug() logger.info(), logger.warning(), logger.error(), logger.critical(): These lines demonstrate logging messages with different severity levels:
  

Advanced Logging with Log File Rotation

The logging library takes a modular approach and provides these components:

• Loggers: class whose objects helps to call the logging functions directly in the application code.
• Handlers: sends log records created by loggers to the appropriate destination.
• Filters: helps to filter log records.
• Formatters: helps to format the log messages.

Log file rotation is crucial when you want to prevent disk Space Overload. Without rotation, log files can grow indefinitely, potentially filling up your disk space. Rotating log files ensures that old log data is archived or deleted, preventing disk space issues. It also helps organize log data, making it easier to manage, search, and analyze. Log files can be named and structured in a way that makes sense for your application. It also helps to maintain performance as large log files can impact application performance. By rotating logs, you keep the log file size manageable, thereby minimizing performance impacts.

Here's an example of advanced logging with log file rotation in Python:

1. Create a file named log_helper.py and add the following code:

import logging
from logging.handlers import RotatingFileHandler
from pathlib import Path


def getLogger():
    
    # Define the directory and file names
    directory = "logs"  # Replace with your desired directory name
    file_name = "my_log.log"  # Replace with your desired file name

    # Create the directory if it doesn't exist
    directory_path = Path(directory)
    directory_path.mkdir(parents=True, exist_ok=True)

    # Create the full file path
    file_path = directory_path / file_name

    # Get the absolute path
    filename = file_path.resolve()

    try:
        # Create logger
        logger = logging.getLogger(__name__)
        logger.setLevel(logging.DEBUG)

        # Create handlers
        str_handler = logging.StreamHandler()
        file_handler = RotatingFileHandler(
            filename, mode='a', maxBytes=100000000, backupCount=4)
        str_handler.setLevel(logging.INFO)
        file_handler.setLevel(logging.INFO)

        # Create formatters and add it to handlers
        str_format = logging.Formatter(
            '%(asctime)s - %(filename)s - %(levelname)s - %(message)s')
        file_format = logging.Formatter(
            '%(asctime)s - %(filename)s - %(levelname)s - %(message)s')
        str_handler.setFormatter(str_format)
        file_handler.setFormatter(file_format)

        # Add handlers to the logger
        logger.addHandler(str_handler)
        logger.addHandler(file_handler)
        return logger
    except Exception as ex:
        print("Error creating logger : %s" % (ex))

In this example, to rotate disk log files in Python, we use RotatingFileHandler class, located in the logging.handlers module. The use of maxBytes and backupCount values in RotatingFileHandler allows the file to rollover at a predetermined size. Rollover never occurs if the value of maxBytes and backupCount is zero. When the current log file size is nearly the maxBytes value in length, the rollover occurs. The value of backupCount argument determines the number of files, the system will create to save old log files by appending the extensions .1, .2, .3, etc., to the filename. For example, with a backupCount of 6 and a base file name of example.log, you would get example.log, example.log.1, example.log.2, example.log.3, example.log.4, example.log.5, and example.log.6. The log messages being written to is always example.log file. When the file size is nearly the maxBytes, it is closed and renamed to example.log.1 and if there is example.log.1 and example.log.2, then they are renamed to example.log.3 and example.log.4.

2. Next, you can import the getLogger() method from the above file into any other file to start logging:

from log_helper import getLogger

logger =  getLogger()

logger.debug("This is a debug message.")
logger.info("This is an informational message.")
logger.warning("This is a warning message.")
logger.error("This is an error message.")
logger.critical("This is a critical message.")