[ad_1]
Use Python Logging like a Professional
I can guess that nearly each Python developer generally makes use of “print” for debugging. There’s nothing flawed with that for prototyping, however for manufacturing, there are way more efficient methods to deal with the logs. On this article, I’ll present 5 sensible the reason why Python “logging” is way more versatile and highly effective and why you completely ought to use it you probably have not began earlier than.
Let’s get into it.
Code
To make issues extra sensible, let’s think about a toy instance. I created a small software that calculates a linear regression for 2 Python lists:
import numpy as np
from sklearn.linear_model import LinearRegression
from typing import Checklist, Optionally availabledef do_regression(arr_x: Checklist, arr_y: Checklist) -> Optionally available[List]:
""" LinearRegression for X and Y lists """
attempt:
x_in = np.array(arr_x).reshape(-1, 1)
y_in = np.array(arr_y).reshape(-1, 1)
print(f"X: {x_in}")
print(f"y: {y_in}")
reg = LinearRegression().match(x_in, y_in)
out = reg.predict(x_in)
print(f"Out: {out}")
print(f"Rating: {reg.rating(x_in, arr_y)}")
print(f"Coef: {reg.coef_}")
return out.reshape(-1).tolist()
besides ValueError as err:
print(f"ValueError: {err}")
return None
if __name__ == "__main__":
print("App began")
ret = do_regression([1,2,3,4], [5,6,7,8])
print(f"LinearRegression outcome: {ret}")
This code works, however can we do it higher? We clearly can. Let’s see 5 benefits of utilizing “logging” as a substitute of “print” on this code.
1. Logging ranges
Let’s change our code a bit:
import loggingdef do_regression(arr_x: Checklist, arr_y: Checklist) -> Optionally available[List]:
"""LinearRegression for X and Y Python lists"""
attempt:
x_in = np.array(arr_x).reshape(-1, 1)
y_in = np.array(arr_y).reshape(-1, 1)
logging.debug(f"X: {x_in}")
...
besides ValueError as err:
logging.error(f"ValueError: {err}")
return None
if __name__ == "__main__":
logging.basicConfig(stage=logging.DEBUG, format='%(message)s')
logging.data("App began")
ret = do_regression([1,2,3,4], [5,6,7,8])
logging.data(f"LinearRegression outcome: {ret}")
Right here I changed “print” calls with “logging” calls. We made a small change, but it surely makes the output way more versatile. Utilizing the “stage” parameter, we will now set completely different logging ranges. For instance, if we use “stage=logging.DEBUG”, then all output shall be seen. After we are certain that our code is prepared for manufacturing, we will change the extent to “logging.INFO”, and debugging messages is not going to be displayed anymore:
And what’s essential is that no code change is required besides the initialization of the logging itself!
By the way in which, all out there constants could be discovered within the logging/__init__.py file:
ERROR = 40
WARNING = 30
INFO = 20
DEBUG = 10
NOTSET = 0
As we will see, the “ERROR” stage is the best; by enabling the “ERROR” log stage, we will suppress all different messages, and solely errors shall be displayed.
2. Formatting
As we will see from the final screenshot, it’s simple to manage the logging output. However we will do way more to enhance that. We will additionally regulate the output by offering the “format” string. For instance, I can specify formatting like this:
logging.basicConfig(stage=logging.DEBUG,
format='[%(asctime)s] %(filename)s:%(lineno)d: %(message)s')
With out every other code adjustments, I can see timestamps, file names, and even the road numbers within the output:
There are about 20 completely different parameters out there, which could be discovered within the “LogRecord attributes” paragraph of the handbook.
3. Saving logs to a file
Python logging is a really versatile module, and its performance could be simply expanded. Let’s say we need to save all our logs right into a file for future evaluation. To do that, we have to add solely two strains of code:
logging.basicConfig(stage=logging.DEBUG,
format='[%(asctime)s] %(message)s',
handlers=[logging.FileHandler("debug.log"),
logging.StreamHandler()])
As we will see, I added a brand new parameter “handlers”. A StreamHandler is displaying the log on the console, and the FileHandler, as we will guess from its title, saves the identical output to the file.
This method is de facto versatile. Loads of completely different “handler” objects can be found in Python, and I encourage readers to check the manual on their very own. And as we already know, logging works virtually robotically; no additional code adjustments are required.
4. Rotating log recordsdata
Saving logs right into a file is an effective possibility, however alas, the disk house isn’t limitless. We will simply resolve this downside through the use of the rotating log file:
from logging.handlers import TimedRotatingFileHandler...
if __name__ == "__main__":
file_handler = TimedRotatingFileHandler(
filename="debug.log",
when="midnight",
interval=1,
backupCount=3,
)
logging.basicConfig(stage=logging.DEBUG,
format='[%(asctime)s] %(message)s',
handlers=[file_handler, logging.StreamHandler()])
All parameters are self-explanatory. A TimedRotatingFileHandler object will create a log file, which shall be modified each midnight, and solely the final three log recordsdata shall be saved. The earlier recordsdata shall be robotically renamed to one thing like “debug.log.2023.03.03”, and after a 3-day interval, they are going to be deleted.
5. Sending logs through socket
Python’s logging is surprisingly versatile. If we don’t need to save logs into a neighborhood file, we will simply add a socket handler, which can ship logs to a different service utilizing a selected IP and port:
from logging.handlers import SocketHandlerlogging.basicConfig(stage=logging.DEBUG, format='[%(asctime)s] %(message)s',
handlers=[SocketHandler(host="127.0.0.1", port=15001),
logging.StreamHandler()])
That’s it; no extra code adjustments are required!
We will additionally create one other software that can hearken to the identical port:
import socket
import logging
import pickle
import struct
from logging import LogRecordport = 15001
stream_handler = logging.StreamHandler()
def create_socket() -> socket.socket:
"""Create the socket"""
sock = socket.socket(socket.AF_INET)
sock.settimeout(30.0)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
return sock
def read_socket_data(conn_in: socket.socket):
"""Learn knowledge from socket"""
whereas True:
knowledge = conn_in.recv(4) # Information: 4 bytes size + physique
if len(knowledge) > 0:
body_len = struct.unpack(">L", knowledge)[0]
knowledge = conn_in.recv(body_len)
report: LogRecord = logging.makeLogRecord(pickle.hundreds(knowledge))
stream_handler.emit(report)
else:
logging.debug("Socket connection misplaced")
return
if __name__ == "__main__":
logging.basicConfig(stage=logging.DEBUG, format='[%(asctime)s] %(message)s',
handlers=[stream_handler])
sock = create_socket()
sock.bind(("127.0.0.1", port)) # Native connections solely
sock.pay attention(1) # One shopper could be related
logging.debug("Logs listening thread began")
whereas True:
attempt:
conn, _ = sock.settle for()
logging.debug("Socket connection established")
read_socket_data(conn)
besides socket.timeout:
logging.debug("Socket listening: no knowledge")
The tough half right here is to make use of the emit methodology, which provides all distant knowledge acquired by a socket to an energetic StreamHandler.
6. Bonus: Log filters
Lastly, a small bonus for readers who had been attentive sufficient to learn till this half. It is usually simple so as to add customized filters to logs. Let’s say we need to log solely X and Y values into the file for future evaluation. It’s simple to create a brand new Filter class, which can save to log solely strings containing “x:” or “y:” data:
from logging import LogRecord, Filterclass DataFilter(Filter):
"""Filter for logging messages"""
def filter(self, report: LogRecord) -> bool:
"""Save solely filtered knowledge"""
return "x:" in report.msg.decrease() or "y:" in report.msg.decrease()
Then we will simply add this filter to the file log. Our console output will keep intact, however the file can have solely “x:” and “y:” values.
file_handler = logging.FileHandler("debug.log")
file_handler.addFilter(DataFilter())logging.basicConfig(stage=logging.DEBUG,
format='[%(asctime)s] %(message)s',
handlers=[file_handler, logging.StreamHandler()])
Conclusion
On this quick article, we realized a number of simple methods to include logs into the Python software. Logging in Python is a really versatile framework, and it’s undoubtedly price spending a while investigating the way it works.
Thanks for studying, and good luck with future experiments.
When you loved this story, be at liberty to subscribe to Medium, and you’ll get notifications when my new articles shall be printed, in addition to full entry to 1000’s of tales from different authors.
[ad_2]
Source link
