概述
最近有个需求是把以前字符串输出的log 改为json 格式,看了别人的例子,还是有些比较茫然,索性就把logging 整个翻了一边,做点小总结.
初看log
在程序中,log 的用处写代码的你用你知道,log 有等级,DEBUG,INFO,...之类,还会记录时间,log 发生的位置,在Python 中用的多的就是logging 这个标准库中的包了.当打log 的时候究竟发生了什么? 是如何把不同级别的log 输出到不同文件里,还能在控制台输出.......
最简单的用法
import logging
logging.basicConfig(format='%(levelname)s:%(message)s',level=logging.DEBUG)
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this,too')
1,第一行导入包 2,第二行利用basicConfig 对输出的格式,和输出级别做了限制 3,后面分别输出了三条不同级别的 log
Logging Levels
共有几个等级,每个等级对应一个Int 型整数,每个等级都会有一个方法与之对应,这样输出的内容就有了不同的等级.
logger 流程,
整个过程,还是不是很详细,贴个图吧,现在看还太早,也说不清真个过程到底发生了什么,先放着,回头来看会比较好懂. loger flow
读代码
代码结构
logging 在源码中有三个文件,结构如下:
├── config.py
├── handlers.py
└── __init__.py
_int.py中实现了基础功能,主要的逻辑就在这个文件中 handlers.py 是一些Handlers (用处后面会明白)用起来很方便的. config.py 是对配置做处理的方法.
objects
LogRecord Objects
每一次log 都会实例化一个Record 对象,这个对象有很多属性,最后对LogRecord 做一下format 就输出了,格式化的log ,里面就基本就是这个对象的属性了。
class LogRecord(object): def __init__(self,name,level,pathname,lineno,msg,args,exc_info,func=None): ct = time.time() self.name = name self.msg = msg if (args and len(args) == 1 and isinstance(args[0],collections.Mapping) and args[0]): args = args[0] self.args = args self.levelname = getLevelName(level) self.levelno = level self.pathname = pathname try: self.filename = os.path.basename(pathname) self.module = os.path.splitext(self.filename)[0] except (TypeError,ValueError,AttributeError): self.filename = pathname self.module = "UnkNown module" self.exc_info = exc_info self.exc_text = None # used to cache the traceback text self.lineno = lineno self.funcName = func self.created = ct self.msecs = (ct - long(ct)) * 1000 self.relativeCreated = (self.created - _startTime) * 1000 if logThreads and thread: self.thread = thread.get_ident() self.threadName = threading.current_thread().name else: self.thread = None self.threadName = None if not logMultiprocessing: self.processName = None else: self.processName = 'MainProcess' mp = sys.modules.get('multiprocessing') if mp is not None: try: self.processName = mp.current_process().name except StandardError: pass if logProcesses and hasattr(os,'getpid'): self.process = os.getpid() else: self.process = None def __str__(self): return '<LogRecord: %s,%s,"%s">'%(self.name,self.levelno,self.pathname,self.lineno,self.msg) def getMessage(self): pass
看代码就发现, 这个类没做什么事情,就是一个model 而已,有一个得到msg 的方法
Formatter Objects
Formatter 就是对Record 专门格式化的对象,它有一个format 方法,我们实现这个方法就能 做到不同的输出,我的需求是做json 格式的log 其实关键就在写一个Formatter 就好了
class Formatter(object): converter = time.localtime def __init__(self,fmt=None,datefmt=None): if fmt: self._fmt = fmt else: self._fmt = "%(message)s" self.datefmt = datefmt def formatTime(self,record,datefmt=None): pass def formatException(self,ei): pass def usesTime(self): return self._fmt.find("%(asctime)") >= 0 def format(self,record): pass
删掉源代码中的实现细节,这个类里面主要的是format 方法,这是默认最基本的Formater ,还有专门对exception ,时间做格式化的方法。具体是哪个,看方法名就很清楚了,具体每个方法怎么实现的,一眼也就懂了。fmt 是制定格式化的,具体怎么指定在最基础的用法中就有例子,datefmt 是对时间格式的指定。
Filter Objects
这个类是Logger 和Handler 的基类,主要有一个Filter 方法,和一个filters 属性
Handler Objects
叫Handler 的类还真的不少,在SocketServer 中也有看到,具体的功能都在Handler 中.在这里,组合所有的Formatter,和控制log 的输出的方向,继承自Filter.
def __init__(self,level=NOTSET): Filterer.__init__(self) self._name = None self.level = _checkLevel(level) self.formatter = None _addHandlerRef(self) self.createLock()
在init方法中看到,Handler 也有一个属性,通过把自身的属性和LogRecord 的level对比来决定是否处理这个LogRecord 的。每个Handler 都有一个Formatter 属性,其实就是上面介绍的Formatter 。Handler 就是来控制LogRecord 和Formatter 的,它还可以控制输出的方式,在后面会有,StreamHandler,FileHandler等。通过名称也就能明白具体能干什么,这就是编程取名的智慧。
Logger Objects
这个类通常会通过getLogger()或者getLogger(name)来得到,不会直接new 一个出来.它会有info(msg,*args,kwargs),warn(msg,*kwargs)等方法,
def __init__(self,level=NOTSET): Filterer.__init__(self) self.name = name self.level = _checkLevel(level) self.parent = Noneou self.handlers = [] self.disabled = 0
从init方法中能看到handlers 属性,这是一个list ,每个LogRecord 通过Handlers 不同的handlers 就能以不同的格式输出到不同的地方了。每个Logger 可以通过addHandler(hdlr)方法来添加各种Handler,知道这些你就基本可以随意定制化了 下面就是我实现的json 格式的Formater,支持控制台颜色变化,当然前提是你的控制终端支持(Ubuntu14.04测试通过)
import re import logging import socket import json import traceback import datetime import time try: from collections import OrderedDict except ImportError: pass RESERVED_ATTRS = ( 'args','asctime','created','exc_info','exc_text','filename','funcName','levelname','levelno','lineno','module','msecs','message','msg','name','pathname','process','processName','relativeCreated','stack_info','thread','threadName') RESERVED_ATTR_HASH = dict(zip(RESERVED_ATTRS,RESERVED_ATTRS)) COLORS ={ 'HEADER' : '\033[95m','INFO' : '\033[94m','DEBUG' : '\033[92m','WARNING' : '\033[93m','ERROR' : '\033[91m','ENDC' : '\033[0m',} def merge_record_extra(record,target,reserved=RESERVED_ATTR_HASH): for key,value in record.__dict__.items(): if (key not in reserved and not (hasattr(key,"startswith") and key.startswith('_'))): target[key] = value return target def get_host_info(): host_name = '' local_ip = '' try: host_name = socket.gethostname() local_ip = socket.gethostbyname(host_name) except Exception,e: pass return host_name,local_ip class JsonFormatterBase(logging.Formatter): def __init__(self,**kwargs): logging.Formatter.__init__(self,**kwargs) self._required_fields = self.parse() self._skip_fields = dict(zip(self._required_fields,self._required_fields)) self._skip_fields.update(RESERVED_ATTR_HASH) def parse(self): standard_formatters = re.compile(r'\((.+?)\)',re.IGNORECASE) return standard_formatters.findall(self._fmt) def add_fields(self,record ): log_record = {} for field in self._required_fields: log_record[field] = record.__dict__.get(field) host_name,local_ip = get_host_info() log_record[u'@hostName'] = host_name log_record[u'@localIp'] = local_ip return log_record #merge_record_extra(record,log_record,reserved=self._skip_fields) def process_log_record(self,log_record): """ Override this method to implement custom logic on the possibly ordered dictionary. """ try: new_record = OrderedDict() except Exception,e: return log_record key_list = [ 'asctime','@hostName','@localIp','threadName',] for k in key_list: new_record[k] = log_record.get(k) new_record.update(log_record) return new_record def jsonify_log_record(self,log_record): """Returns a json string of the log record.""" return json.dumps(log_record,ensure_ascii=False) def format_col(self,message_str,level_name): """ 是否需要颜色 """ return message_str def formatTime(self,datefmt=None): ct = self.converter(record.created) if datefmt: s = time.strftime(datefmt,ct) else: t = time.strftime("%Y-%m-%d %H:%M:%s",ct) s = "%s.%03d" % (t,record.msecs) return s def format(self,record): if isinstance(record.msg,dict): record.message = record.msg elif isinstance(record.msg,list) or isinstance(record.msg,tuple): record.message = record.msg elif isinstance(record.msg,basestring): record.message = record.getMessage().split('\n') elif isinstance(record.msg,Exception): record.message = traceback.format_exc(record.msg).split('\n') else : record.message = repr(record.msg) if "asctime" in self._required_fields: record.asctime = self.formatTime(record,self.datefmt) # # if record.exc_info and not message_dict.get('exc_info'): # message_dict['message'] = traceback.format_exception(*record.exc_info) log_record = self.add_fields(record) log_record = self.process_log_record(log_record) message_str = self.jsonify_log_record(log_record) message_str = self.format_col(message_str,level_name=record.levelname) return message_str class ConsoleFormater(JsonFormatterBase): def __init__(self,**kwargs): JsonFormatterBase.__init__(self,**kwargs) def format_col(self,level_name): if level_name in COLORS.keys(): message_str = COLORS.get(level_name) + message_str + COLORS.get('ENDC') return message_str def jsonify_log_record(self,log_record): return json.dumps(log_record,ensure_ascii=False,indent=4) class JsonFileFormater(JsonFormatterBase): def __init__(self,**kewars): JsonFormatterBase.__init__(self,**kewars) def jsonify_log_record(self,ensure_ascii=False)
配置
很多时候我们并不是这样自己去实现一些Handler ,Formater,之类的代码,用logging 提供的config 就能做到了,如何写config下面举个例子解释下,
SC_LOGGING_CONF = { "version": 1,"disable_existing_loggers": False,"formatters": { "simple": { "format": "%(asctime)s [%(levelname)s] [%(threadName)s:%(thread)d] [%(name)s:%(lineno)d] - %(message)s" } },"handlers": { "console": { "class": "logging.StreamHandler","level": "DEBUG","formatter": "simple","stream": "ext://sys.stdout" },"info_file_handler": { "class": "logging.handlers.RotatingFileHandler","level": "INFO","filename": PATH + "info-" + date.today().isoformat() + ".log","maxBytes": 10485760,"backupCount": 20,"encoding": "utf8" },"error_file_handler": { "class": "logging.handlers.RotatingFileHandler","level": "ERROR","filename": PATH + "errors-" + date.today().isoformat() + ".log","encoding": "utf8" } },"": { "level": "INFO","handlers": ["console","info_file_handler","error_file_handler"] } } }
首先定义了一个formater 叫simaple,然后定义了三个Handler,分别是输出到控制台,输出到文件和info,error的。
logging.config.dictConfig(CONfig.SC_LOGGING_CONF)
通过这句就能让这些配置产生效果了,这也是config.py做的事情,不需要写很多代码也能定制个性化的log.。
以上就是本文的全部内容,希望对大家的学习有所帮助。
总结
以上是编程之家为你收集整理的说一说Python logging全部内容,希望文章能够帮你解决说一说Python logging所遇到的程序开发问题。
如果您也喜欢它,动动您的小指点个赞吧