不幸的是,在Python中遍历字符串相当慢。对于这种事情,正则表达式的速度要快一个数量级。您只需要自己构建角色类即可。该 unicodedata 模块是这个相当有帮助,尤其是 unicodedata.category() 函数。有关类别的说明,请参见Unicode字符数据库。
import unicodedata, re, itertools, sys
all_chars = (chr(i) for i in range(sys.maxunicode))
categories = {'Cc'}
control_chars = ''.join(c for c in all_chars if unicodedata.category(c) in categories)
# or equivalently and much more efficiently
control_chars = ''.join(map(chr, itertools.chain(range(0x00,0x20), range(0x7f,0xa0))))
control_char_re = re.compile('[%s]' % re.escape(control_chars))
def remove_control_chars(s):
return control_char_re.sub('', s)
对于Python2
import unicodedata, re, sys
all_chars = (unichr(i) for i in xrange(sys.maxunicode))
categories = {'Cc'}
control_chars = ''.join(c for c in all_chars if unicodedata.category(c) in categories)
# or equivalently and much more efficiently
control_chars = ''.join(map(unichr, range(0x00,0x20) + range(0x7f,0xa0)))
control_char_re = re.compile('[%s]' % re.escape(control_chars))
def remove_control_chars(s):
return control_char_re.sub('', s)
对于某些用例,最好使用其他类别(例如,来自 控制 组的所有类别),尽管这可能会减慢处理时间并显着增加内存使用。每个类别的字符数:
从注释中添加建议。