假设您知道最终的数组arr
永远不会大于5000x10。然后,您可以预分配最大大小的数组,在遍历循环时将其填充数据,然后arr.resize
在退出循环后将其缩减为发现的大小。
下面的测试表明,无论数组的最终大小如何,这样做都会比构造中间python列表快一点。
同样,arr.resize
取消分配未使用的内存,因此最终的(虽然可能不是中间的)内存占用空间小于所使用的内存占用空间python_lists_to_array
。
这表明numpy_all_the_way
速度更快:
% python -mtimeit -s"import test" "test.numpy_all_the_way(100)"
100 loops, best of 3: 1.78 msec per loop
% python -mtimeit -s"import test" "test.numpy_all_the_way(1000)"
100 loops, best of 3: 18.1 msec per loop
% python -mtimeit -s"import test" "test.numpy_all_the_way(5000)"
10 loops, best of 3: 90.4 msec per loop
% python -mtimeit -s"import test" "test.python_lists_to_array(100)"
1000 loops, best of 3: 1.97 msec per loop
% python -mtimeit -s"import test" "test.python_lists_to_array(1000)"
10 loops, best of 3: 20.3 msec per loop
% python -mtimeit -s"import test" "test.python_lists_to_array(5000)"
10 loops, best of 3: 101 msec per loop
这显示numpy_all_the_way
使用更少的内存:
% test.py
Initial memory usage: 19788
After python_lists_to_array: 20976
After numpy_all_the_way: 20348
test.py:
import numpy as np
import os
def memory_usage():
pid = os.getpid()
return next(line for line in open('/proc/%s/status' % pid).read().splitlines()
if line.startswith('VmSize')).split()[-2]
N, M = 5000, 10
def python_lists_to_array(k):
list_of_arrays = list(map(lambda x: x * np.ones(M), range(k)))
arr = np.array(list_of_arrays)
return arr
def numpy_all_the_way(k):
arr = np.empty((N, M))
for x in range(k):
arr[x] = x * np.ones(M)
arr.resize((k, M))
return arr
if __name__ == '__main__':
print('Initial memory usage: %s' % memory_usage())
arr = python_lists_to_array(5000)
print('After python_lists_to_array: %s' % memory_usage())
arr = numpy_all_the_way(5000)
print('After numpy_all_the_way: %s' % memory_usage())