您可以使用chunksize
参数以块的形式读取CSV文件,并将每个块附加到HDF文件中:
hdf_key = 'hdf_key'
df_cols_to_index = [...] # list of columns (labels) that should be indexed
store = pd.hdfstore(hdf_filename)
for chunk in pd.read_csv(csv_filename, chunksize=500000):
# don't index data columns in each iteration - we'll do it later ...
store.append(hdf_key, chunk, data_columns=df_cols_to_index, index=False)
# index data columns in hdfstore
store.create_table_index(hdf_key, columns=df_cols_to_index, optlevel=9, kind='full')
store.close()