如何使用 append 和 pickle？

问题描述：

我需要将其附加到 pickle 文件中（因为我一次没有完整的字典）。因此，为了执行相同的操作，我编写了以下代码：

 import pickle

 p = {}
 p[1] = 2
 q = {}
 q['a'] = p
 p = {}
 p[2] = 0
 pickle.dump(q, open("save12.p", "ab"))
 f = {}
 f['b'] = p
 pickle.dump(f, open("save12.p", "ab"))

然而，当我加载 pickle 文件时，我没有f在那里找到字典的值。

我该如何将内容附加到 pickle 文件中？

此外，由于我在 Windows 上工作，因此“dbm”之类的数据库不能满足我的需要。

解决方案 1：

Pickle 流是完全独立的，因此解封一次只会解封一个对象。

因此，要解开多个流，您应该反复解开文件，直到收到 EOFError：

>>> f=open('a.p', 'wb')
>>> pickle.dump({1:2}, f)
>>> pickle.dump({3:4}, f)
>>> f.close()
>>> 
>>> f=open('a.p', 'rb')
>>> pickle.load(f)
{1: 2}
>>> pickle.load(f)
{3: 4}
>>> pickle.load(f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
EOFError

所以你的 unpickle 代码可能看起来像

import pickle
objs = []
while 1:
    try:
        objs.append(pickle.load(f))
    except EOFError:
        break

解决方案 2：

#To append to a pickle file
import pickle

p={1:2}
q={3:4}
filename="picklefile"
with open(filename, 'a+') as fp:
    pickle.dump(p,fp)
    pickle.dump(q,fp)


#To load from pickle file
data = []
with open(filename, 'rb') as fr:
    try:
        while True:
            data.append(pickle.load(fr))
    except EOFError:
        pass

解决方案 3：

如果每次追加时都重写整个列表，那么len(list)^2写入磁盘的总字节数就是这个数。如果只是追加到文件，那么保存所有项目所需的总时间就会减少。

如果有人好奇的话，保存加速确实存在n^2，而加载减速则很小。

naive_append:   8.387 seconds
naive_load:     0.024 seconds
smart_append:   0.042 seconds
smart_load:     0.026 seconds

import os, pickle, time, numpy as np

def naive_append(filename, obj):
    objs = []
    if os.path.exists(filename):
        with open(filename, 'rb') as f:
            objs = pickle.load(f)
    objs.append(obj)
    with open(filename, 'wb') as f:
        pickle.dump(objs, f)

def naive_load(filename):
    with open(filename, 'rb') as f:
        return pickle.load(f)

def smart_append(filename, obj):
    with open(filename, 'ab+') as f:
        pickle.dump(obj, f)

def smart_load(filename):
    with open(filename, 'rb') as f:
        while True:
            try:
                yield pickle.load(f)
            except EOFError:
                break

def bench(appender, loader):
    filename="test.log.pkl"
    if os.path.exists(filename):
        os.remove(filename)
    arrays = [np.random.randn(500, 500) for _ in range(100)]

    started = time.time()
    for arr in arrays:
        appender(filename, arr)
    print(f"{appender.__name__}:    {time.time() - started:.3f} seconds")

    started = time.time()
    got_back = list(loader(filename))
    print(f"{loader.__name__}:    {time.time() - started:.3f} seconds")

    assert len(arrays) == len(got_back)
    assert all(a.sum() == b.sum() for a, b in zip(arrays, got_back))

bench(naive_append, naive_load)
bench(smart_append, smart_load)