Background

I had headaches getting some pickled files I created in python 2.7 to actually open in python 3.4. I was getting an error message that looked like this:

UnicodeDecodeError: 'ascii' codec can't decode .....

The pickle contained a dictionary, with string keys. So at first I thought it had something to do with python 3 using utf-8 instead of ascii as the default encoding for strings.

So I tried, the following without any success.

filepath = "/path/to/file.pickle"
with open(filepath, mode = "rb") as fileObj:
    obj = pickle.load(fileObj, encoding="utf-8")
return obj

I then tried the following, which allowed me to open up the contents of the pickle file.

filepath = "/path/to/file.pickle"
with open(filepath, mode = "rb") as fileObj:
    obj = pickle.load(fileObj, encoding="latin1")
return obj

But then when I saved the modified contents back into the pickle file using:

filepath = "/path/to/file.pickle"
with open(filepath, mode="wb") as fileObj:
    pickle.dump(obj, fileObj, protocol=-1)

I was unable to open the pickle in python 2.7.

Solution

It turns out that python 3 has introduced two new protocol versions for creating pickle files. By default, I had been letting the pickle.dump() function use the latest protocol available. The problem is that the latest protocol is not the same in python 2.7 and 3.x series.

The way to save pickle files in a way that is compatible with both python 2.7 and 3.x series is by setting the protocol to 2. This is the highest protocol version that is shared by both python 2.7 and 3.x.

filepath = "/path/to/file.pickle"
with open(filepath, mode="wb") as fileObj:
    pickle.dump(obj, fileObj, protocol=2)

Protocol versions

Here is a summary of the different protocol values, and what versions of python they are compatible with.

• 0
  • This is the original pickle protocol. It encodes the pickles in ASCII file format.
  • Is backwards compatible with earlier versions of Python.
• 1
  • This was the first protocol to save the files in binary format.
  • Also compatible with earlier versions of Python.
• 2
  • Provides much more efficient pickling of newer classes introduced in python 2.x series.
  • compatible with Python >= 2.3.
  • Use this value to be compatible between py2.7 and py3.x
• 3
  • Python >= 3.x series
• 4
  • Python >= 3.4
• -1
  • Uses the highest (latest) protocol available given your current version of python.
  • So if you are using python 2.7, then this will use protocol 2.
  • If you are using python 3.4, then this will use protocol 4.