gsutil TypeError: cannot pickle '_io.TextIOWrapper' object
Sun, Aug 30, 2020
2-minute read
If you’re trying to copy files or otherwise interact with google cloud storage buckets using gsutil
and end up having an error similar to the one below, there is a solution!
~ gsutil -m cp *.avi gs://videoworld-storage/
Traceback (most recent call last):
File "/Users/echo/google-cloud-sdk/platform/gsutil/gsutil", line 21, in <module>
gsutil.RunMain()
File "/Users/echo/google-cloud-sdk/platform/gsutil/gsutil.py", line 123, in RunMain
sys.exit(gslib.__main__.main())
File "/Users/echo/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 429, in main
return _RunNamedCommandAndHandleExceptions(
File "/Users/echo/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 767, in _RunNamedCommandAndHandleExceptions
_HandleUnknownFailure(e)
File "/Users/echo/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 625, in _RunNamedCommandAndHandleExceptions
return command_runner.RunNamedCommand(command_name,
File "/Users/echo/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 411, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/Users/echo/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 1190, in RunCommand
self.Apply(_CopyFuncWrapper,
File "/Users/echo/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1499, in Apply
self._ParallelApply(
File "/Users/echo/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1719, in _ParallelApply
self._CreateNewConsumerPool(process_count, thread_count,
File "/Users/echo/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1384, in _CreateNewConsumerPool
p.start()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
return Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_io.TextIOWrapper' object
The error is triggered when attempting to use -m
flag and the system attempting to use Python3.8. The solution is to force the command to use Python3.7 or other compatible version.
This is accomplished by setting an environment variable either in the shell you need it in, or by adding the lines to .bashrc
or .zshrc
or to the startup script of your choice of shell.
$ export CLOUDSDK_PYTHON=/usr/bin/python3 # on mac
$ export CLOUDSDK_PYTHON=/usr/bin/python3.7 # on linux
Now gsutil -m
will work and your command will use threading!