ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: ‘/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/py4j’Consider using the --user option or check the permissions.

Error:

[ec2-user@ip-172-31-55-239 ~]$ pip install pyspark
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting pyspark
  Downloading https://files.pythonhosted.org/packages/27/67/5158f846202d7f012d1c9ca21c3549a58fd3c6707ae8ee823adcaca6473c/pyspark-3.0.2.tar.gz (204.8MB)
     |████████████████████████████████| 204.8MB 11kB/s
Collecting py4j==0.10.9 (from pyspark)
  Downloading https://files.pythonhosted.org/packages/9e/b6/6a4fb90cd235dc8e265a6a2067f2a2c99f0d91787f06aca4bcf7c23f3f80/py4j-0.10.9-py2.py3-none-any.whl (198kB)
     |████████████████████████████████| 204kB 35.5MB/s
Building wheels for collected packages: pyspark
  Building wheel for pyspark (setup.py) ... done
  Created wheel for pyspark: filename=pyspark-3.0.2-py2.py3-none-any.whl size=205186689 sha256=da3541d6b060509c9e3fabad7631ffa0c3d64c63798ad3cb149f8408e53c5c37
  Stored in directory: /home/ec2-user/.cache/pip/wheels/8b/09/da/c1f2859bcc86375dc972c5b6af4881b3603269bcc4c9be5d16
Successfully built pyspark
Installing collected packages: py4j, pyspark
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/py4j'
Consider using the `--user` option or check the permissions.

Workaround:

Specify –user option with your command or give permissions to the directory where spark is downloaded

[ec2-user@ip-172-31-55-239 ~]$ pip install pyspark  --user
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting pyspark
Collecting py4j==0.10.9 (from pyspark)
  Using cached https://files.pythonhosted.org/packages/9e/b6/6a4fb90cd235dc8e265a6a2067f2a2c99f0d91787f06aca4bcf7c23f3f80/py4j-0.10.9-py2.py3-none-any.whl
Installing collected packages: py4j, pyspark
Successfully installed py4j-0.10.9 pyspark-3.0.2

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s