ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: ‘/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/py4j’Consider using the --user option or check the permissions.
Error:
[ec2-user@ip-172-31-55-239 ~]$ pip install pyspark
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting pyspark
Downloading https://files.pythonhosted.org/packages/27/67/5158f846202d7f012d1c9ca21c3549a58fd3c6707ae8ee823adcaca6473c/pyspark-3.0.2.tar.gz (204.8MB)
|████████████████████████████████| 204.8MB 11kB/s
Collecting py4j==0.10.9 (from pyspark)
Downloading https://files.pythonhosted.org/packages/9e/b6/6a4fb90cd235dc8e265a6a2067f2a2c99f0d91787f06aca4bcf7c23f3f80/py4j-0.10.9-py2.py3-none-any.whl (198kB)
|████████████████████████████████| 204kB 35.5MB/s
Building wheels for collected packages: pyspark
Building wheel for pyspark (setup.py) ... done
Created wheel for pyspark: filename=pyspark-3.0.2-py2.py3-none-any.whl size=205186689 sha256=da3541d6b060509c9e3fabad7631ffa0c3d64c63798ad3cb149f8408e53c5c37
Stored in directory: /home/ec2-user/.cache/pip/wheels/8b/09/da/c1f2859bcc86375dc972c5b6af4881b3603269bcc4c9be5d16
Successfully built pyspark
Installing collected packages: py4j, pyspark
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/py4j'
Consider using the `--user` option or check the permissions.
Workaround:
Specify –user option with your command or give permissions to the directory where spark is downloaded
[ec2-user@ip-172-31-55-239 ~]$ pip install pyspark --user
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting pyspark
Collecting py4j==0.10.9 (from pyspark)
Using cached https://files.pythonhosted.org/packages/9e/b6/6a4fb90cd235dc8e265a6a2067f2a2c99f0d91787f06aca4bcf7c23f3f80/py4j-0.10.9-py2.py3-none-any.whl
Installing collected packages: py4j, pyspark
Successfully installed py4j-0.10.9 pyspark-3.0.2