python - Installing numpy on Amazon EC2

Question

I am having trouble installing numpy on an Amazon EC2 server. I have tried using easy_install, pip, pip inside a virtual env, pip inside another virtual env using python 2.7...

Every time I try, it fails with the error: gcc: internal compiler error: Killed (program cc1), and then further down the line I get a bunch of python errors, with easy_install I get: ImportError: No module named numpy.distutils, and with pip I get: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 72: ordinal not in range(128).

The EC2 instance is running kernel 3.4.43-43.43.amzn1.x86_64. Has anybody solved this problem? Numpy has always been hard for me to install, but I can usually figure it out... at this point I don't care whether it is in it's own virtualenv, I just want to get it installed.

Answer 1

Requirements for installing Numpy

• c compiler (gcc)
• fortran compiler (gfortran)
• python header files (2.4.x - 3.2.x)
• Strongly recommended BLAS or LAPACK

I wrote a script to install virtualenv and scikit-learn along with all the dependencies. You can follow up to the numpy install, which is pretty straight forward. I copied the relevant code below.

sudo yum -y install gcc-c++ python27-devel atlas-sse3-devel lapack-devel
wget https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.11.2.tar.gz
tar xzf virtualenv-1.11.2.tar.gz 
python27 virtualenv-1.11.2/virtualenv.py sk-learn
. sk-learn/bin/activate
pip install numpy

Just copy/paste, hit enter, (get a cup of coffee) and you're ready to go with virtualenv and numpy on EC2.

If you want to verify that numpy found the optimized linear algebra libraries, run:

(sk-learn)[ec2-user@ip-10-99-17-223 ~]$ python -c "import numpy; numpy.show_config()"

if you see something similar to the following you're all set.

atlas_threads_info:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib64/atlas-sse3']
    define_macros = [('ATLAS_INFO', '""3.8.4""')]
    language = f77
    include_dirs = ['/usr/include']
blas_opt_info:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib64/atlas-sse3']
    define_macros = [('ATLAS_INFO', '""3.8.4""')]
    language = c
    include_dirs = ['/usr/include']
atlas_blas_threads_info:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib64/atlas-sse3']
    define_macros = [('ATLAS_INFO', '""3.8.4""')]
    language = c
    include_dirs = ['/usr/include']
lapack_opt_info:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib64/atlas-sse3']
    define_macros = [('ATLAS_INFO', '""3.8.4""')]
    language = f77
    include_dirs = ['/usr/include']
lapack_mkl_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
mkl_info:
  NOT AVAILABLE

For a more detailed explanation, you can read installing-scikit-learn-on-amazon-ec2. I wrote the blog post specifically to remember the installation steps and have a short how-to guide. I try to keep the post and the install script up to date.