python - Pip Requirements.txt --global-option causing installation errors with other packages. "option not recognized"

Question

I'm having difficulties with the --global-option and --install-option settings for a requirements.txt file. Specifying the options for one library is causing other libraries installs to fail.

I'm trying to install Python libraries "grab" and "pycurl". I need to specify that pycurl be installed with option: "--with-nss". I can replicate the error on a completely clean virtual enviroment.

On a new virtual environment With requirements.txt containing:

grab==0.6.25
pycurl==7.43.0 --install-option='--with-nss'

Then installing with:

pip install -r requirements.txt

The following errors will occur.

Installing collected packages: lxml, pycurl, pytils, six, user-agent, weblib, selection, grab
  Running setup.py install for lxml ... done
  Running setup.py install for pycurl ... done
  Running setup.py install for pytils ... error
    Complete output from command /home/ec2-user/test/env/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-8GvFzA/pytils/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('
', '

'), __file__, 'exec'))" install --record /tmp/pip-BCG3Wl-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/ec2-user/test/env/include/site/python2.7/pytils --with-nss:
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help

    error: option --with-nss not recognized

    ----------------------------------------
Command "/home/ec2-user/test/env/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-8GvFzA/pytils/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('
', '
'), __file__, 'exec'))"
install --record /tmp/pip-BCG3Wl-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/ec2-user/test/env/include/site/python2.7/pytils --with-nss" failed with error code 1 in /tmp/pip-build-8GvF
zA/pytils/

My best guess at the root cause is that the option "--with-nss" is being passed to all libraries that require pycurl, and preventing install. The pytils installation fails even though the pycurl install works fine.

Is there anyway to only pass the install options to the one library?

I'm setting this up on an Amazon Elastic Beanstalk instance, so there is no option to manually run each line of the requirements.txt file - the whole install gets run at start up of the application.

Sources for --global-option and --install-option (which I think shouldn't do this): How to maintain pip install options in requirements file made by pip freeze? https://github.com/pypa/pip/blob/develop/docs/reference/pip_install.rst#id28

Answer 1

Your problem comes from the fact that PIP version on EC2 with EB is quite old and does not understand your options.

1. Update pip lib to latest available version with EB commands:

project_dir/.ebextensions/02-python.config:

...
commands:
  01_upgrade_pip_for_venv:
    command: "/opt/python/run/venv/bin/pip install --upgrade pip"
...

2. Now you can leave options in requirement.txt since new version of pip will be able to work with it.

project_dir/requirements.txt:

...
pycurl==7.43.0 --global-option="--with-nss"
...

3. (This may be redundant) Set option in EB console user interfaces or by eb CLI with command:

   eb setenv PYCURL_SSL_LIBRARY=nss

4. Push changes to repository and Rebuild. You may have errors since execution is controlled from external scope and started with old version of PIP. Entry point of execution is outside of app on EC2 instance so I'm not sure how to bring solution that would work from scope of hooks on first deployment... But all you have to do is to deploy again, and it will use proper version of PIP, so it will work from now on, till next rebuild...