ubuntu - Building Python and more on missing modules

Question

I have another thread asking help on "missing zlib". With the nice help the problem has been resolved (almost).

Now I am interested in building Python myself (on Ubuntu 10.10).

A few important questions have caught my attention:

1. After building Python (say 2.7.1), do I need to rebuild Python if I have missing modules?

2. Is there a way to find out what modules will be missing prior to building Python? Say sqlite3. I have sqlite3 installed for the system default (Python 2.6.6), and I can import that into Python 2.6.6 shell. Now I use pythonbrew to build 2.7.1, and in the shell I cannot import sqlite3 because _sqlite3 is not available. I am sure there are a few more important one missing which I need for future development (such as Django..).

I am willing to learn how to build without using pythonbrew.

Please share with me your experience in building another version of Python, and how would you address the problem of missing modules? Is there a practical solution to building Python?

I have never bothered building one myself, so please bear with me. I am beginning to realize the importance of learning and building one myself! Thank you very much!

---

EDIT

First I thank you all of your inputs. They meant a lot. I did the building.

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _curses            _curses_panel   
_tkinter           bsddb185           bz2             
dbm                gdbm               readline        
sunaudiodev        _sqlite3                                    
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

I got sqlite3 and readline away by

sudo apt-get install libreadline6 libreadline6-dev
sudo apt-get install libsqlite3-dev

I tried to import them, but still "no named module xxxx".

At AskUbuntu I actually asked people how to get previous commands because I couldn't use that feature when I am in Python 2.7.1 shell. I believe it's due to readline. Readline

I installed the Python-2.7.1 under this directory: /home/jwxie518/python27/

I looked into setup.py, I found the following lines:

# The sqlite interface
sqlite_setup_debug = False   # verbose debug prints from this script?

# We hunt for #define SQLITE_VERSION "n.n.n"
# We need to find >= sqlite version 3.0.8
sqlite_incdir = sqlite_libdir = None
sqlite_inc_paths = [ '/usr/include',
                     '/usr/include/sqlite',
                     '/usr/include/sqlite3',
                     '/usr/local/include',
                     '/usr/local/include/sqlite',
                     '/usr/local/include/sqlite3',
                   ]

All the paths listed above do not exist. So I guess I have to install sqlite3 manually? I got another reference here (it's in Chinese, however)

# Download the latest and extract
# Go into the extracted directory
./configure --prefix=/home/jwxie518/python27/python
make && make install
# Then edit python-2.7 's setup.py before rebuild it
# Sample (add these two lines to the end....)
'~/share/software/python/sqlite-3.6.20/include',
'~/share/software/python/sqlite-3.6.20/include/sqlite3',

# Then rebuild python like how we did before

I went into my directory where I installed sqlite3. I found include/sqlite3.h only. So I went back and check /usr/include/. I can only find sqlite3.h too.

So what is going on here? Readline is also non-importable.

---

3RD EDIT I started everything over, except I didn't reinstall sqlite3.

# Extract Python-2.7.1
# cd into Python-2.7.1
# ./configure
make >make.out 2>&1
less make.out

make.out is here: http://pastebin.com/raw.php?i=7k3BfxZQ

I still couldn't import sqlite3. So I went into setup.py and made changes:

# We hunt for #define SQLITE_VERSION "n.n.n"
# We need to find >= sqlite version 3.0.8
sqlite_incdir = sqlite_libdir = None
sqlite_inc_paths = [ '/usr/include',
                     '/usr/include/sqlite',
                     '/usr/include/sqlite3',
                     '/usr/local/include',
                     '/usr/local/include/sqlite',
                     '/usr/local/include/sqlite3',
                     '/home/jwxie518/python-mod/include/sqlite',
                     '/home/jwxie518/python-mod/include/sqlite3',
                   ]

Then again, ran everything over (this time I also did make clean)

Output is here: http://pastebin.com/raw.php?i=8ZKgAcWn

According to the output, I don't think the custom path is included.... (for complete output please go to the link above and search for sqlite)

build/temp.linux-i686-2.7/home/jwxie518/Python-2.7.1/Modules/_sqlite/util.o -L/usr/lib -L/usr/local/lib -Wl,-R/usr/lib -lsqlite3 -o build/lib.linux-i686-2.7/_sqlite3.so

I still cannot import sqlite3.

THanks!

---

Thank you very much, Michael Dillon, for helping me out. Your tutorial was neat and clear.

I solved the problem as soon as I realized whenever I tried Python-2.7.1, I was actually using the one installed by Pythonbrew.

The moral of the story is read all the errors. I neglected the errors generated by importing sqlite3. The one installed by Pythonbrew didn't have sqlite3 installed. The development package for sqlite3 was installed after Pythonbrew installed the Python-2.7.1

Thanks.

Answer 1

Here is how to build Python and fix any dependencies. I am assuming that you want this Python to be entirely separate from the Ubuntu release Python, so I am specifying the --prefix option to install it all in /home/python27 using the standard Python layout, i.e. site-packages instead of dist-packages.

1. Get the .tar.gz file into your own home directory.
2. tar zxvf Py*.tar.gz
3. cd Py*1
4. ./configure --prefix=/home/python27
5. make
6. make install

Step 5 is the important one. At the end, it will display a list of any modules that could not be built properly. Often you can fix this by installing an Ubuntu package, and rerunning make.

a. sudo apt-get install something-dev
b. make

It is pretty common to have a problem because you are missing the -dev addon to some module or other. But sometimes you should start over like this:

a. make clean
b. ./configure --prefix=/home/python27
c. make

Starting over never hurts if you are unsure. An important note about step 6. I am not using sudo on this command which means that you will need to have the /home/python27 directory already created with the appropriate ownership.

Don't hesitate to try out ./configure --help |less before building something because there may be interesting options that you could use. One time on a minimal distro I had to do --with-dbmliborder=gdbm:bdb in order to get gdbm working. When you run ./configure, the last few lines will tell you where it put the information that it learned. In the case of Python, Modules/Setup has been useful to figure out how to get a module to build.

Another useful thing is to make clean and then run make >make.out 2>&1 to capture all the output from the full make process. Then, after it is complete, use less or an editor to look for the details on a problem module such as _sqlite. For instance, check all the -I options that are passed to gcc. If the correct include directory is not on the list that would cause a problem. You can edit setup.py to change the list of include directories.

In the past it was more common to have library problems that would be fixed by logging out, logging in again, and running "sudo ldconfig" before doing a complete rebuild.