python - How to use the __import__ function to import a name from a submodule?

Question

I'm trying to replicate from foo.bar import object using the __import__ function and I seem to have hit a wall.

A simpler case from glob import glob is easy: glob = __import__("glob").glob

The problem I'm having is that I am importing a name from a subpackage (i.e. from foo.bar):

So what I'd like is something like

string_to_import = "bar"
object = __import__("foo." + string_to_import).object

But this just imported the top-level foo package, not the foo.bar subpackage:

__import__("foo.bar")
<module 'foo' from 'foo/__init__.pyc'>

Answer 1

How to use python's __import__() function properly?

There are two kinds of uses:

• direct importing
• a hook to alter import behavior

For the most part, you don't really need to do either.

For user-space importing

Best practice is to use importlib instead. But if you insist:

Trivial usage:

>>> sys = __import__('sys')
>>> sys
<module 'sys' (built-in)>

Complicated:

>>> os = __import__('os.path')
>>> os
<module 'os' from '/home/myuser/anaconda3/lib/python3.6/os.py'>
>>> os.path
<module 'posixpath' from '/home/myuser/anaconda3/lib/python3.6/posixpath.py'>

If you want the rightmost child module in the name, pass a nonempty list, e.g. [None], to fromlist:

>>> path = __import__('os.path', fromlist=[None])
>>> path
<module 'posixpath' from '/home/myuser/anaconda3/lib/python3.6/posixpath.py'>

Or, as the documentation declares, use importlib.import_module:

>>> importlib = __import__('importlib')
>>> futures = importlib.import_module('concurrent.futures')
>>> futures
<module 'concurrent.futures' from '/home/myuser/anaconda3/lib/python3.6/concurrent/futures/__init__.py'>

Documentation

The docs for __import__ are the most confusing of the builtin functions.

__import__(...)
    __import__(name, globals=None, locals=None, fromlist=(), level=0) -> module

    Import a module. Because this function is meant for use by the Python
    interpreter and not for general use it is better to use
    importlib.import_module() to programmatically import a module.

    The globals argument is only used to determine the context;
    they are not modified.  The locals argument is unused.  The fromlist
    should be a list of names to emulate ``from name import ...'', or an
    empty list to emulate ``import name''.
    When importing a module from a package, note that __import__('A.B', ...)
    returns package A when fromlist is empty, but its submodule B when
    fromlist is not empty.  Level is used to determine whether to perform 
    absolute or relative imports. 0 is absolute while a positive number
    is the number of parent directories to search relative to the current module.

If you read it carefully, you get the sense that the API was originally intended to allow for lazy-loading of functions from modules. However, this is not how CPython works, and I am unaware if any other implementations of Python have managed to do this.

Instead, CPython executes all of the code in the module's namespace on its first import, after which the module is cached in sys.modules.

__import__ can still be useful. But understanding what it does based on the documentation is rather hard.

Full Usage of __import__

To adapt the full functionality to demonstrate the current __import__ API, here is a wrapper function with a cleaner, better documented, API.

def importer(name, root_package=False, relative_globals=None, level=0):
    """ We only import modules, functions can be looked up on the module.
    Usage: 

    from foo.bar import baz
    >>> baz = importer('foo.bar.baz')

    import foo.bar.baz
    >>> foo = importer('foo.bar.baz', root_package=True)
    >>> foo.bar.baz

    from .. import baz (level = number of dots)
    >>> baz = importer('baz', relative_globals=globals(), level=2)
    """
    return __import__(name, locals=None, # locals has no use
                      globals=relative_globals, 
                      fromlist=[] if root_package else [None],
                      level=level)

To demonstrate, e.g. from a sister package to baz:

baz = importer('foo.bar.baz')    
foo = importer('foo.bar.baz', root_package=True)
baz2 = importer('bar.baz', relative_globals=globals(), level=2)

assert foo.bar.baz is baz is baz2

Dynamic access of names in the module

To dynamically access globals by name from the baz module, use getattr. For example:

for name in dir(baz):
    print(getattr(baz, name))

Hook to alter import behavior

You can use __import__ to alter or intercept importing behavior. In this case, let's just print the arguments it gets to demonstrate we're intercepting it:

old_import = __import__

def noisy_importer(name, locals, globals, fromlist, level):
    print(f'name: {name!r}')
    print(f'fromlist: {fromlist}')
    print(f'level: {level}')
    return old_import(name, locals, globals, fromlist, level)

import builtins
builtins.__import__ = noisy_importer

And now when you import you can see these important arguments.

>>> from os.path import join as opj
name: 'os.path'
fromlist: ('join',)
level: 0
>>> opj
<function join at 0x7fd08d882618>

Perhaps in this context getting the globals or locals could be useful, but no specific uses for this immediately come to mind.