python 2.7 - PyInstaller does NOT work when including Pysnmp

Question

Just trying one of the official documentation site examples:

from pysnmp.hlapi import *

errorIndication, errorStatus, errorIndex, varBinds = next(
    getCmd(SnmpEngine(),
          CommunityData('public'),
          UdpTransportTarget(('192.168.1.14', 161)),
          ContextData(),
          ObjectType(ObjectIdentity('1.3.6.1.2.1.1.1.0')))
)

if errorIndication:
    print(errorIndication)
elif errorStatus:
    print('%s at %s' % (
           errorStatus.prettyPrint(),
           errorIndex and varBinds[int(errorIndex)-1][0] or '?'
       )
    )
else:
    for varBind in varBinds:
       print(' = '.join([ x.prettyPrint() for x in varBind ]))

we can see it does work if its executed through the Python interpreter:

 (Compiler)[user@machine testSNMP]$ python testSNMP.py 
 SNMPv2-MIB::sysDescr."0" = 48-port 10/100/1000 Gigabit Switch

However, if I try to "freeze" it with PyInstaller (PyInstaller --onefile) I get the following error once it is executed:

  (Compiler)[user@machine testSNMP]$ /lab/testSNMP/dist/testSNMP
Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "/lab/testSNMP/build/testSNMP/out00-PYZ.pyz/pysnmp.entity.engine", line 83, in __init__
  File "/lab/testSNMP/build/testSNMP/out00-PYZ.pyz/pysnmp.smi.builder", line 359, in importSymbols
pysnmp.smi.error.MibNotFoundError: No module __SNMP-FRAMEWORK-MIB loaded at <pysnmp.smi.builder.MibBuilder instance at 0x179f518>

It seems that mandatory files in pysnmp/smi/mibs are not explicitly imported to be used. That's why I've split the build procedure in phases. Fist, create the spec file.

(Compiler)[user@machine testSNMP]$ pyi-makespec --onefile getInterfaces.py
wrote /lab/testSNMP/getInterfaces.spec
now run pyinstaller.py to build the executable

Then, I've edited it to import required files following advices from this other stack post (Can't get pysnmp to work with pyinstaller):

# -*- mode: python -*-
import PyInstaller.hooks.hookutils
hiddenimports = ['pysnmp.smi.exval','pysnmp.cache'] + PyInstaller.hooks.hookutils.collect_submodules('pysnmp.smi.mibs') + PyInstaller.hooks.hookutils.collect_submodules('pysnmp.smi.mibs.instances')
a = Analysis(['testSNMP.py'],
            pathex=['/lab/testSNMP'],
            hiddenimports=hiddenimports,
            hookspath=None,
            runtime_hooks=None)
x = Tree('/virtualenvs/Compiler/lib/python2.7/site-packages/pysnmp/smi/mibs',prefix='pysnmp/smi/mibs',excludes='.py')
pyz = PYZ(a.pure)
exe = EXE(pyz,
         a.scripts,
         a.binaries,
         a.zipfiles,
         a.datas,
         x,
         name='testSNMP',
         debug=False,
         strip=None,
         upx=True,
         console=True )

But then, once it's built and executed, I'm getting this other error:

(Compiler)[user@machine testSNMP]$ /lab/testSNMP/dist/testSNMP
Traceback (most recent call last):
  File "<string>", line 15, in <module>
  File "/lab/testSNMP/build/testSNMP/out00-PYZ.pyz/pysnmp.hlapi.asyncore.sync.cmdgen", line 98, in getCmd
  File "/lab/testSNMP/build/testSNMP/out00-PYZ.pyz/pysnmp.hlapi.asyncore.cmdgen", line 135, in getCmd
  File "/lab/testSNMP/build/testSNMP/out00-PYZ.pyz/pysnmp.hlapi.varbinds", line 30, in makeVarBinds
  File "/lab/testSNMP/build/testSNMP/out00-PYZ.pyz/pysnmp.smi.rfc1902", line 689, in resolveWithMib
  File "/lab/testSNMP/build/testSNMP/out00-PYZ.pyz/pysnmp.smi.rfc1902", line 299, in resolveWithMib
  File "/lab/testSNMP/build/testSNMP/out00-PYZ.pyz/pysnmp.smi.compiler", line 44, in addMibCompiler
  File "/lab/testSNMP/build/testSNMP/out00-PYZ.pyz/pysmi.parser.smi", line 21, in __init__
  File "/lab/testSNMP/build/testSNMP/out00-PYZ.pyz/pysmi.lexer.smi", line 83, in __init__
  File "/lab/testSNMP/build/testSNMP/out00-PYZ.pyz/pysmi.lexer.smi", line 100, in reset
  File "/lab/testSNMP/build/testSNMP/out00-PYZ.pyz/ply.lex", line 915, in lex
  File "/lab/testSNMP/build/testSNMP/out00-PYZ.pyz/ply.lex", line 577, in validate_all
  File "/lab/testSNMP/build/testSNMP/out00-PYZ.pyz/ply.lex", line 819, in validate_rules
  File "/lab/testSNMP/build/testSNMP/out00-PYZ.pyz/ply.lex", line 830, in validate_module
  File "/lab/testSNMP/build/testSNMP/out00-PYZ.pyz/inspect", line 690, in getsourcelines
  File "/lab/testSNMP/build/testSNMP/out00-PYZ.pyz/inspect", line 538, in findsource
IOError: could not get source code

What else can I do? Thanks in advance!

Answer 1

To make Jeremy's answer portable, you can modify your specfile like this diff:

+ import os.path
+ import pysmi
+ import pysnmp.smi.mibs

+ def module_base(module):
+   return os.path.abspath(os.path.join(module.__file__, os.pardir))

  coll = COLLECT(
      exe,
      a.binaries,
      a.zipfiles,
      a.datas,
+     Tree(module_base(pysmi), prefix='pysmi'),
+     Tree(module_base(pysnmp.smi.mibs), prefix='pysnmp/smi/mibs'),
      ...

EDIT: turn out adding these to hiddenimports solved the problem for me:

'pysnmp.smi.mibs', 'pysnmp.smi.mibs.instances', 'pysnmp.smi.exval', 'pysnmp.cache'