Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
555 views
in Technique[技术] by (71.8m points)

python - What is the recommended way to persist (pickle) custom sklearn pipelines?

I have built an sklearn pipeline that combines a standard support vector regression component with some custom transformers that create features. This pipeline is then put into an object which is trained and then pickled (this seems to be the recommended way). The unpickled object is used to make predictions.

For distribution, this is turned into an executable file with pyinstaller.

When I call the unpickled regression object from a unit test, it works fine.

However, when I attempt to use the PyInstaller binary to make predictions, I get a long stack trace that ends with:

module = loader.load_module(fullname)   File "messagestream.pxd", line 5, in init scipy.optimize._trlib._trlib ImportError: No module named 'scipy._lib.messagestream'

This feels like some kind of pickling error, probably due to the interaction of pickling with pyinstaller. How can I refactor my code so that my custom pipeline runs just as easily and robustly as a standard sklearn regressor after unpickling?

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

OK, after some googling around it seems to be the case that the root cause is not pickling, it is simply a pyinstaller "hidden imports" issue, but for some reason it only shows up when pickling (don't ask me why).

The following solved the immediate issue for me: edit the .spec file to add the following hidden import with Scipy:

 hiddenimports=['scipy._lib.messagestream']

I also needed some other hidden imports related to other libraries

 hiddenimports=['sklearn.neighbors.typedefs',
                'scipy._lib.messagestream',
                'pandas._libs.tslibs.timedeltas'   ]

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...