Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
451 views
in Technique[技术] by (71.8m points)

python - How to make pipeline for multiple dataframe columns?

I have Dataframe which can be simplified to this:

import pandas as pd

df = pd.DataFrame([{
'title': 'batman',
'text': 'man bat man bat', 
'url': 'batman.com', 
'label':1}, 
{'title': 'spiderman',
'text': 'spiderman man spider', 
'url': 'spiderman.com', 
'label':1},
{'title': 'doctor evil',
 'text': 'a super evil doctor', 
'url': 'evilempyre.com', 
'label':0},])

And I want to try different feature extraction methods: TFIDF, word2vec, Coutvectorizer with different ngram settings, etc. But I want to try it in different combinations: one feature set will contain 'text' data transformed with TFIDF, and 'url' with Countvectoriser and second will have text data converted by w2v, and 'url' by TFIDF and so on. In the end, of course, I want to make a comparison of different preprocessing strategies and choose the best one.

And here are the questions:

  1. Is there a way to do such things using standard sklearn tools like Pipeline?

  2. Is there a common sense in my idea? Maybe there are good ideas how to treat text data with many columns in Dataframes which I am missing?

Many thanks!

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

Take a look at the following link: http://scikit-learn.org/0.18/auto_examples/hetero_feature_union.html

class ItemSelector(BaseEstimator, TransformerMixin):
def __init__(self, key):
    self.key = key

def fit(self, x, y=None):
    return self

def transform(self, data_dict):
    return data_dict[self.key]

The key value accepts a panda dataframe column label. When using it in your pipeline it can be applied as:

('tfidf_word', Pipeline([
            ('selector', ItemSelector(key='column_name')),
            ('tfidf', TfidfVectorizer())), 
            ]))

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...