python - AttributeError: 'PandasExprVisitor' object has no attribute 'visit_Ellipsis', using pandas eval

Question

I have a series of the form:

s

0    [133, 115, 3, 1]
1    [114, 115, 2, 3]
2      [51, 59, 1, 1]
dtype: object

Note that its elements are strings:

s[0]
'[133, 115, 3, 1]'

I'm trying to use pd.eval to parse this string into a column of lists. This works for this sample data.

pd.eval(s)

array([[133, 115, 3, 1],
       [114, 115, 2, 3],
       [51, 59, 1, 1]], dtype=object)

However, on much larger data (order of 10K), this fails miserably!

len(s)
300000

pd.eval(s)
AttributeError: 'PandasExprVisitor' object has no attribute 'visit_Ellipsis'

What am I missing here? Is there something wrong with the function or my data?

Answer 1

TL;DR
This is debattedly a bug in eval. See the open github issue GH16289.

---

Why am I getting this error?
This is because pd.eval cannot parse series with more than 100 rows. Here's an example.

len(s)
300000

pd.eval(s.head(100))  # returns a parsed result

Whereas,

pd.eval(s.head(101))
AttributeError: 'PandasExprVisitor' object has no attribute 'visit_Ellipsis'

This issue persists, regardless of the parser or the engine.

---

What does this error mean?
pd.eval operates on the __repr__ of the Series, rather than the objects contained within it (which is the cause of this bug). The __repr__ truncated rows, replacing them with a ... (ellipsis). This ellipsis is misinterpreted by the engine as an Ellipsis object -

...
Ellipsis

pd.eval('...')
AttributeError: 'PandasExprVisitor' object has no attribute 'visit_Ellipsis'

pd.eval technically is not supposed to parse series of strings (the documentation mentioned it is meant to receive strings) and (as described by the accepted answer) will try to make a reasonable guess at the result instead of rejecting the input outright.

Whether this is intended behavior or incomplete behavior (a lot of pandas methods operate differently based on the input - and eval could work on a series by mapping itself onto each row, which is how I initially assumed this was working anyway) is up for discussion, since there's an open issue tracking this.

---

What can I do to make this to work?
Right now, there isn't a solution (the issue is still open as of 12/28/2017), however, there are a couple of workarounds.

Option 1
ast.literal_eval
This option should work out of the box if you can guarantee that you do not have any malformed strings.

from ast import literal_eval

s.apply(literal_eval)

0    [133, 115, 3, 1]
1    [114, 115, 2, 3]
2      [51, 59, 1, 1]
dtype: object 

If there is a possibility of malformed data, you'll need to write a little error handling code. You can do that with a function -

def safe_parse(x):
    try:
        return literal_eval(x)
    except (SyntaxError, ValueError):
        return np.nan # replace with any suitable placeholder value

Pass this function to apply -

s.apply(safe_parse)
    
0    [133, 115, 3, 1]
1    [114, 115, 2, 3]
2      [51, 59, 1, 1]
dtype: object

ast works for any number of rows, and is slow, but reliable. You can also use pd.json.loads for JSON data, applying the same ideas as with literal_eval.

Option 2
yaml.load
Another great option for parsing simple data, I picked this up from @ayhan a while ago.

import yaml
s.apply(yaml.load)

0    [133, 115, 3, 1]
1    [114, 115, 2, 3]
2      [51, 59, 1, 1]
dtype: object

I haven't tested this on more complex structures, but this should work for almost any basic string representation of data.

You can find the documentation for PyYAML here. Scroll down a bit and you'll find more details on the load function.

---

Note

• If you're working with JSON data, it might be suitable to read your file using pd.read_json or pd.io.json.json_normalize to begin with.

• You can also perform parsing as you read in your data, using read_csv -

    s = pd.read_csv(converters=literal_eval, squeeze=True)
  

  Where the converters argument will apply that function passed on the column as it is read, so you don't have to deal with parsing later.

• Continuing the point above, if you're working with a dataframe, pass a dict -

    df =  pd.read_csv(converters={'col' : literal_eval})
  

  Where col is the column that needs to be parsed You can also pass pd.json.loads (for json data), or pd.eval (if you have 100 rows or less).

---

Credits to MaxU and Moondra for uncovering this issue.