Yes, it seems like np.clip is a lot slower on pandas.Series than on numpy.ndarrays. That's correct but it's actually (at least asymptomatically) not that bad. 8000 elements is still in the regime where constant factors are major contributors in the runtime. I think this is a very important aspect to the question, so I'm visualizing this (borrowing from another answer):
# Setup
import pandas as pd
import numpy as np
def on_series(s):
return np.clip(s, a_min=None, a_max=1)
def on_values_of_series(s):
return np.clip(s.values, a_min=None, a_max=1)
# Timing setup
timings = {on_series: [], on_values_of_series: []}
sizes = [2**i for i in range(1, 26, 2)]
# Timing
for size in sizes:
func_input = pd.Series(np.random.randint(0, 30, size=size))
for func in timings:
res = %timeit -o func(func_input)
timings[func].append(res)
%matplotlib notebook
import matplotlib.pyplot as plt
import numpy as np
fig, (ax1, ax2) = plt.subplots(1, 2)
for func in timings:
ax1.plot(sizes,
[time.best for time in timings[func]],
label=str(func.__name__))
ax1.set_xscale('log')
ax1.set_yscale('log')
ax1.set_xlabel('size')
ax1.set_ylabel('time [seconds]')
ax1.grid(which='both')
ax1.legend()
baseline = on_values_of_series # choose one function as baseline
for func in timings:
ax2.plot(sizes,
[time.best / ref.best for time, ref in zip(timings[func], timings[baseline])],
label=str(func.__name__))
ax2.set_yscale('log')
ax2.set_xscale('log')
ax2.set_xlabel('size')
ax2.set_ylabel('time relative to {}'.format(baseline.__name__))
ax2.grid(which='both')
ax2.legend()
plt.tight_layout()
It's a log-log plot because I think this shows the important features more clearly. For example it shows that np.clip on a numpy.ndarray is faster but it also has a much smaller constant factor in that case. The difference for large arrays is only ~3! That's still a big difference but way less than the difference on small arrays.
However, that's still not an answer to the question where the time difference comes from.
The solution is actually quite simple: np.clip delegates to the clip method of the first argument:
>>> np.clip??
Source:
def clip(a, a_min, a_max, out=None):
"""
...
"""
return _wrapfunc(a, 'clip', a_min, a_max, out=out)
>>> np.core.fromnumeric._wrapfunc??
Source:
def _wrapfunc(obj, method, *args, **kwds):
try:
return getattr(obj, method)(*args, **kwds)
# ...
except (AttributeError, TypeError):
return _wrapit(obj, method, *args, **kwds)
The getattr line of the _wrapfunc function is the important line here, because np.ndarray.clip and pd.Series.clip are different methods, yes, completely different methods:
>>> np.ndarray.clip
<method 'clip' of 'numpy.ndarray' objects>
>>> pd.Series.clip
<function pandas.core.generic.NDFrame.clip>
Unfortunately is np.ndarray.clip a C-function so it's hard to profile it, however pd.Series.clip is a regular Python function so it's easy to profile. Let's use a Series of 5000 integers here:
s = pd.Series(np.random.randint(0, 100, 5000))
For the np.clip on the values I get the following line-profiling:
%load_ext line_profiler
%lprun -f np.clip -f np.core.fromnumeric._wrapfunc np.clip(s.values, a_min=None, a_max=1)
Timer unit: 4.10256e-07 s
Total time: 2.25641e-05 s
File: numpycorefromnumeric.py
Function: clip at line 1673
Line # Hits Time Per Hit % Time Line Contents
==============================================================
1673 def clip(a, a_min, a_max, out=None):
1674 """
...
1726 """
1727 1 55 55.0 100.0 return _wrapfunc(a, 'clip', a_min, a_max, out=out)
Total time: 1.51795e-05 s
File: numpycorefromnumeric.py
Function: _wrapfunc at line 55
Line # Hits Time Per Hit % Time Line Contents
==============================================================
55 def _wrapfunc(obj, method, *args, **kwds):
56 1 2 2.0 5.4 try:
57 1 35 35.0 94.6 return getattr(obj, method)(*args, **kwds)
58
59 # An AttributeError occurs if the object does not have
60 # such a method in its class.
61
62 # A TypeError occurs if the object does have such a method
63 # in its class, but its signature is not identical to that
64 # of NumPy's. This situation has occurred in the case of
65 # a downstream library like 'pandas'.
66 except (AttributeError, TypeError):
67 return _wrapit(obj, method, *args, **kwds)
But for np.clip on the Series I get a totally different profiling result:
%lprun -f np.clip -f np.core.fromnumeric._wrapfunc -f pd.Series.clip -f pd.Series._clip_with_scalar np.clip(s, a_min=None, a_max=1)
Timer unit: 4.10256e-07 s
Total time: 0.000823794 s
File: numpycorefromnumeric.py
Function: clip at line 1673
Line # Hits Time Per Hit % Time Line Contents
==============================================================
1673 def clip(a, a_min, a_max, out=None):
1674 """
...
1726 """
1727 1 2008 2008.0 100.0 return _wrapfunc(a, 'clip', a_min, a_max, out=out)
Total time: 0.00081846 s
File: numpycorefromnumeric.py
Function: _wrapfunc at line 55
Line # Hits Time Per Hit % Time Line Contents
==============================================================
55 def _wrapfunc(obj, method, *args, **kwds):
56 1 2 2.0 0.1 try:
57 1 1993 1993.0 99.9 return getattr(obj, method)(*args, **kwds)
58
59 # An AttributeError occurs if the object does not have
60 # such a method in its class.
61
62 # A TypeError occurs if the object does have such a method
63 # in its class, but its signature is not identical to that
64 # of NumPy's. This situation has occurred in the case of
65 # a downstream library like 'pandas'.
66 except (AttributeError, TypeError):
67 return _wrapit(obj, method, *args, **kwds)
Total time: 0.000804922 s
File: pandascoregeneric.py
Function: clip at line 4969
Line # Hits Time Per Hit % Time Line Contents
==============================================================
4969 def clip(self, lower=None, upper=None, axis=None, inplace=False,
4970 *args, **kwargs):
4971 """
...
5021 """
5022 1 12 12.0 0.6 if isinstance(self, ABCPanel):
5023 raise NotImplementedError("clip is not supported yet for panels")
5024
5025 1 10 10.0 0.5 inplace = validate_bool_kwarg(inplace, 'inplace')
5026
5027 1 69 69.0 3.5 axis = nv.validate_clip_with_axis(axis, args, kwargs)
5028
5029 # GH 17276
5030 # numpy doesn't like NaN as a clip value
5031 # so ignore
5032 1 158 158.0 8.1 if np.any(pd.isnull(lower)):
5033 1 3 3.0 0.2 lower = None
5034 1 26 26.0 1.3 if np.any(pd.isnull(upper)):
5035 upper = None
5036
5037 # GH 2747 (arguments were reversed)
5038 1 1 1.0 0.1 if lower is not None and upper is not None:
5039 if is_scalar(lower) and is_scalar(upper):
5040 lower, upper = min(lower, upper), max(lower, upper)
5041
5042 # fast-path for scalars
5043 1 1 1.0 0.1 if ((lower is None or (is_scalar(lower) and is_number(lower))) and
5044 1 28 28.0 1.4 (upper is None or (is_scalar(upper) and is_number(upper)))):
5045 1 1654 1654.0 84.3 return self._clip_with_scalar(lower, upper, inplace=inplace)
5046
5047 result = self
5048 if lower is not None:
5049 result = result.clip_lower(lower, axis, inplace=inplace)
5050 if upper is not None:
5051 if inplace:
5052 result = self
5053 result = result.clip_upper(upper, axis, inplace=inplace)
5054
5055 return result
Total time: 0.000662153 s
File: pandascoregeneric.py
Function: _clip_with_scalar at line 4920
Line # Hits Time Per Hit % Time Line Contents
==============================================================
4920
