python 2.7 - SWIG and C++ memory leak with vector of pointers

Question

I am using SWIG to interface between C++ and Python. I have created a function which creates a std::vector of object pointers. The objects that are pointed to are not important in this case.

The problem I have is that when the object (someObject) goes out of scope on the Python side it cannot free the memory pointed to by the object/s pointers within the vector, thus causing a memory leak.

Example

• C++ code:

  std::vector < someObject* > createSomeObjectForPython()
  {
     std::vector < someObject* > myVector;
     someObject* instanceOfSomeObject = new someObject();
     myVector.push_back(instanceOfSomeObject);
     return myVector;
  }
  

• From the Python interpreter:

  objectVar = createSomeObjectForPython()
  

When I run this in Python I get this error:

swig/python detected a memory leak of type 'std::vector< someObject *,std::allocator<  someObject * > > *', no destructor found.

This error is because when Python deletes the vector, it can only delete the pointers within the vector and not actually what they point to.

If I could create a destructor for std::vector, this would be the answer, but it’s not possible.

I really need to use vectors of pointers opposed to vectors of objects before anyone suggests this as a solution, particularly because the objects are large and complex, and speed is an issue.

I am using gcc4.4, swigwin 2.0.4, and Python 2.7 on Windows.

Answer 1

The warning you see doesn't lie directly with the fact that you have a vector of pointers. Consider the following SWIG interface file:

%module test

// This just gets passed straight through and not used for wrapping
%{
struct foo {};
%}

struct foo;

%inline %{
  struct foo bar() { struct foo f; return f; }
%}

Using this interface gives:

swig -Wall -python test.i && gcc -Wall -Wextra -std=c99 -shared -o _test.so test_wrap.c -I/usr/include/python2.7 && python2.7
Python 2.7.3 (default, Aug  1 2012, 05:16:07) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> test.bar()
<Swig Object of type 'struct foo *' at 0xb7654a70>
>>> 
swig/python detected a memory leak of type 'struct foo *', no destructor found.

The problem is that SWIG has only seen a declaration, not a definition for struct foo. The default behaviour is for the Python proxy object to free/delete (as appropriate) the underlying object here, but it's not able to deduce how to do that based on only the forward declaration it's seen.

If we extend the test case to include std::vector<foo> the same is observed:

%module test

%{
struct foo {};
%}

struct foo;

%include <std_vector.i>

%inline %{
  foo bar() { return foo(); }
  std::vector<foo> bar2() { 
    return std::vector<foo>(); 
  } 
%}

Which again gives the warning about no destructor:

Python 2.7.3 (default, Aug  1 2012, 05:16:07) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> print test.bar2()
<Swig Object of type 'std::vector< foo,std::allocator< foo > > *' at 0xb7671a70>swig/python detected a memory leak of type 'std::vector< foo,std::allocator< foo > > *', no destructor found.

However we can trivially fix this by making sure a definition of the type is available. For struct foo that's simply making the whole body of the struct visible to SWIG. For std::vector<T> we need to use %template to do that:

%module test

%include <std_vector.i>

%inline %{
  struct foo {};
  foo bar() { return foo(); }
  std::vector<foo> bar2() { 
    return std::vector<foo>(); 
  } 
%}

%template(FooVec) std::vector<foo>;

Which now doesn't warn (or leak for that matter):

Python 2.7.3 (default, Aug  1 2012, 05:16:07) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> print test.bar()
<test.foo; proxy of <Swig Object of type 'foo *' at 0xb76aba70> >
>>> print test.bar2()
<test.FooVec; proxy of <Swig Object of type 'std::vector< foo > *' at 0xb76abab8> >
>>> 

The complication is that in your example you have std::vector<T*>, so we can alter our test case to illustrate that:

%module test

%include <std_vector.i>

%inline %{
  struct foo {};
  foo bar() { return foo(); }
  std::vector<foo*> bar2() { 
    return std::vector<foo*>(1, new foo); 
  } 
%}

%template(FooVec) std::vector<foo*>;

Which we can then run:

Python 2.7.3 (default, Aug  1 2012, 05:16:07) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> print test.bar2()
<test.FooVec; proxy of <Swig Object of type 'std::vector< foo * > *' at 0xb7655a70> >
>>> 

This does leak, but crucially doesn't show the warning you noticed, because as far as SWIG is concerned the std::vector itself has been correctly deleted (the exact same semantics as in C++ in fact).

As far as how to deal with the leak there the options are the same as usual in C++. Personally I'd try to avoid putting raw pointers in a vector unless you really want the objects pointed at to outlive the vector. Basically you can:

1. Not store pointers in the struct
2. Use smart pointers (std::shared_ptr or std::unique_ptr or boost equivalents instead).
3. Manage the memory manually somehow.

We've already done 1 in the second example. With SWIG 2 is pretty simple as well and 3 is a question of writing and wrapping another function in your interface.

%module test

%include <std_vector.i>
%include <std_shared_ptr.i>

%{
#include <memory>
%}

%inline %{
  struct foo {};
  foo bar() { return foo(); }
  std::vector<std::shared_ptr<foo> > bar2() { 
    return std::vector<std::shared_ptr<foo> >(1, std::make_shared<foo>()); 
  } 
%}

%shared_ptr(Foo);
%template(FooVec) std::vector<std::shared_ptr<foo> >;

Python 2.7.3 (default, Aug  1 2012, 05:16:07) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> print test.bar2()
<test.FooVec; proxy of <Swig Object of type 'std::vector< std::shared_ptr< foo >,std::allocator< std::shared_ptr< foo > > > *' at 0xb76f4a70> >
>>> print test.bar2()[0]
<Swig Object of type 'std::vector< std::shared_ptr< foo > >::value_type *' at 0xb76f4a70>
>>> 

Which works, stores shared pointers and doesn't leak.

If you really want to do the third way (I'd avoid it at all cost given that it leaves your interface open to human errors) the easiest way to do it with SWIG is to use %extend, for example:

%module test

%include <std_vector.i>

%inline %{
  struct foo {};
  foo bar() { return foo(); }
  std::vector<foo*> bar2() { 
    return std::vector<foo*>(1, new foo); 
  } 
%}

%template(FooVec) std::vector<foo*>;

%extend std::vector<foo*> {
  void empty_and_delete() {
    for (std::vector<foo*>::iterator it = $self->begin(); 
         it != $self->end(); ++it) {
      delete *it;
    }
    $self->clear();
  }
}

The we can do:

Python 2.7.3 (default, Aug  1 2012, 05:16:07) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> x = test.bar2()
>>> print x.size()
1
>>> x.empty_and_delete()
>>> print x.size()
0
>>> 

Or you could use %pythoncode to modify __del__ to call the function automatically, but that would be a bad idea because it wouldn't affect objects Python never sees at all and could lead to unexpected behaviour in a few cases.