python - How does str(list) work?

Question

Why does str(list) returns how we see list on the console? How does str(list) work? (any reference to the CPython code for str(list))?

>>> x = ['abc', 'def', 'ghi']
>>> str(x)
"['abc', 'def', 'ghi']"

To get the original list back from the str(list) I have to:

>>> from ast import literal_eval
>>> x = ['abc', 'def', 'ghi']
>>> str(x)
"['abc', 'def', 'ghi']"
>>> list(str(x))
['[', "'", 'a', 'b', 'c', "'", ',', ' ', "'", 'd', 'e', 'f', "'", ',', ' ', "'", 'g', 'h', 'i', "'", ']']
>>> literal_eval(str(x))
['abc', 'def', 'ghi']

Why doesn't list(str(list)) turns the str(list) back to the original list?

Or I could use:

>>> eval(str(x))
['abc', 'def', 'ghi']

Is literal_eval the same as eval? Is eval safe to use?

How many times can I do the following? Does the code break if it keep on doing str(list(str(list))))? E.g.

>>> x = 'abc'
>>> list(x)
['a', 'b', 'c']
>>> str(list(x))
"['a', 'b', 'c']"
>>> list(str(list(x)))
['[', "'", 'a', "'", ',', ' ', "'", 'b', "'", ',', ' ', "'", 'c', "'", ']']
>>> str(list(str(list(x))))
'['[', "'", 'a', "'", ',', ' ', "'", 'b', "'", ',', ' ', "'", 'c', "'", ']']'
>>> list(str(list(str(list(x)))))
['[', "'", '[', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", 'a', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", 'b', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", 'c', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ']', "'", ']']
>>> str(list(str(list(str(list(x))))))
'['[', "'", '[', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", 'a', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", 'b', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", 'c', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ']', "'", ']']'
>>> list(str(list(str(list(str(list(x)))))))
['[', "'", '[', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", '[', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', "'", '"', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", '"', "'", ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", 'a', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', "'", '"', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", '"', "'", ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', "'", '"', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", '"', "'", ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", 'b', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', "'", '"', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", '"', "'", ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', "'", '"', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", '"', "'", ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", 'c', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', "'", '"', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", '"', "'", ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ']', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ']', "'", ']']

Answer 1

Well you have a total of 4 questions, let us go one by one.

1. Why does str(list) returns how we see list on the console? How does str(list) work?

What is str() and __str__()?

The str() callable is to return a printable form of the object only! From the docs

str(object) does not always attempt to return a string that is acceptable to eval(); its goal is to return a printable string.

The __str__() function in a class is called whenever you call str() on an object. Again from the documentation

object.__str__(self)

Called by the str() built-in function and by the print statement to compute the “informal” string representation of an object.

What is the list callable?

The list() callable is to create a list from an iterable passed as an argument. Again from the docs

Return a list whose items are the same and in the same order as iterable‘s items

Thus, str(list) gives you a printable form and list(str(list)) will iterate over the string. That is list(str(list)) will give you a list of the individual characters of the printable form of the argument passed.

A small walk-through between the nested calls,

Given list, l = ['a','b'] _{(Apologies for taking a smaller example than that in your question)}.

When you call str(l), it returns a printable form of the list l, that is "['a','b']".

Now you can see clearly that "['a','b']" is a string and is indeed an iterable. Now when you call list on this i.e. list("['a','b']") you get a weird list like ['[', "'", 'a', "'", ',', "'", 'b', "'", ']']. Why does this happen? This happens because the string iterates over its characters, you can test this by using a dummy string,

>>> 'dummy'
'dummy'
>>> list('dummy')
['d', 'u', 'm', 'm', 'y']

Thus when you call the list on a string you get a list of character. Note that again here, when you call str() on list('dummy'), you will not get back your original string 'dummy', so again you will have to use join! Thus recalling the same function will NOT get you back your original object!

So, Calling str() over a list calls the builtin __str__() method of the list?

The answer is NO!

What happens internally when you call str() on a list?

Whenever you call str() on an list object, the steps followed are

1. Call the repr() of each of the list element.
2. Add a fancy [ at the front and another ] at the end of the list.
3. Join all of them with a comma.

As you can see from the source code of the list object in cpython on github. Going through the source code of cpython in hg.python, which is more clear, you can see the following three comments. _{(Thanks to Ashwini for the link on that particular code)}

/* Do repr() on each element.  Note that this may mutate the list,
   so must refetch the list size on each iteration. */ line (382)

/* Add "[]" decorations to the first and last items. */ line (398)

/* Paste them all together with ", " between. */ line (418)

These correspond to the points I mentioned above.

Now what is repr()?

repr() prints the string representation of all the objects. Again from the documentation

Return a string containing a printable representation of an object.

and also note this sentence!

For many types, this function makes an attempt to return a string that would yield an object with the same value when passed to eval(), otherwise the representation is a string enclosed in angle brackets that contains the name of the type of the object together with additional information often including the name and address of the object.

And now your second question here,

2. Why doesn't list(str(list)) turns the str(list) back to the original list?

Internally, str(list) actually creates the repr() representation of the list object. So to get back the list after calling str on the list, you actually need to do eval on it and not a list call.

Workarounds

But we all know that eval is evil, so what is/are the workaround(s)?

1. Using literal_eval

The first work-around would be to use ast.literal_eval. That brings us to your 3rd question,

3. Is literal_eval() the same as eval()? Is eval() safe to use?

ast.literal_eval() is safe unlike the eval() function. The docs themselves mention that it is safe --

Safely evaluate an expression node or a string containing a Python literal or container display

2. Using string functions and builtins

Another workaround can be done using str.split()

>>> x = ['abc', 'def', 'ghi']
>>> a = str(x)
>>> a[2:-2].split("', '")
['abc', 'def', 'ghi']

This is just a simple way to do that for a list of strings. For a list of integers you will need map.

>>> x = [1,2,3]
>>> a =str(x)
>>> list(map(int,a[1:-1].split(', '))) # No need for list call in Py2
[1, 2, 3]

Thus unlike literal_eval these are simple hacks given that you know the elements of the list. If they are heterogeneous in nature like [1, "a", True] then you will have to loop through the split list and discover the element type and then convert it and append the converted element to a final list.

Another place where this fails is when the string itself contains quote characters. As mentioned by nneonneo in a comment

The str.split solution is very fragile and will break if the input contains e.g. strings that contain ", ", or tuples, or other lists, ... It is much better to use ast.literal_eval because that will deal with all the subtleties of the syntax.

And for your final question,

4. Does the code break if you do str(list(str(list)))) again and again?

Not really. The output will grow longer and longer as each time you are creating a list of a str and then again getting the printable version of it. The limitation is your physical machine's limitation only. (which will be soon reached as each step the string length is multiplied by 5.)