pyspark - DataFrame show string representation fails with showString(Integer, Boolean, Boolean) does not exist

Question

I'm trying to capture the string representation generated by the show() function as suggested here, but it seems like the showString method is no longer public.

logger.info('
{}'.format(raw_data._jdf.showString(20, False, False)))

is returning the following error trace

Traceback (most recent call last):
  File "xxxxxxxx/SparkTest/main.py", line 63, in <module>
    logger.info('
{}
'.format(raw_data._jdf.showString(20, False, False)))
  File "xxxxxxxx/envs/SparkTest/lib/python3.8/site-packages/py4j/java_gateway.py", line 1304, in __call__
    return_value = get_return_value(
  File "xxxxxxxx/envs/SparkTest/lib/python3.8/site-packages/pyspark/sql/utils.py", line 128, in deco
    return f(*a, **kw)
  File "xxxxxxxx/envs/SparkTest/lib/python3.8/site-packages/py4j/protocol.py", line 330, in get_return_value
    raise Py4JError(
py4j.protocol.Py4JError: An error occurred while calling o45.showString. Trace:
py4j.Py4JException: Method showString([class java.lang.Integer, class java.lang.Boolean, class java.lang.Boolean]) does not exist
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
    at py4j.Gateway.invoke(Gateway.java:274)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.base/java.lang.Thread.run(Thread.java:832)

is there another alternative to this approach?

question from:https://stackoverflow.com/questions/65836215/dataframe-show-string-representation-fails-with-showstringinteger-boolean-boo

Answer 1

The error says that the method showString(Integer, Boolean, Boolean) does not exist. If you look at showString source code, it takes Integer as second argument but you're passing a boolean.

You want to set truncate=False so simply pass 0 in the second argument:

logger.info('
{}'.format(raw_data._jdf.showString(20, 0, False)))