apache spark - Logging from pandas udf

Question

I am trying to log from a pandas udf called within a python transform.
Because the code is being called on the executor is does not show up in the driver's logs.

I have been looking at some options on SO but so far the closest option is this one

Any idea on how to surface the logs in the driver logs or any other log files available under build is welcome.

import logging
logger = logging.getLogger(__name__)

@pandas_udf(schema, functionType=PandasUDFType.GROUPED_MAP)
def my_udf(my_pdf):
    logger.info('calling my udf')
    do_some_stuff()


results_df = my_df.groupby("Name").apply(my_udf)

question from:https://stackoverflow.com/questions/65918175/logging-from-pandas-udf

Answer 1

Waitting for answers