apache spark - How to extract application ID from the PySpark context

Question

A previous question recommends sc.applicationId, but it is not present in PySpark, only in scala.

So, how do I figure out the application id (for yarn) of my PySpark process?

Answer 1

You could use Java SparkContext object through the Py4J RPC gateway:

>>> sc._jsc.sc().applicationId()
u'application_1433865536131_34483'

Please note that sc._jsc is internal variable and not the part of public API - so there is (rather small) chance that it may be changed in the future.

I'll submit pull request to add public API call for this.