python - Get Max & Min value for each key in the RDD

Question

spark = SparkSession.builder.getOrCreate()
sc = spark.sparkContext
ssc = StreamingContext(sc , 10)
rdd = ssc.sparkContext.parallelize(pd_binance)
rdd.take(1)

Here is a small portion of the result:

[['0.02703300', '1.30900000'],
   ['0.02703300', '0.61800000'],
   ['0.02704600', '3.90800000'],
   ['0.02704700', '4.00000000'],
   ['0.02704700', '7.44600000']

And I want to get the max and minimum value for each key, how?

question from:https://stackoverflow.com/questions/65540894/get-max-min-value-for-each-key-in-the-rdd

Answer 1

You can use reduceByKey:

minimum = rdd.reduceByKey(min)
maximum = rdd.reduceByKey(max)