spark streaming - Custom partiotioning of JavaDStreamPairRDD

Question

In Spark streaming, what's the recommended way to implement a custom partiotioner on DStreams?

I've used the JavaPairRDD.partitionBy(Partitioner) in batch mode but haven't found an equivalent on JavaDStreamPairRDD.

Thanks

Answer 1

Partitions on DStreams are created by the process of getting data from the receiver. The data stream created by each receiver is cut in micro batches of size spark.streaming.blockInterval (200ms by default) each micro batch becomes a partition on the RDD produced for the streaming interval. Hence, the streaming partitioning is a consequence of micro batching and custom partitioner wouldn't make sense at this level.

If you need those partitions in a certain custom shape, you could repartition each RDD of the DStream:

dstream.foreachRDD{rdd => 
val repRDD = rdd.partitionBy(...)
... do stuff ...
}

Be aware that you pay the shuffle price for repartitioning, so use with care.