python - Is it possible to efficiently convert PCollection list to PCollection (just values from the list)?

Question

I know it is possible using a ParDo function that yields elements from the list directly, but is there any other direct or more efficient method to do so?

question from:https://stackoverflow.com/questions/65934576/is-it-possible-to-efficiently-convert-pcollection-list-to-pcollection-just-valu

Answer 1

Following the official Documentation, you can use the Flatten method in order to merge PCollections. Examples are provided in the Beam Programming Guide.

# Flatten takes a tuple of PCollection objects.
# Returns a single PCollection that contains all of the elements in the PCollection objects in that tuple.

merged = (
    (pcoll1, pcoll2, pcoll3)
    # A list of tuples can be "piped" directly into a Flatten transform.
    | beam.Flatten())