i have created a function which is working fine , however, i would like to add if else statement inside function.reduce and im not sure if this is achievable. tried researching on func.reduce but there wasnt any information / use-case available on how i want to function to work. would need some advice if this is possible in python or should i go on another approach .
I have created this function which is working as expected, '
def tempJoin(df,targetField,targetTitle,condition,targetResult,otherwise,show=False,lowercase=False):
case_conditions = list(zip(condition, targetResult))
product_col = functools.reduce(
lambda acc, x: acc.when(col(targetField).like(x[0]), F.lit(x[1])),case_conditions,F,).otherwise(otherwise)
df = df.withColumn(targetTitle, product_col)
if show:
df.show(10)
return df;
i am fully aware that i could go with this approach, but i feel this can be further optimise
def tempJoin(df,targetField,targetTitle,condition,targetResult,otherwise,show=False,lowercase=False):
case_conditions = list(zip(condition, targetResult))
if lowercase :
product_col = functools.reduce(
lambda acc, x: acc.when(lower(col(targetField)).like(x[0]), F.lit(x[1])),case_conditions,F,).otherwise(otherwise)
else :
product_col = functools.reduce(
lambda acc, x: acc.when(col(targetField).like(x[0]), F.lit(x[1])),case_conditions,F,).otherwise(otherwise)
df = df.withColumn(targetTitle, product_col)
if show:
df.show(10)
return df;
what i am trying to achieve is to add if else of 'lowercase=True' , it will triggers within functools itself. something as this shown below
product_col = functools.reduce(
if lowercase:
lambda acc, x: acc.when(lower(col(targetField)).like(x[0]), F.lit(x[1])),case_conditions,F,).otherwise(otherwise)
else:
lambda acc, x: acc.when(col(targetField).like(x[0]), F.lit(x[1])),case_conditions,F,).otherwise(otherwise)
is this achievable ?
question from:
https://stackoverflow.com/questions/65878495/using-if-else-statement-inside-functools-reduce