Welcome To Ask or Share your Answers For Others

yarn - How should a Dask yarncluster be configured in order to use all the resources available from an underlying EMR cluster?

posted Feb 19, 2021 in Technique[技术] by 深蓝 (71.8m points)

yarn - How should a Dask yarncluster be configured in order to use all the resources available from an underlying EMR cluster?

I'm considering running a Dask Yarncluster within an AWS EMR cluster which will be dedicated to it. How can I ensure that the dask_yarn.Yarncluster is scaled to make maximum use of the compute resources available from the underlying EMR cluster?

Do I simply request the same number of workers as I have core nodes / EC2 instances from EMR? Are Yarncluster workers necessarily a 1:1 relationship to EMR nodes? Should I use the adaptive scaling feature with maximum set to infinity?

与恶龙缠斗过久,自身亦成为恶龙；凝视深渊过久,深渊将回以凝视…