google cloud platform - Increasing the memory of a single node in GCP Composer

Question

I have a GCP Composer instance running with 3 nodes (which is the minimum possible). The nodes of this instance have 4GB of memory. However, I have a job that requires 8GB of memory. Is it possible to adjust the memory of just a single node? I don't want to upgrade my whole Composer instance to 8GB, because that would dramatically increase the cost.

question from:https://stackoverflow.com/questions/65886555/increasing-the-memory-of-a-single-node-in-gcp-composer

Answer 1

Thought it is not recommanded, you can upgrade the machine type used by your Cloud Composer cluster by removing and create again de default-pool.

There's an official documentation on how to to this.

It might not show in the Composer UI or in the Composer settings returned by CLI.

Also, you can as suggested in parakeet's comment use a KubernetesPodOperator or GKEPodOperator in order to properly dimension the node pool used to instanciate the pod.