Most efficient way to use Node.js PM2 clustering and Node.js worker_threads module at the same time?

Question

I'm using PM2 cluster module to spawn an instance of my Node.js application for each one of my CPU cores. At the sime time, for CPU-heavy operations (like crypto) I'm using Piscina (a worker pool library module for Node.js, which is based on Node.js worker threads module).

Basically, I create a Piscina instance with some worker threads and call it for crypto operations everytime I hit an specific API.

Could this be unefficient in terms of memory usage, given that I create a worker pool for each one of my app instances? What is the most efficient way to use worker threads and Node.js cluster module at the same time?

question from:https://stackoverflow.com/questions/65834945/most-efficient-way-to-use-node-js-pm2-clustering-and-node-js-worker-threads-modu

Answer 1

The short answer is yes, there is an ineffective memory usage if you are using PM2 or Cluster module as they use process forking instead taking advantages of multithreading.

In your case, the problem is that the worker pool is being created inside the app instance which may lead to some ineffectiveness in memory usage(I think it maybe negligible if pool is configured adequately taking into consideration app instances). You may create another separate node simple app just for handling crypto tasks like a microservice and start it before running app cluster. The cluster workers may communicate with it through some IPC message protocol or better use GRPC/REST.