multithreading - Running 60+ Remote Ubuntu From Unity / C#

Question

I'm trying to running 60+ Ubuntu machine connected by 10Gbps ethernet in Unity with 28 cores.

So I made a task using SSH.NET like this. (Each Ubuntu cmd takes 5+ min)

class UbuntuWorker
{
    async Task Work()
    {
        client.Connect();
        cmd = GetNextJob();
        cmd.BeginExcute();
        while(!cmd.IsCompleted)
            await Task.Delay(3000);
        result = cmd.EndExecute(cmd);
        SaveResult(result);
        client.Disconnect();
    }
    public void GoWork()
    {
        task = Task.Run(() => Work());
    }
}
List<UbuntuWorker> workers = new List<UbuntuWorker>();
List<string> jobs = new List<string>();
LoadWorkers(workers);
LoadJobs(jobs);
foreach (var worker in workers)
    worker.GoWork();
while (jobs.Count() != 0)
{
    foreach (var worker in workers)
        if (worker.hasCompleted)
            worker.GoWork();
        else
            await Task.Delay(300);
}
// These are not exact code but simplified.

Then I made 60+ tasks during runtime and of course, I'm suffering from massive fps drop and freezing. I learned that it's bad design that using task more than cores, but I have to run machines as much as possible at the same time...

Q1. I'm actually new to TPL. In this case, How should I design my code?

Q2. Keeping 60+ connection using SSH.NET is also costly a bit. Is there an alternative approach, for example keeping process after disconnect and check it after 5 min?

question from:https://stackoverflow.com/questions/65830238/running-60-remote-ubuntu-from-unity-c-sharp

Answer 1

A2.

Normally using SSH.NET, when client disconnected before its command has finished, It also stopped.

But I found "nohub" command allows commands keep running after disconnection.

Edit : No, whatever command alone couldn't solve this issue. I solved this eventually by shell class on SSH.NET

So I can reduce number of concurrent connection and FPS of Unity App has recovered.

Checking whether command has finished is other issue. It seems best options is parsing and handle with PID, but I'm using kind of shortcut for now.

A1.

I redesigned entire code and UbuntuWorker(alias) class has seperated to Ubuntu class and Worker class. So I can reduce number of task and I'm still finding optimized number for main machine.

Changed code is like this. (not exact code but simplified)

public class Ubuntu;
public class Worker;
public class Job;
ConCurrentQueue<Ubuntu> ubuntuQueue;
ConCurrentQueue<Job> jobQueue;
int numWorkers = 30; // half of machines
Worker[] workers = new Worker[numWorkers];
LoadUbuntu(ubuntuQueue);
LoadJob(jobQueue);

for (int i = 0; i < numWorkers; ++i)
{
    workers[i] = new Worker();
    workers[i].task = Task.Run(() =>
    {
        Ubuntu holder;
        while (ubuntuQueue.TryDequeue(out holder))
        {
            if (holder.job.isDone)
            {
                SaveResult(holder.job.result);
                if (jobQueue.TryDequeue(out holder.job))
                {
                    holder.client.Connect();
                    using (var shell = holder.client.CreateShellStream("", 0, 0, 0, 0, 0))
                    {
                        shell.WriteLine(holder.job.GetCommand + " & disown -a");
                        await Task.Delay(3000); // give time for launching
                    }
                    holder.client.Disconnect();
                }
                else
                    continue;
            }
            ubuntuQueue.Enqueue(holder);
        }
    });
}

Entire performance has stabilized for now. Still need to prepare for expansion though.

Idea suggested by Dan Puzey (let Ubuntu pull jobs) has not implemented yet because I need to investigate how to make it first. I'm assuming building another C# code on Ubuntu is easiest way so I can reuse SSH.NET APIs.