winapi - Prevent desktop sharing of a particular c# winforms or detect desktop sharing

Question

While developing an examination software I have a requirement to prevent desktop sharing through applications like TeamViewer, AnyDesk, Ammyy Admin etc or at least detection of it. Our examination software is developed in C#, it's a winform renders question one by one.

I don't think detection is that easy as their are so many ways to capture the screen Desktop Duplication API, BitBlt, Direct3D, DirectX, DirectShow and so many.

So I started exploring for preventing my c# winform getting displayed when desktop sharing is started. To do This I have tried following thing so far:

1. As our application will run on windows 7 and above so I took advantage of DWM (Destop Window Manager) by checking desktop composition I set SetWindowDisplayAffinity to WDA_MONITOR to enable protection to my winform. By doing this when ever I start desktop sharing, the computer who has taken remote can see black layer over the form. But not all desktop sharing application have the same behavior. Like TeamViewer behaves as expected but Ammyy Admin, AnyDesk does not. How some applications shows black layer and some does not? Is there anything I can do additionally?

if (winForm != null)
{
    if (Protect)
        result = SetWindowDisplayAffinity(winForm.Handle, WDA_MONITOR);
    else
        result = SetWindowDisplayAffinity(winForm.Handle, WDA_NONE);
}

this approache I coded is not full proof with all desktop sharing applications, so is there any 100% solution to prevent/detect desktop sharing?

If you tried VLC player's "DirectX (DirectDraw) video output" then this output method also does the same thing like SetWindowDisplayAffinity, it puts black layer on video when any desktop sharing app is started and I tested this works with almost 9 desktop sharing apps, any idea how it's done? and can it be done with c# winform?

Update 29-05-2019

After checking VLC code I come to know that they are using hardware overlay using DirectDraw. So I created vc++ project and used d3d9 and created overlay with red color surface, now if I take remote of machine then the form with red color shows black in color. Bingo!! half problem is solved.

Now, I am trying to set transparency to that window so that, overlay will on top of my C# application and candidate can give the exam and if remote is taken then overlay will get shown in black. To make window transparent I used DwmExtendFrameIntoClientArea winapi, but now on remote also it's visible as transparent. Any way out for this?

Answer 1

I would like to add this as the answer because lots of discussions is carried out in comments which are moved to chat, so I would like to summarize it so that upcoming viewers will get the idea about the discussion and the solution I ended up with.

First I would like to clear the thing how VLC's "DirectX (DirectDraw) video output" works? VLC's this particular output mode uses DirectDraw for creating hardware overlay and video is played on it. While screen capturing (not from mobile or external camera) most of the capture mechanism captures screen by requesting GDI for screens, but hardware overlays are directly rendered on GPU and GDI can't access GPU. It's like GDI->CPU+RAM, DirectX->GPU+VRAM that's why in screen capturing VLC's video playing part appears in black. For more info https://stackoverflow.com/a/21535254/1629242

But as RomanR. mentioned in chat "use of an overlay intended for video is NOT going to help to hide UI of an app".

I also confirm the same, as I mentioned in update, I have created a vc++ project which creates hardware overlay using D3D9 if I play video on it then in screen recording shows black area but I didn't find any way to make that overlay transparent so that it can be put on my app or put app content on overlay so that when screen is recorded app contents will not be visible.

Riddel throw a new idea of "the way anti-viruses work and iterate through all the processes and hash check common screen sharing tools" but I personally feels this will not be generic solution as, I have to maintain list of such applications and it's hash, also if any new applications are found then update that list. Also, anyone can edit bytes to break hashes to makes this false.

Solution I accepted:

Windows API provides a function which helps to display the window's contents only on a monitor. i.e. SetWindowDisplayAffinity, this works like charm!!

problem with Minimum supported OS:

Though the MSDN doc mentions "Minimum supported client::Windows 7" but on windows 7 it's not working for all the screen capturing applications. RomanR. said in a comment "This will work correctly starting from Windows 8 (and possibly also Platform Update for Windows 7). On pure Windows 7, you are exposed to free desktop sharing and being unable to secure your content."

So currently, I am done with SetWindowDisplayAffinity I implemented this as:

private const uint WDA_NONE = 0x0;
private const uint WDA_MONITOR = 0x1;

[DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
private static extern bool SetWindowDisplayAffinity(IntPtr hWnd, uint dwAffinity);

public static bool SetDRM(Form winForm, bool Protect)
{
    bool result = false;
    if (winForm != null)
    {
        if (Protect)
            result = SetWindowDisplayAffinity(winForm.Handle, WDA_MONITOR);
        else
            result = SetWindowDisplayAffinity(winForm.Handle, WDA_NONE);
    }
    return result;
}