winapi - RmGetList() API fails when file is locked using Ez file locker but works with another file locking utility

Question

I am experiencing a strange issue with the Restart Manager API: RmGetlist(). To simulate a file locking scenario, i am making use of the following 3rd party file locking utilities:

Ez file locker -http://www.xoslab.com/efl.html -

File locker http://www.jensscheffler.de/filelocker

The strange issue here is that, both these utilities lock a certain file, however, RMGetList() fails with an Access denied error(5) with the first file locking utility(Ez File lock) whereas it works with the second file locking utility.

What could possibly be wrong here? Why would RmGetList() fail with one file locking utility but work with another?

Below is the code that is being used:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Runtime.InteropServices;
using System.Security;
using System.IO;
using System.Windows.Forms;

namespace RMSession
{
    class Program
    {


        public static void GetProcessesUsingFiles(string[] filePaths)
        {
            uint sessionHandle;
            int error = NativeMethods.RmStartSession(out sessionHandle, 0, Guid.NewGuid().ToString("N"));
            if (error == 0)
            {
                try
                {
                    error = NativeMethods.RmRegisterResources(sessionHandle, (uint)filePaths.Length, filePaths, 0, null, 0, null);
                    if (error == 0)
                    {
                        RM_PROCESS_INFO[] processInfo = null;
                        uint pnProcInfoNeeded = 0, pnProcInfo = 0, lpdwRebootReasons = RmRebootReasonNone;
                        error = NativeMethods.RmGetList(sessionHandle, out pnProcInfoNeeded, ref pnProcInfo, null, ref lpdwRebootReasons);
                        while (error == ERROR_MORE_DATA)
                        {
                            processInfo = new RM_PROCESS_INFO[pnProcInfoNeeded];
                            pnProcInfo = (uint)processInfo.Length;
                            error = NativeMethods.RmGetList(sessionHandle, out pnProcInfoNeeded, ref pnProcInfo, processInfo, ref lpdwRebootReasons);
                        }

                        if (error == 0 && processInfo != null)
                        {
                            for (var i = 0; i < pnProcInfo; i++)
                            {
                                RM_PROCESS_INFO procInfo = processInfo[i];
                                Process proc = null;
                                try
                                {
                                    proc = Process.GetProcessById(procInfo.Process.dwProcessId);
                                }
                                catch (ArgumentException)
                                {
                                    // Eat exceptions for processes which are no longer running.
                                }

                                if (proc != null)
                                {
                                    //yield return proc;
                                }
                            }
                        }
                    }
                }
                finally
                {
                    NativeMethods.RmEndSession(sessionHandle);
                }
            }
        }

        private const int RmRebootReasonNone = 0;
        private const int CCH_RM_MAX_APP_NAME = 255;
        private const int CCH_RM_MAX_SVC_NAME = 63;
        private const int ERROR_MORE_DATA = 234;

        [StructLayout(LayoutKind.Sequential)]
        private struct RM_UNIQUE_PROCESS
        {
            public int dwProcessId;
            public System.Runtime.InteropServices.ComTypes.FILETIME ProcessStartTime;
        }

        [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
        private struct RM_PROCESS_INFO
        {
            public RM_UNIQUE_PROCESS Process;
            [MarshalAs(UnmanagedType.ByValTStr, SizeConst = CCH_RM_MAX_APP_NAME + 1)]
            public string strAppName;
            [MarshalAs(UnmanagedType.ByValTStr, SizeConst = CCH_RM_MAX_SVC_NAME + 1)]
            public string strServiceShortName;
            public RM_APP_TYPE ApplicationType;
            public uint AppStatus;
            public uint TSSessionId;
            [MarshalAs(UnmanagedType.Bool)]
            public bool bRestartable;
        }

        private enum RM_APP_TYPE
        {
            RmUnknownApp = 0,
            RmMainWindow = 1,
            RmOtherWindow = 2,
            RmService = 3,
            RmExplorer = 4,
            RmConsole = 5,
            RmCritical = 1000
        }

        [SuppressUnmanagedCodeSecurity]
        private static class NativeMethods
        {
            /// <summary>
            /// Starts a new Restart Manager session.
            /// </summary>
            /// <param name="pSessionHandle">A pointer to the handle of a Restart Manager session. The session handle can be passed in subsequent calls to the Restart Manager API.</param>
            /// <param name="dwSessionFlags">Reserved must be 0.</param>
            /// <param name="strSessionKey">A null-terminated string that contains the session key to the new session. A GUID will work nicely.</param>
            /// <returns>Error code. 0 is successful.</returns>
            [DllImport("RSTRTMGR.DLL", CharSet = CharSet.Unicode, PreserveSig = true, SetLastError = true, ExactSpelling = true)]
            public static extern int RmStartSession(out uint pSessionHandle, int dwSessionFlags, string strSessionKey);

            /// <summary>
            /// Ends the Restart Manager session.
            /// </summary>
            /// <param name="pSessionHandle">A handle to an existing Restart Manager session.</param>
            /// <returns>Error code. 0 is successful.</returns>
            [DllImport("RSTRTMGR.DLL")]
            public static extern int RmEndSession(uint pSessionHandle);

            /// <summary>
            /// Registers resources to a Restart Manager session. 
            /// </summary>
            /// <param name="pSessionHandle">A handle to an existing Restart Manager session.</param>
            /// <param name="nFiles">The number of files being registered.</param>
            /// <param name="rgsFilenames">An array of strings of full filename paths.</param>
            /// <param name="nApplications">The number of processes being registered.</param>
            /// <param name="rgApplications">An array of RM_UNIQUE_PROCESS structures. </param>
            /// <param name="nServices">The number of services to be registered.</param>
            /// <param name="rgsServiceNames">An array of null-terminated strings of service short names.</param>
            /// <returns>Error code. 0 is successful.</returns>
            [DllImport("RSTRTMGR.DLL", CharSet = CharSet.Unicode)]
            public static extern int RmRegisterResources(uint pSessionHandle, uint nFiles, string[] rgsFilenames, uint nApplications, [In] RM_UNIQUE_PROCESS[] rgApplications, uint nServices, string[] rgsServiceNames);

            /// <summary>
            /// Gets a list of all applications and services that are currently using resources that have been registered with the Restart Manager session.
            /// </summary>
            /// <param name="dwSessionHandle">A handle to an existing Restart Manager session.</param>
            /// <param name="pnProcInfoNeeded">A pointer to an array size necessary to receive RM_PROCESS_INFO structures</param>
            /// <param name="pnProcInfo">A pointer to the total number of RM_PROCESS_INFO structures in an array and number of structures filled.</param>
            /// <param name="rgAffectedApps">An array of RM_PROCESS_INFO structures that list the applications and services using resources that have been registered with the session.</param>
            /// <param name="lpdwRebootReasons">Pointer to location that receives a value of the RM_REBOOT_REASON enumeration that describes the reason a system restart is needed.</param>
            /// <returns>Error code. 0 is successful.</returns>
            [DllImport("RSTRTMGR.DLL")]
            public static extern int RmGetList(uint dwSessionHandle, out uint pnProcInfoNeeded, ref uint pnProcInfo, [In, Out] RM_PROCESS_INFO[] rgAffectedApps, ref uint lpdwRebootReasons);
        }


        static void Main(string[] args)
        {
            Console.WriteLine("Starting...");
            string[] file1 = new string[1];
            MessageBox.Show("Debug C#");
            file1[0] = @"C:ProcessMonitor.zip";
            //DirectoryInfo dirInfo = new DirectoryInfo(folder);

            GetProcessesUsingFiles(file1);
            Console.WriteLine("End");``
        }
    }
}

Answer 1

Easy File Locker is "locking" the file only in the informal sense of the word, i.e., it protects the files from access, but it does not do so by obtaining a lock on the file. Instead, it uses a lower-level technology (a file system filter driver) that is broadly similar to the way that anti-virus software protects its files from any unauthorized access. The Restart Manager API is not intended to, and does not, deal with this sort of scenario.

Your application almost certainly doesn't need to deal with this sort of scenario either. This means that Easy File Locker is not an appropriate tool for your particular needs; throw it away.

---

Why would RmGetList() fail with one file locking utility but work with another?

To answer this we need to understand how RmGetList works internally. In your case, we provide it with a filename, and its output is an array of RM_PROCESS_INFO structures. In order to build this array, Windows must determine which processes are using the file. But how does Windows do this?

The function ZwQueryInformationFile (exported by ntdll.dll) can return a lot of information about a file. One of the options in the FILE_INFORMATION_CLASS enumeration is

FileProcessIdsUsingFileInformation

A FILE_PROCESS_IDS_USING_FILE_INFORMATION structure. This value is reserved for system use. This value is available starting with Windows Vista.

and in wdm.h (which is a well known file from the windows WDK) we find

typedef  struct _FILE_PROCESS_IDS_USING_FILE_INFORMATION {
    ULONG NumberOfProcessIdsInList;
    ULONG_PTR ProcessIdList[1];
} FILE_PROCESS_IDS_USING_FILE_INFORMATION, *PFILE_PROCESS_IDS_USING_FILE_INFORMATION;

so this option is exactly which we need!

The algorithm goes like this:

1. open the file with FILE_READ_ATTRIBUTES access (which is sufficient for this information class)
2. call ZwQueryInformationFile(..,FileProcessIdsUsingFileInformation); if NumberOfProcessIdsInList != 0 walk the ProcessIdList
3. open process by ProcessId, query it with ProcessStartTime (see GetProcessTimes) and other properties to fill RM_PROCESS_INFO

So now that we know how that works, let's look at the two utilities you're using.

The second is a very simple app, which provides source code. All it does is to call CreateFile with dwShareMode = 0. That acquires an exclusive lock on the file, ensuring that any attempt to open the file with a dwDesiredAccess that contains FILE_READ_DATA or FILE_WRITE_DATA or DELETE will fail with ERROR_SHARING_VIOLATION. It does not, however, prevent us from opening the file with dwDesiredAccess = FILE_READ_ATTRIBUTES, so the call to RmGetList() will still work properly.

But the first tool, Easy File Locker from XOSLAB, is using a minifilter driver (HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesxlkfs) to restrict access to the file. This driver returns STATUS_ACCESS_DENIED (which is converted to the Win32 ERROR_ACCESS_DENIED) for any attempt to open the file. Because of this, you get an error ERROR_ACCESS_DENIED when RmGetList attempts to open the file at step (1) and (since the API has no idea what to do about this) this error code is returned to you.

That's all there is to it. The tool simply isn't doing what you were expecting it to.