you're reading...
Interop Marshaling

Example Custom Marshaler – The File Handle Marshaler

1. Introduction.

1.1 This is the second “example code” article I am writing to demonstrate custom marshaling.

1.2 In the first example code extension to my “Understanding Custom Marshaling” series of articles, I wrote a custom marshaler that can be used to transform a C# long type (a 64-bit number) to an unmanaged LARGE_INTEGER structure.

1.3 In this article, I shall touch on a subject that some may find practically useful : to transform a managed FileStream type into an unmanaged HANDLE type.

2. The FileHandleMarshaler Class.

2.1 In software development work, file access is a very common activity.

2.2 In C#, one of the ways that files are accessed is via the FileStream class.

2.3 In unmanaged Windows development work, e.g. in C++, files are managed through HANDLEs.

2.4 The custom marshaler that I provide in this article is called FileHandleMarshaler. The basic mission of the FileHandleMarshaler is to marshal a data of type FileStream to an unmanaged HANDLE.

3. Source Codes of the FileHandleMarshaler.

3.1 The following is a listing of the FileHandleMarshaler class :

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using Microsoft.Win32.SafeHandles;
using System.Runtime.InteropServices;

namespace FileHandleMarshalerClassLib
    public class FileHandleMarshaler : ICustomMarshaler
        public object MarshalNativeToManaged(IntPtr pNativeData)
            return null;

        public IntPtr MarshalManagedToNative(object ManagedObj)
            // Managed Object must be a FileStream type.
            if (!(ManagedObj is FileStream))
                throw new ArgumentException("The input argument is not a FileStream object.");

            FileStream file_stream = (FileStream)ManagedObj;

            SafeFileHandle safe_file_handle = file_stream.SafeFileHandle;

            if (!(safe_file_handle.IsClosed) && !(safe_file_handle.IsInvalid))
                bool bSuccess = false;

                safe_file_handle.DangerousAddRef(ref bSuccess);

                IntPtr pHandle = safe_file_handle.DangerousGetHandle();

                return pHandle;

            return IntPtr.Zero;

        public void CleanUpNativeData(IntPtr pNativeData)
            SafeFileHandle safe_file_handle = new SafeFileHandle(pNativeData, false);


        public void CleanUpManagedData(object ManagedObj)
            // Nothing to do

        public int GetNativeDataSize()
            return -1;

        public static ICustomMarshaler GetInstance(string cookie)
            // Always return the same instance
            if (marshaler == null)
                marshaler = new FileHandleMarshaler();

            return marshaler;

        static private FileHandleMarshaler marshaler;

3.2 The following is a summary of this class :

  • It is a one-way marshaler that transforms managed data to unmanaged hence the trivial implementation of the MarshalNativeToManaged() method.
  • The main action is performed inside the MarshalManagedToNative() and the CleanUpNativeData() methods.
  • MarshalManagedToNative() first ensures that the input parameter is of type FileStream. If not, it throws the ArgumentException.
  • MarshalManagedToNative() then transforms the C# FileStream parameter into an unmanaged HANDLE by using the services of the SafeFileHandle class.
  • For more information about the SafeFileHandle class see the SafeFileHandle class and the FileStream.SafeFileHandle property.
  • See also my article Using the SafeHandle Class to Customize the Management of Unmanaged Handles.
  • MarshalManagedToNative() eventually returns an IntPtr that can be construed as an unmanaged HANDLE and can be used in unmanaged code directly.
  • Note well that this returned unmanaged HANDLE is owned by the FileHandleMarshaler instance and will remain so throughout the duration of the external API call.
  • This HANDLE is later released in CleanUpNativeData().

4. Client Code.

4.1 In this section, we present some client code that will invoke the FileHandleMarshaler.

4.2 We need both a C# client as well as an exported DLL function that will be called from inside the C# client.

4.3 The following is a listing of a C++ exported function that will be called in C# :

void __stdcall WriteToFile(HANDLE hFile, const char* lpszString)
	DWORD	dwNumberOfBytesWritten = 0;

  • The purpose of the WriteToFile() function is to write a line of C-style character string into a file.
  • WriteToFile() takes a HANDLE “hFile” as parameter and a constant pointer to a C-style string.
  • It then calls the WriteFile() Windows API to perform the file write.
  • The WriteFile() API takes a handle to a file as first parameter and so it is a very good test of the viability of our FileHandleMarshaler custom marshaler.
  • The hFile parameter is an “in” parameter and so WriteToFile() will must consider it “read-only” and must not alter it in any way.

4.4 The following is how the WriteToFile() function is declared in C# :

[DllImport("TestDLL.dll", EntryPoint = "WriteToFile", CallingConvention = CallingConvention.StdCall, CharSet = CharSet.Ansi)]
private static extern void WriteToFile
        [In][MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef = typeof(FileHandleMarshaler))] FileStream file_stream,
        [In] string str
  • Note that the first parameter “file_stream” is of type FileStream.
  • It signifies that a FileStream object is used at the fundamental level and that we shall be using the services of a custom marshaler to transform it into something else.
  • The second parameter “str” is a string and we will rely on the default marshaler to transform it into a character string.
  • The DllImport CharSet field being set to CharSet.Ansi indicates that the string is to be marshaled in ANSI format.

4.5 The following is a sample C# function that we can use to call the WriteToFile() API :

static void DoTest()
    FileStream file_stream = new FileStream(@"c:\test.txt", FileMode.Append);

    if (file_stream != null)
        WriteToFile(file_stream, "The quick brown fox jumps over the lazy dog.\r\n");
        // Close the file stream when done.
        file_stream = null;
  • Inside DoTest(), a new FileStream object is created and a test text file “c:\test.txt” is specified to be appended.
  • Then the WriteToFile() API is called with the newly created FileStream object together with a string indicating what is to be written to the file.

5. How LongMarshaler Works at Low Level.

5.1 Central to the FileHandleMarshaler are the MarshalManagedToNative() and the CleanUpNativeData() methods.

5.2Let’s examine how the FileHandleMarshaler class works at low-level :

  • When the WriteToFile() API is called, a new FileHandleMarshaler instance is created via the static GetInstance() method.
  • Next, the MarshalManagedToNative() method is called on the instance.
  • The input argument “ManagedObj” must be of type FileStream otherwise an ArgumentException is thrown.
  • We then obtain the SafeFileHandle property of the input FileStream object.
  • According to MSDN documentation, a SafeFileHandle object represents the operating system file handle for the file that the current FileStream object encapsulates.
  • The SafeFileHandle is then checked whether it is still open (via the IsClosed property) and checked for validity (via the IsInvalid property).
  • If these criteria are met, then the underlying OS handle of the file is returned via the DangerousGetHandle() method.
  • Before that, the DangerousAddRef() method must be called to increment the reference count of the file being managed.
  • This is important because we are passing the OS handle of the file to the unmanaged side. Calling DangerousAddRef() will prevent the CLR from inadvertently reclaiming memory used by the handle.
  • Control then passed to the WriteToFile() API which will write a string to the open file.
  • Right after WriteToFile() returns, CleanUpNativeData() will be called.
  • The parameter “pNativeData” is the same value that was returned from MarshalManagedToNative().
  • This is of course the file handle.
  • A SafeFileHandle instance is created out of this file handle and DangerousRelease() is called to decrement the reference count of the file handle.

6. In Conclusion.

6.1 And there we have it. Yet another custom marshaler.

6.2 I do hope that you will find FileHandleMarshaler interesting and inspirational.

6.3 With a little bit of tweaking you may even find it practically useful in your actual projects.

6.4 Drop me a comment anytime you have any ideas on how to improve it further.

About Lim Bio Liong

I've been in software development for nearly 20 years specializing in C , COM and C#. It's truly an exicting time we live in, with so much resources at our disposal to gain and share knowledge. I hope my blog will serve a small part in this global knowledge sharing network. For many years now I've been deeply involved with C development work. However since circa 2010, my current work has required me to use more and more on C# with a particular focus on COM interop. I've also written several articles for CodeProject. However, in recent years I've concentrated my time more on helping others in the MSDN forums. Please feel free to leave a comment whenever you have any constructive criticism over any of my blog posts.


No comments yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: