you're reading...
Interop Marshaling

Understanding Custom Marshaling Part 3

1. Introduction.

1.1 This is the 3rd installment of our series of articles expounding .NET custom marshaling.

1.2 In part 2, we studied how custom marshaling may be performed from unmanaged code to managed. The idea being that the unmanaged code must return an object to managed code.

1.3 We also noted in part 2 that there are 2 ways an object may be returned : via an “out” parameter and via the return value of the function itself.

1.4 In our sample code of part 2, we returned through an “out” parameter. Here in part 3, we shall return by way of the return value.

2. Client Application.

2.1 Just as in part 1, we need a client application and a DLL to which we make a p/invoke call.

2.2 We will use the same client application and the DLL that we used in part 1 but will embellish both with new functions.

2.3 We will specifically be returning a pointer to a C-style null-terminated string by way of a function which returns such a pointer. Towards this end, we shall create the following new API :

const char* __stdcall ReturnString()
	char lpszString [] = "The quick brown fox jumps over the lazy dog";

	char* pszStringToReturn = (char*)::CoTaskMemAlloc(sizeof(lpszString));

	strcpy_s(pszStringToReturn, sizeof(lpszString), lpszString);

	return pszStringToReturn;

Notice that ReturnString() will internally allocate a buffer using CoTaskMemAlloc() and then fill this buffer with the string “The quick brown fox jumps over the lazy dog”. A pointer to this allocated buffer is then directly returned. This return value (a memory address) will be returned in the EAX register.

The CLR, understanding this arrangement, will read the value in the EAX register and treat it as a pointer to unmanaged data.This is illustrated in the diagram below :


We will see this in action later in our example code.

2.4 The C# DllImport declarations for the above function is listed below :

[DllImport("TestDLL.dll", EntryPoint = "ReturnString", CallingConvention = CallingConvention.StdCall)]
[return:MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef = typeof(StringMarshaler))]
private static extern string ReturnString();

Note how we use the MarshalAsAttribute to indicate the marshaler to be used for obtaining the return value from the ReturnString() API.

Notice also that we directly use the managed type (i.e. string) as the return type for the API. However, from point 2.3 we see that the API actually returns a char pointer at low level. This is where the marshaler comes into play and where ICustomMarshaler::MarshalNativeToManaged() is specifically used to convert the unmanaged char pointer to a managed string. More on this later.

2.5 As for the client application, we shall add a new function which is listed below :

static void DoTest_ReturnString()
    string str = ReturnString();


It is very simple indeed. A managed string “str” is declared and is initialized with the return value from ReturnString(). The string “str” is then displayed on the console.

3. Client Application In Action : DoTest_ReturnString().

3.1 Let’s look at DoTest_ReturnString() and examine the steps that will be taken that will lead to control being passed to the ReturnString() API and then back to the C# function :

  • The static StringMarshaler::GetInstance() method will be called by the CLR in order to obtain an instance of the StringMarshaler class.
  • Control will then reach the actual ReturnString() function (inside TestDLL.dll). No parameter is required.
  • ReturnString() allocates a buffer using CoTaskMemAlloc() and then assigns a C-style NULL-terminated string to this buffer :


  • After that the pointer to this buffer is returned and the EAX register is filled with this pointer value :


  • The value in the EAX register is then copied by the CLR into some variable of its own management. This return value is important an will be used in more calls into StringMarshaler.
  • Control then reaches StringMarshaler::MarshalNativeToManaged() with the parameter “pNativeData” (an IntPtr) bearing the same value as the return value of ReturnString() (in our example, this is 0x005afe20) :


  • This turn out of events (i.e. the return value from ReturnString() being directly passed to StringMarshaler::MarshalNativeToManaged() as parameter) is by design of course.
  • It allows StringMarshaler::MarshalNativeToManaged() to use it to create a managed string via Marshal.PtrToStringAnsi().
  • The managed string thus created will later be passed to the DoTest_ReturnString() function but before that happens, StringMarshaler::CleanUpNativeData() will be called to free the C-style string that was allocated (using CoTaskMemAlloc()) in the ReturnString() function :


  • Marshal.FreeCoTaskMem() will be used in the memory de-allocation and we see from the diagram below that after Marshal.FreeCoTaskMem() is called, the memory buffer is freed :


The memory buffer that used to hold the C-style string “The quick brown fox jumps over the lazy dog” has its contents erased.

  • Control finally returns to DoTest_ReturnString() and the cycle of interop marshaling concludes.

3.2 The following are pertinent points about the underlying activities behind the call to DoTest_ReturnString() :

  • Note the DllImport definition for the ReturnString() function (see section 2.4). The returned string is taken to be an “out” value.
  • Hence it is the caller (i.e. the client application) which will own the returned string.
  • Low-level wise, the character pointer returned from ReturnString() will also be owned by the caller.
  • Memory ownership obliges memory freeing. In other words, the owner of a memory buffer is responsible for its eventual de-allocation.
  • This is why StringMarshaler::CleanUpNativeData() must be called to free it on behalf of the client application.
  • Next, note how the ReturnString() API creates this string (point 2.3). It does this by allocating a buffer using CoTaskMemAlloc() and then assigns a C-style string value to this buffer.
  • Just as we saw in part 2, because the low-level ReturnString() uses CoTaskMemAlloc() for buffer allocation, the StringMarshaler::CleanUpNativeData() function must use Marshal.FreeCoTaskMem() to free this buffer.
  • Again, this CoTaskMemAlloc()/Marshal.FreeCoTaskMem() pair is purely a pre-arranged design. Developers are free to use whatever memory allocation/de-allocation mechanism deemed suitable.
  • Even the C++ “new” can be used as long as some means is available (e.g. an exported API) which will perform the appropriate C++ “delete”.
  • Next note that StringMarshaler::MarshalNativeToManaged() is called because the low-level string returned from ReturnString() is a C-style string and as such cannot be used directly in managed code.
  • Marshal.PtrToStringAnsi() must be called to convert the C-style string into a managed one.
  • And finally, Marshal.PtrToStringAnsi() is used because the C-style string allocated in ReturnString() is assumed to be in ANSI format.

4. In Summary.

4.1 With this part, we conclude our study of custom marshaling from unmanaged code to managed.

4.2 In the next installment of this series of articles, we shall study marshaling done in both directions : from managed to unmanaged and back to managed again.


About Lim Bio Liong

I've been in software development for nearly 20 years specializing in C , COM and C#. It's truly an exicting time we live in, with so much resources at our disposal to gain and share knowledge. I hope my blog will serve a small part in this global knowledge sharing network. For many years now I've been deeply involved with C development work. However since circa 2010, my current work has required me to use more and more on C# with a particular focus on COM interop. I've also written several articles for CodeProject. However, in recent years I've concentrated my time more on helping others in the MSDN forums. Please feel free to leave a comment whenever you have any constructive criticism over any of my blog posts.


No comments yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: