//
you're reading...
Interop Marshaling

Passing Structures between Managed and Unmanaged Code

1. Introduction.

1.1 C# developers who use interop frequently often require exchanging structures between managed and unmanaged code. This can sometimes be a difficult and confusing task.

1.2 Let’s take the following structure as an example :

public struct TestStruct
{
  public int[]    m_IntArray;
  public int      m_Int;
}

1.3 This structure looks simple but how do we pass such a structure to a C++ API defined as follows :

void __stdcall TestAPI(/*[in, out]*/ TestStruct* pTestStruct)
{
  for (int i = 0; i < 10; i++)
  {
    (pTestStruct -> m_IntArray)[i]++;
  } 

  (pTestStruct -> m_Int)++;
}

The above API is declared in C# as follows :

[DllImport("TestDLL.dll", CallingConvention = CallingConvention.StdCall)]
private static extern void TestAPI(ref TestStruct test_struct);

This API takes a TestStruct struct as a reference parameter. Internally it will perform modification of the input structure. When the call returns, the C# code is expected to see changes to the TestStruct struct.

1.5 If you simply run the following code :

TestStruct test_struct = new TestStruct();

test_struct.m_IntArray = new int[10]; 

for (int i = 0; i < test_struct.m_IntArray.Length; i++)
{
  (test_struct.m_IntArray)[i] = 0;
} 

test_struct.m_Int = 0;

TestAPI(ref test_struct);

You will very likely run into some error. The problem is that the C++ API treats the TestStruct’s m_IntArray member as a C-style array whereas the .NET runtime will convert it into a SAFEARRAY.

2. Interop Marshaling, Blittable/Non-Blittable Types, Reference/Value Types.

2.1 To understand the nature of the problem, we need to understand the concepts of blittable and non-blittable types as well as reference and value types and how they are related to interop marshaling.

2.2 By “blittable type”, we mean types that have a common representation in both managed and unmanaged memory and do not require conversion when they are passed between managed and unmanaged memory.

2.3 In short, their data can be copied directly from managed memory to unmanaged memory and vice versa without any special processing by the interop marshaler.

2.4 An example of a blittable type is the C# int type. Assuming 32-bit applications, an instance of a C# int takes up 4 bytes, the same as a C++ int type which also takes up 4 bytes. Furthermore, their binary representation in memory are the same. Hence a C# int variable in managed memory can be directly copied (blitted) to a C++ int variable in unmanaged memory. And vice versa.

2.5 Non-blittable-types are just the opposite of blittable-types : i.e. types that do not have a common representation in both managed and unmanaged memory and that may require conversion when they are passed between managed and unmanaged memory. The conversion is performed by the interop marshaler.

2.6 An example of a non-blittable type is the string. A .NET string is intrinsically unicode based and does not have a direct unmanaged equivalent. An unmanaged string may be C-style null-terminated, or it may be a BSTR (COM string). Furthermore an unmanaged string may be unicode-based or multi-byte character-based.

2.7 Another example is the array type. A .NET array is an object and has an internal representation that is not documented albeit the actual array data is as expected : a contiguous range of data packed back-to-back in memory somewhere. However, it has no direct unmanaged equivalent. If not specified using the MarshalAsAttribute, its default unmanaged  equivalent is the SAFEARRAY (see point 1.5 above).

2.8 You might be surprised to note that a .NET char type is actually non-blittable. This is because a .NET char is unicode-based. The same is true of a .NET Boolean which can be represented in unmanaged code in various formats.

2.9 The overarching thing about non-blittable types is that they do not have a single common unmanaged equivalent. Such types, when required to be marshaled over to unmanaged code, will need specifications in order for the interop marshaler to be able to successfully perform the marshaling. These specifications come in the form of MarshalAsAttribute‘s.

2.10 By “reference type”, we mean types the actual data of which are allocated in the managed heap. They are reference counted internally and are de-allocated by the Garbage Collector. Each programing language variable that is assigned to a value of such a type merely holds a reference to that value. The variable does not get assigned another copy of the value.

2.11 The opposite of a reference type is the “value type”. Value type data objects reside on the stack. They are not managed by the Garbage Collector. They are destroyed when they go out of scope. Each programming variable that is assigned a value-type value gets its own copy of that value.

2.12 Now, how are the concepts of blittability and reference and value types relevant to the using structs in unmanaged code ? The following is a summary :

  • When we want to share a managed struct with unmanaged code, a good optimization technique would be to directly pass the struct to the unmanaged code.
  • This can only be done if the member data of the managed struct are all contained within the memory block of the struct itself. That is, the struct self-contains all member data just like a C/C++ class/struct.
  • Now this is only possible if all members are blittable types. If any member is non-blittable, the entire struct is non-blittable.
  • The advantage of a completely blittable struct is that such a struct may be directly passed to unmanaged code. Now in order to directly pass it to unmanaged code, it needs to be pinned in memory, otherwise, not only can it be moved around by the GC, it may even be completely de-allocated by the GC. Pinning it in memory is especially important if the unmanaged API receives a pointer to the struct.
  • The unmanaged code may then reference the struct directly and may even make modifications to it if this is expected. When the unmanaged code returns, the struct may be unpinned from memory.
  • Now, if a struct is non-blittable, it will not be possible to directly pass it to unmanaged code. However, all is not lost. It will still be possible to “pass” the struct to unmanaged code. However, this “struct” is a memory copy of the struct. The creation of such a struct requires the help of the interop marshaler.
  • This copy must contain the actual data of all the members of the struct. It serves as a an unmanaged representation of the original struct. The copy is made with the help of the various MarshalAsAttribute’s.
  • It is this unmanaged representation that gets passed to the unmanaged code. Now, if the unmanaged code is expected to make modifications to this representation, then when the unmanaged code returns, the (now modified) data from the representation must be copied back to the original managed struct.
  • Eventually, the unmanaged representation must be freed.
  • The above description resembles COM paramater marshaling across apartments where parameter data may be re-created in the callee’s stack.

3. The Members of the TestStruct struct.

3.1 Now, let’s examine the TestStruct structure :

public struct TestStruct
{
  public int[] m_IntArray;
  public int m_Int;
}

This structure contains one int member :

public int m_Int;

and one managed array member :

public int[] m_IntArray;

3.2 In C#, the int type is a value type and is a blittable type. When used as a member of a struct, it can directly reside as part of the memory block of the struct itself. Hence m_Int form part of the body of the TestStruct structure. It is destroyed when an instance of the TestStruct struct is destroyed.

3.3 Therefore, when we have code like the following :

TestStruct test_struct = new TestStruct();
test_struct.m_Int = 0;

The value of member “m_Int” (i.e. zero) resides directly as part of the memory block for “test_struct”. The situation is exactly the same as that for the members of a C/C++ structure.

3.4 Now a managed array, when used as a member of a struct, renders the containing struct non-blittable. This is because a managed array is essentially non-blittable. Furthermore it is also a reference type. That is, its actual array data must exist in the managed heap and is merely referenced by the struct member.

3.5 Therefore, when we have code like the following :

test_struct.m_IntArray = new int[10];

The array of 10 int values resides in the managed heap somewhere and is referenced by “test_struct.m_IntArray”. The actual array data of 10 int’s is not part of the memory block of the test_struct object. This is totally unlike what we may be familiar with in C/C++.

3.6 This is the reason why we cannot pin down in memory a struct that contains an array :

// The following code will not work.
GCHandle gch = GCHandle.Alloc(test_struct, GCHandleType.Pinned);
IntPtr p = gch.AddrOfPinnedObject();

In fact, we cannot pin a struct that contains members which are reference types (including the string type).

To pin such a struct would also require that the referenced object be pinned. This might seem acceptable initially until you consider code like the following :

TestStruct test_struct = new TestStruct();
int [] int_array = new int[10];
test_struct.m_IntArray= int_array; //new int[10];

That is, “test_struct.m_IntArray” does not reference a newly allocated int array. Instead, it references an existing one. Pinning “test_struct.m_IntArray” would also logically require that “int_array” be pinned. This may not be a desireable thing.

4. How Do We Marshal the TestStruct structure to an Unmanaged API ?

4.1 Although TestStruct is not blittable (due to the presence of member int_array), it is nevertheless possible to pass such a structure to unmanaged code.

4.2 However, note well that under the covers, an unmanaged copy of TestStruct is created. This copy is actually a representation of TestStruct. We have seen that TestStruct does not have a flat memory layout due to the presence of an array member. Therefore a managed instance of TestStruct cannot be directly used by unmanaged code.

4.3 Hence, in order to pass such a structure to unmanaged code, an unmanaged version of TestStruct is required. The Interop Marshaler will allocate in memory a buffer which is large enough to hold such an unmanaged replica of TestStruct. This is done using Marshal.SizeOf().

4.4  In order to calculate the required size of the buffer, the interop marshaler must know how the individual members of the managed structure are to be represented in the unmanaged counterpart. This includes the positioning of each member and any padding between members. Now this is where the StructLayoutAttribute and the MarshalAsAttribute come in handy. Look up MSDN for the documentation of the StructLayoutAttribute and the MarshalAsAttribute in order to understand how to use them.

4.5 For the TestStruct structure, we applied these 2 attributes in the following way :

[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct TestStruct
{
  [MarshalAs(UnmanagedType.ByValArray, SizeConst = 10)]
  public int[]    m_IntArray;
  public int      m_Int;
}

The StructLayoutAttribute indicate to the interop marshaler that the members of the unmanaged replica are to appear sequentially (in the same order as their appearance in the managed counterpart). The members are also to be packed back-to-back (due to Pack=1).

The MarshalAsAttribute applied to the “m_IntArray” member indicates that this member is to be represented as a C-style array of int type. The SizeConst argument indicates that the array is to have 10 elements.

4.6 When taken together, these information indicate that the unmanaged replica of TestStruct will have the following C/C++ representation :

#pragma pack(1)
struct TestStruct
{
  int m_IntArray[10];
  int m_Int;
};

4.7 Now that we have fully prepared TestStruct for interop marshaling, how do we use it as a parameter to the TestAPI() function ?

4.8 The following is a sample call :

// Allocate a TestStruct struct.
TestStruct test_struct = new TestStruct();
// Assign values into test_struct.
test_struct.m_IntArray = new int[10];
for (int i = 0; i < test_struct.m_IntArray.Length; i++)
{
  (test_struct.m_IntArray)[i] = 0;
} 

test_struct.m_Int = 0;
TestAPI(ref test_struct);

Notice that it is very similar to the code in point 1.5 above. This time, the code will go through successfully and you will see actual changes to the managed test_struct.

4.9 How were all these good effects brought about ? In fact, the only changes were to the declaration of TestStruct (via the StructLayoutAttribute and the MarshalAsAttribute).

4.10 These attributes established what the unmanaged representation of TestStruct will look like in unmanaged memory.

5. The Marshaling Process.

Now, how does the interop marshaler perform the copying of each struct member from the managed TestStruct to the unmanaged TestStruct ? This section describes a possible scenario of how this is actually done under the hood.

5.1 The interop marshaler allocate memory that provides an unmanaged representation of an instance of the TestStruct struct. This is done with the help of Marshal.AllocHGlobal(). The size is determined via Marshal.SizeOf() and it requires the data contained in the StructLayoutAttribute and various MarshalAsAttribute’s.

5.2 The interop marshaler then copies the actual data of the members of the TestStruct instance to this unmanaged memory. The m_Int member is blitted to the unmanaged buffer. The copying of the m_IntArray member is done with specifications provided by the MarshalAsAttribute which was applied to the m_IntArray member. The interop marshaler likely uses the Marshal.StructureToPtr() method.

5.3 The unmanaged API is then invoked passing along a pointer to the unmanaged memory. This pointer is likely an IntPtr type.

5.4 After the API call, because the API has modified the structure, the returned data in the unmanaged structure must be copied back to the managed TestStruct structure. The m_Int member, being blittable, is transferred directly to its managed counterpart. The m_IntArray member is copied back to its managed counterpart with information provided by the MarshalAsAtttibute information contained in the member. This reverse copying from unmanaged memory to the managed struct is likely performed via Marshal.PtrToStructure.

5.5 Finally, the unmanaged TestStruct struct is freed by the interop marshaler. This is done via Marshal.FreeHGlobal().

6. Example Code.

An example code, written to illustrate the concepts discussed in section 5 above, is given below.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Runtime.InteropServices; 

namespace CSConsoleClient01
{
  [StructLayout(LayoutKind.Sequential, Pack = 1)]
  public struct TestStruct
  {
    [MarshalAs(UnmanagedType.ByValArray, SizeConst = 10)]
    public int[] m_IntArray;
    public int m_Int;
  } 

  class Program
  {
    [DllImport("TestDLL01.dll", CallingConvention = CallingConvention.StdCall, EntryPoint = "TestAPI")]
    private static extern void TestAPI(IntPtr pTestStruct); 

    static void DemonstrateCallByIntPtr()
    {
      // Allocate a TestStruct struct.
      TestStruct test_struct = new TestStruct();
      // Assign values into test_struct.
      test_struct.m_IntArray = new int[10];
      for (int i = 0; i < test_struct.m_IntArray.Length; i++)
      {
        (test_struct.m_IntArray)[i] = 0;
      } 

      test_struct.m_Int = 0; 

      // Obtain the size of an unmanaged serialized
      // representation of a TestStruct struct.
      int iSize = Marshal.SizeOf(typeof(TestStruct));
      // Allocate memory (in the Global Heap) for the unmanaged
      // representation of a TestStruct struct.
      IntPtr pTestStruct = Marshal.AllocHGlobal(iSize);
      // Copy the full contents of the members of test_struct
      // to the unmanaged representation of TestStruct.
      // Data from the members of test_struct will be
      // serialized into a flat memory format and stored
      // into the unmanaged serialized representation.
      Marshal.StructureToPtr(test_struct, pTestStruct, false);
      // Call the IntPtr version API.
      TestAPI(pTestStruct);
      // Copy the modified contents of the unmanaged representation
      // of test_struct back to the members of the managed test_struct.
      test_struct = (TestStruct)(Marshal.PtrToStructure(pTestStruct, typeof(TestStruct)));
      // Free the unmanaged representation of test_struct.
      Marshal.FreeHGlobal(pTestStruct);
    } 

    static void Main(string[] args)
    {
      DemonstrateCallByIntPtr();
    }
  }
}

The code for TestAPI() and the TestStruct structure are listed below :

#pragma pack(1) struct TestStruct
{
  int m_IntArray[10];
  int m_Int;
}; 

void __stdcall TestAPI(/*[in, out]*/ TestStruct* pTestStruct)
{
  for (int i = 0; i < 10; i++)
  {
    (pTestStruct -> m_IntArray)[i] = i;
  } 

  pTestStruct -> m_Int = 100;
}
Advertisements

About Lim Bio Liong

I've been in software development for nearly 20 years specializing in C , COM and C#. It's truly an exicting time we live in, with so much resources at our disposal to gain and share knowledge. I hope my blog will serve a small part in this global knowledge sharing network. For many years now I've been deeply involved with C development work. However since circa 2010, my current work has required me to use more and more on C# with a particular focus on COM interop. I've also written several articles for CodeProject. However, in recent years I've concentrated my time more on helping others in the MSDN forums. Please feel free to leave a comment whenever you have any constructive criticism over any of my blog posts.

Discussion

7 thoughts on “Passing Structures between Managed and Unmanaged Code

  1. Hey,
    great blog had some trouble understanding interop but you really helped me however I was wondering What if in the example on this page int m_Int is actually a std::string or a char*. How would you marshal it in the c# code?
    Great blog again

    Posted by apmbaye | April 5, 2013, 8:50 pm
  2. hi, it is a great blog. And I have a problem with passing a pointer to structure. Here is the two c structures:

    typedef struct
    {
    int num; /* section number */
    int len; /* section taps length in use */
    int xId; /* section input data index */
    int tId; /* section taps index */
    } firSect;

    typedef struct _NSPFirState
    {
    int upFactor; /* up */
    int upPhase; /* parameters */
    int downFactor; /* down */
    int downPhase; /* parameters */
    int isMultiRate; /* multi-rate mode flag */
    void* isInit; /* init flag */

    int tapsLen; /* number of filter taps */
    int tapsBlk;
    void* taps; /* taps pointer in use */

    int dlylLen; /* delay line length */
    int inpLen; /* input buffer length */
    int useInp; /* input buffer length used */
    void* dlyl; /* extended dilter delay line */
    void* dlylLimit; /* delay line buffer end pointer */
    void* useDlyl; /* delay line pointer in use */

    firSect* sect; /* FIR sections */

    int tapsFactor; /* taps scale factor */
    int utapsLen; /* number of filter taps (user def) */
    } NSPFirState;

    And here is two method I need to call in C#:
    void nspsFirInit(const float *tapVals, int tapsLen, const float *dlyVals, NSPFirState *statePtr))
    double nspdFir(NSPFirState *statePtr,double samp))

    I handle them in c# like this:
    [DllImport(“nsp.dll”, EntryPoint = “nspsFirInit”, CallingConvention = CallingConvention.StdCall)]
    private static extern void nspsFirInit([MarshalAs(UnmanagedType.LPArray)]double[] tapVals, int tapsLen, [MarshalAs(UnmanagedType.LPArray)]double[] dlyVals, ref NSPFirState statePtr);

    [DllImport(“nsp.dll”, EntryPoint = “nspsFir”, CallingConvention = CallingConvention.StdCall)]
    private static extern double nspsFir(ref NSPFirState statePtr, double samp);

    struct firSect
    {
    public int num; /* section number */
    public int len; /* section taps length in use */
    public int xId; /* section input data index */
    public int tId; /* section taps index */
    } ;

    struct NSPFirState
    {
    public int upFactor; /* up */
    public int upPhase; /* parameters */
    public int downFactor; /* down */
    public int downPhase; /* parameters */
    public int isMultiRate; /* multi-rate mode flag */
    public IntPtr isInit; /* init flag */

    public int tapsLen; /* number of filter taps */
    public int tapsBlk;
    public IntPtr taps; /* taps pointer in use */

    public int dlylLen; /* delay line length */
    public int inpLen; /* input buffer length */
    public int useInp; /* input buffer length used */
    public IntPtr dlyl; /* extended dilter delay line */
    public IntPtr dlylLimit; /* delay line buffer end pointer */
    public IntPtr useDlyl; /* delay line pointer in use */

    //[MarshalAs(UnmanagedType.ByValArray, SizeConst = 1)]
    //public firSect[] sect; /* FIR sections */

    //public IntPtr sect = Marshal.AllocHGlobal(4 * sizeof(int));

    public unsafe firSect* sect;

    public int tapsFactor; /* taps scale factor */
    public int utapsLen; /* number of filter taps (user def) */
    };

    I have no idea how to pass the firSect* sect pointer to my c# code. Could you give some suggestions?

    Posted by shawn shen | July 10, 2014, 1:33 pm
    • Hello Shawn,

      1. The “NSPFirState.sect” should be declared as an IntPtr.

      2. At runtime, what we want is to allocate a firSect structure, fill its members, and then create in unmanaged memory an unmanaged structure which is equivalent to firSect.

      3. This is done by calling a memory allocation method like Marshal.AllocCoTaskMem() or Marshal.AllocHGlobal(). Followed by a call to Marshal.StructureToPtr().

      4. We then set “NSPFirState.sect” to this pointer to the unmanaged firSect.

      5. We then call the nspsFirInit() or nspsFir().

      6. On return from these function calls, the “NSPFirState.sect” field must be converted from an unmanaged firSect structure to its managed version. This is done by calling Marshal.PtrToStructure().

      7. Finally the unmanaged firSect structure must be freed by calling Marshal.FreeCoTaskMem() or Marshal.FreeHGlobal().

      8. The following is an example :

      static void DoTest()
      {
      NSPFirState _NSPFirState = new NSPFirState();
      firSect _firSect = new firSect();

      _firSect.num = 1;
      _firSect.len = 2;
      _firSect.xId = 3;
      _firSect.tId = 4;

      int iSize = Marshal.SizeOf(typeof(firSect));

      // Allocate unmanaged memory for the space
      // required for the firSect structure.
      _NSPFirState.sect = Marshal.AllocCoTaskMem(iSize);
      // Copy the contents of the firSect struct
      // into its unmanaged counterpart (pointed to
      // by p_firSect).
      Marshal.StructureToPtr(_firSect, _NSPFirState.sect, false);

      // Now call nspsFirInit().
      nspsFirInit(null, 0, null, ref _NSPFirState);

      // After the call to nspsFirInit(), _NSPFirState.sect
      // would still point to the unmanaged memory
      // which contains the unmanaged firSect struct.
      //
      // We must now convert _NSPFirState.sect to a managed
      // firSect struct.
      _firSect = (firSect)(Marshal.PtrToStructure(_NSPFirState.sect, typeof(firSect)));

      // Must free the unmanaged memory of p_firSect when no longer needed.
      Marshal.FreeCoTaskMem(_NSPFirState.sect);
      _NSPFirState.sect = IntPtr.Zero;
      }

      Posted by Lim Bio Liong | July 12, 2014, 5:26 am
  3. Hi Bio,
    Is it the same code architecture and structure in TestStruct, if you use a “2D string array” instead of the “1D int array” in your example .. ?
    Thanks.

    Posted by Steve | November 6, 2015, 5:44 pm

Trackbacks/Pingbacks

  1. Pingback: Passing Managed Structures With Arrays To Unmanaged Code Part 1 « limbioliong - June 29, 2011

  2. Pingback: Passing a Pointer to a Structure from C# to C++ Part 1. « limbioliong - August 20, 2011

  3. Pingback: Passing Managed Structures With Strings To Unmanaged Code Part 2 « limbioliong - August 24, 2011

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: