//
you're reading...
Programming Issues/Tips

Quick Serialization/Deserialization of a Managed Structure.

1. Introduction.

1.1 Serialization/deserialization of some data structure is a common and useful programming requirement.

1.2 Serialized data may be transferred over a network to another machine to be deserialized back as a data structure. Or it may be saved to the disk for storage as part of the data file of an application.

1.3 This blog aims to demonstrate a quick way of performing such serialization/deserialization using the Marshal.StructureToPtr() and Marshal.PtrToStructure() methods.

1.4 I will demonstrate how to serialize a managed structure into an array of bytes which can then be saved into an external file.

1.5 I will also demonstrate the converse : the deserialization of an array of bytes into a data structure.

2. Definition of the Test Data Structure.

2.1 Throughout this blog we shall be working with the following data structure :

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode, Pack = 1)]
struct MyDataStruct
{
  [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 32)]
  public string strMember;
  public Int32 int32Member;
  public byte byteMember;
  public double dblMember;
  [MarshalAs(UnmanagedType.ByValArray, ArraySubType = UnmanagedType.I1, SizeConst = 10)]
  public char[] chArrayMember;
}

The following are pertinent points about this structure :

  • This structure has been marked with the StructLayoutAttribute with the LayoutKind.Sequential parameter which indicates that this structure will be arranged with fields laid out in sequence as they appear in the declaration.
  • The use of the StructLayoutAttribute together with LayoutKind.Sequential parameter is very important and without this attribute the C# compiler will specify it automatically together with the LayoutKind.Sequential layout kind argument for a structure.
  • The CharSet argument will influence the way string and char members are serialized. More on this argument will be discussed later when we study the codes that will perform the serialization (see section 3).
  • The Pack argument will influence the amount of space between field members in memory and will eventually affect the size of the serialized data. I have chosen the value of 1 so as to ensure the minimal size for the serialized data.
  • Now there are altogether 5 member fields to this structure. 3 of them (int32Member, byteMember and dblMember) are blittable. 2 of them are not blittable : strMember (type string) and chArrayMember (character array).
  • The blittable members are serialized directly as they appear in binary form.
  • The non-blittable members cannot be serialized without the help of MarshalAsAttributes. I shall expound on this further when we discuss the codes that perform the serialization (see section 3).

2.2 For greater logical integration, I have decided to embed the functions for serialization and deserialization of the MyDataStruct structure inside the structure itself. Hence the final form for MyDataStruct will be something like the following :

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode, Pack = 1)]
struct MyDataStruct
{
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 32)]
    public string strMember;
    public Int32 int32Member;
    public byte byteMember;
    public double dblMember;
    [MarshalAs(UnmanagedType.ByValArray, ArraySubType = UnmanagedType.I1, SizeConst = 10)]
    public char[] chArrayMember;

    // Member function for serialization.
    public byte[] Serialize()
        {
          ...
        }

    // Member function for deserialization.
    public void Deserialize(ref byte[] byteSerializedData)
        {
          ...
        }
}

3. Sample Code For Serialization.

3.1 Listed below is the code for the member function Serialize() :

public byte[] Serialize()
{
    // Get the byte size of a MyDataStruct structure if it is to be
    // marshaled to unmanaged memory.
    Int32 iSizeOMyDataStruct = Marshal.SizeOf(typeof(MyDataStruct));
    // Allocate a byte array to contain the bytes of the unmanaged version
    // of the MyDataStruct structure.
    byte[] byteArrayMyDataStruct = new byte[iSizeOMyDataStruct];
    // Allocate a GCHandle to pin the byteArrayMyDataStruct array
    // in memory in order to obtain its pointer.
    GCHandle gch = GCHandle.Alloc(byteArrayMyDataStruct, GCHandleType.Pinned);
    // Obtain a pointer to the byteArrayMyDataStruct array in memory.
    IntPtr pbyteArrayMyDataStruct = gch.AddrOfPinnedObject();
    // Copy all bytes from the managed MyDataStruct structure into
    // the byte array.
    Marshal.StructureToPtr(this, pbyteArrayMyDataStruct, false);
    // Unpin the byteArrayMyDataStruct array in memory.
    gch.Free();
    // Return the byte array.
    // It contains the serialized bytes of the MyDataStruct structure.
    return byteArrayMyDataStruct;
}

The following is a general synopsis of the Serialize() function :

  • The objective is to produce a byte array which contains the in-memory values of the fields of the MyDataStruct structure.
  • The approach that we take is to express the MyDataStruct structure in its unmanaged form as if we are preparing it for marshaling to unmanaged code.
  • This is done with none other than the Marshal.StructureToPtr() function.
  • Once we have such an unmanaged version of the MyDataStruct structure, it is essentially a series of bytes in unmanaged memory.
  • This series of bytes can be deemed as the serialized form of the structure.

3.2 With the basic objective of the Serialize() function explained, I now expound on the techniques used to achieve this objective :

  • The Serialize() function first creates the byte array which will contain the serialized bytes of the MyDataStruct structure and which will also be the return value of the function.
  • The size of the byte array is determined by the size of the structure as it would be in unmanaged memory. This is determined by the Marshal.SizeOf() function.
  • The Marshal.SizeOf() function requires proper application of the MarshalAsAttributes to the field members of the structure especially those that are not blittable.
  • After the byte array has been created, we will immediately lock it in memory in order to obtain a pointer to it. This is done with the help of the GCHandle class and with effective use of the GCHandle.Alloc() function with the GCHandleType.Pinned parameter.
  • After we have obtained a pointer to the memory-pinned byte array (through the use of the GCHandle.AddrOfPinnedObject() function), we call on Marshal.StructureToPtr() to marshal the MyDataStructure structure to this array data which is treated as if it is unmanaged memory.
  • When this is completed, we can unpin the byte array through GCHandle.Free().
  • The byte array will now contain the serialized data of the structure and it can be returned.

3.3 As mentioned earlier in point 2.1, the blittable members of the structure will be serialized using default marshaling. They will be serialized as they appear in memory. The non-blittable members will require appropriate use of the MarshalAsAttributes in order to be serialized.

3.4 The strMember member is a string with the following MarshalAsAttribute applied :

[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 32)]

This indicates to the interop marshaler that the string is to be marshaled as an inline array of 32 characters. Now, because the MyDataStruct structure is attributed with the StructLayoutAttribute and the CharSet argument set to CharSet.Unicode, this inline array of 32 characters will be Unicode characters. This means that when marshaled, the strMember member will take up 64 (32 x 2) bytes.

3.5 The chArrayMember member is an array of characters. It is marked with the following attribute :

[MarshalAs(UnmanagedType.ByValArray, ArraySubType = UnmanagedType.I1, SizeConst = 10)]

Due to the above description of the MarshalAsAttribute, chArrayMember will be serialized as an inline array of 10 characters. Now with the ArraySubType argument being set to equal UnmanagedType.I1, chArrayMember will be serialized as an array of single-byte characters instead of 2-byte Unicode characters (the default). This means that when marshaled, the chArrayMember member will take up exactly 10 bytes.

This MarshalAsAttribute was used to demonstrate to the user its power and the flexibility the user possesses in expressing how the structure is to be serialized.

3.6 The following is a sample code that uses the MyDataStruct.Serialize() function to serialize a MyDataStruct structure and then save the serialized data to an external disk file named “MyDataStruct.bin” :

public static void TestSerializeMyDataStruct()
{
    // Allocate a new MyDataStruct structure.
    MyDataStruct my_data_struct = new MyDataStruct();

    // Assign values to the fields of the structure.
    my_data_struct.strMember = "Hello World";
    my_data_struct.int32Member = 100;
    my_data_struct.byteMember = (byte)'A';
    my_data_struct.dblMember = 0.123456;
    my_data_struct.chArrayMember = new char[10];
    for (int i = 0; i < my_data_struct.chArrayMember.Length; i++)
    {
        my_data_struct.chArrayMember[i] = (char)('A' + (char)i);
    }

    // Serialize the MyDataStruct structure into an array
    // of bytes.
    byte[] byArraySerializedData = my_data_struct.Serialize();

    // Save the array contents into an external file.
    FileStream file_stream = File.OpenWrite("MyDataStruct.bin");
    // Write the full contents of byArraySerializedData into the file stream.
    file_stream.Write(byArraySerializedData, 0, byArraySerializedData.Length);
    // Close the file stream when done.
    file_stream.Close();
}

3.7 When viewed in memory, the contents of byArraySerializedData would be as they are displayed in the 2 diagrams below :

4. Sample Code For Deserialization.

4.1 Listed below is the code for the member function Deserialize() :

public void Deserialize(ref byte[] byteSerializedData)
{
    // Allocate a GCHandle of the Pinned Type
    // for the byteSerializedData byte array.
    // This is possible for a byte array
    // because it is a blittable type.
    GCHandle gch = GCHandle.Alloc(byteSerializedData, GCHandleType.Pinned);
    // Get a pointer to the byteSerializedData array.
    IntPtr pbyteSerializedData = gch.AddrOfPinnedObject();
    // Convert the array data of byteSerializedData
    // directly into a MyDataStruct structure.
    // The interop marshaler will use the MarshalAsAttribute
    // of the fields of the MyDataStruct structure to
    // perform field data conversions.
    this = (MyDataStruct)Marshal.PtrToStructure(pbyteSerializedData, typeof(MyDataStruct));
    // Free the GCHandle.
    gch.Free();
}

The following is a general synopsis of the Deserialize() function :

  • The objective is to take a byte array which contains the serialized data and then reconstruct a MyDataStruct structure from it.
  • The approach that we take is to treat the serialized data as if it contains the marshaled form of a MyDataStruct structure in unmanaged memory.
  • We then use the Marshal.PtrToStructure() function to convert the marshaled data into a managed MyDataStruct structure.

4.2 With the basic objective of the Deserialize() function explained, I now expound on the techniques used to achieve this objective :

  • The input byte array byteSerializedData is first pinned in memory with the help of the GCHandle class.
  • It is possible to pin this array because its members are bytes which is blittable.
  • Once pinned, we obtain a pointer to its memort using GCHandle.AddrOfPinnedObject().
  • We then use Marshal.PtrToStructure() to convert the serialized data into a managed MyDataStruct structure.
  • The serialized data is treated as if it contains the unmanaged version of a MyDataStruct structure.
  • The call to Marshal.PtrToStructure() is successful because of the specifications of an unmanaged version of the MyDataStruct structure as given by the StructLayoutAttribute applied to the MyDataStruct structure as well as the various MarshalAsAttributes applied to the non-blittable field members.

4.3 The following is a sample code which reads the byte data from a previously saved “MyDataStruct.bin” file and then uses it to reconstruct a MyDataStruct structure :

public static void TestDeserializeMyDataStructure()
{
    FileStream file_stream = File.OpenRead("MyDataStruct.bin");

    // Allocate a byte array the size of the length of the file.
    byte[] byArraySerializedData = new byte[file_stream.Length];
    // Read the full contents of file into the file stream.
    file_stream.Read(byArraySerializedData, 0, byArraySerializedData.Length);
    // Close the file stream when done.
    file_stream.Close();

    // Deserialize the contents of byArraySerializedData into
    // a MyDataStruct structure.
    MyDataStruct my_data_struct = new MyDataStruct();
    my_data_struct.Deserialize(ref byArraySerializedData);
}

5. In Conclusion.

5.1 From this article, I hope the reader will appreciate the power of the Marshal.StructureToPtr() and the Marshal.PtrToStructure() functions in enabling quick serialization and deserialization.

5.2 I also hope the reader notes the importance of the StructLayoutAttribute as well as the various MarshalAsAttributes.

5.3 Indeed the MarshalAsAttribute is the key to the success of the quick serialization/deserialization process.

 

Advertisements

About Lim Bio Liong

I've been in software development for nearly 20 years specializing in C , COM and C#. It's truly an exicting time we live in, with so much resources at our disposal to gain and share knowledge. I hope my blog will serve a small part in this global knowledge sharing network. For many years now I've been deeply involved with C development work. However since circa 2010, my current work has required me to use more and more on C# with a particular focus on COM interop. I've also written several articles for CodeProject. However, in recent years I've concentrated my time more on helping others in the MSDN forums. Please feel free to leave a comment whenever you have any constructive criticism over any of my blog posts.

Discussion

2 thoughts on “Quick Serialization/Deserialization of a Managed Structure.

  1. This was really helpful, thank you.

    Posted by Stuart Johnson | March 16, 2014, 11:59 am
  2. With the strMember being given a set size, can it be changed to be a variable length string? Say a TCHAR* on the C++ side, and have the C# side deallocate it when it is done with the struct that contains the variable length string?

    Posted by diego | September 29, 2014, 4:07 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: