Monday, August 03, 2009

Binding C++ APIs, the COM way

A couple of days ago, during a routine "aaagh, we still don't have a nice way to do C# bindings for C++ APIs" discussion, Miguel asked me how hard would it be to leverage COM to bind C++ APIs. I've been known to mess around with COM, as when I did Mono.WebBrowser/Gecko C# bindings, but I never did get around to do little test apps to try and streamline the whole process of using COM to bind a C++ API, so I jumped at the chance and got some interesting results.

COM, despite all the bad connotations surrounding it, is actually really simple: it is just a contract stating that any COM-conforming C++ class has at least 3 methods: QueryInterface, AddRef and Release. No matter how many members the class might have, those 3 are always present at the top of the class' vtable, so Mono's COM interop layer always knows where they are and can invoke them directly. And since the vtable layout for the class is known, any other method on that class can also be invoked in this way, bypassing name-mangling and other issues.

COM-comforming C++ classes can be described in C# via interfaces that have the same layout as the C++ class, so Mono knows exactly where the methods are in the vtable when invoking. Furthermore, COM support is pretty much transparent in C# - once you've defined your interfaces, you don't even realize you're using a COM object, it's just another object that you invoke methods on. Mono does all the marshalling for you, so you don't have to pass IntPtrs around, you just use the types you defined and everything will be marshalled for you behind the scenes.

Show me the code!

Let's say you have a little C++ library you'd like to use from C#:

class File {
public:
  int Open();
  int Close();
};

The C++ COM class

The first thing you need to do is create a COM class which will serve as a proxy between C# and your nice little library.

class COMFile {
public:
  virtual int QueryInterface (void* id, void** result) {
    *result = this;
    return 0;
  }
  virtual int AddRef () { return 1; }
  virtual int Release () { return 0; }

  virtual int Open () { return file->Open(); }
  virtual int Close () { return file->Close(); }

  COMFile (File* f) : file(f) {}

private:
  File* file;
};

All methods that need to be "exported" are marked as virtual, and the layout is what you would expect: the 3 methods on top that make this a COM class, plus the 2 methods that are proxying the calls to the library's File class.

AddRef and Release are standard refcounting methods - these will be called by Mono as needed when you invoke things that end up creating objects of this type. I'm just returning fixed values here, but it's important to note that when Release makes the refcount go to 0, the object should be released.

QueryInterface allows Mono's COM interop layer to figure out if a pointer can be cast to a specified type - via behind the scenes magic (and a little code), it enables a dynamic type system. This example is very simple and doesn't use inheritance, but with a complex binding you'll certainly have inheritance, and there is where QueryInterface comes in, for instance allowing for upcasts if your COM class inherits from several different classes.

You'll notice in the C# interface below that it is marked with a Guid - this id is unique to every class, and your C++ class definition should also have the same id. When QueryInterface is invoked, the id argument is the Guid of the type you want to cast to, so you can check if your C++ class is of the correct type by comparing ids, or if it is a subclass and you need to cast the result (or you don't support it at all, in which case you'd return null).

The C# interface

[Guid ("00000000-0000-0000-0000-000000001111")]
[InterfaceType (ComInterfaceType.InterfaceIsUnknown)]
[ComImport()]
public interface COMFile {
  [PreserveSigAttribute]
  [MethodImpl (MethodImplOptions.InternalCall, MethodCodeType = MethodCodeType.Runtime)]
  int Open();

  [PreserveSigAttribute]
  [MethodImpl (MethodImplOptions.InternalCall, MethodCodeType = MethodCodeType.Runtime)]
  int Close();
}

"Wait a minute... where are the 3 methods?", I can hear you thinking. Well, on the C# side you don't need them. The interface is marked ComImport(), so Mono already knows it's a COM class and it will add them for you, no questions asked. The C# interface only needs the definitions of the methods you want to access, and nothing else.

Putting it all together

Now that you have all your definitions in place, the only thing you need is to get a reference to a COMFile object. For this you're going to need to add a P/Invoke call to a C function on your proxy code that gives you a pointer to an instance of that class. You only need to do that for top level objects, because any objects that are returned via COM calls are directly available to you.

C++ proxy library

extern "C" {
  COMFile* getptr() {
    return new COMFile (new File ());
  }
}

C# test app

[DllImport ("myglue")]
[return:MarshalAs(UnmanagedType.Interface)]
static extern COMFile getptr();

public static void Main() {
  COMFile file = getptr();
  int return = file.Open();
  ...
}

And there you go, the "file" variable is now talking to your COM class which is proxying all calls directly to your library. The glue code is very straightforward and can be easily autogenerated.

You can download a complete working sample here:
comtest-0.1.tar.gz
comtestsharp-0.1.tar.gz

Build and install both packages to the same prefix, then go to $prefix/lib/ and do " ln -s comtestsharp/* . ". Then just do"mono comtestsharp.exe" and you should see the output of the Open and Close calls.

Update: BTW, neither QI nor AddRef/Release are actually implemented properly in this little sample. The unused parameter "id" is the Guid that is getting requested, and QI should always check it against the current instance.

20 comments:

Bert Huijben said...

In COM QueryInterface returns a result code to allow checking if the requested interface is really supported. And always returning 'this' will not work for other interfaces than the default interface. (It has to be casted to the interface before returning the reference)

Martin Adoue said...

Also, while you are at it, you can fix Microsoft's mistake and make all COM classes implement IDisposable. That way deterministic finalization (key for many COM/C++ classes) can be easily achieved using, well, "using".

andreia|gaita said...

@Bert:

Yes, you're right, it should return int. When the interface is not supported, the pointer should be set to null, which gives the same behaviour as returning an error code (throws a COMException), so I can get away with not returning it, but returning it allows the exception to have an error code.

Wrt returning 'this', this is a quick and dirty example. It doesn't implement refcounting either, nor does it clean up any objects. I don't expect to be seeing this little sample copy-pasted to production code, hopefully ;)

andreia|gaita said...

@Martin:

I take it you're talking about C# classes that implement and/or store COM interfaces/objects?

Anonymous said...

Would seriously recommend SWIG
http://www.swig.org/

We've used it to automatically generate bindings for massive C++ baselines in both Windows and Linux. It takes your C++ header files as input, so as your interface changes, your bindings are automatically updated.

mdi said...

Question: are there any restrictions on what the COM exposed APIs have to return?

Do they all have to return some sort of int/HRESULT, or can they return other things like strings and structs?

andreia|gaita said...

@mdi:

You can return whatever you want - the limit is the same as for pinvokes, basically, and the same goes for in and out parameters. If what you're returning can't be automatically marshalled, you can always use HandleRef or IntPtr to do manual conversions, just like you would for pinvokes.

Some example code is at http://www.mono-project.com/COM_Interop.

Mono.WebBrowser uses COM to talk to Gecko, and you can see those interfaces at http://anonsvn.mono-project.com/viewvc/trunk/mcs/class/Mono.WebBrowser/Mono.Mozilla/interfaces/

Jae said...

+1 on SWIG, we've used on several large cross platform c++ APIs with the resulting c# wrapper also being cross platform.

MyID.config.php said...

If anyone wants to earn some Kudos for both the Drizzle & Mono communities, there is no C# client for the Drizzle database. It needs hooking up to the C library libdrizzle
https://launchpad.net/libdrizzle

Anonymous said...

is this possible in mono?? http://www.codeproject.com/KB/dotnet/DllExporter.aspx

works fine in windows tho...

Anders Rune Jensen said...

Nice article, but I agree that this needs to be autogenerated so the SWIG link is very interesting indeed :)

AJ said...

Interesting and clear, but, how many times are we going to make the same mistakes with OLE/COM design?

http://www.relisoft.com/Win32/olerant.html

andreia|gaita said...

@AJ:

Besides the fact that we're not talking about OLE anywhere here, that article makes little sense, frankly.

QueryInterface is called to verify that the current object instance can be cast to the type that you're asking it: this is the same dynamic casting concept as typeid/dynamic_cast that you get when you enable rtti on c++, the ability to do reflection on objects. This particular OLE rant is a bit meaningless, as you don't really query interfaces, you query objects for information about their interfaces - or rather, the interop layer does it for you whenever you request a cast or pass around an COM-type object.

Can you subvert a type system like this? Yes you can, completely. But then again, you can also pass void* around and pretend it's something else than what it actually is. You can pass IntPtr around on every call and use them any way you want.

There's a reason why the c++ standard did not include rtti initially: any type of dynamic introspection system will potentially allow the developer to completely mess up the type system and do the most crazy hacks - but rtti was later added on because it is actually useful, if used responsibly. You can shoot yourself in the foot all you want with this sort of thing, but don't blame the tool for what you do with it.

Also, I find it really funny that people love to hate stuff just because other people hate stuff. Should I start calling it DTS (dynamic type system) instead of COM? Or maybe DiTS - DiTS sounds good :D Maybe then people would appreciate it for what it is and not for what anyone else says it is, and develop their own opinions about the thing.

Martin Adoue said...

@andreia: exactly. If the wrapper classes implement IDisposable, life in much easier. In an old protect I actually reimplemented MS's tool (tlbimp? Can't remember now) to do exactly that.

andreia|gaita said...

@Martin:
Yes, of course, all wrapper classes need to be IDisposable, otherwise it's a huge headache. This article doesn't talk about wrapper classes, but I should get around to talking about that one of these days. :P

Iyan said...

though i code mostly in C, i enjoy your article much

Daniel Ferreira Monteiro Alves said...

Hey man! I liked your post, but I have one question, because I don't know COM very well, so:

What is the performance hit of using a COM interface instead of a p/invoke call?

andreia|gaita said...

@Daniel

This is all just shuffling pointers around and calling function pointers on vtables, so performance wise it's pretty much the same thing. The performance impact on pinvoke is the need for memory pinning and copying, which also happens here.

Daniel Ferreira Monteiro Alves said...

By start... I very, very sorry about my last comment hehe... I didn't realize that I was talking with a woman instead of a man... I just read the Gaita part of the name... so again, sorry by my mistake! :P

Now looking at your code and the possibilities for the C++ bindings... I am creating bindings for some C++ libraries used in games, like Ogre, OIS, FMod and Bullet...

The COM way is a good way to use functionalities from the unmanaged side, but it does not fix the problems with inheritance like we saw...

Your work with the cppinterop project is interesting but it's not working (at least on the trunk of the git repository)... maybe because I'm using the gccxml on mac.

I will check more about the theme and I would like to talk with you about how we could make something about it... I am finishing my Master Degree on Computer Science on Brazil (yes, I can talk portuguese \o/ ), so I would like to help you with any possible thing at my reach.

andreia|gaita said...

The cppinterop essentially supersedes this, supporting interfaces, class instantiation and subclassing. It's still a work in progress, it should be finished in the next few months.

Regarding the mac, there's no support at the moment for name mangling there, if you're targetting non-gcc libraries that won't probably work, and I'm not sure if gcc on the mac is supported, I haven't tested there yet.

Mangling needs to be implemented for every compiler, so we currently support gcc/linux and msvc/windows. We take patches!