collapse Blogs I Read
collapse Table of Contents

Jonathan Pryor's web log

Unix Signal Handling In C#

In the beginning, Unix introduced signal(2), which permits a process to respond to external "stimuli", such as a keyboard interrupt (SIGINT), floating-point error (SIGFPE), dereferencing the NULL pointer (SIGSEGV), and other asynchronous events. And lo, it was...well, acceptable, really, but there wasn't anything better, so it at least worked. (Microsoft, when faced with the same problem of allowing processes to perform some custom action upon an external stimuli, invented Structured Exception Handling.)

Then, in a wrapping binge, I exposed it for use in C# with Stdlib.signal(), so that C# code could register signal handlers to be invoked when a signal occurred.

The problem? By their very nature, signals are asynchronous, so even in a single-threaded program, you had to be very careful about what you did, as your "normal" thread was certainly in the middle of doing something. For example, calling malloc(3) was almost certainly a bad idea, because if the process was in the middle of a malloc call already, you'd have a reentrant malloc call which could corrupt the heap.

This reentrant property impacts all functions in the process, including system calls. Consequently, a list of functions that were "safe" for invocation from signal handlers was standardized, and is listed in the above signal man page; it includes functions such as read(2) and write(2), but not functions like e.g. pwrite(2).

Consequently, these limitations and a few other factors led to the general recommendation that signal handlers should be as simple as possible, such as writing to global variable which the main program occasionally polls.

What's this have to do with Stdlib.signal(), and why was it a mistake to expose it? The problem is the P/Invoke mechanism, which allows marshaling C# delegates as a function pointer that can be invoked from native code. When the function pointer is invoked, the C# delegate is eventually executed.

However, before the C# delegate can be executed, a number of of steps needs to be done first:

  1. The first thing it does is to ensure the application domain for the thread where the signal handler executes actually matches the appdomain the delegate comes from, if it isn't it may need to set it and do several things that we can't guarantee are signal context safe...
  2. If the delegate is of an instance method we also need to retrieve the object reference, which may require taking locks...

In the same email, lupus suggests an alternate signal handling API that would be safe to use from managed code. Later, I provided a possible implementation. It amounts to treating the UnixSignal instance as a glorified global variable, so that it can be polled to see if the signal has been generated:

UnixSignal signal = new UnixSignal (Signum.SIGINT);
while (!signal.IsSet) {
  /* normal processing */
}

There is also an API to permit blocking the current thread until the signal has been emitted (which also accepts a timeout):

UnixSignal signal = new UnixSignal (Signum.SIGINT);
// Wait for SIGINT to be generated within 5 seconds
if (signal.WaitOne (5000, false)) {
    // SIGINT generated
}

Groups of signals may also be waited on:

UnixSignal[] signals = new UnixSignal[]{
    new UnixSignal (Signum.SIGINT),
    new UnixSignal (Signum.SIGTERM),
};

// block until a SIGINT or SIGTERM signal is generated.
int which = UnixSignal.WaitAny (signals, -1);

Console.WriteLine ("Got a {0} signal!", signals [which].Signum);

This isn't as powerful as the current Stdlib.signal() mechanism, but it is safe to use, doesn't lead to potentially ill-defined or unwanted behavior, and is the best that we can readily provide for use by managed code.

Mono.Unix.UnixSignal is now in svn-HEAD and the mono-1-9 branch, and should be part of the next Mono release.

Posted on 08 Feb 2008 | Path: /development/mono/ | Permalink

Mono and Mixed Mode Assembly Support

An occasional question on #mono@irc.gnome.org and ##csharp@irc.freenode.net is whether Mono will support mixed-mode assemblies, as generated by Microsoft's Managed Extensions for C++ compiler (Visual Studio 2001, 2003), and C++/CLI (Visual Studio 2005, 2008).

The answer is no, and mixed mode assemblies will likely never be supported.

Why?

First, what's a mixed mode assembly? A mixed mode assembly is an assembly that contains both managed (CIL) and unmanaged (machine language) code. Consequently, they are not portable to other CPU instruction sets, just like normal C and C++ programs and libraries.

Next, why use them? The primary purpose for mixed mode assemblies is as "glue", to e.g. use a C++ library class as a base class of a managed class. This allows the managed class to extend unmanaged methods, allowing the managed code to be polymorphic with respect to existing unmanaged functions. This is extremely useful in many contexts. However, as something like this involves extending a C++ class, it requires that the compiler know all about the C++ compiler ABI (name mangling, virtual function table generation and placement, exception behavior), and thus effectively requires native code. If the base class is within a separate .dll, this will also require that the mixed mode assembly list the native .dll as a dependency, so that the native library is also loaded when the assembly is loaded.

The other thing that mixed mode assemblies support is the ability to export new C functions so that other programs can LoadLibrary() the assembly and GetProcAddress the exported C function.

Both of these capabilities require that the shared library loader for the platform support Portable Executable (PE) files, as assemblies are PE files. If the shared library loader supports PE files, then the loader can ensure that when the assembly is loaded, all listed dependent libraries are also loaded (case 1), or that native apps will be able to load the assembly as if it were a native DLL and resolve DLL entry points against it.

This requirement is met on Windows, which uses the PE file format for EXE and DLL files. This requirement is not met on Linux, which uses ELF, nor is it currently met on Mac OS X, which uses Mach-O.

So why can't mixed mode assemblies be easily supported in Mono? Because ld.so doesn't like PE.

The only workarounds for this would be to either extend assemblies so that ELF files can contain both managed and unmanaged code, or to extend the shared library loader to support the loading of PE files. Using ELF as an assembly format may be useful, but would restrict portability of such ELF-assemblies to only Mono/Linux; .NET could never make use of them, nor could Mono on Mac OS X. Similarly, extending the shared library loader to support PE could be done, but can it support loading both PE and ELF (or Mach-O) binaries into a single process? What happens if a PE file loaded into an "ELF" process requires KERNEL32.DLL? Extending the shared library loader isn't a panacea either.

This limitation makes mixed mode assemblies of dubious value. It is likely solvable, but there are for more important things for Mono to focus on.

Posted on 27 Jan 2008 | Path: /development/mono/ | Permalink

So you want to parse a command line...

If you develop command-line apps, parsing the command-line is a necessary evil (unless you write software so simple that it doesn't require any options to control its behavior). Consequently, I've written and used several parsing libraries, including Mono.GetOptions, Perl's Getopt::Long library, and some custom written libraries or helpers.

So what's wrong with them? The problem with Mono.GetOptions is that it has high code overhead: in order to parse a command line, you need a new type (which inherits from Mono.GetOptions.Options) and annotate each field or property within the type with an Option attribute, and let Mono.GetOptions map each command-line argument to a field/property within the Options subclass. See monodocer for an example; search for Opts to find the subclass.

The type-reflector parser is similarly code heavy, if only in a different way. The Mono.Fuse, lb, and omgwtf parsers are one-offs, either specific to a particular environment (e.g. integration with the FUSE native library) or not written with any eye toward reuse.

Which leaves Perl's Getopt::Long library, which I've used for a number of projects, and quite like. It's short, concise, requires no object overhead, and allows seeing at a glance all of the options supported by a program:

use Getopt::Long;
my $data    = "file.dat";
my $help    = undef;
my $verbose = 0;

GetOptions (
	"file=s"    => \$data,
	"v|verbose" => sub { ++$verbose; },
	"h|?|help"  => $help
);

The above may be somewhat cryptic at first, but it's short, concise, and lets you know at a glance that it takes three sets of arguments, one of which takes a required string parameter (the file option).

So, says I, what would it take to provide similar support in C#? With C# 3.0 collection initializers and lambda delegates, I can get something that feels rather similar to the above GetOpt::Long code:

string data = null;
bool help   = false;
int verbose = 0;

var p = new Options () {
	{ "file=",      (v) => data = v },
	{ "v|verbose",  (v) => { ++verbose } },
	{ "h|?|help",   (v) => help = v != null },
};
p.Parse (argv).ToArray ();

Options.cs has the goods, plus unit tests and additional examples (via the tests).

Options is both more and less flexible than Getopt::Long. It doesn't support providing references to variables, instead using a delegate to do all variable assignment. In this sense, Options is akin to Getopt::Long while requiring that all options use a sub callback (as the v|verbose option does above).

Options is more flexible in that it isn't restricted to just strings, integers, and floating point numbers. If there is a TypeConverter registered for your type (to perform string->object conversions), then any type can be used as an option value. To do so, merely declare that type within the callback:

int count = 0;

var p = new Options () {
	{ "c|count=", (int v) => count = v },
};

As additional crack, you can provide an (optional) description of the option so that Options can generate help text for you:

var p = new Options () {
	{ "really-long-option", "description", (v) => {} },
	{ "h|?|help", "print out this message and exit", (v) => {} },
};
p.WriteOptionDescriptions (Console.Out);

would generate the text:

      --really-long-option   description
  -h, -?, --help             print out this message and exit

Options currently supports:

All un-handled parameters are returned from the Options.Parse method, which is implemented as an iterator (hence the calls to .ToArray() in the above C# examples, to force processing).

Posted on 07 Jan 2008 | Path: /development/mono/ | Permalink

Re-Introducing monodocer

In the beginning... Mono was without documentation. Who needed it when Microsoft had freely available documentation online? (That's one of the nice things about re-implementing -- and trying to stay compatible with -- a pre-existing project: reduced documentation requirements. If you know C# under .NET, you can use C# under Mono, by and large, so just take an existing C# book and go on your way...)

That's not an ideal solution, as MSDN is/was slow. Very slow. Many seconds to load a single page slow. (And if you've ever read the .NET documentation on MSDN where it takes many page views just to get what you're after... You might forget what you're looking for before you find it.) A local documentation browser is useful.

Fortunately, the ECMA 335 standard comes to the rescue (somewhat): it includes documentation for the types and methods which were standardized under ECMA, and this documentation is freely available and re-usable.

The ECMA documentation consists of a single XML file (currently 7.2MB) containing all types and type members. This wasn't an ideal format for writing new documentation, so the file was split up into per-type files; this is what makes up the monodoc svn module (along with many documentation improvements since, particularly types and members that are not part of the ECMA standard.

However, this ECMA documentation import was last done many years ago, and the ECMA documentation has improved since then. (In particular, it now includes documentation for many types/members added in .NET 2.0.) We had no tools to import any updates.

Monodocer

Shortly after the ECMA documentation was originally split up into per-type files, Mono needed a way to generate documentation stubs for non-ECMA types within both .NET and Mono-specific assemblies. This was (apparently) updater.exe.

Eventually, Joshua Tauberer created monodocer, which both creates ECMA-style documentation stubs (in one file/type format) and can update documentation based on changes to an assembly (e.g. add a new type/member to an assembly and the documentation is updated to mention that new type/member).

By 2006, monodocer had (more-or-less) become the standard the generating and updating ECMA-style documentation, so when I needed to write Mono.Fuse documentation I used monodocer...and found it somewhat lacking in support for Generics. Thus begins my work on improving monodocer.

monodocer -importecmadoc

Fast-forward to earlier this year. Once monodocer could support generics, we could generate stubs for all .NET 2.0 types. Furthermore, ECMA had updated documentation for many core .NET 2.0 types, so...what would it take to get ECMA documentation re-imported?

This turned out to be fairly easy, with supported added in mid-May to import ECMA documentation via a -importecmadoc:FILENAME parameter. The problem was that this initial version was slow; quoting the ChangeLog, "WARNING: import is currently SLOW." How slow? ~4 Minutes to import documentation for System.Array.

This might not be too bad, except that there are 331 types in the ECMA documentation file, documenting 3797 members (fields, properties, events, methods, constructors, etc.). 4 minutes per type is phenominally slow.

Optimizing monodocer -importecmadoc

Why was it so slow? -importecmadoc support was originally modeled after -importslashdoc support, which is as follows: lookup every type and member in System.Reflection order, create an XPath expression for this member, and execute an XPath query against the documentation we're importing. If we get a match, import the found node.

The slowdown was twofold: (1) we loaded the entire ECMA documentation into a XmlDocument instance (XmlDocument is a DOM interface, and thus copies the entire file into memory), and (2) we were then accessing the XmlDocument randomly.

The first optimization is purely algorithmic: don't import documentation in System.Reflection order, import it in ECMA documentation order. This way, we read the ECMA documentation in a single pass, instead of randomly.

As is usually the case, algorithmic optimizations are the best kind: it cut down the single-type import from ~4 minutes to less than 20 seconds.

I felt that this was still too slow, as 20s * 331 types is nearly 2 hours for an import. (This is actually faulty reasoning, as much of that 20s time was to load the XmlDocument in the first place, which is paid for only once, not for each type.) So I set out to improve things further.

First was to use a XPathDocument to read the ECMA documentation. Since I wasn't editing the document, I didn't really need the DOM interface that XmlDocument provides, and some cursory tests showed that XPathDocument was much faster than XmlDocument for parsing the ECMA documentation (about twice as fast). This improved things, cutting single-type documentation import from ~15-20s to ~10-12s. Not great, but better.

Convinced that this still wasn't fast enough, I went to the only faster XML parser within .NET: XmlTextReader, which is a pull-parser lacking any XPath support. This got a single-file import down to ~7-8s.

I feared that this would still need ~45 minutes to import, but I was running out of ideas so I ran a full documentation import for mscorlib.dll to see what the actual runtime was. Result: ~2.5 minutes to import ECMA documentation for all types within mscorlib.dll. (Obviously the ~45 minute estimate was a little off. ;-)

Conclusion

Does this mean that we'll have full ECMA documentation imported for the next Mono release? Probably not. There are still a few issues with the documentation import where it skips members that ideally would be imported (for instance, documentation for System.Security.Permissions.FileIOPermissionAttribute.All isn't imported because Mono provides a get accessor while ECMA doesn't). The documentation also needs to be reviewed after import to ensure that the import was successful (a number of bugs have been found and fixed while working on these optimizations).

Hopefully it won't take me too long to get things imported...

Posted on 15 Jul 2007 | Path: /development/mono/ | Permalink

POSIX Says The Darndest Things

make check was reported to be failing earlier this week, and Mono.Posix was one of the problem areas:

1) MonoTests.Mono.Unix.UnixGroupTest.ListAllGroups_ToString : #TLAU_TS:
Exception listing local groups: System.IO.FileNotFoundException: Nie ma
takiego pliku ani katalogu ---> Mono.Unix.UnixIOException: Nie ma
takiego pliku ani katalogu [ENOENT].
  at Mono.Unix.UnixMarshal.ThrowExceptionForLastError () [0x00000] in
/home/koxta/mono-1.2.4/mcs/class/Mono.Posix/Mono.Unix/UnixMarshal.cs:456
  at Mono.Unix.UnixGroupInfo.GetLocalGroups () [0x0001c] in
/home/koxta/mono-1.2.4/mcs/class/Mono.Posix/Mono.Unix/UnixGroupInfo.cs:127
  at MonoTests.Mono.Unix.UnixGroupTest.ListAllGroups_ToString ()
[0x0000a] in
/home/koxta/mono-1.2.4/mcs/class/Mono.Posix/Test/Mono.Unix/UnixGroupTest.cs:32
  at MonoTests.Mono.Unix.UnixGroupTest.ListAllGroups_ToString ()
[0x0003c] in
/home/koxta/mono-1.2.4/mcs/class/Mono.Posix/Test/Mono.Unix/UnixGroupTest.cs:37
  at <0x00000> <unknown method>
  at (wrapper managed-to-native)
System.Reflection.MonoMethod:InternalInvoke (object,object[])
  at System.Reflection.MonoMethod.Invoke (System.Object obj,
BindingFlags invokeAttr, System.Reflection.Binder binder,
System.Object[] parameters, System.Globalization.CultureInfo culture)
[0x00040] in
/home/koxta/mono-1.2.4/mcs/class/corlib/System.Reflection/MonoMethod.cs:144

Further investigation narrowed things down to Mono_Posix_Syscall_setgrent() in support/grp.c:

int
Mono_Posix_Syscall_setgrent (void)
{
	errno = 0;
	setgrent ();
	return errno == 0 ? 0 : -1;
}

I did this because setgrent(3) can fail, even though it has a void return type; quoting the man page:

Upon error, errno may be set. If one wants to check errno after the call, it should be set to zero before the call.

Seems reasonably straightforward, no? Clear errno, do the function call, and if errno is set, an error occurred.

Except that this isn't true. On Gentoo and Debian, calling setgrent(3) may set errno to ENOENT (no such file or directory), because setgrent(3) tries to open the file /etc/default/nss. Consequently, Mono.Unix.UnixGroupInfo.GetLocalGroups reported an error (as can be seen in the above stack trace).

Further discussion with some Debian maintainers brought forth the following detail: It's only an error if it's a documented error. So even though setgrent(3) set errno, it wasn't an error because ENOENT isn't one of the documented error values for setgrent(3).

"WTF!," says I.

So I dutifully go off and fix it, so that only documented errors result in an error:

int
Mono_Posix_Syscall_setgrent (void)
{
	errno = 0;
	do {
		setgrent ();
	} while (errno == EINTR);
	mph_return_if_val_in_list5(errno, EIO, EMFILE, ENFILE, ENOMEM, ERANGE);
	return 0;
}

...and then I go through the rest of the MonoPosixHelper code looking for other such erroneous use of errno and error reporting. There are several POSIX functions with void return types that are documented as generating no errors, and others are like setgrent(3) where they may generate an error.

It's unfortunate that POSIX has void functions that can trigger an error. It makes binding POSIX more complicated than it should be.

Posted on 29 Jun 2007 | Path: /development/mono/ | Permalink

Mono.Fuse 0.4.1

Now with MacFUSE support!

Mono.Fuse is a C# binding for FUSE. This is a minor update over the previous Mono.Fuse 0.4.0 release.

The highlight for this release is cursory MacFUSE support, which allows Mono.Fuse to work on Mac OS X. Unfortunately, it's not complete support, and I would appreciate any assistance in fixing the known issues (details below).

Mac OS X HOWTO

To use Mono.Fuse on Mac OS X, do the following:

  1. Download and install Mono 1.2.3.1 or later. Other releases can be found at the Mono Project Downloads Page.
  2. Download and install MacFUSE 0.2.4 or later. Other releases can be found at the macfuse download page.
  3. Download, extract, and configure Mono.Fuse 0.4.1:
    1. curl www.jprl.com/Projects/mono-fuse/mono-fuse-0.4.1.tar.gz > mono-fuse-0.4.1.tar.gz
    2. tar xzf mono-fuse-0.4.1.tar.gz
    3. cd mono-fuse-0.4.1.tar.gz
    4. PKG_CONFIG_PATH=/usr/local/lib/pkgconfig CFLAGS="-D__FreeBSD__=10 -O -g" ./configure --prefix=`pwd`/install-root
      • Note: PKG_CONFIG_PATH is needed so that fuse.pc will be found by pkg-config.
      • Note: CFLAGS is used as per the macfuse FAQ.
      • Note: You can choose any other --prefix you want.
    5. make
  4. Once Mono.Fuse has been built, you can run the sample programs as described in the README:
    1. cd example/HelloFS/
    2. mkdir t
    3. ./hellofs t &
    4. ls t
    5. cat t/hello

Known Issues

HelloFS works, but RedirectFS and RedirectFS-FH do not. Trying to execute them results in a SIGILL within Mono.Unix.Native.Syscall.pread when trying to read a file:

  1. cd example/RedirectFS
  2. mkdir t
  3. MONO_TRACE_LISTENER=Console.Out:+++ ./redirectfs t ~/ &
    • Note: MONO_TRACE_LISTENER set so that exception messages from Mono.Fuse.FileSystem will be printed to stdout. See the mono(1) man page for more information about MONO_TRACE_LISTENER.
  4. ls t        # works
  5. cat t/some-file-that-exists
    • Generates a SIGILL.

I would appreciate any assistance in fixing this issue.

Download

Mono.Fuse 0.4.1 is available from http://www.jprl.com/Projects/mono-fuse/mono-fuse-0.4.1.tar.gz. It can built with Mono 1.1.13 and later. Apple Mac OS X support has only been tested with Mono 1.2.3.1.

GIT Repository

The GIT repository for Mono.Fuse is at http://www.jprl.com/Projects/mono-fuse.git.

Posted on 13 Apr 2007 | Path: /development/mono/ | Permalink

When Comparisons Fail

One of the unsung helper programs for Mono.Fuse and Mono.Unix is create-native-map ( man page), which takes an assembly, looks for DllImport-attributed methods, and generates C structure and function prototypes for those methods and related types. This allows e.g. the internal Mono.Posix.dll methods to be kept in sync with their implementation methods in MonoPosixHelper, checked by the compiler to ensure type consistency.

One of the "features" of create-native-map is support for integer overflow checking. For example, if you have a C# type:

[Map ("struct foo")]
struct Foo {
  public int member;
}

then create-native-map will generate the (excerpted) C code (it generates much more):

struct Foo {
  int member;
};

int ToFoo (struct foo *from, struct Foo *to)
{
  _cnm_return_val_if_overflow (int, from->member, -1);
  to->member = from->member;
	return 0;
}

This could be handy, as if the actual type of struct foo::member differed from int, we could tell at runtime if the value of from->member wouldn't fit within to->member. That was the hope, anyway. (Yes, this flexibility is required, as many Unix structures only standardize member name and type, but not necessarily the actual type. For example, struct stat::st_nlink is of type nlink_t, which will vary between platforms, but Mono.Unix.Native.Stat.st_nlink can't change between platforms, it needs to expose an ABI-agnostics interface for portability. Consequently, overflow checking is desirable when doing Statstruct stat conversions, and vice versa, to ensure that nothing is lost.)

The reality is that _cnm_return_val_if_overflow() was horribly buggy and broke if you looked at it wrong (i.e. it worked for me and would fail on many of the build machines running !Linux). Consequently _cnm_return_val_if_overflow() was converted into a no-op unless DEBUG is defined before/during the Mono 1.2.0 release.

Why discuss this now? Because Mono.Fuse 0.4.0 shipped with a broken version of create-native-map, which is the primary reason that it doesn't work with MacFUSE.

But because I'm a glutton-for-punishment/insane, I thought I'd take a look into making overflow checking work again (though it still won't be enabled unless DEBUG is defined). I wrote some tests, got them working on Linux, and tried to run them on Intel Mac OS X. The result: all but one worked. The reason it failed is inexplicable: a failing comparison. G_MININT64 can't be directly compared against 0:

$ cat ovf.c
# include <glib.h>
# include <limits.h>
# include <stdio.h>

int main ()
{
  long long v = G_MININT64;
  printf (" LLONG_MIN < 0? %i\n", (int) (LLONG_MIN < 0));
  printf ("G_MININT64 < 0? %i\n", (int) (G_MININT64 < 0));
  printf ("         v < 0? %i\n", (int) (v < 0));
}

$ gcc -o ovf ovf.c `pkg-config --cflags --libs glib-2.0`
$ ./ovf
 LLONG_MIN < 0? 1
G_MININT64 < 0? 0
         v < 0? 1

Now that's a w-t-f: G_MININT64 < 0 is FALSE. Simply bizarre...

Meanwhile, I should have a Mono.Fuse 0.4.1 release out "soon" to fix these problems, permitting Mono.Fuse to work properly with MacFUSE.

Posted on 12 Apr 2007 | Path: /development/mono/ | Permalink

Novell, Microsoft, & Patents

The news is out: hell has frozen over. Novell and Microsoft have announced a "patent cooperation agreement," whereby Microsoft won't sue Novell customers for patent infringement and Novell won't sue Microsoft customers for patent infringement.

I first heard about this on mono-list, and immediately replied with the obvious (to me) response.

Note: I am not a lawyer [0], so consequently everything I say is bunk, but I have been paying some attention to various lawsuits over the years.

That out of the way, take a step back and ignore Microsoft and Novell for the moment. Assume that you're a patent holder, and you decide that your patent has been infringed. Who do you sue? There are three possible defendants:

  1. Sue the developer. (Example: Stac Electronics vs. Microsoft.)
  2. Sue the distributor. This is frequently identical to (1) as the developer is the distributor, but the rise of Free and Open Source software introduces this distinction.
  3. Sue the customer of (1) and/or (2). The example I remembered hearing several years ago was Timeline vs. Microsoft [1].

The summary is this: software patents are evil, allowing virtually anyone to sue virtually everyone else. There are no assurances of safety anywhere. Software from large companies (Internet Explorer) can be sued as easily as software from a no-name company or the open-source community (see Eolas vs. Microsoft).

With that background out of the way, what does this Microsoft/Novell deal mean? It means exactly what they say: Novell won't sue Microsoft customers, and Microsoft won't sue Novell customers. Anyone else can still sue Microsoft, Novell, and their customers, so Novell and Microsoft customers really aren't any safer than they were before. Novell customers are a little safer -- the monster in the closet of a Microsoft lawsuit is no longer an issue -- but no one is completely safe. It just provides peace of mind, but it isn't -- and cannot -- be a complete "solution" to the threat of patent lawsuits. (The only real solution is the complete abolition of all software patents, which is highly unlikely.)

What does this mean for hobbyists who contribute to Mono, Samba, Wine, Linux, and other projects (like me)? It means I'm protected as part of this agreement, as my code is distributed as part of openSUSE. This also means that anyone other than Microsoft can sue me if I happen to violate a patent.

What about hobbyists whose code isn't part of openSUSE? Nothing has changed -- they're as subject to a lawsuit as they were a week ago.

What about other companies such as Red Hat? Nothing has changed for them, either. Red Hat is still safe, as it is a member of the Open Invention Network, which was created to deal with the potential for patent lawsuits from any party. OIN is a more complete solution for most parties involved than the Microsoft and Novell agreement, as it involves more parties.

The problem with OIN is that it only covers the members of OIN. Red Hat is protected, but any distributors of Red Hat code are not (such as CentOS), and neither are the customers of Red Hat (unless the customer has a patent protection contract with their supplier). Consequently, OIN serves to protect the original developers (1), but not any "downstream" distributors (2) or their customers (3).

But what about the GPL, section 7? Doesn't the Microsoft/Novell agreement violate it?

7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program.

The simple solution is that this doesn't apply, as this agreement doesn't touch this clause at all. It's not a consequence of a court judgment, there is no allegation of patent infringement, and I haven't heard of any conditions that Microsoft requires Novell to follow in order for the code to be freely distributed. The Microsoft/Novell agreement primarily covers their customers, not their code, so there isn't a problem.

Notes:
[0] But I did stay at a Holiday Inn last night!
[1] Computerworld Article.

Posted on 04 Nov 2006 | Path: /development/mono/ | Permalink

Mono.Fuse 0.4.0

Mono.Fuse is a C# binding for FUSE. This is the fourth major release

This release contains a few major changes to the public API for consistency and clarification purposes, the biggest of which is renaming Mono.Fuse.FileSystemEntry to Mono.Fuse.DirectoryEntry (which of course required changing Mono.Fuse.FileSystem.OnReadDirectory(), again!). Some of the Mono.Fuse.FileSystem properties were also renamed for consistency.

I'm still making no promises for API stability. The FileSystem virtual methods should be fairly stable, but the properties may continue to be flexible as I document them more fully (as I'm not entirely sure what the ramifications are for some of them, such as FileSystem.ReaddirSetsInode vs. FileSystem.SetsInode, and answering these questions will require reading the FUSE source).

API Changes from the previous release:

See the commit diff for specifics.

Download

Mono.Fuse 0.4.0 is available from http://www.jprl.com/Projects/mono-fuse/mono-fuse-0.4.0.tar.gz. It can built with Mono 1.1.13 and later.

GIT Repository

A GIT repository for Mono.Fuse is at http://www.jprl.com/Projects/mono-fuse.git.

Posted on 20 Sep 2006 | Path: /development/mono/ | Permalink

Naming, Mono.Fuse Documentation

I find naming to be difficult. What should a type be named, what should a member be named? What's consistent? What names are easily understandable in any context? What names sow confusion instead of clarity?

This is why writing documentation, as annoying as it can be (lots of repetition), is also useful: it forces a different viewpoint on the subject.

For example, Mono's monodocer program uses System.Reflection to generate an initial documentation stub for use within monodoc. As such, it shows you the public types which are actually within an assembly, not just what you thought was in the assembly, like the compiler-generated default constructors which are so easy to forget about.

I've documented every public type and member within Mono.Fuse: Mono.Fuse Documentation.

And if you haven't guessed by now, the types have changed, because writing documentation forces a different viewpoint on the subject, and shows out all of the glaring inconsistencies within the API. So much for my hope that the API would be reasonably stable after the 0.3.0 release. <Sigh>

Consequently, the docs are only ~90% useful for the most recent 0.3.0 release, as they document the forthcoming 0.4.0 release. I hope to get 0.4.0 out reasonably soon, though I still have to write release notes.

The GIT repository has been updated so that HEAD contains the 0.4.0 sources, so if you're really interested in using the current API, you can git-clone it for now.

Mono.Fuse Home Page

Posted on 20 Sep 2006 | Path: /development/mono/ | Permalink

Miguel's ReflectionFS

After the Mono.Fuse 0.2.1 release, Miguel de Icaza wrote a small Mono.Fuse program that exposed System.Reflection information as a filesystem.

With the Mono.Fuse 0.3.0 release, this sample no longer works, as the Mono.Fuse API changed. Thus, here is an updated version of the sample:


$ cp `pkg-config --variable=Libraries mono-fuse` .
$ gmcs ReflectionFS.cs -r:Mono.Fuse.dll -r:Mono.Posix.dll
$ mkdir t
$ mono ReflectionFS.exe t &
$ ls t/mscorlib/System.Diagnostics.ConditionalAttribute
ConditionString      GetType             ToString
Equals               get_TypeId          TypeId
get_ConditionString  IsDefaultAttribute
GetHashCode          Match
$ fusermount -u t

Note that the above requires that PKG_CONFIG_PATH contain a directory with the mono-fuse.pc file (created during Mono.Fuse installation), and that libMonoFuseHelper.so should be in LD_LIBRARY_PATH or a directory listed in /etc/ld.so.conf.

Mono.Fuse also contains other sample programs. In particular, RedirectFS-FH.cs is a straightforward port of FUSE's fusexmp_fh.c sample program, and shows a way to "redirect" a new mountpoint to display the contents of another existing directory. RedirectFS-FH.cs is actually an improvement, as fusexmp_fh.c just shows the contents of the / directory at any new mount point, while RedirectFS-FH can redirect to any other directory.

Posted on 11 Sep 2006 | Path: /development/mono/ | Permalink

Mono.Fuse 0.3.0

Mono.Fuse is a C# binding for FUSE. This is the third major release.

This release completely changes the public API for consistency and performance. Hopefully this will be the last API change, though I would appreciate any feedback on the current Mono.Fuse.FileSystem.OnReadDirectory API.

API Changes from the previous release:

Download

Mono.Fuse 0.3.0 is available from http://www.jprl.com/Projects/mono-fuse/mono-fuse-0.3.0.tar.gz. It can built with Mono 1.1.13 and later.

GIT Repository

A GIT repository for Mono.Fuse is at http://www.jprl.com/Projects/mono-fuse.git.

Posted on 11 Sep 2006 | Path: /development/mono/ | Permalink

Mono.Fuse, Take 2.1!

At Miguel's request, I've created a version of Mono.Fuse that doesn't depend upon Mono runtime changes. This should make it possible for more people to try it out.

Don't forget to read the README, as it contains build instructions and a description of how to run the included example program.

I should caution that the API isn't stable (I suspect Mono.Fuse.FileSystem.OnRead should probably become Mono.Fuse.FileSystem.OnReadFile, for one), and I look forward to any and all API suggestions that you can provide.

Two final notes: Mono.Fuse depends on FUSE, and FUSE is a Linux kernel module, so you'll need to run:

/sbin/modprobe fuse

as the root user before you can use any FUSE programs. You'll also need to install the FUSE user-space programs, as you must use the fusermount program to unmount a directory that has been mounted by FUSE, e.g.:

fusermount -u mount-point
Posted on 01 Sep 2006 | Path: /development/mono/ | Permalink

Mono.Fuse, Take 2!

See the original announcement for more on what Mono.Fuse is (in short: a C# binding for FUSE).

This is an update, releasing Mono.Fuse 0.2.0, and (more importantly) an updated set of patches to mcs, mono, and now mono-tools. The mcs and mono patches are required to build & run Mono.Fuse, while the mono-tools patch is optional and only necessary if you want to view the create-native-map.exe program.

See here for all patches and an overview.

The major change between this set of patches and the original set is one of approach: the original set tried to make the native MonoPosixHelper API public, which was deemed as unacceptable (as there's too much cruft in there that we don't want to maintain).

The new approach only adds public APIs to the Mono.Unix.Native.NativeConvert type, permitting managed code to copy any existing native instance of supported structures. For example:

  Mono.Unix.Native.NativeConvert.Copy (IntPtr source, 
      out Mono.Unix.Native.Stat destination);

copies a pointer to an existing native struct stat and copies it into the managed Mono.Unix.Native.Stat instance. There are equivalent methods to do the managed → native conversion as well.

Since this approach requires making far fewer public API changes to Mono.Posix and MonoPosixHelper (i.e. no public API changes to MonoPosixHelper, as it's an internal/private library), I hope that this will be more acceptable.

Here's to a quick review!

Updated to add a link to the overview page.

Posted on 01 Sep 2006 | Path: /development/mono/ | Permalink

Announcing Mono.Fuse

Mono.Fuse is a binding for the FUSE library, permitting user-space file systems to be written in C#.

Why?

I read Robert Love's announcement of beaglefs, a FUSE program that exposes Beagle searches as a filesystem. My first thought: Why wasn't that done in C# (considering that the rest of Beagle is C#)?

What about SULF?

Stackable User-Level Filesystem, or SULF, is a pre-existing FUSE binding in C#, started by Valient Gough in 2004.

Mono.Fuse has no relation to SULF, for three reasons:

  1. It goes to great efforts to avoid a Mono.Posix.dll dependency, duplicating Mono.Unix.Native.Stat (Fuse.Stat), Mono.Unix.Native.Statvfs (Fuse.StatFS), and many methods from Mono.Unix.Native.Syscall (Fuse.Wrapper).
  2. I don't like the SULF API. (Not that I spent a great deal of time looking at it, but what I did see I didn't like.)
  3. SULF wraps the FUSE kernel-level interface, while Mono.Fuse wraps the higher level libfuse C interface.

I find (1) the most appalling, if only because I'm the Mono.Posix maintainer and I'd like to see my work actually used. :-)

Once I started writing Mono.Fuse, I discovered a good reason to avoid Mono.Posix: it's currently impossible to use the native MonoPosixHelper shared library from outside of Mono. I figured this would be a good opportunity to rectify that, making it easier for additional libraries to build upon the Mono.Posix infrastructure.

Implementation

Mono.Fuse requires patches to the mcs and mono modules, changes which need to be proposed and discussed.

mono

The biggest problem with the mono module is that no headers are installed, making it difficult to make use of libMonoPosixHelper.so.

Changes:

map.h is the current map.h file generated by make-map.exe, with some major additions (detailed in the mcs section).

helper.h is the main include file, which includes map.h and declares all types/functions which cannot be generated by make-map.exe.

mono-config.h is necessary because it needs to contain platform-specific macros. In particular, Linux needs:

int
Mono_Posix_ToStatvfs (struct statvfs *to, struct Mono_Posix_Statvfs *to);

while OS X and *BSD need:

int
Mono_Posix_ToStatvfs (struct statfs *to, struct Mono_Posix_Statvfs *to);

Note struct statvfs vs. struct statfs. The mono/posix/helper.h header needs to "paper over" the difference, and thus needs to know which type the platform prefers. helper.h thus looks like:

#ifdef MONO_HAVE_STATVFS
  struct statvfs;
  int Mono_Posix_ToStatvfs (struct statvfs *from, 
      struct Mono_Posix_Statvfs *to);
#endif
#ifdef MONO_HAVE_STATFS
  struct statfs;
  int Mono_Posix_ToStatvfs (struct statfs *from, 
      struct Mono_Posix_Statvfs *to);
#endif

One of MONO_HAVE_STATVFS or MONO_HAVE_STATFS would be defined in mono-config.h.

mcs

There are two major changes:

The MapAttribute attribute is public so that make-map.exe can use a publically exposed API for code generation purposes which can be used by other libraries (Mono.Fuse makes use of these changes).

make-map.exe can also generate structure declarations and delegate declarations in addition to P/Invoke function declarations, allowing for a better, automated interface between C and C#.

Previously, [Map] could only be used on enumerations.

Now, [Map] can be used on classes, structures, and delegates, to create a C declaration of the C# type, suitable for P/Invoke purposes, e.g. the C# code:

[Map] struct Stat {public FilePermissions mode;}

would generate the C declaration

struct Namespace_Stat {unsigned int mode;};

The MapAttribute.NativeType property is used to specify that type conversion functions should be generated, thus:

[Map ("struct stat")] struct Stat {public FilePermissions mode;}

would generate

struct Namespace_Stat {unsigned int mode;};
int Namespace_ToStat (struct stat *from, struct Namespace_Stat *to);
int Namespace_FromStat (struct Namespace_Stat *from, struct stat *to);

along with the actual implementations of Namespace_ToStat() and Namespace_FromStat().

The MapAttribute.NativeSymbolPrefix property is used to specify the C "namespace" to use:

[Map (NativeSymbolPrefix="Foo")] struct Stat {FilePermissiond mode;}

generates

struct Foo_Stat {unsigned int mode;};

This prefix is also used for the conversion functions.

(You may be wondering why NativeSymbolPrefix exists at all. This is for reasonable symbol versioning -- make-map.exe currently has a "hack" in place to rename Mono.Unix(.Native) to Mono_Posix, a hack I'd like to remove, and NativeSymbolPrefix allows the Mono.Unix.Native types to have a Mono_Posix C namespace in a reasonably general manner.)

The previously internal Mono.Unix.HeaderAttribute has been removed. The HeaderAttribute.Includes and HeaderAttribute.Defines properties have been replaced with make-map.exe command-line arguments. In particular, HeaderAttribute.Includes has been replaced with --autoconf-header, --impl-header, --impl-macro, --public-header, and --public-macro (the first three modify the generated .c file, while the latter two modify the generated .h file).

Finally, make-map.exe has been renamed and moved from mcs/class/Mono.Posix/Mono.Unix.Native/make-map.exe to mcs/tools/create-native-map/create-native-map.exe.

HOWTO

  1. Go to http://www.jprl.com/Projects/mono-fuse for the patches and source download.
  2. Apply mcs.patch to a mcs checkout, rebuild, and install.
  3. Apply mono.patch to a mono checkout, rebuild, and install.
  4. Build mono-fuse-0.1.0.tar.gz in "the standard manner" (./configure ; make ; make install).

Questions

Posted on 29 Aug 2006 | Path: /development/mono/ | Permalink

Performance Comparison: IList<T> Between Arrays and List<T>

Rico Mariani recently asked a performance question: given the following code, which is faster, Sum(array), which converts a ushort[] to an IList<T>, or Sum(list), which uses the implicit conversion between List<T> and IList<T>.

using System;
using System.Collections.Generic;

class Test {
  static int Sum (IList<ushort> indeces)
  {
    int result = 0;
    for (int i = 0; i < indeces.Count; ++i)
      result += indeces [i];
    return result;
  }

  const int Size = 500000;

  public static void Main ()
  {
    ushort[] array= new ushort [Size];
    DateTime start = DateTime.UtcNow;
    Sum (array);
    DateTime end = DateTime.UtcNow;
    Console.WriteLine ("    ushort[]: {0}", end-start);

    List<ushort> list = new List<ushort> (Size);
    for (int i = 0; i < Size; ++i) list.Add (0);
    start = DateTime.UtcNow;
    Sum (list);
    end = DateTime.UtcNow;
    Console.WriteLine ("List<ushort>: {0}", end-start);
  }
}

Note that the question isn't about comparing the performance for constructing a ushort[] vs. a List<T>, but rather the use of an IList<ushort> backed by a ushort[] vs a List<ushort>.

The answer for Mono is that, oddly enough, List<ushort> is faster than ushort[]:

    ushort[]: 00:00:00.0690370
List<ushort>: 00:00:00.0368170

The question is, why?

The answer is, "magic." System.Array is a class with magical properties that can't be duplicated by custom classes. For example, all arrays, such as ushort[], inherit from System.Array, but only have an explicitly implemented IList indexer. What looks like an indexer usage results in completely different IL code; the compiler is involved, and must generate different code for an array access.

For example, an array access generates the IL code:

// int i = array [0];
ldarg.0    // load array
ldc.i4.0   // load index 0
ldelem.i4  // load element array [0]
stloc.0    // store into i

While an IList indexer access generates this IL code:

// object i = list [0];
ldarg.0    // load list
ldc.i4.0   // load index 0
callvirt instance object class [mscorlib]System.Collections.IList::get_Item(int32)
           // call IList.this [int]
stloc.0    // store into i

This difference in IL allows the JIT to optimize array access, since different IL is being generated only for arrays.

In .NET 2.0, System.Array got more magic: all array types implicitly implement IList<T> for the underlying array type, which is why the code above works (ushort[] implicitly implements IList<ushort>). However, this is provided by the runtime and is "magical," in that System.Reflection won't see that System.Array implements any generics interfaces. Magic.

On Mono, this is implemented via an indirection: arrays may derive from System.Array.InternalArray<T> instead of System.Array. InternalArray<T> implements IList<T>, permitting the implicit conversion from ushort[] to IList<ushort>.

However, this indirection has a performance impact: System.Array.InternalArray<T>.get_Item invokes System.Array.InternalArray<T>.GetGenericValueImpl, which is an internal call. This is the source of the overhead, as can be seen with mono --profile=default:stat program.exe:

prof counts: total/unmanaged: 172/97
     27 15.79 % mono
     12  7.02 % mono(mono_metadata_decode_row
     11  6.43 % Enumerator:MoveNext ()
     10  5.85 % Test:Sum (System.Collections.Generic.IList`1)
     10  5.85 % (wrapper managed-to-native) InternalArray`1:GetGenericValueImpl (int,uint16&)
     10  5.85 % InternalEnumerator:MoveNext ()

To conclude, List<ushort> is faster than ushort[], when accessed via an IList<ushort> reference, because the ushort[] can't be accessed as a normal array, procluding the usual runtime optimizations:

It should be also noted that because of this "magic," all arrays under .NET 2.0 have more overhead than the same arrays under .NET 1.1, because of the need to support the "magic" generics interfaces. This could be optimized to save memory, such that if you never access the array via a generic interface no memory is used, but Mono has not performed such an optimization yet.

Update: After discussing this on #mono, Paolo Molaro implemented an optimization which makes the array usage much faster. Now it's only slightly slower than IList<T>:

    ushort[]: 00:00:00.0133390
List<ushort>: 00:00:00.0132830

Update 2: Rico Mariani has posted his .NET performance analysis. The key take home point? "Arrays are magic."

Posted on 10 Mar 2006 | Path: /development/mono/ | Permalink

System.Diagnostics Tracing Support

As I wrote Mono's original Trace support infrastructure, I should probably get around to implementing the much improved 2.0 version. Which means I first need to understand it. Fortunately Mike Rousos is documenting how it works:

Posted on 21 Sep 2005 | Path: /development/mono/ | Permalink

Mono.Unix Reorganization

Brad Abrams, author of the Framework Design Guidelines, recently posted his precon slides for the recent PDC.

I quickly read through it to see how it well Mono.Unix follows it. I also recently ran FxCop on Mono.Posix.dll, with many interesting results.

One major point that Abrams' slides pointed out is on page 53:

This is completely different from how Mono.Unix currently operates, as it places both low-level classes such as Syscall and high-level classes such as UnixStream into the same namespace. The only difference between thw low-level and high-level is the Unix prefix present on the high-level classes. This is a problem.

It's a problem because when looking at the class view or the documentation you get lost looking at the dozens of low-level types such as AccessMode, ConfStr, and Syscall, as the high-level wrapper classes -- having a Unix prefix, will be after most of the types developers (hopefully) won't be interested in.

My solution is to separate the low-level classes into a Mono.Unix.Native namespace. The Mono.Unix namespace will be used for high-level types following CLS conventions (such as PascalCased types and methods) such as UnixFileSystemInfo, and for .NET integration classes such as UnixStream.

This change went into mono-HEAD today. All of the existing low-level Mono.Unix types have been marked [Obsolete], with messages directing users to use the appropriate Mono.Unix.Native types. Alas, some of these low-level types are used in properties or as the return types of methods in the high-level classes. These have been marked [Obsolete] for now, with a message stating that the property type or method return type will change in the next release. "Next release" in this case will be 1.1.11 or 1.2 (as I'm assuming the release of 1.1.10, which is when developers will actually see these messages if they don't follow mono-HEAD).

I'm also interested in better CLS compliance in the high-level classes. At present many of them are [CLSCompliant(false)] because they use non-CLS-compatible types such as uint or ulong. Should these be changed to CLS-compliant types? Any such changes should be done now (i.e. before 1.1.10), to allow any migration time.

Posted on 20 Sep 2005 | Path: /development/mono/ | Permalink

Major Change to Nullable Types

Poor Martin Baulig -- Microsoft has changed the design of System.Nullable so that it's far more integrated into the runtime. This change impacts things as fundamental as the box and unbox instructions...

Posted on 12 Aug 2005 | Path: /development/mono/ | Permalink

Frogger under Mono

DotGNU Portable.NET provides a .NET Curses wrapper for creating nifty console-based programs using the ncurses library. It is possible to run this under Mono.

Alas, the provided configure script doesn't work very well with Mono, so we need to do things manually. The following will allow you to build Portable.NET's Curses.dll ncurses wrapper and run the bundled Frogger game.

  1. Download and extract pnetcurses:
    $ wget http://www.southern-storm.com.au/download/pnetcurses-0.0.2.tar.gz
    $ tar xzf pnetcurses-0.0.2.tar.gz
    $ cd pnetcurses-0.0.2
    			
  2. Build the libcsharpncurses.so helper library. You can ignore the warnings produced by GCC.
    $ cd help
    $ gcc -shared -o libcsharpncurses.so *.c -lncurses
    input.c: In function `CursesHelpGetNextChar':
    input.c:69: warning: integer constant is too large for "long" type
    $ cd ..
    
  3. Build the Curses.dll wrapper library:
    $ cd src
    $ mcs -t:library *.cs -out:Curses.dll
    Compilation succeeded
    $ cd ..
    
  4. Create a Curses.dll.config file so that Mono can load the appropriate native libraries. This file should go into the same directory as Curses.dll.
    $ cd src
    $ cat > Curses.dll.config <<EOF
    <configuration>
      <dllmap dll="cygncurses5.dll" 
        target="libncurses.so"/>
      <dllmap dll="libcsharpcurses-0-0-1.dll"
        target="libcsharpncurses.so"/>
    </configuration>
    EOF
    $ cd ..
    
  5. Build the Frogger.exe demo program:
    $ cd frogger
    $ cp ../src/Curses.dll* .
    $ mcs -t:exe -out:Frogger.exe -r:Curses.dll *.cs
    Compilation succeeded
    $ cd ..
    			
  6. Execute Frogger.exe with Mono:
    $ cd frogger
    $ LD_LIBRARY_PATH=`pwd`/../help mono Frogger.exe
    			
Posted on 08 Apr 2005 | Path: /development/mono/ | Permalink

Mono.Unix Documentation Stubs

I've just added the Mono.Unix documentation stubs. Next step is to start documenting the members, mostly by copy/pasting the current Mono.Posix documentation.

Lessons learned:

  1. Documentation can be out of date, especially the documentation to update existing documentation. Oops. We're supposed to use the monodocer program.
  2. The correct command to generate/update documentation is: monodocer -assembly:assembly-name -path:directory-name.
  3. The monodocer program didn't like me (it generated a NullReferenceException because of a class in the global namespace). Patch in svn-trunk.
  4. Documentation is an alternate view of a class library

That final point is the major point: through it, I realized that several methods in Mono.Unix which should be private were instead public. Oops. Obviously I need to document things more often...

Posted on 30 Jan 2005 | Path: /development/mono/ | Permalink