Jonathan Pryor's web log

HackWeek V

jonpryor@vt.edu (Jonathan Pryor) — Tue, 15 Jun 2010 19:23:00 -0500

Last week was HackWeek V, during which I had small goals, yet had most of the time eaten by unexpected "roadblocks."

The week started with my mis-remembering OptionSet behavior. I had thought that there was a bug with passing options containing DOS paths, as I thought the path would be overly split:

string path = null;
var o = new OptionSet () {
	{ "path=", v => path = v },
};
o.Parse (new[]{"-path=C:\path"});

Fortunately, my memory was wrong: this works as expected. Yay.

What fails is if the option supports multiple values:

string key = null, value = null;
var o = new OptionSet () {
	{ "D=", (k, v) => {key = k; value = v;} },
};
o.Parse (new[]{"-DFOO=C:\path"});

The above fails with a OptionException, because the DOS path is split, so OptionSet attempts to send 3 arguments to an option expecting 2 arguments. This isn't allowed.

The patch to fix the above is trivial (most of that patch is for tests). However, the fix didn't work at first.

Enter roadblock #1: String.Split() can return too many substrings. Oops.

So I fixed it. That only killed a day...

Next up, I had been sent an email showing that OptionSet had some bugs when removing by index. I couldn't let that happen...and being in a TDD mood, I first wrote some unit tests to describe what the IList<T> semantics should be. Being in an over-engineering mood, I wrote a set of "contract" tests for IList<T> in Cadenza, fixed some Cadenza bugs so that Cadenza would pass the new ListContract, then merged ListContract with the OptionSet tests.

Then I hit roadblock #2 when KeyedCollection<TKey, TItem> wouldn't pass my ListContract tests, as it wasn't exception safe. Not willing to give up on ListContract, I fixed KeyedCollection so it would now pass my ListContract tests, improving compatibility with .NET in the process, which allowed me to finally fix the OptionSet bugs.

I was then able to fix a mdoc export-html bug in which index files wouldn't always be updated, before starting to investigate mdoc assemble wanting gobs of memory.

While pondering how to figure out why mdoc assemble wanted 400MB of memory, I asked the folks on ##csharp on freenode if there were any Mono bugs preventing their SpikeLite bot from working under Mono. They kindly directed me toward a bug in which AppDomain.ProcessExit was being fired at the wrong time. This proved easier than I feared (I feared it would be beyond me).

Which left me with pondering a memory "leak." It obviously couldn't be a leak with a GC and no unmanaged memory to speak of, but what was causing so much memory to be used? Thus proceeded lots of Console.WriteLine(GC.GetTotalMemory(false)) calls and reading the output to see where the memory use was jumping (as, alas I found Mono's memory profiler to be less than useful for me, and mono's profiler was far slower than a normal run). This eventually directed me to the problem:

I needed, at most, two XmlNode values from an XmlDocument. An XmlDocument loaded from a file that could be very small or large-ish (0.5MB). Thousands of such files. At once.

That's when it dawned on me that storing XmlNodes in a Dictionary loaded from thousands of XmlDocuments might not be such a good idea, as each XmlNode retains a reference to the XmlDocument it came from, so I was basically copying the entire documentation set into memory, when I only needed a fraction of it. Doh!

The fix was straightforward: keep a temporary XmlDocument around and call XmlDocument.ImportNode to preserve just the data I needed.

Memory use plummeted to less than one tenth what was previously required.

Along the way I ran across and reported an xbuild bug (since fixed), and filed a regression in gmcs which prevented Cadenza from building.

Overall, a productive week, but not at all what I had originally intended.

Defending XML-based Build Systems

jonpryor@vt.edu (Jonathan Pryor) — Mon, 26 Apr 2010 21:24:00 -0500

Justin Etheredge recently suggested that we Say Goodbye to NAnt and MSBuild for .NET Builds With IronRuby. Why? because they're based on XML.

He goes on to mention several problems with XML-based build systems, principally:

It's not code (unless you're using XSLT, and having maintained XSLTs if you need to write them you have my condolences...)
Lose existing tooling: editors, debuggers, libraries, etc.
Limit creation of custom rules.
Require that we write custom tools to workaround the limitations of XML build systems.

His solution: use Ruby to describe your build process.

My reaction? No, no, for the love of $deity NO!

Why? Three reasons: GNU Autotools, Paul E. McKenney's excellent parallel programming series, and SQL.

Wait, what? What do those have to do with build systems? Everything, and nothing.

The truly fundamental problem is this: "To a man with a hammer, everything looks like a nail" (reportedly a quote from Mark Twain, but that's neither here nor there). In this case, the "hammer" is "writing code." But it's more than that: it's writing imperative code, specifically Ruby code (though the particular language isn't the problem I have, rather the imperative aspect).

Which is, to me, the fundamental problem: it's a terrible base for any form of higher-level functionality. Suppose you want to build your software in parallel (which is where Paul McKenney's series comes in). Well, you can't, because your entire build system is based on imperative code, and unless all the libraries you're using were written with that in mind...well, you're screwed. The imperative code needs to run, and potentially generate any side effects, and without a "higher-level" description of what those side effects entail it can't sanely work.

Want to add a new file to your build (a fairly common thing to do in an IDE, along with renaming files?) Your IDE needs to be able to understand the imperative code. If it doesn't, it just broke your build script. Fun!

OK, what about packaging? Well, in order to know what the generated files are (and where they're located), you'll have to run the entire script and (somehow) track what files were created.

Want to write an external tool that does something hitherto unknown? (As a terrible example, parse all C# code for #if HAVE_XXX blocks so that a set of feature tests can be automatically extracted.) Well, tough -- you have to embed an IronRuby interpreter, and figure out how to query the interpreter for the information you want (e.g. all the source files).

etc., etc.

My problem with imperative languages is that they're not high-level enough. McKenney asks what the good multicore programming languages are; the answer is SQL because it's dedicated ~solely to letting you describe the question but leaves the implementation of the answer to the question up to the SQL database. It's not imperative, it's declarative (at least until you hit esoteric features such as cursors, but in principal you can generally stick to a declarative subset).

OK, so I want a higher-level language to describe targets and dependencies, and supports faster builds. To a large degree, make(1) supports all that, and it's the basis of Autotools. Surely I like that, right?

The problem with autotools is that it's a mixture of declarative and imperative code, with Unix shell scripts forming the backbone of the imperative code (aka the target rules), and these are inherently Unix specific. (Possibly Linux specific, much to my consternation.) Plus, the format is virtually unreadable by anything other than make(1), what with all the language extensions...

So why XML?

Because it's not code, it's data, which (somewhat) lowers the barrier of entry for writing external tools which can parse the format and Do New Things without needing to support some language which might not even run on the platform you're using.

Because it's easily parseable AND verifiable, it's (somewhat) safer for external automated tools to manipulate the file without screwing you over "accidentally" -- e.g. adding and removing files from the build via an IDE.

Because custom rules are limited, there is a smaller "grammar" for external tools to understand, making it simpler to write and maintain them. It also encourages moving "non-target targets" out of the build system, simplifying file contents (and facilitating interaction with e.g. IDEs).

Am I arguing that XML-based build systems are perfect? Far from it. I'm instead arguing that small, purpose-specific languages can (and often are) Good Things™, particularly if they permit interoperability between a variety of tools and people. XML allows this, if imperfectly. An IronRuby-based build system does not.

mdoc Repository Format History

jonpryor@vt.edu (Jonathan Pryor) — Wed, 20 Jan 2010 15:23:00 -0500

Time to wrap up this overly long series on mdoc. We covered:

To close out this series, where did the mdoc repository format come from? It mostly came from Microsoft, actually.

Taking a step back, "in the beginning," as it were, the Mono project saw the need for documentation in January 2002. I wasn't involved then, but perusing the archives we can see that csc /doc output was discarded early because it wouldn't support translation into multiple languages. NDoc was similarly discarded because it relied on csc /doc documentation. I'm sure a related problem at the time was that Mono's C# compiler didn't support the /doc compiler option (and wouldn't begin to support /doc until April 2004), so there would be no mechanism to extract any inline documentation anyway.

By April 2003 ECMA standardization of the Common Language Infrastructure was apparently in full force, and the standardization effort included actual class library documentation. The ECMA documentation is available within ECMA-335.zip. The ECMA-335 documentation also included a DTD for the documentation contained therein, and it was a superset of the normal C# XML documentation. The additional XML elements provided what XML documentation lacked: information available from the assembly, such as actual parameter types, return types, base class types, etc. There was one problem with ECMA-335 XML, though: it was gigantic, throwing everything into a single 7MB+ XML file.

To make this format more version-control friendly (can you imagine maintaining and viewing diffs on a 7+MB XML file?), Mono "extended" the ECMA-335 documentation format by splitting it into one file per type. This forms the fundamental basis of the mdoc repository format (and is why I say that the repository format came from Microsoft, as Microsoft provided the documentation XML and DTD to ECMA). This is also why tools such as mdoc assemble refer to the format as ecma. The remainder of the Mono extensions were added in order to fix various documentation bugs (e.g. to distinguish between ref vs. out parameters, to better support generics), etc.

In closing this series, I would like to thank everyone who has ever worked on Monodoc and the surrounding tools and infrastructure. It wouldn't be anywhere near as useful without them.

Assembly Versioning with mdoc

jonpryor@vt.edu (Jonathan Pryor) — Tue, 19 Jan 2010 17:12:00 -0500

Previously, we mentioned as an aside that the Type.xml files within an mdoc repository contained //AssemblyVersion elements. Today we will discuss what they're for.

The //AssemblyVersion element records exactly one thing: which assembly versions a type and member was found in. (The assembly version is specified via the AssemblyVersionAttribute attribute.) With a normal assembly versioning policy, this allows monodoc to show two things: which version added the type/member, and (by inference) which version(s) removed the member.

For example, consider the NetworkStream.Close method. This method was present in .NET 1.0 which overrode Stream.Close. However, in .NET 2.0 the override was removed entirely.

The //AssemblyVersion attribute allows the mdoc repository to track such versioning changes; for example, consider the mdoc-generated NetworkStream.xml file. The //Member[@MemberName='Close']/AssemblyInfo/AssemblyVersion elements contain only an entry for 1.0.5000.0 (corresponding to .NET 1.1) on line 536. Compare to the //Member[@MemberName='CanWrite']/AssemblyInfo/AssemblyVersion elements (for the NetworkStream.CanWrite property) which has //AssemblyVersion elements for 1.0.5000.0 and 2.0.0.0. From this, we can deduce that NetworkStream.Close was present in .NET 1.1, but was removed in .NET 2.0.

When viewing type and member documentation, monodoc and the ASP.NET front end will show the assembly versions that have the member:

There are two limitations with the version tracking:

It only tracks types and members. For example, attributes, base classes, and interfaces may be added or removed across versions; these are not currently tracked.
It uses the assembly version to fill the <AssemblyVersion> element.

The second point may sound like a feature (isn't it the point?), but it has one downfall: auto-generated assembly versions. You can specify an auto-generated assembly version by using the * for some components in the AssemblyVersionAttribute constructor:

[assembly: AssemblyVersion("1.0.*.*")]

If you do this, every time you rebuild the assembly the compiler will dutifully generate a different assembly number. For example, the first time you might get a compiler version of 1.0.3666.19295, while the second recompilation the compiler will generate 1.0.3666.19375. Since mdoc assigns no meaning to the version numbers, it will create //AssemblyVersion elements for each distinct version found.

The "advantage" is that you can know on which build a member was added. (If you actually care...)

The disadvantage is a major bloating of the mdoc repository, as you add at least 52*(1+M) bytes to each file in the mdoc repository for each unique assembly version (where M is the number of members within the file, as each member is separately tracked). It will also make viewing the documentation distracting; imagine seeing 10 different version numbers for a member, which all differ in the build number. That much noise would make the feature ~useless.

As such, if you're going to use mdoc, I highly suggest not using auto-generated assembly version numbers.

Next time, we'll wrap up this series with a history of the mdoc repository format.

Caching mdoc's ASP.NET-generated HTML

jonpryor@vt.edu (Jonathan Pryor) — Fri, 15 Jan 2010 17:50:00 -0500

Last time we discussed configuring the ASP.NET front-end to display monodoc documentation. The display of extension methods within monodoc and the ASP.NET front-end is fully dynamic. This has it's pros and cons.

On the pro side, if/when you install additional assembled documentatation sources, those sources will be searched for extension methods and they will be shown on all matching types. This is very cool.

On the con side, searching for the extension methods and converting them into HTML takes time -- there is a noticable delay when viewing all members of a type if there are lots of extension methods. On heavily loaded servers, this may be detrimental to overall performance.

If you're running the ASP.NET front-end, you're not regularly adding documentation, and you have Mono 2.6, you can use the mdoc export-html-webdoc command to pre-render the HTML files and cache the results. This will speed up future rendering.

For example, consider the url http://localhost:8080/index.aspx?link=T:System.Collections.Generic.List`1/* (which shows all of the List<T> members). This is a frameset, and the important frame here is http://localhost:8080/monodoc.ashx?link=T:System.Collections.Generic.List`1/* which contains the member listing (which includes extension methods). On my machine, it takes ~2.0s to download this page:

$ time curl -s \
	'http://localhost:8080/monodoc.ashx?link=T:System.Collections.Generic.List`1/*' \
	> /dev/null

real	0m2.021s
user	0m0.003s
sys	0m0.002s

In a world where links need to take less than 0.1 seconds to be responsive, this is...pretty bad.

After running mdoc export-html-webdoc netdocs.zip (which contains the List<T> docs):

$ time curl -s \
	'http://localhost:8080/monodoc.ashx?link=T:System.Collections.Generic.List`1/*' \
	> /dev/null

real	0m0.051s
user	0m0.003s
sys	0m0.006s

That's nearly 40x faster, and within the 0.1s guideline.

Cache Generation: to generate the cache files, run mdoc export-html-web ASSEMBLED-FILES. ASSEMBLED-FILES consists of the .tree or .zip files which are generated by mdoc assemble and have been installed into $prefix/lib/monodoc/sources:

$ mdoc export-html-webdoc $prefix/lib/monodoc/sources/Demo.zip

(Where $prefix is your Mono installation prefix, e.g. /usr/lib/monodoc/sources/Demo.zip.)

This will create a directory tree within $prefix/lib/monodoc/sources/cache/Demo. Restarting the ASP.NET front-end will allow it to use the cache.

If you don't want to generate the cache in another directory, use the -o=PREFIX option. This is useful if you're updating an existing cache on a live server and you don't want to overwrite/replace the existing cache (it's a live server!) -- generate the cache elsewhere, then move the files when the server is offline.

If you have lots of time on your hands, you could process all assembled documentation with:

$ mdoc export-html-webdoc $prefix/lib/monodoc/sources/*.zip

Limitations: It should be noted that this is full of limitations, so you should only use it if performance is really important. Limitations include:

The existence of the cache subdirectories are more important than any timestamps; if the .zip file is newer than the corresponding cache directory, the cache contents will still be returned.
It's privy to monodoc internals, and thus you may need to regenerate all cached documentation whenever you add or remove .zip files. For example, since it can be used to show extension methods, and any set of documentation can contain extension methods, adding or removing assembled documentation files may render the cached output out of date.
mdoc export-html-webdoc processing is slow. Processing the 2.4KB Demo.zip takes a speedy 1.2s. Processing the 5.8MB netdocs.zip (51MB uncompressed, containing 4810 types with 45267 members, including List<T> documentation) takes an astounding 247m (over 4 hours). The resulting cache/netdocs directory is 316MB.

Next time, we'll cover mdoc's support for assembly versioning.

Configuring the ASP.NET front-end for mdoc

jonpryor@vt.edu (Jonathan Pryor) — Thu, 14 Jan 2010 14:43:00 -0500

Last time, we assembled our documentation and installed it for use with monodoc. This is a prerequisite for ASP.NET support (as they both use the same system-wide documentation directory).

Once the documentation is installed (assuming a Linux distro or OSX with the relevant command-line tools installed), you can trivially host a web server which will display the documentation:

$ svn co http://anonsvn.mono-project.com/source/branches/mono-2-4/mono-tools/webdoc/
# output omitted...
$ cd webdoc
$ xsp2

You will need to change the svn co command to use the same version of Mono that is present on your system. For example, if you have Mono 2.6 installed, change the mono-2-4 to mono-2-6.

Once xsp2 is running, you can point your web browser to http://localhost:8080 to view documentation. This will show the same documentation as monodoc did last time:

For "real" use, setting up using Apache with mod_mono may be preferred (or any of the other options listed at Mono's ASP.NET support page). Configuring mod_mono or anything other than xsp2 is beyond my meager abilities.

Next time, we'll discuss improving the ASP.NET front-end's page rendering performance.

Assembling Documentation with mdoc

jonpryor@vt.edu (Jonathan Pryor) — Wed, 13 Jan 2010 14:43:00 -0500

We previously discussed exporting the mdoc repository into static HTML files using mdoc export-html and into a Microsoft XML Documentation file with mdoc export-msxdoc. Today, we'll discuss exporting documentation with mdoc assemble.

mdoc assemble is used to assemble documentation for use with the monodoc Documentation browser and the ASP.NET front-end. This involves the following steps:

Running mdoc assemble.
Writing a .source file.
Installing the files.

Unfortunately we're taking a diversion from the Windows world, as the monodoc browser and the ASP.NET front-end won't run under Windows (due to limitations in the monodoc infrastructure). I will attempt to fix these limitations in the future.

Running mdoc assemble: mdoc assemble has three arguments of interest:

-f=FORMAT is used to specify the format of the files to assemble. When documenting assemblies you can skip this, as the default format is for mdoc repositories. This is useful if you want to assemble other materials, such as man pages or plain HTML files.
-o=PREFIX is used to specify the output prefix. mdoc assemble generates two files, a .tree and a .zip file. The PREFIX value is the basename to use for these to files.
The list of files or directories to process. Whether these need to be files or directories (and related semantics) depends upon the format specified; see the FORMATS section of the mdoc-assemble(1) man page for details.

For our current documentation, we would run:

$ mdoc assemble -o Demo Documentation/en.docs

This will create the files Demo.tree and Demo.zip in the current working directory.

The .source file is used to tell the documentation browser where in the tree the documentation should be inserted. It's an XML file that contains two things: a (set of) /monodoc///node elements describing where in the tree the documentation should be inserted, and /monodoc/source elements which specify the files to use. For example:

<?xml version="1.0"?>
<monodoc>
  <node label="Demo Library" name="Demo-lib" parent="libraries" />
  <source provider="ecma" basefile="Demo" path="Demo-lib"/>
</monodoc>

The /monodoc/node element describes where in the monodoc tree the documentation should be placed. It has three attributes, two of which are required:

label is the text to display in the tree view.
name is the name of the node, so that other nodes and the /monodoc/source/@path attribute may refer to it.
parent is optional, and contains the //node/@name value of the node which should be the parent of this node. This is used to provide a degree of structure. It should be a value from $prefix/lib/monodoc/monodoc.xml in the //node/@name attribute values. Currently these include:
- languages for programming language references, e.g. The C# Language Specification.
- libraries for class library documentation.
- man for man pages and other command references.
- tools and various for anything that doesn't fit in the above descriptions.
If not specified and this is the /monodoc/node element, this defaults to the various node. If this is a nested //node element, the @parent attribute defaults to the parent //node element (../@name).

The /monodoc/source element describes what file basename to use when looking for the .tree and .zip files. (By convention the .source, .tree, and .zip files share the same basename, but this is not required. The .tree and .zip files must share the same basename, but the .source basename may differ, and will differ if e.g. one .source file pulls in several .tree/.zip pairs.) It has three attributes, all of which are required:

basefile is the file basename of the .tree and .zip files.
path is the //node/@name value that will be associated with the docs within basefile.tree and basefile.zip.
provider is the format provided to mdoc assemble. For assembly documentation, this should be ecma.

Installing the files. Files need to be installed into $prefix/lib/monodoc/sources. You can obtain this directory with pkg-config(1):

$ cp Demo.source Demo.tree Demo.zip \
    `pkg-config monodoc --variable=sourcesdir`

Now when we run monodoc, we can navigate to the documentation that was just installed:

Additionally, those paying attention on January 10 will have noticed that the With() method we documented is an extension method. Monodoc supports displaying extension methods on the relevant type documentation. In this case, With() is an extension on TSource, which is, for all intents and purposes, System.Object. Thus, if we view the System.Object docs within our local monodoc browser, we will see the With() extension method:

In fact, we will see With() listed as an extension method on all types (which is arguably a bug, as static types can't have instance methods...).

Furthermore, mdoc export-html will also list extension methods. However, mdoc export-html is far more limited: it will only look for extension methods within the mdoc repositories being processing, and it will only list those methods as extension methods on types within the mdoc repository. Consequently, mdoc export-html will not list e.g. IEnumerable<T> extension methods on types that implement IEnumerable<T>. (It simply lacks the information to do so.)

Examples of mdoc export-html listings of extension methods can be found in the mdoc unit tests and the Cadenza.Collections.CachedSequence<T> docs (which lists a million extension methods because Cadenza.Collections.EnumerableCoda contains a million extension methods on IEnumerable<T>).

Next time, we'll discuss setting up the ASP.NET front end under Linux.

Exporting mdoc Repositories to Microsoft XML Documentation

jonpryor@vt.edu (Jonathan Pryor) — Tue, 12 Jan 2010 13:20:00 -0500

Previously, we discussed how to write documentation and get it into the documentation repository. We also discussed exporting the documentation into static HTML files using mdoc export-html. Today, we'll discuss mdoc export-msxdoc.

mdoc export-msxdoc is used to export the documentation within the mdoc repository into a .xml file that conforms to the same schema as csc /doc. This allows you, if you so choose, to go entirely to externally managed documentation (instead of inline XML) and still be able to produce your Assembly.xml file so that Visual Studio/etc. can provide code completion against your assembly.

There are two ways to invoke it:

$ mdoc export-msxdoc Documentation/en
$ mdoc export-msxdoc -o Demo.xml Documentation/en

The primary difference between these is what files are generated. Within each Type.xml file of the mdoc repository (e.g. ObjectCoda.xml) is a /Type/AssemblyInfo/AssemblyName element.

The first command (lacking -o Demo.xml) will generate a set of .xml files, where the filenames are based on the values of the /Type/AssemblyInfo/AssemblyName element values, in this case Demo.xml. Additionally, a NamespaceSummaries.xml file is generated, containing documentation for any namespaces that were documented (which come from the ns-*.xml files, e.g. ns-Cadenza.xml).

The second command (which specifies -o Demo.xml) will only generate the specified file (in this case Demo.xml).

For this mdoc repository, there is no actual difference between the commands (as only one assembly was documented within the repository), except for the generation of the NamespaceSummaries.xml file. However, if you place documentation from multiple assemblies into the same mdoc repository, the first command will properly generate .xml files for each assembly, while the latter will generate only a single .xml file containing the documentation from all assemblies.

Next time, we'll cover mdoc assemble.

Customizing mdoc's Static HTML Output

jonpryor@vt.edu (Jonathan Pryor) — Mon, 11 Jan 2010 13:19:00 -0500

Last time, we wrote documentation for our Demo.dll assembly. What if we want to improve the looks of those docs, e.g. to change the colors or add additional navigation links for site consistency purposes?

mdoc export-html uses three mechanisms to control output:

--ext=FILE-EXTENSION is used to change the file extension of generated files from .html to FILE-EXTENSION. This is useful if you want to generate e.g. .aspx files instead of .html files (the default).
--template=TEMPLATE-FILE specifies an XSLT to use for the layout of all generated files. If not specified, then the XSLT returned by mdoc export-html --default-template is used.
HTML CSS class names are used throughout the documentation, allowing various elements to be customized by providing an alternate stylesheet. The mdoc export-html man page lists the CSS classes that are used.

The XSLT needs to consume an XML document that has the following structure:

<Page>
    <CollectionTitle>Collection Title</CollectionTitle>
    <PageTitle>Page Title</PageTitle>
    <Summary>Page Summary</Summary>
    <Signature>Type Declaration</Signature>
    <Remarks>Type Remarks</Remarks>
    <Members>Type Members</Members>
    <Copyright>Documentation Copyright</Copyright>
</Page>

The contents of each of the //Page/* elements contains HTML or plain text nodes. Specifically:

/Page/CollectionTitle

Contains the Assembly and Namespace name links.

/Page/PageTitle

Contains the type name/description.

/Page/Summary

Contains the type <summary/> documentation.

/Page/Signature

Contains the type signature, e.g. whether it's a struct or class, implemented interfaces, etc.

/Page/Remarks

Contains type-level <remarks/>.

/Page/Members

Contains the documentation for all of the members of the type, including a table for all of the members.

/Page/Copyright

Contains copyright information taken from the mdoc repository, specifically from index.xml's /Overview/Copyright element.

By providing a custom --template XSLT and/or by providing an additional CSS file, you have some degree of control over the resulting documentation.

I'll be the first to admit that this isn't a whole lot of flexibility; there is no control over what CSS class names are used, nor is there any control over what is generated within the /Page//* elements. What this model does allow is for controlling the basic page layout, e.g. to add a site-wide menu system, allowing documentation to be consistent with the rest of the site.

For example, my site uses custom templates to provide a uniform look-and-feel with the rest of their respective sites for the Mono.Fuse and NDesk.Options documentation.

Next time, we'll cover mdoc export-msxdoc.

mdoc XML Schema

jonpryor@vt.edu (Jonathan Pryor) — Sun, 10 Jan 2010 14:22:00 -0500

Previously, I mentioned that you could manually edit the XML files within the mdoc repository.

What I neglected to mention is that there are only parts of the XML files that you should edit, and that there is an XML Schema file available for all docs.

The mdoc(5) man page lays out which files within the repository (and which parts of those files) are editable. In summary, all ns-*.xml files and the //Docs nodes of all other .xml files are editable, and they should contain ye normal XML documentation elements (which are also documented within the mdoc(5) man page).

The XML Schema can be found in Mono's SVN, at http://anonsvn.mono-project.com/source/trunk/mcs/tools/mdoc/Resources/monodoc-ecma.xsd.

Writing Documentation for mdoc

jonpryor@vt.edu (Jonathan Pryor) — Sun, 10 Jan 2010 13:20:00 -0500

Last time, we create an assembly and used mdoc to generate a documentation repository containing stubs. Stubs have some utility -- you can view the types, members, and parameter types that are currently present -- but they're far from ideal. We want actual documentation.

Unfortunately, mdoc isn't an AI, and can't write documentation for you. It manages documentation; it doesn't create it.

How do we get actual documentation into the respository? There are three ways:

Manually edit the XML files within the repository directory (if following from last time, this would be all .xml files within the Documentation/en directory.
Use monodoc --edit Documentation/en.
We can continue writing XML documentation within our source code.

Manually editing the files should be self-explanatory; it's not exactly ideal, but it works, and is how I write most of my documentation.

When using monodoc --edit Documentation/en, the contents of Documentation/en will be shown sorted in the tree view by it's assembly name, e.g. in the Mono Documentation → Demo node. When viewing documentation, there are [Edit] links that, when clicked, will allow editing the node (which directly edits the files within Documentation/en.

However, I can't recommend monodoc as an actual editor. It's usability is terrible, and has one major usability flaw: when editing method overloads, most of the documentation will be the same (or similar enough that you'll want to copy everything anyway), e.g. <summary/>, <param/>, etc. The monodoc editor doesn't allow copying all of this at once, but only each element individually. It makes for a very slow experience.

Which brings us to inline XML documentation. mdoc update supports importing XML documentation as produced by csc /doc. So let's edit our source code to add inline documentation:

using System;

namespace Cadenza {
    /// <summary>
    ///  Extension methods on <see cref="T:System.Object" />.
    /// </summary>
    public static class ObjectCoda {
        /// <typeparam name="TSource">The type to operate on.</typeparam>
        /// <typeparam name="TResult">The type to return.</typeparam>
        /// <param name="self">
        ///   A <typeparamref name="TSource" /> containing the value to manipulate.
        ///   This value may be <see langword="null" /> (unlike most other
        ///   extension methods).
        /// </param>
        /// <param name="selector">
        ///   A <see cref="T:System.Func{TSource,TResult}" /> which will be
        ///   invoked with <paramref name="self" /> as a parameter.
        /// </param>
        /// <summary>
        ///   Supports chaining otherwise temporary values.
        /// </summary>
        /// <returns>
        ///   The value of type <typeparamref name="TResult" /> returned by
        ///   <paramref name="selector" />.
        /// </returns>
        /// <remarks>
        ///   <para>
        ///     <c>With</c> is useful for easily using an intermediate value within
        ///     an expression "chain" without requiring an explicit variable
        ///     declaration (which is useful for reducing in-scope variables, as no
        ///     variable is explicitly declared).
        ///   </para>
        ///   <code lang="C#" src="../../example.cs#With" />
        /// </remarks>
        /// <exception cref="T:System.ArgumentNullException">
        ///   <paramref name="selector" /> is <see langword="null" />.
        /// </exception>
        public static TResult With<TSource, TResult>(
                this TSource self, 
                Func<TSource, TResult> selector)
        {
            if (selector == null)
                throw new ArgumentNullException ("selector");
            return selector (self);
        }
    }
}

(As an aside, notice that our file ballooned from 14 lines to 45 lines because of all the documentation. This is why I prefer to keep my documentation external to the source code, as it really bloats the source. Certainly, the IDE can hide comments, but I find that this defeats the purpose of having comments in the first place.)

Compile it into an assembly (use csc if running on Windows), specifying the /doc parameter to extract XML documentation comments:

$ gmcs /t:library /out:Demo.dll /doc:Demo.xml demo.cs

Update our documentation repository, but import Demo.xml:

$ mdoc update -o Documentation/en -i Demo.xml Demo.dll --exceptions=added
Updating: Cadenza.ObjectCoda
Members Added: 0, Members Deleted: 0

(No members were added or deleted as we're only changing the documentation, and didn't add any types or members to the assembly.)

Now when we view ObjectCoda.xml, we can see the documentation that was present in the source code.

However, notice one other change. In the documentation we wrote, we had:

        ///   <code lang="C#" src="../../example.cs#With" />

Yet, within ObjectCoda.xml, we have:

          <code lang="C#" src="../../example.cs#With">Console.WriteLine(
    args.OrderBy(v => v)
    .With(c => c.ElementAt (c.Count()/2)));
</code>

What's going on here? What's going on is that mdoc will search for all <code/> elements. If they contain a //code/@src attribute, the specified file is read in and inserted as the //code element's value. The filename specified in the //code/@src attribute is relative to the documentation repository root. A further extension is that, for C# code, if the filename has an "anchor", a #region block of the same name is searched for within the source code.

The ../../example.cs file referenced in the //code/@src value has the contents:

using System;
using System.Linq;
using Cadenza;

class Demo {
    public static void Main (string[] args)
    {
        #region With
        Console.WriteLine(
            args.OrderBy(v => v)
            .With(c => c.ElementAt (c.Count()/2)));
        #endregion
    }
}

This makes keeping documentation examples actually compiling trivial to support. For example, I'll have documentation refer to my unit tests, e.g.

<code lang="C#" src="../../Test/Cadenza/ObjectTest.cs#With" />

One final point worth mentioning: you can import documentation as often as you want. The imported documentation will always overwrite whatever is already present within the documentation repository. Consequently, if you want to use mdoc for display purposes but want to continue using inline XML documentation, always import the compiler-generated .xml file.

Now, we can update our HTML documentation:

$ mdoc export-html -o html Documentation/en
Cadenza.ObjectCoda

The current Demo.dll documentation.

Next time, we'll cover customizing the static HTML output.

Using mdoc

jonpryor@vt.edu (Jonathan Pryor) — Sat, 09 Jan 2010 17:28:00 -0500

As mentioned last time, mdoc is an assembly-based documentation management system. Thus, before you can use mdoc you need an assembly to document. Let's write some C# source:

using System;

namespace Cadenza {
    public static class ObjectCoda {
        public static TResult With<TSource, TResult>(
                this TSource self, 
                Func<TSource, TResult> selector)
        {
            if (selector == null)
                throw new ArgumentNullException ("selector");
            return selector (self);
        }
    }
}

Compile it into an assembly (use csc if running on Windows):

$ gmcs /t:library /out:Demo.dll demo.cs

Now that we have an assembly, we can create the mdoc repository for the Demo.dll assembly, which will contain documentation stubs for all publically visible types and members in the assembly:

$ mdoc update -o Documentation/en Demo.dll --exceptions=added
New Type: Cadenza.ObjectCoda
Member Added: public static TResult With<TSource,TResult> (this TSource self, Func<TSource,TResult> selector);
Namespace Directory Created: Cadenza
New Namespace File: Cadenza
Members Added: 1, Members Deleted: 0

mdoc update is the command for for synchronizing the documentation repository with the assembly; it can be run multiple times. The -o option specifies where to write the documentation repository. Demo.dll is the assembly to process; any number of assemblies can be specified. The --exceptions argument analyzes the IL to statically determine which exception types can be generated from a member. (It is not without some limitations; see the "--exceptions" documentation section.) The added argument to --exceptions tells mdoc to add <exception/> elements only for types and members that have been added to the repository, not to all types and members in the assembly. This is useful for when you've removed <exception/> documentation and don't want mdoc to re-add them.

We choose Documentation/en as the documentation repository location so that we can easily support localizing the documentation into multiple languages: each directory underneath Documentation would be named after an ISO 639-1 code, e.g. en is for English. This is only a convention, and is not required; any directory name can be used.

Notice that, since mdoc is processing assemblies, it will be able to work with any language that can generate assemblies, such as Visual Basic.NET and F#. It does not require specialized support for each language.

Now we have a documentation repository containing XML files; a particularly relevant file is ObjectCoda.xml, which contains the documentation stubs for our added type. I won't show the output here, but if you view it there are three important things to note:

The XML is full of type information, e.g. the /Type/Members/Member/Parameters/Parameter/@Type attribute value.
The XML contains additional non-documentation information, such as the //AssemblyVersion elements. This will be discussed in a future blog posting.
The //Docs elements are a container for the usual C# XML documentation elements.

Of course, a documentation repository isn't very useful on it's own. We want to view it! mdoc provides three ways to view documentation:

mdoc export-html: This command generates a set of static HTML files for all types and members found within the documentation repository.
mdoc assemble: This command "assembles" the documentation repository into a .zip and .tree file for use with the monodoc Documentation browser and the ASP.NET front-end (which powers http://www.go-mono.com/docs).
mdoc export-msxdoc: This generates the "traditional" XML file which contains only member documentation. This is for use with IDEs like Visual Studio, so that the IDE can show summary documentation while editing.

We will cover mdoc assemble and mdoc export-msxdoc in future installments. For now, to generate static HTML:

$ mdoc export-html -o html Documentation/en
Cadenza.ObjectCoda

The current Demo.dll documentation.

Next time we will cover how to write actual documentation instead of just documentation stubs.

TekPub's Mastering LINQ Challenge

jonpryor@vt.edu (Jonathan Pryor) — Sat, 09 Jan 2010 03:12:00 -0500

Justin Etheredge has posted TekPub's Mastering LINQ Challenge, in which he lays out a "little LINQ challenge." The rules:

You have to blog about a single LINQ query which starts with Enumerable.Range(1,n) and produces a list of prime numbers from the range. Thus, this blog posting. (Otherwise I'd rely on my twitter response.)
You can't cheat. This is determined by me, and includes hardcoding values in the results. You'll know if you cheated. Part of me wonders if just being me qualifies as cheating, but that might imply that my computer self has too large an ego ^{</ob-##csharp-meme>}.
Uses no custom LINQ methods. Here I ponder what constitutes a "custom LINQ method." Is any extension method a custom LINQ method? Any utility code?
Will return all of the prime numbers of the sequence. It doesn't have to be super optimal, but it has to be correct. Boy is it not super optimal (it's a one liner!), but some improvements could make it better (e.g. Memoization, hence the prior question about whether extension methods constitute a "custom LINQ method").
Be one of the first 5 people to blog a correct answer and then tweet this "I just solved the @tekpub LINQ challenge: <link to post>" will get any single TekPub screencast. The time of your solution will be based on your tweet! So be prompt!
As far as timliness, I'm writing this blog entry over four hours after my tweet, so, uh, so much for timliness.
You must link to both TekPub's website and this post in your blog post.
Done, and done.

So, the quick and dirty, not at all efficent answer (with longer identifiers as I certainly have more than 140 characters to play with:

Enumerable.Range(1, n).Where(value => 
    value <= 3
        ? true
        : Enumerable.Range(2, value - 2)
          .All(divisor => value % divisor != 0))

In English, we take all integers between 1 and n. Given a value from that sequence, if the value is less than 3, it's prime. If it's greater than three, take all numbers from 2 until value-1 and see if any of them divides value with no remainder. If none of them divide with no remainder, value is prime.

We need to use value-2 in the nested Enumerable.Range call so that we skip the value itself (since we're starting at 2).

Now, we can improve upon this in a fairly straightforward fashion if we can use additional code. For example, if we use Bart de Smet's Memoize extension method on System.Func<T, TResult>, we can skip the repeated nested Enumerable.Range call on every value, as prime numbers don't change (and thus are prime candidates for caching ;-):

Func<int, bool> isPrime = value => 
    value <= 3
        ? true
        : Enumerable.Range(2, value - 2)
          .All(divisor => value % divisor != 0))
isPrime = isPrime.Memoize();
Enumerable.Range(1, n).Where(value => isPrime(value));

Whether this latter answer matches the rules depends upon the definition of "single LINQ query" (does the definition of isPrime need to be part of the LINQ query, or just its use?) and whether Bart's Memoize extension method qualifies as a "custom LINQ method" (I don't think it is...). The downside to the memoization is that it's basically a memory leak in disguise, so I still wouldn't call it "optimal," just that it likely has better performance characteristics than my original query...

What is mdoc?

jonpryor@vt.edu (Jonathan Pryor) — Fri, 08 Jan 2010 14:22:00 -0500

mdoc is an assembly-based documentation management system, which recently added support for .NET .

I say "assembly based" because an alternative is source-based, which is what "normal" C# XML documentation, JavaDoc, and perlpod provide. Unlike these source-based systems, in mdoc documentation for public types and members are not present within source code. Instead, documentation is stored externally (to the source), in a directory of XML files (hereafter refered to as the mdoc repository).

Furthermore, mdoc provides commands to:

synchronize the mdoc repository with an assembly being documented (adding to the repository any types and members added to the assembly);
to export the mdoc repository into HTML or Microsoft's XML format (for use with an IDE like Visual Studio and MonoDevelop);
to validate the mdoc repository XML schema;
to assemble the mdoc repository for use with the Monodoc Documentation Browser and the ASP.NET front-end.

Why the mdoc repository?

Why have a directory of XML files as the mdoc repository? The mdoc repository comes from the need to satisfy two goals:

The compiler-generated /doc XML contains no type information.
Having types is very useful for HTML output/etc., so the type information must come from somewhere.

Said "somewhere" could be the actual assemblies being documented, but this has other downsides (e.g. it would complicate supporting different versions of the same assembly). mdoc uses the repository to contain both documentation and full type information, so that the source assemblies are only needed to update the repository (and nothing else).

Why use mdoc?

Which provides enough background to get to the point: why use mdoc?

You would primarily want to use mdoc if you want to view your documentation outside of an IDE, e.g. within a web browser or stand-alone documentation browser. Most mdoc functionality is geared toward making documentation viewable (e.g. mdoc export-html and mdoc assemble), and making the documentation that is viewed more useful (such as the full type information provided by mdoc update and the generation of <exception/> elements for documentation provided by mdoc update --exceptions).

Next time, we'll discuss how to use mdoc.

Re-Introducing mdoc

jonpryor@vt.edu (Jonathan Pryor) — Thu, 07 Jan 2010 14:19:00 -0500

Many moons ago, Jon Skeet announced Noda Time. In it he asked:

How should documentation be created and distributed?

Is Sandcastle the best way of building docs? How easy is it to get it running so that any developer can build the docs at any time? (I've previously tried a couple of times, and failed miserable.)

Would Monodoc be a better approach?

Thus I pondered, "how well does mdoc support Windows users?"

The answer: not very well, particularly in an interop scenario.

So, lots of bugfixing and a false-start later, and I'd like to announce mdoc for .NET. All the power of mdoc, cross-platform.

Note that these changes did not make it into Mono 2.6, and won't be part of a formal Mono release until Mono 2.8. Consequently, if you want to run things under .NET, you should use the above ZIP archive. (You can, of course, install Mono on Windows and then use mdoc via Mono, you just won't be able to run mdoc under .NET.)

The changes made since Mono 2.6 include:

Always generate Unix line endings within mdoc-generated XML files. This allows directories to be shared between Windows and Linux (e.g. with Samba) without causing lots of differences due to end-of-line changes.
Fix the mdoc export-html XML stylesheets so that they would work under .NET (as mdoc uses XSLT and started relying on several Mono bugs).
Use XslCompiledTransform, as .NET's XslTransform is really slow. (Mono's XslCompiledTransform is just a wrapper of XslTransform, so this doesn't change things under Mono, but under .NET this brought a 3+ minute execution down to 1.7 seconds.)
Add support for properly converting <see cref="N:SomeNamespace"/> links into HTML links.

Next time, we'll cover what mdoc is, and why you'd want to use it.

Linq to SQL on Mono 2.6: NerdDinner on Mono

jonpryor@vt.edu (Jonathan Pryor) — Tue, 15 Dec 2009 20:30:00 -0500

NerdDinner is an ASP.NET MVC sample, licensed under the Ms-PL with sources hosted at CodePlex.

Back on May 14th, I wrote that NerdDinner could be run under Mono using trunk.

Now, I'm pleased to note that the just-released Mono 2.6 includes these changes. Furthermore, thanks to ankit's progress on xbuild, installation and setup is easier than before:

Build (or otherwise obtain) Mono 2.6. The Parallel Mono Environments page may be helpful.
Download the NerdDinner 1.0 sources through a web browser. (curl or wget won't work.)

Extract the NerdDinner sources:

$ mkdir -p $HOME/tmp
$ cd $HOME/tmp
$ unzip "/path/to/NerdDinner 1.0.zip"

Build NerdDinner 1.0:
```
$ cd "$HOME/tmp/NerdDinner 1.0"
$ xbuild NerdDinner/NerdDinner.csproj
```
(Unfortunately we can't build just run xbuild (or build NerdDinner.sln) as this requires access to the MSTest assemblies used by the NerdDinner unit tests, which aren't currently present on Mono.)
Only the web portion runs under Mono, as does the data access layer (System.Data.Linq, more affectionately known as Linq to SQL). The database is still Microsoft SQL Server. Go forth and configure the NerdDinner server (if you don't already have one configured).
Back on the Linux side of things, edit $HOME/tmp/NerdDinner 1.0/NerdDinner/ConnectionStrings.config, and change the NerdDinnerConnectionString connection string to:
```
<add name="NerdDinnerConnectionString"
    connectionString="Data Source=gourry\SQLEXPRESS;
    Initial Catalog=NerdDinner;
    User ID=gourry\jonp;
    Password=123456; 
    Integrated Security=true"/>
```
You will need to adjust the machine name in the Data Source parameter to contain your actual computer name, and change the User ID and Password to whatever values you chose when configuring SQL Server.
Configure a MembershipProvider for NerdDinner username/password storage.
Run the web app:
```
$ cd "$HOME/tmp/NerdDinner 1.0/NerdDinner"
$ MONO_IOMAP=all xsp2
```
The MONO_IOMAP environment variable is needed because some link targets used within NerdDinner require case insensitivity.

Some things worth noting since May. First, openSUSE has released openSUSE 11.2, which is apparently more stringent than 11.1. Consequently, you may need to open the firewall so that port 8080 is accessible. You can do this by:

Opening YaST.
Starting the Firewall applet.
In the Allowed Services area, add the HTTP Server and Mono XSP2 ASP.NET Host Service services.
Click Next, then Finish.

One other oddity I encountered is that a url of http://localhost:8080 isn't permitted; using telnet(1) shows that it attempts to connect to ::1... (i.e. a IPv6 address), and the connection is refused. Instead, I needed to connect to http://127.0.0.1:8080.

Mono.Data.Sqlite & System.Data in MonoTouch 1.2 [Preview]

jonpryor@vt.edu (Jonathan Pryor) — Thu, 22 Oct 2009 03:25:00 -0500

One of the new features that will be present in MonoTouch 1.2 is inclusion of the System.Data and Mono.Data.Sqlite assemblies. This is a preview release of System.Data et. al; it may not fully work. Known limitations are at the end of this post.

What Does This Mean?

It means that the following assemblies will be included in MonoTouch 1.2, and thus usable by MonoTouch applications:

System.Data.dll: Reduced; see below.
System.Transactions.dll: Unchanged.
Mono.Data.Tds.dll: Unchanged.
Mono.Data.Sqlite.dll: Unchanged, but see below.

Example?

Sure:

using System;
using System.Data;
using System.IO;
using Mono.Data.Sqlite;

class Demo {
    static void Main (string [] args)
    {
        var connection = GetConnection ();
        using (var cmd = connection.CreateCommand ()) {
            connection.Open ();
            cmd.CommandText = "SELECT * FROM People";
            using (var reader = cmd.ExecuteReader ()) {
                while (reader.Read ()) {
                    Console.Error.Write ("(Row ");
                    Write (reader, 0);
                    for (int i = 1; i < reader.FieldCount; ++i) {
                        Console.Error.Write(" ");
                        Write (reader, i);
                    }
                    Console.Error.WriteLine(")");
                }
            }
            connection.Close ();
        }
    }		

    static SqliteConnection GetConnection()
    {
        var documents = Environment.GetFolderPath (
                Environment.SpecialFolder.Personal);
        string db = Path.Combine (documents, "mydb.db3");
        bool exists = File.Exists (db);
        if (!exists)
            SqliteConnection.CreateFile (db);
        var conn = new SqliteConnection("Data Source=" + db);
        if (!exists) {
            var commands = new[] {
                "CREATE TABLE People (PersonID INTEGER NOT NULL, FirstName ntext, LastName ntext)",
                "INSERT INTO People (PersonID, FirstName, LastName) VALUES (1, 'First', 'Last')",
                "INSERT INTO People (PersonID, FirstName, LastName) VALUES (2, 'Dewey', 'Cheatem')",
                "INSERT INTO People (PersonID, FirstName, LastName) VALUES (3, 'And', 'How')",
            };
            foreach (var cmd in commands)
                using (var c = conn.CreateCommand()) {
                    c.CommandText = cmd;
                    c.CommandType = CommandType.Text;
                    conn.Open ();
                    c.ExecuteNonQuery ();
                    conn.Close ();
                }
        }
        return conn;
    }

    static void Write(SqliteDataReader reader, int index)
    {
        Console.Error.Write("({0} '{1}')", 
                reader.GetName(index), 
                reader [index]);
    }
}

The above code creates the Documents/mydb.db3 SQLite database, populates it if it doesn't already exist, then executes a SQL query against the database using normal, standard, ADO.NET mechanisms.

What's Missing?

Functionality is missing from System.Data.dll and Mono.Data.Sqlite.dll.

Functionality missing from System.Data.dll consists of:

Anything requiring System.CodeDom (e.g. System.Data.TypedDataSetGenerator)
XML config file support (e.g. System.Data.Common.DbProviderConfigurationHandler)
System.Data.Common.DbProviderFactories (depends on XML config file support)
System.Data.OleDb
System.Data.Odbc
The System.EnterpriseServices.dll dependency was removed from System.Data.dll, resulting in the removal of the SqlConnection.EnlistDistributedTransaction(ITransaction) method.

Meanwhile, Mono.Data.Sqlite.dll suffered no source code changes, but instead may be host to a number of runtime issues (the primary reason this is a preview release). Mono.Data.Sqlite.dll binds SQLite 3.5. iPhoneOS, meanwhile, ships with SQLite 3.0. Suffice it to say, some things have changed between the two versions. ;-)

Thus, the real question is this: what's missing in SQLite 3.0? The following functions are used by Mono.Data.Sqlite.dll but are missing from iPhoneOS's SQLite:

sqlite3_column_database_name
sqlite3_column_database_name16
sqlite3_column_origin_name
sqlite3_column_origin_name16
sqlite3_column_table_name
sqlite3_column_table_name16
sqlite3_key
sqlite3_rekey
sqlite3_table_column_metadata

Where are these functions used (i.e. what can't you use from Mono.Data.Sqlite)? These appear to be related to database schema querying, e.g. determining at runtime which columns exist on a given table, such as Mono.Data.Sqlite.SqliteConnection.GetSchema (overriding DbConnection.GetSchema) and Mono.Data.Sqlite.SqliteDataReader.GetSchemaTable (overriding DbDataReader.GetSchemaTable). In short, it seems that anything using DataTable is unlikely to work.

Why Provide Mono.Data.Sqlite?

Why not? We realize that there are pre-existing SQLite solutions, but felt that many people would prefer to use the ADO.NET code they're already familiar with. Bringing System.Data and Mono.Data.Sqlite to MonoTouch permits this.

What About Data Binding?

Data binding with e.g. a UITableView is not currently implemented.

Conclusion

I suck at conclusions. :-)

Hope you enjoy this preview!

Linq to SQL on Mono Update: NerdDinner on Mono

jonpryor@vt.edu (Jonathan Pryor) — Thu, 14 May 2009 18:47:00 -0500

NerdDinner is an ASP.NET MVC sample, licensed under the Ms-PL with sources hosted at CodePlex.

It is now possible to run the web portions of NerdDinner 1.0 on Linux with Mono trunk, thanks to Marek Habersack and Gonzalo Paniagua Javier's help with Mono's ASP.NET and ASP.NET MVC support, and the DbLinq community's assistance with Linq to SQL support.

This shows a growing level of maturity within Mono's Linq to SQL implementation.

Build Mono from trunk. The Parallel Mono Environments page may be helpful.
Download the NerdDinner 1.0 sources through a web browser. (curl or wget won't work.)

Extract the NerdDinner sources:

$ mkdir -p $HOME/tmp
$ cd $HOME/tmp
$ unzip "/path/to/NerdDinner 1.0.zip"

Build NerdDinner 1.0:

$ cd "$HOME/tmp/NerdDinner 1.0/NerdDinner"
$ mkdir bin
$ gmcs -t:library -out:bin/NerdDinner.dll -debug+ -recurse:'*.cs' \
    -r:System -r:System.Configuration -r:System.Core \
    -r:System.Data -r:System.Data.Linq -r:System.Web \
    -r:System.Web.Abstractions -r:System.Web.Mvc \
    -r:System.Web.Routing

As mentioned in the introduction, only the web portion runs under Mono, as does the data access layer (System.Data.Linq, more affectionately known as Linq to SQL). The database is still Microsoft SQL Server. Find yourself a Windows machine, install SQL Server 2008 (Express is fine), and perform the following bits of configuration:
1. Create the database files:
  1. Copy the NerdDinner_log.ldf and NerdDinner.mdf files from the $HOME/tmp/NerdDinner 1.0/NerdDinner/App_Data directory to your Windows machine, e.g. C:\tmp.
  2. Within Windows Explorer, go to C:\tmp, right-click the C:\tmp folder, click Properties, click the Security tab, click Edit..., and add the Full Control, MOdify, Read & execute, List folder contents, Read, and Write permissions to the User group. Click OK.
  3. Repeat the above permissions modifications for the NerdDinner_log.ldf and NerdDinner.mdf files in C:\tmp.
2. Add the NerdDinner database files to Microsoft SQL Server:
  1. Start Microsoft SQL Server Management Studio (Start → All Programs → Microsoft SQL Server 2008 → SQL Server Management Studio).
  2. Connect to your database instance.
  3. Within the Object Explorer (View → Object Explorer), right-click the database name and click Attach....
  4. Within the Attach Databases dialog, click the Add... button, and choose C:\tmp\NerdDinner.mdf in the Locate Database Files dialog. Click OK in both the Locate Database Files dialog and the Attach Databases dialog.
3. Enable mixed-mode authentication:
  1. Start Microsoft SQL Server Management Studio.
  2. Connect to your database instance.
  3. Within the Object Explorer, right-click the database name and click Properties.
  4. In the Server Properties dialog, select the Security page.
  5. In the Server authentication section, select the SQL Server and Windows Authentication mode radio button.
  6. Click OK.
  7. Restart SQL Server by right-clicking on the database name and clicking Restart.
4. Add a SQL Server user:
  1. Within SQL Server Management Studio, connect to the database instance.
  2. Within the Object Explorer, expand the Security → Logins tree node.
  3. Right-click the Logins node, and click New Login....
  4. In the Login - New dialog, enter a login name. We'll use jonp for discussion purposes. Select the SQL Server authentication dialog button, and enter a password in the Password and Conform Password text boxes. For discussion purposes we'll use 123456.
  5. Still within the Login - New dialog, select the User Mapping page. In the Users mapped to this login section, select the checkbox in the Map column corresponding to the NerdDinner database. Within the Database role membership for: NerdDinner section, select the db_datareader and db_datawriter roles. Click OK.
5. Enable remote access to SQL Server (see also):
  1. Configure SQL Server:
    1. Start SQL Server Configuration Manager (Start → All Programs → Microsoft SQL Server 2008 → Configuration Tools → SQL Server Configuration Manager).
    2. In the left-hand pane, select the SQL Server Configuration manager (Local) → SQL Server Network Configuration → Protocols for Database Instance Name node.
    3. In the right pane, double click the TCP/IP Protocol Name.
    4. In the Protocol tab, set Enabled to Yes. Click OK.
    5. In the left-hand pane, go to the SQL Server Configuration Manager (Local) → SQL Server Services node.
    6. In the right pane, double-click SQL Server Browser.
    7. In the Service tab, set the Start Mode property to Automatic. Click OK.
    8. Right-click SQL Server Browser, and click Start.
    9. Right-click SQL Server, and click Restart.
  2. Configure Windows Firewall
    1. Within Windows Control Panel, open the Windows Firewall applet.
    2. Click the Allow a program through Windows Firewall link.
    3. In the Windows Firewall Settings dialog, click the Exceptions tab.
    4. Click Add program..., and add the following programs:
      - sqlbrowser.exe (C:\Program Files\Microsoft SQL Server\90\Shared\sqlbrowser.exe)
      - sqlservr.exe (C:\Program Files\Microsoft SQL Server\MSSQL10.SQLEXPRESS\MSSQL\Binn\sqlservr.exe)
    5. Click OK.
Back on the Linux side of things, edit $HOME/tmp/NerdDinner 1.0/NerdDinner/ConnectionStrings.config, and change the NerdDinnerConnectionString connection string to:
```
<add name="NerdDinnerConnectionString"
    connectionString="Data Source=gourry\SQLEXPRESS;Initial Catalog=NerdDinner;User ID=jonp;Password=123456;"/>
```
You will need to adjust the machine name in the Data Source parameter to contain your actual computer name, and change the User ID and Password to whatever values you chose in §5.E.iv.

NerdDinner makes use of ASP.NET's MembershipProvider functionality, so a SQLite database needs to be created to contain the username and password information for the NerdDinner site. This is detailed at the ASP.NET FAQ and Guide: Porting ASP.NET Applications pages:

$ cd "$HOME/tmp/NerdDinner 1.0/NerdDinner/App_Data

# Create the commands needed to configure the SQLite database:
$ cat > aspnetdb.sql <<EOF
CREATE TABLE Users (
 pId                                     character(36)           NOT NULL,
 Username                                character varying(255)  NOT NULL,
 ApplicationName                         character varying(255)  NOT NULL,
 Email                                   character varying(128)  NOT NULL,
 Comment                                 character varying(128)  NULL,
 Password                                character varying(255)  NOT NULL,
 PasswordQuestion                        character varying(255)  NULL,
 PasswordAnswer                          character varying(255)  NULL,
 IsApproved                              boolean                 NULL, 
 LastActivityDate                        timestamptz             NULL,
 LastLoginDate                           timestamptz             NULL,
 LastPasswordChangedDate                 timestamptz             NULL,
 CreationDate                            timestamptz             NULL, 
 IsOnLine                                boolean                 NULL,
 IsLockedOut                             boolean                 NULL,
 LastLockedOutDate                       timestamptz             NULL,
 FailedPasswordAttemptCount              integer                 NULL,
 FailedPasswordAttemptWindowStart        timestamptz             NULL,
 FailedPasswordAnswerAttemptCount        integer                 NULL,
 FailedPasswordAnswerAttemptWindowStart  timestamptz             NULL,
 CONSTRAINT users_pkey PRIMARY KEY (pId),
 CONSTRAINT users_username_application_unique UNIQUE (Username, ApplicationName)
);

CREATE INDEX users_email_index ON Users (Email);
CREATE INDEX users_islockedout_index ON Users (IsLockedOut);

CREATE TABLE Roles (
 Rolename                                character varying(255)  NOT NULL,
 ApplicationName                         character varying(255)  NOT NULL,
 CONSTRAINT roles_pkey PRIMARY KEY (Rolename, ApplicationName)
);

CREATE TABLE UsersInRoles (
 Username                                character varying(255)  NOT NULL,
 Rolename                                character varying(255)  NOT NULL,
 ApplicationName                         character varying(255)  NOT NULL,
 CONSTRAINT usersinroles_pkey PRIMARY KEY (Username, Rolename, ApplicationName),
 CONSTRAINT usersinroles_username_fkey FOREIGN KEY (Username, ApplicationName) REFERENCES Users (Username, ApplicationName) ON DELETE CASCADE,
 CONSTRAINT usersinroles_rolename_fkey FOREIGN KEY (Rolename, ApplicationName) REFERENCES Roles (Rolename, ApplicationName) ON DELETE CASCADE
);

CREATE TABLE Profiles (
 pId                                     character(36)           NOT NULL,
 Username                                character varying(255)  NOT NULL,
 ApplicationName                         character varying(255)  NOT NULL,
 IsAnonymous                             boolean                 NULL,
 LastActivityDate                        timestamptz             NULL,
 LastUpdatedDate                         timestamptz             NULL,
 CONSTRAINT profiles_pkey PRIMARY KEY (pId),
 CONSTRAINT profiles_username_application_unique UNIQUE (Username, ApplicationName),
 CONSTRAINT profiles_username_fkey FOREIGN KEY (Username, ApplicationName) REFERENCES Users (Username, ApplicationName) ON DELETE CASCADE
);

CREATE INDEX profiles_isanonymous_index ON Profiles (IsAnonymous);

CREATE TABLE ProfileData (
 pId                                     character(36)           NOT NULL,
 Profile                                 character(36)           NOT NULL,
 Name                                    character varying(255)  NOT NULL,
 ValueString                             text                    NULL,
 ValueBinary                             bytea                   NULL,
 CONSTRAINT profiledata_pkey PRIMARY KEY (pId),
 CONSTRAINT profiledata_profile_name_unique UNIQUE (Profile, Name),
 CONSTRAINT profiledata_profile_fkey FOREIGN KEY (Profile) REFERENCES Profiles (pId) ON DELETE CASCADE
);
EOF

# Create the SQLite database:
$ sqlite3 aspnetdb.sqlite
sqlite> .read aspnetdb.sql
sqlite> .quit

Run the web app:
```
$ MONO_IOMAP=all xsp2
```
The MONO_IOMAP environment variable is needed because some link targets used within NerdDinner require case insensitivity.

Where are all the fuel efficient cars?

jonpryor@vt.edu (Jonathan Pryor) — Sun, 12 Apr 2009 15:15:00 -0500

With my Verizon TV service I get BBC America, which includes the wonderful show Top Gear.

A few weeks ago I saw their endurance race to Blackpool, a 750 mile trip from Basel, Switzerland to Blackpool, UK. (Which is odd, as Google Maps implies that the trip would be 1319km, or 819.5 miles.)

Jeremy Clarkson chose a Jaguar XJ6 TDvi (sorry, no direct link), which gets 32.3mpg, or 8.7 l/100km.

James May chose a Subaru Legacy Diesel (click Economy, then 2.0D R for the mileage), which gets 56.6 mpg, or 5.0 l/100km.

Richard Hammond chose a Volkswagen Polo Bluemotion, which was mentioned as getting 74mpg (though the above site lists 88.3 mpg, or 3.2 l/100km).

Unfortunately, these mileages are using UK gallons, which are larger than US gallons. So, using a handy online calculator, we see that the Jaguar gets ~27mpg US, the Subaru gets ~47mpg US, and the VW gets ~73.5mpg US.

Are there any equivalents to these vehicles in the USA?

Jaguar lists 25 mpg for some models (aside: the US site is far more link friendly than the UK site), which is comparable to the UK Jaguar, so it's covered.

Subaru doesn't offer a diesel engine, so nothing is comparable to the 47mpg that the UK Subaru Legacy gets.

For Volkswagan, the nearest US equivalent appears to be the Jetta TDI, which gets 41mpg, a far cry from the 73.5 of the Bluemotion.

Thus, the question: Why don't we have these cars in USA?

Mono 2.4 and mdoc-update

jonpryor@vt.edu (Jonathan Pryor) — Tue, 31 Mar 2009 17:13:00 -0500

Mono 2.4 was released, and among the unlisted changes was that mdoc-update has migrated from using Reflection to using Mono.Cecil.

There are multiple advantages and disadvantages to this migration. The disadvantages include slower execution (when I tested, Mono.Cecil took ~10% longer to do the same task as Reflection) and increased dependencies (Mono.Cecil is now required).

I believe that these disadvantages are outweighed by the advantages. Firstly, the migration makes my life significantly easier. One of the major limitations of Reflection is that only one mscorlib.dll can be loaded into a process. This means that, in order to support generating documentation from mscorlib.dll 1.0, there needs to be a version of mdoc-update that runs under .NET 1.0. Similarly, to document mscorlib.dll 2.0, I need a different version of mdoc-update which runs under .NET 2.0. And when .NET 4.0 is released (with yet another version of mscorlib.dll), I'll need...yet another version of mdoc-update to run under .NET 4.0. This is less than ideal, and using Mono.Cecil allows me to have one program which supports every version of mscorlib.dll.

This also means that I can use C# 3.0 features within mdoc-update, as I no longer need to ensure that (most of) mdoc-update can run under the .NET 1.0 profile.

Most people won't care about making my life easier, but I do. ;-)

For everyone else, the most important result of the Mono.Cecil migration is that mdoc-update now has a suitable base for advanced documentation generation scenarios which make use of IL analysis. The first feature making use of it is new --exceptions functionality, which analyzes member IL to determine which exceptions could be generated, and creates stub <exception/> XML documentation based on that analysis. This feature is experimental (see the documentation), and contains a number of corner cases, but I've already found it useful for writing Mono.Rocks documentation.

DbLinq and Mono

jonpryor@vt.edu (Jonathan Pryor) — Fri, 13 Mar 2009 00:51:00 -0500

.NET 3.5 introduced Language Integrated Query (LINQ), which allowed for querying groupings of data across diverse "paradigms" -- collections (arrays, lists, etc.), XML, and relational data, called LINQ to SQL. LINQ to SQL support is within the System.Data.Linq assembly, which is one of the assemblies Mono is currently implementing.

However, LINQ to SQL has one limitation: it only works with Microsoft SQL Server and Microsoft SQL Server Compact Edition, leaving numerous other databases users unable to use this assembly.

Enter DbLinq, an effort to provide LINQ to SQL functionality for other databases, including Firebird, Ingres, MySQL, Oracle, PostgreSql, SQLite, and SQL Server. DbLinq provides a System.Data.Linq-compatible implementation for these databases (compatible implying the same types and methods, but located within a different namespace).

Which brings us to Mono. Mono is using DbLinq as the foundation for Mono's System.Data.Linq.dll implementation, allowing Mono's System.Data.Linq.dll to support all the databases that DbLinq supports. Mono also has sqlmetal (based on DbLinq's DbMetal.exe sources), which can be used to generate C# types to interact with databases.

DbLinq On Mono

MonoDevelop can load the DbLinq solutions. However, it has a problem with building all of the projects within the solution. At the time of this writing, MonoDevelop can build the following assemblies: DbLinq.dll, DbLinq.Sqlite_test_mono_strict.dll, DbLinq.SqlServer.dll, DbLinq.SqlServer_test.dll, DbLinq.SqlServer_test_ndb.dll, DbLinq.SqlServer_test_strict.dll, and DbLinq_test_ndb_strict.dll. The *_test* assemblies are unit tests, so this leaves the core DbLinq.dll assembly and SQL Server support.

Thus, DbLinq is usually built with Visual Studio.NET (the free Visual Studio Express can be used). Once built, you can run some of the unit tests under Mono:

cd $path_to_dblinq2007_checkout/build.dbg

# Core tests
$ for test in DbLinq_test.dll DbLinq_test_ndb_strict.dll DbMetal_test.dll ; do \
nunit-console2 $test \
done
# Verbose output omitted

# SQLite tests
$ nunit-console2 DbLinq.Sqlite_test_mono.dll
# Verbose output omitted

# Plus many tests for the other providers...

Most of the tests require an accessible database, so I've been limiting my current tests to SQLite (as setup is easier).

DbLinq In Mono

As mentioned before, DbLinq is being used to implement Mono's System.Data.Linq.dll. (For those reading the DbLinq source, the Mono-specific bits are within MONO_STRICT conditional code.) This allows us to write code that depends only on .NET assemblies (though this is of dubious value, as the mechanisms used to support SQLite and other databases won't work with .NET proper, but it's still a cute trick).

To play along, you'll need Mono trunk.

Grab a SQLite database file to use with LINQ to SQL:
wget http://dblinq2007.googlecode.com/svn/trunk/src/Northwind.db3
Use sqlmetal to generate C# bindings for the database:
sqlmetal /namespace:nwind /provider:Sqlite "/conn:Data Source=Northwind.db3" /code:nwind.cs

Write some code to interact with the generated source code:

// File: nwind-app.cs
// Compile as: 
//    gmcs nwind-app.cs nwind.cs -r:System.Data \
//    	-r:System.Data.Linq -r:Mono.Data.Sqlite
using System;
using System.Data.Linq;
using System.Linq;

using Mono.Data.Sqlite;

using nwind;

class Test {
    public static void Main ()
    {
        var conn = new SqliteConnection (
                "DbLinqProvider=Sqlite;" + 
                "Data Source=Northwind.db3"
        );
        Main db = new Main (conn);
        var pens =
            from p in db.Products 
            where p.ProductName == "Pen"
            select p;
        foreach (var pen in pens) {
            Console.WriteLine ("     CategoryID: {0}",  pen.CategoryID);
            Console.WriteLine ("   Discontinued: {0}",  pen.Discontinued);
            Console.WriteLine ("      ProductID: {0}",  pen.ProductID);
            Console.WriteLine ("    ProductName: {0}",  pen.ProductName);
            Console.WriteLine ("QuantityPerUnit: {0}",  pen.QuantityPerUnit);
            Console.WriteLine ("   ReorderLevel: {0}",  pen.ReorderLevel);
            Console.WriteLine ("     SupplierID: {0}",  pen.SupplierID);
            Console.WriteLine ("      UnitPrice: {0}",  pen.UnitPrice);
            Console.WriteLine ("   UnitsInStock: {0}",  pen.UnitsInStock);
            Console.WriteLine ("   UnitsOnOrder: {0}",  pen.UnitsOnOrder);
        }
    }
}

Compile:
gmcs nwind-app.cs nwind.cs -r:System.Data -r:System.Data.Linq -r:Mono.Data.Sqlite

Run:

$ mono nwind-app.exe 
     CategoryID: 
   Discontinued: False
      ProductID: 1
    ProductName: Pen
QuantityPerUnit: 10
   ReorderLevel: 
     SupplierID: 1
      UnitPrice: 
   UnitsInStock: 12
   UnitsOnOrder: 2

Notice that we use the database connection string to specify the database vendor to use, specifically the DbLinqProvider value specifies the database vendor, and must be present when connecting to a database other than Microsoft SQL Server (which is the default vendor).

If using the DataContext(string) constructor directly (and not through a generated subclass as used above), you should also provide the DbLinqConnectionType parameter, which is the assembly-qualified type name to use for the IDbConnection implementation. This allows you to use multiple different IDbConnection implementations that use similar SQL implementations, e.g. Mono.Data.Sqlite.dll and System.Data.SQLite.dll, both of which wrap the SQLite database.

Extension Method Documentation

jonpryor@vt.edu (Jonathan Pryor) — Sun, 25 Jan 2009 21:16:00 -0500

C# 3.0 adds a new language feature called extension methods. Extension methods allow the "addition" of new instance methods to any type, without modifying the type itself. This is extremely powerful, arguably crack-adled, and exists because Visual Studio users can't do anything without code completion (tongue firmly in cheek).

It's also extremely useful, permitting LINQ and the even more crack-adled thinking in Mono.Rocks (much of which I wrote, and I'm not entirely sure if the "crack" is in jest or not; sometimes I wonder...).

To create an extension method, you first create a static class. A method within the static class is an extension method if the first parameter's type has a this modifier:

static class MyExtensions {
    public static string Implode (this IEnumerable<string> self, string separator)
    {
        return string.Join (separator, self.ToArray ());
    }
}

Usage is as if it were a normal instance method:

string[] a        = {"This", "is", "my", "sentence."};
string   imploded = a.Implode (" ");
// imploded == "This is my sentence."

Extension methods are entirely syntactic sugar. (Nice syntactic sugar, nonetheless...). As such, it doesn't in any way modify the type that is being extended. Consequently, it cannot access private members, nor is the extension method returned when reflecting over the extended type. For example, typeof(string[]).GetMethod("Implode") will return null, as System.Array doesn't have an Implode method.

Furthermore, extension methods are only available if you have a using declaration for the namespace the extension method type resides in. So if the above MyExtensions type resides in the Example namespace, and a source file doesn't have using Example;, then the Implode extension method isn't available.

Earlier I alluded that Visual Studio users can't do anything without code completion. Extension methods are thus a boon, as they (potentially) make it easier to find new functionality, as no new types need to be introduced or known about in advance. However, you still need to have an appropriate using declaration to bring the methods "in scope," so how does a developer know what namespaces to use? The same way a developer knows which type to use for anything: documentation.

MSDN online documentation has been enhanced to show which extension methods are applicable for a given type, e.g. The extension methods for IEnumerable<T>. Mono has similar documentation support.

This isn't particularly interesting, though. Part of the utility and flexibility is that any type, in any namespace, can be extended with extension methods, and the extension methods themselves can be contained in any type.

Obviously, MSDN and Mono documentation online can't know about extension methods that are not part of the core framework. Thus, if the e.g. Mono.Cecil or Gendarme frameworks provided extension methods, the online documentation sites won't be helpful.

Which brings us to a Mono 2.0 feature (yes, I'm only now announcing a feature that shipped 3 months ago):

Mono Documentation Tools: the Mono Documentation framework has been upgraded to support documenting generics and extension methods.

This support consists of four things:

Enhancing mdoc update to generate an /Overview/ExtensionMethods element within index.xml. The /Overview/ExtensionMethods element contains <ExtensionMethod/> elements which in turn contains //Targets/Target elements specifying which types the extension method is an instance method on, and a <Member/> element which is a subset of the actual extension method documentation. Developers don't need to edit this copy; it's handled entirely by mdoc update.
Enhancing mdoc assemble to look for the //ExtensionMethod elements and insert them into the ExtensionMethods.xml file within the generated .zip file.
Enhancing the XML documentation to HTML generation process so that the extension methods are listed. This allows all of monodoc and mod, online documentation, and mdoc export-html to use the underlying infrastructure.
Enhance monodoc.dll to load all ExtensionMethods.xml files from all installed .zip files. This the allows monodoc and online documentation mechanisms to show extension methods for all installed documentation sources.

The short of it is that this requires no workflow change to get extension methods listed on all extended types. Just create extension methods, document them as if they were normal static methods (as they are normal static methods, and can be invoked as such), assemble the documentation, and install the documentation.

There is one wrinkle, though: since the index.xml file contains a subset of the <Member/> documentation, you need to rerun mdoc update after editing extension method documentation so that index.xml will have the correct documentation when mdoc assemble is run. Otherwise the "summary" extension method documentation may differ from the actual intended documentation. This may be improved in a future release.

How To Defend Against Software Patent FUD

jonpryor@vt.edu (Jonathan Pryor) — Tue, 20 Jan 2009 03:24:00 -0500

You don't.

Bwa-ha-ha-ha-ha-ha-ha¹⁰⁰⁰.

Context: for years, Mono has been the target of FUD because of potential software patent issues. For years the Mono community has attempted to defend from these attack, sometimes successfully.

Recently, someone asked on mono-list about ways to pre-emptively answer the FUD so that it would become a non-issue. I responded, and had several people suggest that I blog it. Here we go.

To begin, there are several problems with defending against software patent FUD, starting with software patents themselves:

Software patents suck.
Software patents really suck. (Specifically, The "Don't Look" Problem section.)
Software patents really, really suck. (Related)
The anti-Mono FUDsters apparently can't see the forest for the trees.

I imagine that most people reading this will agree with the first three points, so it is the fourth that I will attempt to focus on.

Specifically, the anti-Mono FUDsters seem to spend so much time on a tree (Microsoft) that they either miss or minimize the forest of actual patent problems, patent trolls, etc.

So for once, I'll (non-seriously) throw the FUD:

A long time ago, Wang created a patent that "covered a method by which a program can get help from another computer application to complete a task." Microsoft licensed the patent from Wang. Sun did not. In 1997, Eastman Kodak Company bought Wang, thus acquiring this patent. Kodak then sued Sun, claiming that Java infringed this patent. Kodak won, and they later settled out of court.

Now, for my non-serious steaming pile of FUD, in the form of a question: Did Sun acquire the ability to sublicense these patents from Kodak? If Sun can sublicense the patents, then GPL'd Java is fine. If Sun can't, then Java cannot be GPL'd, and any company making use of Java could be subject to a lawsuit from Kodak.

(I would hope that this is yes, but I have no idea, and the lack of patent sub-licensing has come up before.)

So do we need to worry about Java? I have no idea. I mention it to raise a larger point:

It Doesn't Matter. Anyone can hold a patent, for anything, and sue anyone at any time. Thus, Gnome is not free of patent issues, KDE is not free of patent issues, Linux is not free of patent issues, Python is not free of patent issues, Ruby is not free of patent issues.... Nothing is free of patent issues.

(Consider: do you think that the Python Software Foundation has signed a patent license with Kodak? Has Red Hat? I doubt it. Furthermore, I find it hard to believe that something as flexible as Python wouldn't violate the aforementioned Wang patent, especially when you get into COM interop/etc. on Windows...)

Having said the above, a related question becomes: How do you avoid violating someone's patents? You don't (insert more laughter). You could try restricting yourself to only using software that's at least 20 years old, but you won't gain many users that way. It also won't work, for at least two reasons: (1) submarine patents -- not all patents that would have been in effect 20 years ago have necessarily expired (though submarine patents shouldn't exist for ~too much longer); and (2) look at the drug patent industry, where to prevent patented drugs from "going generic" the drug companies take the patent-expired drug(s), combine them with other drugs, then patent the result. I don't think it will take too long for Software companies to start doing this if they feel that it's necessary, and once they do, even using known-patent-expired programs won't be safe, as merely combining them together may be covered by an unexpired patent. Yay.

The only other way to avoid software patents is to perform a patent search, which is extremely tricky (as software patents are deliberately vague), and if you miss a patent and get sued over it, you're now liable for treble damages. You're almost always better to not look at software patents. (Isn't it funny how something that was supposed to "promote the Progress of Science and useful Arts" can't be used by those it's supposed to help? Isn't it hilarious?)

With all this in mind, you can see why patent FUD is hard to fight, because there's no way to dismiss it. Software patents are a reality, they're ugly, but they can't be avoided. (Yet they must be ignored, to avoid increased liability.) My problem is that the anti-Mono people only seem to focus on patents with respect to Mono and Microsoft, ignoring the rest of the software industry. They're ignoring the (gigantic) forest so that they can pay attention to a single tree, Microsoft.

What I find even "funnier" is that Microsoft supposedly holds a number of patents in a number of areas frequently used by open-source projects, such as HTML, CSS, C++, XML, and others. So why don't we ever see any suggestions to avoid these technologies because the Big Bad Microsoft might sue?

For that matter, (again) considering how vague software patents tend to be, wouldn't many Microsoft patents on .NET stand a chance at being applicable toward Java, Python, and other projects? (Again) Why just focus on Mono?

Final note: I am a Software Engineer, not a patent lawyer. Feel free to ignore the entire rant, but I would appreciate it if a little more thought went into all the anti-Mono propaganda.

openSUSE 11.1: Where'd my hostname go?

jonpryor@vt.edu (Jonathan Pryor) — Mon, 22 Dec 2008 04:01:00 -0500

After playing with the openSUSE 11.1 beta releases and final release, I finally installed it onto my main workstation. Funny how actually using it ~full-time shows things that were previous missed...

In this case, what greeted me when I opened a shell was:

jon@linux-jcq7$

This was rather unexpected, as this wasn't the hostname I wanted. No matter, this was normal after a fresh install (and has been happening for eons). So off I go to YaST to edit the Network Settings (/sbin/yast2 lan)...

Previously (i.e. on openSUSE 10.1, 10.2, 10.3, and 11.0), I could go to the Hostname/DNS tab, to the Hostname and Domain Name section, and specify a Hostname. (Whereupon everything would break until the next reboot as Gnome didn't seem to like the hostname changing on it, but at least I had the right hostname!)

Under openSUSE 11.1, this is disabled when NetworkManager controls things. (Again, this was not the case under 11.0 and prior releases, even when using NetworkManager to control things.)

So how do we change the hostname? Perusing Control Center brought forth the Network Connections applet → Wired tab → connection name (e.g. System eth0) → Edit → IPv4 Settings tab's DHCP Client ID textbox. This was nice to find -- I'd often wondered why setting the DHCP Client Identifier within YaST Network Settings seemingly had no effect; DHCP Client ID does work -- but it had no effect during bootup (presumably because NetworkManager isn't running early enough to set the hostname), so my shell prompt was still wrong.

Similarly, the "traditional" technique of hand-editing /etc/hosts (or using the new-fangled Hostnames YaST applet) seemed to have no effect on the system name after a reboot.

So how do we really change the hostname? Edit /etc/HOSTNAME, which is a single line file containing the fully-qualified hostname to use during bootup.

Icecream & Firewalls

jonpryor@vt.edu (Jonathan Pryor) — Sat, 20 Dec 2008 04:01:00 -0500

Earlier this year, Michael Meeks described how to use icecream to speed up builds. One problem was that originally it required disabling the firewall on most systems. There was an update mentioning that setting FW_CONFIGURATIONS_EXT could be used to open up the appropriate ports in the firewall so that things would Just Work. Alas, that doesn't work for me on openSUSE 11.1.

Thus, if using the openSUSE Firewall Allowed Services configuration doesn't work (which is what setting FW_CONFIGURATIONS_EXT modifies), there is one alternate strategy to use before disabling the firewall: manually specify the scheduler system on the daemon systems within the icecream configuration file:

sudo sed -i 's/ICECREAM_SCHEDULER_HOST=""/ICECREAM_SCHEDULER_HOST="SCHEDULER"/' /etc/sysconfig/icecream

Replace SCHEDULER with the appropriate host name or IP address of your scheduler system.

Announcing NDesk.Options 0.2.1

jonpryor@vt.edu (Jonathan Pryor) — Tue, 21 Oct 2008 02:26:00 -0500

I am pleased to announce the release of NDesk.Options 0.2.1. NDesk.Options is a C# program option parser library, inspired by Perl's Getopt::Long option parser.

To download, visit the NDesk.Options web page:

http://www.ndesk.org/Options

Usage

See http://www.ndesk.org/Options and the OptionSet documentation for examples.

What's New?

There have been several minor changes since the previous 0.2.0 release:

The OptionSet base class has been changed from Collection<Option> to KeyedCollection<string, Option>, as KeyedCollection<string, Option> is conceptually closer to what OptionSet supports: one or more strings as aliases for a single Option.
OptionSet.GetOptionForName() has been deprecated in favor of using KeyedCollection.Item(string).
C# 2.0 Compatibility. The unit tests require a C# 3.0 compiler, but the actual option parser and related classes now only require a C# 2.0 compiler.
Default argument handling support. This is useful for argument runs, in which the meaning of later options depends upon a prior argument, e.g.: mdoc-assemble --format=ecma A B --format=man C (where A and B are processed with --format=ecma in effect, while C is processed with --format=man in effect).
The Option.Description property can now contain value formatting codes which are used by OptionSet.WriteOptionDescriptions().
The Option.Description property is now automatically line-wrapped within OptionSet.WriteOptionDescriptions().
ndesk-options.pc fixes for pkg-config.
Unit tests now depend on NUnit and have been split out into separate files.

Threading: Lock Nesting

jonpryor@vt.edu (Jonathan Pryor) — Wed, 28 May 2008 03:24:00 -0500

a.k.a. Why the Java 1.0 collections were rewritten...

Threading is an overly complicated subject, covered in great detail at other locations and in many books. However, there is one subject that either I haven't seen discussed too often, or somehow have managed to miss while reading the plethora of threading sources, something I'll call lock nesting depth:

lock nesting depth

The number of locks that must be acquired and held simultaneously in order to perform a given operation.

In general, the lock nesting depth should be kept as small as possible; anything else results in extra, possibly unnecessary/extraneous locks, which serve only to slow down performance for no added benefit.

First, an aside: why does threading code require locks? To maintain data invariants for data shared between threads, preventing the data from being corrupted. Note that this is not necessarily the same as producing "correct" data, as there may be internal locks to prevent internal data corruption but the resulting output may not be "correct" (in as much as it isn't the output that we want).

The prototypical example of "non-corrupting but not correct" output is when multiple threads write to the (shared) terminal:

using System;
using System.Threading;

class Test {
	public static void Main ()
	{
		Thread[] threads = new Thread[]{
			new Thread ( () => { WriteMessage ("Thread 1"); } ),
			new Thread ( () => { WriteMessage ("Thread 2"); } ),
		};
		foreach (var t in threads)
			t.Start ();
		foreach (var t in threads)
			t.Join ();
	}

	static void WriteMessage (string who)
	{
		Console.Write ("Hello from ");
		Console.Write (who);
		Console.Write ("!\n");
	}
}

Output for the above program can vary from the sensible (and desirable):

$ mono ls.exe 
Hello from Thread 2!
Hello from Thread 1!
$ mono ls.exe 
Hello from Thread 1!
Hello from Thread 2!

To the downright "corrupt":

Hello from Hello from Hello from Hello from Thread 2!
Thread 1!

(This can happen when Thread 1 is interrupted by Thread 2 before it can write out its entire message.)

Notice what's going on here: as far as the system is concerned, what we're doing is safe -- no data is corrupted, my terminal/shell/operating system/planet isn't going to go bonkers, everything is well defined. It's just that in this circumstance "well defined" doesn't match what I, as the developer/end user, desired to see: one of the first two sets of output.

The solution, as always, is to either add a a lock within WriteMessage to ensure that the output is serialized as desired:

	static object o = new object ();
	static void WriteMessage (string who)
	{
		lock (o) {
			Console.Write ("Hello from ");
			Console.Write (who);
			Console.Write ("!\n");
		}
	}

Or to instead ensure that the message can't be split up, working within the predefined semantics of the terminal:

	static void WriteMessage (string who)
	{
		string s = "Hello from " + who + "!\n";
		Console.Write (s);
	}

(Which can oddly generate duplicate messages on Mono; not sure what's up with that... More here.)

For the WriteMessage that uses locks, the lock nesting depth is 2, and this can't be readily improved (because Console.Write is static, and thus must be thread safe as any thread could execute it at any time).

Returning to this entry's subtitle, why were the Java 1.0 collections rewritten? Because they were all internally thread safe. This had it's uses, should you be sharing a Hashtable or Vector between threads, but even then it was of limited usefulness, as it only protected the internal state for a single method call, not any state that may require more than one function call. Consider this illustrative code which counts the number of times a given token is encountered:

Hashtable data = new Hashtable ();
for (String token : tokens) {
    if (data.containsKey (token)) {
        Integer n = (Integer) data.get (token);
        data.put (token, new Integer (n.intValue() + 1));
    }
    else {
        data.put (token, new Integer (1));
    }
}

Yes, Hashtable is thread safe and thus won't have its data corrupted, but it can still corrupt your data should multiple threads execute this code against a shared data instance, as there is a race with the data.containsKey() call, where multiple threads may evaluate the same token "simultaneously" (read: before the following data.put call), and thus each thread would try to call data.put (token, new Integer (1)). The result: a missed token.

The solution is obvious: another lock, controlled by the developer, must be used to ensure valid data:

Object lock = new Object ();
Hashtable data = new Hashtable ();
for (String token : tokens) {
    synchronized (lock) {
        if (data.containsKey (token)) {
            Integer n = (Integer) data.get (token);
            data.put (token, new Integer (n.intValue() + 1));
        }
        else {
            data.put (token, new Integer (1));
        }
    }
}

Consequently, for all "non-trivial" code (where "non-trivial" means "requires more than one method to be called on the collection object in an atomic fashion") will require a lock nesting depth of two. Furthermore, the lock nesting depth would always be at least one, and since many functions were not invoked between multiple threads, or the collection instance local to that particular method, the synchronization within the collection was pure overhead, providing no benefit.

Which is why in Java 1.2, all of the new collection classes such as ArrayList and HashMap are explicitly unsynchronized, as are all of the .NET 1.0 and 2.0 collection types unless you use a synchronized wrapper such as System.Collections.ArrayList.Synchronized (which, again, is frequently of dubious value if you ever need to invoke more than one method against the collection atomically).

Finally, the Threading Design Guidelines of the .NET Framework Design Guidelines for Class Library Developers (book) suggests that all static members be thread safe, but instance member by default should not be thread safe:

Instance state does not need to be thread safe. By default, class libraries should not be thread safe. Adding locks to create thread-safe code decreases performance, increases lock contention, and creates the possibility for deadlock bugs to occur. In common application models, only one thread at a time executes user code, which minimizes the need for thread safety. For this reason, the .NET Framework class libraries are not thread safe by default.

Obviously, there are exceptions -- for example, if a static method returns a shared instance of some class, then all of those instance members must be thread safe as they can be accessed via the static method (System.Reflection.Assembly must be thread safe, as an instance of Assembly is returned by the static method Assembly.GetExecutingAssembly). By default, though, instance members should not be thread safe.

HackWeek Summary

jonpryor@vt.edu (Jonathan Pryor) — Tue, 19 Feb 2008 15:51:00 -0500

In case you missed it, last week was "Hackweek" at Novell.

My week was less "hacking" and more "spit-and-polish." In particular:

Wrote Mono.Unix.UnixSignal documentation. This only took ~1/2 a day.
Released NDesk.Options 0.2.0, which in turn involved the vicious cycle of write documentation (and samples), realize I'm missing something or don't like the name, change the library, update the documentation, repeat...
Consequently, something I thought would only take another half a day wound up taking 3 days, with some changes happening at the last moment (the System.Action`2 removal, thus removing the need to have two different builds depending on whether .NET 2.0 or .NET 3.0 is targeted).
Improve monodocs2html so that the output sucks...well, less. I won't say that it's actually good now, but it is an improvement. For comparison purposes, NDesk.Options.OptionSet documentation output from:
- monodocs2html 1.2.6
- monodocs2html svn
In particular, note that in the 1.2.6 output that there are three examples as part of the Remarks documentation, but it's rather difficult to tell when one example ends and the next begins (due to the lack of an Example header). The svn version fixes this, adds JavaScript bling so that all sections can be collapsed and expanded, and adds a mini "index" to the top-left so that it's easier to get to the relevant sections (such as skipping the three lengthy examples that precede the Members listing).

I had wanted to do other things as well, such as migrate the monodoc-related programs to use NDesk.Options instead of Mono.GetOptions for option parsing, but such efforts will have to wait until later...

Announcing NDesk.Options 0.2.0

jonpryor@vt.edu (Jonathan Pryor) — Thu, 14 Feb 2008 21:42:00 -0500

I am pleased to announce the release of NDesk.Options 0.2.0. NDesk.Options is a C# program option parser library, inspired by Perl's Getopt::Long option parser.

To download, visit the NDesk.Options web page:

http://www.ndesk.org/Options

Usage

See http://www.ndesk.org/Options and the OptionSet documentation for examples.

What's New?

There have been numerous changes since the previous 0.1.0 release:

Mono 1.9 is now required to build. (An svn release was previously required anyway, so this isn't a surprising requirement.)
Simplify the API by removing all OptionSet.Add() methods which provided an OptionContext to the callback function; this includes:
- OptionSet.Add(string, Action<string, OptionContext>)
- OptionSet.Add(string, string, Action<string, OptionContext>)
- OptionSet.Add<T>(string, Action<T, OptionContext>)
- OptionSet.Add<T>(string, string, Action<T, OptionContext>)
If you really need access to an OptionContext, you can Add your own Option and override Option.OnParseComplete(OptionContext).
By Miguel's request, change the semantics for Options with optional values (arguments that have a type value of `:'). Previously, Options accepting an optional value were virtually identical to Options accepting a required value; the only place they would differ is at the end of the command line, where if a value was missing for a Option with an optional value no error would occur, while an Option with a required value would generate an error.

Now, we introduce the notion of greediness: required values are greedy, and will eat any number of following arguments in order to fulfill their requirements. Optional values are not greedy, and will only extract a value from the current argument.

By way of example:
```
string color = null;
var p = new OptionSet () {
    { "-color:", v => color = v },
};
p.Parse (new string[]{"--color=auto"});     // 1
p.Parse (new string[]{"--color", "auto"});  // 2
```
In NDesk.Options 0.1.0, (1) and (2) would be identical and color would be given the value auto. In 0.2.0, they are not identical: (1) would assign the value auto to color, while (2) would assign null to color. This permits consistency with GNU ls(1)'s ls --color behavior.

If a required option were to be specified (by using = instead of :), then (1) and (2) would again have identical results.
NDesk.Options 0.1.0 restricted option bundling to boolean Options. This restriction has been relaxed so that (1) Options accepting both optional and required values may be bundled with boolean Options, and (2) the optional or required value may be bundled as well. As before, only single character Options may be bundled.
The logic is as follows: given an argument such as -cvfname:
1. cvfname must not match a registered Option. If it does match a registered option, then that is the Option that will (eventually) be invoked.
2. c must be a registered option. If it isn't, then -cvfname is returned from OptionSet.Parse(IEnumerable<string>).
3. Each character is looked up; if it's a boolean Option, then the associated action is invoked with a non-null value.
4. If instead the character is an Option accepting one or more optional or required values, then the rest of the argument (not including the Option character) is used as the value. This also follows the greediness of optional vs. required values: optional values will only use the current argument, while required values may use the following argument(s) if e.g. the Option's character is the last character in the sequence.
5. If a non-Option character is encountered that is not (a) the first character in the sequence, or (b) used as the value for a previous Option, then an OptionException is thrown.
This does The Right Thing for tar(1)-like option handling, with tar -cvfname ... creating (with verbose output) the file with the name name.
Options may now accept (or require) more than one value. The Option (string, string, int) constructor allows specifying how many values are accepted/required (depending on whether the Option has optional or required values).
The Option values are available through the OptionContext.OptionValues collection.
Direct support for Options accepting/required two values within OptionSet:
This now permits reasonable handling of cc(1)-style parameters:
```
var macros = new Dictionary<string, string> ();
var p = new OptionSet () {
    { "D:", (k, v) => { if (k != null) macros.Add (k, v); } },
};
p.Parse (new string[]{"-DNAME1", "-DNAME2=VALUE2"});
    // Adds the keys "NAME1" (with null value) 
    // and "NAME2" (with value "VALUE2") to `macros'.
```
Note that an optional value is used; if D= were specified, two values would be required, so -DNAME1 -DNAME2=VALUE2 would insert one macro -- NAME1 -- with the value -DNAME2=VALUE2.
When an Option permits more than one value, it may provide a list of value separator strings, strings that may be used to separate the multiple values. If no separators are listed, = and : are used as the default (thus permitting the previous -DNAME2=VALUE2 example to work; -DNAME2:VALUE2 would have had the same result).
The value separator strings follow the : or = in the Option prototype. They consist of:
- The string within { and }.
- Any other invidividual character.
Thus, the prototype M:+-*/ would use +, -, *, or / to split values, so -M5+2, -M5-2, -M5*2, and -M5/2 all provide two values (5 and 2) to the M option.

The prototype N={-->}{=>} would parse both -NA-->B and -NA=>B so that A and B are provided as the two values to the N option.

As a special construct, the separator {} requires that each value be a separate argument. (This makes no sense for Options with optional values.)
Naming consistency improvements: an argument is an unparsed string, an option is a parsed argument that corresponds to a registered Option, and a prototype is a description of an Option, describing all aliases, the value type, and value separators. This has resulted in method argument name changes.
Removal of .NET 3.5 support. Instead of using System.Action`2 from System.Core.dll (or providing an internal equivalent), I've just defined a OptionAction<TKey, TValue> type. This simplifies assembly versioning.

Unix Signal Handling In C#

jonpryor@vt.edu (Jonathan Pryor) — Fri, 08 Feb 2008 20:32:00 -0500

In the beginning, Unix introduced signal(2), which permits a process to respond to external "stimuli", such as a keyboard interrupt (SIGINT), floating-point error (SIGFPE), dereferencing the NULL pointer (SIGSEGV), and other asynchronous events. And lo, it was...well, acceptable, really, but there wasn't anything better, so it at least worked. (Microsoft, when faced with the same problem of allowing processes to perform some custom action upon an external stimuli, invented Structured Exception Handling.)

Then, in a wrapping binge, I exposed it for use in C# with Stdlib.signal(), so that C# code could register signal handlers to be invoked when a signal occurred.

The problem? By their very nature, signals are asynchronous, so even in a single-threaded program, you had to be very careful about what you did, as your "normal" thread was certainly in the middle of doing something. For example, calling malloc(3) was almost certainly a bad idea, because if the process was in the middle of a malloc call already, you'd have a reentrant malloc call which could corrupt the heap.

This reentrant property impacts all functions in the process, including system calls. Consequently, a list of functions that were "safe" for invocation from signal handlers was standardized, and is listed in the above signal man page; it includes functions such as read(2) and write(2), but not functions like e.g. pwrite(2).

Consequently, these limitations and a few other factors led to the general recommendation that signal handlers should be as simple as possible, such as writing to global variable which the main program occasionally polls.

What's this have to do with Stdlib.signal(), and why was it a mistake to expose it? The problem is the P/Invoke mechanism, which allows marshaling C# delegates as a function pointer that can be invoked from native code. When the function pointer is invoked, the C# delegate is eventually executed.

However, before the C# delegate can be executed, a number of of steps needs to be done first:

The first thing it does is to ensure the application domain for the thread where the signal handler executes actually matches the appdomain the delegate comes from, if it isn't it may need to set it and do several things that we can't guarantee are signal context safe...

If the delegate is of an instance method we also need to retrieve the object reference, which may require taking locks...

In the same email, lupus suggests an alternate signal handling API that would be safe to use from managed code. Later, I provided a possible implementation. It amounts to treating the UnixSignal instance as a glorified global variable, so that it can be polled to see if the signal has been generated:

UnixSignal signal = new UnixSignal (Signum.SIGINT);
while (!signal.IsSet) {
  /* normal processing */
}

There is also an API to permit blocking the current thread until the signal has been emitted (which also accepts a timeout):

UnixSignal signal = new UnixSignal (Signum.SIGINT);
// Wait for SIGINT to be generated within 5 seconds
if (signal.WaitOne (5000, false)) {
    // SIGINT generated
}

Groups of signals may also be waited on:

UnixSignal[] signals = new UnixSignal[]{
    new UnixSignal (Signum.SIGINT),
    new UnixSignal (Signum.SIGTERM),
};

// block until a SIGINT or SIGTERM signal is generated.
int which = UnixSignal.WaitAny (signals, -1);

Console.WriteLine ("Got a {0} signal!", signals [which].Signum);

This isn't as powerful as the current Stdlib.signal() mechanism, but it is safe to use, doesn't lead to potentially ill-defined or unwanted behavior, and is the best that we can readily provide for use by managed code.

Mono.Unix.UnixSignal is now in svn-HEAD and the mono-1-9 branch, and should be part of the next Mono release.