Home
RSS SyndicationJonathan Pryor's web log
HackWeek V
Last week was HackWeek V, during which I had small goals, yet had most of the time eaten by unexpected "roadblocks."
The week started with my mis-remembering OptionSet behavior. I had thought that there was a bug with passing options containing DOS paths, as I thought the path would be overly split:
string path = null;
var o = new OptionSet () {
{ "path=", v => path = v },
};
o.Parse (new[]{"-path=C:\path"});
Fortunately, my memory was wrong: this works as expected. Yay.
What fails is if the option supports multiple values:
string key = null, value = null;
var o = new OptionSet () {
{ "D=", (k, v) => {key = k; value = v;} },
};
o.Parse (new[]{"-DFOO=C:\path"});
The above fails with a OptionException, because the DOS path is split, so OptionSet attempts to send 3 arguments to an option expecting 2 arguments. This isn't allowed.
The patch to fix the above is trivial (most of that patch is for tests). However, the fix didn't work at first.
Enter roadblock #1: String.Split() can return too many substrings. Oops.
So I fixed it. That only killed a day...
Next up, I had been sent an email showing that OptionSet had some bugs when removing by index. I couldn't let that happen...and being in a TDD mood, I first wrote some unit tests to describe what the IList<T> semantics should be. Being in an over-engineering mood, I wrote a set of "contract" tests for IList<T> in Cadenza, fixed some Cadenza bugs so that Cadenza would pass the new ListContract, then merged ListContract with the OptionSet tests.
Then I hit roadblock #2 when KeyedCollection<TKey, TItem> wouldn't pass my ListContract tests, as it wasn't exception safe. Not willing to give up on ListContract, I fixed KeyedCollection so it would now pass my ListContract tests, improving compatibility with .NET in the process, which allowed me to finally fix the OptionSet bugs.
I was then able to fix a mdoc export-html bug in which index files wouldn't always be updated, before starting to investigate mdoc assemble wanting gobs of memory.
While pondering how to figure out why mdoc assemble wanted 400MB of memory, I asked the folks on ##csharp on freenode if there were any Mono bugs preventing their SpikeLite bot from working under Mono. They kindly directed me toward a bug in which AppDomain.ProcessExit was being fired at the wrong time. This proved easier than I feared (I feared it would be beyond me).
Which left me with pondering a memory "leak." It obviously couldn't be a leak with a GC and no unmanaged memory to speak of, but what was causing so much memory to be used? Thus proceeded lots of Console.WriteLine(GC.GetTotalMemory(false)) calls and reading the output to see where the memory use was jumping (as, alas I found Mono's memory profiler to be less than useful for me, and mono's profiler was far slower than a normal run). This eventually directed me to the problem:
I needed, at most, two XmlNode values from an XmlDocument. An XmlDocument loaded from a file that could be very small or large-ish (0.5MB). Thousands of such files. At once.
That's when it dawned on me that storing XmlNodes in a Dictionary loaded from thousands of XmlDocuments might not be such a good idea, as each XmlNode retains a reference to the XmlDocument it came from, so I was basically copying the entire documentation set into memory, when I only needed a fraction of it. Doh!
The fix was straightforward: keep a temporary XmlDocument around and call XmlDocument.ImportNode to preserve just the data I needed.
Memory use plummeted to less than one tenth what was previously required.
Along the way I ran across and reported an xbuild bug (since fixed), and filed a regression in gmcs which prevented Cadenza from building.
Overall, a productive week, but not at all what I had originally intended.
Defending XML-based Build Systems
Justin Etheredge recently suggested that we Say Goodbye to NAnt and MSBuild for .NET Builds With IronRuby. Why? because they're based on XML.
He goes on to mention several problems with XML-based build systems, principally:
- It's not code (unless you're using XSLT, and having maintained XSLTs if you need to write them you have my condolences...)
- Lose existing tooling: editors, debuggers, libraries, etc.
- Limit creation of custom rules.
- Require that we write custom tools to workaround the limitations of XML build systems.
His solution: use Ruby to describe your build process.
My reaction? No, no, for the love of $deity NO!
Why? Three reasons: GNU Autotools, Paul E. McKenney's excellent parallel programming series, and SQL.
Wait, what? What do those have to do with build systems? Everything, and nothing.
The truly fundamental problem is this: "To a man with a hammer, everything looks like a nail" (reportedly a quote from Mark Twain, but that's neither here nor there). In this case, the "hammer" is "writing code." But it's more than that: it's writing imperative code, specifically Ruby code (though the particular language isn't the problem I have, rather the imperative aspect).
Which is, to me, the fundamental problem: it's a terrible base for any form of higher-level functionality. Suppose you want to build your software in parallel (which is where Paul McKenney's series comes in). Well, you can't, because your entire build system is based on imperative code, and unless all the libraries you're using were written with that in mind...well, you're screwed. The imperative code needs to run, and potentially generate any side effects, and without a "higher-level" description of what those side effects entail it can't sanely work.
Want to add a new file to your build (a fairly common thing to do in an IDE, along with renaming files?) Your IDE needs to be able to understand the imperative code. If it doesn't, it just broke your build script. Fun!
OK, what about packaging? Well, in order to know what the generated files are (and where they're located), you'll have to run the entire script and (somehow) track what files were created.
Want to write an external tool that does something hitherto unknown? (As a terrible example, parse all C# code for #if HAVE_XXX blocks so that a set of feature tests can be automatically extracted.) Well, tough -- you have to embed an IronRuby interpreter, and figure out how to query the interpreter for the information you want (e.g. all the source files).
etc., etc.
My problem with imperative languages is that they're not high-level enough. McKenney asks what the good multicore programming languages are; the answer is SQL because it's dedicated ~solely to letting you describe the question but leaves the implementation of the answer to the question up to the SQL database. It's not imperative, it's declarative (at least until you hit esoteric features such as cursors, but in principal you can generally stick to a declarative subset).
OK, so I want a higher-level language to describe targets and dependencies, and supports faster builds. To a large degree, make(1) supports all that, and it's the basis of Autotools. Surely I like that, right?
The problem with autotools is that it's a mixture of declarative and imperative code, with Unix shell scripts forming the backbone of the imperative code (aka the target rules), and these are inherently Unix specific. (Possibly Linux specific, much to my consternation.) Plus, the format is virtually unreadable by anything other than make(1), what with all the language extensions...
So why XML?
Because it's not code, it's data, which (somewhat) lowers the barrier of entry for writing external tools which can parse the format and Do New Things without needing to support some language which might not even run on the platform you're using.
Because it's easily parseable AND verifiable, it's (somewhat) safer for external automated tools to manipulate the file without screwing you over "accidentally" -- e.g. adding and removing files from the build via an IDE.
Because custom rules are limited, there is a smaller "grammar" for external tools to understand, making it simpler to write and maintain them. It also encourages moving "non-target targets" out of the build system, simplifying file contents (and facilitating interaction with e.g. IDEs).
Am I arguing that XML-based build systems are perfect? Far from it. I'm instead arguing that small, purpose-specific languages can (and often are) Good Things™, particularly if they permit interoperability between a variety of tools and people. XML allows this, if imperfectly. An IronRuby-based build system does not.
mdoc Repository Format History
Time to wrap up this overly long series on mdoc. We covered:
- Introduction
- Overview
- Simple usage
- Importing documentation
- Repository XML Schema
- Static HTML customization
- Exporting to Microsoft's XML Documentation format
- Assembling documentation
- Viewing documentation via ASP.NET
- Caching ASP.NET content
- Assembly versioning
To close out this series, where did the mdoc repository format come from? It mostly came from Microsoft, actually.
Taking a step back, "in the beginning," as it were, the Mono project saw the need for documentation in January 2002. I wasn't involved then, but perusing the archives we can see that csc /doc output was discarded early because it wouldn't support translation into multiple languages. NDoc was similarly discarded because it relied on csc /doc documentation. I'm sure a related problem at the time was that Mono's C# compiler didn't support the /doc compiler option (and wouldn't begin to support /doc until April 2004), so there would be no mechanism to extract any inline documentation anyway.
By April 2003 ECMA standardization of the Common Language Infrastructure was apparently in full force, and the standardization effort included actual class library documentation. The ECMA documentation is available within ECMA-335.zip. The ECMA-335 documentation also included a DTD for the documentation contained therein, and it was a superset of the normal C# XML documentation. The additional XML elements provided what XML documentation lacked: information available from the assembly, such as actual parameter types, return types, base class types, etc. There was one problem with ECMA-335 XML, though: it was gigantic, throwing everything into a single 7MB+ XML file.
To make this format more version-control friendly (can you imagine maintaining and viewing diffs on a 7+MB XML file?), Mono "extended" the ECMA-335 documentation format by splitting it into one file per type. This forms the fundamental basis of the mdoc repository format (and is why I say that the repository format came from Microsoft, as Microsoft provided the documentation XML and DTD to ECMA). This is also why tools such as mdoc assemble refer to the format as ecma. The remainder of the Mono extensions were added in order to fix various documentation bugs (e.g. to distinguish between ref vs. out parameters, to better support generics), etc.
In closing this series, I would like to thank everyone who has ever worked on Monodoc and the surrounding tools and infrastructure. It wouldn't be anywhere near as useful without them.
Assembly Versioning with mdoc
Previously, we mentioned as an aside that the Type.xml files within an mdoc repository contained //AssemblyVersion elements. Today we will discuss what they're for.
The //AssemblyVersion element records exactly one thing: which assembly versions a type and member was found in. (The assembly version is specified via the AssemblyVersionAttribute attribute.) With a normal assembly versioning policy, this allows monodoc to show two things: which version added the type/member, and (by inference) which version(s) removed the member.
For example, consider the NetworkStream.Close method. This method was present in .NET 1.0 which overrode Stream.Close. However, in .NET 2.0 the override was removed entirely.
The //AssemblyVersion attribute allows the mdoc repository to track such versioning changes; for example, consider the mdoc-generated NetworkStream.xml file. The //Member[@MemberName='Close']/AssemblyInfo/AssemblyVersion elements contain only an entry for 1.0.5000.0 (corresponding to .NET 1.1) on line 536. Compare to the //Member[@MemberName='CanWrite']/AssemblyInfo/AssemblyVersion elements (for the NetworkStream.CanWrite property) which has //AssemblyVersion elements for 1.0.5000.0 and 2.0.0.0. From this, we can deduce that NetworkStream.Close was present in .NET 1.1, but was removed in .NET 2.0.
When viewing type and member documentation, monodoc and the ASP.NET front end will show the assembly versions that have the member:
There are two limitations with the version tracking:
- It only tracks types and members. For example, attributes, base classes, and interfaces may be added or removed across versions; these are not currently tracked.
- It uses the assembly version to fill the <AssemblyVersion> element.
The second point may sound like a feature (isn't it the point?), but it has one downfall: auto-generated assembly versions. You can specify an auto-generated assembly version by using the * for some components in the AssemblyVersionAttribute constructor:
[assembly: AssemblyVersion("1.0.*.*")]
If you do this, every time you rebuild the assembly the compiler will dutifully generate a different assembly number. For example, the first time you might get a compiler version of 1.0.3666.19295, while the second recompilation the compiler will generate 1.0.3666.19375. Since mdoc assigns no meaning to the version numbers, it will create //AssemblyVersion elements for each distinct version found.
The "advantage" is that you can know on which build a member was added. (If you actually care...)
The disadvantage is a major bloating of the mdoc repository, as you add at least 52*(1+M) bytes to each file in the mdoc repository for each unique assembly version (where M is the number of members within the file, as each member is separately tracked). It will also make viewing the documentation distracting; imagine seeing 10 different version numbers for a member, which all differ in the build number. That much noise would make the feature ~useless.
As such, if you're going to use mdoc, I highly suggest not using auto-generated assembly version numbers.
Next time, we'll wrap up this series with a history of the mdoc repository format.
Caching mdoc's ASP.NET-generated HTML
Last time we discussed configuring the ASP.NET front-end to display monodoc documentation. The display of extension methods within monodoc and the ASP.NET front-end is fully dynamic. This has it's pros and cons.
On the pro side, if/when you install additional assembled documentatation sources, those sources will be searched for extension methods and they will be shown on all matching types. This is very cool.
On the con side, searching for the extension methods and converting them into HTML takes time -- there is a noticable delay when viewing all members of a type if there are lots of extension methods. On heavily loaded servers, this may be detrimental to overall performance.
If you're running the ASP.NET front-end, you're not regularly adding documentation, and you have Mono 2.6, you can use the mdoc export-html-webdoc command to pre-render the HTML files and cache the results. This will speed up future rendering.
For example, consider the url http://localhost:8080/index.aspx?link=T:System.Collections.Generic.List`1/* (which shows all of the List<T> members). This is a frameset, and the important frame here is http://localhost:8080/monodoc.ashx?link=T:System.Collections.Generic.List`1/* which contains the member listing (which includes extension methods). On my machine, it takes ~2.0s to download this page:
$ time curl -s \ 'http://localhost:8080/monodoc.ashx?link=T:System.Collections.Generic.List`1/*' \ > /dev/null real 0m2.021s user 0m0.003s sys 0m0.002s
In a world where links need to take less than 0.1 seconds to be responsive, this is...pretty bad.
After running mdoc export-html-webdoc netdocs.zip (which contains the List<T> docs):
$ time curl -s \ 'http://localhost:8080/monodoc.ashx?link=T:System.Collections.Generic.List`1/*' \ > /dev/null real 0m0.051s user 0m0.003s sys 0m0.006s
That's nearly 40x faster, and within the 0.1s guideline.
Cache Generation: to generate the cache files, run mdoc export-html-web ASSEMBLED-FILES. ASSEMBLED-FILES consists of the .tree or .zip files which are generated by mdoc assemble and have been installed into $prefix/lib/monodoc/sources:
$ mdoc export-html-webdoc $prefix/lib/monodoc/sources/Demo.zip
(Where $prefix is your Mono installation prefix, e.g. /usr/lib/monodoc/sources/Demo.zip.)
This will create a directory tree within $prefix/lib/monodoc/sources/cache/Demo. Restarting the ASP.NET front-end will allow it to use the cache.
If you don't want to generate the cache in another directory, use the -o=PREFIX option. This is useful if you're updating an existing cache on a live server and you don't want to overwrite/replace the existing cache (it's a live server!) -- generate the cache elsewhere, then move the files when the server is offline.
If you have lots of time on your hands, you could process all assembled documentation with:
$ mdoc export-html-webdoc $prefix/lib/monodoc/sources/*.zip
Limitations: It should be noted that this is full of limitations, so you should only use it if performance is really important. Limitations include:
- The existence of the cache subdirectories are more important than any timestamps; if the .zip file is newer than the corresponding cache directory, the cache contents will still be returned.
- It's privy to monodoc internals, and thus you may need to regenerate all cached documentation whenever you add or remove .zip files. For example, since it can be used to show extension methods, and any set of documentation can contain extension methods, adding or removing assembled documentation files may render the cached output out of date.
- mdoc export-html-webdoc processing is slow. Processing the 2.4KB Demo.zip takes a speedy 1.2s. Processing the 5.8MB netdocs.zip (51MB uncompressed, containing 4810 types with 45267 members, including List<T> documentation) takes an astounding 247m (over 4 hours). The resulting cache/netdocs directory is 316MB.
Next time, we'll cover mdoc's support for assembly versioning.
Configuring the ASP.NET front-end for mdoc
Last time, we assembled our documentation and installed it for use with monodoc. This is a prerequisite for ASP.NET support (as they both use the same system-wide documentation directory).
Once the documentation is installed (assuming a Linux distro or OSX with the relevant command-line tools installed), you can trivially host a web server which will display the documentation:
$ svn co http://anonsvn.mono-project.com/source/branches/mono-2-4/mono-tools/webdoc/ # output omitted... $ cd webdoc $ xsp2
You will need to change the svn co command to use the same version of Mono that is present on your system. For example, if you have Mono 2.6 installed, change the mono-2-4 to mono-2-6.
Once xsp2 is running, you can point your web browser to http://localhost:8080 to view documentation. This will show the same documentation as monodoc did last time:
For "real" use, setting up using Apache with mod_mono may be preferred (or any of the other options listed at Mono's ASP.NET support page). Configuring mod_mono or anything other than xsp2 is beyond my meager abilities.
Next time, we'll discuss improving the ASP.NET front-end's page rendering performance.
Assembling Documentation with mdoc
We previously discussed exporting the mdoc repository into static HTML files using mdoc export-html and into a Microsoft XML Documentation file with mdoc export-msxdoc. Today, we'll discuss exporting documentation with mdoc assemble.
mdoc assemble is used to assemble documentation for use with the monodoc Documentation browser and the ASP.NET front-end. This involves the following steps:
- Running mdoc assemble.
- Writing a .source file.
- Installing the files.
Unfortunately we're taking a diversion from the Windows world, as the monodoc browser and the ASP.NET front-end won't run under Windows (due to limitations in the monodoc infrastructure). I will attempt to fix these limitations in the future.
Running mdoc assemble: mdoc assemble has three arguments of interest:
- -f=FORMAT is used to specify the format of the files to assemble. When documenting assemblies you can skip this, as the default format is for mdoc repositories. This is useful if you want to assemble other materials, such as man pages or plain HTML files.
- -o=PREFIX is used to specify the output prefix. mdoc assemble generates two files, a .tree and a .zip file. The PREFIX value is the basename to use for these to files.
- The list of files or directories to process. Whether these need to be files or directories (and related semantics) depends upon the format specified; see the FORMATS section of the mdoc-assemble(1) man page for details.
For our current documentation, we would run:
$ mdoc assemble -o Demo Documentation/en.docs
This will create the files Demo.tree and Demo.zip in the current working directory.
The .source file is used to tell the documentation browser where in the tree the documentation should be inserted. It's an XML file that contains two things: a (set of) /monodoc///node elements describing where in the tree the documentation should be inserted, and /monodoc/source elements which specify the files to use. For example:
<?xml version="1.0"?> <monodoc> <node label="Demo Library" name="Demo-lib" parent="libraries" /> <source provider="ecma" basefile="Demo" path="Demo-lib"/> </monodoc>
The /monodoc/node element describes where in the monodoc tree the documentation should be placed. It has three attributes, two of which are required:
- label is the text to display in the tree view.
- name is the name of the node, so that other nodes and the /monodoc/source/@path attribute may refer to it.
- parent is optional, and contains the //node/@name
value of the node which should be the parent of this node.
This is used to provide a degree of structure. It should be a value from
$prefix/lib/monodoc/monodoc.xml in the //node/@name
attribute values. Currently these include:
- languages for programming language references, e.g. The C# Language Specification.
- libraries for class library documentation.
- man for man pages and other command references.
- tools and various for anything that doesn't fit in the above descriptions.
The /monodoc/source element describes what file basename to use when looking for the .tree and .zip files. (By convention the .source, .tree, and .zip files share the same basename, but this is not required. The .tree and .zip files must share the same basename, but the .source basename may differ, and will differ if e.g. one .source file pulls in several .tree/.zip pairs.) It has three attributes, all of which are required:
- basefile is the file basename of the .tree and .zip files.
- path is the //node/@name value that will be associated with the docs within basefile.tree and basefile.zip.
- provider is the format provided to mdoc assemble. For assembly documentation, this should be ecma.
Installing the files. Files need to be installed into $prefix/lib/monodoc/sources. You can obtain this directory with pkg-config(1):
$ cp Demo.source Demo.tree Demo.zip \
`pkg-config monodoc --variable=sourcesdir`
Now when we run monodoc, we can navigate to the documentation that was just installed:
Additionally, those paying attention on January 10 will have noticed that the With() method we documented is an extension method. Monodoc supports displaying extension methods on the relevant type documentation. In this case, With() is an extension on TSource, which is, for all intents and purposes, System.Object. Thus, if we view the System.Object docs within our local monodoc browser, we will see the With() extension method:
In fact, we will see With() listed as an extension method on all types (which is arguably a bug, as static types can't have instance methods...).
Furthermore, mdoc export-html will also list extension methods. However, mdoc export-html is far more limited: it will only look for extension methods within the mdoc repositories being processing, and it will only list those methods as extension methods on types within the mdoc repository. Consequently, mdoc export-html will not list e.g. IEnumerable<T> extension methods on types that implement IEnumerable<T>. (It simply lacks the information to do so.)
Examples of mdoc export-html listings of extension methods can be found in the mdoc unit tests and the Cadenza.Collections.CachedSequence<T> docs (which lists a million extension methods because Cadenza.Collections.EnumerableCoda contains a million extension methods on IEnumerable<T>).
Next time, we'll discuss setting up the ASP.NET front end under Linux.
Exporting mdoc Repositories to Microsoft XML Documentation
Previously, we discussed how to write documentation and get it into the documentation repository. We also discussed exporting the documentation into static HTML files using mdoc export-html. Today, we'll discuss mdoc export-msxdoc.
mdoc export-msxdoc is used to export the documentation within the mdoc repository into a .xml file that conforms to the same schema as csc /doc. This allows you, if you so choose, to go entirely to externally managed documentation (instead of inline XML) and still be able to produce your Assembly.xml file so that Visual Studio/etc. can provide code completion against your assembly.
There are two ways to invoke it:
$ mdoc export-msxdoc Documentation/en $ mdoc export-msxdoc -o Demo.xml Documentation/en
The primary difference between these is what files are generated. Within each Type.xml file of the mdoc repository (e.g. ObjectCoda.xml) is a /Type/AssemblyInfo/AssemblyName element.
The first command (lacking -o Demo.xml) will generate a set of .xml files, where the filenames are based on the values of the /Type/AssemblyInfo/AssemblyName element values, in this case Demo.xml. Additionally, a NamespaceSummaries.xml file is generated, containing documentation for any namespaces that were documented (which come from the ns-*.xml files, e.g. ns-Cadenza.xml).
The second command (which specifies -o Demo.xml) will only generate the specified file (in this case Demo.xml).
For this mdoc repository, there is no actual difference between the commands (as only one assembly was documented within the repository), except for the generation of the NamespaceSummaries.xml file. However, if you place documentation from multiple assemblies into the same mdoc repository, the first command will properly generate .xml files for each assembly, while the latter will generate only a single .xml file containing the documentation from all assemblies.
Next time, we'll cover mdoc assemble.
Customizing mdoc's Static HTML Output
Last time, we wrote documentation for our Demo.dll assembly. What if we want to improve the looks of those docs, e.g. to change the colors or add additional navigation links for site consistency purposes?
mdoc export-html uses three mechanisms to control output:
- --ext=FILE-EXTENSION is used to change the file extension of generated files from .html to FILE-EXTENSION. This is useful if you want to generate e.g. .aspx files instead of .html files (the default).
- --template=TEMPLATE-FILE specifies an XSLT to use for the layout of all generated files. If not specified, then the XSLT returned by mdoc export-html --default-template is used.
- HTML CSS class names are used throughout the documentation, allowing various elements to be customized by providing an alternate stylesheet. The mdoc export-html man page lists the CSS classes that are used.
The XSLT needs to consume an XML document that has the following structure:
<Page>
<CollectionTitle>Collection Title</CollectionTitle>
<PageTitle>Page Title</PageTitle>
<Summary>Page Summary</Summary>
<Signature>Type Declaration</Signature>
<Remarks>Type Remarks</Remarks>
<Members>Type Members</Members>
<Copyright>Documentation Copyright</Copyright>
</Page>
The contents of each of the //Page/* elements contains HTML or plain text nodes. Specifically:
- /Page/CollectionTitle
- Contains the Assembly and Namespace name links.
- /Page/PageTitle
- Contains the type name/description.
- /Page/Summary
- Contains the type <summary/> documentation.
- /Page/Signature
- Contains the type signature, e.g. whether it's a struct or class, implemented interfaces, etc.
- /Page/Remarks
- Contains type-level <remarks/>.
- /Page/Members
- Contains the documentation for all of the members of the type, including a table for all of the members.
- /Page/Copyright
- Contains copyright information taken from the mdoc repository, specifically from index.xml's /Overview/Copyright element.
By providing a custom --template XSLT and/or by providing an additional CSS file, you have some degree of control over the resulting documentation.
I'll be the first to admit that this isn't a whole lot of flexibility; there is no control over what CSS class names are used, nor is there any control over what is generated within the /Page//* elements. What this model does allow is for controlling the basic page layout, e.g. to add a site-wide menu system, allowing documentation to be consistent with the rest of the site.
For example, my site uses custom templates to provide a uniform look-and-feel with the rest of their respective sites for the Mono.Fuse and NDesk.Options documentation.
Next time, we'll cover mdoc export-msxdoc.
mdoc XML Schema
Previously, I mentioned that you could manually edit the XML files within the mdoc repository.
What I neglected to mention is that there are only parts of the XML files that you should edit, and that there is an XML Schema file available for all docs.
The mdoc(5) man page lays out which files within the repository (and which parts of those files) are editable. In summary, all ns-*.xml files and the //Docs nodes of all other .xml files are editable, and they should contain ye normal XML documentation elements (which are also documented within the mdoc(5) man page).
The XML Schema can be found in Mono's SVN, at http://anonsvn.mono-project.com/source/trunk/mcs/tools/mdoc/Resources/monodoc-ecma.xsd.
Writing Documentation for mdoc
Last time, we create an assembly and used mdoc to generate a documentation repository containing stubs. Stubs have some utility -- you can view the types, members, and parameter types that are currently present -- but they're far from ideal. We want actual documentation.
Unfortunately, mdoc isn't an AI, and can't write documentation for you. It manages documentation; it doesn't create it.
How do we get actual documentation into the respository? There are three ways:
- Manually edit the XML files within the repository directory (if following from last time, this would be all .xml files within the Documentation/en directory.
- Use monodoc --edit Documentation/en.
- We can continue writing XML documentation within our source code.
Manually editing the files should be self-explanatory; it's not exactly ideal, but it works, and is how I write most of my documentation.
When using monodoc --edit Documentation/en, the contents of Documentation/en will be shown sorted in the tree view by it's assembly name, e.g. in the Mono Documentation → Demo node. When viewing documentation, there are [Edit] links that, when clicked, will allow editing the node (which directly edits the files within Documentation/en.
However, I can't recommend monodoc as an actual editor. It's usability is terrible, and has one major usability flaw: when editing method overloads, most of the documentation will be the same (or similar enough that you'll want to copy everything anyway), e.g. <summary/>, <param/>, etc. The monodoc editor doesn't allow copying all of this at once, but only each element individually. It makes for a very slow experience.
Which brings us to inline XML documentation. mdoc update supports importing XML documentation as produced by csc /doc. So let's edit our source code to add inline documentation:
using System;
namespace Cadenza {
/// <summary>
/// Extension methods on <see cref="T:System.Object" />.
/// </summary>
public static class ObjectCoda {
/// <typeparam name="TSource">The type to operate on.</typeparam>
/// <typeparam name="TResult">The type to return.</typeparam>
/// <param name="self">
/// A <typeparamref name="TSource" /> containing the value to manipulate.
/// This value may be <see langword="null" /> (unlike most other
/// extension methods).
/// </param>
/// <param name="selector">
/// A <see cref="T:System.Func{TSource,TResult}" /> which will be
/// invoked with <paramref name="self" /> as a parameter.
/// </param>
/// <summary>
/// Supports chaining otherwise temporary values.
/// </summary>
/// <returns>
/// The value of type <typeparamref name="TResult" /> returned by
/// <paramref name="selector" />.
/// </returns>
/// <remarks>
/// <para>
/// <c>With</c> is useful for easily using an intermediate value within
/// an expression "chain" without requiring an explicit variable
/// declaration (which is useful for reducing in-scope variables, as no
/// variable is explicitly declared).
/// </para>
/// <code lang="C#" src="../../example.cs#With" />
/// </remarks>
/// <exception cref="T:System.ArgumentNullException">
/// <paramref name="selector" /> is <see langword="null" />.
/// </exception>
public static TResult With<TSource, TResult>(
this TSource self,
Func<TSource, TResult> selector)
{
if (selector == null)
throw new ArgumentNullException ("selector");
return selector (self);
}
}
}
(As an aside, notice that our file ballooned from 14 lines to 45 lines because of all the documentation. This is why I prefer to keep my documentation external to the source code, as it really bloats the source. Certainly, the IDE can hide comments, but I find that this defeats the purpose of having comments in the first place.)
Compile it into an assembly (use csc if running on Windows), specifying the /doc parameter to extract XML documentation comments:
$ gmcs /t:library /out:Demo.dll /doc:Demo.xml demo.cs
Update our documentation repository, but import Demo.xml:
$ mdoc update -o Documentation/en -i Demo.xml Demo.dll --exceptions=added Updating: Cadenza.ObjectCoda Members Added: 0, Members Deleted: 0
(No members were added or deleted as we're only changing the documentation, and didn't add any types or members to the assembly.)
Now when we view ObjectCoda.xml, we can see the documentation that was present in the source code.
However, notice one other change. In the documentation we wrote, we had:
/// <code lang="C#" src="../../example.cs#With" />
Yet, within ObjectCoda.xml, we have:
<code lang="C#" src="../../example.cs#With">Console.WriteLine(
args.OrderBy(v => v)
.With(c => c.ElementAt (c.Count()/2)));
</code>
What's going on here? What's going on is that mdoc will search for all <code/> elements. If they contain a //code/@src attribute, the specified file is read in and inserted as the //code element's value. The filename specified in the //code/@src attribute is relative to the documentation repository root. A further extension is that, for C# code, if the filename has an "anchor", a #region block of the same name is searched for within the source code.
The ../../example.cs file referenced in the //code/@src value has the contents:
using System;
using System.Linq;
using Cadenza;
class Demo {
public static void Main (string[] args)
{
#region With
Console.WriteLine(
args.OrderBy(v => v)
.With(c => c.ElementAt (c.Count()/2)));
#endregion
}
}
This makes keeping documentation examples actually compiling trivial to support. For example, I'll have documentation refer to my unit tests, e.g.
<code lang="C#" src="../../Test/Cadenza/ObjectTest.cs#With" />
One final point worth mentioning: you can import documentation as often as you want. The imported documentation will always overwrite whatever is already present within the documentation repository. Consequently, if you want to use mdoc for display purposes but want to continue using inline XML documentation, always import the compiler-generated .xml file.
Now, we can update our HTML documentation:
$ mdoc export-html -o html Documentation/en Cadenza.ObjectCoda
The current Demo.dll documentation.
Next time, we'll cover customizing the static HTML output.
Using mdoc
As mentioned last time, mdoc is an assembly-based documentation management system. Thus, before you can use mdoc you need an assembly to document. Let's write some C# source:
using System;
namespace Cadenza {
public static class ObjectCoda {
public static TResult With<TSource, TResult>(
this TSource self,
Func<TSource, TResult> selector)
{
if (selector == null)
throw new ArgumentNullException ("selector");
return selector (self);
}
}
}
Compile it into an assembly (use csc if running on Windows):
$ gmcs /t:library /out:Demo.dll demo.cs
Now that we have an assembly, we can create the mdoc repository for the Demo.dll assembly, which will contain documentation stubs for all publically visible types and members in the assembly:
$ mdoc update -o Documentation/en Demo.dll --exceptions=added New Type: Cadenza.ObjectCoda Member Added: public static TResult With<TSource,TResult> (this TSource self, Func<TSource,TResult> selector); Namespace Directory Created: Cadenza New Namespace File: Cadenza Members Added: 1, Members Deleted: 0
mdoc update is the command for for synchronizing the documentation repository with the assembly; it can be run multiple times. The -o option specifies where to write the documentation repository. Demo.dll is the assembly to process; any number of assemblies can be specified. The --exceptions argument analyzes the IL to statically determine which exception types can be generated from a member. (It is not without some limitations; see the "--exceptions" documentation section.) The added argument to --exceptions tells mdoc to add <exception/> elements only for types and members that have been added to the repository, not to all types and members in the assembly. This is useful for when you've removed <exception/> documentation and don't want mdoc to re-add them.
We choose Documentation/en as the documentation repository location so that we can easily support localizing the documentation into multiple languages: each directory underneath Documentation would be named after an ISO 639-1 code, e.g. en is for English. This is only a convention, and is not required; any directory name can be used.
Notice that, since mdoc is processing assemblies, it will be able to work with any language that can generate assemblies, such as Visual Basic.NET and F#. It does not require specialized support for each language.
Now we have a documentation repository containing XML files; a particularly relevant file is ObjectCoda.xml, which contains the documentation stubs for our added type. I won't show the output here, but if you view it there are three important things to note:
- The XML is full of type information, e.g. the /Type/Members/Member/Parameters/Parameter/@Type attribute value.
- The XML contains additional non-documentation information, such as the //AssemblyVersion elements. This will be discussed in a future blog posting.
- The //Docs elements are a container for the usual C# XML documentation elements.
Of course, a documentation repository isn't very useful on it's own. We want to view it! mdoc provides three ways to view documentation:
- mdoc export-html: This command generates a set of static HTML files for all types and members found within the documentation repository.
- mdoc assemble: This command "assembles" the documentation repository into a .zip and .tree file for use with the monodoc Documentation browser and the ASP.NET front-end (which powers http://www.go-mono.com/docs).
- mdoc export-msxdoc: This generates the "traditional" XML file which contains only member documentation. This is for use with IDEs like Visual Studio, so that the IDE can show summary documentation while editing.
We will cover mdoc assemble and mdoc export-msxdoc in future installments. For now, to generate static HTML:
$ mdoc export-html -o html Documentation/en Cadenza.ObjectCoda
The current Demo.dll documentation.
Next time we will cover how to write actual documentation instead of just documentation stubs.
TekPub's Mastering LINQ Challenge
Justin Etheredge has posted TekPub's Mastering LINQ Challenge, in which he lays out a "little LINQ challenge." The rules:
- You have to blog about a single LINQ query which starts with Enumerable.Range(1,n) and produces a list of prime numbers from the range. Thus, this blog posting. (Otherwise I'd rely on my twitter response.)
- You can't cheat. This is determined by me, and includes hardcoding values in the results. You'll know if you cheated. Part of me wonders if just being me qualifies as cheating, but that might imply that my computer self has too large an ego </ob-##csharp-meme>.
- Uses no custom LINQ methods. Here I ponder what constitutes a "custom LINQ method." Is any extension method a custom LINQ method? Any utility code?
- Will return all of the prime numbers of the sequence. It doesn't have to be super optimal, but it has to be correct. Boy is it not super optimal (it's a one liner!), but some improvements could make it better (e.g. Memoization, hence the prior question about whether extension methods constitute a "custom LINQ method").
- Be one of the first 5 people to blog a correct answer and
then tweet this "I just solved the @tekpub LINQ challenge: <link to
post>" will get any single TekPub screencast. The time of your
solution will be based on your tweet! So be prompt!
As far as timliness, I'm writing this blog entry over four hours after my tweet, so, uh, so much for timliness.
- You must link to both TekPub's
website and
this post in your blog post.
Done, and done.
So, the quick and dirty, not at all efficent answer (with longer identifiers as I certainly have more than 140 characters to play with:
Enumerable.Range(1, n).Where(value =>
value <= 3
? true
: Enumerable.Range(2, value - 2)
.All(divisor => value % divisor != 0))
In English, we take all integers between 1 and n. Given a value from that sequence, if the value is less than 3, it's prime. If it's greater than three, take all numbers from 2 until value-1 and see if any of them divides value with no remainder. If none of them divide with no remainder, value is prime.
We need to use value-2 in the nested Enumerable.Range call so that we skip the value itself (since we're starting at 2).
Now, we can improve upon this in a fairly straightforward fashion if we can use additional code. For example, if we use Bart de Smet's Memoize extension method on System.Func<T, TResult>, we can skip the repeated nested Enumerable.Range call on every value, as prime numbers don't change (and thus are prime candidates for caching ;-):
Func<int, bool> isPrime = value =>
value <= 3
? true
: Enumerable.Range(2, value - 2)
.All(divisor => value % divisor != 0))
isPrime = isPrime.Memoize();
Enumerable.Range(1, n).Where(value => isPrime(value));
Whether this latter answer matches the rules depends upon the definition of "single LINQ query" (does the definition of isPrime need to be part of the LINQ query, or just its use?) and whether Bart's Memoize extension method qualifies as a "custom LINQ method" (I don't think it is...). The downside to the memoization is that it's basically a memory leak in disguise, so I still wouldn't call it "optimal," just that it likely has better performance characteristics than my original query...
What is mdoc?
mdoc is an assembly-based documentation management system, which recently added support for .NET .
I say "assembly based" because an alternative is source-based, which is what "normal" C# XML documentation, JavaDoc, and perlpod provide. Unlike these source-based systems, in mdoc documentation for public types and members are not present within source code. Instead, documentation is stored externally (to the source), in a directory of XML files (hereafter refered to as the mdoc repository).
Furthermore, mdoc provides commands to:
- synchronize the mdoc repository with an assembly being documented (adding to the repository any types and members added to the assembly);
- to export the mdoc repository into HTML or Microsoft's XML format (for use with an IDE like Visual Studio and MonoDevelop);
- to validate the mdoc repository XML schema;
- to assemble the mdoc repository for use with the Monodoc Documentation Browser and the ASP.NET front-end.
Why the mdoc repository?
Why have a directory of XML files as the mdoc repository? The mdoc repository comes from the need to satisfy two goals:
- The compiler-generated /doc XML contains no type information.
- Having types is very useful for HTML output/etc., so the type information must come from somewhere.
Said "somewhere" could be the actual assemblies being documented, but this has other downsides (e.g. it would complicate supporting different versions of the same assembly). mdoc uses the repository to contain both documentation and full type information, so that the source assemblies are only needed to update the repository (and nothing else).
Why use mdoc?
Which provides enough background to get to the point: why use mdoc?
You would primarily want to use mdoc if you want to view your documentation outside of an IDE, e.g. within a web browser or stand-alone documentation browser. Most mdoc functionality is geared toward making documentation viewable (e.g. mdoc export-html and mdoc assemble), and making the documentation that is viewed more useful (such as the full type information provided by mdoc update and the generation of <exception/> elements for documentation provided by mdoc update --exceptions).
Next time, we'll discuss how to use mdoc.
Re-Introducing mdoc
Many moons ago, Jon Skeet announced Noda Time. In it he asked:
How should documentation be created and distributed?
- Is Sandcastle the best way of building docs? How easy is it to get it running so that any developer can build the docs at any time? (I've previously tried a couple of times, and failed miserable.)
- Would Monodoc be a better approach?
Thus I pondered, "how well does mdoc support Windows users?"
The answer: not very well, particularly in an interop scenario.
So, lots of bugfixing and a false-start later, and I'd like to announce mdoc for .NET. All the power of mdoc, cross-platform.
Note that these changes did not make it into Mono 2.6, and won't be part of a formal Mono release until Mono 2.8. Consequently, if you want to run things under .NET, you should use the above ZIP archive. (You can, of course, install Mono on Windows and then use mdoc via Mono, you just won't be able to run mdoc under .NET.)
The changes made since Mono 2.6 include:
- Always generate Unix line endings within mdoc-generated XML files. This allows directories to be shared between Windows and Linux (e.g. with Samba) without causing lots of differences due to end-of-line changes.
- Fix the mdoc export-html XML stylesheets so that they would work under .NET (as mdoc uses XSLT and started relying on several Mono bugs).
- Use XslCompiledTransform, as .NET's XslTransform is really slow. (Mono's XslCompiledTransform is just a wrapper of XslTransform, so this doesn't change things under Mono, but under .NET this brought a 3+ minute execution down to 1.7 seconds.)
- Add support for properly converting <see cref="N:SomeNamespace"/> links into HTML links.
Next time, we'll cover what mdoc is, and why you'd want to use it.
Linq to SQL on Mono 2.6: NerdDinner on Mono
NerdDinner is an ASP.NET MVC sample, licensed under the Ms-PL with sources hosted at CodePlex.
Back on May 14th, I wrote that NerdDinner could be run under Mono using trunk.
Now, I'm pleased to note that the just-released Mono 2.6 includes these changes. Furthermore, thanks to ankit's progress on xbuild, installation and setup is easier than before:
- Build (or otherwise obtain) Mono 2.6. The Parallel Mono Environments page may be helpful.
- Download the NerdDinner 1.0 sources through a web browser. (curl or wget won't work.)
- Extract the NerdDinner sources:
$ mkdir -p $HOME/tmp $ cd $HOME/tmp $ unzip "/path/to/NerdDinner 1.0.zip"
- Build NerdDinner 1.0:
(Unfortunately we can't build just run xbuild (or build NerdDinner.sln) as this requires access to the MSTest assemblies used by the NerdDinner unit tests, which aren't currently present on Mono.)$ cd "$HOME/tmp/NerdDinner 1.0" $ xbuild NerdDinner/NerdDinner.csproj
- Only the web portion runs under Mono, as does the data access layer (System.Data.Linq, more affectionately known as Linq to SQL). The database is still Microsoft SQL Server. Go forth and configure the NerdDinner server (if you don't already have one configured).
- Back on the Linux side of things, edit $HOME/tmp/NerdDinner
1.0/NerdDinner/ConnectionStrings.config, and change the
NerdDinnerConnectionString connection string to:
You will need to adjust the machine name in the Data Source parameter to contain your actual computer name, and change the User ID and Password to whatever values you chose when configuring SQL Server.<add name="NerdDinnerConnectionString" connectionString="Data Source=gourry\SQLEXPRESS; Initial Catalog=NerdDinner; User ID=gourry\jonp; Password=123456; Integrated Security=true"/> - Configure a MembershipProvider for NerdDinner username/password storage.
- Run the web app:
The MONO_IOMAP environment variable is needed because some link targets used within NerdDinner require case insensitivity.$ cd "$HOME/tmp/NerdDinner 1.0/NerdDinner" $ MONO_IOMAP=all xsp2
Some things worth noting since May. First, openSUSE has released openSUSE 11.2, which is apparently more stringent than 11.1. Consequently, you may need to open the firewall so that port 8080 is accessible. You can do this by:
- Opening YaST.
- Starting the Firewall applet.
- In the Allowed Services area, add the HTTP Server and Mono XSP2 ASP.NET Host Service services.
- Click Next, then Finish.
One other oddity I encountered is that a url of http://localhost:8080 isn't permitted; using telnet(1) shows that it attempts to connect to ::1... (i.e. a IPv6 address), and the connection is refused. Instead, I needed to connect to http://127.0.0.1:8080.
Mono.Data.Sqlite & System.Data in MonoTouch 1.2 [Preview]
One of the new features that will be present in MonoTouch 1.2 is inclusion of the System.Data and Mono.Data.Sqlite assemblies. This is a preview release of System.Data et. al; it may not fully work. Known limitations are at the end of this post.
What Does This Mean?
It means that the following assemblies will be included in MonoTouch 1.2, and thus usable by MonoTouch applications:
- System.Data.dll: Reduced; see below.
- System.Transactions.dll: Unchanged.
- Mono.Data.Tds.dll: Unchanged.
- Mono.Data.Sqlite.dll: Unchanged, but see below.
Example?
Sure:
using System;
using System.Data;
using System.IO;
using Mono.Data.Sqlite;
class Demo {
static void Main (string [] args)
{
var connection = GetConnection ();
using (var cmd = connection.CreateCommand ()) {
connection.Open ();
cmd.CommandText = "SELECT * FROM People";
using (var reader = cmd.ExecuteReader ()) {
while (reader.Read ()) {
Console.Error.Write ("(Row ");
Write (reader, 0);
for (int i = 1; i < reader.FieldCount; ++i) {
Console.Error.Write(" ");
Write (reader, i);
}
Console.Error.WriteLine(")");
}
}
connection.Close ();
}
}
static SqliteConnection GetConnection()
{
var documents = Environment.GetFolderPath (
Environment.SpecialFolder.Personal);
string db = Path.Combine (documents, "mydb.db3");
bool exists = File.Exists (db);
if (!exists)
SqliteConnection.CreateFile (db);
var conn = new SqliteConnection("Data Source=" + db);
if (!exists) {
var commands = new[] {
"CREATE TABLE People (PersonID INTEGER NOT NULL, FirstName ntext, LastName ntext)",
"INSERT INTO People (PersonID, FirstName, LastName) VALUES (1, 'First', 'Last')",
"INSERT INTO People (PersonID, FirstName, LastName) VALUES (2, 'Dewey', 'Cheatem')",
"INSERT INTO People (PersonID, FirstName, LastName) VALUES (3, 'And', 'How')",
};
foreach (var cmd in commands)
using (var c = conn.CreateCommand()) {
c.CommandText = cmd;
c.CommandType = CommandType.Text;
conn.Open ();
c.ExecuteNonQuery ();
conn.Close ();
}
}
return conn;
}
static void Write(SqliteDataReader reader, int index)
{
Console.Error.Write("({0} '{1}')",
reader.GetName(index),
reader [index]);
}
}
The above code creates the Documents/mydb.db3 SQLite database, populates it if it doesn't already exist, then executes a SQL query against the database using normal, standard, ADO.NET mechanisms.
What's Missing?
Functionality is missing from System.Data.dll and Mono.Data.Sqlite.dll.
Functionality missing from System.Data.dll consists of:
- Anything requiring System.CodeDom (e.g. System.Data.TypedDataSetGenerator)
- XML config file support (e.g. System.Data.Common.DbProviderConfigurationHandler)
- System.Data.Common.DbProviderFactories (depends on XML config file support)
- System.Data.OleDb
- System.Data.Odbc
- The System.EnterpriseServices.dll dependency was removed from System.Data.dll, resulting in the removal of the SqlConnection.EnlistDistributedTransaction(ITransaction) method.
Meanwhile, Mono.Data.Sqlite.dll suffered no source code changes, but instead may be host to a number of runtime issues (the primary reason this is a preview release). Mono.Data.Sqlite.dll binds SQLite 3.5. iPhoneOS, meanwhile, ships with SQLite 3.0. Suffice it to say, some things have changed between the two versions. ;-)
Thus, the real question is this: what's missing in SQLite 3.0? The following functions are used by Mono.Data.Sqlite.dll but are missing from iPhoneOS's SQLite:
- sqlite3_column_database_name
- sqlite3_column_database_name16
- sqlite3_column_origin_name
- sqlite3_column_origin_name16
- sqlite3_column_table_name
- sqlite3_column_table_name16
- sqlite3_key
- sqlite3_rekey
- sqlite3_table_column_metadata
Where are these functions used (i.e. what can't you use from Mono.Data.Sqlite)? These appear to be related to database schema querying, e.g. determining at runtime which columns exist on a given table, such as Mono.Data.Sqlite.SqliteConnection.GetSchema (overriding DbConnection.GetSchema) and Mono.Data.Sqlite.SqliteDataReader.GetSchemaTable (overriding DbDataReader.GetSchemaTable). In short, it seems that anything using DataTable is unlikely to work.
Why Provide Mono.Data.Sqlite?
Why not? We realize that there are pre-existing SQLite solutions, but felt that many people would prefer to use the ADO.NET code they're already familiar with. Bringing System.Data and Mono.Data.Sqlite to MonoTouch permits this.
What About Data Binding?
Data binding with e.g. a UITableView is not currently implemented.
Conclusion
I suck at conclusions. :-)
Hope you enjoy this preview!
Linq to SQL on Mono Update: NerdDinner on Mono
NerdDinner is an ASP.NET MVC sample, licensed under the Ms-PL with sources hosted at CodePlex.
It is now possible to run the web portions of NerdDinner 1.0 on Linux with Mono trunk, thanks to Marek Habersack and Gonzalo Paniagua Javier's help with Mono's ASP.NET and ASP.NET MVC support, and the DbLinq community's assistance with Linq to SQL support.
This shows a growing level of maturity within Mono's Linq to SQL implementation.
- Build Mono from trunk. The Parallel Mono Environments page may be helpful.
- Download the NerdDinner 1.0 sources through a web browser. (curl or wget won't work.)
- Extract the NerdDinner sources:
$ mkdir -p $HOME/tmp $ cd $HOME/tmp $ unzip "/path/to/NerdDinner 1.0.zip"
- Build NerdDinner 1.0:
$ cd "$HOME/tmp/NerdDinner 1.0/NerdDinner" $ mkdir bin $ gmcs -t:library -out:bin/NerdDinner.dll -debug+ -recurse:'*.cs' \ -r:System -r:System.Configuration -r:System.Core \ -r:System.Data -r:System.Data.Linq -r:System.Web \ -r:System.Web.Abstractions -r:System.Web.Mvc \ -r:System.Web.Routing - As mentioned in the introduction, only the web portion runs under Mono,
as does the data access layer (System.Data.Linq, more
affectionately known as Linq to SQL). The database is still Microsoft
SQL Server. Find yourself a Windows machine, install SQL Server 2008
(Express
is fine), and perform the following bits of configuration:
- Create the database files:
- Copy the NerdDinner_log.ldf and NerdDinner.mdf files from the $HOME/tmp/NerdDinner 1.0/NerdDinner/App_Data directory to your Windows machine, e.g. C:\tmp.
- Within Windows Explorer, go to C:\tmp, right-click the C:\tmp folder, click Properties, click the Security tab, click Edit..., and add the Full Control, MOdify, Read & execute, List folder contents, Read, and Write permissions to the User group. Click OK.
- Repeat the above permissions modifications for the NerdDinner_log.ldf and NerdDinner.mdf files in C:\tmp.
- Add the NerdDinner database files to Microsoft SQL Server:
- Start Microsoft SQL Server Management Studio (Start → All Programs → Microsoft SQL Server 2008 → SQL Server Management Studio).
- Connect to your database instance.
- Within the Object Explorer (View → Object Explorer), right-click the database name and click Attach....
- Within the Attach Databases dialog, click the Add... button, and choose C:\tmp\NerdDinner.mdf in the Locate Database Files dialog. Click OK in both the Locate Database Files dialog and the Attach Databases dialog.
- Enable mixed-mode authentication:
- Start Microsoft SQL Server Management Studio.
- Connect to your database instance.
- Within the Object Explorer, right-click the database name and click Properties.
- In the Server Properties dialog, select the Security page.
- In the Server authentication section, select the SQL Server and Windows Authentication mode radio button.
- Click OK.
- Restart SQL Server by right-clicking on the database name and clicking Restart.
- Add a SQL Server user:
- Within SQL Server Management Studio, connect to the database instance.
- Within the Object Explorer, expand the Security → Logins tree node.
- Right-click the Logins node, and click New Login....
- In the Login - New dialog, enter a login name. We'll use jonp for discussion purposes. Select the SQL Server authentication dialog button, and enter a password in the Password and Conform Password text boxes. For discussion purposes we'll use 123456.
- Still within the Login - New dialog, select the User Mapping page. In the Users mapped to this login section, select the checkbox in the Map column corresponding to the NerdDinner database. Within the Database role membership for: NerdDinner section, select the db_datareader and db_datawriter roles. Click OK.
- Enable remote access to SQL Server (see
also):
- Configure SQL Server:
- Start SQL Server Configuration Manager (Start → All Programs → Microsoft SQL Server 2008 → Configuration Tools → SQL Server Configuration Manager).
- In the left-hand pane, select the SQL Server Configuration manager (Local) → SQL Server Network Configuration → Protocols for Database Instance Name node.
- In the right pane, double click the TCP/IP Protocol Name.
- In the Protocol tab, set Enabled to Yes. Click OK.
- In the left-hand pane, go to the SQL Server Configuration Manager (Local) → SQL Server Services node.
- In the right pane, double-click SQL Server Browser.
- In the Service tab, set the Start Mode property to Automatic. Click OK.
- Right-click SQL Server Browser, and click Start.
- Right-click SQL Server, and click Restart.
- Configure Windows Firewall
- Within Windows Control Panel, open the Windows Firewall applet.
- Click the Allow a program through Windows Firewall link.
- In the Windows Firewall Settings dialog, click the Exceptions tab.
- Click Add program..., and add the following
programs:
- sqlbrowser.exe (C:\Program Files\Microsoft SQL Server\90\Shared\sqlbrowser.exe)
- sqlservr.exe (C:\Program Files\Microsoft SQL Server\MSSQL10.SQLEXPRESS\MSSQL\Binn\sqlservr.exe)
- Click OK.
- Configure SQL Server:
- Create the database files:
- Back on the Linux side of things, edit $HOME/tmp/NerdDinner
1.0/NerdDinner/ConnectionStrings.config, and change the
NerdDinnerConnectionString connection string to:
You will need to adjust the machine name in the Data Source parameter to contain your actual computer name, and change the User ID and Password to whatever values you chose in §5.E.iv.<add name="NerdDinnerConnectionString" connectionString="Data Source=gourry\SQLEXPRESS;Initial Catalog=NerdDinner;User ID=jonp;Password=123456;"/> - NerdDinner makes use of ASP.NET's MembershipProvider functionality, so a
SQLite database needs to be created to contain the username and password
information for the NerdDinner site. This is detailed at the
ASP.NET
FAQ and Guide: Porting ASP.NET Applications pages:
$ cd "$HOME/tmp/NerdDinner 1.0/NerdDinner/App_Data # Create the commands needed to configure the SQLite database: $ cat > aspnetdb.sql <<EOF CREATE TABLE Users ( pId character(36) NOT NULL, Username character varying(255) NOT NULL, ApplicationName character varying(255) NOT NULL, Email character varying(128) NOT NULL, Comment character varying(128) NULL, Password character varying(255) NOT NULL, PasswordQuestion character varying(255) NULL, PasswordAnswer character varying(255) NULL, IsApproved boolean NULL, LastActivityDate timestamptz NULL, LastLoginDate timestamptz NULL, LastPasswordChangedDate timestamptz NULL, CreationDate timestamptz NULL, IsOnLine boolean NULL, IsLockedOut boolean NULL, LastLockedOutDate timestamptz NULL, FailedPasswordAttemptCount integer NULL, FailedPasswordAttemptWindowStart timestamptz NULL, FailedPasswordAnswerAttemptCount integer NULL, FailedPasswordAnswerAttemptWindowStart timestamptz NULL, CONSTRAINT users_pkey PRIMARY KEY (pId), CONSTRAINT users_username_application_unique UNIQUE (Username, ApplicationName) ); CREATE INDEX users_email_index ON Users (Email); CREATE INDEX users_islockedout_index ON Users (IsLockedOut); CREATE TABLE Roles ( Rolename character varying(255) NOT NULL, ApplicationName character varying(255) NOT NULL, CONSTRAINT roles_pkey PRIMARY KEY (Rolename, ApplicationName) ); CREATE TABLE UsersInRoles ( Username character varying(255) NOT NULL, Rolename character varying(255) NOT NULL, ApplicationName character varying(255) NOT NULL, CONSTRAINT usersinroles_pkey PRIMARY KEY (Username, Rolename, ApplicationName), CONSTRAINT usersinroles_username_fkey FOREIGN KEY (Username, ApplicationName) REFERENCES Users (Username, ApplicationName) ON DELETE CASCADE, CONSTRAINT usersinroles_rolename_fkey FOREIGN KEY (Rolename, ApplicationName) REFERENCES Roles (Rolename, ApplicationName) ON DELETE CASCADE ); CREATE TABLE Profiles ( pId character(36) NOT NULL, Username character varying(255) NOT NULL, ApplicationName character varying(255) NOT NULL, IsAnonymous boolean NULL, LastActivityDate timestamptz NULL, LastUpdatedDate timestamptz NULL, CONSTRAINT profiles_pkey PRIMARY KEY (pId), CONSTRAINT profiles_username_application_unique UNIQUE (Username, ApplicationName), CONSTRAINT profiles_username_fkey FOREIGN KEY (Username, ApplicationName) REFERENCES Users (Username, ApplicationName) ON DELETE CASCADE ); CREATE INDEX profiles_isanonymous_index ON Profiles (IsAnonymous); CREATE TABLE ProfileData ( pId character(36) NOT NULL, Profile character(36) NOT NULL, Name character varying(255) NOT NULL, ValueString text NULL, ValueBinary bytea NULL, CONSTRAINT profiledata_pkey PRIMARY KEY (pId), CONSTRAINT profiledata_profile_name_unique UNIQUE (Profile, Name), CONSTRAINT profiledata_profile_fkey FOREIGN KEY (Profile) REFERENCES Profiles (pId) ON DELETE CASCADE ); EOF # Create the SQLite database: $ sqlite3 aspnetdb.sqlite sqlite> .read aspnetdb.sql sqlite> .quit
- Run the web app:
The MONO_IOMAP environment variable is needed because some link targets used within NerdDinner require case insensitivity.$ MONO_IOMAP=all xsp2
Mono 2.4 and mdoc-update
Mono 2.4 was released, and among the unlisted changes was that mdoc-update has migrated from using Reflection to using Mono.Cecil.
There are multiple advantages and disadvantages to this migration. The disadvantages include slower execution (when I tested, Mono.Cecil took ~10% longer to do the same task as Reflection) and increased dependencies (Mono.Cecil is now required).
I believe that these disadvantages are outweighed by the advantages. Firstly, the migration makes my life significantly easier. One of the major limitations of Reflection is that only one mscorlib.dll can be loaded into a process. This means that, in order to support generating documentation from mscorlib.dll 1.0, there needs to be a version of mdoc-update that runs under .NET 1.0. Similarly, to document mscorlib.dll 2.0, I need a different version of mdoc-update which runs under .NET 2.0. And when .NET 4.0 is released (with yet another version of mscorlib.dll), I'll need...yet another version of mdoc-update to run under .NET 4.0. This is less than ideal, and using Mono.Cecil allows me to have one program which supports every version of mscorlib.dll.
This also means that I can use C# 3.0 features within mdoc-update, as I no longer need to ensure that (most of) mdoc-update can run under the .NET 1.0 profile.
Most people won't care about making my life easier, but I do. ;-)
For everyone else, the most important result of the Mono.Cecil migration is that mdoc-update now has a suitable base for advanced documentation generation scenarios which make use of IL analysis. The first feature making use of it is new --exceptions functionality, which analyzes member IL to determine which exceptions could be generated, and creates stub <exception/> XML documentation based on that analysis. This feature is experimental (see the documentation), and contains a number of corner cases, but I've already found it useful for writing Mono.Rocks documentation.
DbLinq and Mono
.NET 3.5 introduced Language Integrated Query (LINQ), which allowed for querying groupings of data across diverse "paradigms" -- collections (arrays, lists, etc.), XML, and relational data, called LINQ to SQL. LINQ to SQL support is within the System.Data.Linq assembly, which is one of the assemblies Mono is currently implementing.
However, LINQ to SQL has one limitation: it only works with Microsoft SQL Server and Microsoft SQL Server Compact Edition, leaving numerous other databases users unable to use this assembly.
Enter DbLinq, an effort to provide LINQ to SQL functionality for other databases, including Firebird, Ingres, MySQL, Oracle, PostgreSql, SQLite, and SQL Server. DbLinq provides a System.Data.Linq-compatible implementation for these databases (compatible implying the same types and methods, but located within a different namespace).
Which brings us to Mono. Mono is using DbLinq as the foundation for Mono's System.Data.Linq.dll implementation, allowing Mono's System.Data.Linq.dll to support all the databases that DbLinq supports. Mono also has sqlmetal (based on DbLinq's DbMetal.exe sources), which can be used to generate C# types to interact with databases.
DbLinq On Mono
MonoDevelop can load the DbLinq solutions. However, it has a problem with building all of the projects within the solution. At the time of this writing, MonoDevelop can build the following assemblies: DbLinq.dll, DbLinq.Sqlite_test_mono_strict.dll, DbLinq.SqlServer.dll, DbLinq.SqlServer_test.dll, DbLinq.SqlServer_test_ndb.dll, DbLinq.SqlServer_test_strict.dll, and DbLinq_test_ndb_strict.dll. The *_test* assemblies are unit tests, so this leaves the core DbLinq.dll assembly and SQL Server support.
Thus, DbLinq is usually built with Visual Studio.NET (the free Visual Studio Express can be used). Once built, you can run some of the unit tests under Mono:
cd $path_to_dblinq2007_checkout/build.dbg
# Core tests
$ for test in DbLinq_test.dll DbLinq_test_ndb_strict.dll DbMetal_test.dll ; do \
nunit-console2 $test \
done
# Verbose output omitted
# SQLite tests
$ nunit-console2 DbLinq.Sqlite_test_mono.dll
# Verbose output omitted
# Plus many tests for the other providers...
Most of the tests require an accessible database, so I've been limiting my current tests to SQLite (as setup is easier).
DbLinq In Mono
As mentioned before, DbLinq is being used to implement Mono's System.Data.Linq.dll. (For those reading the DbLinq source, the Mono-specific bits are within MONO_STRICT conditional code.) This allows us to write code that depends only on .NET assemblies (though this is of dubious value, as the mechanisms used to support SQLite and other databases won't work with .NET proper, but it's still a cute trick).
To play along, you'll need Mono trunk.
- Grab a SQLite database file to use with LINQ to SQL:
wget http://dblinq2007.googlecode.com/svn/trunk/src/Northwind.db3
- Use sqlmetal to generate C# bindings for the database:
sqlmetal /namespace:nwind /provider:Sqlite "/conn:Data Source=Northwind.db3" /code:nwind.cs
- Write some code to interact with the generated source code:
// File: nwind-app.cs // Compile as: // gmcs nwind-app.cs nwind.cs -r:System.Data \ // -r:System.Data.Linq -r:Mono.Data.Sqlite using System; using System.Data.Linq; using System.Linq; using Mono.Data.Sqlite; using nwind; class Test { public static void Main () { var conn = new SqliteConnection ( "DbLinqProvider=Sqlite;" + "Data Source=Northwind.db3" ); Main db = new Main (conn); var pens = from p in db.Products where p.ProductName == "Pen" select p; foreach (var pen in pens) { Console.WriteLine (" CategoryID: {0}", pen.CategoryID); Console.WriteLine (" Discontinued: {0}", pen.Discontinued); Console.WriteLine (" ProductID: {0}", pen.ProductID); Console.WriteLine (" ProductName: {0}", pen.ProductName); Console.WriteLine ("QuantityPerUnit: {0}", pen.QuantityPerUnit); Console.WriteLine (" ReorderLevel: {0}", pen.ReorderLevel); Console.WriteLine (" SupplierID: {0}", pen.SupplierID); Console.WriteLine (" UnitPrice: {0}", pen.UnitPrice); Console.WriteLine (" UnitsInStock: {0}", pen.UnitsInStock); Console.WriteLine (" UnitsOnOrder: {0}", pen.UnitsOnOrder); } } } - Compile:
gmcs nwind-app.cs nwind.cs -r:System.Data -r:System.Data.Linq -r:Mono.Data.Sqlite
- Run:
$ mono nwind-app.exe CategoryID: Discontinued: False ProductID: 1 ProductName: Pen QuantityPerUnit: 10 ReorderLevel: SupplierID: 1 UnitPrice: UnitsInStock: 12 UnitsOnOrder: 2
Notice that we use the database connection string to specify the database vendor to use, specifically the DbLinqProvider value specifies the database vendor, and must be present when connecting to a database other than Microsoft SQL Server (which is the default vendor).
If using the DataContext(string) constructor directly (and not through a generated subclass as used above), you should also provide the DbLinqConnectionType parameter, which is the assembly-qualified type name to use for the IDbConnection implementation. This allows you to use multiple different IDbConnection implementations that use similar SQL implementations, e.g. Mono.Data.Sqlite.dll and System.Data.SQLite.dll, both of which wrap the SQLite database.
Extension Method Documentation
C# 3.0 adds a new language feature called extension methods. Extension methods allow the "addition" of new instance methods to any type, without modifying the type itself. This is extremely powerful, arguably crack-adled, and exists because Visual Studio users can't do anything without code completion (tongue firmly in cheek).
It's also extremely useful, permitting LINQ and the even more crack-adled thinking in Mono.Rocks (much of which I wrote, and I'm not entirely sure if the "crack" is in jest or not; sometimes I wonder...).
To create an extension method, you first create a static class. A method within the static class is an extension method if the first parameter's type has a this modifier:
static class MyExtensions {
public static string Implode (this IEnumerable<string> self, string separator)
{
return string.Join (separator, self.ToArray ());
}
}
Usage is as if it were a normal instance method:
string[] a = {"This", "is", "my", "sentence."};
string imploded = a.Implode (" ");
// imploded == "This is my sentence."
Extension methods are entirely syntactic sugar. (Nice syntactic sugar, nonetheless...). As such, it doesn't in any way modify the type that is being extended. Consequently, it cannot access private members, nor is the extension method returned when reflecting over the extended type. For example, typeof(string[]).GetMethod("Implode") will return null, as System.Array doesn't have an Implode method.
Furthermore, extension methods are only available if you have a using declaration for the namespace the extension method type resides in. So if the above MyExtensions type resides in the Example namespace, and a source file doesn't have using Example;, then the Implode extension method isn't available.
Earlier I alluded that Visual Studio users can't do anything without code completion. Extension methods are thus a boon, as they (potentially) make it easier to find new functionality, as no new types need to be introduced or known about in advance. However, you still need to have an appropriate using declaration to bring the methods "in scope," so how does a developer know what namespaces to use? The same way a developer knows which type to use for anything: documentation.
MSDN online documentation has been enhanced to show which extension methods are applicable for a given type, e.g. The extension methods for IEnumerable<T>. Mono has similar documentation support.
This isn't particularly interesting, though. Part of the utility and flexibility is that any type, in any namespace, can be extended with extension methods, and the extension methods themselves can be contained in any type.
Obviously, MSDN and Mono documentation online can't know about extension methods that are not part of the core framework. Thus, if the e.g. Mono.Cecil or Gendarme frameworks provided extension methods, the online documentation sites won't be helpful.
Which brings us to a Mono 2.0 feature (yes, I'm only now announcing a feature that shipped 3 months ago):
Mono Documentation Tools: the Mono Documentation framework has been upgraded to support documenting generics and extension methods.
This support consists of four things:
- Enhancing mdoc update to generate an /Overview/ExtensionMethods element within index.xml. The /Overview/ExtensionMethods element contains <ExtensionMethod/> elements which in turn contains //Targets/Target elements specifying which types the extension method is an instance method on, and a <Member/> element which is a subset of the actual extension method documentation. Developers don't need to edit this copy; it's handled entirely by mdoc update.
- Enhancing mdoc assemble to look for the //ExtensionMethod elements and insert them into the ExtensionMethods.xml file within the generated .zip file.
- Enhancing the XML documentation to HTML generation process so that the extension methods are listed. This allows all of monodoc and mod, online documentation, and mdoc export-html to use the underlying infrastructure.
- Enhance monodoc.dll to load all ExtensionMethods.xml files from all installed .zip files. This the allows monodoc and online documentation mechanisms to show extension methods for all installed documentation sources.
The short of it is that this requires no workflow change to get extension methods listed on all extended types. Just create extension methods, document them as if they were normal static methods (as they are normal static methods, and can be invoked as such), assemble the documentation, and install the documentation.
There is one wrinkle, though: since the index.xml file contains a subset of the <Member/> documentation, you need to rerun mdoc update after editing extension method documentation so that index.xml will have the correct documentation when mdoc assemble is run. Otherwise the "summary" extension method documentation may differ from the actual intended documentation. This may be improved in a future release.
How To Defend Against Software Patent FUD
You don't.
Bwa-ha-ha-ha-ha-ha-ha¹⁰⁰⁰.
Context: for years, Mono has been the target of FUD because of potential software patent issues. For years the Mono community has attempted to defend from these attack, sometimes successfully.
Recently, someone asked on mono-list about ways to pre-emptively answer the FUD so that it would become a non-issue. I responded, and had several people suggest that I blog it. Here we go.
To begin, there are several problems with defending against software patent FUD, starting with software patents themselves:
- Software patents suck.
- Software patents really suck. (Specifically, The "Don't Look" Problem section.)
- Software patents really, really suck. (Related)
- The anti-Mono FUDsters apparently can't see the forest for the trees.
I imagine that most people reading this will agree with the first three points, so it is the fourth that I will attempt to focus on.
Specifically, the anti-Mono FUDsters seem to spend so much time on a tree (Microsoft) that they either miss or minimize the forest of actual patent problems, patent trolls, etc.
So for once, I'll (non-seriously) throw the FUD:
A long time ago, Wang created a patent that "covered a method by which a program can get help from another computer application to complete a task." Microsoft licensed the patent from Wang. Sun did not. In 1997, Eastman Kodak Company bought Wang, thus acquiring this patent. Kodak then sued Sun, claiming that Java infringed this patent. Kodak won, and they later settled out of court.
Now, for my non-serious steaming pile of FUD, in the form of a question: Did Sun acquire the ability to sublicense these patents from Kodak? If Sun can sublicense the patents, then GPL'd Java is fine. If Sun can't, then Java cannot be GPL'd, and any company making use of Java could be subject to a lawsuit from Kodak.
(I would hope that this is yes, but I have no idea, and the lack of patent sub-licensing has come up before.)
So do we need to worry about Java? I have no idea. I mention it to raise a larger point:
It Doesn't Matter. Anyone can hold a patent, for anything, and sue anyone at any time. Thus, Gnome is not free of patent issues, KDE is not free of patent issues, Linux is not free of patent issues, Python is not free of patent issues, Ruby is not free of patent issues.... Nothing is free of patent issues.
(Consider: do you think that the Python Software Foundation has signed a patent license with Kodak? Has Red Hat? I doubt it. Furthermore, I find it hard to believe that something as flexible as Python wouldn't violate the aforementioned Wang patent, especially when you get into COM interop/etc. on Windows...)
Having said the above, a related question becomes: How do you avoid violating someone's patents? You don't (insert more laughter). You could try restricting yourself to only using software that's at least 20 years old, but you won't gain many users that way. It also won't work, for at least two reasons: (1) submarine patents -- not all patents that would have been in effect 20 years ago have necessarily expired (though submarine patents shouldn't exist for ~too much longer); and (2) look at the drug patent industry, where to prevent patented drugs from "going generic" the drug companies take the patent-expired drug(s), combine them with other drugs, then patent the result. I don't think it will take too long for Software companies to start doing this if they feel that it's necessary, and once they do, even using known-patent-expired programs won't be safe, as merely combining them together may be covered by an unexpired patent. Yay.
The only other way to avoid software patents is to perform a patent search, which is extremely tricky (as software patents are deliberately vague), and if you miss a patent and get sued over it, you're now liable for treble damages. You're almost always better to not look at software patents. (Isn't it funny how something that was supposed to "promote the Progress of Science and useful Arts" can't be used by those it's supposed to help? Isn't it hilarious?)
With all this in mind, you can see why patent FUD is hard to fight, because there's no way to dismiss it. Software patents are a reality, they're ugly, but they can't be avoided. (Yet they must be ignored, to avoid increased liability.) My problem is that the anti-Mono people only seem to focus on patents with respect to Mono and Microsoft, ignoring the rest of the software industry. They're ignoring the (gigantic) forest so that they can pay attention to a single tree, Microsoft.
What I find even "funnier" is that Microsoft supposedly holds a number of patents in a number of areas frequently used by open-source projects, such as HTML, CSS, C++, XML, and others. So why don't we ever see any suggestions to avoid these technologies because the Big Bad Microsoft might sue?
For that matter, (again) considering how vague software patents tend to be, wouldn't many Microsoft patents on .NET stand a chance at being applicable toward Java, Python, and other projects? (Again) Why just focus on Mono?
Final note: I am a Software Engineer, not a patent lawyer. Feel free to ignore the entire rant, but I would appreciate it if a little more thought went into all the anti-Mono propaganda.
openSUSE 11.1: Where'd my hostname go?
After playing with the openSUSE 11.1 beta releases and final release, I finally installed it onto my main workstation. Funny how actually using it ~full-time shows things that were previous missed...
In this case, what greeted me when I opened a shell was:
jon@linux-jcq7$
This was rather unexpected, as this wasn't the hostname I wanted. No matter, this was normal after a fresh install (and has been happening for eons). So off I go to YaST to edit the Network Settings (/sbin/yast2 lan)...
Previously (i.e. on openSUSE 10.1, 10.2, 10.3, and 11.0), I could go to the Hostname/DNS tab, to the Hostname and Domain Name section, and specify a Hostname. (Whereupon everything would break until the next reboot as Gnome didn't seem to like the hostname changing on it, but at least I had the right hostname!)
Under openSUSE 11.1, this is disabled when NetworkManager controls things. (Again, this was not the case under 11.0 and prior releases, even when using NetworkManager to control things.)
So how do we change the hostname? Perusing Control Center brought forth the Network Connections applet → Wired tab → connection name (e.g. System eth0) → Edit → IPv4 Settings tab's DHCP Client ID textbox. This was nice to find -- I'd often wondered why setting the DHCP Client Identifier within YaST Network Settings seemingly had no effect; DHCP Client ID does work -- but it had no effect during bootup (presumably because NetworkManager isn't running early enough to set the hostname), so my shell prompt was still wrong.
Similarly, the "traditional" technique of hand-editing /etc/hosts (or using the new-fangled Hostnames YaST applet) seemed to have no effect on the system name after a reboot.
So how do we really change the hostname? Edit /etc/HOSTNAME, which is a single line file containing the fully-qualified hostname to use during bootup.
Icecream & Firewalls
Earlier this year, Michael Meeks described how to use icecream to speed up builds. One problem was that originally it required disabling the firewall on most systems. There was an update mentioning that setting FW_CONFIGURATIONS_EXT could be used to open up the appropriate ports in the firewall so that things would Just Work. Alas, that doesn't work for me on openSUSE 11.1.
Thus, if using the openSUSE Firewall Allowed Services configuration doesn't work (which is what setting FW_CONFIGURATIONS_EXT modifies), there is one alternate strategy to use before disabling the firewall: manually specify the scheduler system on the daemon systems within the icecream configuration file:
sudo sed -i 's/ICECREAM_SCHEDULER_HOST=""/ICECREAM_SCHEDULER_HOST="SCHEDULER"/' /etc/sysconfig/icecream
Replace SCHEDULER with the appropriate host name or IP address of your scheduler system.
Announcing NDesk.Options 0.2.1
I am pleased to announce the release of NDesk.Options 0.2.1. NDesk.Options is a C# program option parser library, inspired by Perl's Getopt::Long option parser.
To download, visit the NDesk.Options web page:
http://www.ndesk.org/Options
Usage
See http://www.ndesk.org/Options and the OptionSet documentation for examples.
What's New?
There have been several minor changes since the previous 0.2.0 release:
- The OptionSet base class has been changed from Collection<Option> to KeyedCollection<string, Option>, as KeyedCollection<string, Option> is conceptually closer to what OptionSet supports: one or more strings as aliases for a single Option.
- OptionSet.GetOptionForName() has been deprecated in favor of using KeyedCollection.Item(string).
- C# 2.0 Compatibility. The unit tests require a C# 3.0 compiler, but the actual option parser and related classes now only require a C# 2.0 compiler.
- Default argument handling support. This is useful for argument runs, in which the meaning of later options depends upon a prior argument, e.g.: mdoc-assemble --format=ecma A B --format=man C (where A and B are processed with --format=ecma in effect, while C is processed with --format=man in effect).
- The Option.Description property can now contain value formatting codes which are used by OptionSet.WriteOptionDescriptions().
- The Option.Description property is now automatically line-wrapped within OptionSet.WriteOptionDescriptions().
- ndesk-options.pc fixes for pkg-config.
- Unit tests now depend on NUnit and have been split out into separate files.
Threading: Lock Nesting
a.k.a. Why the Java 1.0 collections were rewritten...
Threading is an overly complicated subject, covered in great detail at other locations and in many books. However, there is one subject that either I haven't seen discussed too often, or somehow have managed to miss while reading the plethora of threading sources, something I'll call lock nesting depth:
- lock nesting depth
- The number of locks that must be acquired and held simultaneously in order to perform a given operation.
In general, the lock nesting depth should be kept as small as possible; anything else results in extra, possibly unnecessary/extraneous locks, which serve only to slow down performance for no added benefit.
First, an aside: why does threading code require locks? To maintain data invariants for data shared between threads, preventing the data from being corrupted. Note that this is not necessarily the same as producing "correct" data, as there may be internal locks to prevent internal data corruption but the resulting output may not be "correct" (in as much as it isn't the output that we want).
The prototypical example of "non-corrupting but not correct" output is when multiple threads write to the (shared) terminal:
using System;
using System.Threading;
class Test {
public static void Main ()
{
Thread[] threads = new Thread[]{
new Thread ( () => { WriteMessage ("Thread 1"); } ),
new Thread ( () => { WriteMessage ("Thread 2"); } ),
};
foreach (var t in threads)
t.Start ();
foreach (var t in threads)
t.Join ();
}
static void WriteMessage (string who)
{
Console.Write ("Hello from ");
Console.Write (who);
Console.Write ("!\n");
}
}
Output for the above program can vary from the sensible (and desirable):
$ mono ls.exe Hello from Thread 2! Hello from Thread 1! $ mono ls.exe Hello from Thread 1! Hello from Thread 2!
To the downright "corrupt":
Hello from Hello from Hello from Hello from Thread 2! Thread 1!
(This can happen when Thread 1 is interrupted by Thread 2 before it can write out its entire message.)
Notice what's going on here: as far as the system is concerned, what we're doing is safe -- no data is corrupted, my terminal/shell/operating system/planet isn't going to go bonkers, everything is well defined. It's just that in this circumstance "well defined" doesn't match what I, as the developer/end user, desired to see: one of the first two sets of output.
The solution, as always, is to either add a a lock within WriteMessage to ensure that the output is serialized as desired:
static object o = new object ();
static void WriteMessage (string who)
{
lock (o) {
Console.Write ("Hello from ");
Console.Write (who);
Console.Write ("!\n");
}
}
Or to instead ensure that the message can't be split up, working within the predefined semantics of the terminal:
static void WriteMessage (string who)
{
string s = "Hello from " + who + "!\n";
Console.Write (s);
}
(Which can oddly generate duplicate messages on Mono; not sure what's up with that... More here.)
For the WriteMessage that uses locks, the lock nesting depth is 2, and this can't be readily improved (because Console.Write is static, and thus must be thread safe as any thread could execute it at any time).
Returning to this entry's subtitle, why were the Java 1.0 collections rewritten? Because they were all internally thread safe. This had it's uses, should you be sharing a Hashtable or Vector between threads, but even then it was of limited usefulness, as it only protected the internal state for a single method call, not any state that may require more than one function call. Consider this illustrative code which counts the number of times a given token is encountered:
Hashtable data = new Hashtable ();
for (String token : tokens) {
if (data.containsKey (token)) {
Integer n = (Integer) data.get (token);
data.put (token, new Integer (n.intValue() + 1));
}
else {
data.put (token, new Integer (1));
}
}
Yes, Hashtable is thread safe and thus won't have its data corrupted, but it can still corrupt your data should multiple threads execute this code against a shared data instance, as there is a race with the data.containsKey() call, where multiple threads may evaluate the same token "simultaneously" (read: before the following data.put call), and thus each thread would try to call data.put (token, new Integer (1)). The result: a missed token.
The solution is obvious: another lock, controlled by the developer, must be used to ensure valid data:
Object lock = new Object ();
Hashtable data = new Hashtable ();
for (String token : tokens) {
synchronized (lock) {
if (data.containsKey (token)) {
Integer n = (Integer) data.get (token);
data.put (token, new Integer (n.intValue() + 1));
}
else {
data.put (token, new Integer (1));
}
}
}
Consequently, for all "non-trivial" code (where "non-trivial" means "requires more than one method to be called on the collection object in an atomic fashion") will require a lock nesting depth of two. Furthermore, the lock nesting depth would always be at least one, and since many functions were not invoked between multiple threads, or the collection instance local to that particular method, the synchronization within the collection was pure overhead, providing no benefit.
Which is why in Java 1.2, all of the new collection classes such as ArrayList and HashMap are explicitly unsynchronized, as are all of the .NET 1.0 and 2.0 collection types unless you use a synchronized wrapper such as System.Collections.ArrayList.Synchronized (which, again, is frequently of dubious value if you ever need to invoke more than one method against the collection atomically).
Finally, the Threading Design Guidelines of the .NET Framework Design Guidelines for Class Library Developers (book) suggests that all static members be thread safe, but instance member by default should not be thread safe:
Instance state does not need to be thread safe. By default, class libraries should not be thread safe. Adding locks to create thread-safe code decreases performance, increases lock contention, and creates the possibility for deadlock bugs to occur. In common application models, only one thread at a time executes user code, which minimizes the need for thread safety. For this reason, the .NET Framework class libraries are not thread safe by default.
Obviously, there are exceptions -- for example, if a static method returns a shared instance of some class, then all of those instance members must be thread safe as they can be accessed via the static method (System.Reflection.Assembly must be thread safe, as an instance of Assembly is returned by the static method Assembly.GetExecutingAssembly). By default, though, instance members should not be thread safe.
HackWeek Summary
In case you missed it, last week was "Hackweek" at Novell.
My week was less "hacking" and more "spit-and-polish." In particular:
- Wrote Mono.Unix.UnixSignal documentation. This only took ~1/2 a day.
- Released NDesk.Options 0.2.0, which in turn involved the
vicious cycle
of write documentation (and samples), realize I'm missing something or don't
like the name, change the library, update the documentation, repeat...
Consequently, something I thought would only take another half a day wound up taking 3 days, with some changes happening at the last moment (the System.Action`2 removal, thus removing the need to have two different builds depending on whether .NET 2.0 or .NET 3.0 is targeted).
- Improve monodocs2html so that the output sucks...well, less. I won't say
that it's actually good now, but it is an improvement. For comparison
purposes, NDesk.Options.OptionSet documentation output from:
In particular, note that in the 1.2.6 output that there are three examples as part of the Remarks documentation, but it's rather difficult to tell when one example ends and the next begins (due to the lack of an Example header). The svn version fixes this, adds JavaScript bling so that all sections can be collapsed and expanded, and adds a mini "index" to the top-left so that it's easier to get to the relevant sections (such as skipping the three lengthy examples that precede the Members listing).
I had wanted to do other things as well, such as migrate the monodoc-related programs to use NDesk.Options instead of Mono.GetOptions for option parsing, but such efforts will have to wait until later...
Announcing NDesk.Options 0.2.0
I am pleased to announce the release of NDesk.Options 0.2.0. NDesk.Options is a C# program option parser library, inspired by Perl's Getopt::Long option parser.
To download, visit the NDesk.Options web page:
http://www.ndesk.org/Options
Usage
See http://www.ndesk.org/Options and the OptionSet documentation for examples.
What's New?
There have been numerous changes since the previous 0.1.0 release:
- Mono 1.9 is now required to build. (An svn release was previously required anyway, so this isn't a surprising requirement.)
-
Simplify the API by removing all OptionSet.Add() methods which provided an OptionContext to the callback function; this includes:
- OptionSet.Add(string, Action<string, OptionContext>)
- OptionSet.Add(string, string, Action<string, OptionContext>)
- OptionSet.Add<T>(string, Action<T, OptionContext>)
- OptionSet.Add<T>(string, string, Action<T, OptionContext>)
If you really need access to an OptionContext, you can Add your own Option and override Option.OnParseComplete(OptionContext).
By Miguel's request, change the semantics for Options with optional values (arguments that have a type value of `:'). Previously, Options accepting an optional value were virtually identical to Options accepting a required value; the only place they would differ is at the end of the command line, where if a value was missing for a Option with an optional value no error would occur, while an Option with a required value would generate an error.
Now, we introduce the notion of greediness: required values are greedy, and will eat any number of following arguments in order to fulfill their requirements. Optional values are not greedy, and will only extract a value from the current argument.
By way of example:
string color = null; var p = new OptionSet () { { "-color:", v => color = v }, }; p.Parse (new string[]{"--color=auto"}); // 1 p.Parse (new string[]{"--color", "auto"}); // 2In NDesk.Options 0.1.0, (1) and (2) would be identical and color would be given the value auto. In 0.2.0, they are not identical: (1) would assign the value auto to color, while (2) would assign null to color. This permits consistency with GNU ls(1)'s ls --color behavior.
If a required option were to be specified (by using = instead of :), then (1) and (2) would again have identical results.
- NDesk.Options 0.1.0 restricted option bundling to boolean
Options. This restriction has been relaxed so that
(1) Options accepting both optional and required values may be
bundled with boolean Options, and (2) the optional or required
value may be bundled as well. As before, only single character
Options may be bundled.
The logic is as follows: given an argument such as -cvfname:
- cvfname must not match a registered Option. If it does match a registered option, then that is the Option that will (eventually) be invoked.
- c must be a registered option. If it isn't, then -cvfname is returned from OptionSet.Parse(IEnumerable<string>).
- Each character is looked up; if it's a boolean Option, then the associated action is invoked with a non-null value.
- If instead the character is an Option accepting one or more optional or required values, then the rest of the argument (not including the Option character) is used as the value. This also follows the greediness of optional vs. required values: optional values will only use the current argument, while required values may use the following argument(s) if e.g. the Option's character is the last character in the sequence.
- If a non-Option character is encountered that is not (a) the first character in the sequence, or (b) used as the value for a previous Option, then an OptionException is thrown.
This does The Right Thing for tar(1)-like option handling, with tar -cvfname ... creating (with verbose output) the file with the name name.
- Options may now accept (or require) more than one value. The
Option (string, string, int) constructor allows specifying how many
values are accepted/required (depending on whether the Option has
optional or required values).
The Option values are available through the OptionContext.OptionValues collection.
- Direct support for Options accepting/required two values within
OptionSet:
- OptionSet.Add(string, OptionAction<string, string>)
- OptionSet.Add(string, string, OptionAction<string, string>)
- OptionSet.Add<TKey, TValue> (string, OptionAction<TKey, TValue>)
- OptionSet.Add<TKey, TValue> (string, string, OptionAction<TKey, TValue>)
This now permits reasonable handling of cc(1)-style parameters:
var macros = new Dictionary<string, string> (); var p = new OptionSet () { { "D:", (k, v) => { if (k != null) macros.Add (k, v); } }, }; p.Parse (new string[]{"-DNAME1", "-DNAME2=VALUE2"}); // Adds the keys "NAME1" (with null value) // and "NAME2" (with value "VALUE2") to `macros'.Note that an optional value is used; if D= were specified, two values would be required, so -DNAME1 -DNAME2=VALUE2 would insert one macro -- NAME1 -- with the value -DNAME2=VALUE2.
- When an Option permits more than one value, it may provide a
list of value separator strings, strings that may be used to
separate the multiple values. If no separators are listed, = and
: are used as the default (thus permitting the previous
-DNAME2=VALUE2 example to work; -DNAME2:VALUE2 would
have had the same result).
The value separator strings follow the : or = in the Option prototype. They consist of:
- The string within { and }.
- Any other invidividual character.
Thus, the prototype M:+-*/ would use +, -, *, or / to split values, so -M5+2, -M5-2, -M5*2, and -M5/2 all provide two values (5 and 2) to the M option.
The prototype N={-->}{=>} would parse both -NA-->B and -NA=>B so that A and B are provided as the two values to the N option.
As a special construct, the separator {} requires that each value be a separate argument. (This makes no sense for Options with optional values.)
- Naming consistency improvements: an argument is an unparsed string, an option is a parsed argument that corresponds to a registered Option, and a prototype is a description of an Option, describing all aliases, the value type, and value separators. This has resulted in method argument name changes.
- Removal of .NET 3.5 support. Instead of using System.Action`2 from System.Core.dll (or providing an internal equivalent), I've just defined a OptionAction<TKey, TValue> type. This simplifies assembly versioning.
Unix Signal Handling In C#
In the beginning, Unix introduced signal(2), which permits a process to respond to external "stimuli", such as a keyboard interrupt (SIGINT), floating-point error (SIGFPE), dereferencing the NULL pointer (SIGSEGV), and other asynchronous events. And lo, it was...well, acceptable, really, but there wasn't anything better, so it at least worked. (Microsoft, when faced with the same problem of allowing processes to perform some custom action upon an external stimuli, invented Structured Exception Handling.)
Then, in a wrapping binge, I exposed it for use in C# with Stdlib.signal(), so that C# code could register signal handlers to be invoked when a signal occurred.
The problem? By their very nature, signals are asynchronous, so even in a single-threaded program, you had to be very careful about what you did, as your "normal" thread was certainly in the middle of doing something. For example, calling malloc(3) was almost certainly a bad idea, because if the process was in the middle of a malloc call already, you'd have a reentrant malloc call which could corrupt the heap.
This reentrant property impacts all functions in the process, including system calls. Consequently, a list of functions that were "safe" for invocation from signal handlers was standardized, and is listed in the above signal man page; it includes functions such as read(2) and write(2), but not functions like e.g. pwrite(2).
Consequently, these limitations and a few other factors led to the general recommendation that signal handlers should be as simple as possible, such as writing to global variable which the main program occasionally polls.
What's this have to do with Stdlib.signal(), and why was it a mistake to expose it? The problem is the P/Invoke mechanism, which allows marshaling C# delegates as a function pointer that can be invoked from native code. When the function pointer is invoked, the C# delegate is eventually executed.
However, before the C# delegate can be executed, a number of of steps needs to be done first:
- The first thing it does is to ensure the application domain for the thread where the signal handler executes actually matches the appdomain the delegate comes from, if it isn't it may need to set it and do several things that we can't guarantee are signal context safe...
- If the delegate is of an instance method we also need to retrieve the object reference, which may require taking locks...
In the same email, lupus suggests an alternate signal handling API that would be safe to use from managed code. Later, I provided a possible implementation. It amounts to treating the UnixSignal instance as a glorified global variable, so that it can be polled to see if the signal has been generated:
UnixSignal signal = new UnixSignal (Signum.SIGINT);
while (!signal.IsSet) {
/* normal processing */
}
There is also an API to permit blocking the current thread until the signal has been emitted (which also accepts a timeout):
UnixSignal signal = new UnixSignal (Signum.SIGINT);
// Wait for SIGINT to be generated within 5 seconds
if (signal.WaitOne (5000, false)) {
// SIGINT generated
}
Groups of signals may also be waited on:
UnixSignal[] signals = new UnixSignal[]{
new UnixSignal (Signum.SIGINT),
new UnixSignal (Signum.SIGTERM),
};
// block until a SIGINT or SIGTERM signal is generated.
int which = UnixSignal.WaitAny (signals, -1);
Console.WriteLine ("Got a {0} signal!", signals [which].Signum);
This isn't as powerful as the current Stdlib.signal() mechanism, but it is safe to use, doesn't lead to potentially ill-defined or unwanted behavior, and is the best that we can readily provide for use by managed code.
Mono.Unix.UnixSignal is now in svn-HEAD and the mono-1-9 branch, and should be part of the next Mono release.
Announcing NDesk.Options 0.1.0
I am pleased to announce the release of NDesk.Options 0.1.0. NDesk.Options is a C# program option parser library, inspired by Perl's Getopt::Long option parser.
To download, visit the NDesk.Options web page:
http://www.ndesk.org/Options
Usage
See http://www.ndesk.org/Options and the OptionSet documentation for examples.
What's New?
There have been numerous changes since the previous prototype release:
- Full member documentation.
- All errors are reported via OptionException.
- Options has been renamed to OptionSet.
- OptionSet.Parse(IEnumerable<string>) now returns a List<string> instead of an IEnumerable<string>.
- When a registered option follows an option requiring a value, the
registered option is used as the option value instead of triggering an
error. Thus:
var p = new OptionSet () { { "-n=", v => { /* ignore */ } }, { "-v", v => { /* ignore */ } }, }; p.Parse (new string[]{"-n", "-v"});would previously have triggered an exception, but now uses -v as the value of the -n option. This is consistent with Getopt::Long.
- TypeConverter exceptions are now wrapped within an OptionException, and the Message property contains a useful error message.
- OptionException message localization support.
- Add a OptionContext class that provides contextual information about the current option.
- Add a set of OptionSet.Add() methods that have callbacks that accept an OptionContext parameter.
- Add a set of virtual methods to Option and OptionSet to permit use by subclasses. The OptionSet class-level documentation has an example.
Mono and Mixed Mode Assembly Support
An occasional question on #mono@irc.gnome.org and ##csharp@irc.freenode.net is whether Mono will support mixed-mode assemblies, as generated by Microsoft's Managed Extensions for C++ compiler (Visual Studio 2001, 2003), and C++/CLI (Visual Studio 2005, 2008).
The answer is no, and mixed mode assemblies will likely never be supported.
Why?
First, what's a mixed mode assembly? A mixed mode assembly is an assembly that contains both managed (CIL) and unmanaged (machine language) code. Consequently, they are not portable to other CPU instruction sets, just like normal C and C++ programs and libraries.
Next, why use them? The primary purpose for mixed mode assemblies is as "glue", to e.g. use a C++ library class as a base class of a managed class. This allows the managed class to extend unmanaged methods, allowing the managed code to be polymorphic with respect to existing unmanaged functions. This is extremely useful in many contexts. However, as something like this involves extending a C++ class, it requires that the compiler know all about the C++ compiler ABI (name mangling, virtual function table generation and placement, exception behavior), and thus effectively requires native code. If the base class is within a separate .dll, this will also require that the mixed mode assembly list the native .dll as a dependency, so that the native library is also loaded when the assembly is loaded.
The other thing that mixed mode assemblies support is the ability to export new C functions so that other programs can LoadLibrary() the assembly and GetProcAddress the exported C function.
Both of these capabilities require that the shared library loader for the platform support Portable Executable (PE) files, as assemblies are PE files. If the shared library loader supports PE files, then the loader can ensure that when the assembly is loaded, all listed dependent libraries are also loaded (case 1), or that native apps will be able to load the assembly as if it were a native DLL and resolve DLL entry points against it.
This requirement is met on Windows, which uses the PE file format for EXE and DLL files. This requirement is not met on Linux, which uses ELF, nor is it currently met on Mac OS X, which uses Mach-O.
So why can't mixed mode assemblies be easily supported in Mono? Because ld.so doesn't like PE.
The only workarounds for this would be to either extend assemblies so that ELF files can contain both managed and unmanaged code, or to extend the shared library loader to support the loading of PE files. Using ELF as an assembly format may be useful, but would restrict portability of such ELF-assemblies to only Mono/Linux; .NET could never make use of them, nor could Mono on Mac OS X. Similarly, extending the shared library loader to support PE could be done, but can it support loading both PE and ELF (or Mach-O) binaries into a single process? What happens if a PE file loaded into an "ELF" process requires KERNEL32.DLL? Extending the shared library loader isn't a panacea either.
This limitation makes mixed mode assemblies of dubious value. It is likely solvable, but there are for more important things for Mono to focus on.
So you want to parse a command line...
If you develop command-line apps, parsing the command-line is a necessary evil (unless you write software so simple that it doesn't require any options to control its behavior). Consequently, I've written and used several parsing libraries, including Mono.GetOptions, Perl's Getopt::Long library, and some custom written libraries or helpers.
So what's wrong with them? The problem with Mono.GetOptions is that it has high code overhead: in order to parse a command line, you need a new type (which inherits from Mono.GetOptions.Options) and annotate each field or property within the type with an Option attribute, and let Mono.GetOptions map each command-line argument to a field/property within the Options subclass. See monodocer for an example; search for Opts to find the subclass.
The type-reflector parser is similarly code heavy, if only in a different way. The Mono.Fuse, lb, and omgwtf parsers are one-offs, either specific to a particular environment (e.g. integration with the FUSE native library) or not written with any eye toward reuse.
Which leaves Perl's Getopt::Long library, which I've used for a number of projects, and quite like. It's short, concise, requires no object overhead, and allows seeing at a glance all of the options supported by a program:
use Getopt::Long;
my $data = "file.dat";
my $help = undef;
my $verbose = 0;
GetOptions (
"file=s" => \$data,
"v|verbose" => sub { ++$verbose; },
"h|?|help" => $help
);
The above may be somewhat cryptic at first, but it's short, concise, and lets you know at a glance that it takes three sets of arguments, one of which takes a required string parameter (the file option).
So, says I, what would it take to provide similar support in C#? With C# 3.0 collection initializers and lambda delegates, I can get something that feels rather similar to the above GetOpt::Long code:
string data = null;
bool help = false;
int verbose = 0;
var p = new Options () {
{ "file=", (v) => data = v },
{ "v|verbose", (v) => { ++verbose } },
{ "h|?|help", (v) => help = v != null },
};
p.Parse (argv).ToArray ();
Options.cs has the goods, plus unit tests and additional examples (via the tests).
Options is both more and less flexible than Getopt::Long. It doesn't support providing references to variables, instead using a delegate to do all variable assignment. In this sense, Options is akin to Getopt::Long while requiring that all options use a sub callback (as the v|verbose option does above).
Options is more flexible in that it isn't restricted to just strings, integers, and floating point numbers. If there is a TypeConverter registered for your type (to perform string->object conversions), then any type can be used as an option value. To do so, merely declare that type within the callback:
int count = 0;
var p = new Options () {
{ "c|count=", (int v) => count = v },
};
As additional crack, you can provide an (optional) description of the option so that Options can generate help text for you:
var p = new Options () {
{ "really-long-option", "description", (v) => {} },
{ "h|?|help", "print out this message and exit", (v) => {} },
};
p.WriteOptionDescriptions (Console.Out);
would generate the text:
--really-long-option description
-h, -?, --help print out this message and exit
Options currently supports:
- Parameters of the form: -flag, --flag, /flag, -flag=value, --flag=value, /flag=value, -flag:value, --flag:value, /flag:value, -flag value, --flag value, /flag value.
- "boolean" parameters of the form: -flag, --flag, and /flag. Boolean parameters can have a `+' or `-' appended to explicitly enable or disable the flag (in the same fashion as mcs -debug+). For boolean callbacks, the provided value is non-null for enabled, and null for disabled.
- "value" parameters with a required value (append `=' to the option name) or an optional value (append `:' to the option name). The option value can either be in the current option (--opt=value) or in the following parameter (--opt value). The actual value is provided as the parameter to the callback delegate, unless it's (1) optional and (2) missing, in which case null is passed.
- "bundled" parameters which must start with a single `-' and consists of only single characters. In this manner, -abc would be a shorthand for -a -b -c.
- Option processing is disabled when -- is encountered.
All un-handled parameters are returned from the Options.Parse method, which is implemented as an iterator (hence the calls to .ToArray() in the above C# examples, to force processing).
In Defense Of git
On Friday at the OpenOffice.org Conference, we had two sessions discussing the future of Source Code Managers in OpenOffice.org: Child workspaces and the OOo SCM system by Jens-Heiner Rechtien and git: the Source Code Manager for OOo? by Jan Holesovsky (kendy).
In the Q&A section after the git presentation, there was a lot of heated debate in which it seemed that Jan and Jens were talking "past" each other. As a git backer, I thought I'd try to bring some clarity to things.
It seemed that Jens has one fundamental problem with git, which itself is fundamental to its operation: commits are not transferred to the remote module; instead, you need an explicit git-push command to send all local changes to the remote repository. Jens claimed three implications of this (that I remember):
- git did not permit line-by-line authorship information, as with cvs annotate or svn blame.
- Developers would not see changes made by other developers as soon as they happen.
- QA and Release Engineering wouldn't be alerted as soon as developers made any change on any child workspace.
The line-by-line authorship information is possible in git with the git blame or git annotate commands (they are synonyms for each other). I suspect I misinterpreted this part of the debate, as all parties should have known that git supported this.
Which leaves the other two issues, which (again) are fundamental to git: a commit does not send any data to the repository. Thus we get to the title of this blog entry: this is a Good Thing™.
Local commits are world changing in a very small way: they're insanely fast, much faster than Subversion. (For example, committing a one-line change to a text file under a Subversion remote directory took me 4.775s; a similar change under git is 0.246s -- 19x faster -- and this is a small Subversion module, ~1.5MB, hosted on the ximian.com Subversion repo, which never seems as loaded as the openoffice.org servers.)
What can you do when your commits are at least 19x faster? You commit more often. You commit when you save your file (or soon thereafter). You commit when you code is 99.995% guaranteed to be WRONG.
Why do this? Because human memory is limited. Most studies show that the average person can remember 7±2 items at a time before they start forgetting things. This matters because a single bug may require changes to multiple different files, and even within a single file your memory will be filled with such issues as what's the scope of this variable?, what's the type of this variable?, what's this method do?, what bug am I trying to fix again?, etc. Human short-term memory is very limited.
So what's the poor developer to do? Most bugs can be partitioned in some way, e.g. into multiple methods or blocks of code, and each such block/sub-problem is solved sequentially -- you pick one sub-problem, solve it, test it (individually if possible), and continue to the next sub-problem. During this process and when you're finished you'll review the patch (is it formatted nicely?, could this code be cleaned up to be more maintainable?), then finally commit your single patch to the repository. It has to be done this way because if you commit at any earlier point in time, someone else will get your intermediate (untested) changes, and you'll break THEIR code flow. This is obviously bad.
During this solve+test cycle, I frequently find that I'll make a set of changes to a file, save it, make other changes, undo them, etc. I never close my file, because (and here's the key point) cvs diff shows me too many changes. It'll show me the changes I made yesterday as well as the changes I made 5 minutes ago, and I need to keep those changes separate -- the ones from yesterday (probably) work, the ones from 5 minutes ago (probably) don't, and the only way I can possibly remember which is the set from 5 minutes ago is to hit Undo in my editor and find out. :-)
So git's local commits are truly world-changing for me: I can commit something as soon as I have it working for a (small) test case, at which point I can move on to related code and fix that sub-problem, even (especially) if it's a change in the same file. I need an easy way to keep track of which are the solved problems (the stuff I fixed yesterday) and the current problem. I need this primarily because the current problem filled my 7±2 memory slots, and I'm unable to easily remember what I did yesterday. (I'm only human! And "easily remember" means "takes less than 0.1s to recall." If you need to think you've already lost.)
This is why I think the other two issues -- developers don't see other changes instantly, and neither does QA -- are a non-issue. It's a feature.
So let's bring in a well-used analogy to programming: writing a book. You write a paragraph, spell check it, save your document, go onto another paragraph/chapter, repeat for a bit, then review what was written. At any part of this process, you'll be ready to Undo your changes because you changed your mind. Changes may need to occur across the entire manuscript.
Remote commits are equivalent to sending each saved manuscript to the author's editor. If someone is going to review/use/depend upon your change, you're going to Damn Well make sure that it Works/is correct before you send that change.
Which brings us to the workflow dichotomy between centralized source code managers (cvs, svn) and distributed managers (git et. al). Centralized source managers by design require more developer effort, because the developer needs to manually track all of the individual changes of a larger work/patch before sending it upstream (as described above).
Decentralized source managers instead help the developer with the tedious effort of tracking individual changes, because the developer can commit without those changes being seen/used by anyone else. The commit instead gets sent when the developer is done with the feature.
This is why I prefer git to Subversion. git allows me to easily work with my 7±2 short-term memory limitations, by allowing me to commit "probably working but not fully tested" code so that I don't need to review those changes at the next cvs diff for the current problem I'm working on.
Comparing Java and C# Generics
Or, What's Wrong With Java Generics?
What Are Generics
Java 5.0 and C# 2.0 have both added Generics, which permit a multitude of things:
- Improved compiler-assisted checking of types.
- Removal of casts from source code (due to (1)).
- In C#, performance advantages (discussed later).
This allows you to replace the error-prone Java code:
List list = new ArrayList ();
list.add ("foo");
list.add (new Integer (42)); // added by "mistake"
for (Iterator i = list.iterator (); i.hasNext (); ) {
String s = (String) i.next ();
// ClassCastException for Integer -> String
// work on `s'
System.out.println (s);
}
with the compiler-checked code:
// constructed generic type
List<String> list = new ArrayList<String> ();
list.add ("foo");
list.add (42); // error: cannot find symbol: method add(int)
for (String s : list)
System.out.println (s);
The C# equivalent code is nigh identical:
IList<string> list = new List<string> ();
list.Add ("foo");
list.Add (42); // error CS1503: Cannot convert from `int' to `string'
foreach (string s in list)
Console.WriteLine (s);
Terminology
A Generic Type is a type (classes and interfaces in Java and C#, as well as delegates and structs in C#) that accepts Generic Type Parameters. A Constructed Generic Type is a Generic Type with Generic Type Arguments, which are Types to actually use in place of the Generic Type Parameters within the context of the Generic Type.
For simple generic types, Java and C# have identical syntax for declaring and using Generic Types:
class GenericClass<TypeParameter1, TypeParameter2>
{
public static void Demo ()
{
GenericClass<String, Object> c =
new GenericClass<String, Object> ();
}
}
In the above, GenericClass is a Generic Type, TypeParameter1 and TypeParameter2 are Generic Type Parameters for GenericClass, and GenericClass<String, Object> is a Constructed Generic Type with String as a Generic Type Argument for the TypeParameter1 Generic Type Parameter, and Object as the Generic Type Argument for the TypeParameter2 Generic Type Parameter.
It is an error in C# to create a Generic Type without providing any Type Arguments. Java permits creating Generic Types without providing any Type Arguments; these are called raw types:
Map rawMap = new HashMap <String, String> ();
Java also permits you to leave out Generic Type Arguments from the right-hand-side. Both raw types and skipping Generic Type Arguments elicit a compiler warning:
Map<String, String> correct = new HashMap<String, String> ();
// no warning, lhs matches rhs
Map<String, String> incorrect = new HashMap ();
// lhs doesn't match rhs; generates the warning:
// Note: gen.java uses unchecked or unsafe operations.
// Note: Recompile with -Xlint:unchecked for details.
Compiling the above Java code with -Xlint:unchecked produces:
gen.java:9: warning: [unchecked] unchecked conversion
found : java.util.HashMap
required: java.util.Map<java.lang.String,java.lang.String>
Map<String, String> incorrect = new HashMap ();
Note that all "suspicious" code produces warnings, not errors, under Java. Only provably wrong code generate compiler errors (such as adding an Integer to a List<String>).
(Also note that "suspicious" code includes Java <= 1.4-style use of collections, i.e. all collections code that predates Java 5.0. This means that you get lots of warnings when migrating Java <= 1.4 code to Java 5.0 and specifying -Xlint:unchecked.)
Aside from the new use of `<', `>', and type names within constructed generic type names, the use of generic types is essentially identical to the use of non-generic types, though Java has some extra flexibility when declaring variables.
Java has one added wrinkle as well: static methods of generic classes cannot reference the type parameters of their enclosing generic class. C# does not have this limitation:
class GenericClass<T> {
public static void UseGenericParameter (T t) {}
// error: non-static class T cannot be
// referenced from a static context
}
class Usage {
public static void UseStaticMethod () {
// Valid C#, not valid Java
GenericClass<int>.UseGenericParameter (42);
}
}
Generic Methods
Java and C# both support generic methods, in which a (static or instance) method itself accepts generic type parameters, though they differ in where the generic type parameters are declared. Java places the generic type parameters before the method return type:
class NonGenericClass {
public static <T> T max (T a, T b) {/*...*/}
}
while C# places them after the method name:
class NonGenericClass {
static T Max<T> (T a, T b) {/*...*/}
}
Generic methods may exist on both generic- and non-generic classes and interfaces.
Constraints
What can you do with those Generic Type Parameters within the class or method body? Not much:
- Declare variables using the generic type parameter in most nested scopes (e.g. instance class member, method parameter, local variable).
- Invoke methods that exist on java.lang.Object (Java) or System.Object (C#).
- Assign the default value to variables with the generic parameter type. The default value is null in Java, and default(Generic Type Parameter) in C#. (C# requires this syntax because null isn't a valid value for value types such as int.)
class GenericJavaClass<T> {
T[] arrayMember = null;
T singleMember = null;
public static void Demo ()
{
T localVariable = 42; // error
T localVariable2 = null;
AcceptGenericTypeParameter (localVariable);
}
public static void AcceptGenericTypeParameter (T t)
{
System.out.println (t.toString ()); // ok
System.out.println (t.intValue ());
// error: cannot find symbol
}
}
class GenericCSharpClass<T> {
T[] arrayMember = null;
T singleMember = default(T);
public static void Demo ()
{
T localVariable = 42; // error
T localVariable2 = default(T);
AcceptGenericTypeParameter (localVariable);
}
public static void AcceptGenericTypeParameter (T t)
{
System.out.println (t.ToString ()); // ok
System.out.println (t.GetTypeCode ()); // error: cannot find symbol
}
}
So how do we call non-Object methods on objects of a generic type parameter?
- Cast the variable to a type that has the method you want (and accept the potentially resulting cast-related exceptions).
- Place a constraint on the generic type parameter. A constraint is a compile-time assertion that the generic type argument will fulfill certain obligations. Such obligations include the base class of the generic type argument, any implemented interfaces of the generic type argument, and (in C#) whether the generic type argument's type has a default constructor, is a value type, or a reference type.
Java Type Constraints
Java type and method constraints are specified using a "mini expression language" within the `<' and `>' declaring the generic type parameters. For each type parameter that has constraints, the syntax is:
TypeParameter ListOfConstraints
Where ListOfConstraints is a `&'-separated list of one of the following constraints:
- Specifying a base class or implemented interface on the Generic Type Argument by using: extends BaseOrInterfaceType
(`&' must be used instead of `,' because `,' separates each generic type parameter.)
The above constraints also apply to methods, and methods can use some additional constraints described below.
class GenericClass<T extends Number & Comparable<T>> {
void print (T t) {
System.out.println (t.intValue ()); // OK
}
}
class Demo {
static <U, T extends U>
void copy (List<T> source, List<U> dest) {
for (T t : source)
dest.add (t);
}
static void main (String[] args) {
new GenericClass<Integer>().print (42);
// OK: Integer extends Number
new GenericClass<Double>().print (3.14159);
// OK: Double extends Number
new GenericClass<String>().print ("string");
// error: <T>print(T) in gen cannot be applied
// to (java.lang.String)
ArrayList<Integer> ints = new ArrayList<Integer> ();
Collections.addAll (ints, 1, 2, 3);
copy (ints, new ArrayList<Object> ());
// OK; Integer inherits from Object
copy (ints, new ArrayList<String> ());
// error: <U,T>copy(java.util.List<T>,
// java.util.List<U>) in cv cannot be
// applied to (java.util.ArrayList<java.lang.Integer>,
// java.util.ArrayList<java.lang.String>)
}
}
C# Constraints
C# generic type parameter constraints are specified with the context-sensitive where keyword, which is placed after the class name or after the method's closing `)'. For each type parameter that has constraints, the syntax is:
where TypeParameter : ListOfConstraints
Where ListOfConstraints is a comma-separated list of one of the following constraints:
- Specifying inheritance relationships, such as a mandatory base class or interface by listing the type name or another generic type parameter.
- Requiring that the type argument be a value type by specifying struct.
- Requiring that the type argument be a reference type by using class.
- Requiring that the type argument provide a default constructor by using new().
class GenericClass<T> : IComparable<GenericClass<T>>
where T : IComparable<T>
{
private GenericClass () {}
void Print (T t)
{
Console.WriteLine (t.CompareTo (t));
// OK; T must implement IComparable<T>
}
public int CompareTo (GenericClass<T> other)
{
return 0;
}
}
class Demo {
static void OnlyValueTypes<T> (T t)
where T : struct
{
}
static void OnlyReferenceTypes<T> (T t)
where T : class
{
}
static void Copy<T, U> (IEnumerable<T> source,
ICollection<U> dest)
where T : U, IComparable<T>, new()
where U : new()
{
foreach (T t in source)
dest.Add (t);
}
static T CreateInstance<T> () where T : new()
{
return new T();
}
public static void Main (String[] args)
{
new GenericClass<int>.Print (42);
// OK: Int32 implements IComparable<int>
new GenericClass<double>.Print (3.14159);
// OK: Double implements IComparable<double>
new GenericClass<TimeZone>.Print (
TimeZone.CurrentTimeZone);
// error: TimeZone doesn't implement
// IComparable<TimeZone>
OnlyValueTypes (42); // OK: int is a struct
OnlyValueTypes ("42");
// error: string is a reference type
OnlyReferenceTypes (42);
// error: int is a struct
OnlyReferenceTypes ("42"); // OK
CreateInstance<int> ();
// OK; int has default constructor
CreateInstance<GenericClass<int>> ();
// error CS0310: The type `GenericClass<int>'
// must have a public parameterless constructor
// in order to use it as parameter `T' in the
// generic type or method
// `Test.CreateInstance<T>()'
// In theory, you could do `Copy' instead of
// `Copy<...>' below, but it depends on the
// type inferencing capabilities of your compiler.
Copy<int,object> (new int[]{1, 2, 3},
new List<object> ());
// OK: implicit int -> object conversion exists.
Copy<int,AppDomain> (new int[]{1, 2, 3},
new List<AppDomain> ());
// error CS0309: The type `int' must be
// convertible to `System.AppDomain' in order
// to use it as parameter `T' in the generic
// type or method `Test.Copy<T,U>(
// System.Collections.Generic.IEnumerable<T>,
// System.Collections.Generic.ICollection<U>)'
}
}
Java Wildcards (Java Method Constraints)
Java has additional support for covarient- and contravariant generic types on method declarations.
By default, you cannot assign an instance of one constructed generic type to an instance of another generic type where the generic type arguments differ:
// Java, though s/ArrayList/List/ for C# List<String> stringList = new ArrayList<String> (); List<Object> objectList = stringList; // error
The reason for this is quite obvious with a little thought: if the above were permitted, you could violate the type system:
// Assume above...
stringList.add ("a string");
objectList.add (new Object ());
// and now `stringList' contains a non-String object!
This way leads madness and ClassCastExceptions. :-)
However, sometimes you want the flexibility of having different generic type arguments:
static void cat (Collection<Reader> sources) throws IOException {
for (Reader r : sources) {
int c;
while ((c = r.read()) != -1)
System.out.print ((char) c);
}
}
Many types implement Reader, e.g. StringReader and FileReader, so we might want to do this:
Collection<StringReader> sources =
new Collection<StringReader> ();
Collections.addAll (sources,
new StringReader ("foo"),
new StringReader ("bar"));
cat (sources);
// error: cat(java.util.Collection<java.io.Reader>)
// in gen cannot be applied to
// (java.util.Collection<java.io.StringReader>)
There are two ways to make this work:
- use Collection<Reader> instead of
Collection<StringReader>:
Collection<Reader> sources = new Collection<Reader> (); Collections.addAll (sources, new StringReader ("foo"), new StringReader ("bar")); cat (sources); - Use wildcards.
Unbounded Wildcards
If you don't care about the specific generic type arguments involved, you can use `?' as the type parameter. This is an unbounded wildcard, because the `?' can represent anything:
static void printAll (Collection<?> c) {
for (Object o : c)
System.out.println (o);
}
The primary utility of unbounded wildcards is to migrate pre-Java 5.0 collection uses to Java 5.0 collections (thus removing the probably thousands of warnings -Xlint:unchecked produces) in the easiest manner.
This obviously won't help for cat (above), but it's also possible to "bind" the wildcard, to create a bounded wildcard.
Bounded Wildcards
You create a bounded wildcard by binding an upper- or lower- bound to an unbounded wildcard. Upper bounds are specified via extends, while lower bounds are specified via super. Thus, to allow a Collection parameter that accepts Reader instances or any type that derives from Reader:
static void cat (Collection<? extends Reader> c)
throws IOException
{
/* as before */
}
This permits the more desirable use:
Collection<StringReader> sources =
new Collection<StringReader> ();
Collections.addAll (sources,
new StringReader ("foo"),
new StringReader ("bar"));
cat (sources);
Bounded wildcards also allow you to reduce the number of generic parameters you might otherwise want/need a generic method; compare this Demo.copy to the previous Java Demo.copy implementation:
class Demo {
static <T> void copy (List<? extends T> source,
List<? super T> dest)
{
for (T t : source)
dest.add (t);
}
}
C# Equivalents
C# has no direct support for bounded or unbounded wildcards, and thus doesn't permit declaring class- or method-level variables that make use of them. However, if you can make the class/method itself generic, you can create equivalent functionality.
A Java method taking an unbounded wildcard would be mapped to a generic C# method with one generic type parameter for each unbound variable within the Java method:
static void PrintAll<T> (IEnumerable<T> list) {
foreach (T t in list) {
Console.WriteLine (t);
}
}
This permits working with any type of IEnumerable<T>, e.g. List<int> and List<string>.
A Java method taking an upper bounded wildcard can be mapped to a generic C# method with one generic type parameter for each bound variable, then using a derivation constraint on the type parameter:
static void Cat<T> (IList<T> sources)
where T : Stream
{
for (Stream s : sources) {
int c;
while ((c = r.ReadByte ()) != -1)
Console.Write ((char) c);
}
}
A Java method taking a lower bounded wildcard can be mapped to a generic C# method taking two generic type parameters for each bound variable (one is the actual type you care about, and the other is the super type), then using a derivation constraint between your type variables:
static void Copy<T,U> (IEnumerable<T> source,
ICollection<U> dest)
where T : U
{
foreach (T t in source)
dest.Add (t);
}
Generics Implementation
How Java and C# implement generics has a significant impact on what generics code can do and what can be done at runtime.
Java Implementation
Java Generics were originally designed so that the .class file format wouldn't need to be changed. This would have meant that Generics-using code could run unchanged on JDK 1.4.0 and earlier JDK versions.
However, the .class file format had to change anyway (for example, generics permits you to overload methods based solely on return type), but they didn't revisit the design of Java Generics, so Java Generics remains a compile-time feature based on Type Erasure.
Sadly, you need to know what type erasure is in order to actually write much generics code.
With Type Erasure, the compiler transforms your code in the following manner:
- Generic types (classes and interfaces) retain the same name, so you cannot have a generic class Foo and a non-generic Foo<T> in the same package -- these are the same type. This is the raw type.
- All instances of generic types become their corresponding raw type. So a List<String> becomes a List. (Thus all "nested" uses of generic type parameters -- in which the generic type parameter is used as a generic type argument of another generic type -- are "erased".)
- All instances of generic type parameters in both class and method
scope become instances of their closest matching type:
- If the generic type parameter has an extends constraint, then instances of the generic type parameter become instances of the specified type.
- Otherwise, java.lang.Object is used.
- Generic methods also retain the same name, and thus there cannot be any overloading of methods between those using generic type parameters (after the above translations have occurred) and methods not using generic type parameters (see below for example).
- Runtime casts are inserted by the compiler to ensure that the runtime types are what you think they are. This means that there is runtime casting that you cannot see (the compiler inserts the casts), and thus generics confer no performance benefit over non-generics code.
For example, the following generics class:
class GenericClass<T, U extends Number> {
T tMember;
U uMember;
public T getFirst (List<T> list) {
return list.get (0);
}
// in bytecode, this is overloading based on return type
public U getFirst (List<U> list) {
return list.get (0);
}
//
// This would be an error -- doesn't use generic type parameters
// and has same raw argument list as above two methods:
//
// public Object getFirst (List list) {
// return list.get (0);
// }
//
public void printAll (List<U> list) {
for (U u : list) {
System.out.println (u);
}
}
}
Is translated by the compiler into the equivalent Java type:
class GenericClass {
Object tMember;
Number uMember; // as `U extends Number'
public Object getFirst (List list) {
return list.get (0);
}
public Number getFirst (List list) {
// note cast inserted by compiler
return (Number) list.get (0);
}
public void printAll (List list) {
for (Iterator i=list.iterator (); i.hasNext (); ) {
// note cast inserted by compiler
Number u = (Number) i.next ();
System.out.println (u);
}
}
}
.NET Implementation
.NET adds a number of new instructions to its intermediate language to support generics. Consequently generics code cannot be directly used by languages that do not understand generics, though many generic .NET types also implement the older non-generic interfaces so that non-generic languages can still use generic types, if not directly.
The extension of IL to support generics permits type-specific code generation. Generic types and methods can be constructed over both reference (classes, delegates, interfaces) and value types (structs, enumerations).
Under .NET, there will be only one "instantiation" (JIT-time code generation) of a class which will be used for all reference types. (This can be done because (1) all reference types have the same representation as local variables/class fields, a pointer, and (2) generics code has a different calling convention in which additional arguments are implicitly passed to methods to permit runtime type operations.) Consequently, a List<string> and a List<object> will share JIT code.
No additional implicit casting is necessary for this code sharing, as the IL verifier will prevent violation of the type system. It is not Java Type Erasure.
Value types will always get a new JIT-time instantiation, as the sizes of value types will differ. Consequently, List<int> and List<short> will not share JIT code.
Currently, Mono will always generate new instantiations for generic types, for both value and reference types (i.e. JIT code is never shared). This may change in the future, and will have no impact on source code/IL. (It will impact runtime performance, as more memory will be used.)
However, there are still some translations performed by the compiler, These translations have been standardized for Common Language Subset use, though these specific changes are not required:
The actual type name for generic types is the original type name, followed by ``' and the number of generic type parameters. Thus List<T> has the IL name List`1. This allows multiple different generic types to share the same name as long as they have a different number of generic type parameters, e.g. you can have the types Foo, Foo<T>, and Foo<T,U> all in the same namespace.
This impacts type lookup through reflection: Type.GetType("System.Collections.Generic.List") will fail, while Type.GetType("System.Collections.Generic.List`1") works.
- The actual method name for generic methods is unchanged, though in XML Documentation purposes it is modified to have ```' and the number of generic type parameters appended to the method name.
Runtime Environment
Generics implementations may have some additional runtime support.
Java Runtime Environment
Java generics are a completely compile-time construct. You cannot do anything with generic type parameters that rely in any way on runtime information. This includes:
- Creating instances of generic type parameters.
- Creating arrays of generic type parameters.
- Quering the runtime class of a generic type parameter.
- Using instanceof with generic type parameters.
In short, all of the following produce compiler errors in Java:
static <T> void genericMethod (T t) {
T newInstance = new T (); // error: type creation
T[] array = new T [0]; // error: array creation
Class c = T.class; // error: Class querying
List<T> list = new ArrayList<T> ();
if (list instanceof List<String>) {}
// error: illegal generic type for instanceof
}
Array Usage
The above has some interesting implications on your code. For example, how would you create your own type-safe collection (i.e. how is ArrayList<T> implemented)?
By accepting the unchecked warning -- you cannot remove the warning. Fortunately you'll only see the warning when you compile your class, and users of your class won't see the unchecked warnings within your code. There are two ways to do it, the horribly unsafe way and the safe way.
The horribly unsafe way works for simple cases:
static <T> T[] unsafeCreateArray (T type, int size) {
return (T[]) new Object [size];
}
This seems to work for typical generics code:
static <T> void seemsToWork (T t) {
T[] array = unsafeCreateArray (t, 10);
array [0] = t;
}
But it fails horribly if you ever need to use a non-generic type:
static void failsHorribly () {
String[] array = unsafeCreateArray ((String) null, 10);
// runtime error: ClassCastException
}
The above works if you can guarantee that the created array will never be cast to a non-Object array type, so it's useful in some limited contexts (e.g. implementing java.util.ArrayList), but that's the extent of it.
If you need to create the actual runtime array type, you need to use java.lang.reflect.Array.newInstance and java.lang.Class<T>:
static <T> T[] safeCreateArray (Class<T> c, int size) {
return (T[])java.lang.reflect.Array.newInstance(c,size);
}
static void actuallyWorks () {
String[] a1 = safeCreateArray(String.class, 10);
}
Note that this still generates a warning by the compiler, but no runtime exception will occur.
C# Runtime Environment
.NET provides extensive runtime support for generics code, permitting you to do everything that Java doesn't:
static void GenericMethod<T> (T t)
where T : new()
{
T newInstance = new T (); // OK - new() constraint.
T[] array = new T [0]; // OK
Type type = typeof(T); // OK
List<T> list = new List<T> ();
if (list is List<String>) {} // OK
}
C# also has extensive support for querying generic information at runtime via System.Reflection, such as with System.Type.GetGenericArguments().
What C# doesn't support is non-default constructor declaration, non-interface or base-type method declaration, and static method declaration. Since operator overloading is based on static methods, this means that you cannot generically use arithmetic unless you introduce your own interface to perform arithmetic:
// This is what I'd like:
class Desirable // NOT C#
{
public static T Add<T> (T a, T b)
where T : .op_Addition(T,T)
{
return a + b;
}
}
// And this is what we currently need to do:
interface IArithmeticOperations<T> {
T Add (T a, T b);
// ...
}
class Undesirable {
public static T Add<T> (IArithmeticOperations<T> ops,
T a, T b)
{
return ops.Add (a, b);
}
}
Summary
The generics capabilities in Java and .NET differ significantly. Syntax wise, Java and C# generics initially look quite similar, and share similar concepts such as constraints. The semantics of generics is where they differ most, with .NET permitting full runtime introspection of generic types and generic type parameters in ways that are obvious in their utility (instance creation, array creation, performance benefits for value types due to lack of boxing) and completely lacking in Java.
In short, all that Java generics permit is greater type safety with no new capabilities, with an implementation that permits blatant violation of the type system with nothing more than warnings:
List<String> stringList = new ArrayList<String> ();
List rawList = stringList;
// only triggers a warning
List<Object> objectList = rawList;
// only triggers a warning
objectList.add (new Object ());
for (String s : stringList)
System.out.println (s);
// runtime error: ClassCastException due to Object.
This leads to the recommendation that you remove all warnings from your code, but if you try to do anything non-trivial (apparently typesafe arrays is non-trivial), you get into scenarios where you cannot remove all warnings.
Contrast this with C#/.NET, where the above code isn't possible, as there are no raw types, and converting a List<string> to a List<object> would (1) require an explicit cast (as opposed to the complete lack of casts in the above Java code), and (2) generate an InvalidCastException at runtime from the explicit cast.
Furthermore, C#/.NET convey additional performance benefits due to the lack of required casts (as the verifier ensures everything is kosher) and support for value types (Java generics don't work with the builtin types like int), thus removing the overhead of boxing, and C# permits faster, more elegant, more understandable, and more maintainable code.
Links
- http://java.sun.com/j2se/1.5.0/docs/guide/language/generics.html
- http://java.sun.com/j2se/1.5/pdf/generics-tutorial.pdf
- http://www.angelikalanger.com/GenericsFAQ/JavaGenericsFAQ.pdf
- http://tromey.com/blog/?p=297
- http://java.sun.com/j2se/1.5.0/docs/api/
- http://java.sun.com/j2se/1.5.0/docs/guide/language/
Problems with Traditional Object Oriented Ideas
I've been training with Michael Meeks, and he gave Hubert and I an overview of the history of OpenOffice.org.
One of the more notable comments was the binfilter module, which is a stripped-down copy of StarOffice 5.2 (so if you build it you wind up with an ancient version of StarOffice embedded within your current OpenOffice.org build).
Why is a embedded StarOffice required? Because of mis-informed "traditional" Object Oriented practice. :-)
Frequently in program design, you'll need to save state to disk and read it back again. Sometimes this needs to be done manually, and sometimes you have a framework to help you (such as .NET Serialization). Normally, you design the individual classes to read/write themselves to external storage. This has lots of nice benefits, such as better encapsulation (the class doesn't need to expose it's internals), the serialization logic is in the class itself "where it belongs," etc. It's all good.
Except it isn't. By tying the serialization logic to your internal data structures, you severely reduce your ability to change your internal data structures for optimization, maintenance, etc.
Which is why OpenOffice.org needs to embed StarOffice 5.2: the StarOffice 5.2 format serialized internal data structures, but as time went on they wanted to change the internal structure for a variety of reasons, The result: they couldn't easily read or write their older storage format without having a copy of the version of StarOffice that generated that format.
The take away from this is that if you expect your software to change in any significant way (and why shouldn't you?), then you should aim to keep your internal data structures as far away from your serialization format as possible. This may complicate things, or it may require "duplicating" code (e.g. your real data structure, and then a [Serializable] version of the "same" class -- with the data members but not the non-serialization logic -- to be used when actually saving your state), but failure to do so may complicate future maintenance.
(Which is why Advanced .NET Remoting suggests thinking about serialization formats before you publish your first version...)
Re-Introducing monodocer
In the beginning... Mono was without documentation. Who needed it when Microsoft had freely available documentation online? (That's one of the nice things about re-implementing -- and trying to stay compatible with -- a pre-existing project: reduced documentation requirements. If you know C# under .NET, you can use C# under Mono, by and large, so just take an existing C# book and go on your way...)
That's not an ideal solution, as MSDN is/was slow. Very slow. Many seconds to load a single page slow. (And if you've ever read the .NET documentation on MSDN where it takes many page views just to get what you're after... You might forget what you're looking for before you find it.) A local documentation browser is useful.
Fortunately, the ECMA 335 standard comes to the rescue (somewhat): it includes documentation for the types and methods which were standardized under ECMA, and this documentation is freely available and re-usable.
The ECMA documentation consists of a single XML file (currently 7.2MB) containing all types and type members. This wasn't an ideal format for writing new documentation, so the file was split up into per-type files; this is what makes up the monodoc svn module (along with many documentation improvements since, particularly types and members that are not part of the ECMA standard.
However, this ECMA documentation import was last done many years ago, and the ECMA documentation has improved since then. (In particular, it now includes documentation for many types/members added in .NET 2.0.) We had no tools to import any updates.
Monodocer
Shortly after the ECMA documentation was originally split up into per-type files, Mono needed a way to generate documentation stubs for non-ECMA types within both .NET and Mono-specific assemblies. This was (apparently) updater.exe.
Eventually, Joshua Tauberer created monodocer, which both creates ECMA-style documentation stubs (in one file/type format) and can update documentation based on changes to an assembly (e.g. add a new type/member to an assembly and the documentation is updated to mention that new type/member).
By 2006, monodocer had (more-or-less) become the standard the generating and updating ECMA-style documentation, so when I needed to write Mono.Fuse documentation I used monodocer...and found it somewhat lacking in support for Generics. Thus begins my work on improving monodocer.
monodocer -importecmadoc
Fast-forward to earlier this year. Once monodocer could support generics, we could generate stubs for all .NET 2.0 types. Furthermore, ECMA had updated documentation for many core .NET 2.0 types, so...what would it take to get ECMA documentation re-imported?
This turned out to be fairly easy, with supported added in mid-May to import ECMA documentation via a -importecmadoc:FILENAME parameter. The problem was that this initial version was slow; quoting the ChangeLog, "WARNING: import is currently SLOW." How slow? ~4 Minutes to import documentation for System.Array.
This might not be too bad, except that there are 331 types in the ECMA documentation file, documenting 3797 members (fields, properties, events, methods, constructors, etc.). 4 minutes per type is phenominally slow.
Optimizing monodocer -importecmadoc
Why was it so slow? -importecmadoc support was originally modeled after -importslashdoc support, which is as follows: lookup every type and member in System.Reflection order, create an XPath expression for this member, and execute an XPath query against the documentation we're importing. If we get a match, import the found node.
The slowdown was twofold: (1) we loaded the entire ECMA documentation into a XmlDocument instance (XmlDocument is a DOM interface, and thus copies the entire file into memory), and (2) we were then accessing the XmlDocument randomly.
The first optimization is purely algorithmic: don't import documentation in System.Reflection order, import it in ECMA documentation order. This way, we read the ECMA documentation in a single pass, instead of randomly.
As is usually the case, algorithmic optimizations are the best kind: it cut down the single-type import from ~4 minutes to less than 20 seconds.
I felt that this was still too slow, as 20s * 331 types is nearly 2 hours for an import. (This is actually faulty reasoning, as much of that 20s time was to load the XmlDocument in the first place, which is paid for only once, not for each type.) So I set out to improve things further.
First was to use a XPathDocument to read the ECMA documentation. Since I wasn't editing the document, I didn't really need the DOM interface that XmlDocument provides, and some cursory tests showed that XPathDocument was much faster than XmlDocument for parsing the ECMA documentation (about twice as fast). This improved things, cutting single-type documentation import from ~15-20s to ~10-12s. Not great, but better.
Convinced that this still wasn't fast enough, I went to the only faster XML parser within .NET: XmlTextReader, which is a pull-parser lacking any XPath support. This got a single-file import down to ~7-8s.
I feared that this would still need ~45 minutes to import, but I was running out of ideas so I ran a full documentation import for mscorlib.dll to see what the actual runtime was. Result: ~2.5 minutes to import ECMA documentation for all types within mscorlib.dll. (Obviously the ~45 minute estimate was a little off. ;-)
Conclusion
Does this mean that we'll have full ECMA documentation imported for the next Mono release? Probably not. There are still a few issues with the documentation import where it skips members that ideally would be imported (for instance, documentation for System.Security.Permissions.FileIOPermissionAttribute.All isn't imported because Mono provides a get accessor while ECMA doesn't). The documentation also needs to be reviewed after import to ensure that the import was successful (a number of bugs have been found and fixed while working on these optimizations).
Hopefully it won't take me too long to get things imported...
Mono.Fuse 0.4.2
Mono.Fuse is a C# binding for FUSE. This is a minor update over the previous Mono.Fuse 0.4.1 release.
This is a minor release to fix configure support.
Aside: A Walk through Mono.Posix History
As mentioned in the Mono.Fuse 0.1.0 release, one of the side-goals was to make sure that Mono.Unix.Native was complete enough to be usable. One of the great discoveries was that it wasn't, which led to the addition of some new NativeConvert methods.
However, Mono.Fuse and the new Mono.Posix development were concurrent, and in getting the new NativeConvert methods added some of them were dropped. Originally, there would be 4 methods to convert between managed and native types:
- void Copy (IntPtr source, out ManagedType dest);
- void Copy (ref ManagedType source, IntPtr dest);
- bool TryCopy (IntPtr source, out ManagedType dest);
- bool TryCopy (ref ManagedType source, IntPtr dest);
This is what Mono.Fuse 0.2.1 and later releases assumed, and they used the NativeConvert.Copy methods.
Unfortunately, it was felt that having 4 methods/type (Stat, Statvfs, Utimbuf, Pollfd, Timeval...) would add a lot of new methods, so Mono.Posix only accepted the TryCopy variants, and not the Copy variants.
This implicitly broke Mono.Fuse, but I unfortunately didn't notice. Combined with a configure check that only checked whether libMonoPosixHelper.so exported one of the required underlying copy functions, most people didn't notice it either (as the installed libMonoPosixHelper.so didn't have the exports, so the check always failed, causing Mono.Fuse to use it's fallback methods).
Now that newer Mono releases are available, the configure check does find the libMonoPosixHelper.so exports, so it tries to use the NativeConvert methods...and triggers a compilation error, as the methods it's trying to use don't exist.
Mea culpa.
Download
Mono.Fuse 0.4.2 is available from http://www.jprl.com/Projects/mono-fuse/mono-fuse-0.4.2.tar.gz. It can built with Mono 1.1.13 and later. Apple Mac OS X support has only been tested with Mono 1.2.3.1.
GIT Repository
The GIT repository for Mono.Fuse is at http://www.jprl.com/Projects/mono-fuse.git.
POSIX Says The Darndest Things
make check was reported to be failing earlier this week, and Mono.Posix was one of the problem areas:
1) MonoTests.Mono.Unix.UnixGroupTest.ListAllGroups_ToString : #TLAU_TS: Exception listing local groups: System.IO.FileNotFoundException: Nie ma takiego pliku ani katalogu ---> Mono.Unix.UnixIOException: Nie ma takiego pliku ani katalogu [ENOENT]. at Mono.Unix.UnixMarshal.ThrowExceptionForLastError () [0x00000] in /home/koxta/mono-1.2.4/mcs/class/Mono.Posix/Mono.Unix/UnixMarshal.cs:456 at Mono.Unix.UnixGroupInfo.GetLocalGroups () [0x0001c] in /home/koxta/mono-1.2.4/mcs/class/Mono.Posix/Mono.Unix/UnixGroupInfo.cs:127 at MonoTests.Mono.Unix.UnixGroupTest.ListAllGroups_ToString () [0x0000a] in /home/koxta/mono-1.2.4/mcs/class/Mono.Posix/Test/Mono.Unix/UnixGroupTest.cs:32 at MonoTests.Mono.Unix.UnixGroupTest.ListAllGroups_ToString () [0x0003c] in /home/koxta/mono-1.2.4/mcs/class/Mono.Posix/Test/Mono.Unix/UnixGroupTest.cs:37 at <0x00000> <unknown method> at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (object,object[]) at System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00040] in /home/koxta/mono-1.2.4/mcs/class/corlib/System.Reflection/MonoMethod.cs:144
Further investigation narrowed things down to Mono_Posix_Syscall_setgrent() in support/grp.c:
int
Mono_Posix_Syscall_setgrent (void)
{
errno = 0;
setgrent ();
return errno == 0 ? 0 : -1;
}
I did this because setgrent(3) can fail, even though it has a void return type; quoting the man page:
Upon error, errno may be set. If one wants to check errno after the call, it should be set to zero before the call.
Seems reasonably straightforward, no? Clear errno, do the function call, and if errno is set, an error occurred.
Except that this isn't true. On Gentoo and Debian, calling setgrent(3) may set errno to ENOENT (no such file or directory), because setgrent(3) tries to open the file /etc/default/nss. Consequently, Mono.Unix.UnixGroupInfo.GetLocalGroups reported an error (as can be seen in the above stack trace).
Further discussion with some Debian maintainers brought forth the following detail: It's only an error if it's a documented error. So even though setgrent(3) set errno, it wasn't an error because ENOENT isn't one of the documented error values for setgrent(3).
"WTF!," says I.
So I dutifully go off and fix it, so that only documented errors result in an error:
int
Mono_Posix_Syscall_setgrent (void)
{
errno = 0;
do {
setgrent ();
} while (errno == EINTR);
mph_return_if_val_in_list5(errno, EIO, EMFILE, ENFILE, ENOMEM, ERANGE);
return 0;
}
...and then I go through the rest of the MonoPosixHelper code looking for other such erroneous use of errno and error reporting. There are several POSIX functions with void return types that are documented as generating no errors, and others are like setgrent(3) where they may generate an error.
It's unfortunate that POSIX has void functions that can trigger an error. It makes binding POSIX more complicated than it should be.
Mono.Fuse 0.4.1
Now with MacFUSE support!
Mono.Fuse is a C# binding for FUSE. This is a minor update over the previous Mono.Fuse 0.4.0 release.
The highlight for this release is cursory MacFUSE support, which allows Mono.Fuse to work on Mac OS X. Unfortunately, it's not complete support, and I would appreciate any assistance in fixing the known issues (details below).
Mac OS X HOWTO
To use Mono.Fuse on Mac OS X, do the following:
- Download and install Mono 1.2.3.1 or later. Other releases can be found at the Mono Project Downloads Page.
- Download and install MacFUSE 0.2.4 or later. Other releases can be found at the macfuse download page.
- Download, extract, and configure Mono.Fuse 0.4.1:
- curl www.jprl.com/Projects/mono-fuse/mono-fuse-0.4.1.tar.gz > mono-fuse-0.4.1.tar.gz
- tar xzf mono-fuse-0.4.1.tar.gz
- cd mono-fuse-0.4.1.tar.gz
- PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
CFLAGS="-D__FreeBSD__=10 -O -g"
./configure --prefix=`pwd`/install-root
- Note: PKG_CONFIG_PATH is needed so that fuse.pc will be found by pkg-config.
- Note: CFLAGS is used as per the macfuse FAQ.
- Note: You can choose any other --prefix you want.
- make
- Once Mono.Fuse has been built, you can run the sample programs as
described in the README:
- cd example/HelloFS/
- mkdir t
- ./hellofs t &
- ls t
- cat t/hello
Known Issues
HelloFS works, but RedirectFS and RedirectFS-FH do not. Trying to execute them results in a SIGILL within Mono.Unix.Native.Syscall.pread when trying to read a file:
- cd example/RedirectFS
- mkdir t
- MONO_TRACE_LISTENER=Console.Out:+++ ./redirectfs t ~/ &
- Note: MONO_TRACE_LISTENER set so that exception messages from Mono.Fuse.FileSystem will be printed to stdout. See the mono(1) man page for more information about MONO_TRACE_LISTENER.
- ls t # works
- cat t/some-file-that-exists
- Generates a SIGILL.
I would appreciate any assistance in fixing this issue.
Download
Mono.Fuse 0.4.1 is available from http://www.jprl.com/Projects/mono-fuse/mono-fuse-0.4.1.tar.gz. It can built with Mono 1.1.13 and later. Apple Mac OS X support has only been tested with Mono 1.2.3.1.
GIT Repository
The GIT repository for Mono.Fuse is at http://www.jprl.com/Projects/mono-fuse.git.
When Comparisons Fail
One of the unsung helper programs for Mono.Fuse and Mono.Unix is create-native-map ( man page), which takes an assembly, looks for DllImport-attributed methods, and generates C structure and function prototypes for those methods and related types. This allows e.g. the internal Mono.Posix.dll methods to be kept in sync with their implementation methods in MonoPosixHelper, checked by the compiler to ensure type consistency.
One of the "features" of create-native-map is support for integer overflow checking. For example, if you have a C# type:
[Map ("struct foo")]
struct Foo {
public int member;
}
then create-native-map will generate the (excerpted) C code (it generates much more):
struct Foo {
int member;
};
int ToFoo (struct foo *from, struct Foo *to)
{
_cnm_return_val_if_overflow (int, from->member, -1);
to->member = from->member;
return 0;
}
This could be handy, as if the actual type of struct foo::member differed from int, we could tell at runtime if the value of from->member wouldn't fit within to->member. That was the hope, anyway. (Yes, this flexibility is required, as many Unix structures only standardize member name and type, but not necessarily the actual type. For example, struct stat::st_nlink is of type nlink_t, which will vary between platforms, but Mono.Unix.Native.Stat.st_nlink can't change between platforms, it needs to expose an ABI-agnostics interface for portability. Consequently, overflow checking is desirable when doing Stat → struct stat conversions, and vice versa, to ensure that nothing is lost.)
The reality is that _cnm_return_val_if_overflow() was horribly buggy and broke if you looked at it wrong (i.e. it worked for me and would fail on many of the build machines running !Linux). Consequently _cnm_return_val_if_overflow() was converted into a no-op unless DEBUG is defined before/during the Mono 1.2.0 release.
Why discuss this now? Because Mono.Fuse 0.4.0 shipped with a broken version of create-native-map, which is the primary reason that it doesn't work with MacFUSE.
But because I'm a glutton-for-punishment/insane, I thought I'd take a look into making overflow checking work again (though it still won't be enabled unless DEBUG is defined). I wrote some tests, got them working on Linux, and tried to run them on Intel Mac OS X. The result: all but one worked. The reason it failed is inexplicable: a failing comparison. G_MININT64 can't be directly compared against 0:
$ cat ovf.c
# include <glib.h>
# include <limits.h>
# include <stdio.h>
int main ()
{
long long v = G_MININT64;
printf (" LLONG_MIN < 0? %i\n", (int) (LLONG_MIN < 0));
printf ("G_MININT64 < 0? %i\n", (int) (G_MININT64 < 0));
printf (" v < 0? %i\n", (int) (v < 0));
}
$ gcc -o ovf ovf.c `pkg-config --cflags --libs glib-2.0`
$ ./ovf
LLONG_MIN < 0? 1
G_MININT64 < 0? 0
v < 0? 1
Now that's a w-t-f: G_MININT64 < 0 is FALSE. Simply bizarre...
Meanwhile, I should have a Mono.Fuse 0.4.1 release out "soon" to fix these problems, permitting Mono.Fuse to work properly with MacFUSE.
openSUSE 10.2 Windows Key Solution
Celso Pinto has told me the solution to my Windows key woes:
gconftool-2 --type bool --set /apps/metacity/general/enable_windows_keys "false"
That is, set the GConf key /apps/metacity/general/enable_windows_keys to false, the documentation for which is:
If true, then pressing the Windows flag keys will cause the panel's main menu to appear.
(This obviously proves that I didn't look hard enough in GConf.)
The downside is that this isn't exposed in a UI anywhere. Which, on reflection, is a good thing -- it shoudn't be exposed, as the key shouldn't be necessary!
<rant>
The presence of the key is a hack, plain and simple. I like the idea of having the Win key display the panel's main menu. It's just like Windows (which isn't always a bad thing). The problem is execution: under Windows, the Windows Start Menu isn't displayed until the Win key is released, i.e. on key-up. This allows using it as a command modifier, e.g. Win+d (show desktop), Win+r (run command), Win+e (open Explorer), and the Start Menu is never displayed for any of these shortcuts.
The problem with the Metacity implementation is that the main menu is displayed on key-down, preventing it from being used as a modifier.
The simple solution? Follow Windows practice, and only display the panel's main menu on Window key-up, and only if no other keys are pressed when the Win key is released. This allows Win+r, Win+d, Win+F1, etc. to all be valid shortcuts, while Win (press + release) can display the main menu, and it all Just Works, without the requirement to set a crappy GConf key.
</rant>
openSUSE 10.2 Windows Key Workaround
Massimiliano Mantione suggested a workaround for the Gnome main-menu in openSUSE 10.2 always "eating" the Windows key: add an additional modifier, such as Ctrl or Alt. This works, so Alt+Win+F1 can be used to switch to workspace 1. Additionally, this can be typed with one hand, making it significantly better than using Menu+F1 for the same task.
The downside is that there is an ordering requirement: whatever modifier you use with the Win key must be typed before the Win key, so Alt+Win+F1 switches to workspace 1, while Win+Alt+F1 (Win key typed first) results in displaying the main menu, with the Alt+F1 ignored. This is annoying.
What's odd is the Keyboard Shortcuts applet: Alt+Win+r is <Alt><Mod4><Hyper>r -- an extra modifier, <Hyper>, is synthesized. (Before openSUSE 10.2, Win+F1 would be <Mod4>r, so the <Mod4> makes sense. Furthermore, it wasn't always possible to use the Win key when entering shortcuts within the Keyboard Shortcuts applet, but it was possible to enter it manually within gconf-editor by using <Mod4> as a "stand-in" for the Win key.)
Care and Feeding of openSUSE 10.2
I upgraded to openSUSE 10.2 recently, and since I installed from scratch I had to figure some things out so that it would be usable. Sadly, Google isn't very helpful with some of these issues...
See SUSE Linux Rants for steps to install media codecs so that you can play videos and DVDs.
Then comes the stuff that Google doesn't help with...
My machine has two ethernet cards: eth0 is the external-facing device (connected to a cable modem), while eth1 is the internal-facing device, which is connected to a Wireless Router and desktop machines. I need to setup IP Forwarding so the internal machines can access the Internet, plus setup some Samba shares so the Windows Machines can access shared files.
IP Masquerading/Network Address Translation (NAT)
- Start YaST
- Start the Network Devices → Network Card Program in YaST.
- Select Traditional Method with ifup. Click Next.
- Select the internal network card. Click Edit.
- In the Address Tab:
- Select the Static Address Setup radio button.
- Set the IP Address field to 192.168.1.1.
- Set the Subnet Mask field to 255.255.255.0.
- Click the Routing button.
- Check the Enable IP Forwarding check box.
- Click OK to exit Routing Configuration.
- In the General Tab:
- Set Firewall Zone to Internal Zone (Unprotected).
- Set Device Activation to At Boot Time.
- Click Next.
- Click Finish.
- Start the Security and Users → Firewall program in YaST.
- In the Start-Up node, within the Service Start area, check the When Booting checkbox.
- In the Interfaces node, make sure that eth0 is the External Zone, while eth1 is Internal Zone. Note: eth0 and eth1 aren't used here; instead, the devices names are used here (e.g. Realtek RT8139).
- In the Masquerading node, check the Masquerade Networks check box.
- Click Next, then Accept.
- Make sure that eth0 has an IP address, and that you can view a web page.
- Start the Network Services → DHCP Server program in YaST.
- In the Start-Up node, within the Service Start section, check the When Booting check box.
- In the Card Selection node, select the device which is eth1, which should be the one with an actual IP address listed (eth0 is probably DHCP), check the Open Firewall for Selected Interfaces check box, and click the Select button.
- Read the file /etc/resolv.conf. There should be at least one line starting with nameserver, followed by an IP Address, e.g. nameserver 1.2.3.4.
- In the Global Settings node, set the Primary Name Server IP textbox to the first nameserver entry in /etc/resolv.conf, and set the Secondary Name Server IP textbox to the second nameserver entry in /etc/resolv.conf. Set the Default Gateway field to the IP Address of eth1, e.g. 192.168.1.1.
- In the Dynamic DHCP node, set the First IP Address field to 192.168.1.10, and the Last IP Address field to 192.168.1.254.
- Click Finish.
You should now be able to connect to the router from any internal machine via DHCP and connect to the outside world.
HTTP Server with mod_userdir
mod_userdir is an Apache module that allows accessing the /home/username/public_html directory as the URL http://server/~username/. It should be easy to setup, and yet I always have grief setting it up...
- Start the Network Services → HTTP Server in YaST.
- In the Listen Ports and Addresses Tab, make sure that HTTP Service is Enabled, Listen on Ports has a Network Address value of All Addresses, and Port is set to 80. Check the Open Port in Firewall checkbox.
- Click Finish.
- Edit /etc/sysconfig/apache2 as the root user, and
change line 32 from:
to:APACHE_CONF_INCLUDE_FILES=""
APACHE_CONF_INCLUDE_FILES="extra/httpd-userdir.conf"
- Run the following commands as the root user:
- /sbin/SuSEconfig
- /usr/sbin/rcapache2 restart
Windows Shares
I have internal Windows machines which need to use files on the router (it doubles as a file server), so I obviously need a Windows share. This involves configuring Samba.
- Start the Network Services → Samba Server program in YaST.
- In the Shares tab, check the Allow Users to Share Their Directories check box.
- Click Finish.
- Create a user (Security and Users → User Management) for each remote user that needs to access the Windows Share (assuming you don't want Guest access to the share).
- At the command line, type the following as the root user for
each remote user
- /usr/bin/smbpasswd -a username
You should now be able to connect from any remote machine as the user username, using the same password that was entered in the smbpasswd command.
openSUSE 10.2 Complaints
Drive Partitioning
I have a genetic deficiency: the default options are never adequate. :-)
In particular, I have three drives: two ISA and an external FireWire drive. /dev/hda had ~13 partitions (/home, /boot, swap, and a set of /, /usr, /tmp, /var etc. directories for two separate installations). Obviously, this is what I had wanted to install to. /dev/hdb had my backups.
What does openSUSE want to install onto? /dev/hdb (my backups), of course! No, that is not happening.
Of course, I also wanted to completely redo /dev/hda to make use of a Logical Volume Manager. This was also more complicated than it needed to be. When using the Expert installer, you can only create a Logical Volume Manager on an already existing partition. So I couldn't delete every partition on /dev/hda, create two new partitions (/boot and the LVM partition), and setup the LVM in the new partition. That wouldn't work. It should work, but it doesn't. So I had to create the two partitions, continue with the installation, abort the installation, restart, and then I could use a LVM on /dev/hda2!
WTF?!
At least it worked though.
Using the Windows key
"Back in the day," when my roommate was trying to convert me to Linux from Windows, I spent a fair amount of time at the "console", outside of X11 or any other GUI environment. One of the nice features that Linux had that FreeBSD and other systems lacked at the time was virtual consoles: pressing Alt+FN (e.g. Alt+F4) would immediately switch to virtual console N. This made it really convenient to run multiple programs concurrently within separate shells, without messing around with &, bg, etc.
I thought this was an excellent feature, so when I moved to using X11 more extensively I wanted to do the same thing with virtual desktops. So I did. The only downside to this is that some apps wanted to make use of Alt+FN shortcuts themselves (e.g. KDE uses Alt+F4 to exit programs), but this was something I was willing to live with.
More recently, I switched to using the Windows key as the virtual desktop modifier, so that Win+FN would witch to virtual desktop N, and Shift+Win+FN would move the current window to virtual desktop N.
I also use the Win key for a host of shortcuts borrowed from Windows, e.g. Win+r to open the Run Program dialog, Win+d to show the Desktop, Win+a to move the current window above all other windows (I use sloppy mouse focus/"focus follows mouse"), Win+b to move the current window below all other windows, etc. I use it a lot.
This is where things go horribly wrong: under openSUSE 10.2, the Win key is always eaten by the Gnome main menu program. Oops.
Even worse, I can't find a way to fix this. Changing the Keyboard Layout in the Keyboard Preferences application doesn't change anything, and none of the Layout Options → Alt/Win key behavior options seem helpful.
The best "compromise" is the Alt is mapped to the right Win-key and Super to Menu option, which still sucks, for two reasons. First, the Menu key can no longer be used to open the context menu (fortunately Shift+F10 still works for this), and two, all of my shortcuts which used to take one hand now take two. Win+F1..Win+F5, Win+r, Win+a, Win+b, etc., could all be done with my left hand. Since the Menu key is on the right, next to the right Ctrl key, I now require two hands for these shortcuts. Which means I can't easily keep one hand on the keyboard and one on the mouse...
Grr....
Please, someone fix this. It's fundamentally broken. I've been able to use Win as a modifier for literally years, and now I can't, because one broken app decides that the key is only meaningful to it. And removing the Gnome main menu doesn't help, the menu is still displayed.
:-(
Novell, Microsoft, & Patents
The news is out: hell has frozen over. Novell and Microsoft have announced a "patent cooperation agreement," whereby Microsoft won't sue Novell customers for patent infringement and Novell won't sue Microsoft customers for patent infringement.
I first heard about this on mono-list, and immediately replied with the obvious (to me) response.
Note: I am not a lawyer [0], so consequently everything I say is bunk, but I have been paying some attention to various lawsuits over the years.
That out of the way, take a step back and ignore Microsoft and Novell for the moment. Assume that you're a patent holder, and you decide that your patent has been infringed. Who do you sue? There are three possible defendants:
- Sue the developer. (Example: Stac Electronics vs. Microsoft.)
- Sue the distributor. This is frequently identical to (1) as the developer is the distributor, but the rise of Free and Open Source software introduces this distinction.
- Sue the customer of (1) and/or (2). The example I remembered hearing several years ago was Timeline vs. Microsoft [1].
The summary is this: software patents are evil, allowing virtually anyone to sue virtually everyone else. There are no assurances of safety anywhere. Software from large companies (Internet Explorer) can be sued as easily as software from a no-name company or the open-source community (see Eolas vs. Microsoft).
With that background out of the way, what does this Microsoft/Novell deal mean? It means exactly what they say: Novell won't sue Microsoft customers, and Microsoft won't sue Novell customers. Anyone else can still sue Microsoft, Novell, and their customers, so Novell and Microsoft customers really aren't any safer than they were before. Novell customers are a little safer -- the monster in the closet of a Microsoft lawsuit is no longer an issue -- but no one is completely safe. It just provides peace of mind, but it isn't -- and cannot -- be a complete "solution" to the threat of patent lawsuits. (The only real solution is the complete abolition of all software patents, which is highly unlikely.)
What does this mean for hobbyists who contribute to Mono, Samba, Wine, Linux, and other projects (like me)? It means I'm protected as part of this agreement, as my code is distributed as part of openSUSE. This also means that anyone other than Microsoft can sue me if I happen to violate a patent.
What about hobbyists whose code isn't part of openSUSE? Nothing has changed -- they're as subject to a lawsuit as they were a week ago.
What about other companies such as Red Hat? Nothing has changed for them, either. Red Hat is still safe, as it is a member of the Open Invention Network, which was created to deal with the potential for patent lawsuits from any party. OIN is a more complete solution for most parties involved than the Microsoft and Novell agreement, as it involves more parties.
The problem with OIN is that it only covers the members of OIN. Red Hat is protected, but any distributors of Red Hat code are not (such as CentOS), and neither are the customers of Red Hat (unless the customer has a patent protection contract with their supplier). Consequently, OIN serves to protect the original developers (1), but not any "downstream" distributors (2) or their customers (3).
But what about the GPL, section 7? Doesn't the Microsoft/Novell agreement violate it?
7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program.
The simple solution is that this doesn't apply, as this agreement doesn't touch this clause at all. It's not a consequence of a court judgment, there is no allegation of patent infringement, and I haven't heard of any conditions that Microsoft requires Novell to follow in order for the code to be freely distributed. The Microsoft/Novell agreement primarily covers their customers, not their code, so there isn't a problem.
Notes:
[0] But I did stay at a Holiday Inn last night!
[1] Computerworld
Article.
Mono.Fuse 0.4.0
Mono.Fuse is a C# binding for FUSE. This is the fourth major release
This release contains a few major changes to the public API for consistency and clarification purposes, the biggest of which is renaming Mono.Fuse.FileSystemEntry to Mono.Fuse.DirectoryEntry (which of course required changing Mono.Fuse.FileSystem.OnReadDirectory(), again!). Some of the Mono.Fuse.FileSystem properties were also renamed for consistency.
I'm still making no promises for API stability. The FileSystem virtual methods should be fairly stable, but the properties may continue to be flexible as I document them more fully (as I'm not entirely sure what the ramifications are for some of them, such as FileSystem.ReaddirSetsInode vs. FileSystem.SetsInode, and answering these questions will require reading the FUSE source).
API Changes from the previous release:
See the commit diff for specifics.
- FileSystem:
- Now abstract. There are no abstract methods, but it's pointless to create instances of this type, as it wouldn't do anything.
- Make constructors protected (to emphasize the above).
- Rename FileSystemName property to Name.
- Rename ImmediateRemoval property to ImmediatePathRemoval.
- Rename DirectIO property to EnableDirectIO.
- Rename Umask property to DefaultUmask.
- Add EnableKernelCache property.
- Rename UserId property to DefaultUserId.
- Rename GroupId property to DefaultGroupId.
- Rename EntryTimeout property to PathTimeout.
- Rename DeletedNameTimeout property to DeletedPathTimeout.
- Change types of Timeout properties to double from int; changes PathTimeout, DeletedPathTimeout, AttributeTimeout.
- Rename OnCloseDirectory() method to OnReleaseDirectory() (for consistency with OnReleaseHandle()).
- Rename Exit() method to Stop().
- Rename parameters of many methods to match the method names -- e.g. OnReadSymbolicLink() takes a link parameter instead of path, OnOpenDirectory() takes a directory parameter instead of path, etc.
- FileSystemEntry:
- Renamed to DirectoryEntry.
- Renamed Path property to Name.
- Add argument validation to DirectoryEntry constructor.
- Remove implicit conversion to string (as constructor may generate exceptions).
- FileSystemOperationContext:
- Make sealed (this should only be created internally).
- Constructor is now internal.
- OpenedFileInfo:
- Make sealed (this should only be created internally).
- Constructor now internal.
- Remove OpenReadOnly, OpenWriteOnly, OpenReadWrite properties.
- Add OpenAccess property.
Download
Mono.Fuse 0.4.0 is available from http://www.jprl.com/Projects/mono-fuse/mono-fuse-0.4.0.tar.gz. It can built with Mono 1.1.13 and later.
GIT Repository
A GIT repository for Mono.Fuse is at http://www.jprl.com/Projects/mono-fuse.git.
Naming, Mono.Fuse Documentation
I find naming to be difficult. What should a type be named, what should a member be named? What's consistent? What names are easily understandable in any context? What names sow confusion instead of clarity?
This is why writing documentation, as annoying as it can be (lots of repetition), is also useful: it forces a different viewpoint on the subject.
For example, Mono's monodocer program uses System.Reflection to generate an initial documentation stub for use within monodoc. As such, it shows you the public types which are actually within an assembly, not just what you thought was in the assembly, like the compiler-generated default constructors which are so easy to forget about.
I've documented every public type and member within Mono.Fuse: Mono.Fuse Documentation.
And if you haven't guessed by now, the types have changed, because writing documentation forces a different viewpoint on the subject, and shows out all of the glaring inconsistencies within the API. So much for my hope that the API would be reasonably stable after the 0.3.0 release. <Sigh>
Consequently, the docs are only ~90% useful for the most recent 0.3.0 release, as they document the forthcoming 0.4.0 release. I hope to get 0.4.0 out reasonably soon, though I still have to write release notes.
The GIT repository has been updated so that HEAD contains the 0.4.0 sources, so if you're really interested in using the current API, you can git-clone it for now.
Mono.Fuse 0.3.0
Mono.Fuse is a C# binding for FUSE. This is the third major release.
This release completely changes the public API for consistency and performance. Hopefully this will be the last API change, though I would appreciate any feedback on the current Mono.Fuse.FileSystem.OnReadDirectory API.
API Changes from the previous release:
- FileSystem:
- The EnableDebugOutput property has been renamed to EnableFuseDebugOutput. To get FileSystem trace output, set the MONO_TRACE_LISTENER environment variable.
- The Options dictionary property has been renamed to FuseOptions, as this only holds options which are passed directly to the FUSE library. Any non-FUSE-supported options will be flagged as an error.
- A bool MultiThreaded property has been added, and the StartMultithreaded() method has been removed. Instead, Start consults the MultiThreaded property to determine whether FUSE should be started in a multithreaded context. You can use the -s command line argument to disable this, or explicitly set the MultiThreaded property to false. MultiThreaded is true by default.
- Nearly every protected virtual method has been renamed,
(and the parameters modified) in order to organize the methods by
name. Here are the groupings and renamings:
Previous Current Directory: methods which only act upon direcctories. OnCloseDirectory(string path, OpenedFileInfo info); OnCloseDirectory(string path, OpenedPathInfo info); OnCreateDirectory(string path, FilePermissions mode); OnOpenDirectory(string path, OpenedFileInfo info); OnOpenDirectory(string path, OpenedPathInfo info); OnReadDirectory(string path, out string[] paths, OpenedFileInfo info); OnReadDirectory(string path, OpenedPathInfo info, out IEnumerable<FileSystemEntry> paths); OnRemoveDirectory(string path); OnSynchronizeDirectory(string path, bool onlyUserData, OpenedFileInfo info); OnSynchronizeDirectory(string path, OpenedPathInfo info, bool onlyUserData); File: methods which only act upon file names. OnCreateFileNode(string path, FilePermissions perms, ulong dev); OnCreateSpecialFile(string path, FilePermissions perms, ulong dev); OnRemoveFile(string path); OnTruncateFile(string path, long length); FileSystem: methods which only act upon the file system as a whole. OnGetFileSystemStatistics(string path, out Statvfs buf); OnGetFileSystemStatus(string path, out Statvfs buf); Link: methods which deal with manipulation hard and symbolic links. OnCreateHardLink(string oldpath, string newpath); OnCreateSymbolicLink(string oldpath, string newpath); OnReadSymbolicLink(string path, StringBuilder buf, ulong bufsize); OnReadSymbolicLink(string path, out string target); Handle: methods which deal with a "handle" to an opened file. OnCreate(string path, FilePermissions mode, OpenedFileInfo info); OnCreateHandle(string path, OpenedPathInfo info, FilePermissions mode); OnFlush(string path, OpenedFileInfo info); OnFlushHandle(string path, OpenedPathInfo info); OnGetFileDescriptorAttributes(string path, out Stat buf, OpenedFileInfo info); OnGetHandleStatus(string path, OpenedPathInfo info, out Stat buf); OnOpen(string path, OpenedFileInfo info); OnOpenHandle(string path, OpenedPathInfo info); OnRead(string path, byte[] buf, long offset, OpenedFileInfo info, out int bytesWritten); OnReadHandle(string path, OpenedPathInfo info, byte[] buf, long offset, out int bytesWritten); OnWrite(string path, byte[] buf, long offset, OpenedFileInfo info, out int bytesRead); OnWriteHandle(string path, OpenedPathInfo info, byte[] buf, long offset, out int bytesRead); OnRelease(string path, OpenedFileInfo info); OnReleaseHandle(string path, OpenedPathInfo info); OnSynchronizeFileDescriptor(string path, bool onlyUserData, OpenedFileInfo info); OnSynchronizeHandle(string path, OpenedPathInfo info, bool onlyUserData); OnTruncateFileDescriptor(string path, long length, OpenedFileInfo info); OnTruncateHandle(string path, OpenedPathInfo info, long length); Path: methods which deal with a file system entry names as strings, and thus manipulate both files and directories. OnAccess(string path, AccessModes mode); OnAccessPath(string path, AccessModes mode); OnGetFileAttributes(string path, out Stat stat); OnGetPathStatus(string path, out Stat stat); OnRenameFile(string oldpath, string newpath); OnRenamePath (string oldpath, string newpath); OnChangePermissions(string path, FilePermissions mode); OnChangePathPermissions(string path, FilePermissions mode); OnChangeOwner(string path, long owner, long group); OnChangePathOwner(string path, long owner, long group); OnChangeTimes(string path, ref Utimbuf buf); OnChangePathTimes(string path, ref Utimbuf buf); OnGetExtendedAttributes(string path, string name, byte[] value, out int bytesWritten); OnGetPathExtendedAttribute(string path, string name, byte[] value, out int bytesWritten); OnListExtendedAttributes(string path, byte[] list, out int bytesWritten); OnListPathExtendedAttributes(string path, out string[] names); OnSetExtendedAttributes(string path, string name, byte[] value, XattrFlags flags); OnSetPathExtendedAttribute(string path, string name, byte[] value, XattrFlags flags); OnRemoveExtendedAttributes(string path, string name); OnRemovePathExtendedAttribute(string path, string name);
- FileSystemEntry: New type to associate a path string with a Mono.Unix.Native.Stat instance. Used by OnReadDirectory.
- FileSystemOperationContext:
- Remove the PrivateData member.
- UserId, GroupId, and ProcessId are now read-only properties instead of fields.
- ProcessId is now int, for constency with Mono.Unix.
- OpenedFileInfo has been renamed to OpenedPathInfo, as this type is used by both handle and directory methods. The long OpenedFileInfo.FileHandle property has been changed into a IntPtr Handle property.
Download
Mono.Fuse 0.3.0 is available from http://www.jprl.com/Projects/mono-fuse/mono-fuse-0.3.0.tar.gz. It can built with Mono 1.1.13 and later.
GIT Repository
A GIT repository for Mono.Fuse is at http://www.jprl.com/Projects/mono-fuse.git.
Miguel's ReflectionFS
After the Mono.Fuse 0.2.1 release, Miguel de Icaza wrote a small Mono.Fuse program that exposed System.Reflection information as a filesystem.
With the Mono.Fuse 0.3.0 release, this sample no longer works, as the Mono.Fuse API changed. Thus, here is an updated version of the sample:
$ cp `pkg-config --variable=Libraries mono-fuse` . $ gmcs ReflectionFS.cs -r:Mono.Fuse.dll -r:Mono.Posix.dll $ mkdir t $ mono ReflectionFS.exe t & $ ls t/mscorlib/System.Diagnostics.ConditionalAttribute ConditionString GetType ToString Equals get_TypeId TypeId get_ConditionString IsDefaultAttribute GetHashCode Match $ fusermount -u t
Note that the above requires that PKG_CONFIG_PATH contain a directory with the mono-fuse.pc file (created during Mono.Fuse installation), and that libMonoFuseHelper.so should be in LD_LIBRARY_PATH or a directory listed in /etc/ld.so.conf.
Mono.Fuse also contains other sample programs. In particular, RedirectFS-FH.cs is a straightforward port of FUSE's fusexmp_fh.c sample program, and shows a way to "redirect" a new mountpoint to display the contents of another existing directory. RedirectFS-FH.cs is actually an improvement, as fusexmp_fh.c just shows the contents of the / directory at any new mount point, while RedirectFS-FH can redirect to any other directory.
Mono.Fuse, Take 2!
See the original announcement for more on what Mono.Fuse is (in short: a C# binding for FUSE).
This is an update, releasing Mono.Fuse 0.2.0, and (more importantly) an updated set of patches to mcs, mono, and now mono-tools. The mcs and mono patches are required to build & run Mono.Fuse, while the mono-tools patch is optional and only necessary if you want to view the create-native-map.exe program.
See here for all patches and an overview.
The major change between this set of patches and the original set is one of approach: the original set tried to make the native MonoPosixHelper API public, which was deemed as unacceptable (as there's too much cruft in there that we don't want to maintain).
The new approach only adds public APIs to the Mono.Unix.Native.NativeConvert type, permitting managed code to copy any existing native instance of supported structures. For example:
Mono.Unix.Native.NativeConvert.Copy (IntPtr source,
out Mono.Unix.Native.Stat destination);
copies a pointer to an existing native struct stat and copies it into the managed Mono.Unix.Native.Stat instance. There are equivalent methods to do the managed → native conversion as well.
Since this approach requires making far fewer public API changes to Mono.Posix and MonoPosixHelper (i.e. no public API changes to MonoPosixHelper, as it's an internal/private library), I hope that this will be more acceptable.
Here's to a quick review!
Updated to add a link to the overview page.
Mono.Fuse, Take 2.1!
At Miguel's request, I've created a version of Mono.Fuse that doesn't depend upon Mono runtime changes. This should make it possible for more people to try it out.
Don't forget to read the README, as it contains build instructions and a description of how to run the included example program.
I should caution that the API isn't stable (I suspect Mono.Fuse.FileSystem.OnRead should probably become Mono.Fuse.FileSystem.OnReadFile, for one), and I look forward to any and all API suggestions that you can provide.
Two final notes: Mono.Fuse depends on FUSE, and FUSE is a Linux kernel module, so you'll need to run:
/sbin/modprobe fuse
as the root user before you can use any FUSE programs. You'll also need to install the FUSE user-space programs, as you must use the fusermount program to unmount a directory that has been mounted by FUSE, e.g.:
fusermount -u mount-point
Announcing Mono.Fuse
Mono.Fuse is a binding for the FUSE library, permitting user-space file systems to be written in C#.
Why?
I read Robert Love's announcement of beaglefs, a FUSE program that exposes Beagle searches as a filesystem. My first thought: Why wasn't that done in C# (considering that the rest of Beagle is C#)?
What about SULF?
Stackable User-Level Filesystem, or SULF, is a pre-existing FUSE binding in C#, started by Valient Gough in 2004.
Mono.Fuse has no relation to SULF, for three reasons:
- It goes to great efforts to avoid a Mono.Posix.dll dependency, duplicating Mono.Unix.Native.Stat (Fuse.Stat), Mono.Unix.Native.Statvfs (Fuse.StatFS), and many methods from Mono.Unix.Native.Syscall (Fuse.Wrapper).
- I don't like the SULF API. (Not that I spent a great deal of time looking at it, but what I did see I didn't like.)
- SULF wraps the FUSE kernel-level interface, while Mono.Fuse wraps the higher level libfuse C interface.
I find (1) the most appalling, if only because I'm the Mono.Posix maintainer and I'd like to see my work actually used. :-)
Once I started writing Mono.Fuse, I discovered a good reason to avoid Mono.Posix: it's currently impossible to use the native MonoPosixHelper shared library from outside of Mono. I figured this would be a good opportunity to rectify that, making it easier for additional libraries to build upon the Mono.Posix infrastructure.
Implementation
Mono.Fuse requires patches to the mcs and mono modules, changes which need to be proposed and discussed.
mono
The biggest problem with the mono module is that no headers are installed, making it difficult to make use of libMonoPosixHelper.so.
Changes:
- Modify configure to generate a mono-config.h file, installed as $libdir/mono/include/mono-config.h. (Basic idea "borrowed" from GLib's $libdir/glib-2.0/include/glibconfig.h).
- Add a mono-posix-helper.pc file.
- Install the files $includedir/mono/posix/helper.h and $includedir/mono/posix/map.h.
map.h is the current map.h file generated by make-map.exe, with some major additions (detailed in the mcs section).
helper.h is the main include file, which includes map.h and declares all types/functions which cannot be generated by make-map.exe.
mono-config.h is necessary because it needs to contain platform-specific macros. In particular, Linux needs:
int Mono_Posix_ToStatvfs (struct statvfs *to, struct Mono_Posix_Statvfs *to);
while OS X and *BSD need:
int Mono_Posix_ToStatvfs (struct statfs *to, struct Mono_Posix_Statvfs *to);
Note struct statvfs vs. struct statfs. The mono/posix/helper.h header needs to "paper over" the difference, and thus needs to know which type the platform prefers. helper.h thus looks like:
#ifdef MONO_HAVE_STATVFS
struct statvfs;
int Mono_Posix_ToStatvfs (struct statvfs *from,
struct Mono_Posix_Statvfs *to);
#endif
#ifdef MONO_HAVE_STATFS
struct statfs;
int Mono_Posix_ToStatvfs (struct statfs *from,
struct Mono_Posix_Statvfs *to);
#endif
One of MONO_HAVE_STATVFS or MONO_HAVE_STATFS would be defined in mono-config.h.
mcs
There are two major changes:
- The addition of one public attribute to the API:
// targets Class, Delegate, Enum, Field, Struct class Mono.Unix.Native.MapAttribute { public MapAttribute (); public MapAttribute (string nativeType); public string NativeType {get;} public string NativeSymbolPrefix {get; set;} } - A major revamp to make-map.exe.
The MapAttribute attribute is public so that make-map.exe can use a publically exposed API for code generation purposes which can be used by other libraries (Mono.Fuse makes use of these changes).
make-map.exe can also generate structure declarations and delegate declarations in addition to P/Invoke function declarations, allowing for a better, automated interface between C and C#.
Previously, [Map] could only be used on enumerations.
Now, [Map] can be used on classes, structures, and delegates, to create a C declaration of the C# type, suitable for P/Invoke purposes, e.g. the C# code:
[Map] struct Stat {public FilePermissions mode;}
would generate the C declaration
struct Namespace_Stat {unsigned int mode;};
The MapAttribute.NativeType property is used to specify that type conversion functions should be generated, thus:
[Map ("struct stat")] struct Stat {public FilePermissions mode;}
would generate
struct Namespace_Stat {unsigned int mode;};
int Namespace_ToStat (struct stat *from, struct Namespace_Stat *to);
int Namespace_FromStat (struct Namespace_Stat *from, struct stat *to);
along with the actual implementations of Namespace_ToStat() and Namespace_FromStat().
The MapAttribute.NativeSymbolPrefix property is used to specify the C "namespace" to use:
[Map (NativeSymbolPrefix="Foo")] struct Stat {FilePermissiond mode;}
generates
struct Foo_Stat {unsigned int mode;};
This prefix is also used for the conversion functions.
(You may be wondering why NativeSymbolPrefix exists at all. This is for reasonable symbol versioning -- make-map.exe currently has a "hack" in place to rename Mono.Unix(.Native) to Mono_Posix, a hack I'd like to remove, and NativeSymbolPrefix allows the Mono.Unix.Native types to have a Mono_Posix C namespace in a reasonably general manner.)
The previously internal Mono.Unix.HeaderAttribute has been removed. The HeaderAttribute.Includes and HeaderAttribute.Defines properties have been replaced with make-map.exe command-line arguments. In particular, HeaderAttribute.Includes has been replaced with --autoconf-header, --impl-header, --impl-macro, --public-header, and --public-macro (the first three modify the generated .c file, while the latter two modify the generated .h file).
Finally, make-map.exe has been renamed and moved from mcs/class/Mono.Posix/Mono.Unix.Native/make-map.exe to mcs/tools/create-native-map/create-native-map.exe.
HOWTO
- Go to http://www.jprl.com/Projects/mono-fuse for the patches and source download.
- Apply mcs.patch to a mcs checkout, rebuild, and install.
- Apply mono.patch to a mono checkout, rebuild, and install.
- Build mono-fuse-0.1.0.tar.gz in "the standard manner" (./configure ; make ; make install).
Questions
- Is it OK for create-native-map.exe to be a .NET 2.0 app?
- How should we cope with unstable APIs which make use of native
code? The
Application Deployment Guidelines don't address
this issue, nor the related issue of what should be done with
64-bit binaries. In particular, for an AMD64 machine, which
directory layout should be used for an assembly + native lib
combo?
- What Mono seems to do, with the GAC in $prefix/lib no matter
what the architecture:
/usr/lib/mono-fuse/Mono.Fuse.dll /usr/lib64/libMonoFuseHelper.so - Be consistent, and toss everything in @libdir@:
/usr/lib64/mono-fuse/Mono.Fuse.dll /usr/lib64/libMonoFuseHelper.so
- What Mono seems to do, with the GAC in $prefix/lib no matter
what the architecture:
Performance Comparison: IList<T> Between Arrays and List<T>
Rico Mariani recently asked a performance question: given the following code, which is faster, Sum(array), which converts a ushort[] to an IList<T>, or Sum(list), which uses the implicit conversion between List<T> and IList<T>.
using System; using System.Collections.Generic; class Test { static int Sum (IList<ushort> indeces) { int result = 0; for (int i = 0; i < indeces.Count; ++i) result += indeces [i]; return result; } const int Size = 500000; public static void Main () { ushort[] array= new ushort [Size]; DateTime start = DateTime.UtcNow; Sum (array); DateTime end = DateTime.UtcNow; Console.WriteLine (" ushort[]: {0}", end-start); List<ushort> list = new List<ushort> (Size); for (int i = 0; i < Size; ++i) list.Add (0); start = DateTime.UtcNow; Sum (list); end = DateTime.UtcNow; Console.WriteLine ("List<ushort>: {0}", end-start); } }
Note that the question isn't about comparing the performance for constructing a ushort[] vs. a List<T>, but rather the use of an IList<ushort> backed by a ushort[] vs a List<ushort>.
The answer for Mono is that, oddly enough, List<ushort> is faster than ushort[]:
ushort[]: 00:00:00.0690370
List<ushort>: 00:00:00.0368170
The question is, why?
The answer is, "magic." System.Array is a class with magical properties that can't be duplicated by custom classes. For example, all arrays, such as ushort[], inherit from System.Array, but only have an explicitly implemented IList indexer. What looks like an indexer usage results in completely different IL code; the compiler is involved, and must generate different code for an array access.
For example, an array access generates the IL code:
// int i = array [0]; ldarg.0 // load array ldc.i4.0 // load index 0 ldelem.i4 // load element array [0] stloc.0 // store into i
While an IList indexer access generates this IL code:
// object i = list [0];
ldarg.0 // load list
ldc.i4.0 // load index 0
callvirt instance object class [mscorlib]System.Collections.IList::get_Item(int32)
// call IList.this [int]
stloc.0 // store into i
This difference in IL allows the JIT to optimize array access, since different IL is being generated only for arrays.
In .NET 2.0, System.Array got more magic: all array types implicitly implement IList<T> for the underlying array type, which is why the code above works (ushort[] implicitly implements IList<ushort>). However, this is provided by the runtime and is "magical," in that System.Reflection won't see that System.Array implements any generics interfaces. Magic.
On Mono, this is implemented via an indirection: arrays may derive from System.Array.InternalArray<T> instead of System.Array. InternalArray<T> implements IList<T>, permitting the implicit conversion from ushort[] to IList<ushort>.
However, this indirection has a performance impact: System.Array.InternalArray<T>.get_Item invokes System.Array.InternalArray<T>.GetGenericValueImpl, which is an internal call. This is the source of the overhead, as can be seen with mono --profile=default:stat program.exe:
prof counts: total/unmanaged: 172/97
27 15.79 % mono
12 7.02 % mono(mono_metadata_decode_row
11 6.43 % Enumerator:MoveNext ()
10 5.85 % Test:Sum (System.Collections.Generic.IList`1)
10 5.85 % (wrapper managed-to-native) InternalArray`1:GetGenericValueImpl (int,uint16&)
10 5.85 % InternalEnumerator:MoveNext ()
To conclude, List<ushort> is faster than ushort[], when accessed via an IList<ushort> reference, because the ushort[] can't be accessed as a normal array, procluding the usual runtime optimizations:
- IList<T>.get_Current → List<T>.get_Current → direct T[] access, optimized by the runtime as a normal array access; vs.
- IList<T>.get_Current → Array.InternalArray<T>.get_Current → Array.InternalArray<T>.GetGenericValueImpl (internal call).
It should be also noted that because of this "magic," all arrays under .NET 2.0 have more overhead than the same arrays under .NET 1.1, because of the need to support the "magic" generics interfaces. This could be optimized to save memory, such that if you never access the array via a generic interface no memory is used, but Mono has not performed such an optimization yet.
Update: After discussing this on #mono, Paolo Molaro implemented an optimization which makes the array usage much faster. Now it's only slightly slower than IList<T>:
ushort[]: 00:00:00.0133390
List<ushort>: 00:00:00.0132830
Update 2: Rico Mariani has posted his .NET performance analysis. The key take home point? "Arrays are magic."
Reverse Engineering
Occasionally I run across blatant misunderstandings about what reverse engineering is, for example a recent slashdot post regarding Wine benchmarks, and on mono-list.
The confusion exists because there are two forms of reverse engineering: white box reverse engineering and black box reverse engineering (similar in many respects to white box testing and black box testing).
White box reverse engineering, when applied to software, frequently involves decompiling the binary to view either the underlying assembly or some higher level representation. (For example, there are Java decompilers which can recreate the original Java source, absent any comments.) The results of White box reverse engineering cannot be used in a commercial product, as it creates copyright violation possilibities (since you may inadvertantly copy the original product). Nevertheless, it can be very useful in a security context, such as the creation of the unofficial WMF patch, or for determining how a virus operates.
Black box reverse engineering is closer to the scientific method, only applied to hardware or software:
- Gather information and resources, typically by reading any available public documentation.
- Form a hypothesis, such as how something is implemented.
- Perform experiment and collect data. For software, the "experiment" is typically a test program of some sort.
- Analyze data.
- Interpret data and draw conclusions.
- Repeat 2..5 until the underlying hardware or software is understood.
If black-box reverse engineering wasn't allowed, then any market that depends on interoperability would be killed. Samba, Wine, Mono: dead. The PC Clone industry? Couldn't happen without Compaq reverse-engineering the original BIOS. Garage door openers? Forget it (though there was an interesting use of the DMCA to try and prevent the creation of compatible garage door openers).
Reverse engineering is a necessary component to a free market. Without it, monopolies cannot be challenged, compatibility can not be assured (compatibility is at the whim of the original creator), and the market is stifled.
Programming Language Comparison
Last year, Joel Spolsky mentioned the The Perils of JavaSchools, and mentioned the mid-term he took at Penn:
1a. (MIT-Scheme) Using the following function, implement sum-of-squares which calculates the sum of squares of a list:
(define (accumulate combiner null-value l) (if (null? l) null-value (combiner (car l) (accumulate combiner null-value (cdr l))))) (define (sum-of-squares l) (accumulate (lambda (x y) (+ (* x x) y)) 0 l)) (sum-of-squares '(1 2 3 4 5))
Simple, straightforward (if you're familiar with the functional programming paradigm), and 14 lines of code.
This begs the question: what's the similar flavor code in C# and Java?
C# 2.0 allows anonymous delegates, which allows an almost direct conversion of the Scheme version, with the caveats that this version is limited to ints (while Scheme will handle mixed numeric types), and it's a more verbose 25 lines, largely because of the required class structure:
// Sum Of Squares using System; using System.Collections; class Exam { delegate int Combiner (int a, int b); static int Accumulate (Combiner combiner, int null_value, IEnumerator l) { if (!l.MoveNext()) return null_value; return combiner ((int) l.Current, Accumulate (combiner, null_value, l)); } static int SumOfSquares (IEnumerable e) { return Accumulate ( delegate (int a, int b) { return a*a + b; }, 0, e.GetEnumerator ()); } public static void Main () { Console.WriteLine (SumOfSquares (new int[]{1, 2, 3, 4, 5})); } }
Notice that I'm using IEnumerator.Current as the (car) of the list and the IEnumerator instance is the (cdr) of the list. This is a reasonable mapping, and allows any .NET collection type to be used with the Exam.SumOfSquares method.
Java is considerably more verbose to stay in the same style as the Scheme program, requiring 37 lines. It's similar to C# in that it uses the standard Iterator interface to represent a list, with next() mapping to (car) and the Iterator instance containing (cdr). This program also uses an interface declaration and an anonymous inner class to implement the callback (to maintain the feel from the Scheme program):
import java.util.*; class sos { static interface Combiner { int combine (int a, int b); } static int accumulate (Combiner combiner, int null_value, Iterator l) { if (!l.hasNext()) return null_value; Object next = l.next (); return combiner.combine ( ((Integer) next).intValue (), accumulate (combiner, null_value, l)); } static int sumOfSquares (Iterator l) { return accumulate ( new Combiner () { public int combine (int a, int b) { return a*a + b; } }, 0, l ); } public static void main (String[] args) { List l = new ArrayList (); l.add (new Integer (1)); l.add (new Integer (2)); l.add (new Integer (3)); l.add (new Integer (4)); l.add (new Integer (5)); System.out.println (sumOfSquares (l.iterator())); } }
The real "killer" for Java is the lack of integration between the language and the class libraries. .NET arrays implement the collections interfaces, which is why the C# Exam.SumOfSquares method can accept an array of ints, as well as a System.Collections.ArrayList, a System.Collections.Generic.List<int>, or anything else implementing the collections interfaces (such as C# 2.0 iterators).
Java, on the other hand, requires manually constructing and populating a java.util.ArrayList or some other type to obtain an Iterator; there's no way to convert an existing array into an iterator, short of writing your own wrapper class.
This language "comparison" really doesn't mean anything -- the functional paradigm isn't widely used (sadly) -- but functional programming might not be used because the common "mainstream" languages support it so poorly. C# is providing better functional support, going so far as supporting a limited form of closures (and C# 3.0 will make lambda functions even easier to define). Java, sadly, does not seem to be making the functional paradigm easier to use, at least not from what I've seen.
System.Diagnostics Tracing Support
As I wrote Mono's original Trace support infrastructure, I should probably get around to implementing the much improved 2.0 version. Which means I first need to understand it. Fortunately Mike Rousos is documenting how it works:
Mono.Unix Reorganization
Brad Abrams, author of the Framework Design Guidelines, recently posted his precon slides for the recent PDC.
I quickly read through it to see how it well Mono.Unix follows it. I also recently ran FxCop on Mono.Posix.dll, with many interesting results.
One major point that Abrams' slides pointed out is on page 53:
- Avoid having types designed for advanced scenarios in the same namespace as types intended for common programming tasks.
- Do ensure that each main feature area namespace contains only types that are used in the most common scenarios. Types used in advanced scenarios should be placed in subnamespaces.
This is completely different from how Mono.Unix currently operates, as it places both low-level classes such as Syscall and high-level classes such as UnixStream into the same namespace. The only difference between thw low-level and high-level is the Unix prefix present on the high-level classes. This is a problem.
It's a problem because when looking at the class view or the documentation you get lost looking at the dozens of low-level types such as AccessMode, ConfStr, and Syscall, as the high-level wrapper classes -- having a Unix prefix, will be after most of the types developers (hopefully) won't be interested in.
My solution is to separate the low-level classes into a Mono.Unix.Native namespace. The Mono.Unix namespace will be used for high-level types following CLS conventions (such as PascalCased types and methods) such as UnixFileSystemInfo, and for .NET integration classes such as UnixStream.
This change went into mono-HEAD today. All of the existing low-level Mono.Unix types have been marked [Obsolete], with messages directing users to use the appropriate Mono.Unix.Native types. Alas, some of these low-level types are used in properties or as the return types of methods in the high-level classes. These have been marked [Obsolete] for now, with a message stating that the property type or method return type will change in the next release. "Next release" in this case will be 1.1.11 or 1.2 (as I'm assuming the release of 1.1.10, which is when developers will actually see these messages if they don't follow mono-HEAD).
I'm also interested in better CLS compliance in the high-level classes. At present many of them are [CLSCompliant(false)] because they use non-CLS-compatible types such as uint or ulong. Should these be changed to CLS-compliant types? Any such changes should be done now (i.e. before 1.1.10), to allow any migration time.
Major Change to Nullable Types
Poor Martin Baulig -- Microsoft has changed the design of System.Nullable so that it's far more integrated into the runtime. This change impacts things as fundamental as the box and unbox instructions...
Frogger under Mono
DotGNU Portable.NET provides a .NET Curses wrapper for creating nifty console-based programs using the ncurses library. It is possible to run this under Mono.
Alas, the provided configure script doesn't work very well with Mono, so we need to do things manually. The following will allow you to build Portable.NET's Curses.dll ncurses wrapper and run the bundled Frogger game.
- Download and extract pnetcurses:
$ wget http://www.southern-storm.com.au/download/pnetcurses-0.0.2.tar.gz $ tar xzf pnetcurses-0.0.2.tar.gz $ cd pnetcurses-0.0.2
- Build the libcsharpncurses.so helper library. You can ignore
the warnings produced by GCC.
$ cd help $ gcc -shared -o libcsharpncurses.so *.c -lncurses input.c: In function `CursesHelpGetNextChar': input.c:69: warning: integer constant is too large for "long" type $ cd ..
- Build the Curses.dll wrapper library:
$ cd src $ mcs -t:library *.cs -out:Curses.dll Compilation succeeded $ cd ..
- Create a Curses.dll.config file so that Mono can load the
appropriate native libraries. This file should go into the same directory
as Curses.dll.
$ cd src $ cat > Curses.dll.config <<EOF <configuration> <dllmap dll="cygncurses5.dll" target="libncurses.so"/> <dllmap dll="libcsharpcurses-0-0-1.dll" target="libcsharpncurses.so"/> </configuration> EOF $ cd .. - Build the Frogger.exe demo program:
$ cd frogger $ cp ../src/Curses.dll* . $ mcs -t:exe -out:Frogger.exe -r:Curses.dll *.cs Compilation succeeded $ cd ..
- Execute Frogger.exe with Mono:
$ cd frogger $ LD_LIBRARY_PATH=`pwd`/../help mono Frogger.exe
Mono.Unix Documentation Stubs
I've just added the Mono.Unix documentation stubs. Next step is to start documenting the members, mostly by copy/pasting the current Mono.Posix documentation.
Lessons learned:
- Documentation can be out of date, especially the documentation to update existing documentation. Oops. We're supposed to use the monodocer program.
- The correct command to generate/update documentation is: monodocer -assembly:assembly-name -path:directory-name.
- The monodocer program didn't like me (it generated a NullReferenceException because of a class in the global namespace). Patch in svn-trunk.
- Documentation is an alternate view of a class library
That final point is the major point: through it, I realized that several methods in Mono.Unix which should be private were instead public. Oops. Obviously I need to document things more often...
