So, here I am writing some code to download files from remote servers, and I wandered across one of the dumbest designs in the .NET framework.

When using WebRequest (really, the child class HttpWebRequest ) write some code to make an HTTP request of the faraway server (I’m showing the non-async way to keep it simple):

var webRequest = (HttpWebRequest) WebRequest.Create("http://microsoft.com/foo");
webRequest.AllowAutoRedirect = true;
webRequest.Method = WebRequestMethods.Http.Get;

var webResponse = webRequest.GetResponse() as HttpWebResponse;

if( webResponse != null ) {
    Console.WriteLine("Status: {0}", webResponse.StatusCode );

    // ... finish reading from the webResponse....
}

That’s fine and dandy. Except one thing; if the request returns 404 NOT FOUND, GetResponse throws an Exception!?!

Now, I can understand how someone might think that not finding the requested resource is exceptional behavior, but it’s not a condition that I can check for before making the request itself. It’s certainly not in the same class of responses as timing out, or not having network connectivity.

So, through no fault of my own, the code throws an exception without even letting me check to see the result. Which is at best, rude and at worst, the lousiest way to handle that which is not exceptional behavior.

Eric Lippert calls these “Vexing Exceptions”

This only gets worse when using the Task Parallel Library in .NET 4.0 :

var webRequest = (HttpWebRequest) WebRequest.Create("http://microsoft.com/foo");
webRequest.AllowAutoRedirect = true;
webRequest.Method = WebRequestMethods.Http.Get;

var task = Task.Factory.FromAsync<WebResponse>(webRequest.BeginGetResponse, webRequest.BetterEndGetResponse , this)
    .ContinueWith(asyncResult => {
        var webResponse = asyncResult.Result as HttpWebResponse;
        Console.WriteLine("Status: {0}", webResponse.StatusCode );
        // ... finish reading from the webResponse....
    } );

So now, rather than checking the response, I have to check the asyncResult.IsFaulted property, grab the asyncResult.Exception property (which is actually an AggregateException), call aggregateException.Flatten() and iterate thru the collection to find the WebException, and ask that object for the WebResponse and from there, get the damn StatusCode… just so I can find out WHY the damn call failed!

I searched all the way thru the source code in the .NET framework, and I didn’t see any workaround… then it hit me. All we have to do is make the call not throw a bleedin’ exception when it actually created a WebResponse.

A quick little static class with a couple extension methods fixes the problem nicely (taken from the CoApp project for where I wrote it):

//-----------------------------------------------------------------------
//
//     Copyright (c) 2011 Garrett Serack. All rights reserved.
//
//
//     The software is licensed under the Apache 2.0 License (the "License")
//     You may not use the software except in compliance with the License.
//
//-----------------------------------------------------------------------

namespace CoApp.Toolkit.Extensions {
    using System;
    using System.Net;

    public static class WebRequestExtensions {
        public static WebResponse BetterEndGetResponse(this WebRequest request, IAsyncResult asyncResult) {
            try {
                return request.EndGetResponse(asyncResult);
            }
            catch (WebException wex) {
                if( wex.Response != null ) {
                    return wex.Response;
                }
                throw;
            }
        }

        public static WebResponse BetterGetResponse(this WebRequest request) {
            try {
                return request.GetResponse();
            }
            catch (WebException wex) {
                if( wex.Response != null ) {
                    return wex.Response;
                }
                throw;
            }
        }
    }
}

Now, just call webRequest.BetterGetResponse() or webRequest.BetterEndGetResponse() for the async version. Presto, it returns the WebResponse object instead of throwing (and will still throw for truly exceptional reasons.)

If you want to use it with the TPL’s FromAsync methods, prefix your webRequest.BetterEndGetResponse with the a cast to (Func<IAsyncResult, WebResponse>) before it so that the compiler can curry the extension method into the appropriate delegate for you:

var task = Task.Factory.FromAsync<WebResponse>(
    webRequest.BeginGetResponse, (Func<IAsyncResult, WebResponse>)webRequest.BetterEndGetResponse , this)
    .ContinueWith(asyncResult => {
        // ... yada yada yada....
    } );

Of course, this does mean that you MUST check the webResponse.StatusCode, and that you probably still need want an exception handler, but if you’re trying to handle truly exceptional cases differently than simple response conditions, I think you’ll like what you see here.

I’d like to introduce a new member of our team; Tim Rogers.

Tim is a new full time employee at Microsoft who started this week, and as he just found out yesterday (his first day in the office), is going to be working as a full-time test resource for CoApp.

Tim has a few words about himself;

“Tim Rogers is new to Microsoft, fresh out of college.  He has been living in Iowa, attending Iowa State University.  Major interests include technical theater, bicycles, strategy games, and reading.  Tim insists that he is a boring and unimaginative person, so throwing rocks and other things at him is to be encouraged.  This bio was written by the aforementioned boring and unimaginative person.  Feel free to adjust or make things up to your hearts content.“

Apparently, Tim likes to talk about himself in the third person.

Tim is going to start by tackling the orchestration and test writing for the CoApp engine—the component that gets the most end-user exposure.

As he gets up to speed, he’ll likely start blogging and filling in the documentation where he can as well.

The Garrett welcomes Tim!

We’re nearing the point where we’re able to take open source projects and produce happy-shiny packages for them—uh yeah, nearing.  It’s gonna be a couple more weeks shaking out the tools that generate new Visual Studio project files for a given project and wire all that stuff up.

The good news though, is that we’re ready to have people start shallow-forking projects and sticking them into Github.

What’s a “shallow-fork” you say?

A fork happens when developers take a copy of source code from one software package and starts independent development on it, creating a distinct piece of software.  A shallow-fork is when the independent development continually brings forward changes from the original project, and attempts not to stray very far from it.  We do this so that we can make appropriate Windows changes (like new build scripts, or better API support) keep compatibility and not break the original project.  If the upstream project is willing to accept changes, great! If not (for whatever reason)… great! Either way, the package maintainer for the CoApp project will aim to keep the projects in sync as much as possible.

Linux distributions do this sort of thing all the time, in order to build packages for their specific version of Linux.  We’re essentially following in the same footsteps, but following the procedure that I’m setting out here, so that we can eventually produce packages of software for CoApp.

Can You Help?

We’re looking for people to start the leg-work for building packages—right now, this means creating a fork in github, and doing the work to get it compiling under Windows—no matter how it’s done (i.e., you can use a makefile, VS project files, a batch file… whatever). The idea is to just get a successful build that can be replicated on a known configuration–Visual Studio 2010 (even VS 2010 express), and the Windows SDK 7.1 … We’ll be automating the work later that creates well-built VS 2010 project files (and eventually other builds like mingw, etc), but that work will leverage the effort done now to create consistent shallow-forks.

What do you need?

1. The aforementioned Visual Studio 2010 (the express version will do), and the Windows SDK 7.1 . 
2. An account on github, and the ability to check stuff in (you can use msysgit or tortoisegit , or any other git client for Windows—even the Mercurial plugin hg-git, which lets you use git repositories as if they were mercurial repositories. (Note: I tried the hg-git plugin, and had to use the version here to get it to work) 
3. The ability to build software using Visual Studio—we need command line builds, but if you get it working from inside the IDE, we can show you how to automate that from the command line in a single step.
4. The ability to follow the pretty-darn-simple process below to produce the necessary build.
5. A current build of the CoApp tools. You can compile them yourself, or just download my latest snapshot build. Make it easy on yourself, put them in a folder that’s in the %PATH%.

Becoming a package maintainer

The first thing is to get yourself added as a CoApp package-maintainer on github—no worries, signing up for this isn’t a lifelong commitment. If you are interested in keeping it moving over the long haul, that’s great, but even if you can just do up a few projects and get them checked in, we’ll figure out the long term maintenance strategy later.

In order to get added to the package maintainer group you can send me a message on github (fearthecowboy) or via email, or heck, even just send me a tweet me. All I need is your github account id, and I’ll add you to the group.

Wait! Before You Begin…

Since shallow-forking is intended to continually merge in changes from upstream, please keep changes to the project and its files to an absolute minimum—don’t do any unnecessary changes, don’t reorganize the files, and don’t try to rebuild a ‘cleaner’ build process.  You only need to get it to the point where it builds a viable output.  As we progress, we’ll capture the data from the process and build shiny new project files that are much nicer—expending extra effort now doesn’t get you anywhere.

If you run into trouble, and need help

Check on IRC—#coapp on freenode.net (try the webclient if you don’t have IRC installed)—if I’m there, I’m more than willing you help you thru the steps (and fix stuff that needs fixing), or failing that on the mailing list.

The Process for Shallow-Forking for CoApp packages

Quick Note: This is the first iteration of the Shallow-Fork process—there are a number of deeper issues when forking projects that have multiple active branches, several configurations, etc.  What we’re trying to do here is lay out a common approach so that as we proceed, we can automate as much as possible over the long run.   This procedure is likely to evolve quite a bit as we account for more scenarios and add support for additional compilers, configurations and build types.

1. Find an open source C/C++ library or application that you’d like to fork.  You can select one off the list of things that we know we really need, or something else that of particular interest to you, and for which you’d like to see a CoApp package created.  If the project that you want to build has things that it depends on, you may need to go ahead and provide the shallow-fork for that as well—we do not want to rely on binary builds that someone else has provided, that would be kinda silly. 
2. Create a project in Github. Depending on what you’re trying to make a fork of, you’re going to do this in one of a few different ways.  If the project is already on github, create a fork for it in the coapp-packages organization. If the source is posted as a zip file, unpack it and create the project manually. If it’s in another VCS, there may be a generally-acceptable method to creating a fork in git.  Bottom line, get the project into Github in a project in the coapp-packages organization.
3. Clone the source code into your working directory.
4. Create a folder in the root of the project called COPKG. 
5. Create a text file in the folder called .buildinfo—this is the file that you will be putting the build steps into.The format of the .buildinfo file can be thought of something akin to a .css file.  Simple values need not be quoted (but it’s ok to do so).  It supports comments ( double-slash and /* */), C# style string literals (may be preceded by an @ symbol to make a string multi-line).

   Inside the .buildinfo file,  insert the following:

   #product-info  {
       product-name: "";
       version: "";
       original-source-location: "";
       original-source-website: "";
       license: "";
       packager: "";
   }

   You can fill in the values of each of the fields. An example for libjpeg (done by Rafael)

   #product-info  {
       product-name: "libjpeg";
       version: "8c";
       original-source-location: "http://ijg.org/files/jpegsr8c.zip";
       original-source-website: "http://ijg.org";
       license: "Custom license, see README";
       packager: "Rafael Rivera <rafael@withinwindows.com>";
   }

6. Create one or more Build ConfigurationsA “Build Configuration” is analogous to a build configuration with Visual Studio—You can create builds for x86, x64, as well as for different flavors (static library vs. dynamic library) etc.  It’s not necessary to explicitly create multiple build configuration for x86 and x64 if project compiles well under both without different commands (save for the selection of which platform).  You do want multiple configurations when the steps change between platforms (or you know that different files get processed—in some libraries, there are separate files for x86 vs. x64), or if you have different expected outputs—i.e. some libraries allow you to generate a static and a dynamic version of the same library).  Generally speaking, err on the side of caution and don’t create multiple build configurations excessively—for projects that behave themselves, we’ll autogenerate the alternate configurations when we get around to creating new Visual Studio projects.

   The build configuration syntax:

   name  {
       /* optional -- defaults to vc10-x86 */
       compiler: compiler-tag ;
               // currently supported values are vc10-x86 and vc10-x64
               //... we'll add more compilers in the future.
   
       /* optional -- only used if this project has dependencies on others
                   -- may be specified mutiple times */
       uses: bld-cfg="..\path" ;
               // bld-cfg is the build configuration in dependent project
               // the 'blg-cfg=' part can be omitted to depend on the default
               // path is to the root path of the dependent project that
               // contains a COPKG\.buildinfo file.
   
       targets: { ... } ;
               // a comma seperated list of the binary files that are output
               // that are of significance
   
       build-command: "";
               // either a single command line, or a batch script
               // that has the commands to build the targets listed above.
   
       clean-command: "";
               // either a single command line, or a batch script
               // that has the commands to remove all temporary files
               // created during a build.
   }

   The above libjpeg example continued – note that there isn’t any uses option:

   x86 {
       compiler: vc10-x86;
   
       targets: {
           // main library
           "Release\jpeg.lib",
   
           // extra utilities
           "cjpeg\Release\cjpeg.exe",
           "djpeg\Release\djpeg.exe",
           "jpegtran\Release\jpegtran.exe",
           "rdjpgcom\Release\rdjpgcom.exe",
           "wrjpgcom\Release\wrjpgcom.exe",
       };
   
       build-command:@"
           copy jconfig.vc jconfig.h
           copy makejsln.v10 makejsln.sln
           copy makeasln.v10 makeasln.sln
           copy makejvcx.v10 jpeg.vcxproj
           copy makecvcx.v10 cjpeg.vcxproj
           copy makedvcx.v10 djpeg.vcxproj
           copy maketvcx.v10 jpegtran.vcxproj
           copy makewvcx.v10 wrjpgcom.vcxproj
           copy makervcx.v10 rdjpgcom.vcxproj
           msbuild /p:Platform=Win32 /p:Configuration=Release makejsln.sln
           msbuild /p:Platform=Win32 /p:Configuration=Release makeasln.sln
       ";
   
       clean-command:@"
           attrib -S -H -R *
           del /Q *.sdf *.suo *.sln *.vcxproj *.user jconfig.h 2>NUL
           rmdir /S /Q Release 2>NUL
   
           rmdir /S /Q cjpeg 2>NUL
           rmdir /S /Q djpeg 2>NUL
           rmdir /S /Q ipch 2>NUL
           rmdir /S /Q jpegtran 2>NUL
           rmdir /S /Q rdjpgcom 2>NUL
           rmdir /S /Q wrjpgcom 2>NUL
       ";
   };

    

7. Once you have the .buildinfo file complete, you can run the pTk tool from the command line (make sure you’re in the root of the project).You should be able to run a build:

   C:\forks\libjpeg>ptk clean
   
   CoApp Project pTk Version 1.0.2.906 for x64
   Copyright (c) Garrett Serack, CoApp Contributors 2010-2011. All rights reserved
   CoApp portingToolkit for porting apps
   -------------------------------------------------------------------------------
   
       ( build happens here ... )
   
   Project Built.
   
   C:\forks\libjpeg>

   … and a clean :

   C:\forks\libjpeg>ptk clean
   
   CoApp Project pTk Version 1.0.2.906 for x64
   Copyright (c) Garrett Serack, CoApp Contributors 2010-2011. All rights reserved
   CoApp portingToolkit for porting apps
   -------------------------------------------------------------------------------
   
       ( build happens here ... )
   
   Project Built.
   
   C:\forks\libjpeg>

   … and a verify (where it builds, verifies the build targets, cleans, and verifies everything is clean):

   C:\forks\libjpeg>ptk verify
   CoApp Project pTk Version 1.0.2.906 for x64
   Copyright (c) Garrett Serack, CoApp Contributors 2010-2011. All rights reserved
   CoApp portingToolkit for porting apps
   -------------------------------------------------------------------------------
    ( build happens here ... )
   
   Targets Verified.
   Project Verified.
   
   C:\forks\libjpeg>

    

8. Once it does what you expect, commit and push the clean version up to github, and give me a shout.

If you are interested in showing up:

Time: 7:30 PM-10 PM
Where: iQmetrix offices
250 Howe Street – Suite 1210
Vancouver, BC
V6C 3R8 CANADA

I’d love to have ya’ll out!

And, Pizza & drinks are provided!

G

As you can see, we’re on a new site; this is powered by Orchard, an open source CMS for Windows.  I chose this, primarily because it’s built to run on Azure as well as Windows Server, and the kind folks at Microsoft have given me an Azure account with plenty of resources.  I also spun up a Screwturn Wiki for documentation (it’s what the Orchard guys use for their site, so I followed suit). 

The new site is a tad bare right now, but hopefully we’ll get this all put together over the next couple of weeks.

A new code repository

As I mentioned on my blog last week, we also moved our code repository over to github:

• While I liked a lot of the things about Launchpad, the website is feeling slower and slower some days, and Bazaar, while offering the features that I like, isn’t getting the attention (and developer resources) that git is.  Combined with the fantastic innovation happening at Github, it’s undeniably the go-to place for open source development these days.
  
• And, having done some recent tests with git on Windows, it’s clear that it’s stable and feature rich enough for all my purposes.

Along with that I also talk about how to checkout the code and compile it up.  Yes, it’s still an iceberg (most of the code isn’t about what you see) but beneath that surface there is a huge amount of functionality lurking.

Current Status

The last six months have had a flurry of code development done, including essentially the entire CoApp engine—originally we were hoping that the volunteer work by the group at the University of Syracuse would produce a functional prototype,  it turned out to be too-aggressive of a goal.  Consequently, the entire engine was written in three months, and diverted my attention away from other things (this site being one of them)

Where we at:

CoApp Core:

• CoApp toolkit – all of CoApp’s shared code ends up in the toolkit project. A cornucopia of functionality, it provides a tremendous amount of simple, reusable functionality that is shared across all projects.
• CoApp Engine – we have a managed version of the engine with the basic v1 functionality complete, I’d say that it’s in a solid beta stage at this point. 
• CoApp Command Line Client – the command line client works pretty good, if a tad verbose on the command line. It’s pretty optimized for the happy-path at this point, but still a pretty good beta.

Publisher Tools:

• mkPackage – the first tool to create packages for CoApp has been working, but it turned out it took an awful lot of effort to build a package, so we’ve depricated that, and gone back to the drawing board. The result, is Autopackage—a tool designed to do what we really wanted, which is creating packages without messy XML, no need to understand MSI or WiX, or even code signing. 
• Autopackage—is designed to eliminate 100% of the understanding and guesswork in creating packages for consumption.  It’s already functional and able to produce packages, and is nearing the ‘alpha’ stage.  Eric has done an amazing amount of work in a short time to produce something that is going to get a lot of attention.
• simplesigner – even simple code signing operations are a hassle in Windows, and when you add .NET strong-naming into the mix, it’s all a little difficult to get a hold of, so I wrote the simplesigner tool.. It does exactly what it says, makes digital signing software on Windows simple.  For .NET executable binaries, it both signs and strong-names the result, and eliminates the guesswork and frustration entirely.

Developer Tools:

• Scan—Trevor knocked out the first beta of this tool way back in October,  It does exactly as I‘d hoped, a very useful scan of a source tree, and brings back a very large amount of useful data.  On top of it’s use as part of mkSpec, it’s also quite handy on it’s own to get an idea of what a project uses, and how it all fits together. Solid 1.0 material.
• Trace—The ultimate evolution of my original 32 bit trace utility, Rafael and I have put an insane amount of work into this tool.  Trace can watch a program, and all of its child processes and record every single file access (and how it was accessed), and record the environment, and command lines for every process.  It works for nearly every type of application we’ve thrown at it: 32bit, 64bit, .NET, cygwin, native… The data it gets back is extremely valuable—we use it primarily to feed into the mkSpec tool to produce project files, but it has a lot of use on it’s own.  It even captures some data that you can’t get from Sysinternals’ ProcMon. We’re pretty much 1.0 gold here.

  mkSpec—The tool that generates a ‘compiler-independent’ project file from scan and trace data.  It’s getting really close to beta stage—I just need to leverage the latest trace info and we should be on our way.

  mkProject – the secret sauce of the entire CoApp project.  mkProject takes project info from mkSpec and produces Visual Studio project files that are consistent, clean and support easy use of advanced features like PGO optimization, etc.  Still a work in progress—probably a few weeks away from a beta.

• testpackagemaker – development of the package manager requires a lot of testing examples, and this tool simplifies the generation of native and .NET executables and libraries that support side-by-side so we can build packages for testing. Solid 1.0 stuff here.
• pTk – a recent addition to the CoApp developer tool lineup, pTk (aka the porting Toolkit)  is a build automation tool (no, not like make or msbuild).  pTk provides a method for package maintainers to express ‘how-to-build’ a given project so that the process can be automated by other tools without understanding anything about the build whatsoever (this will let us automate the trace/mkSpec/mkProject process *a lot*.)  Rather than having the package maintainer/developer express a build process in its terms, it simply is a way of letting the developer write down the commands to build a given project (as a command, or batch script, or whatever) and provides an automatable wrapper for that.  This is definitely release candidate material.

Productivity Tools:

• quicktool – During development, it’s always so useful to share code, screenshots etc. when working remotely (either live, via Skype, Lync or IRC) or even via Twitter or email. quicktool provides a system-wide hot-key to uploading images, formatted source snippets, and shortening URLs without having to bring up a separate tool or browser to do so.  Faster and more convenient than clumsy websites like pastebin, it allows developers to easily share information at a single keypress.

That’s about it for today… over the next few days I’ll be posting some tutorials (ie, how to shallow fork a Project for CoApp) and some more information on how you can get involved if you’re inclined.

Fixing Bzr for Windows

Bzr on Windows supports fast import—except that the installer doesn’t install the fast-import python module when it installs python.  Easy enough to fix. You need to grab a copy of the fast-import module from launchpad:

C:\tmp>bzr clone lp:python-fastimport
Connected (version 2.0, client Twisted)
Authentication (publickey) successful!
Secsh channel 1 opened.
Branched 301 revision(s).

And then combine it with the library.zip file that’s already in the Bazaar install directory using zip (I used the command line zip utility from infozip) …

C:\tmp>cd python-fastimport

C:\tmp\python-fastimport>zip -r "c:\Program Files (x86)\Bazaar\lib\library.zip" *

Once you’ve done that, it’s really trivial to move a .bzr repository to a git one:

C:\tmp>mkdir someproject

C:\tmp>cd someproject

C:\tmp\someproject>git init
Initialized empty Git repository in C:/tmp/someproject/.git/

C:\tmp\someproject>bzr fast-export --plain YOUR-BZR-REPO | git fast-import

Where YOUR-BZR-REPO is the path to your bazaar repository–it can be a path to a local repo, or a remote location (like, lp:projectname).

It will spit out some information, and then you need to:

C:\tmp\someproject>git checkout master
Already on 'master'

Now, you’ve got a shiny new git repo from your bazaar repo, and you can play with it as you will.

While I liked a lot of the things about Launchpad, the website is feeling slower and slower some days, and Bazaar, while offering the features that I like, isn’t getting the attention (and developer resources) that git is.  Combined with the fantastic innovation happening at Github, it’s undeniably the go-to place for open source development these days.

And, having done some recent tests with git on Windows, it’s clear that it’s stable and feature rich enough for all my purposes.

Updated for use on Github:

The following is the instructions on how to build the current CoApp bits:

0. You need to have the following tools installed in order to build CoApp:

Visual Studio 2010 (I’m told that VS 2010 Express will work)

Windows SDK 7.0 or higher — http://bit.ly/bOoxJT

Windows WDK 7.1 — http://bit.ly/cU1lvH

msysgit for Windows 1.7.4 — http://code.google.com/p/msysgit/downloads/detail?name=Git-1.7.4-preview20110204.exe
(this is a command line git client –there are also other GUIs available.)

Putty (& Pageant) — http://bit.ly/awE3jd –get the putty-0.60-installer.exe

NOTE: I had issues with the SSH client in msysgit; you may need to run the command :

c:\coapp> SET GIT_SSH=PLINK.EXE

(assuming that PLINK is installed in your PATH somewhere)

1.0. Create and account for Github.

1.1. Run Pageant, load your private key.

2. Download the following script:

https://github.com/downloads/fearthecowboy/coapp/coapp-src.cmd

3. Unzip the script into your working directory (where you want to check out the source to)

4. From the command line:

 C:\coapp> coapp-src

   Usage:
   ------
   coapp-src.cmd [OPTION]

   where [OPTION] is one or more of:

       core        - just the core projects to build the package manager
       gui         - the prototype work on the GUI client
       tools       - the developer and publisher tools
       guts        - the guts of the bootstrap and installershim
       other       - garrett's other tools and oddities
       all         - all of the above

       update      - merges updates for any projects that are already checked out.

    You can check out a portion of the code (or all, if you want) by using the script:

   C:\coapp> coapp-src core

Will check out just the core (coapp-cli, coapp-toolkit and coapp-solution).

4. Open the coapp-solution\coapp-tools.sln solution file in Visual Studio 2010

If you didn’t get all of the projects, you’ll see an error when visual studio tries to load

projects that are not checked out; this is ok, you can ignore the error (or delete the

projects that are missing from your solution file)

5. Build it. (ctrl-shift-b)

Make sure you build the debug version, you won’t be able to build the release

(you’d need my private cert, and it does some funky stuff during the build process)

6. The output will be in [root]\output\any\debug\bin:

And you should be able to run the coapp.exe in that directory:

You’ll notice the packages that I installed from the http://coapp.org website are installed.

All coapp binaries (except for the bootstrapper itself) are built as ‘any’ (meaning they will run 64bit on x64 systems, and 32bit on x86  systems)

So, the CoTask task factory also lets you pass the event listeners as part of the task creation call, you just have to instantiate the TestEvents class and use an object initializer to set the value of the event handler:

// *** Example Two ***
// Listening to events
// but attaching before the task is started
var taskTwo = CoTask.Factory.StartNew(() => {
    for (var q = 10; q > 0; q--) {
        // simulate a longer running operation
        Thread.Sleep(200);

        // that occasionally notifies anyone who cares to listen.
        TestEvents.Invoke.Message1("Message From taskTwo", q);
    }
}, new TestEvents {
    Message1 = (s, i) => {
        Console.WriteLine("The Task said: [{0}] and [{1}] ", s, i);
    }
} );

Console.WriteLine("\r\n(Main Thread Completed)======>Press Enter to end.\r\n");
Console.ReadLine();

In line 12 you can see where I just tack on the instance of the TestEvents class as the last parameter in the call to StartNew() .  By passing the event listener earlier, this allows the caller to not miss out on any events:

Much Better!

Now, what if the class that had the events in it had more than one event:

 public class TestEvents : MessageHandlers<TestEvents> {
     public Action<string, int> Message1;
     public Action<string, string> Message2;
     public Action<string> Message3;
 }

Well, can go ahead and implement the ones you want, and leave the ones you don’t:

// *** Example Three ***
// Listening to multiple events
var taskThree = CoTask.Factory.StartNew(() => {
    for (var q = 10; q > 0; q--) {
        // simulate a longer running operation
        Thread.Sleep(200);

        // fire a different message for different values:
        switch(q % 3) {
            case 0:
                TestEvents.Invoke.Message1("Message1 From taskThree", q);
                break;
            case 1:
                TestEvents.Invoke.Message2("Message2 From taskThree", "Just more text");
                break;
            case 2:
                TestEvents.Invoke.Message3("Message3 From taskThree");
                break;

        }
    }
}, new TestEvents {
    Message1 = (aString, anInt) => {
        Console.WriteLine("The Task said a string and an int: [{0}] and [{1}] ", aString, anInt);
    },
    Message2 = (aString1, aString2) => {
        Console.WriteLine("The Task said two strings: [{0}] and [{1}] ", aString1, aString2);
    }
} );

Console.WriteLine("\r\n(Main Thread Completed)======>Press Enter to end.\r\n");
Console.ReadLine();

And when you run that, you see that we’re able to handle the messages that we’re interested in:

Note the lack of messages for Message3.

You can also have the Task actually ask for a result from the listener as well—if nobody’s listening, it’ll return the ‘default’ value (0 or null) for the type requested:

public class TestEvents : MessageHandlers<TestEvents> {
    public Action<string, int> Message1;
    public Action<string, string> Message2;
    public Action<string> Message3;
    public Func<int, bool> ShouldWeCancel;
}

And, a little code showing how it’s used:

// *** Example Four ***
// Listening to multiple events
var taskFour = CoTask.Factory.StartNew(() => {
    for (var q = 0; q > 100; q++) {
        // simulate a longer running operation
        Thread.Sleep(200);

        if( TestEvents.Invoke.ShouldWeCancel(q)) {
            // Note: this is not a really good way to actually cancel a Task
            //       you should really use a TaskCancellationSource object
            //       and do it right.
            break;
        }

        // fire a different message for different values:
        switch(q % 3) {
            case 0:
                TestEvents.Invoke.Message1("Message1 From taskFour", q);
                break;
            case 1:
                TestEvents.Invoke.Message2("Message2 From taskFour", "Just more text");
                break;
            case 2:
                TestEvents.Invoke.Message3("Message3 From taskFour");
                break;
        }
    }
}, new TestEvents {
    Message1 = (aString, anInt) => {
        Console.WriteLine("The Task said a string and an int: [{0}] and [{1}] ", aString, anInt);
    },
    Message2 = (aString1, aString2) => {
        Console.WriteLine("The Task said two strings: [{0}] and [{1}] ", aString1, aString2);
    },
    ShouldWeCancel = (anInt) => {
        if( anInt > 12 ) {
            return true;
        }
        return false;
    }
} );

Console.WriteLine("\r\n(Main Thread Completed)======>Press Enter to end.\r\n");
Console.ReadLine();

Voila! The task is now asking for information from the listener:

You can see that the Task broke out of the loop when the listener responded to the ShouldWeCancel query with true.

There are obviously a few limitations in using a function to get data back from the listener—it’s only able to get a value back from the first listener that responds (I’ve got an idea for how to aggregate up stuff, but not an actual personal use case yet, so unless someone asks, that’ll be on the ‘todo later’ list … ) … and you can’t tell the difference between a zero or null response and the lack of response. Still, kinda handy in a couple of cases.

Next time, I’ll show you how events bubble their way up—from any code that the Task ends up calling, as well as from any child tasks that are created.

Before we get too deep…

I should note that asynchronous code in .NET 4 make rather extensive use of Lambda Expressions—If you’re not familiar with lambdas, you’d better go get familiar, because this stuff is gonna be a tad hard to understand without it. It’s ok, I’ll wait…

And, I’m not going to go into too much detail about the basics of using the TPL; I recommend that you watch Ander’s video (to get an idea of the motivation) and perhaps read thru Eric Lippert’s blog posts on Asynchrony (and really, start with his earlier posts on Continuation Passing Style)—if you are a .NET developer, this stuff will change your life.

The following is a simple example for how we start an asynchronous task using the TPL:

// *** Example Zero ***
// Starting a task using the TPL
var taskZero = Task.Factory.StartNew(() => {
    for (var q = 10; q > 0; q--) {
        // simulate a longer running operation
        Thread.Sleep(200);

        // Wish we could tell someone about
        // what's going on in this operation
        // and Writeline is hardly a good idea
        // when you are in another thread.
        Console.WriteLine("Progress Message from taskZero", q);
    }
});

Console.WriteLine("(Main Thread Completed)======>Press Enter to end.\r\n");
Console.ReadLine();

Which will produce output like:

You’ll notice that the main thread completes (and is sitting around waiting at the ReadLine() ) while the Task executes.

And getting to the point…

Ok, so now I can illustrate some code that I’ve written as part of CoApp that fills a void where I was creating complex asynchronous operations and wanted to be able to broadcast arbitrary events from that operation, without having to pass an object along to each step in the operation.

Let me put this another way…

I’d like to start an operation—say, “Install a package”—and I’d like the calling context to be able to subscribe to events for that particular operation (regardless of what events the operation directly (or indirectly) broadcasts.  Yes, I *could* create a class that has a bunch events, and pass that from function to function in the entire call tree for that operation, but that would be terribly unwieldy, and frankly, would be a real pain when I wanted to reuse other asynchronous tasks as child tasks, and they may not take the same class for events.

Additionally, that asynchronous operation may spawn off additional asynchronous operations of its own—perhaps to download some dependencies—and it’d be nice to be able to pick up download progress notifications so we could show some UI that keeps the user informed as to what’s actually going on.

Hrm… Howsabout I explain this with some code.  Let’s assume my little task has an event that it’d like to notify. First, we declare a class for the messages:

public class TestEvents : MessageHandlers<TestEvents> {
    public Action<string, int> Message1;
}

And then an example of the task firing the event:

// *** Example One ***
// Firing an event
var taskOne = CoTask.Factory.StartNew(() => {
    for (var q = 10; q > 0; q--) {
        // simulate a longer running operation
        Thread.Sleep(200); 

        // that occasionally notifies anyone who cares to listen.
        TestEvents.Invoke.Message1("Message From taskOne", q);
    }
});

Now,  you should notice two things (‘specially since I highlighted the lines…). First, I use my own CoTask Task factory instead of the TPL’s Task—this just lets me track tasks a bit better, and handles some background work for all this stuff. Functionally, it should be identical, and still returns the same Task objects. Second, I can now use the class I created before to fire off a message (in this case, Message1 takes a string and an int for parameters) using some mysterious static Invoke dohickey on my TestEvents class .

Of course, nothing is listening to the events yet, let’s wire that up:

// perhaps we'll slack off for a while
Thread.Sleep(400); 

// start listening to messages after the task has started.
// (we probably missed some!)
((TestEvents)taskOne).Message1 += (aString, anInt ) => {
    Console.WriteLine("The Task said: [{0}] and [{1}] ", aString, anInt);
};

Console.WriteLine("(Main Thread Completed)\r\n**Press Enter to end**")
Console.ReadLine();

When we run this, we’ll end up with some rather interesting output:

Shiny! You’ll notice in this example, that we’ve actually missed a couple of events (exacerbated by the Thread.Sleep() in line 2.). Regardless, the main thread completes and waits, and you can see my event handler in the lambda expression is getting called.
The beauty of this approach is that we can see how I can launch a longer running operation, and accept events from it, without having to explicitly tell the code in the operation about the event listener.

In the next part as I dig deeper, I’ll show how we can ensure we never miss our messages, and how we can listen for more than one at a time.

As I mentioned in my last post, CoApp’s design relies heavily on the use of Symbolic Links (symlinks)—a feature which is implemented fairly well in Windows Vista and beyond, but is missing in earlier versions of Windows.

Cygwin tried to work around the limitation by using shortcuts (.LNK files), which are files used by the Windows Shell for all the shortcuts in the start menu.  Not a bad way to go, but unfortunately, not very compatible with a lot of things, since most programs don’t specially recognize .LNK files, although the POSIX emulation in cygwin did.

I was thinkin’ that any acceptable solution should work with as much software as possible, without requiring special understanding, nor should any additional software be required (ie, a filter driver of some sort).

Our options then, are somewhat limited. On Windows XP and Windows Server 2003 we *do* have features like junctions (like symlinks, but only for directories, and only on the same volume), hard links (where more than one directory entry points to the same physical file) and alternate streams (the ability to store extra data with a file).

Hmmm.

Junctions seem to work fairly well for directories, with the caveat that actually manipulating junctions requires some tricky code with DeviceIOControl calls.

For file symlinks, I’ve used NTFS hardlinks, but when I create the link, I store some additional metadata with the file itself in an alternate stream (under the name “legacySymlinkInfo”) which allows tools to find out what the canonical filename really is, and also to find out what other files are linked to the same physical file.

Of course, non-CoApp software could still manipulate these files for little regard as to their status as ‘symlinks’, so the legacySymlinkInfo really can’t always be relied upon. But we can make a pretty good effort to ‘clean-as-we-go’ to keep that information up-to-date. If a file gets renamed or moved, the metadata may not be entirely accurate, so when we check the data, we make an attempt to validate that it is accurate, and update it with what we know. If it sounds like this is dodgy, I wouldn’t disagree, but in practice this seems to work really well.

Lastly, we need to be able to redirect a symlink, even when it is in use.  This seems trivial enough to do with junctions, but with files, we have to resort to moving the file out of the way, and creating a new linked file (which we can do, even if the file locked).

Admittedly, there are a few limitations that this design has:

Symlinks are only supported on the local system (no cross volume links, nor can they link to UNC paths)

Symlinks must be absolute references (not linking to relative paths)

The relationship between the linked file and the canonical file is potentially ephemeral, but simple examination of the files allows us to rebuild the information.

In the worst case scenario, if files get unlinked, they still work as regular files.

 

Luckily, CoApp’s requirements for what symbolic links need to do doesn’t have to be as flexible as full blown symlinks, and these limitations are not a hindrance at all.

I’ve just added this code to the CoApp toolkit (written in C#), and I’ll be playing with it quite a bit over the next few weeks. Once I’m sure that everything works as I like, if there is interest, I may implement a pure native version of the code so that other utilities and software can do the same thing and play nice too.

You can either comment here, come join mailing list (join the team at https://launchpad.net/~coapp-developers) and continue the conversation!