April 13th, 2011 By Garrett Categories: CoApp, Development

In my last post, I showed how we’re binding event handlers to the Task after it’s already been started. This of course, is probably not what you want, as it’s possible to lose some events if the task starts up quick enough.

So, the CoTask task factory also lets you pass the event listeners as part of the task creation call, you just have to instantiate the TestEvents class and use an object initializer to set the value of the event handler:

// *** Example Two ***
// Listening to events
// but attaching before the task is started
var taskTwo = CoTask.Factory.StartNew(() => {
    for (var q = 10; q > 0; q--) {
        // simulate a longer running operation
        Thread.Sleep(200);

        // that occasionally notifies anyone who cares to listen.
        TestEvents.Invoke.Message1("Message From taskTwo", q);
    }
}, new TestEvents {
    Message1 = (s, i) => {
        Console.WriteLine("The Task said: [{0}] and [{1}] ", s, i);
    }
} );

Console.WriteLine("\r\n(Main Thread Completed)======>Press Enter to end.\r\n");
Console.ReadLine();

In line 12 you can see where I just tack on the instance of the TestEvents class as the last parameter in the call to StartNew() .  By passing the event listener earlier, this allows the caller to not miss out on any events:

task_2

Much Better!

Now, what if the class that had the events in it had more than one event:

 public class TestEvents : MessageHandlers<TestEvents> {
     public Action<string, int> Message1;
     public Action<string, string> Message2;
     public Action<string> Message3;
 }

Well, can go ahead and implement the ones you want, and leave the ones you don’t:

// *** Example Three ***
// Listening to multiple events
var taskThree = CoTask.Factory.StartNew(() => {
    for (var q = 10; q > 0; q--) {
        // simulate a longer running operation
        Thread.Sleep(200);

        // fire a different message for different values:
        switch(q % 3) {
            case 0:
                TestEvents.Invoke.Message1("Message1 From taskThree", q);
                break;
            case 1:
                TestEvents.Invoke.Message2("Message2 From taskThree", "Just more text");
                break;
            case 2:
                TestEvents.Invoke.Message3("Message3 From taskThree");
                break;

        }
    }
}, new TestEvents {
    Message1 = (aString, anInt) => {
        Console.WriteLine("The Task said a string and an int: [{0}] and [{1}] ", aString, anInt);
    },
    Message2 = (aString1, aString2) => {
        Console.WriteLine("The Task said two strings: [{0}] and [{1}] ", aString1, aString2);
    }
} );

Console.WriteLine("\r\n(Main Thread Completed)======>Press Enter to end.\r\n");
Console.ReadLine();

And when you run that, you see that we’re able to handle the messages that we’re interested in:

task_3

Note the lack of messages for Message3.

You can also have the Task actually ask for a result from the listener as well—if nobody’s listening, it’ll return the ‘default’ value (0 or null) for the type requested:

public class TestEvents : MessageHandlers<TestEvents> {
    public Action<string, int> Message1;
    public Action<string, string> Message2;
    public Action<string> Message3;
    public Func<int, bool> ShouldWeCancel;
}

And, a little code showing how it’s used:

// *** Example Four ***
// Listening to multiple events
var taskFour = CoTask.Factory.StartNew(() => {
    for (var q = 0; q > 100; q++) {
        // simulate a longer running operation
        Thread.Sleep(200);

        if( TestEvents.Invoke.ShouldWeCancel(q)) {
            // Note: this is not a really good way to actually cancel a Task
            //       you should really use a TaskCancellationSource object
            //       and do it right.
            break;
        }

        // fire a different message for different values:
        switch(q % 3) {
            case 0:
                TestEvents.Invoke.Message1("Message1 From taskFour", q);
                break;
            case 1:
                TestEvents.Invoke.Message2("Message2 From taskFour", "Just more text");
                break;
            case 2:
                TestEvents.Invoke.Message3("Message3 From taskFour");
                break;
        }
    }
}, new TestEvents {
    Message1 = (aString, anInt) => {
        Console.WriteLine("The Task said a string and an int: [{0}] and [{1}] ", aString, anInt);
    },
    Message2 = (aString1, aString2) => {
        Console.WriteLine("The Task said two strings: [{0}] and [{1}] ", aString1, aString2);
    },
    ShouldWeCancel = (anInt) => {
        if( anInt > 12 ) {
            return true;
        }
        return false;
    }
} );

Console.WriteLine("\r\n(Main Thread Completed)======>Press Enter to end.\r\n");
Console.ReadLine();

Voila! The task is now asking for information from the listener:

task_4

You can see that the Task broke out of the loop when the listener responded to the ShouldWeCancel query with true.

 

There are obviously a few limitations in using a function to get data back from the listener—it’s only able to get a value back from the first listener that responds (I’ve got an idea for how to aggregate up stuff, but not an actual personal use case yet, so unless someone asks, that’ll be on the ‘todo later’ list … :D ) … and you can’t tell the difference between a zero or null response and the lack of response. Still, kinda handy in a couple of cases.

 

Next time, I’ll show you how events bubble their way up—from any code that the Task ends up calling, as well as from any child tasks that are created.

Comments Off
April 13th, 2011 By Garrett Categories: CoApp, Development

During the development of CoApp, I’ve enthusiastically embraced the .NET 4.0 Task Parallel Library (aka, the TPL).  It’s a set of APIs that make developers more productive by significantly simplifying the process of adding parallelism and concurrency to applications. I got religion around this last fall when I saw the PDC presentation that Anders Hejlsberg gave on “The Future of C# and VB”.  In there Anders talks about the shift to a fully asynchronous programming methodology. While the technology that he spoke of isn’t yet ready for production use today, the Task Parallel Library in .NET 4.0 provides the underlying framework that we can take advantage of right now.

Before we get too deep…

I should note that asynchronous code in .NET 4 make rather extensive use of Lambda Expressions—If you’re not familiar with lambdas, you’d better go get familiar, because this stuff is gonna be a tad hard to understand without it. It’s ok, I’ll wait…

 

And, I’m not going to go into too much detail about the basics of using the TPL; I recommend that you watch Ander’s video (to get an idea of the motivation) and perhaps read thru Eric Lippert’s blog posts on Asynchrony (and really, start with his earlier posts on Continuation Passing Style)—if you are a .NET developer, this stuff will change your life.

The following is a simple example for how we start an asynchronous task using the TPL:

// *** Example Zero ***
// Starting a task using the TPL
var taskZero = Task.Factory.StartNew(() => {
    for (var q = 10; q > 0; q--) {
        // simulate a longer running operation
        Thread.Sleep(200);

        // Wish we could tell someone about
        // what's going on in this operation
        // and Writeline is hardly a good idea
        // when you are in another thread.
        Console.WriteLine("Progress Message from taskZero", q);
    }
});

Console.WriteLine("(Main Thread Completed)======>Press Enter to end.\r\n");
Console.ReadLine();

Which will produce output like:

task_0

You’ll notice that the main thread completes (and is sitting around waiting at the ReadLine() ) while the Task executes.

And getting to the point…

Ok, so now I can illustrate some code that I’ve written as part of CoApp that fills a void where I was creating complex asynchronous operations and wanted to be able to broadcast arbitrary events from that operation, without having to pass an object along to each step in the operation.

Let me put this another way…

I’d like to start an operation—say, “Install a package”—and I’d like the calling context to be able to subscribe to events for that particular operation (regardless of what events the operation directly (or indirectly) broadcasts.  Yes, I *could* create a class that has a bunch events, and pass that from function to function in the entire call tree for that operation, but that would be terribly unwieldy, and frankly, would be a real pain when I wanted to reuse other asynchronous tasks as child tasks, and they may not take the same class for events.

Additionally, that asynchronous operation may spawn off additional asynchronous operations of its own—perhaps to download some dependencies—and it’d be nice to be able to pick up download progress notifications so we could show some UI that keeps the user informed as to what’s actually going on.

 

Hrm… Howsabout I explain this with some code.  Let’s assume my little task has an event that it’d like to notify. First, we declare a class for the messages:

public class TestEvents : MessageHandlers<TestEvents> {
    public Action<string, int> Message1;
}

And then an example of the task firing the event:

// *** Example One ***
// Firing an event
var taskOne = CoTask.Factory.StartNew(() => {
    for (var q = 10; q > 0; q--) {
        // simulate a longer running operation
        Thread.Sleep(200); 

        // that occasionally notifies anyone who cares to listen.
        TestEvents.Invoke.Message1("Message From taskOne", q);
    }
});

Now,  you should notice two things (‘specially since I highlighted the lines…). First, I use my own CoTask Task factory instead of the TPL’s Task—this just lets me track tasks a bit better, and handles some background work for all this stuff. Functionally, it should be identical, and still returns the same Task objects. Second, I can now use the class I created before to fire off a message (in this case, Message1 takes a string and an int for parameters) using some mysterious static Invoke dohickey on my TestEvents class .

Of course, nothing is listening to the events yet, let’s wire that up:

// perhaps we'll slack off for a while
Thread.Sleep(400); 

// start listening to messages after the task has started.
// (we probably missed some!)
((TestEvents)taskOne).Message1 += (aString, anInt ) => {
    Console.WriteLine("The Task said: [{0}] and [{1}] ", aString, anInt);
};

Console.WriteLine("(Main Thread Completed)\r\n**Press Enter to end**")
Console.ReadLine();

When we run this, we’ll end up with some rather interesting output:

task_1

Shiny! You’ll notice in this example, that we’ve actually missed a couple of events (exacerbated by the Thread.Sleep() in line 2.). Regardless, the main thread completes and waits, and you can see my event handler in the lambda expression is getting called.
The beauty of this approach is that we can see how I can launch a longer running operation, and accept events from it, without having to explicitly tell the code in the operation about the event listener.

In the next part as I dig deeper, I’ll show how we can ensure we never miss our messages, and how we can listen for more than one at a time.

April 7th, 2011 By Garrett Categories: CoApp

(cross-posted from the mailing list)

As I mentioned in my last post, CoApp’s design relies heavily on the use of Symbolic Links (symlinks)—a feature which is implemented fairly well in Windows Vista and beyond, but is missing in earlier versions of Windows.

Cygwin tried to work around the limitation by using shortcuts (.LNK files), which are files used by the Windows Shell for all the shortcuts in the start menu.  Not a bad way to go, but unfortunately, not very compatible with a lot of things, since most programs don’t specially recognize .LNK files, although the POSIX emulation in cygwin did.

I was thinkin’ that any acceptable solution should work with as much software as possible, without requiring special understanding, nor should any additional software be required (ie, a filter driver of some sort).

Our options then, are somewhat limited. On Windows XP and Windows Server 2003 we *do* have features like junctions (like symlinks, but only for directories, and only on the same volume), hard links (where more than one directory entry points to the same physical file) and alternate streams (the ability to store extra data with a file).

Hmmm.

Junctions seem to work fairly well for directories, with the caveat that actually manipulating junctions requires some tricky code with DeviceIOControl calls.

 

For file symlinks, I’ve used NTFS hardlinks, but when I create the link, I store some additional metadata with the file itself in an alternate stream (under the name “legacySymlinkInfo”) which allows tools to find out what the canonical filename really is, and also to find out what other files are linked to the same physical file.

Of course, non-CoApp software could still manipulate these files for little regard as to their status as ‘symlinks’, so the legacySymlinkInfo really can’t always be relied upon. But we can make a pretty good effort to ‘clean-as-we-go’ to keep that information up-to-date. If a file gets renamed or moved, the metadata may not be entirely accurate, so when we check the data, we make an attempt to validate that it is accurate, and update it with what we know. If it sounds like this is dodgy, I wouldn’t disagree, but in practice this seems to work really well.

 

Lastly, we need to be able to redirect a symlink, even when it is in use.  This seems trivial enough to do with junctions, but with files, we have to resort to moving the file out of the way, and creating a new linked file (which we can do, even if the file locked).

 

Admittedly, there are a few limitations that this design has:

Symlinks are only supported on the local system (no cross volume links, nor can they link to UNC paths)

Symlinks must be absolute references (not linking to relative paths)

The relationship between the linked file and the canonical file is potentially ephemeral, but simple examination of the files allows us to rebuild the information.

In the worst case scenario, if files get unlinked, they still work as regular files.

 

Luckily, CoApp’s requirements for what symbolic links need to do doesn’t have to be as flexible as full blown symlinks, and these limitations are not a hindrance at all.

I’ve just added this code to the CoApp toolkit (written in C#), and I’ll be playing with it quite a bit over the next few weeks. Once I’m sure that everything works as I like, if there is interest, I may implement a pure native version of the code so that other utilities and software can do the same thing and play nice too.


You can either comment here, come join mailing list (join the team at https://launchpad.net/~coapp-developers) and continue the conversation!

April 4th, 2011 By Garrett Categories: CoApp

(cross-posted from the mailing list)

When CoApp packages are installed, they install into a predetermined location based on the package metadata—this ensures that all packages play by the rules, and allow us to use Windows features to support things side-by-side installation of applications (ie, havin’ two versions of the same application installed concurrently), and also to ensure that we never, ever require a reboot to install software (even if a previous version of the software is currently running).

 

For most packages, this follows a format something like:

c:\apps\.installed\PUBLISHERNAME\PRODUCT-VERSION-PLATFORM\

which means that for a given package (say, CoApp’s command line toolkit) the files would get installed into:

c:\apps\.installed\OUTERCURVE FOUNDATION\coapp.toolkit-1.0.2.338-any
c:\apps\.installed\OUTERCURVE FOUNDATION\coapp.toolkit-1.0.2.338-any\coapp.exe

 

Of course, this predefined location is nether attractive, nor trivial for the end user (or other software) to use, so we like to ensure that packages can expose themselves in a consistent and familiar manner.  The process that CoApp has to accomplish this is called Package Composition.

In CoApp V1, Package Composition is fairly limited; it’s intended to create canonical directories, EXE links and shortcuts for packages so that they can be easily be referenced by users and developers without much trouble. In CoApp V2, Package Composition will provide a method for publishers to define how third-party plugins, libraries and extensions can be cleanly published without polluting the application’s install directory with files that really don’t belong there.

 

In CoApp V1, we see a package expose itself by creating a canonical location for the package and optionally publish EXEs in the PATH.  The canonical location for a given (application) package follows the following format:

c:\apps\PRODUCT\

Pretty easy, eh? … So that means we see CoApp’s command line toolkit visible here:

c:\apps\coapp.toolkit\
c:\apps\coapp.toolkit\coapp.exe

And the EXE should show itself in the PATH, by appearing in the common:

c:\Apps\bin

directory (which is placed in the PATH), like this:

c:\Apps\bin\coapp.exe

Of course, we really don’t want multiple *copies* of all these things littering the drive, so we’re using NTFS symbolic links (aka symlinks) to have the file system merely appear that way (somewhat like UNIX).

Symlinks On Windows

Starting with Windows Vista, NTFS supports symbolic links (often called soft-links), in a manner extremely similar to UNIX (and brethren). Yay! This is awesome; we can easily use symlinks to redirect the files and directories that we’re interested in back to the original installed location (c:\apps\.installed…) and not duplicate files.

This is also the secret mojo we’ve needed to allow applications to install side-by-side with other versions of itself.  It turns out that if you run an EXE via a symlink, it doesn’t lock the link, it locks the target. This means we can remove the linked version in c:\apps\bin and create a new link to the new version, and the currently running program doesn’t freak out.

Now, my pappy always used to say “this is all goin’ down too easy; I’m still waitin’ to find the mouse at the bottom of this can of beans!

The proverbial mouse in this case is that Windows XP and Windows Server 2003 don’t actually support symbolic links—but they do have a couple of thinks we can use to achieve nearly the same goal.

For directories, there are junctions. Junctions are really close to symbolic links (they have some limitations, like you can’t have a junction to a file across UNC, and they have to be absolute paths). Fortunately, this is minor potatoes—we are only interested in having directories locally, and we don’t need to use relative paths.

For files, it was lookin’ like we was plumb out of luck—except we do have something called hard links. Hard links actually allow more than one directory entry on the hard disk to physically point to the same file on disk.  While this gets around the trouble with actually duplicating the file on disk, if you run an application via the hard link (ie, run c:\apps\bin\coapp.exe ) and try to remove the link, it’ll fail while the application is running.

Luckily, there is something we can do to a file that is locked. We can move it—or rename it—even while it’s in use.  Using this, we can move the locked file out of the way, create a link to the new version, and we’re happy as horses.  If the file is locked, we’ll queue it up to actually get deleted on reboot (by using MoveFileEx which stuffs it into the registry at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\PendingFileRenameOperations ).

The only trouble that I can see is when you make a hard-link, there really isn’t an efficient way to find out what else is pointing to the same file. We’d need to be able to find linked copies when we want to remove a package, so that the ‘linked’ versions get cleaned up at package removal time too.

In order to address that, we use another NTFS feature—alternate streams.  This lets us store a little bit of information with the file itself—in this case, we’ll store the other known locations that are linked to the file—so that when we go to remove the ‘official’ version, we can clean up all the symlinked copies too.

 


You can either comment here, come join mailing list (join the team at https://launchpad.net/~coapp-developers) and continue the conversation!

Comments Off
March 23rd, 2011 By Garrett Categories: Development

The following is the instructions on how to build the current CoApp bits:

0. You need to have the following tools installed in order to build CoApp:

[I think that it’s only necessary to use register with LaunchPad + use SSH if you want to be able to push patches back; which makes Steps 1.0 – 1.2 optional]

1.0. Create/Register a SSH public key with Launchpad.

1.1. Run Pageant, load your private key.

1.2 Log into bzr/Launchpad

(make sure that pageant is running with your private key loaded)


 C:\coapp> bzr lp-login <username>

2. Download the following script:

http://coapp.org/coapp-src.cmd.zip

3. Unzip the script into your working directory (where you want to check out the source to)

4. From the command line:


 C:\coapp> coapp-src

   Usage:
   ------

   coapp-src.cmd [OPTION]

   where [OPTION] is one or more of:

       core        - just the core projects to build the package manager
       gui         - the prototype work on the GUI client
       tools       - the developer and publisher tools
       guts        - the guts of the bootstrap and installershim
       other       - garrett's other tools and oddities
       all         - all of the above

       update      - merges updates for any projects that are already checked out.

You can check out a portion of the code (or all, if you want) by using the script:

C:\coapp> coapp-src core

Will check out just the core (coapp-cli, coapp-toolkit and coapp-solution).

NOTE: the bzr checkouts are happening in the background in minimized windows. Wait until they are done (look at the taskbar, and you’ll see them)

4. Open the coapp-solution\coapp-tools.sln solution file in Visual Studio 2010

If you didn’t get all of the projects, you’ll see an error when visual studio tries to load projects that are not checked out; this is ok, you can ignore the error (or delete the projects that are missing from your solution file)

5. Build it. (ctrl-shift-b)

Make sure you build the debug version, you won’t be able to build the release (you’d need my private cert, and it does some funky stuff during the build process)

6. The output will be in [root]\output\any\debug\bin:

And you should be able to run the coapp.exe in that directory:

You’ll notice the packages that I installed from the http://coapp.org website are installed.

All coapp binaries (except for the bootstrapper itself) are built as ‘any’ (meaning they will run 64bit on x64 systems, and 32bit on x86  systems)

Comments Off
March 23rd, 2011 By Garrett Categories: Status

I’m moving to a new blog host right now… should have stuff up and running pretty quickly.

 

G

Comments Off
December 9th, 2010 By Garrett Categories: Uncategorized

The question occasionally comes up regarding how to create Zip files from C# without using GPL (or LGPL) code.   If license isn’t an issue, the simple answer is always is to use ZipLib (http://www.icsharpcode.net/OpenSource/SharpZipLib/) which is a wonderful, and well known library for manipulating zip files.

The other solution is to grab WiX  3.5 (http://wix.codeplex.com/) and use the assemblies out of the WiX SDK:

Microsoft.Deployment.Compression.dll
Microsoft.Deployment.Compression.Zip.dll

Which contain the classes you’d need to manipulate zip files (and really easy to boot!)

test.cs
using System;
using System.IO;
using System.Collections.Generic;
using Microsoft.Deployment.Compression.Zip;

namespace TestZip {
    public class Test {
        public static void Main(string[] args) {
            var zipFile = new ZipInfo("test.zip");
            var filesToArchive = new List<string>() { @"test.exe", @"test.cs" };
            zipFile.PackFiles(null, filesToArchive, null); //create the zip file
            zipFile.Unpack(@".\testfolder\"); //unpack the zip file into a folder
        }
    }
}
 

Pretty simple, eh?

July 29th, 2010 By Garrett Categories: CoApp

(cross-posted from the mailing list)

It’s been suggested a few times over the last several weeks that CoApp won’t provide any value for the .NET community. With the emergence of several alternative projects to provide a form of package management (NPack, OpenWrap, nu-net, and others), I figured I’d better make some time to explain how CoApp and .NET get along.

 

Q: Isn’t CoApp about packaging applications?

A: Well, oddly enough, the original vision for CoApp was less concerned about actually installing applications, but rather installing shared components.  The ability to install an application comes by virtue that applications tend to be collections of components (shared or private), and a decent package management system should cover these goals easily. 

 

Q: If CoApp is such a great idea for .NET why is all the work focused on native applications ?

A: The reasons that our initial work that we’re doing is focused around native shared libraries and applications, is that they require quite a bit more adaptation to correctly work using modern C compilers and Side-by-Side technology. Once we’ve got the tools to build properly behaving projects to produce binaries that are useful in this fashion, we’ll be in a good place to actually produce packages themselves.

.NET Assemblies, by their very nature are already beautifully designed to be adapted into CoApp packages.  Strong-named assemblies install into the GAC—which is really just the .NET implementation of the Windows Side-by-Side technology–by design. CoApp .NET packages simply install the assemblies into the GAC, where all applications can share them in the way that was intended.

 

Q: What if I don’t want stuff to be in the GAC?

A: I’d ask you to reconsider.  I can appreciate the desire to maintain control over every aspect of building and distributing your application.  On the other hand libraries that designed to be shared shouldn’t require the consumer of the library to do anything, except for consume them.  The publisher of the shared library shoulders the responsibility of maintaining the library and publishing security updates as well new versions.  As with native Side-by-Side assemblies, the publisher can indicate what version of a library should be used when a particular version is requested.

 

Q: What possible reason should I choose using CoApp over another .NET package management ?

A: Well, like I’ve said, a plethora of implementations is always a good thing. From what I can see, most of the other package management systems are focused on assisting developers in getting simplified access to shared .NET components.  CoApp takes a larger scope, hoping to serve developers, end-users, and IT administrators alike.  Our design includes the ability to update any components without the necessity of shutting down or rebooting processes (or even god-forbid, the system itself).  CoApp is designed to apply to software regardless of the language it’s written in, from native (C,C++,etc) to managed code (.NET; C#, VB.NET, etc) ,to dynamic languages (Perl, Python, etc) and web apps (ASP.NET, PHP, and more).

 

Q: Why have you chosen MSI files rather than something simpler and more convenient (like Zip+Manifest)?

A: Oh, trust me, I really wanted to.  I realize that Windows Installer is kindof a bloated beast that has a lot of downsides; we’ve has chosen MSI as the packaging format because it is handles so many other situations very well–one noteable point, is that on XP & Windows 2003, the only way to install a native Side-by-Side components is by using MSIs. We’ve taken steps to lessen the burden by deliberately limiting the scope of what we are using in MSI to not encumber packages in a painful mess.

As well, by using MSIs we gain the ability to leverage things like group policies, Windows Logo certification, transactional installations, and trivial adoption by other non-CoApp consumers—there is nothing that would stop someone from using CoApp packages for some of their dependencies, and without having do anything other than install the MSIs.

 



Hey, rather than commenting here, come join mailing list (join the team at https://launchpad.net/~coapp-developers) and continue the conversation!

Comments Off