April 13th, 2011 By Garrett Categories: CoApp, Development

During the development of CoApp, I’ve enthusiastically embraced the .NET 4.0 Task Parallel Library (aka, the TPL).  It’s a set of APIs that make developers more productive by significantly simplifying the process of adding parallelism and concurrency to applications. I got religion around this last fall when I saw the PDC presentation that Anders Hejlsberg gave on “The Future of C# and VB”.  In there Anders talks about the shift to a fully asynchronous programming methodology. While the technology that he spoke of isn’t yet ready for production use today, the Task Parallel Library in .NET 4.0 provides the underlying framework that we can take advantage of right now.

Before we get too deep…

I should note that asynchronous code in .NET 4 make rather extensive use of Lambda Expressions—If you’re not familiar with lambdas, you’d better go get familiar, because this stuff is gonna be a tad hard to understand without it. It’s ok, I’ll wait…

 

And, I’m not going to go into too much detail about the basics of using the TPL; I recommend that you watch Ander’s video (to get an idea of the motivation) and perhaps read thru Eric Lippert’s blog posts on Asynchrony (and really, start with his earlier posts on Continuation Passing Style)—if you are a .NET developer, this stuff will change your life.

The following is a simple example for how we start an asynchronous task using the TPL:

// *** Example Zero ***
// Starting a task using the TPL
var taskZero = Task.Factory.StartNew(() => {
    for (var q = 10; q > 0; q--) {
        // simulate a longer running operation
        Thread.Sleep(200);

        // Wish we could tell someone about
        // what's going on in this operation
        // and Writeline is hardly a good idea
        // when you are in another thread.
        Console.WriteLine("Progress Message from taskZero", q);
    }
});

Console.WriteLine("(Main Thread Completed)======>Press Enter to end.\r\n");
Console.ReadLine();

Which will produce output like:

task_0

You’ll notice that the main thread completes (and is sitting around waiting at the ReadLine() ) while the Task executes.

And getting to the point…

Ok, so now I can illustrate some code that I’ve written as part of CoApp that fills a void where I was creating complex asynchronous operations and wanted to be able to broadcast arbitrary events from that operation, without having to pass an object along to each step in the operation.

Let me put this another way…

I’d like to start an operation—say, “Install a package”—and I’d like the calling context to be able to subscribe to events for that particular operation (regardless of what events the operation directly (or indirectly) broadcasts.  Yes, I *could* create a class that has a bunch events, and pass that from function to function in the entire call tree for that operation, but that would be terribly unwieldy, and frankly, would be a real pain when I wanted to reuse other asynchronous tasks as child tasks, and they may not take the same class for events.

Additionally, that asynchronous operation may spawn off additional asynchronous operations of its own—perhaps to download some dependencies—and it’d be nice to be able to pick up download progress notifications so we could show some UI that keeps the user informed as to what’s actually going on.

 

Hrm… Howsabout I explain this with some code.  Let’s assume my little task has an event that it’d like to notify. First, we declare a class for the messages:

public class TestEvents : MessageHandlers<TestEvents> {
    public Action<string, int> Message1;
}

And then an example of the task firing the event:

// *** Example One ***
// Firing an event
var taskOne = CoTask.Factory.StartNew(() => {
    for (var q = 10; q > 0; q--) {
        // simulate a longer running operation
        Thread.Sleep(200); 

        // that occasionally notifies anyone who cares to listen.
        TestEvents.Invoke.Message1("Message From taskOne", q);
    }
});

Now,  you should notice two things (‘specially since I highlighted the lines…). First, I use my own CoTask Task factory instead of the TPL’s Task—this just lets me track tasks a bit better, and handles some background work for all this stuff. Functionally, it should be identical, and still returns the same Task objects. Second, I can now use the class I created before to fire off a message (in this case, Message1 takes a string and an int for parameters) using some mysterious static Invoke dohickey on my TestEvents class .

Of course, nothing is listening to the events yet, let’s wire that up:

// perhaps we'll slack off for a while
Thread.Sleep(400); 

// start listening to messages after the task has started.
// (we probably missed some!)
((TestEvents)taskOne).Message1 += (aString, anInt ) => {
    Console.WriteLine("The Task said: [{0}] and [{1}] ", aString, anInt);
};

Console.WriteLine("(Main Thread Completed)\r\n**Press Enter to end**")
Console.ReadLine();

When we run this, we’ll end up with some rather interesting output:

task_1

Shiny! You’ll notice in this example, that we’ve actually missed a couple of events (exacerbated by the Thread.Sleep() in line 2.). Regardless, the main thread completes and waits, and you can see my event handler in the lambda expression is getting called.
The beauty of this approach is that we can see how I can launch a longer running operation, and accept events from it, without having to explicitly tell the code in the operation about the event listener.

In the next part as I dig deeper, I’ll show how we can ensure we never miss our messages, and how we can listen for more than one at a time.

April 7th, 2011 By Garrett Categories: CoApp

(cross-posted from the mailing list)

As I mentioned in my last post, CoApp’s design relies heavily on the use of Symbolic Links (symlinks)—a feature which is implemented fairly well in Windows Vista and beyond, but is missing in earlier versions of Windows.

Cygwin tried to work around the limitation by using shortcuts (.LNK files), which are files used by the Windows Shell for all the shortcuts in the start menu.  Not a bad way to go, but unfortunately, not very compatible with a lot of things, since most programs don’t specially recognize .LNK files, although the POSIX emulation in cygwin did.

I was thinkin’ that any acceptable solution should work with as much software as possible, without requiring special understanding, nor should any additional software be required (ie, a filter driver of some sort).

Our options then, are somewhat limited. On Windows XP and Windows Server 2003 we *do* have features like junctions (like symlinks, but only for directories, and only on the same volume), hard links (where more than one directory entry points to the same physical file) and alternate streams (the ability to store extra data with a file).

Hmmm.

Junctions seem to work fairly well for directories, with the caveat that actually manipulating junctions requires some tricky code with DeviceIOControl calls.

 

For file symlinks, I’ve used NTFS hardlinks, but when I create the link, I store some additional metadata with the file itself in an alternate stream (under the name “legacySymlinkInfo”) which allows tools to find out what the canonical filename really is, and also to find out what other files are linked to the same physical file.

Of course, non-CoApp software could still manipulate these files for little regard as to their status as ‘symlinks’, so the legacySymlinkInfo really can’t always be relied upon. But we can make a pretty good effort to ‘clean-as-we-go’ to keep that information up-to-date. If a file gets renamed or moved, the metadata may not be entirely accurate, so when we check the data, we make an attempt to validate that it is accurate, and update it with what we know. If it sounds like this is dodgy, I wouldn’t disagree, but in practice this seems to work really well.

 

Lastly, we need to be able to redirect a symlink, even when it is in use.  This seems trivial enough to do with junctions, but with files, we have to resort to moving the file out of the way, and creating a new linked file (which we can do, even if the file locked).

 

Admittedly, there are a few limitations that this design has:

Symlinks are only supported on the local system (no cross volume links, nor can they link to UNC paths)

Symlinks must be absolute references (not linking to relative paths)

The relationship between the linked file and the canonical file is potentially ephemeral, but simple examination of the files allows us to rebuild the information.

In the worst case scenario, if files get unlinked, they still work as regular files.

 

Luckily, CoApp’s requirements for what symbolic links need to do doesn’t have to be as flexible as full blown symlinks, and these limitations are not a hindrance at all.

I’ve just added this code to the CoApp toolkit (written in C#), and I’ll be playing with it quite a bit over the next few weeks. Once I’m sure that everything works as I like, if there is interest, I may implement a pure native version of the code so that other utilities and software can do the same thing and play nice too.


You can either comment here, come join mailing list (join the team at https://launchpad.net/~coapp-developers) and continue the conversation!

April 4th, 2011 By Garrett Categories: CoApp

(cross-posted from the mailing list)

When CoApp packages are installed, they install into a predetermined location based on the package metadata—this ensures that all packages play by the rules, and allow us to use Windows features to support things side-by-side installation of applications (ie, havin’ two versions of the same application installed concurrently), and also to ensure that we never, ever require a reboot to install software (even if a previous version of the software is currently running).

 

For most packages, this follows a format something like:

c:\apps\.installed\PUBLISHERNAME\PRODUCT-VERSION-PLATFORM\

which means that for a given package (say, CoApp’s command line toolkit) the files would get installed into:

c:\apps\.installed\OUTERCURVE FOUNDATION\coapp.toolkit-1.0.2.338-any
c:\apps\.installed\OUTERCURVE FOUNDATION\coapp.toolkit-1.0.2.338-any\coapp.exe

 

Of course, this predefined location is nether attractive, nor trivial for the end user (or other software) to use, so we like to ensure that packages can expose themselves in a consistent and familiar manner.  The process that CoApp has to accomplish this is called Package Composition.

In CoApp V1, Package Composition is fairly limited; it’s intended to create canonical directories, EXE links and shortcuts for packages so that they can be easily be referenced by users and developers without much trouble. In CoApp V2, Package Composition will provide a method for publishers to define how third-party plugins, libraries and extensions can be cleanly published without polluting the application’s install directory with files that really don’t belong there.

 

In CoApp V1, we see a package expose itself by creating a canonical location for the package and optionally publish EXEs in the PATH.  The canonical location for a given (application) package follows the following format:

c:\apps\PRODUCT\

Pretty easy, eh? … So that means we see CoApp’s command line toolkit visible here:

c:\apps\coapp.toolkit\
c:\apps\coapp.toolkit\coapp.exe

And the EXE should show itself in the PATH, by appearing in the common:

c:\Apps\bin

directory (which is placed in the PATH), like this:

c:\Apps\bin\coapp.exe

Of course, we really don’t want multiple *copies* of all these things littering the drive, so we’re using NTFS symbolic links (aka symlinks) to have the file system merely appear that way (somewhat like UNIX).

Symlinks On Windows

Starting with Windows Vista, NTFS supports symbolic links (often called soft-links), in a manner extremely similar to UNIX (and brethren). Yay! This is awesome; we can easily use symlinks to redirect the files and directories that we’re interested in back to the original installed location (c:\apps\.installed…) and not duplicate files.

This is also the secret mojo we’ve needed to allow applications to install side-by-side with other versions of itself.  It turns out that if you run an EXE via a symlink, it doesn’t lock the link, it locks the target. This means we can remove the linked version in c:\apps\bin and create a new link to the new version, and the currently running program doesn’t freak out.

Now, my pappy always used to say “this is all goin’ down too easy; I’m still waitin’ to find the mouse at the bottom of this can of beans!

The proverbial mouse in this case is that Windows XP and Windows Server 2003 don’t actually support symbolic links—but they do have a couple of thinks we can use to achieve nearly the same goal.

For directories, there are junctions. Junctions are really close to symbolic links (they have some limitations, like you can’t have a junction to a file across UNC, and they have to be absolute paths). Fortunately, this is minor potatoes—we are only interested in having directories locally, and we don’t need to use relative paths.

For files, it was lookin’ like we was plumb out of luck—except we do have something called hard links. Hard links actually allow more than one directory entry on the hard disk to physically point to the same file on disk.  While this gets around the trouble with actually duplicating the file on disk, if you run an application via the hard link (ie, run c:\apps\bin\coapp.exe ) and try to remove the link, it’ll fail while the application is running.

Luckily, there is something we can do to a file that is locked. We can move it—or rename it—even while it’s in use.  Using this, we can move the locked file out of the way, create a link to the new version, and we’re happy as horses.  If the file is locked, we’ll queue it up to actually get deleted on reboot (by using MoveFileEx which stuffs it into the registry at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\PendingFileRenameOperations ).

The only trouble that I can see is when you make a hard-link, there really isn’t an efficient way to find out what else is pointing to the same file. We’d need to be able to find linked copies when we want to remove a package, so that the ‘linked’ versions get cleaned up at package removal time too.

In order to address that, we use another NTFS feature—alternate streams.  This lets us store a little bit of information with the file itself—in this case, we’ll store the other known locations that are linked to the file—so that when we go to remove the ‘official’ version, we can clean up all the symlinked copies too.

 


You can either comment here, come join mailing list (join the team at https://launchpad.net/~coapp-developers) and continue the conversation!

Comments Off
March 23rd, 2011 By Garrett Categories: Development

The following is the instructions on how to build the current CoApp bits:

0. You need to have the following tools installed in order to build CoApp:

[I think that it’s only necessary to use register with LaunchPad + use SSH if you want to be able to push patches back; which makes Steps 1.0 – 1.2 optional]

1.0. Create/Register a SSH public key with Launchpad.

1.1. Run Pageant, load your private key.

1.2 Log into bzr/Launchpad

(make sure that pageant is running with your private key loaded)


 C:\coapp> bzr lp-login <username>

2. Download the following script:

http://coapp.org/coapp-src.cmd.zip

3. Unzip the script into your working directory (where you want to check out the source to)

4. From the command line:


 C:\coapp> coapp-src

   Usage:
   ------

   coapp-src.cmd [OPTION]

   where [OPTION] is one or more of:

       core        - just the core projects to build the package manager
       gui         - the prototype work on the GUI client
       tools       - the developer and publisher tools
       guts        - the guts of the bootstrap and installershim
       other       - garrett's other tools and oddities
       all         - all of the above

       update      - merges updates for any projects that are already checked out.

You can check out a portion of the code (or all, if you want) by using the script:

C:\coapp> coapp-src core

Will check out just the core (coapp-cli, coapp-toolkit and coapp-solution).

NOTE: the bzr checkouts are happening in the background in minimized windows. Wait until they are done (look at the taskbar, and you’ll see them)

4. Open the coapp-solution\coapp-tools.sln solution file in Visual Studio 2010

If you didn’t get all of the projects, you’ll see an error when visual studio tries to load projects that are not checked out; this is ok, you can ignore the error (or delete the projects that are missing from your solution file)

5. Build it. (ctrl-shift-b)

Make sure you build the debug version, you won’t be able to build the release (you’d need my private cert, and it does some funky stuff during the build process)

6. The output will be in [root]\output\any\debug\bin:

And you should be able to run the coapp.exe in that directory:

You’ll notice the packages that I installed from the http://coapp.org website are installed.

All coapp binaries (except for the bootstrapper itself) are built as ‘any’ (meaning they will run 64bit on x64 systems, and 32bit on x86  systems)

Comments Off
March 23rd, 2011 By Garrett Categories: Status

I’m moving to a new blog host right now… should have stuff up and running pretty quickly.

 

G

Comments Off
December 9th, 2010 By Garrett Categories: Uncategorized

The question occasionally comes up regarding how to create Zip files from C# without using GPL (or LGPL) code.   If license isn’t an issue, the simple answer is always is to use ZipLib (http://www.icsharpcode.net/OpenSource/SharpZipLib/) which is a wonderful, and well known library for manipulating zip files.

The other solution is to grab WiX  3.5 (http://wix.codeplex.com/) and use the assemblies out of the WiX SDK:

Microsoft.Deployment.Compression.dll
Microsoft.Deployment.Compression.Zip.dll

Which contain the classes you’d need to manipulate zip files (and really easy to boot!)

test.cs
using System;
using System.IO;
using System.Collections.Generic;
using Microsoft.Deployment.Compression.Zip;

namespace TestZip {
    public class Test {
        public static void Main(string[] args) {
            var zipFile = new ZipInfo("test.zip");
            var filesToArchive = new List<string>() { @"test.exe", @"test.cs" };
            zipFile.PackFiles(null, filesToArchive, null); //create the zip file
            zipFile.Unpack(@".\testfolder\"); //unpack the zip file into a folder
        }
    }
}
 

Pretty simple, eh?

July 29th, 2010 By Garrett Categories: CoApp

(cross-posted from the mailing list)

It’s been suggested a few times over the last several weeks that CoApp won’t provide any value for the .NET community. With the emergence of several alternative projects to provide a form of package management (NPack, OpenWrap, nu-net, and others), I figured I’d better make some time to explain how CoApp and .NET get along.

 

Q: Isn’t CoApp about packaging applications?

A: Well, oddly enough, the original vision for CoApp was less concerned about actually installing applications, but rather installing shared components.  The ability to install an application comes by virtue that applications tend to be collections of components (shared or private), and a decent package management system should cover these goals easily. 

 

Q: If CoApp is such a great idea for .NET why is all the work focused on native applications ?

A: The reasons that our initial work that we’re doing is focused around native shared libraries and applications, is that they require quite a bit more adaptation to correctly work using modern C compilers and Side-by-Side technology. Once we’ve got the tools to build properly behaving projects to produce binaries that are useful in this fashion, we’ll be in a good place to actually produce packages themselves.

.NET Assemblies, by their very nature are already beautifully designed to be adapted into CoApp packages.  Strong-named assemblies install into the GAC—which is really just the .NET implementation of the Windows Side-by-Side technology–by design. CoApp .NET packages simply install the assemblies into the GAC, where all applications can share them in the way that was intended.

 

Q: What if I don’t want stuff to be in the GAC?

A: I’d ask you to reconsider.  I can appreciate the desire to maintain control over every aspect of building and distributing your application.  On the other hand libraries that designed to be shared shouldn’t require the consumer of the library to do anything, except for consume them.  The publisher of the shared library shoulders the responsibility of maintaining the library and publishing security updates as well new versions.  As with native Side-by-Side assemblies, the publisher can indicate what version of a library should be used when a particular version is requested.

 

Q: What possible reason should I choose using CoApp over another .NET package management ?

A: Well, like I’ve said, a plethora of implementations is always a good thing. From what I can see, most of the other package management systems are focused on assisting developers in getting simplified access to shared .NET components.  CoApp takes a larger scope, hoping to serve developers, end-users, and IT administrators alike.  Our design includes the ability to update any components without the necessity of shutting down or rebooting processes (or even god-forbid, the system itself).  CoApp is designed to apply to software regardless of the language it’s written in, from native (C,C++,etc) to managed code (.NET; C#, VB.NET, etc) ,to dynamic languages (Perl, Python, etc) and web apps (ASP.NET, PHP, and more).

 

Q: Why have you chosen MSI files rather than something simpler and more convenient (like Zip+Manifest)?

A: Oh, trust me, I really wanted to.  I realize that Windows Installer is kindof a bloated beast that has a lot of downsides; we’ve has chosen MSI as the packaging format because it is handles so many other situations very well–one noteable point, is that on XP & Windows 2003, the only way to install a native Side-by-Side components is by using MSIs. We’ve taken steps to lessen the burden by deliberately limiting the scope of what we are using in MSI to not encumber packages in a painful mess.

As well, by using MSIs we gain the ability to leverage things like group policies, Windows Logo certification, transactional installations, and trivial adoption by other non-CoApp consumers—there is nothing that would stop someone from using CoApp packages for some of their dependencies, and without having do anything other than install the MSIs.

 



Hey, rather than commenting here, come join mailing list (join the team at https://launchpad.net/~coapp-developers) and continue the conversation!

Comments Off
July 17th, 2010 By Garrett Categories: Uncategorized

I wrote this a couple years ago, and I ended up in a #cls10 session related to it, so I thought I’d post it unedited. Feel free to destroy it.

 

 

Idea:
    A self-regulating community meritocracy model for collaboration and cooperation.
   
    This can be applied to any type of collaborative organization where individuals efforts
    are rewarded when the organization benefits. Examples:
        Content Wikis : Wikipedia, etc
        Projects : Software development, etc.
        Organizations that work together for common benefit: ISO, ECMA, etc.
   
    This model is designed to significantly reduce the ability for a Pariticipant to game the
    system for self-benefit. (Examples of which: Attempts at ISO Vote stacking by MS)
   
Participants:
    Individuals – People who particpate in the community
    Organizations – Collections of associated individuals who are both responsible to the
                    organization and responsible for the organization’s behavior.
                   
                    Organizations have the authority to grant some or all of their Achievement
                    benefits to a member Participant. (completey subject to the Organization’s
                    philosophy)
   
Concepts:
    Action – An activity that a Participant is involved with:
        Contribution – a type of action that is adding to the value of the community,
        [Recognition|Transgression] – the recognition of an activity a pariticipant, filed by someone in the community
       
        Bill – a request for a policy change or ammendment to the fabric of the community
             –these are never negative, but the introduction of bills as a nusance can be considered anti-social
   
    Merit – a quantifiable contribution to the community
    Karma – a instantaneous measurement of a Participant’s [value?] in the community
        – [short-term] karma – the value that represents a participant’s [value] within the [time window]
        – [long-term] karma – the sum/eternal acheivement karma that is prior to the [time window]
    [Potential] – given the evaluation of past performance, the expected future value.
            – value is weighted in [time window] increments:
                current [time window] – 40%
                current -1 [time window] – 30%
                current -2 [time window] – 20%
                remaining [time window] – 10%

    [Time Window] – the length of time that is considered to be “right now” {~90 days?}
   
    [Achievement Level] – the state that a Participant holds inside the community.
                          there are four levels, each has a weight associated with it [name:weight]
        – [Fellow:10]   the highest level of achievement
                        considered fully trusted and can act on behalf of the community
                        unrestricted Contributions access.
                        only Individuals can be Fellows
                        By virtue of creating the community, founders tend to be fellows

        – [Trusted:6]   Considered extremely trustworthy
                        Contributions are assumed to be accepted, but is still placed in the [review queue]

        – [Associate:2] Proven record of positive actions to the community.
                        Contributions are publicly viewable, are required to be reviewed prior to acceptance
        – [Volenteer:1] Any member of the community without a proven track record.
   
        – [Owner:100]   NOT PART OF PUBLIC COMMUNITIES
            – Some Organizations may still require individuals that are granted ‘super’ status (ie, benevolent dictator)
            – Public communities should never have Owners.
           
    [Review Queue]  – Contributions that should be given a [Rating]
   
    [Rating] – Quantifiable examination of an Action
        – Intent:
            Positive – The community is not harmed by the contribution {hmm}
            Negative – The community is harmed by the contribution {considered anti-social}
        – Acceptance
            Excellent – This is a valuable addition to the community,
                        of a quality that requires no revision before acceptance
            Good      – This is a valuable addition to the community,
                        with feedback the contributor should evaluate prior to acceptance
                        but can be accepted without additional change
            [Almost]  – This is on the right track to being accepted,
                        with feedback to the contributor that must be rectified before acceptance is possible
                            (this may be a conversation, or a modification to the contribution)
            [Insufficient] – this contribution does not meet community standards for acceptance
                            The contribution rejected, and must be resubmitted.
        – Significance – the amount of Merit that the contribution has.
            [Extreme:10] – this is a crowning acheivement that has the highest level of benefit to the community
            [High:6] – this is very valuable to the community
            [Fair:2] – this is valuable to the community
            [Mild:1] – the value is low, but not zero
            [NoEffect:0] – the contribution does not actively add value to the community
        – Comments – constructive feedback on the contribution (no ‘A+++ crap,’ no ‘stupid xxx’)
       
    [Sponsor] – a Participant will have multiple Participants who are have acted to approve an acheivement promotion
                Sponsors are then required to act in the event of Negative Actions. Actions can be:
                    – Censure – a revocation of sponsorship
                    – Penalty/Counselling – Application of penalty to the participant’s karma, and creating a
                                            conversation to correct behavior.
                    – Forgiveness – the sponsors can forgive the transgression, usually accompanied by
                                    a positive action on part of the participant.
                    – Defense – Sponsors can come to the defence of a Participant, which can potentially reverse
                                the ‘negative’ intent of the action; (may require approval by quorum)
               
                Because a participant has several sponsors; 
               
              – if a Sponsor fails to act, the lack of action can become a ‘negative action’ of their own

Comments Off