(cross-posted from the mailing list)

It’s been suggested a few times over the last several weeks that CoApp won’t provide any value for the .NET community. With the emergence of several alternative projects to provide a form of package management (NPack, OpenWrap, nu-net, and others), I figured I’d better make some time to explain how CoApp and .NET get along.

Q: Isn’t CoApp about packaging applications?

A: Well, oddly enough, the original vision for CoApp was less concerned about actually installing applications, but rather installing shared components.  The ability to install an application comes by virtue that applications tend to be collections of components (shared or private), and a decent package management system should cover these goals easily. 

Q: If CoApp is such a great idea for .NET why is all the work focused on native applications ?

A: The reasons that our initial work that we’re doing is focused around native shared libraries and applications, is that they require quite a bit more adaptation to correctly work using modern C compilers and Side-by-Side technology. Once we’ve got the tools to build properly behaving projects to produce binaries that are useful in this fashion, we’ll be in a good place to actually produce packages themselves.

.NET Assemblies, by their very nature are already beautifully designed to be adapted into CoApp packages.  Strong-named assemblies install into the GAC—which is really just the .NET implementation of the Windows Side-by-Side technology--by design. CoApp .NET packages simply install the assemblies into the GAC, where all applications can share them in the way that was intended.

Q: What if I don’t want stuff to be in the GAC?

A: I’d ask you to reconsider.  I can appreciate the desire to maintain control over every aspect of building and distributing your application.  On the other hand libraries that designed to be shared shouldn’t require the consumer of the library to do anything, except for consume them.  The publisher of the shared library shoulders the responsibility of maintaining the library and publishing security updates as well new versions.  As with native Side-by-Side assemblies, the publisher can indicate what version of a library should be used when a particular version is requested.

Q: What possible reason should I choose using CoApp over another .NET package management ?

A: Well, like I’ve said, a plethora of implementations is always a good thing. From what I can see, most of the other package management systems are focused on assisting developers in getting simplified access to shared .NET components.  CoApp takes a larger scope, hoping to serve developers, end-users, and IT administrators alike.  Our design includes the ability to update any components without the necessity of shutting down or rebooting processes (or even god-forbid, the system itself).  CoApp is designed to apply to software regardless of the language it’s written in, from native (C,C++,etc) to managed code (.NET; C#, VB.NET, etc) ,to dynamic languages (Perl, Python, etc) and web apps (ASP.NET, PHP, and more).

Q: Why have you chosen MSI files rather than something simpler and more convenient (like Zip+Manifest)?

A: Oh, trust me, I really wanted to.  I realize that Windows Installer is kindof a bloated beast that has a lot of downsides; we’ve has chosen MSI as the packaging format because it is handles so many other situations very well--one noteable point, is that on XP & Windows 2003, the only way to install a native Side-by-Side components is by using MSIs. We’ve taken steps to lessen the burden by deliberately limiting the scope of what we are using in MSI to not encumber packages in a painful mess.

As well, by using MSIs we gain the ability to leverage things like group policies, Windows Logo certification, transactional installations, and trivial adoption by other non-CoApp consumers—there is nothing that would stop someone from using CoApp packages for some of their dependencies, and without having do anything other than install the MSIs.

Hey, rather than commenting here, come join mailing list (join the team at https://launchpad.net/~coapp-developers) and continue the conversation!

I wrote this a couple years ago, and I ended up in a #cls10 session related to it, so I thought I'd post it unedited. Feel free to destroy it.

Idea:
    A self-regulating community meritocracy model for collaboration and cooperation.
   
    This can be applied to any type of collaborative organization where individuals efforts
    are rewarded when the organization benefits. Examples:
        Content Wikis : Wikipedia, etc
        Projects : Software development, etc.
        Organizations that work together for common benefit: ISO, ECMA, etc.
   
    This model is designed to significantly reduce the ability for a Pariticipant to game the
    system for self-benefit. (Examples of which: Attempts at ISO Vote stacking by MS)
   
Participants:
    Individuals - People who particpate in the community
    Organizations - Collections of associated individuals who are both responsible to the
                    organization and responsible for the organization's behavior.
                   
                    Organizations have the authority to grant some or all of their Achievement
                    benefits to a member Participant. (completey subject to the Organization's
                    philosophy)
   
Concepts:
    Action - An activity that a Participant is involved with:
        Contribution - a type of action that is adding to the value of the community,
        [Recognition|Transgression] - the recognition of an activity a pariticipant, filed by someone in the community
       
        Bill - a request for a policy change or ammendment to the fabric of the community
             --these are never negative, but the introduction of bills as a nusance can be considered anti-social
   
    Merit - a quantifiable contribution to the community
    Karma - a instantaneous measurement of a Participant's [value?] in the community
        - [short-term] karma - the value that represents a participant's [value] within the [time window]
        - [long-term] karma - the sum/eternal acheivement karma that is prior to the [time window]
    [Potential] - given the evaluation of past performance, the expected future value.
            - value is weighted in [time window] increments:
                current [time window] - 40%
                current -1 [time window] - 30%
                current -2 [time window] - 20%
                remaining [time window] - 10%

    [Time Window] - the length of time that is considered to be "right now" {~90 days?}
   
    [Achievement Level] - the state that a Participant holds inside the community.
                          there are four levels, each has a weight associated with it [name:weight]
        - [Fellow:10]   the highest level of achievement
                        considered fully trusted and can act on behalf of the community
                        unrestricted Contributions access.
                        only Individuals can be Fellows
                        By virtue of creating the community, founders tend to be fellows

        - [Trusted:6]   Considered extremely trustworthy
                        Contributions are assumed to be accepted, but is still placed in the [review queue]

        - [Associate:2] Proven record of positive actions to the community.
                        Contributions are publicly viewable, are required to be reviewed prior to acceptance
        - [Volenteer:1] Any member of the community without a proven track record.
   
        - [Owner:100]   NOT PART OF PUBLIC COMMUNITIES
            - Some Organizations may still require individuals that are granted 'super' status (ie, benevolent dictator)
            - Public communities should never have Owners.
           
    [Review Queue]  - Contributions that should be given a [Rating]
   
    [Rating] - Quantifiable examination of an Action
        - Intent:
            Positive - The community is not harmed by the contribution {hmm}
            Negative - The community is harmed by the contribution {considered anti-social}
        - Acceptance
            Excellent - This is a valuable addition to the community,
                        of a quality that requires no revision before acceptance
            Good      - This is a valuable addition to the community,
                        with feedback the contributor should evaluate prior to acceptance
                        but can be accepted without additional change
            [Almost]  - This is on the right track to being accepted,
                        with feedback to the contributor that must be rectified before acceptance is possible
                            (this may be a conversation, or a modification to the contribution)
            [Insufficient] - this contribution does not meet community standards for acceptance
                            The contribution rejected, and must be resubmitted.
        - Significance - the amount of Merit that the contribution has.
            [Extreme:10] - this is a crowning acheivement that has the highest level of benefit to the community
            [High:6] - this is very valuable to the community
            [Fair:2] - this is valuable to the community
            [Mild:1] - the value is low, but not zero
            [NoEffect:0] - the contribution does not actively add value to the community
        - Comments - constructive feedback on the contribution (no 'A+++ crap,' no 'stupid xxx')
       
    [Sponsor] - a Participant will have multiple Participants who are have acted to approve an acheivement promotion
                Sponsors are then required to act in the event of Negative Actions. Actions can be:
                    - Censure - a revocation of sponsorship
                    - Penalty/Counselling - Application of penalty to the participant's karma, and creating a
                                            conversation to correct behavior.
                    - Forgiveness - the sponsors can forgive the transgression, usually accompanied by
                                    a positive action on part of the participant.
                    - Defense - Sponsors can come to the defence of a Participant, which can potentially reverse
                                the 'negative' intent of the action; (may require approval by quorum)
               
                Because a participant has several sponsors; 
               
              - if a Sponsor fails to act, the lack of action can become a 'negative action' of their own

(cross-posted from the mailing list)

It has been nearly three months since launching the CoApp project, and in that time I’ve been absolutely amazed at the response that we’ve gotten, and the community folks that have jumped on board. As a matter of fact, it’s been far busier than I had hoped, and was prepared for.

Initially, I thought that we’d see a few people interested, and maybe two or three volunteer to become committers, and we’d start slowly and build up to actually producing something. Well, within a week or two of the kickoff of the project we saw an explosion of news articles and blog posting that appeared was very cool. 130 people joined the developer mailing list, and we’ve got 13 committers.

Unexpected rapid growth causes its own problems, first being the lack of something for 13 people to actually start working on, and a definite gap between the vision that I had started with and what everyone understood to be the ‘plan’.  On top of that, the sheer number of people both internally and externally to Microsoft who wanted to know more and how they could start using it made it a challenge to answer their questions, make the appearances to talk about CoApp, put enough design in place to unblock committers, and oh, wait, actually produce code!

Early on, one of the problems I had is that folks didn’t quite know exactly what we would be producing, so I wrote out seven points that describe the scope that the CoApp project is focused on:

1. Provide a specification for package types and how applications are packaged, so that they can behave themselves in a common ecosystem.
2. Provide tools to make packaging easier
3. Shallow-fork a lot of OSS projects to provide clean, well-maintained Windows binaries
4. Provide tools to make shallow-forking & maintaining packages easier
5. A user experience and a service that will allow end-users to manage their installed packages trivially (including updating...etc)
6. A specification for metadata so that any publisher who conforms to the package specification can build and distribute packages using via the client tools (5)
7. Binary and source packages of applications and libraries forked in (3)

With that in mind, we could move forward.

In mid-May Microsoft hosted the CoApp Design and Development summit, where 13 folks from around the world came to Redmond where we spent two days going over my initial vision, adding in everyone’s ideas, and finally fleshing out some of the design and the plan forward. As is typical with this type of project, folks are more interested in coding than documenting, but we’re slowly filling in the blanks.(Our rough notes on Day 1 and Day 2. Oooh-And some pictures!)

To match Point #1 on our list of what CoApp is, we have to crystallize what exactly a CoApp-style package looks like. Initially, I had thought that there would be several package types (apps, shared libraries, source code, static libraries, drivers, etc…) but during the summit we came up with a change to that.  A package can contain multiple roles; each role installs a particular type of thing.  So, there are several role types (App, shared library, source code, developer library, and drivers) This subtle yet important difference allows collections of things to be versioned as a group and cuts down on the number of packages for a particular purpose.  We also studied in depth how complicated things like Perl and Python will need to be assembled in order to provide the flexibility and power they require.

Some of the details (notably device drivers, the concepts of feature advertisement and product composition necessary for things like Python) we’re putting off until later to have a bit of experience and understanding how well our design works for the less complex packages.  So, other than that I’m fairly sure that we’ve got a pretty good idea of how those packages are going to look.  Certainly it’s enough information for me to hand-roll some MSIs that we can use for testing.

I’ve also spent a bunch of time setting up project infrastructure and the skeleton code and build system for the sub-projects on Launchpad.  It’s now possible to actually build CoApp, even though nothing actually does anything.  This allows some unblocking of the developers who wanted to get cracking.

Currently, we’re working on Point #2, which is split into a couple of separate tracks. The CoApp client engine (which is the client component that handles all the magic at install time) is being led by Elizabeth Smith. Elizabeth is a well-known open source (PHP, gtk and others) developer who has wide expertise in C and Windows.  The other track is the developer tools where I’m working with a handful of developers to begin fleshing out their design and start cranking out code over the summer.

I would also like to thank all of the committers who came out to the Design & Development Summit in May: Elizabeth Smith, Rafael Rivera, Adam Kennedy, Trent Nelson, Philip Allison, Adam Baxter, Jonathan Ben-Joseph, Ted Bullock, Nasser Dassi, Trevor Dennis, Olaf Vanderspek, Kevin Moore, Mark Stone and Rob Mensching.  Your efforts are greatly appreciated, and I’m looking forward to continuing to work together.

Hi, my name is Garrett, and I have a problem.

I love scripting stuff. And, I by stuff, I mean everything. Really, I often use it as an exercise to understanding whatever it is that I’m trying to learn—if I can automate it, I’ve got a strong feeling that I understand it.

As you may or may not know, my absolute favorite language is JavaScript (and its many incarnations, like JScript). My love for JavaScript even predates the existence of JavaScript—I heavily used a scripting language called C-- (C minus minus aka ‘Cmm’, later renamed to ScriptEase). A somewhat forgotten company called Nombas had written it. A clip from the wayback machine (http://bit.ly/dCjRd9):

Cmm and JavaScript were so similar anyway, it was inevitable that one of them would perform the necessary course corrections and become like the other.  Regardless of the specific flavor, for me, the notion of a dynamic c-style scripting language has always had extremely strong appeal.

Over the last 15 or so years, I continued to use JavaScript (or JScript) whenever I needed a script, and I’ve been very happy.  As well, I continued to refine some of my scripting techniques and create some rather clever hacks to do a lot of the things I needed to do.

The real problem began when I needed to do some batch scripting—that is some useful scripting of command lines and whatnot. On Windows (out of the box), there are only a couple of options: Batch scripting with CMD, and in later years, PowerShell. For a long time, I used CMD Batch Scripting for lots of this, but it’s so … so … nasty. You really don’t want to do serious work in there, it’s just not up to the task.

PowerShell should have made me ecstatic—a .NET language, with access to pretty much everything I ever wanted in a scripting language, tons of support, and clearly the answer to my problems.  Unfortunately, that syntax breaks my brain. I just can’t get natural with it, no matter how much I’ve tried.

So, a couple of years ago, I started writing a nifty library that allowed me to make complicated batch scripts using JScript; and it’s been working out nicely. I get access to everything in the OS that I need, the language is installed on every version of Windows that I have used, and it doesn’t require any magic to make work.  Over time, I’ve added in code to do some ‘theoretically impossible’ tasks in JScript, including:

• full support for batch scripting (capturing and using the results of other command line apps)
• full use of environment variables in scripting
• formatted string handling
• Binary file read and write (and they said it couldn’t be done)
• some basic file assertions
• http downloading
• logging
• PE executable identification
• process management
• an extensible library interface and libraries for Restore Points, Speech (SAPI), JSON, MD5, Hyper-V VHD handling, and a twisted on-the-fly C# compiler.

And probably some other things I can’t remember.

Here’s where it starts to get weird.

Then, last night I was doing some more batch scripting at home, and I had my insane idea. What if I wrote a script that transformed a batch-like script language into JScript, and executed it. And, just because I’m that crazy, I’ll write it in JScript too.

And now, a scan hundred or so lines later, my latest script language is born. I call it ‘gs’ – short for gScript… a cousin to my other (unreleased) scripting language g#.

A quick little example script:

// comments are still slashies

// you can execute command lines just like regular batch languages:
// (this just prints the robocopy usage)
robocopy 

// Or you can capture the results of a command:
// (this uses the command processor to get a list of files (as text))
var $RESULT = cmd /c dir /b c:\*.zip

// there are built-in commands:
// so far cd, md, rd, erase, echo, dir
echo {$RESULT}


// and when you want to do some jscript, start a jscript block:
js {
    // this is a jscript code block
    //any legal jscript is ok here!

    // of course, you have access to the js.js function library:
    print("hello World");

    // and functions work just fine:
    function foo() {
    
        // you can even run batch commands from where with hash-bang:
        $WGETHELP = #! wget --help

        #! cmd.exe /c ver

        // output from the last command is always captured as an array in
        // $StdOut and as a string in $StdOutString
        print($StdOutString);

        return $WGETHELP;
    }
   
}

// oh, and for loops work outside the js blocks in the batch-world too:
// prints out text:
for(var i=0;i<10;i++)
    echo {$RESULT} {i}

using the gs script runner to run this:

C:\>gs test.gs

( or, to dump the processed script out )

C:\>gs /debug test.gs

// comments are still slashies

// you can execute command lines just like regular batch languages:
// (this just prints the robocopy usage)
$$("robocopy");

// Or you can capture the results of a command:
// (this uses the command processor to get a list of files (as text))
var $RESULT = $$.RunQuiet("cmd /c dir /b c:\\*.zip");

// there are built-in commands:
// so far cd, md, rd, erase, echo, dir
print('{$RESULT}');


// and when you want to do some jscript, start a jscript block:

    // this is a jscript code block
    //any legal jscript is ok here!

    // of course, you have access to the js.js function library:
    print("hello World");

    // and functions work just fine:
    function foo() {

        // you can even run batch commands from where with hash-bang:
        $WGETHELP = $$.RunQuiet("wget --help");

        $$("cmd.exe /c ver");

        // output from the last command is always captured as an array in
        // $StdOut and as a string in $StdOutString
        print($StdOutString);

        return $WGETHELP;
    }

// oh, and for loops work outside the js blocks in the batch-world too:
// prints out text:
for(var i=0;i<10;i++)
print('{$RESULT} {i}');

I’ve skimmed over so much of how this all works, but feel free to play with the code.  Some of this I published earlier in my gsToolkit project on Codeplex.  I’m using all of this in the CoApp project for a variety of tasks, so I’m publishing the whole set in a CoApp project on Launchpad

(cross-posted from the mailing list)

I’m doing some initial work for long term planning today—really this will help drive my focus over the next year or two.

Assuming that CoApp produces a beautifully functioning package manager and ecosystem, what open source packages would you like to see made available for Windows via CoApp?

I’ve stated before that my personal focus is around PHP, Apache and Python (courtesy of the folks kind enough to pay for my time on the project).

What would you like? What software or libraries would you find compelling? What is #1 in your list? On the server? On the client? In the cloud?

What is the priority of that (say on a scale of one to five stars)?

Feel free to tweet your answers tagged with #CoApp #PackageIdea

Hey, rather than commenting here, come join mailing list (join the team at https://launchpad.net/~coapp-developers) and continue the conversation!

In a couple of weeks the CoApp Design & Development Summit that will take place here at Microsoft, I’ll have 15 or so people from around the world to thrash thru my some of my crazy ideas regarding package management on Windows.

Scheduling this even has slowed down discussions on the CoApp mailing list—which is ok, rather than trying to settle much before hand, this gives me the opportunity to actually think about a lot of the other scenarios that will need to be handled, and to brainstorm how I think I’d deal with them. Which isn’t to say that’s how they will get dealt with… the difference between my imagination and reality can be pretty wide.

That being said, some of these ideas I have actually started a bit more than 12 or 13 years ago—sometime just before Microsoft started publishing their first information about Microsoft Installer Technology. I was working a contract for SHL, and we were building a system for a client that had 22,000 desktops where they wanted people to be able to run any of the several hundred applications that were available, but without needing to explicitly install them.  My team and I built a system we called “Zaal"—a Zero Administration Application Launcher.

Essentially, we made a system for building packages somewhat similar to support what kids today call ‘portable applications’, but really designed around getting apps from the network …

The way it worked, the network login script would quickly populate the start menu (this was on Win95 and WinNT 4.0) with icons for all the applications that were available to the entire company (yeah, that was a lot… but this is what the client wanted), but rather than the shortcut pointing to the actual application, we pointed them all to a launcher, along with some parameters that said what they wanted to run. The launcher was a tiny, tight little piece of C code that looked to see if the app was installed, if not it would grab it off the network, and silently install it, and finally it would run it.

Now—ya gotta remember, this was all before this funky-groovy MSI technology. We built a bunch of tools that took a snapshot of the entire ‘corporate standard’ PC (files & registry), we’d install the application, diff the results, package up what got changed—sometimes with a fair amount of hand tweaking for some apps (not mentioning any names … LOTUS NOTES [...uh...sorry Ray!]). So, in the end we’d have these packages which could be just-in-time installed with zero UI, and the customer was quite happy.

At the time, I tried to convince the company to turn it into an actual product—but alas, they had little foresight.

But still, looking back, it feels to me like not much has changed. Those packages we created NEVER asked anything at install time. They installed to a predetermined place. They installed with the most logical settings. We even had a secure service which took care of writing registry keys that were normally administrator only, so the app launcher never had to run any higher than “standard user” privileges.

I learned a hell of a lot from the couple years I spent at SHL.  Most of which is, this stuff is a lot simpler than people like to think it is.

Over the last decade or so, application installation and deployment has gotten more and more complex—but not because installing applications has become more complex—I think that everyone just got carried away with a lot of stuff that wasn’t necessary, or didn’t really understand the best way to have done something. And, as the tools became increasingly flexible, the demand for unnecessary features began to creep in, and the cycle begins anew.

I guess that’s runaway feature creep.

Don’t get me wrong, there are a lot of really cool things that have gone into Windows’ Installer Technology over the last decade, and I’m damn thankful for those.  I think that we can leverage what’s important while still driving back to simplicity and streamlining everything.

I figured I’d take a quick break, and review the fine wisdom my old pappy has imparted to my blog:

Like my pappy always used to tell me “It may not be the easy way, but it’s the cowboy way.” … Well, WinSxS may not be the easy way (yet!), but it will most assuredly be the CoApp way. (http://fearthecowboy.com/post/CoApp-FAQ-Can-you-explain-how-Side-by-side-%28WinSxS%29-works.aspx)

My pappy always used to tell me, “Never squat with your spurs on”. Good advice, and here’s a bit more, so pay attention. (http://fearthecowboy.com/post/CoApp-FAQ-Can-you-explain-how-Side-by-side-%28WinSxS%29-works.aspx)

It reminds me of somethin' my pappy told me... "Always take a good look at what you're about to eat. It's not so important to know what it is, but it sure is crucial to know what it was." That kind of advice works really well for security, just as it did for those odd-tastin'' sausages we barbeque'd up that night. (http://fearthecowboy.com/post/What-is-the-Master-Keyc29d-that-is-used-to-create-the-key-pair-and-PPID.aspx)

My pappy always used to say "Don't judge people by their relatives." -- good advice at the best of times.(http://fearthecowboy.com/post/How-to-move-the-herd-one-open-source-project-at-a-time.aspx)

It reminds me of somethin' my pappy told me... "There are three kinds of men: The ones that learn by reading. The few who learn by observation. The rest of them have to pee on the electric fence."  Well, I think I ended up in the last category, but at least I learned somethin'...(http://fearthecowboy.com/post/Random-Hacking-Windows-Media-Encoder-and-DirectShow.aspx)

Perhaps you should what my pappy always used to tell me: “Don't try to understand 'em, Just rope, throw, and brand 'em” Now, it’s entirely possible he was talkin’ about cattle and not code at the time, but I figure it’s good advice at the best of times. (http://fearthecowboy.com/post/CoApp-FAQ-Why-demand-all-code-be-signed.aspx)

‘Course, my pappy always used to tell me “it don’t take a genius to spot a goat in flock of sheep” … Sure, it’s easy to see what the problem is, question is, how do we go about fixin’ it?(http://fearthecowboy.com/post/Whate28099s-this-e28098CoAppe28099-all-about.aspx)

I remember what my pappy told me about tryin' to change the world: "If you're ridin' ahead of the herd, take a look back every now and then to make sure it's still there with ya".  Well, I'll keep checkin', but y'all gotta try to keep up.(http://fearthecowboy.com/post/Open-Source-at-Microsoft-Herdin-cats-or-Cow-Chips.aspx)

As my pappy use'd to say "Lettin' the cat outta the bag is a whole lot easier than puttin' it back".  If I give a site a password, and they require me to divulge the password back to them before granting me access, It's my butt between the bull and the fence if the connection between me and them ain't secure. (http://fearthecowboy.com/post/What-happens-when-my-laptop-gets-stolen.aspx)

Uh-oh. My pappy used to remind me of two things about gettin' into trouble. First, "Never slap a man who's chewing tobacco." .. that's good advice you can't afford to forget. The second is a little more on-topic, "If you find yourself in a hole, stop diggin' ." (http://fearthecowboy.com/post/Me-and-my-PPID-Can-I-rely-on-it.aspx)

(cross-posted from the mailing list)

Windows Side-by-side (WinSxS) technology is a really shiny piece of technology that is not well enough understood, and often misused. This comes from a variety of reasons, one of which is the documentation—while quite excellent—only makes sense if you actually understand everything about WinSxS before hand. When I started brainstormin’ how to make CoApp go, I had an inklin’ that WinSxS would solve many of the most painful problems, it was just a matter of me figurin’ it out. Like my pappy always used to tell me “It may not be the easy way, but it’s the cowboy way.” … Well, WinSxS may not be the easy way (yet!), but it will most assuredly be the CoApp way.

Q:What exactly is Windows Side-by-side (WinSxS)?

A:I’ll let the good folks from Wikipedia take a stab at that:

Hmm. I’m not sure that cleared it all up. Simply speakin’, WinSxS lets us take a DLL, give it a version number, digitally sign it, and then tell Windows to install it into a little box somewhere, and when we come lookin’ for it, it’ll give it to us.  Now, when we make a new version of that same DLL, we can tell Windows that when someone comes lookin’ for the older one, that they should use this one instead.

Now, before some real WinSxS expert rides up and starts tellin’ us that it does a wagonload better than that, I’ll capitulate and say sure—it most certainly does a heck of a lot more, but we’re gonna take pony steps first. This time around, I’m gonna give ya’ll the easy explanation. Damn city slickers…

Q: So WinSxS stores DLL files?

A: Well, technically speakin’, Windows can store more that DLLs with WinSxS—and consequently, rather than just call them DLLs, they call them Assemblies. And so WinSxS stores Assemblies for us.

Q: Hang on, that’s startin’ to sound a lot like that .NET thing…

A: Well, that’s because .NET assemblies and WinSxS are very similar.  Good eye. Now, stop fussin’.

Q: How does WinSXS store these Assemblies?

A: There are two implementations of WinSxS, they both work the same for the devleloper, with some minor behind-the-scenes implementation differences that aren’t important, and I’ll explain the Windows 7/Vista/2008/2008 R2 model.  In the C:\Windows directory, you’ll find a subdirectory called WinSXS. Inside there is what appears to be an extraordinarily large number of directories, each holding a few files. And it seems a bit massive, and some of it seems redundant—but it ain’t.

WinSxS manages the files in there, so you don’t go about muckin’ with em.  On top of that, it’s not really using up all the space you think it is, ‘cause it’s using hard links to manage multiple connections to the same files, so leave em alone. You’re not gonna ‘save space’ by messin’ with em.

Q: Are you sure I can’t clean that up? It seems like it’s takin’ like 11 gigs of space.

A: You know, my pappy always used to tell me, “Never squat with your spurs on”. Good advice, and here’s a bit more, so pay attention. First of all, this ain’t 1989—disk space just ain’t that expensive. Hell, even if you needed a couple more gigs, you could probably find it somewhere else a lot easier, than muckin’ with somethin’ you really shouldn’t. And all that hard link’n is really hiding the fact that it’s not really using that much space.

Q: That WinSxS folder is a mess. How does it find anything?

A: When WinSxS stores stuff in there it stores it by using it’s assemblyIdentity. The assemblyIdentity consists of the name, the processorArchitecture (amd64/x86), the version (##.##.##.##), the publicKeyToken that was used to sign the assembly, and optionally the language of the assembly (en-us, fr-ca, etc...).

The publicKeyToken is a 16-character hexadecimal string representing the last 8 bytes of the SHA-1 hash of the public key under which the assembly is signed.

Q: All things being equal, each version is ‘unique’?

A: Yeah, pretty much.  Even though everything else can stay constant, every version of an Assembly is uniquely accessible.

Q: How does a developer set all that information for an assembly?

A: By attaching a particular kind of resource to the Assembly called an assembly manifest. The assembly manifest gives the Assembly its identity.

Q: How does a developer choose the version of the Assembly they want to bind with?

A: By attaching an application manifest to the application, the LoadLibrary API will ask the WinSxS system to load the correct Assembly at run time.

Q: How does the publisher of the shared Assembly redirect old bindings to new Assemblies?

A: By specifying the policy in a publisher configuration file that gets registered along side the Assembly.  The publisher can redirect specific versions, or a range of versions of old assemblies to a new one.

Q: Can the developer consuming an Assembly say “no, I really want a very specific version” ?

A: Yes. But I’m not going to explain that right now. You’re trying to solve the wrong problem, and it’s a slippery slope of pain and suffering. If you continue down this path, I’m gonna hog-tie you and throw you in another wagon.

Q: Didn’t you say that we were going to use WinSxS to support multiple compilers, but same version of the same assembly?

A: Yep.  It turns out there is a really easy way for us to support multiple compilers (VC9, VC10, MinGW, etc) with the same Assembly name, version number and architecture by using multiple certificates; one for each different compiler. So, the CoApp project will have several certificates, each publicKeyToken will support a different compiler. If that sounds complicated or troublesome, trust me, it ain’t.

Q: Isn’t WinSxS the reason that Visual C++ dependencies are so problematic?

A: Yep.  That’s because … uh, well, let’s just say I disagree with how they used them.  Visual C++ would bind to a very specific version number. They would put something like this in the application manifest:

<assemblyIdentity type='win32' name='Microsoft.VC80.CRT' version='8.0.50727.42' processorArchitecture='x86' publicKeyToken='1fc8b3b9a1e18e3b' />

Now, as you can plainly see they use version 8.90.50727.42. Let’s assume you want to run an app, and it needs the C++ redist files, and what if moments before, you had installed the C++ redistributables for the first time, with the version number 8.90.50727.40 ?  When the application didn’t work, you’d think to yourself, but I just installed them (and you' really didn’t know what version it was), why is it not working? There isn’t an easy way for end-users to track this down, and it makes people go insane.

You’re probably thinking that well, you need to get the newest build, right?  But you know what? The app probably would have run perfectly with the previous version, and if it was that damn important to have the absolute up-to-the-moment build, they should have given you the updated runtime in the app’s installer.

Q: What should Visual C++ be doing?

A: Well, in my humble opinion, they should have been binding to version 8.90.0.0 and let WinSxS use publisher policies to point to a newer version.  If an application has a known problem with an old redist, they should be shipping the latest one with their app, and let WinSxS take care of it all.

Q: The Visual Studio team fixed that in VC10 right?

A: *blink*.  Um… sortof.  The VC team stopped using WinSxS for the C++ runtime.  I don’t really agree with what they’ve done, but heck, maybe I’m just a back-woods hick who doesn’t really understand the problem.  After all, how much can you really know about this stuff without one of those fancy college educations?

Q: But… you still want to use this for CoApp?

A: You betcha. WinSxS will work, as long as we are playin’ by the rules, and we’ll make sure there are some tools for divinin’ what’s wrong if somebody tries somethin’ fishy.

Q: Is this all there is to WinSXS?

A: No, not by a wide margin. But it’s enough for you to get a good grip on what it’s for, and why it’s so damn central to CoApp.

Hey, rather than commenting here, come join mailing list (join the team at https://launchpad.net/~coapp-developers) and continue the conversation!

(cross-posted from the mailing list)

We had a discussion on the mailing list about maintaining a Symlink to the most recent version of a particular library, and this was of some concern.

The installation directory (where libraries and their associated files) are installed has absolutely zero effect on binding choices for picking up the actual DLL used for a particular shared library.  The directory layout is actually a convenience for a developer who wishes to simply use the latest, CoApp packaged version.

Shared library binding is handled by WinSXS, where a developer chooses a particular version of a library to bind to. So, for example let's say that you want to compile up Python with OpenSSL  0.98h . You can specifically choose the path for the import library by including the version, and you are bound to that version. Now, there is one slight caveat.

WinSXS allows a publisher of a shared library to set the policy of the shared library so that if they put in a bug fix, but do alter the binary interface (ABI) then at run time, the application picks up the latest ("Most recommended") binary compatible version of the library.

This means that you bind Python to OpenSSL 0.98h. A bug is found, the publisher of OpenSSL releases 0.98k, and since the ABI hasn't changed, the major & minor versions haven't changed [0][98], just the revision [k]. So once the new library is installed, Python will use it.

This is the behavior we want

However, when OpenSSL 0.99a ships, by virtue of having the minor # changed [98]->[99] the policy should not forward use on to the new version, and Python would continue to use 0.98k.

This is extremely important, because when we’re using shared libraries, it's the author's/publisher's  responsibility to provide security and bug-fix updates in a binary compatible way, and it's the consumer's responsibility to use the libraries in a consistent fashion so we preserve that intent.

Yes, this relinquishes control from the consumer to the publisher, but that is a better balance anyway.

So, if you really really really need to bind to a very specific version, and you are dead set against preserving the shared library model, while it is possible to force bind dynamically to a specific version, what you are really wanting is to be statically bound.

Lemme say that again: If you don’t' intend to work with shared libraries in the prescribed way, you really want to be statically bound, and should select the foo_a.lib. You will actually gain additional performance, and will still be able to work in-process with other applications using the dynamic library version without collision.

Shared libraries need to be used in a shared-responsibility fashion.

Hey, rather than commenting here, come join mailing list (join the team at https://launchpad.net/~coapp-developers) and continue the conversation!

(cross-posted to the mailing list)

I’m answering this in a one-off manner a little too often, so I’ll dump it all here.

Q: What is Code Signing?

A: For the long-winded answer, check out Wikipedia’s article on Code Signing. All that aside, it’s a way of attaching a cryptographic signature to a binary (EXE, DLL ... etc) that allows anyone validate what organization compiled the code, when it was compiled, and that it hasn’t been tampered since it was signed.  The certificate must be issued from a Certificate Authority—pretty much the same gang of folks who churn out SSL certificates. Similar benefits are often accomplished in the Linux world by a couple of cryptographic operations: An MD5 hash, which someone can verify that a binary has not been tampered with, and less commonly, PGP/GPG signatures for files.

Q: What are the requirements to signing code?

A: Signing code itself (on Windows) is pretty easy using the signtool that comes in the Windows SDK...provided you have a certificate. To get a certificate, you can either generate a certificate yourself (often called self-issued or self-signed) or you can get one from a reputable CA. The CA will validate that you are who you claim to be (by whatever methods they deem necessary) before issuing the certificate. The certificate is good for a fixed period (typically for a year) and can be renewed. When you sign a binary with the signtool, it can use a time server to validate that the binary was signed when the certificate was still valid, and the binary won’t expire when the certificate expires.  In the event of malice or loss-of-possession of a certificate, the CA can revoke the certificate by adding it to a Certificate Revocation List (CRL).

Q: What benefits will this provide?

A: Signed code will allow us as users to identify who is responsible for a particular binary or package. It also lets us uniquely identify a binary regardless of name—two publishers can create the same binary, but they would sign it with different certificates, thereby providing us a way of selecting one over another.

Q: Does this completely halt malware?

A: It should slam the doors pretty damn tight. Anonymity is how malware gets around, either by corrupting an unsigned binary in the first place, or by shipping software that isn’t signed and having an end user execute it.  Signed binaries don’t allow malware operators the pleasure of hiding behind the fact nobody knows who they are, and if they did acquire a certificate fraudulently and signed a binary, the certificate can be revoked, prohibiting the binaries issued from validating.

Q: But, I can’t afford a code signing certificate!

A: I’m committed to making sure that cost is not a blocker for code-signing open source applications. I’ve identified a few avenues that this can happen, some of which I’m investigating right now.

Q: Why can’t I simply use a self-signed/self-issued certificate?

A: While we could validate that the same certificate (and by extension, because a persons possesses the signing key, a person) has signed a particular package as another, it doesn’t give us a trustable way to identify who that was.  In the same fashion a self issued SSL cert protects against casual man-in-the-middle attacks, it doesn’t identify who you are connecting with, and would permit a non-casual man-in-the-middle attack or redirection.

Q: So who’s gonna sign all those apps?

A: Publishers of CoApp-compatible packages will be able to sign them. Anyone who acquires the appropriate certificates can be a publisher. As I’ve noted elsewhere, Publishers don’t have to be the author of the application.  The CoApp project will be one such publisher of Open Source libraries and applications, and will do so for a very wide range of software.

Q: You’re saying CoApp is gonna sign my apps?

A: Sure, provided that it meets all of the requirements for packaging and automation of the build process. We will likely implement a code-validation process where it would have to pass both an automated scan and a manual verification by a peer or peer committee, to avoid any shenanigans. As well, we are confident that other open source organizations like Apache or Eclipse or others will be able to do the same.

Q: LA LA LA LA LA I’m not listening. Most open source projects can’t afford a code signing certificate!

A: Stop that. I’ve already told you that it will not be an issue. Code Signing is not an option.

Q: Hey, waitaminute. What about .NET code. Doesn’t that Strong Naming stuff obliviate the need for Code Signing with Certificates?

A: Strong naming is not code signing. Although digital signatures are used to generate strong names, strong naming does not use digital certificate, instead it uses self-created digital keys . This means that it is not really possible to establish a chain of trust for the public key; and therefore there is no way to bind the identity of the software publisher to the private key being used to sign the assembly (see the question about self-signing above). Since strong names do not support digital certificates, they do not support expiration or revocation of key material either. It is always recommended that strong names be used along with Authenticode code signing, where possible.

If’n you’re still having problems, perhaps you should what my pappy always used to tell me:

“Don't try to understand 'em, Just rope, throw, and brand 'em”

Now, it’s entirely possible he was talkin’ about cattle and not code at the time, but I figure it’s good advice at the best of times.

Hey, rather than commenting here, come join mailing list (join the team at https://launchpad.net/~coapp-developers) and continue the conversation!