Fear the Cowboy

Life of Microsoft Open Source Developer

When scripting goes bad…er, insane.

clock June 24, 2010 09:14 by author Garrett Serack

Hi, my name is Garrett, and I have a problem.

I love scripting stuff. And, I by stuff, I mean everything. Really, I often use it as an exercise to understanding whatever it is that I’m trying to learn—if I can automate it, I’ve got a strong feeling that I understand it.

As you may or may not know, my absolute favorite language is JavaScript (and its many incarnations, like JScript). My love for JavaScript even predates the existence of JavaScript—I heavily used a scripting language called C-- (C minus minus aka ‘Cmm’, later renamed to ScriptEase). A somewhat forgotten company called Nombas had written it. A clip from the wayback machine (http://bit.ly/dCjRd9):

 

In 1992 and 1993 Nombas developed a language named Cmm (for C minus minus, or "C without the hard stuff") for use as an embeddable scripting language, showing that it was possible to have a full-powered language that was simple enough to replace macro languages. Years later we would change the language name to ScriptEase, because Cmm was "too negative, and the letter 'C' scared people". Cmm was first released in a shareware product called CEnvi, which won awards and fame and is now available as ScriptEase:Desktop.

When Netscape's first commercial browsers were released we made a version of CEnvi that could handle short scripts embedded within web pages. By embedded scripts within the page we allowed the client side to handle processing, rather than making all dynamic interaction happen on the server. This brought immediate client-side interaction with the user. The advantages of client-side handling were made obvious by Nombas' "Espresso Pages", and Netscape soon began work on their own version, which they called LiveScript, and then renamed to JavaScript just before its final release.

Cmm and JavaScript were so similar anyway, it was inevitable that one of them would perform the necessary course corrections and become like the other.  Regardless of the specific flavor, for me, the notion of a dynamic c-style scripting language has always had extremely strong appeal.

Over the last 15 or so years, I continued to use JavaScript (or JScript) whenever I needed a script, and I’ve been very happy.  As well, I continued to refine some of my scripting techniques and create some rather clever hacks to do a lot of the things I needed to do.

The real problem began when I needed to do some batch scripting—that is some useful scripting of command lines and whatnot. On Windows (out of the box), there are only a couple of options: Batch scripting with CMD, and in later years, PowerShell. For a long time, I used CMD Batch Scripting for lots of this, but it’s so … so … nasty. You really don’t want to do serious work in there, it’s just not up to the task.

PowerShell should have made me ecstatic—a .NET language, with access to pretty much everything I ever wanted in a scripting language, tons of support, and clearly the answer to my problems.  Unfortunately, that syntax breaks my brain. I just can’t get natural with it, no matter how much I’ve tried.

So, a couple of years ago, I started writing a nifty library that allowed me to make complicated batch scripts using JScript; and it’s been working out nicely. I get access to everything in the OS that I need, the language is installed on every version of Windows that I have used, and it doesn’t require any magic to make work.  Over time, I’ve added in code to do some ‘theoretically impossible’ tasks in JScript, including:

  • full support for batch scripting (capturing and using the results of other command line apps)
  • full use of environment variables in scripting
  • formatted string handling
  • Binary file read and write (and they said it couldn’t be done)
  • some basic file assertions
  • http downloading
  • logging
  • PE executable identification
  • process management
  • an extensible library interface and libraries for Restore Points, Speech (SAPI), JSON, MD5, Hyper-V VHD handling, and a twisted on-the-fly C# compiler.

And probably some other things I can’t remember.

Here’s where it starts to get weird.

Then, last night I was doing some more batch scripting at home, and I had my insane idea. What if I wrote a script that transformed a batch-like script language into JScript, and executed it. And, just because I’m that crazy, I’ll write it in JScript too.

And now, a scan hundred or so lines later, my latest script language is born. I call it ‘gs’ – short for gScript… a cousin to my other (unreleased) scripting language g#.

A quick little example script:

test.gs
// comments are still slashies

// you can execute command lines just like regular batch languages:
// (this just prints the robocopy usage)
robocopy 

// Or you can capture the results of a command:
// (this uses the command processor to get a list of files (as text))
var $RESULT = cmd /c dir /b c:\*.zip

// there are built-in commands:
// so far cd, md, rd, erase, echo, dir
echo {$RESULT}


// and when you want to do some jscript, start a jscript block:
js {
    // this is a jscript code block
    //any legal jscript is ok here!

    // of course, you have access to the js.js function library:
    print("hello World");

    // and functions work just fine:
    function foo() {
    
        // you can even run batch commands from where with hash-bang:
        $WGETHELP = #! wget --help

        #! cmd.exe /c ver

        // output from the last command is always captured as an array in
        // $StdOut and as a string in $StdOutString
        print($StdOutString);

        return $WGETHELP;
    }
   
}

// oh, and for loops work outside the js blocks in the batch-world too:
// prints out text:
for(var i=0;i<10;i++)
    echo {$RESULT} {i}

using the gs script runner to run this:

Script Run Example:
C:\>gs test.gs

( or, to dump the processed script out )

C:\>gs /debug test.gs

// comments are still slashies

// you can execute command lines just like regular batch languages:
// (this just prints the robocopy usage)
$$("robocopy");

// Or you can capture the results of a command:
// (this uses the command processor to get a list of files (as text))
var $RESULT = $$.RunQuiet("cmd /c dir /b c:\\*.zip");

// there are built-in commands:
// so far cd, md, rd, erase, echo, dir
print('{$RESULT}');


// and when you want to do some jscript, start a jscript block:

    // this is a jscript code block
    //any legal jscript is ok here!

    // of course, you have access to the js.js function library:
    print("hello World");

    // and functions work just fine:
    function foo() {

        // you can even run batch commands from where with hash-bang:
        $WGETHELP = $$.RunQuiet("wget --help");

        $$("cmd.exe /c ver");

        // output from the last command is always captured as an array in
        // $StdOut and as a string in $StdOutString
        print($StdOutString);

        return $WGETHELP;
    }

// oh, and for loops work outside the js blocks in the batch-world too:
// prints out text:
for(var i=0;i<10;i++)
print('{$RESULT} {i}');

I’ve skimmed over so much of how this all works, but feel free to play with the code.  Some of this I published earlier in my gsToolkit project on Codeplex.  I’m using all of this in the CoApp project for a variety of tasks, so I’m publishing the whole set in a CoApp project on Launchpad




Using JScript as a batch scripting language (Part III)

clock May 18, 2009 13:19 by author Garrett Serack

Now that I’ve shown how to build a cool .Format() method for strings, we can put it to good use in a lot of places.

In batch scripting, it’s really nice to be able to make nearly every call support replacement arguments in a consistent fashion. In cmd.exe batch scripts, we use %var% all over the place. In JScript batch scripting, we simply use {$VAR} instead, and put a little bit of code in the top of our functions to help out with that.

First, the one-stop-universal-arguments helper, which we can add to the source from before:

Scripting.js
function ArgsToArray(x) { return Array.prototype.slice.call(x);} 

// FormatArguments must be passed either:
//     one argument: 
//          containing the arguments object from the caller
//          where the first argument of that should be the format string.
//          == or ==
//          just the format string.
//
//     two arguments:
//          the first argument is the format string
//          the second argument is the argument collection from the caller.
function FormatArguments(args, moreargs) {
    var result = "";

    if (arguments.length == 1) {
        if (typeof (args) == "object") {
            args = ArgsToArray(args);
            result = "" + (args.shift());

            if (args.length == 1 && arguments[0].length > 0)
                args = arguments[0];

            return result.Format(args);
        } else return ("" + args).Format();
    } else if (arguments.length == 2) {
        if (typeof (args) == "string" && typeof (moreargs) == "object") {
            result = args;
            args = ArgsToArray(moreargs);
            args.shift();

            return result.Format(args);
        }
    }
    throw "Invalid Argument passed to FormatArguments";
}

FormatArguments()  gives us the ability to do variable substitution in any function, in a very flexible way. You can simply use the function to give you a completed string:

Test-4.js
// use the first string as the format string, 
// the rest are potential value substitutions
function Test1() {
    var foo = FormatArguments(arguments)
    WScript.echo( foo );
}

Test1("the path is {$PATH}");

// both arguments are used as format strings. any 
// parameter substitution should use numbers starting
// at {1}, since the 'destfile' parameter is techincally {0}
function Test2(srcfile, destfile) {
    var srcfile = FormatArguments(srcfile, arguments);
    var destfile = FormatArguments(destfile, arguments);
    var msg = "Copy {0} to {1}".Format(srcfile, destfile);
    WScript.echo(msg);
}

Test2("{$WINDIR}\\system32\\notepad.exe" , "{$USERPROFILE}\\desktop\\notepad.exe");

Knowing that, we can create a few functions that will use the FormatArguments function:

Scripting.js
// Some global objects we'll need 
var WSHShell = WScript.CreateObject("WScript.Shell");
var WinShell = WScript.CreateObject("Shell.Application");
var procEnvironment=WSHShell.Environment("PROCESS")
var fso = new ActiveXObject("Scripting.FileSystemObject");

function Print() {
    WScript.echo(FormatArguments(arguments));
}

function cd() {
    WSHShell.CurrentDirectory = FormatArguments(arguments);
}

function pwd() {
    return WSHShell.CurrentDirectory;
}

function erase(file) {
    file = FormatArguments(arguments);
    if (exists(file))
        fso.DeleteFile(file);
}

function rmdir(folder) {
    folder = FormatArguments(arguments);
    if (folderExists(folder))
        fso.DeleteFolder(folder, true);
}

function exists(file) {
    return fso.FileExists(FormatArguments(arguments));
}

function folderExists(folder) {
    return fso.FolderExists(FormatArguments(arguments));
}

function mkdir(folder) {
    folder = FormatArguments(arguments);
    if (!folderExists(folder))
        fso.CreateFolder(folder);
}

Now, we can do some pretty nifty little batch scripting:

test-5.js
// Bootstrap Scripting Library
eval(new ActiveXObject("Scripting.FileSystemObject").OpenTextFile("Scripting.js", 1, false).ReadAll());

$ORIG_DIR = pwd();

if( exists("{$WINDIR}\\log.txt") ) {
    Print("Deleting log file");
    erase("{$WINDIR}\\log.txt");
}

mkdir("{$TEMP}\\tempdir");

Next time, we’ll see how we can call on command-line tools to do our bidding, and play with the results.




Using JScript as a batch scripting language (Part II)

clock May 15, 2009 16:14 by author Garrett Serack

Last time, I wrote about synthesizing an #include facility along with handling environment variables in a trivial way.

This time, let’s look at filling in a couple more gaps in JScript’s basic scripting functionality.

What’s wrong with String?

The String class in JavaScript/JScript is … ok. If you work in .NET enough, you’ll eventually get to the point where you’d like a couple of functions that seem to be missing. First, is Trim, which I’ve always thought as obvious:

Scripting.js – String Prototypes
//----------------------------------------------------------------------------
// String Prototypes



String.prototype.Trim = function() { return (this || "").replace(/^\s+|\s+$/g, ""); }

And secondly, is something I miss dearly from when I’m in .NET, is a string Format function. Now, I’ve seen a couple of these that were pretty simple, but I wanted to be able to do some really cool Format strings (and, since the rest of my scripting will rely on them quite a bit, I’m going for the gold here.

Scripting.js – String Prototypes
String.prototype.Format = function() {
    var args = (arguments.length == 1 && typeof (arguments[0]) == "object") ? arguments[0] : arguments;

    result = this;
    while (z = /{(.*?)}/.exec(result))
        try { result = result.replace(z[0], isNaN(z[1]) ? eval(z[1]) : (args[z[1]]||"??<" + z[1] + ">??" )); }
    catch (x) { result = result.replace(z[0], "??<" + z[1] + ">??"); }
    return result.replace("??<", "{").replace(">??", "}");
}

So, with this tasty little script in my pocket, I can do ad-hoc format strings that use one of a few types of replacement:

test-3.js
// simple parameter # replacement like .NET
var foo = "My Name is {0}. Please to meet you {1}".Format("Garrett", "Mr. Serack");
WScript.echo(foo);

// *global* variable replacement
ABC = "this is a test";
var bar = "'{ABC}' is really a replacement string".Format();
WScript.echo(bar);

// really, any legal expression in there is fine.
bar = "'{100+200/75}' is really a replacement value?".Format();
WScript.echo(bar);

// and if it doesn't match, just leave it alone...
bar = "'{teehee}' isn't really a replacement value.".Format();
WScript.echo(bar);

// even if it's just a number.
bar = "'{1}' isn't really an index.".Format();
WScript.echo(bar);

Which spits out the following:

Ouptut
My Name is Garrett. Please to meet you Mr. Serack
'this is a test' is really a replacement string
'102.66666666666667' is really a replacement value?
'{teehee}' isn't really a replacement value.
'{1}' isn't really an index.

Basically, it just looks for anything in braces, and checks to see if it’s a number. if it is, it tries to substitute in the n’th parameter passed in. Otherwise it just does an eval() on it—which, will replace expressions, or global variables. If something goes haywire, it just leaves it alone (well, it does some switcheroo stuff to get past the while loop, but it puts it back in the end).

Next time, we’ll see how to use this… everywhere.




Using JScript as a batch scripting language (Part I)

clock May 8, 2009 10:02 by author Garrett Serack

As I mentioned a few days ago, I chose JScript to script of the optimized PHP build process that I've built. JScript in-box on pretty much every modern Windows operating system, and provides a great deal of flexibility and benefits for a scripting language:

- it's syntax is C like. Very tasty.

- it gives access to *a lot* of functionality via COM and WMI.

- if you know enough about Windows Scripting Host and JavaScript, you can accomplish darn near anything if you want it bad enough.

- JScript's regular expressions. While not the universe's most powerful, they are certainly an integral part of the language.

- Prototypes allow you to do things to classes that can significantly boost productivity.

It however does lack a few of the basic things that I'd like to see in a batch* scripting language:

- an #include mechanism.

- Easy interaction with environment variables.

- Interaction and leveraging of external processes

- Analogs to the built in command-line functions like DIR, MKDIR, ERASE, RMDIR, etc.

* And, by "batch" scripting, I mean the scripting of external commands and programs to automate something that would otherwise be done by hand.

I was thinking about all of this over the last few months and started experimenting. Along the way I came up with a basic library of functions that address the deficiencies in a rather clever way.

First, let's fix the lack of #include

JScript (well, and VBScript) really didn't do us any favors by not supplying us with the ability to reuse code in a simple fashion (And yes, I know about .WSC components, and I'm not keen on how *that* turned out. Ask me about that again later some time)

Anyway, #including another JScript file is pretty easy if you know what to do. Not pretty, but easy:

//---[Test01.js]----------------------------------------------------------
// includes the scripting library
eval( new ActiveXObject("Scripting.FileSystemObject").OpenTextFile("Scripting.js", 1, false).ReadAll() );

The eval function gives us the ability to just run code that we pass in at runtime. This will give us a few little bumps along the way later, but for the most part, is pretty darn good.

 

How about those environment variables

The WScript.Shell object has some methods that let us get at environment variables, but I wouldn't exactly consider them "Script Friendly". So, the first thing I did, was create a some basic functionality for exposing the environment as global variables. (Some of how this gets useful, comes a bit later.)

//----------------------------------------------------------------------------
// Global variables
var GLOBALS=this;
var WSHShell = WScript.CreateObject("WScript.Shell");
var procEnvironment=WSHShell.Environment("PROCESS")
GlobalInit();

// Loads the environment from into UPPERCASE variables in the global namespace.
// each variable is prefixed with a $
function loadEnvironment() {
env = CollectionToStringArray(procEnvironment);
for(each in env) {
var v= env[each];
if(typeof(v)=='string')
if ((p = v.indexOf('=')) == 0 )
continue;
else
GLOBALS['$'+v.substring(0,p).toUpperCase()] = v.substring(p+1) ;
}
}

// Sets environment variables with the all string variables in the global namespace
// that start with a $
function setEnvironment() {
for(each in GLOBALS) {
var t = typeof(GLOBALS[each]);
if(t =='string' t=='number') {
if( each.indexOf("$") == 0 ) {
if( IsNullOrEmpty(GLOBALS[each]) )
procEnvironment.Remove(each.substring(1));
else
procEnvironment(each.substring(1)) = GLOBALS[each];
}
}
}
}

// takes one of those funky-groovy COM collections and gives back a JScript
// array of strings.
function CollectionToStringArray(collection){
var result = new Array();
for( e = new Enumerator(collection); !e.atEnd(); e.moveNext() )
result.push(""+e.item());
return result;
}

// returns true if the string is null or empty
// Yeah, I was thinking of c# when I wrote this.
function IsNullOrEmpty(str) {
return (str "").length == 0;
}

// Our function for bootstrapping the required environment.
function GlobalInit() {
loadEnvironment();
}

Now, we can easily access environment variables:

//---[Test02.js]----------------------------------------------------------
// includes the scripting library
eval( new ActiveXObject("Scripting.FileSystemObject").OpenTextFile("Scripting.js", 1, false).ReadAll() );

WScript.echo( "Path is :" + $PATH );

Next time, I'll show how I added code to execute and capture other external commands, and show a few cool functions that make playing in JScript a bit simpler.





The Cowboy

What I'm Tweetering about...

 

follow me on Twitter

Calendar

<<  September 2010  >>
MoTuWeThFrSaSu
303112345
6789101112
13141516171819
20212223242526
27282930123
45678910

View posts in large calendar

Sign in