A couple weeks ago I was reading Paul Graham's excellent book On Lisp, and I was struck with the thought that many of the features of Lisp he seems to really like are also provided by JavaScript. These include dynamic typing, lexical variable scoping, first-class functions, closures, etc. But macros are of course completely missing from JavaScript.

How hard would it be to add Lisp-like macros to JavaScript, I wondered? I'm learning about Lisp macros for the first time here, so I didn't really know, but I thought it would be fun to try.

This will be the first of several posts describing "JaM", my attempt to add Lisp-like macros to JavaScript.

It seems to me there a few essential ingredients to make a macro system for JavaScript that is as powerful as Lisp's:

1. Macros are written in JavaScript with access to all functions defined up to that point.
2. The source code that the macros read and write is in a format that is easy for JavaScript to manipulate, such as a JSON parse tree, not just a big string.

Number 1 is important, because otherwise you end up with something like C or C++ macros -- a weak mini-language that looks and acts differently than the host language. I suppose it's better than nothing, but it doesn't have nearly the language-building power that Lisp macros do.

As for number 2, I suppose it would be possible to set up a system where each macro could match based on a regex or something equally flexible, but this would have a couple of drawback. First, most macros wouldn't need that flexibility and would end up reimplementing the same patterns as other macros, including tricky things like matching up pairs of parens, curly braces, etc. Second, I don't want to try to read code that's being worked over by macros that are that flexible. I'd like to be able to assume that each macro is constrained in its scope, following roughly the same rules as the core JavaScript language. Lisp macros certainly have this restriction, so I'm not worried about this causing my macro system to be too weak.

Let's start off with a function JaM.include("foo.js"), which will fetch the JavaScript file, expand any macros in it, and then run it through the regular JavaScript eval(). Except it's not quite that simple because we want to be able to mix function definitions and macro definitions after each other in any order so they can build on each other. So we'll need to split "foo.js" into top-level expressions that can be correctly evaluated. I'll leave the details of this step for another time, but what's important is that for each top-level expression we will expand any macros and then evaluate it before moving on to the next expression.

This will probably be very slow -- certainly slower than just letting the JavaScript interpreter have at foo.js directly. But if that's the sort of thing that worries you, perhaps you'll be comforted by these thoughts: First, as described here the parsing, macro expansion, and separate evals happen once when a new source file is being loaded. Once the code is up and running there may be no need to do further macro expansion. Second, we could add another step to the processing of each top-level expression: after expanding any macros in an expression we could save the resulting code before evaluating it. Doing this with each expression in turn would result in a file full of fully-expanded normal JavaScript that browsers could run directly and at full speed.

Ok, that was pretty dull; perhaps I can whet your appetite with some sample code? Let's say you really like perl's unless keyword, and you wish JavaScript had something similar. With JaM, you can add it yourself:

defMacro unless( expr, block: body ) {
  return {{
    if( ! ( #expr ) ) {
      #body
    }
  }};
}

unless( true == false ) {
  alert( 'Hello, macro-enabled world!' );
}

It may not be the most useful example one can imagine, and even so I feel I've got a fair bit more to explain before it'll make much sense. We'll build our way up to defMacro, {{ ... }}, and the # operator over the next few posts. Hopefully, though, all of the above will be sufficient for you to understand where I'm headed.

¶ 12:30 PM Tuesday, October 10, 2006 by Chouser