n01senet

An array of functions.

2008-05-14T18:24:00.000-07:00

Write a function (in your favorite language) that takes an integer and returns an array of that length, where each item in the array is a function that returns its own index in the array.

I proposed the above as a challenge for some friends, and got a few nice responses.

bash

The first answer was in bash by agriffis. After some adjustments, he settled on:

makearray() {
  declare x arr=$1
  eval "$arr=()"
  for ((x=0;x<$2;x++)); do
    eval "${arr}_func_$x() { echo $x; }; $arr[x]=${arr}_func_$x"
  done
}

To use it, pass in an array name and your desired array size:

makearray myarray 10

This is a bit of a cheat in that it actually creates an array of function names, not functions or closures. But it's bash, so you should be impressed it's even vaguely possible. To call the function at index 3:

${myarray[3]}

...and that echoes 3 to stdout, just as it should.

Python

Brett wrote a couple solutions in Python, settling on:

lambda x : map(lambda y : lambda : y, range(x))

This is pretty compact, so let's replace a couple of the lambdas with named function definitions:

def makefunc( y ):
  return (lambda : y)

def makearray( x ):
  return map( makefunc, range(x) )

This does exactly the same thing as Brett's original solution, but takes up more space doing it. range(x) gives us a sequence of integers, each of which is their own index. So now all we need to do is convert each of those integers into a function. map() helps us do this by passing each of the integers to makefunc, and returning an array of the results.

So makefunc's job is to take an integer y and return a function. The lambda in makefunc has nothing to the left of its colon (:), so the returned function needs no arguments.

my_array = makearray( 10 )
one_of_the_funcs = my_array[ 3 ]
one_of_the_funcs()  # returns 3

Or you can pack it all back together:

(lambda x : map(lambda y : lambda : y, range(x)))( 10 )[ 3 ]()

JavaScript

I originally thought of this challenge as a way to test someone's skill with JavaScript, so I thought I'd better write a JavaScript solution:

function makearray(x) {
  function makefunc(y) {
    return function(){ return y; }
  }

  var a=[];
  for( var i = 0; i < x; ++i ) {
    a[i] = makefunc(i);
  }
  return a;
}

This is a lot like the longer Python solution, but there's no special lambda syntax, so I have to use the "function" and "return" keywords everywhere. Also you can't generally rely on there being a map function, so I have a for-loop that looks a lot like agriffis' bash solution.

There's no particular benefit to putting the makefunc definition inside makearray, except that it hides that name so nothing outside makearray can call it. The syntax for using it is actually identical to Python's:

makearray(10)[3]()

Watch out, though, because JavaScript makes it easy to do closures incorrectly. It might seem like we don't need makefunc at all, and that this would work:

function makebadarray(x) {
  var a=[];
  for( var i = 0; i < x; ++i ) {
    a[i] = function(){ return i; } // not what you want
  }
  return a;
}

The problem with this is that all of the functions in the array will be referring to the same variable i, whose final value is 10. So although we'll have 10 different functions, they'll all return the same value:

makebadarray(10)[3](); // returns 10 (not 3)

Perl

I first heard about closures while learning Perl, so it's only polite to include it:

sub makearray{ map { my $y = $_; sub{$y} } (0..$_[0]-1) }

Perl's map acts just like Python's, but it has special syntax so you can just say "map {...}" instead of "map( lambda y : ... )". Unfortunately, instead of naming your own variable, the special $_ variable is set to each element of your sequence in turn.

sub{$y} in Perl means the same thing as (lambda : y) in Python.

Oh, and you don't get to name your function parameters either, so makearray's first parameter is called $_[0]. We pass this to the .. operator which is like the range() function (in this case).

The syntax for using this is not identical to Python's:

&{(makearray(10))[3]}

Clojure

I'm on a Clojure kick these days, so of course I had to write:

(defn makearray [x] (map #(fn [] %) (range x)))

The elements here should all be familiar from the Python solution, except the very compact syntax #(fn [] %) which is just a shortcut for (fn [y] (fn [] y)).

A quick aside: It is tempting to say (fn [y] #(y)), but #(y) is a function that executes the value of y instead of simply returning it.

Anyway, a more important detail is that this Clojure makearray doesn't actually return an array. All the previous solutions we've looked at create all ten functions, regardless of which ones we end up calling. In contrast, the Clojure makearray returns a lazy sequence, and doesn't create any functions at all until they're accessed.

So our standard usage:

((nth (makearray 10) 3))

...returns 3 just as the others did, but it only had to create the functions up through the one we asked for.

One practical way to take advantage of this is to change makearray to no longer take an argument, and instead allow it to generate an un-ending series of functions. To do this we simply replace the bounded (range x) with (iterate inc 0). Let's also call the function makeseq since that's more accurate:

(defn makeseq [] (map #(fn [] %) (iterate inc 0)))

Now call it:

((nth (makeseq) 99))

...returns 99, and since no more functions are required (by nth in this case), no more work is done.

Just to drive this home, realize that since makeseq takes no arguments, it's returning an equivalent sequence every time. This means it doesn't need to be a function at all:

(def myseq (map #(fn [] %) (iterate inc 0)))

This sets myseq to the potentially infinite sequence, but at this point none of the functions have actually been created yet. The functions are created on demand:

((nth myseq 25))  ; returns 25
((first myseq))   ; returns 0
(second myseq)    ; returns a function that, when
                  ; called, will return 1

So how would you complete this challenge in your favorite language?

Ubuntu java plugin on Debian Lenny amd64

2008-03-27T15:22:00.000-07:00

Things I tried today:

apt-get install sun-java6-jre; no plugin included
apt-get install sun-java5-jre; no plugin inculded
apt-get install ia32-sun-java6-bin nspluginwrapper; this combination doesn't work because Java uses the OJI plugin API instead of the NPAPI that nspluginwrapper supports
download various packages from java.sun.com and install them using make-jpkg; oddly, this doesn't change the fact that Sun's Java doesn't provide an amd64 plugin, duh!

Finally, thanks to Dave Medberry's suggestion, I downloaded the Ubuntu Blackdown amd64 package from http://packages.ubuntu.com/hardy/j2re1.4 and installed it directly with dpkg. The final manual step was:

sudo ln -s /usr/lib/j2se/1.4/jre/plugin/amd64/mozilla/libjavaplugin_oji.so /usr/lib/iceweasel/plugins/

It might not be the latest version, but it works...!

Why you want macros (even if you don't know it yet)

2008-03-12T08:45:00.000-07:00

This is a slightly edited transcript of a conversation with my friend Brian.

Let's say you want to write your own "or" function. Let's say you're not satisfied with C++'s || because you want the actual value that is true, not just the boolean "true". You want to be able to say:

    QString x = y || "default";

And to make things easy, let's not worry about the goofy operator syntax. You'd be happy with:

    QString x = my_or( y, "default" );

Now, you could write that, right? Let's assume it only works on QStrings too, just to make things easy:

    QString my_or( const QString& a, const QString& b ) {
      return a.isEmpty() ? b : a;
    }

But what if b is a big ol' calculation:

    QString x = my_or( y, calculateDefault() );

You want to short-circuit, and only call calculateDefault() if y is false. In C++ you're screwed. Can't do it.

Also screwed in Java, C# (as far as I know), perl, ruby, etc. You need macros, or something similar. [See below for what I learned about C#]

Now, there are half measures that will let you squeak by in this very simple case. You can in fact use C pre-processor macros. Gross, but you can:

    #define my_or( a, b )  ((a)?(a):(b))

Oops, I just evaluated a twice.

In javascript (or perl or ruby) you can use closures:

    function my_or( fa, fb ) {
      var a = fa();
      return a ? a : fb();
    }

and then call it with this happy formulation:

    var x = my_or( function() { return y; },
                   function() { return calculateDefault(); } );

ew.

Now if the only reason you want a macro is for delayed evaluation (which is all you need in this case), Scala gives you that one particular feature.

    def my_or( a: => String, b: => String ) = {
      if( a ) a; else b;
    }

That tricky little => tossed in there makes the argument "lazy" and gives us what we need in this case. But of course there are other things that macros provide that the lazy argument feature does not.

Brian: So you would argue that macros should be part of every modern computer language

oh. Well... Hm, I'd never thought of phrasing it that way.

Having macros makes a language much more flexible and powerful.

Brian: Are macros in other languages like lisp implemented in a pre-processor fashion?

Lisp macros can call any regular lisp function. This a fundamental difference compared to, for example, C pre-proc macros which can't use anything except other pre-proc macros. Usage of lisp macros are expanded before the result is evaluated, so it's "pre" in that sense, but they can use other functions and macros and in that sense they're very much mixed into the evaluation system.

It's interesting to note that Clojure, for example, has no interpreter. It compiles everything into Java bytecode on the fly before it runs it. But it still has full-on lisp macros.

Brian: It would be interesting to see what the Scala folks think about macros

Yeah, I whined about them quite a bit in the Scala IRC channel. Somebody seemed to think someone was working on them. *shrug*

Brian: Runtime macros in C# 3.0

Interesting.

So the two drawbacks of C#'s solution (compared to Clojure) are (1) special syntax for calling a "macro" and (2) expansion happens at runtime. But it's a bit better off than say JavaScript which can't do C++ tricks because it has no pre-processor, and can't do C#'s tricks because it doesn't have access to parse trees.

Brian: That's what you really want -- some sort of access to the parse tree.

Right, access to the parse tree. exactly.

For the record, Clojure's built in or already does what we want, and is in fact implemented as a macro. You can find the real version in Clojure's boot.clj file. That one's already pretty simple, but here's a slightly simplified version that handles just the two-argument examples above:

    (defmacro my-or [a b] `(let [av# ~a] (if av# av# ~b)))

clojure is the best lisp yet

2008-02-13T07:48:00.000-08:00

I've dabbled in a few lisp dialects, but I generally become frustrated with some deficiency or other. I had been holding out some hope for Paul Graham's arc, but when I tried it out, I wasn't exactly blown away.

Common Lisp has a rather bloated library of functions, many with disgustingly long names. Arc has indeed successfully addressed this problem, and so has clojure: the longest function name I could find in Clojure's succinct and powerful library was the outlier "clear-agent-errors".

Lisp is old enough that many dialects have trouble with modern character encodings. Arc doesn't support anything but ASCII (so far anyway), and although elisp does it seems to regard UTF-8 with suspicion. Clojure supports UTF-8 out of the box, even inlined in your source code.

Many lisps have a shocking shortage of useful library modules for common tasks, especially related to networking or particular file formats, when compared to Perl, Python, etc., and creating bindings for existing C or other native libraries can be pretty tricky. Clojure allows you to directly use any part of any existing Java library, no special bindings needed.

With a large variety of libraries comes name collisions. Arc apparently hasn't addressed this problem yet, while clojure has all the namespace support you'll need.

JavaScript and other modern languages have demonstrated to me the power of hashes and the utility of being able to easily initialize them with literals in my code, yet few lisps make this convenient. Clojure has rich support for sorted maps, hash maps, and struct maps (which allow you to efficiently store a large number of hashs with common keys), including succinct syntax for creating new ones.

Besides cleaning up some of lisp's historic weaknesses, clojure provides several innovative new features. For example, the idea of data structure interfaces allows you to write code that can work the same on a variety of concrete types. The (conj ...) function adds an item to a collection, whether that collection is a list, a vector, or a hash.

I also found it interesting that Clojure's built in data structures are all immutable. The API makes working with these convenient, and should allow you to avoid common hassles when dealing with concurrency.

There are many other little details that clojure seems to have tweaked for the better compared to older lisps. The syntax for defining macros is a little tidier, (cond... ) is a bit more succinct, and it even has support for literal regular expressions.

I don't know that many lisps, and none of them very deeply. But for what it's worth, clojure is my favorite.

experimenting with vnc

2008-02-07T06:35:00.000-08:00

My home setup consists of a server in the basement with thin clients in the upstairs office. I love this setup because it means the office is entirely solid state; no fans or disks to make noise. It's the ultimate silent PC.

Method 1: XDMCP

I typically don't run applications on the thin clients, rather I run an X server with -broadcast to get an XDMCP session from my server. From that point on, everything (gnome, window manager, browser, xterms, etc.) runs on the server with $DISPLAY pointing at the thin client. This is pretty simple to set up, just change gdm.conf on the server:

--- gdm.conf.dpkg-dist  2007-05-29 05:08:37.000000000 -0400
+++ gdm.conf    2008-02-07 09:56:42.000000000 -0500
@@ -59,2 +59,3 @@
 [xdmcp]
+Enable=true
 
@@ -74,2 +75,3 @@
 [servers]
+0=inactive

and on the thin client, in /etc/rc.local:

X -broadcast

Method 2: VNC

Running X remotely, even on a 100 Mbit network, can get slow. In particular, Firefox just crawls. Making a new tab (blank page), chug chug. Rendering pages, chug chug. It's plenty usable, and you can get used to it, but it's a shock to go back to a normal PC and see the speedup that local rendering buys. (Xterms, btw, are plenty fast on remote X because they're just sending characters to the server, which it then renders locally.)

So I started experimenting with VNC. To my delight, VNC restores Firefox's rendering speed. In fact, VNC is fast all around. I see some very minor latency in my xterms since they're now sending graphics over the link instead of characters, but it's barely a price to pay for the overall speedup.

Here's what I did to make this work. First the server, I install apt-get install vnc4server, then create three entries in inetd.conf to correspond to the screens I have attached to the thin clients:

server # cat >> /etc/inetd.conf <<EOF
5910 stream tcp nowait nobody /usr/bin/Xvnc4 Xvnc4 -inetd \
  -depth 24 -geometry 1024x768 -fp tcp/oliva:7100 \
  -query localhost -once -securitytypes none
5912 stream tcp nowait nobody /usr/bin/Xvnc4 Xvnc4 -inetd \
  -depth 24 -geometry 1280x1024 -dpi 85 -fp tcp/oliva:7100 \
  -query localhost -once -securitytypes none
5919 stream tcp nowait nobody /usr/bin/Xvnc4 Xvnc4 -inetd \
  -depth 24 -geometry 1920x1200 -dpi 93 -fp tcp/oliva:7100 \
  -query localhost -once -securitytypes none
EOF
server # /etc/init.d/openbsd-inetd restart

On the thin client I apt-get install xvnc4viewer, then replace the -broadcast line in /etc/rc.local with:

# this corresponds to the 1920x1200 entry, suitable for a 24" LCD
xinit /usr/bin/xvnc4viewer -fullcolor -fullscreen server:5919

Still investigating...

Not everything is perfect using VNC. Here are a few problems I've run into:

caps-lock: I normally override caps-lock to be mod4 to control my window manager. It turns out that xvnc4viewer doesn't forward the caps-lock to Xvnc4, instead it's handled on the thin client's X server. That means that it can't be remapped inside the VNC session, instead I have to do it partly on the thin client: ssh t5710 "xmodmap -e 'remove lock = Caps_Lock' -e 'keysym Caps_Lock = Super_R' then inside the VNC session: xmodmap -e 'add mod4 = Super_L Super_R'.
mouse: My mouse has a bunch of buttons, which normally I can remap with xmodmap. With this setup, though, I only get 5 buttons (left, middle, right, scroll up, scroll down). I haven't figured out a workaround for this yet.

how to use a custom theme in rx4rdf

2007-07-06T17:30:00.000-07:00

I've started experimenting with rx4rdf, and although there are more useful things I could do first, I started trying to replace the default visual theme with a custom one just for n01se.net. After much stumbling around, I found what I needed to add to the site-config.py file in order for my own XSLT (really RxSLT) and CSS stylesheets to work. First I used __addItem__ to make rx4rdf aware of the two new stylesheet files, and how to treat each of them:

  __addItem__('n01senet-theme.xsl', loc="path:n01senet-theme.xsl",
    format='http://www.w3.org/1999/XSL/Transform',
    disposition='template'),

  __addItem__('n01senet-theme.css', loc="path:n01senet-theme.css",
    format='text', disposition='complete'),

The server will now expect to find those files in my site's "content" directory. Next I created a new "theme" made up of those two stylesheets:

  __addRxML__(replace = '@themes', contents = '''
  base:n01senet-theme:
    a: wiki:SiteTheme
    wiki:uses-css-stylesheet: base:n01senet-theme.css
    wiki:uses-site-template-stylesheet: base:n01senet-theme.xsl
    rdfs:comment: `custom theme for n01se.net
  ''')

Note by using "replace =" above, I disabled the three themes that ship with rx4rdf. That's ok with me, since I won't be needing them for this site. Finally, I configured my sitevars to point to the new theme:

  __addRxML__(replace = '@sitevars', contents = '''
  base:site-template:
    wiki:header-image: `n01senet-logo2.png
    #wiki:header-text: `n01se.net rocks!
    wiki:footer-text: `© 2007 n01se.net
    wiki:uses-theme: base:n01senet-theme
    #wiki:uses-skin:  base:skin-lightblue.css
  ''')

I also commented out the uses-skin predicate because I don't need another CSS file included. This may not be the best way to do this, but since I couldn't find any existing documentation on how it should be done, I thought it would be worth writing up what worked for me. I started off with n01senet-theme.css and .xsl files copied from rx4rdf-0.6.0/rhizome/themes/default/theme.css and .xsl files, and started modifying them to look like I want. I'm not done yet, but it looks like I'll have all the flexibility I need (a.k.a. enough rope to hang myself)

regex engine timing charts

2007-05-08T19:15:00.000-07:00

Charts by Google Spreadsheets (taller columns are worse):

regex engine timing

2007-05-08T11:04:00.000-07:00

Writing a regex-based sudoku solver with very little language-dependent code opened the door to benchmarking the various PCRE engines. Without much work, I was able to test perl, ruby, python and python-psyco.

Here are the results, unit is seconds:

                sudre.pl  sudre.rb  sudre.py  sudre.psyco
empty.input     0.012     0.028     0.316     0.240
full.input      0.032     0.076     0.312     0.264
level-1.input1  0.028     0.072     0.328     0.268
level-1.input2  0.028     0.068     0.328     0.268
level-1.input3  0.024     0.052     0.328     0.248
level-1.input4  0.008     0.020     0.316     0.240
level-1.input5  0.008     0.016     0.308     0.236
level-2.input1  0.012     0.016     0.320     0.228
level-2.input2  0.112     0.300     0.400     0.356
level-2.input3  0.172     0.464     0.440     0.412
level-2.input4  0.016     0.048     0.316     0.252
level-2.input5  0.064     0.184     0.340     0.288
level-3.input1  0.212     0.552     0.484     0.424
level-3.input2  0.028     0.072     0.336     0.244
level-3.input3  1.404     3.564     1.428     1.492
level-3.input4  0.072     0.172     0.360     0.296
level-3.input5  3.676     8.725     3.044     3.332
level-4.input1  0.612     1.504     0.840     0.776
level-4.input2  1.580     4.036     1.604     1.736
level-4.input3  0.076     0.220     0.332     0.316
level-4.input4  0.052     0.140     0.352     0.284
level-4.input5  0.076     0.196     0.368     0.292
                -----     -----     -----     -----
          mean  0.377     0.933     0.600     0.568

Some quick observations:

Perl is fastest on most of the puzzles, but python is far more consistent. My first guess is that python takes a lot longer to compile the regex, but may execute it faster after it's compiled.
Ruby-1.8 (current stable) crashes with a stack overflow trying to compile the regex. Ruby-1.9 was used for this test.
It would be interesting to make some graphs out of this... volunteers?

sudoku by regex

2007-05-03T09:03:00.000-07:00

About a year ago, I proposed a contest on #noise to write a sudoku solver. LIM wrote us a driver and generator. Chouser, Mr_Bones_, and owend wrote solvers. I started one in LISP but never finished it, to my shame...

One question that's always been in my mind is "How could we use a regex to solve a sudoku?" The main problem seems to be the input: Regular expressions are a language for searching, so you need the search space expressed in a way that the regex can process it. The naive approach expresses all the possible puzzles on the input, which is way too much data.

I usually think of a sudoku like this, where zeros are squares that need to be solved:

0 0 0 4 0 0 0 0 0
0 0 0 0 0 2 0 7 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 4 1 0
0 0 9 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 6 7
0 0 0 0 0 0 0 0 0
0 7 6 3 0 0 0 0 0

The problem with this representation is that the zeros aren't searchable. The regular expression can't tell that they mean "any number 1 through 9". So for a first step, my solver replaces them:

$puz =~ s{\d}{$& ? $&.'        ' : '123456789'}ge;

So the massaged input looks like this:

123456789 123456789 123456789 4         123456789 123456789 123456789 123456789 123456789
123456789 123456789 123456789 123456789 123456789 2         123456789 7         123456789
123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789
123456789 123456789 123456789 123456789 123456789 123456789 4         1         123456789
123456789 123456789 9         123456789 123456789 123456789 123456789 123456789 123456789
123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789
123456789 123456789 123456789 123456789 123456789 123456789 123456789 6         7
123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789 123456789
123456789 7         6         3         123456789 123456789 123456789 123456789 123456789

Now each cell contains the full list of possibilities for that cell. This is searchable. And with negative lookahead assertions, it doesn't have to waste nearly as much time backtracking.

The final output, after the second substitution, is:

9 8 7 4 6 5 3 2 1
6 5 4 1 3 2 9 7 8
3 2 1 9 8 7 6 5 4
8 6 5 7 9 3 4 1 2
7 4 9 8 2 1 5 3 6
2 1 3 6 5 4 7 8 9
5 3 8 2 4 9 1 6 7
1 9 2 5 7 6 8 4 3
4 7 6 3 1 8 2 9 5

Neat, eh? So finally, here are the two scripts implementing this solution. The first script is the solver itself. It accepts a puzzle on stdin formatted like the input above, converts it internally to the searchable format, and generates on stdout the solved puzzle. The second script generates the regular expression which is inserted into the first script.

By the way, while this isn't the fastest solver out there by any means (it's just brute force, after all), it solves most puzzles in approximate 0.01 seconds. That's 1/10 of the time genre.pl takes to generate the regular expression in the first place.

Scratching the mutt, part 2: Patch and configuration

2007-02-05T16:39:00.000-08:00

This is part 2 of a series that started here.

Before jumping into the patch and configuration details for using sendbox, let's take one step back and review the reasons for sending via IMAP. Sending email via IMAP gets mixed reviews in the community because it is non-standard, seldom supported by mail user agents (such as mutt), and supported by few IMAP servers (I'm only aware of Courier IMAP). The argument is that mail should be sent via SMTP and received via IMAP, and isn't that good enough?

I think there are good reasons to consider using IMAP for sending:

only a single connection to the server needed
avoid reconfiguring the local MTA to use the appropriate outgoing SMTP server to match my From address
avoid messing with SMTP-AUTH
avoid sending the message twice over the wire, i.e. once over SMTP and once over IMAP to put it in the "sent" folder
works well for offline situations, don't need to cook up separate offline methods for both IMAP and SMTP

For me, these reasons trump the naysayers' arguments. Overall, sending email via IMAP is more elegant than splitting the operations between IMAP and SMTP, and elegance in software attracts me irresistably.

Let's start with the patch then talk about configuration details to make use of it. After all, the patch introduces latent functionality that won't take effect unless you change your configuration. But configuration is useless without the patch.

Obtaining a patched mutt

Here are three options for obtaining a patched mutt:

If you're on Gentoo, use mercurial to fetch the n01se.net overlay and use emerge:

hg clone http://n01se.net/gentoo/overlay overlay-noise
export PORTDIR_OVERLAY="$PWD/overlay-noise $PORTDIR_OVERLAY"
emerge mutt

If you're on Debian, use my n01se.net apt repository:

echo deb http://n01se.net/agriffis/debian sid main >> /etc/apt/sources.list
echo deb-src http://n01se.net/agriffis/debian sid main >> /etc/apt/sources.list
apt-get update
apt-get install mutt

Apply the patches and build it yourself. This patches was made on 1.5.12 but applies cleanly and works on 1.5.13 too.
```
wget http://n01se.net/agriffis/mutt/mutt-1.5.12-sendbox.patch
```

Configuring your Outbox in mutt

There are three ways to use Courier's Outbox (if you learn of another IMAP server that supports the Outbox concept, I'd love to hear about it). The first way is to simply specify it in your configuration. For an IMAP configuration, that would be:

set sendbox=imaps://imap.domain.com/INBOX/Outbox
set use_sendbox=yes

This simplistic configuration works if the administrator of your mail server has enabled Outbox by default. Unfortunately this is seldom the case. More commonly, you'll need to the second way, which is to ssh into the server and start imapd yourself to enable Outbox. This is simpler than it sounds, just add to the above:

set tunnel="ssh -q server.domain.com   env OUTBOX=.Outbox SENDMAIL=/usr/sbin/sendmail /usr/sbin/courier-imapd .maildir"

To make this work with multiple servers, you can use account hooks.

If you're an offlineimap user, your offlineimaprc should have something like this:

[Account email]
localrepository = local
remoterepository = remote

[Repository local]
type = Maildir
localfolders = ~/mail
sep = .

[Repository remote]
type = IMAP
preauthtunnel = ssh -q server.domain.com   'env OUTBOX=.Outbox SENDMAIL=/usr/sbin/sendmail /usr/sbin/courier-imapd .maildir'
nametrans = lambda foldername: re.sub('^INBOX\.', '', foldername)

and your muttrc can then contain simply:

set sendbox=~/mail/Outbox
set use_sendbox=yes

Just don't forget to run offlineimap after sending mail from mutt, otherwise it won't leave your local Outbox!

Scratching the mutt, part 1: Introducing sendbox

2006-10-31T17:03:00.000-08:00

Two years ago I documented my configuration of mutt, offlineimap and courier imap. This combination gave me complete offline access (critical on a laptop) including folders and sending new messages. Additionally it scaled painlessly to multiple servers and clients, without my needing to think about keeping them in sync. Life was good, for a while...

More recently, I decided that I'd like to use mutt with IMAP directly on some client machines (workstations) and offlineimap on other machines (laptops). I rewrote my muttrc to be preprocessed with m4, and everything was rosy, except for sending via Outbox...

The method I was using depended on a helper script (mailout) which called safecat to deposit mail in the Outbox. That script no longer suffices, because it needs to save to IMAP rather than local maildir. Of course we want to use mutt's existing connection to the server rather than create another connection in the script! So I needed to find a way to make it work within mutt.

What started as an elegant hack quickly turned ugly. I tried setting $sendmail=/bin/true and used $record to fcc the message to the IMAP Outbox. This hack generally worked. The fcc-status patch is needed to prevent "Status: RO" (screws up the recipients that use mboxes; the mail shows up as already-read). Additionally mutt doesn't provide a method to fcc to more than one folder, so to get an fcc other than the Outbox, I used the write-fcc function in the compose menu. That was borked though because it lacks the Message-ID, encryption/signing, date, etc!

Moreover it all falls apart when bouncing (remailing) messages, because that code path calls mutt_invoke_sendmail directly, and of course no fcc is done.

Finally I realized that these hackish methods were going nowhere fast, so I implemented Outbox support directly in mutt. I call it "sendbox" because I think that's more explicit. Plus mutt already has an internal variable Outbox that corresponds to the $record setting, and I wanted to avoid confusion.

To use this feature, one does something like:

set sendbox=imaps://imap.domain.com/INBOX/Outbox
set use_sendbox=yes

...then all mail is sent by saving it to the Outbox instead of calling sendmail.

Part 2 in this series will explain this configuration in more detail and provide the patches to make it happen. Later parts will document some other interesting (at least to me) parts of my mutt setup.

JaM part 2: The token tree.

2006-10-18T19:21:00.000-07:00

This is part 2 of a series that started here.

First, a quick correction: I claimed that Lisp macros didn't have the ability change the basic syntax of Lisp. I believed that because I hadn't yet learned about "read" macros. Using "read" macros in Lisp you can indeed adjust some pretty basic syntax rules. That can't be done in JaM yet, but it's something that can probably be added in later.

So let's look more closely at the first stages of what JaM does. As stated previously, JaM.include("foo.js") fetches a JavaScript source file as text. It then passes that text to JaM.eval( ... ). The eval function does a lexical analysis of the text (currently using code from JSLint) to generate a JavaScript array of token objects.

A JaM token object is basically just a JSLint token object. For each word read from the JavaScript source, it contains the word itself, the line number and column where it was seen, and other bits of meta-data. Here's a couple of examples:

{
  value: "function",
  line: 99,
  character: 16,
  reserved: true,
  identifier: true
}

{
  value: "'bar'",
  line: 134,
  character: 32,
  type: '(string)'
}

I'm not sure if all this will be needed or useful in the long run, but JSLint produces it, and I have no good reason to throw any of it away. So far the "value" has been the only part that is has regularly been useful in the macros I've written.

Next, eval uses a few simple grouping rules to generate a tree of nested arrays of token objects. For example, the JavaScript expression "alert('hi!');" would be translated into the following tree:

  [
    { value: "alert", ... },
    [
      { value: "(", ... }
      [
        [
          { value: "'hi!'", ... }
        ]
      ],
      { value: ")", ... }
    ],
    { value: ";", ... }
  ]

These grouping rules have to be simple and loose because they must correctly parse not only normal JavaScript but also the code which will be the input to all our macros. (This is the stage where we could be using Lisp-like "read" macros instead of it being hardcoded as it is in JaM currently).

For example, if we plan to define a unless macro, JaM at this stage does not know what the word unless means, but it needs to generate an appropriate tree anyway. So for this expression:

  unless( false ) { alert( 'hi' ); }

JaM generates the tree below. For brevity I'll show just the value of each token object:

  [
    "unless",
    ["(",[["false"]],")"],
    ["{",
      [
        [
          "alert",
          ["(",[["'hi'"]],")"],
          ";"
        ],
        []
      ],
    "}"]
  ]

This nested tree of token objects is the data structure that JaM macros operate on. What exactly a macro might do with with this structure will be covered in the next installment.

JaM: Lisp-like macros for JavaScript, part 1

2006-10-10T12:30:00.000-07:00

A couple weeks ago I was reading Paul Graham's excellent book On Lisp, and I was struck with the thought that many of the features of Lisp he seems to really like are also provided by JavaScript. These include dynamic typing, lexical variable scoping, first-class functions, closures, etc. But macros are of course completely missing from JavaScript.

How hard would it be to add Lisp-like macros to JavaScript, I wondered? I'm learning about Lisp macros for the first time here, so I didn't really know, but I thought it would be fun to try.

This will be the first of several posts describing "JaM", my attempt to add Lisp-like macros to JavaScript.

It seems to me there a few essential ingredients to make a macro system for JavaScript that is as powerful as Lisp's:

Macros are written in JavaScript with access to all functions defined up to that point.
The source code that the macros read and write is in a format that is easy for JavaScript to manipulate, such as a JSON parse tree, not just a big string.

Number 1 is important, because otherwise you end up with something like C or C++ macros -- a weak mini-language that looks and acts differently than the host language. I suppose it's better than nothing, but it doesn't have nearly the language-building power that Lisp macros do.

As for number 2, I suppose it would be possible to set up a system where each macro could match based on a regex or something equally flexible, but this would have a couple of drawback. First, most macros wouldn't need that flexibility and would end up reimplementing the same patterns as other macros, including tricky things like matching up pairs of parens, curly braces, etc. Second, I don't want to try to read code that's being worked over by macros that are that flexible. I'd like to be able to assume that each macro is constrained in its scope, following roughly the same rules as the core JavaScript language. Lisp macros certainly have this restriction, so I'm not worried about this causing my macro system to be too weak.

Let's start off with a function JaM.include("foo.js"), which will fetch the JavaScript file, expand any macros in it, and then run it through the regular JavaScript eval(). Except it's not quite that simple because we want to be able to mix function definitions and macro definitions after each other in any order so they can build on each other. So we'll need to split "foo.js" into top-level expressions that can be correctly evaluated. I'll leave the details of this step for another time, but what's important is that for each top-level expression we will expand any macros and then evaluate it before moving on to the next expression.

This will probably be very slow -- certainly slower than just letting the JavaScript interpreter have at foo.js directly. But if that's the sort of thing that worries you, perhaps you'll be comforted by these thoughts: First, as described here the parsing, macro expansion, and separate evals happen once when a new source file is being loaded. Once the code is up and running there may be no need to do further macro expansion. Second, we could add another step to the processing of each top-level expression: after expanding any macros in an expression we could save the resulting code before evaluating it. Doing this with each expression in turn would result in a file full of fully-expanded normal JavaScript that browsers could run directly and at full speed.

Ok, that was pretty dull; perhaps I can whet your appetite with some sample code? Let's say you really like perl's unless keyword, and you wish JavaScript had something similar. With JaM, you can add it yourself:

defMacro unless( expr, block: body ) {
  return {{
    if( ! ( #expr ) ) {
      #body
    }
  }};
}

unless( true == false ) {
  alert( 'Hello, macro-enabled world!' );
}

It may not be the most useful example one can imagine, and even so I feel I've got a fair bit more to explain before it'll make much sense. We'll build our way up to defMacro, {{ ... }}, and the # operator over the next few posts. Hopefully, though, all of the above will be sufficient for you to understand where I'm headed.

encrypted /home

2006-07-17T07:57:00.000-07:00

I've done this now on a few systems, mostly laptops. My reasoning is that any data I care about (particularly email) are stored in my home directory. If the laptop were to get out of my possession, it would likely be shut off at the time, and I don't use suspend/resume. So encrypting /home suffices to protect me if the laptop is lost or stolen. It's also easier and more performance-friendly than encrypting the root dir or swap.

When I installed my current laptop, I knew I'd encrypt /home eventually so I left a spare partition, hda3. If you don't have one, you'll need to shrink a filesystem to create a new partition or logical volume. When you're ready with that, the next step is to create the encrypted mapping.

modprobe aes # or aes-i586 on x86 systems
emerge cryptsetup-luks
cryptsetup --verify-passphrase luksFormat /dev/hda3
cryptsetup luksOpen /dev/hda3 home
mkfs.ext3 -j -m 1 -O dir_index,filetype,sparse_super /dev/mapper/home

Next copy the existing /home to the encrypted filesystem. Be sure not to alter any important data in /home since you'll lose any changes made after the copy. You can enforce this by switching to single user mode but I don't bother personally.

mkdir /mnt/newhome
mount /dev/mapper/home /mnt/newhome
cp -ax /home/. /mnt/newhome
umount /mnt/newhome

Next let Gentoo know about it, then reboot to start using the encrypted /home:

printf 'mount=home\nsource=/dev/hda3\ntype=luks\n' >> /etc/conf.d/cryptfs
echo '/dev/mapper/home /home ext3 noatime 0 0' >> /etc/fstab
echo 'aes' >> /etc/modules.autoload.d/kernel-2.6  # or aes-i586
mv /home /oldhome
mkdir /home
reboot

Finally, when you've verified that it's working, be sure to remove the old home which still contains unencrypted data!

rm -rf /oldhome

md + lvm2

2006-06-26T13:21:00.000-07:00

It's just beautiful what happens when you mix Linux md with lvm2. I just added two 15K 146G disks to my system. They show up as sdc and sdd. Now I do this:

for x in sdc sdd; do
    parted /dev/$x mklabel msdos mkpart primary 0 100% set 1 raid on print quit
done
cd /dev; MAKEDEV md
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdc1 /dev/sdd1
pvcreate /dev/md0
vgcreate raid1 /dev/md0
for x in agriffis amg; do
    lvcreate -L 50G -n $x raid1
    mkfs.ext3 -L $x -O dir_index /dev/raid1/$x
    echo "LABEL=$x /home/$x ext3 noatime 1 2" >>/etc/fstab
done
mount -a

That leaves nearly 50G of free space on the physical volume into which I can grow either filesystem eventually, or add a third, depending on future needs.

Commentify contest

2006-06-12T13:53:00.000-07:00

A little while back (in April), we had a contest on #noise that started with David's post regarding a script to comment out words in a C file. The general applicability of this script is questionable, but it was still a fun contest. Here are the instructions:

Write a command-line filter which will comment out all matches of a pattern given on the command-line, reading a C program from stdin and filtering to stdout. Whether the pattern is a literal, glob, regular expression is up to the contestant.
The filter should ignore matches that are already contained within a comment (including C++ style comments) or double-quoted string.

The existing gawk implementation on David's blog is an interesting starting point, though it doesn't fulfill all the requirements above.

Sample I/O:

simple:input /output ("add")
wacky:input /output ("wacky")
deranged:input /output ("int")

The final results are in now! In order of longest to shortest:

http://n01se.net/paste/9Cc (owen, awk, 169 chars)
http://n01se.net/paste/MX (chouser, perl, 91 chars)
http://n01se.net/paste/Lf (bob, perl, 91 chars)
http://n01se.net/paste/OI (agriffis, perl, 85 chars)
http://n01se.net/paste/Kf (aaron+kanaka, sed+cpp, 52 chars)

Placeholder post

2006-05-20T15:04:00.000-07:00

Blogger requires me to make a post before I can preview the template or subscribe to the RSS feed. Is this fair enough or lame to the max? Discuss...