Corked!

At the time of writing, my blog suddenly got very ugly… That’s because I’m in the process of switching to a static site generator I’ve written from scratch, called cork.

Cork is written in Coffeescript and of course runs in Node.js. The idea is to have a pluggable static site generator that is itself opinionated about what makes up a “site” (static pages, blog posts/archives/category lists, etc), but then defers to plugins to discover the content in the site directory, render it, lay it out, and deploy it. The goal is to have a site generator that makes a “better” trade-off when it comes to opionated (and thus convenient if opionions align with end-user) vs flexible.

Why? Well, I tried Octopress first. Octopress is fantastic, but very much at the extreme end of opinionated, as soon as I went against the grain, I was frequently frustrated and spent a lot of time Googling stuff. For example, at the time of research, using LESS in Octopress was a real pain, it couldn’t be plugged in to the rendering pipeline. I also tried Docpad, but it’s too flexible and requires a lot of work in layout code to query content and pull together a standard personal site + blog.

So, right now you’re looking at my new blog, generated using Cork and deployed to S3, and then fronted by Cloudfront. Excuse the ugliness, I’ll be working on the stylish touches when I’ve nailed down all the functionality I need.

Fun With Polymaps

Recently I had a few ideas I wanted to try out in geographical data visualization. I’ve never really done any data vis stuff, other than a few charts here and there, so I had no clue where to even begin.

At Wotif, I’ve been heavily involved in the Reviews system, which went live a couple of months ago. What I wanted to do was color a world map to signify the density of Reviews for hotels in each country. After some pathetic Google-fu, I ran into the Geochart visualization from Google Chart Tools. The Geochart visualization was a good start, but I quickly discovered just how limited it was when it comes to client-side interaction. I was sure there must be something more, but having never had much to do with mapping on the interwebz, I could not even begin to figure out the question I needed to ask, let alone the answer.

Thankfully, I discovered that what I was trying to display was a Choropleth Map. Once I knew this, it didn’t take long before I stumbled upon this fantastic blog post from someone trying to do exactly the same thing as I. It was through this article that I had my eyes opened to a whole world of amazing mapping tools and libraries. The one that struck me as the most interesting was none other than Polymaps, primarily because of the way it leveraged SVG for some pretty sexy looking “slippy” maps.

Even once I’d gotten my foothold in this wealth of information, I was still pretty lost. Answering my initial question lead to a myriad more. What is GeoJSON? How do tiles work? Where does a Tile Cache come into play in all this? Wading through the mass amounts of information was pretty daunting.

In the end I focussed back in on what I was trying to do. I wanted a simplistic world map that I could mutate and accept user input on. Poring over the Polymaps examples, I determined that GeoJSON would help me render an world map with country outlines. A bit more Googling and I found a fantastic Github repo - johan/world.geo.json, with all countries in a GeoJSON file.

Armed with this GeoJSON file, I was easily able to fumble about and mash my face against the keyboard until I got the desired result, a nice Choropleth map that users could mouseover to see more information! I bundled this up in a nice little dashboard with some other statistics I cooked up. I’ve since shown it to a few people at Wotif and they seem to like it.

The initial experiment only whetted my appetite though, and I’m finding myself quite engrossed in the world of amateur cartography. I’ve since started cooking up some interesting real-time displays using Polymaps, overlaying SVG drawing onto the map and doing some fun stuff with it, like smooth panning/zooming between points and extents. I will blog more about some of the cool stuff I’ve come up with.

Bye internets!

Switching From Gnome 3 to MATE

So my Linux distro of choice for quite a while has been Ubuntu. Unfortunately it seems that lately Canonical has embarked on a mission to destroy the desktop experience with religious fervor. I have no qualms in saying that I think Unity is the worst thing to ever happen to Ubuntu - it’s a complete unmitigated disaster. How it managed to become the default overnight without rioting in the streets is beyond me - but I digress.

Of course there are other alternatives in the Desktop Linux scene; offerings such as XFCE, LXDE and KDE all have their place. I personally don’t have the time to familiarize myself with yet another desktop environment, especially given that my goal is to be running a Mac at work and at home. I had grown quite fond of GNOME 2 in previous releases of Ubuntu, so my next logical step was to use GNOME 3.

The problem with GNOME 3 is that of immaturity. It’s still pretty rough around the edges (it crashes at least a couple of times a day for me under regular use), has very poor display driver support, and doesn’t quite have the community momentum for themes and plugins like its predecessor had.

I’ve been using GNOME 3 for the past few months (at work and home) with the hopes that my gripes with it would be addressed shortly but unfortunately I’m yet to observe even incremental improvements in stability or performance. My work machine (which I’m using 8 hours a day) constantly needs gnome-shell restarts, and sometimes it locks up so bad I actually have to kill the whole session and lose everything I had open. Recently my patience has worn thin enough that I started conspiring to try something (anything, really) new on my machine at work. The proverbial straw was broken when somehow my NVIDIA display drivers (custom installed in order to address a host of issues with gnome-shell) was broken so badly by an Ubuntu update that I had to boot into recovery to fix it. But wouldn’t you know it, recovery console isn’t working with my setup for some reason - it wouldn’t detect my keyboard!

I had read about the MATE project on the Linux Mint blog a few months back, so today I wondered how it had come along. The prospect of a pure GNOME 2 environment running in the latest Ubuntu sounded very promising indeed. I decided it was finally time to make the jump.

With a small amount of tinkering, I had a fully functional MATE session up and running in Ubuntu 11.10. It took a whopping 10 minutes to get MATE fully operational. All I had to do was the following:

  • sudo bash -c ‘echo “deb http://tridex.net/repo/ubuntu/ oneiric main” >> /etc/apt/sources.list’

  • sudo apt-get update

  • sudo apt-get install mate-archive-keyring

  • sudo apt-get install mate-core

  • Logout, select MATE from the session list and log back in

The thing that struck me immediately is HOW INSANELY FAST IT IS. Especially compared to GNOME 3/Unity. After getting Compiz up and running and using the desktop for an hour or two, I suddenly realized just how sorely I’d missed the polished and refined awesomeness that is GNO- er, MATE.

I am in the process of installing MATE on my home machine now so I can get as much exposure to MATE as possible over the coming weeks. I will be documenting my experiences here, and sharing all the gotchas I find as I go.

Glassfish 2 & Grails 2 Logging

Grails 2 is now released, and no doubt many people will be upgrading from 1.3.7 shortly. This blog post serves as a quick HOWTO on a fairly common issue involving deploying a Grails 2 app to a Glassfish 2 container. That issue is getting logging working correctly.

A quick rundown on how Grails 2 logging works:

  • SLF4J is used to handle logging

  • By default Grails uses Log4j as the logging provider.

  • Jakarta Commons Logging (JCL) is redirected to SLF4J using jcl-over-slf4j

  • java.util.logging is redirected to SLF4J in dev mode, but disabled in production environment by default

With this default setup, you’ll quickly discover that Grails logging ends up in standard out in Glassfish, which means that you can’t use Glassfish admin to configure log levels at all with your deployed application. This is obviously not an ideal situation.

Glassfish only supports java.util.logging, so we need to tell Grails to log to this in order to play nicely with Glassfish.

Previously in Grails 1.3.7, we were using the technique Reiney Sadder outlined in his excellent blog post to remove log4j and install slf4j-jdk14 to redirect all logging to SLF4J into JUL. Unfortunately this technique no longer works in Grails 2, as Log4j support in Grails no longer comes from slf4j-log4j, but rather a core Grails plugin called “grails-plugin-log4j”.

So, how do we achieve this in Grails 2? Simple! Just add the following to your BuildConfig.groovy:

1
2
3
4
5
6
7
8
9
10
11
12
13
// Inside your grails.project.dependency.resolution closure:

inherits("global") {
  if(Environment.current == Environment.PRODUCTION) {
      excludes "grails-plugin-log4j", "log4j"
  }
}

dependencies {
  if(Environment.current == Environment.PRODUCTION) {
      runtime "org.slf4j:slf4j-jdk14:1.6.4"
  }
}

This will disable log4j entirely in production environment and add the slf4j-jdk14 JUL bridge.

NOTE: Make sure you definitely have the following line added to your Config.groovy:

1
grails.logging.jul.usebridge = false

You really should have this disabled for production anyway, as it’s not very performant.

The BuildConfig.groovy change above completely disable log4j, so any log settings you have defined using the Log4j DSL in Config.groovy will not be recognized at all. This is why I only enabled it in production environment, it’s still nice to be able to easily tweak logging settings for your development environment. If you really want to though, you can configure your own JUL logging.properties and just enable the SLF4J JUL bridge permanently.

Have fun!

Inotify + Node + FTP = Easy-mode Remote Dev Work

Pre-ramble

Haven’t updated my blog in a while, but I had a fun little win tonight that I thought I’d share with the lovely readers of this blog (you know I love all 3 of you).

I am currently working fulltime at Wotif.com in Brisbane, and I’m extremely fortunate to be immersed in the weird and wondrous world of Groovy/Grails for the majority of application development I’m involved in. To make things even more awesome-er we’re currently working a on a project that uses CouchDB/ElasticSearch as our primary data provider. Life is bliss. Well, mostly.

The Problem

I still have work come in occasionally from a long-time client I respect enough to give some time to on the weekend. Unfortunately this work is in the form of PHP4 code most of the time. And no, the pain doesn’t just end there, usually the work is on a fairly large production Joomla 1.0 site. Needless to say, the technical debt flows freely.

Anyway, this particular site is a real pain to work on, as it’s not really in a state where it can be run up locally without a fair amount of pain and misery. I haven’t done much work for this particular client in the last few months, and thus I didn’t really have a proper LAMP stack running on my machine at home (nor do I really want to). A more recent project I’d done for this client last year was a Joomla 1.5 site, where I had the luxury of “doing it right” - I had set the project up to easily be run up locally using some bash scripts, UnionFS, and a little bit of voodoo. But no such win was to be had for this Joomla 1.0 site.

The work I needed to perform on the Joomla 1.0 site was considerable enough that the prospect of firing up Filezilla and manually FTPing changes was unbearable. In the past I’ve used Eclipse with an obscure plugin called ESFTP to push my changes to the server as I develop. However this still has an obnoxious required manual step of clicking a button every time I want to push a file to the server.

I figured there had to be an easier way. Then I remembered seeing some cool inotify stuff in Node.js a while ago.

The Solution

I thought to myself: “Wouldn’t it be cool if I had a little Node app running that monitored my project on the filesystem and FTP’d changes to the codebase as I made them?”.

So I decided to cook something up. A couple of hours later I came up with this:

I didn’t end up using the libinotify bindings for Node.js, as it was a little too low level for a quick prototype. The main pain point was the fact that inotify isn’t actually recursive, so you actually have to put together your own code that recursively creates watch descriptors for the directory structure, glue in new watch descriptors as new directories get created, and delete old descriptors as directories disappear. I instead opted to use the awesome inotifywait tool (which comes from the inotify-tools package in Ubuntu) which handles all the un-fun parts of inotify and instead sends nice little status updates on stdout.

Oh, and I was getting bizzare issues with the node-ftp library from NPM, so I just grabbed the latest from the git repo and threw it in with the script.

So now I just fire up this script with the FTP details and paths. It just sits there patiently and creates/deletes directories as needed, and pushes file changes/deletions as they occur.

Now this could definitely have easily been done using pretty much any language, but I think it’s a pretty neat and elegant little CoffeeScript/Node solution :)

Speaking of Node/CoffeeScript, I’ve been working on a little project in all the spare time I can get. I’m excited to blog about some of the cool stuff I’ve found/done in that regard in the coming weeks!

That’s all for now, internets.

Listening for End of Response With Node/Express.JS

I’m currently working with CoffeeScript, Node, Express, and Redis to deliver on a quick’n’easy contract I’ve been put in charge of. This is the first time I’ve used any of these technologies in a proper commercial deliverables type project, and I have to say, it’s been an absolute delight.

An issue I ran into was I wanted to reduce the boilerplate on handling requests, so I wrote a quick route middleware in Express to create a client connection to Redis, assigning the connection to the request object for easy use . I also wanted to be clever and have this same middleware clean up after itself when the request ended. That is, I wanted the middleware to QUIT the Redis connection when the response had been sent.

Consulting the Express/Connect/Node docs yielded no clues as to how to do this, the closest hint I found was from the Node docs indicating that a http.ServerResponse is a WritableStream. I noticed WritableStreams had a “close” event that is supposed to be called when the Stream is no longer writable. I assumed that if you call .end() on a response then it should trigger this event, so my initial middleware looked like this:

1
2
3
4
5
6
7
8
9
10
11
setupRedisClient = (req, res, next) =>
  req.redisClient = require("redis").createClient()

  cleanup = =>
      console.log "it worked!"
      req.redisClient.quit()

  res.on "close", cleanup
  res.on "error", cleanup

  next()

I tried using this middleware in a route, and was sad to see that the event was not being triggered.

As a last resort I started digging through the Node source, and lo and behold! I found what I was looking for in lib/http.js. Turns out when you .end() your http response, it will emit a “finish” event.

Now my Redis middleware looks like so:

1
2
3
4
5
6
7
8
9
10
setupRedisClient = (req, res, next) =>
  req.redisClient = require("redis").createClient()

  cleanup = =>
      req.redisClient.quit()

  res.on "finish", cleanup
  res.on "error", cleanup

  next()

Incoming routes that need a Redis connection simply add this middleware, and hey presto! They can use req.redisClient to their hearts content. Once a response is sent back to the client, or an error occurs with the request, the Redis connection will be cleaned up automagically! Hurrah!

Facebook Open Graph HTML5 Validation Problems

I’m currently working on a site currently that uses a few Facebook “Like” buttons scattered around the place. As such the pages that can be “Liked” need a modest amount of Open Graph metadata embedded in them.

The fun part of this is that the Open Graph specification states that OG metadata should be embedded on the page like so:

1
2
3
4
<meta property="og:title" content="The Rock" />
<meta property="og:type" content="movie" />
<meta property="og:url" content="http://www.imdb.com/title/tt0117500/" />
<meta property="og:image" content="http://ia.media-imdb.com/images/rock.jpg" />

The stupid thing is this is actually invalid HTML5. The HTML5 spec states that meta tags must use the name/content attributes, and because OG is based off RDFa, an XHTML draft, OG mandates that its metadata uses the property attribute instead of the name attribute. After Googling around a bit I came to the “conclusion” that the only way to get a page to validate with Open Graph tags was to switch to XHTML.

Screw that.

Instead I tried just changing the property attributes back to name and run it through the Facebook URL Linter tool. Turns out, Facebook will grumble at you, but they will still parse OG data out of standard meta tags, and Like Buttons will use that metadata just fine.

So, in summary, if you’re authoring a straight HTML5 (non-XHTML) site, you have a choice between pissing off W3C, or pissing off Facebook. It’s your call.

I’ve put up two examples for reference, one uses meta property (non-standard), the other uses meta name (standard).

No.de Coupon

Huzzah!

Got my no.de coupon in the mail a couple of days ago.

Now I just gotta figure out what I wanna host at http://sammeh.no.de/….

Creating a Proper Buffer in a Node C++ Addon

Despite the wordy title, it’s actually a fairly simple problem, with a fairly simple solution.

Let’s say you have some binary data you want to provide to Node Javascript. No problem, Node has Buffers for that. Digging through the Node.js source code, you find node_buffer.h, which promises a utopia of an ObjectWrap goodness; you can even memcpy your binary data directly to it using Buffer::Data(bufferObject).

“Fantastic! I’ll rock one of those buffers and simply return bufferObject->handle_!”, I hear you exclaim. Not so fast stud.

If the client were to use this Buffer, they’d get a nasty surprise. It’s not a Buffer. You see, Node.js has re-implemented Buffers since 0.2. The Buffer you’re playing with from node_buffer.h is actually a SlowBuffer. As the name implies, it’s working directly on the heap-allocated memory chunk, so alot of operations on it are quite inefficient. Worse still, the interface provided on SlowBuffer is actually different to the Node.js documentation. Allow me to explain.

The Buffer you’re used to dealing with from Node.js user code actually originates from buffer.js. These Buffers are actually just “views” on a proper SlowBuffer, so operations like slicing are literally as quick as allocating a new Buffer object that views the SlowBuffer at a different offset and max length.

So how do you create one of these badboys from C++ to pass directly back to JS calling code? Glad you asked. Like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
// Some data we want to provide to Node.js userland code.
// This can be binary of course.
const char *data = "Hello world!";
int length = strlen(data);

// This is Buffer that actually makes heap-allocated raw binary available
// to userland code.
node::Buffer *slowBuffer = node::Buffer::New(length);

// Buffer:Data gives us a yummy void* pointer to play with to our hearts
// content.
memcpy(node::Buffer::Data(slowBuffer), data, length);

// Now we need to create the JS version of the Buffer I was telling you about.
// To do that we need to actually pull it from the execution context.
// First step is to get a handle to the global object.
v8::Local<v8::Object> globalObj = v8::Context::GetCurrent()->Global();

// Now we need to grab the Buffer constructor function.
v8::Local<v8::Function> bufferConstructor = v8::Local<v8::Function>::Cast(globalObj->Get(v8::String::New("Buffer")));

// Great. We can use this constructor function to allocate new Buffers.
// Let's do that now. First we need to provide the correct arguments.
// First argument is the JS object Handle for the SlowBuffer.
// Second arg is the length of the SlowBuffer.
// Third arg is the offset in the SlowBuffer we want the .. "Fast"Buffer to start at.
v8::Handle<v8::Value> constructorArgs[3] = { slowBuffer->handle_, v8::Integer::New(length), v8::Integer::New(0) };

// Now we have our constructor, and our constructor args. Let's create the 
// damn Buffer already!
v8::Local<v8::Object> actualBuffer = bufferConstructor->NewInstance(3, constructorArgs);

// This Buffer can now be provided to the calling JS code as easy as this:
return scope.Close(actualBuffer);

And that’s all folks!

Node.js

Since I haven’t updated my blog for a few months I figure now would be a good time to do a bit of a brain dump on what is interesting to me nowadays.

Currently I’m completely immersed in the weird and wondrous world of Node.js. If you are even remotely interested in anything related to web development/engineering, you should already know about Node. Briefly, it’s a server side JavaScript (SSJS) implementation built on top of Google’s V8 JavaScript engine. It’s blisteringly fast, and has already been employed in some big projects to solve some pretty insane scaling problems engineers are facing in large websites.

Currently I’m just getting myself acquainted with Node.js, I’ve written some random libs that are on my Github. Currently I’m writing a native Node extension called node-gitteh, which provides bindings to the excellent C library libgit2. I’ll be using these bindings to manipulate Git repositories from Node as part of a little project I’m going to undertake (more on that later).

Writing these bindings has been interesting, given that I’m writing C++ code for the first time in years, and having more trouble remembering how to use an STL map<> than I am wrangling the bizarro V8 API. I think this definitely warrants a tip of the hat to Google, the internals of V8 are pretty accessible; my only gripe with V8 is a pretty painful lack of hand-holding documentation, however there were plenty of examples of stuff on Github from other kindred Node spirits who’ve written bindings for things like GD, Mysql, libxml2 and the like.

The thing that has impressed me about Node the most is the amount of talent the community is comprised of. Node is only just over a year old and there is already a mass of quality libraries and frameworks. One I particularly love is Vows, which has made TDD an absolute breeze in Node. If you’re just starting out with Node, I thoroughly recommend you get into the habit of using Vows to test your code, ideally writing the tests before you even launch into your next Javascripty wet dream. Seriously, it’s worth it.

Of course, Node is not without it’s faults. The biggest one currently is lack of threaded-ness, or any kind of concurrency control from JavaScript. Many will argue this is a Good Thing, as it abstracts away the misery that is semaphores, locks, re-entrancy and other goodies that come with thread-safety. However I still think there should be first-class support for concurrent operations in Node user land (read: JavaScript code).

There are some solutions out there that utilize nodes ability to spawn child processes and communicate/control them effectively, to the effect of running a pool of Node processes. However I view these as a kludge, as Node processes do have a pretty decent memory footprint on initialization. Given that Chrome has a way of running completely sandboxes JavaScript execution contexts in parallel (that is, a blocking script in one frame wouldn’t block other frames), I’m sure there’s an elegant solution to be found.

That’s all for now!