samcday

So I’ve been using Amazon Web Services Elastic Beanstalk for a few PHP and Java apps. I’ve decided I definitely don’t like it. Here’s why.

Well actually, a quick bit of background. I love how straightforward and powerful Heroku is (and I can maybe even forgive them for that whole dyno routing debacle), but the dealbreaker for me is it being US-based. I live in Australia, and while our routing to the East Coast isn’t terrible, it’s still not ideal to serve a predominantly Australian website from there.

Small history lesson aside, here are the main reasons why I dislike Elastic Beanstalk.

Slow

Okay okay, this might be quibbling a bit, because any kind of orchestrated deployment platform like Beanstalk/Heroku is probably much faster than even the most experienced sysadmin could run up a LAMP stack, configure it, and deploy your crappy PHP app to. That said, Beanstalk is quite painfully slow to start up a new app, or make changes to, especially when you compare it to Heroku.

Weird

Vague title yes, but allow me to explain. Elastic Beanstalk does things I just don’t understand:

• Destroys and recreates EC2 images unnecessarily. Why do I need a whole new instance when I simply change a security group?
• Why does git aws.push repush the entire repo if there was no changes since last push?

Some of these might just be me doing things wrong, but that ties in well with my next point, and it’s the big one.

TERRIBLE Documentation

Oh my GOD. AWS has notoriously bad documentation in general, but Elastic Beanstalk HAS to be an elaborate joke Amazon is playing on the world. What little documentation there actually is on entire swathes of functionality provided by Elastic Beanstalk is chaotically organized. There’s two different command line toolsets available to interact with Elastic Beanstalk. The newer one has tragic documentation coverage, but is quite powerful when you scrounge enough information on how to use it.

I really cannot overstate how poor the Elastic Beanstalk documentation is. It’s extremely frustrating as I’ve been spoiled by fairly excellent documentation resources on the internets. Again, contrast Elastic Beanstalk with Heroku on the documentation front - there’s really no comparison.

Okay, Some Positives

So as much as Elastic Beanstalk can be frustrating, it does have some positive aspects. It’s a great way to deploy applications to AWS, when comparing to tedious approaches like manually setting up an AWS environment, or using complex solutions like Netflix Asgard, or CloudFormation. Also, it really is the only decent PaaS available in Australia.

I have noticed that Elastic Beanstalk has been receiving fairly frequent updates and improvements, so perhaps the shortcomings in documentation for it could be attributed to the velocity in which the core service is being iterated on.

/rant

As much as I don’t like Elastic Beanstalk in its current state, I will continue to use it for applications I develop/maintain that have a predominantly Australian audience. I will cross my fingers and hope that Elastic Beanstalk service and documentation improve over the coming months, or that AWS OpsWorks evolves to be a better alternative.

If you’re trying to configure SSSD and you get an error message in your syslog that kinda looks like this:

May  3 00:55:51 www1 sssd: Cannot load configuration database

… Then you just need to make sure your sssd.conf file has a newline at the end. Not even kidding.

echo >> /etc/sssd/sssd.conf 

I had this particular issue on Ubuntu 12.04 Server. Might be specific to that distro, although I did find this sssd ticket that seems to indicate it was an upstream issue, but then again it was closed as invalid. So I dunno. The invalid resolution fixed the issue for me. YMMV.

If you’re using MATE Desktop in a virtual machine with VMware, you might notice that when you open the Unity Applications menu on your host, you get an empty list.

Luckily, there’s an easy fix.

1
2
3
4
5

cat >> ~/.profile <<MATEFIX
# Fix VMware Unity mode for MATE.
XDG_CURRENT_DESKTOP=GNOME
export XDG_CURRENT_DESKTOP
MATEFIX

Run the above, log out and back in to your desktop, and re-enter VMware Unity. Ta-da!

The issue is that a VMware tools utility, /usr/bin/vmware-xdg-detect-de, does not recognize MATE (even though MATE is just GNOME2), so by setting the environment variable XDG_CURRENT_DESKTOP in your .profile, you force VMware to recognize your desktop as GNOME.

I’ve always wished I could be better at design. Growing up, my artistic style extended as far as drawing mazes and stick figures - most of the latter looking like they had been horribly deformed by a tragic car accident.

The thing is though, I have always been painfully aware that creating something interesting, something people can get excited about, something people will show to their friends, is just as much about the design as it is the functionality. I had this revelation very early on in my career, but I’ve never had the patience to learn enough about design to apply it practically to things I create, in either my career or my personal endeavours.

I’m trying to change that now though, and with amazing resources like Hack Design, I may actually have a shot at learning a thing or two about design.

Kerning?

Tonight I was going through the second lesson on typography, and came across the fascinating Kerning Game site. It presents a few simple exercises, in which words of differing serif/sans serif fonts have had their kerning messed up, and your task is to restore them to their optimal spacing. I’m usually terrified of trying stuff like this, as I don’t like being reminded of things I’m not good at, but I decided to give it a go.

As it would turn out, I’m a little better at kerning type than I am at drawing stick figures, as I managed to score 93/100 overall for the 10 exercises presented! I’m not sure if this site was sneakily placed in the Hack Design course to give an early morale booster, but hidden agendas be damned! I’m feeling really encouraged and am going to stick at this design thing!

Google’s Go programming language, while not new, has been making the rounds on Hacker News headlines quite alot lately. I’ve been wanting to try a new language, but was put off by the not-quite-1.0-yet status of Rust, and Scala/Clojure didn’t quite pique my interest enough to sit down and give them a proper go.

Other than a few testimonials from companies like Airbrake/Cloudflare/DNSimple saying that Go is awesome for use case X and blah blah blah … I didn’t really know what the fuss was all about with Go, other than it’s good for concurrency, and had some interesting ideas around syntax and dependency management.

I finally took the plunge and went through the Go tour and I’m definitely impressed. Go-lang is a pretty freakin’ sweet language!

Some reasons why it’s awesome include:

• Static compilation - no more pip/bundle/npm package management issues. No more juggling the correct version of a runtime, Go compiles ALL OF THE THINGS into a single static binary.
• Syntax - apart from the backward-ness in variable definitions, Go syntax is very clean and very readable.
• Code style - Go ships with gofmt, which means programmers no longer have to quibble over code style nonsense, the language designers decided on it for you.
• Sweet tooling - the go compiler suite comes with lots of awesome functionality straight out of the box.
• Concurrency - I didn’t know what CSP was before I picked up Go, but I would definitely choose goroutines and channels over Java java.util.concurrent / Node callback-soup / Node fibers any day.

Some parts about Go that I’m concerned about, or don’t fully understand:

• Dependency versioning - while Go has a very opionated appreoach to dependency management, it also sports a very rudimentary package manager of sorts (I consider go get package management, feel free to tell me I’m wrong), it seems that handling dependency versioning is a bit dodgy.
• Runtime performance - Go site itself mentions that its still early days for its garbage collection and allocator. That said many benchmarks are looking very promising for Go 1.0, and also much better for Go 1.1.
• Package maturity - I’ve learnt this one from personal experience - you need to be extemely cautious about which 3rd-party packages you pull in for an immature programming language. I started using Node.js as an early-adopter, and while NPM is exploding with amazing packages now, early on there were a lot of poorly written libraries, and worse, some libraries written in C that had all kinds of concurrency issues. Since Go’s recommended approach is to re-implement all of the things in pure Go, I suspect that many people will find themselves re-implementing/porting stuff all over the place.

With that said, I haven’t actually had a chance to use Go in anger on something interesting yet, though I do intend to rectify that ASAP. I look forward to open sourcing something interesting in Go on my github profile in the near future!

So you have a Unix-based host sitting there. It’s got some files on it. You wanna edit those files. Let’s pretend you’re not editing config files though, because you should be using Git/Puppet/Chef for that. Instead, I’m assuming you’re wanting to maybe make a few adhoc changes somewhere in the filesystem. I’m also assuming it’s no longer 1994 and you don’t use FTP for productivity/development out of principle.

Enter sshfs. Briefly, SSHFS is a FUSE driver that allows you to mount a remote filesystem locally, using SSH as the transport. The benefits of this should be immediately obvious:

• Instantly secure.
• Ubiquitous support - only requires OpenSSH server on the remote end.
• Easy to setup - in most cases there won’t be an iota of configuration required if you’ve already got your pubkey on the remote.

Installing sshfs is straightward enough on Ubuntu:

1

(sudo) apt-get install sshfs

Mounting a remote location is also easy as pie:

1
2
3
4
5

mkdir -p /local/mount/point
sshfs remoteuser@remote.host:/remote/location /local/mount/point
ls /local/mount/point

# zomgawd the remote filesystem location lists on your local mount point how does that even oh my god what

Bonus points

So the above is all well and good for most use cases, but there’s a caveat. What if you want to edit files as a specific user on the remote host, but you either don’t have direct ssh access to that user, or it doesn’t have a shell (the latter would be more likely if you were editing system config files, but we already established you’re not doing that)? Turns out this is something that sshfs can handle, too!

What you can do is override some sshfs settings and escalate your privileges to the other user when connecting to the remote end. That way, whenever you make changes locally, the changes are applied using the correct user on the remote end. This avoids permission issues, and also means new files will be created with the correct user!

In order for this to work, you’ll need to have sudo permissions with a user on the remote host. Specifically, you’ll need:

• privileges that DON’T require a TTY, or a password.
• the path to the sftp-server binary on your remote host.

Determining the sftp-server path is up to you, as a hint though, you’ll probably find it in the following locations…

Ubuntu:

/usr/lib/openssh/sftp-server

CentOS 5:

/usr/libexec/openssh/sftp-server

Others

locate sftp-server

Then you’ll need to make sure your sudoers file has something like the following:

# Ensure user "remoteuser" does not require a TTY to run sudo.
Defaults:remoteuser !requiretty

# Ensure user "remoteuser" can execute sftp-server without a password.
remoteuser ALL=(ALL) NOPASSWD: /your/path/to/sftp-server

You can ensure the disabling of requiretty works by running the following in your workstation:

ssh remoteuser@remote.host "sudo -v"

If you don’t get any errors, then you can assume all is peachy.

With that out of the way, you can now execute this on your workstation:

shfs remoteuser@remote.host:/some/path /local/mount -o sftp_server="/usr/bin/sudo -u otheruser /path/to/sftp-server"

Voila! Now when you make changes to /local/mount on your workstation, the changes will be made using the otheruser on the remote host.

Go forth and securely edit remote filesystems with wild abandon!

A couple of months I blogged about my super exciting Cork static site generator, and about how I was now dogfooding it for my personal blog. Well, that was 4 months ago, and not a whole lot has happened since then. It turns out having a full-time job, and several other projects of varying ambitiousness, hampers efforts like Cork for me. The end result was my blog looked ugly as hell, and as a result I neglected it. Again.

So now I’m going for a slightly more pragmatic approach. I’m using Octopress to generate my blog. Jekyll is well and truly the most popular static site generator out there, and Octopress gets you from 0 to sexy-blog-plus-mobile-responsive-and-zomg-sexy-asides in 3 seconds. The plugins are mature, so now I should be able to keep onto myself to write a blog post now and then, without the excuse of a half written personal project to stop me.

I haven’t given up on Cork yet though, as a long-term goal I’d still like to finish implementing it, and use it for my blog again. However by the time I get to it, there may be other exciting new site generators out there that I can look into.

At the time of writing, my blog suddenly got very ugly… That’s because I’m in the process of switching to a static site generator I’ve written from scratch, called cork.

Cork is written in Coffeescript and of course runs in Node.js. The idea is to have a pluggable static site generator that is itself opinionated about what makes up a “site” (static pages, blog posts/archives/category lists, etc), but then defers to plugins to discover the content in the site directory, render it, lay it out, and deploy it. The goal is to have a site generator that makes a “better” trade-off when it comes to opionated (and thus convenient if opionions align with end-user) vs flexible.

Why? Well, I tried Octopress first. Octopress is fantastic, but very much at the extreme end of opinionated, as soon as I went against the grain, I was frequently frustrated and spent a lot of time Googling stuff. For example, at the time of research, using LESS in Octopress was a real pain, it couldn’t be plugged in to the rendering pipeline. I also tried Docpad, but it’s too flexible and requires a lot of work in layout code to query content and pull together a standard personal site + blog.

So, right now you’re looking at my new blog, generated using Cork and deployed to S3, and then fronted by Cloudfront. Excuse the ugliness, I’ll be working on the stylish touches when I’ve nailed down all the functionality I need.

Recently I had a few ideas I wanted to try out in geographical data visualization. I’ve never really done any data vis stuff, other than a few charts here and there, so I had no clue where to even begin.

At Wotif, I’ve been heavily involved in the Reviews system, which went live a couple of months ago. What I wanted to do was color a world map to signify the density of Reviews for hotels in each country. After some pathetic Google-fu, I ran into the Geochart visualization from Google Chart Tools. The Geochart visualization was a good start, but I quickly discovered just how limited it was when it comes to client-side interaction. I was sure there must be something more, but having never had much to do with mapping on the interwebz, I could not even begin to figure out the question I needed to ask, let alone the answer.

Thankfully, I discovered that what I was trying to display was a Choropleth Map. Once I knew this, it didn’t take long before I stumbled upon this fantastic blog post from someone trying to do exactly the same thing as I. It was through this article that I had my eyes opened to a whole world of amazing mapping tools and libraries. The one that struck me as the most interesting was none other than Polymaps, primarily because of the way it leveraged SVG for some pretty sexy looking “slippy” maps.

Even once I’d gotten my foothold in this wealth of information, I was still pretty lost. Answering my initial question lead to a myriad more. What is GeoJSON? How do tiles work? Where does a Tile Cache come into play in all this? Wading through the mass amounts of information was pretty daunting.

In the end I focussed back in on what I was trying to do. I wanted a simplistic world map that I could mutate and accept user input on. Poring over the Polymaps examples, I determined that GeoJSON would help me render an world map with country outlines. A bit more Googling and I found a fantastic Github repo - johan/world.geo.json, with all countries in a GeoJSON file.

Armed with this GeoJSON file, I was easily able to fumble about and mash my face against the keyboard until I got the desired result, a nice Choropleth map that users could mouseover to see more information! I bundled this up in a nice little dashboard with some other statistics I cooked up. I’ve since shown it to a few people at Wotif and they seem to like it.

The initial experiment only whetted my appetite though, and I’m finding myself quite engrossed in the world of amateur cartography. I’ve since started cooking up some interesting real-time displays using Polymaps, overlaying SVG drawing onto the map and doing some fun stuff with it, like smooth panning/zooming between points and extents. I will blog more about some of the cool stuff I’ve come up with.

Bye internets!

So my Linux distro of choice for quite a while has been Ubuntu. Unfortunately it seems that lately Canonical has embarked on a mission to destroy the desktop experience with religious fervor. I have no qualms in saying that I think Unity is the worst thing to ever happen to Ubuntu - it’s a complete unmitigated disaster. How it managed to become the default overnight without rioting in the streets is beyond me - but I digress.

Of course there are other alternatives in the Desktop Linux scene; offerings such as XFCE, LXDE and KDE all have their place. I personally don’t have the time to familiarize myself with yet another desktop environment, especially given that my goal is to be running a Mac at work and at home. I had grown quite fond of GNOME 2 in previous releases of Ubuntu, so my next logical step was to use GNOME 3.

The problem with GNOME 3 is that of immaturity. It’s still pretty rough around the edges (it crashes at least a couple of times a day for me under regular use), has very poor display driver support, and doesn’t quite have the community momentum for themes and plugins like its predecessor had.

I’ve been using GNOME 3 for the past few months (at work and home) with the hopes that my gripes with it would be addressed shortly but unfortunately I’m yet to observe even incremental improvements in stability or performance. My work machine (which I’m using 8 hours a day) constantly needs gnome-shell restarts, and sometimes it locks up so bad I actually have to kill the whole session and lose everything I had open. Recently my patience has worn thin enough that I started conspiring to try something (anything, really) new on my machine at work. The proverbial straw was broken when somehow my NVIDIA display drivers (custom installed in order to address a host of issues with gnome-shell) was broken so badly by an Ubuntu update that I had to boot into recovery to fix it. But wouldn’t you know it, recovery console isn’t working with my setup for some reason - it wouldn’t detect my keyboard!

I had read about the MATE project on the Linux Mint blog a few months back, so today I wondered how it had come along. The prospect of a pure GNOME 2 environment running in the latest Ubuntu sounded very promising indeed. I decided it was finally time to make the jump.

With a small amount of tinkering, I had a fully functional MATE session up and running in Ubuntu 11.10. It took a whopping 10 minutes to get MATE fully operational. All I had to do was the following:

• sudo bash -c ‘echo “deb http://tridex.net/repo/ubuntu/ oneiric main” >> /etc/apt/sources.list’

• sudo apt-get update

• sudo apt-get install mate-archive-keyring

• sudo apt-get install mate-core

• Logout, select MATE from the session list and log back in

The thing that struck me immediately is HOW INSANELY FAST IT IS. Especially compared to GNOME 3/Unity. After getting Compiz up and running and using the desktop for an hour or two, I suddenly realized just how sorely I’d missed the polished and refined awesomeness that is GNO- er, MATE.

I am in the process of installing MATE on my home machine now so I can get as much exposure to MATE as possible over the coming weeks. I will be documenting my experiences here, and sharing all the gotchas I find as I go.