Announcing Clusterphone

A couple of weeks ago I released a new Node.js library called clusterphone, to assist with intra-cluster messaging. It provides a clean, robust API to pass messages from workers to master, and vice-versa. The README has all the juicy specifics.

So what can you do with clusterphone that you can’t do with the built-in Node.js cluster messaging? Here’s a short list:

  • Dispatch messages to specific handlers on both master and worker ends.
  • Acknowledge the receipt and handling of messages.
  • Send messages to workers straight away, without worrying about them being online yet (messages are queued and sent when worker is ready).
  • Namespace messaging, so your library has no risk of conflicting with other libraries / the application using the messaging layer.
  • Acknowledge / wait for acknowledgement API supports either Node-style callbacks or Promises.

I wrote this library because I wanted to trivialize Node.js cluster messaging. I found some existing libraries out there, but none of them had all the features I wanted. Worse, very few of them had enough tests for me to be very confident in them. At the time of writing, clusterphone has 98% test coverage.

If you end up using the library, or have some thoughts about it, please feel free to get in touch!

On Finishing Things

I, like many I’m sure, have a terrible habit. That habit is starting things and never finishing them.

There, it’s out there. I’ve said it. It’s done!

I’ve started so many projects in the past year, and looking back I haven’t finished a single one.

The pattern for all of these aborted projects is almost always the same too. An idea takes shape in my mind throughout the day, I research it, validate it, and away I go! Furious coding sessions into the wee hours of the morning. Every waking free moment spent thinking and dreaming about this amazing project. Then suddenly … nothing. I lose steam overnight and forget about the excitement of the project. Several weeks later, an idea takes shape in my mind …

That ends now. I will be working on my most recent idea, a hosted Dash Docset generator for javadoc, over the next few weeks and reporting my progress here, to keep myself accountable. I will be planning out a realistic roadmap to get this project shipped and available for public use.

After that, I will be picking up the second most recent idea I worked on a couple of months ago, and I’ll be shipping that one too. Then, I’ll pick the third most recent idea, and so on.

Why this sudden drive to finish what I’ve started? There’s a couple of reasons actually. For one, the past few months I’ve been heavily into dieting for weight loss and running to increase my fitness. I’ve been setting goals and meeting them, and the feeling is fantastic. It has a measurable impact on my personal and professional life. Secondly, the ability to nail out the last 10% of a project (which, as we all know, is generally 90% of the work) is an important skill I’ll need as I mature professionally. Thirdly, I have aspirations to build some small side ventures that generate passive income. Of course this will never happen if my ideas never make it past a GitHub repository ;)

Off I go! Soon I shall publish an article tracking my progress on finishing project #1!

Atlassian.

So. I’m working at Atlassian now.


Initial thoughts.

I’m sitting in my apartment, having finished my 3rd day at the Atlassian office a few hours ago. I want to write something that conveys the insane mix of elation, trepidation, pride and self-doubt I’m presently feeling. I don’t believe I have the requisite eloquence to actually capture my feelings in writing, but, to hell with it! I’ll give it a whirl.

One of the first things that hit me on my first day at Atlassian is just how huge this place is. The second thing that occurred to me is the staggering number of incredibly talented people there are, sitting within 50m of me in all directions - Atlassian has a few floors in the Sydney office, so when I say all directions, I really mean left, right, up, and down. Right now it really feels like every single switched-on invidivual in the world works here.

Knowing that so many incredibly gifted people work here is a mixed blessing for me. On the one hand, it’s liberating to know I’m surrounded by so many empowered guys and gals. On the other hand though, I haven’t ever been surrounded by this many geniuses in one location. I’m inexplicably finding this to be rather confronting. There were a few people who were much smarter than me in my previous organization, but honestly in most settings I felt I was one of the “smarter-er” ones, which lead to me being a little too confident of my own abilities. At Atlassian I kind of feel like the village idiot. I almost feel as if I’m being literally barraged by the sheer amount of intelligence in this place. It positively oozes out of every crack of the office.

I keep reminding myself that after rigorous interviewing, I too am now planting my ass in a chair, contained by the very same office where these titans of engineering, design, finance, HR, management and leadership also sit (or stand, there’s a LOT of standing desks here). But I still keep wondering - ”Should I be here? Am I smart enough? Am I creative enough? Will I be able to deliver the results this powerhouse will expect, no, demand of me?”. Ultimately I think Atlassian is the perfect place for me, and I am going to do very well here. With that in mind, I don’t think this is a place I (or anyone) could ever get complacent in. I will constantly feel challenged and will be forever striving to better myself - to understand more, to experience more, to do more.

One of the biggest reasons that the sheer intellect of those around me appears so vast is because of how much Atlassian “dogfoods” their own products. One of their biggest products, Confluence, a Wiki collaboration tool, is heavily used internally to disseminate knowledge of … well, everything. Seriously. Atlassians have concisely documented noteworthy places I can eat near the office, insightful dossiers on the executive team, upcoming local gigs (and people who are going to them), and everything in between. I’m to undertake several phases of initiation into the company, parts of which have been prepared by the HR team and other parts by the Engineering department, all of which is made easily available in Confluence.

Enough melodrama.

Fair enough. Let me talk about some of the tangible awesome that is Atlassian.

The on-boarding process.

I rocked up to the office Monday morning at 10am sharp. I was then taken through a basic initiation presentation with a couple of lovely people from HR. After this I was taken on a whirlwind tour through the level 6 and 7 office floors, where I was then handed off to another meeting. This one explained the basics of the systems I’ll be dealing with on an everyday basis, and access details for them. From here I was then planted at my new desk to get the basics set up.

No sooner had I settled in was I already being whisked off to lunch and 1-on-1 meetings with my development manager to begin my mentorship. For the rest of the first afternoon I was busy working through an expansive “Survival Essentials” checklist that directed me to pertinent knowledge areas in the Confluence instance I mentioned earlier.

This checklist also set me onto my “bootcamp” process. It’s a 4 week program which involves attending many hosted talks - fellow Atlassians run these on what appears to be a fortnightly roster, going through the Dragon Quest, fixing my first bugs, and finally shipping a feature. That’s right. I’m going to be designing, implementing, polishing, and SHIPPING a feature to Atlassian customers within 3 and a half weeks from now. Atlassian has a lot of freakin’ customers, even if only a tiny fraction of them check out my new feature (whether it be bundled with a version of Confluence/JIRA or made available as a plugin), that’s a whole lotta eyeballs on something I created.

Along the way in my initiation I’ve met many people who are extremely friendly and helpful. Other than the reservations I expressed earlier (the part where I was whining about how many really smart people I’m surrounded by now), I haven’t felt awkward or out of place for even a second. The whole process has been incredibly smooth so far.

The Perks

You’ve probably heard of a few of them. The fully-stocked kitchen (breakfast, lunch, goodie bar, icecream freezer), the beer, the Aeron chairs, the ping pong table, the pool table. They’re all pretty sweet. I don’t know what to say besides that, other than: Why the hell doesn’t every other company invest in this kind of stuff?

I haven’t taken any 20% time, and I haven’t experienced any of the TGIF events (the Friday just before I joined was Bouncy Castle Boxing) yet, but I’ll definitely be reporting back on this blog when I do.

The Culture

Whether it’s the wallboards that intone in a robotic voice at predefined times “Pushups. Do it.”, the hilarious flowcharts in the bathrooms, the standing desks, the Nerf gun I was given in my first day welcome pack, the whole-wall whiteboards covered in goals (many of which have already been satisfied) or the entertaining themed meeting rooms (I had meetings today in the Nebuchadnezzar and Aperture, yesterday was meetings in Tardis and Rapture) you can’t really walk a single step here and not feel the Atlassian vibe.

Of course one of the biggest aspects of Atlassian culture is the ShipIt days. A 24-hour hackathon to build new features for Atlassian projects is a logical choice for a company that demands, and fosters, creativity and results from all in its employ. It just so happens there’s one tomorrow, and I’ll be joining a team of 4 others to hack up an interesting and exciting new feature for Confluence.

TL;DR

Your friends weren’t lying to you. Atlassian really is the best place to work in Australia. You should come work here.

Reasons Why AWS Elastic Beanstalk Sucks

So I’ve been using Amazon Web Services Elastic Beanstalk for a few PHP and Java apps. I’ve decided I definitely don’t like it. Here’s why.

Well actually, a quick bit of background. I love how straightforward and powerful Heroku is (and I can maybe even forgive them for that whole dyno routing debacle), but the dealbreaker for me is it being US-based. I live in Australia, and while our routing to the East Coast isn’t terrible, it’s still not ideal to serve a predominantly Australian website from there.

Small history lesson aside, here are the main reasons why I dislike Elastic Beanstalk.

Slow

Okay okay, this might be quibbling a bit, because any kind of orchestrated deployment platform like Beanstalk/Heroku is probably much faster than even the most experienced sysadmin could run up a LAMP stack, configure it, and deploy your crappy PHP app to. That said, Beanstalk is quite painfully slow to start up a new app, or make changes to, especially when you compare it to Heroku.

Weird

Vague title yes, but allow me to explain. Elastic Beanstalk does things I just don’t understand:

  • Destroys and recreates EC2 images unnecessarily. Why do I need a whole new instance when I simply change a security group?
  • Why does git aws.push repush the entire repo if there was no changes since last push?

Some of these might just be me doing things wrong, but that ties in well with my next point, and it’s the big one.

TERRIBLE Documentation

Oh my GOD. AWS has notoriously bad documentation in general, but Elastic Beanstalk HAS to be an elaborate joke Amazon is playing on the world. What little documentation there actually is on entire swathes of functionality provided by Elastic Beanstalk is chaotically organized. There’s two different command line toolsets available to interact with Elastic Beanstalk. The newer one has tragic documentation coverage, but is quite powerful when you scrounge enough information on how to use it.

I really cannot overstate how poor the Elastic Beanstalk documentation is. It’s extremely frustrating as I’ve been spoiled by fairly excellent documentation resources on the internets. Again, contrast Elastic Beanstalk with Heroku on the documentation front - there’s really no comparison.

Okay, Some Positives

So as much as Elastic Beanstalk can be frustrating, it does have some positive aspects. It’s a great way to deploy applications to AWS, when comparing to tedious approaches like manually setting up an AWS environment, or using complex solutions like Netflix Asgard, or CloudFormation. Also, it really is the only decent PaaS available in Australia.

I have noticed that Elastic Beanstalk has been receiving fairly frequent updates and improvements, so perhaps the shortcomings in documentation for it could be attributed to the velocity in which the core service is being iterated on.

/rant

As much as I don’t like Elastic Beanstalk in its current state, I will continue to use it for applications I develop/maintain that have a predominantly Australian audience. I will cross my fingers and hope that Elastic Beanstalk service and documentation improve over the coming months, or that AWS OpsWorks evolves to be a better alternative.

SSSD: Cannot Load Configuration Database (FIX)

If you’re trying to configure SSSD and you get an error message in your syslog that kinda looks like this:

May  3 00:55:51 www1 sssd: Cannot load configuration database

… Then you just need to make sure your sssd.conf file has a newline at the end. Not even kidding.

echo >> /etc/sssd/sssd.conf 

I had this particular issue on Ubuntu 12.04 Server. Might be specific to that distro, although I did find this sssd ticket that seems to indicate it was an upstream issue, but then again it was closed as invalid. So I dunno. The invalid resolution fixed the issue for me. YMMV.

Fixing VMware Unity Menu With MATE Desktop

If you’re using MATE Desktop in a virtual machine with VMware, you might notice that when you open the Unity Applications menu on your host, you get an empty list.

Luckily, there’s an easy fix.

1
2
3
4
5
cat >> ~/.profile <<MATEFIX
# Fix VMware Unity mode for MATE.
XDG_CURRENT_DESKTOP=GNOME
export XDG_CURRENT_DESKTOP
MATEFIX

Run the above, log out and back in to your desktop, and re-enter VMware Unity. Ta-da!

The issue is that a VMware tools utility, /usr/bin/vmware-xdg-detect-de, does not recognize MATE (even though MATE is just GNOME2), so by setting the environment variable XDG_CURRENT_DESKTOP in your .profile, you force VMware to recognize your desktop as GNOME.

Typography, Kerning, and Other Design-y Things.

I’ve always wished I could be better at design. Growing up, my artistic style extended as far as drawing mazes and stick figures - most of the latter looking like they had been horribly deformed by a tragic car accident.

The thing is though, I have always been painfully aware that creating something interesting, something people can get excited about, something people will show to their friends, is just as much about the design as it is the functionality. I had this revelation very early on in my career, but I’ve never had the patience to learn enough about design to apply it practically to things I create, in either my career or my personal endeavours.

I’m trying to change that now though, and with amazing resources like Hack Design, I may actually have a shot at learning a thing or two about design.

Kerning?

Tonight I was going through the second lesson on typography, and came across the fascinating Kerning Game site. It presents a few simple exercises, in which words of differing serif/sans serif fonts have had their kerning messed up, and your task is to restore them to their optimal spacing. I’m usually terrified of trying stuff like this, as I don’t like being reminded of things I’m not good at, but I decided to give it a go.

As it would turn out, I’m a little better at kerning type than I am at drawing stick figures, as I managed to score 93/100 overall for the 10 exercises presented! I’m not sure if this site was sneakily placed in the Hack Design course to give an early morale booster, but hidden agendas be damned! I’m feeling really encouraged and am going to stick at this design thing!

Golang

Google’s Go programming language, while not new, has been making the rounds on Hacker News headlines quite alot lately. I’ve been wanting to try a new language, but was put off by the not-quite-1.0-yet status of Rust, and Scala/Clojure didn’t quite pique my interest enough to sit down and give them a proper go.

Other than a few testimonials from companies like Airbrake/Cloudflare/DNSimple saying that Go is awesome for use case X and blah blah blah … I didn’t really know what the fuss was all about with Go, other than it’s good for concurrency, and had some interesting ideas around syntax and dependency management.

I finally took the plunge and went through the Go tour and I’m definitely impressed. Go-lang is a pretty freakin’ sweet language!

Some reasons why it’s awesome include:

  • Static compilation - no more pip/bundle/npm package management issues. No more juggling the correct version of a runtime, Go compiles ALL OF THE THINGS into a single static binary.
  • Syntax - apart from the backward-ness in variable definitions, Go syntax is very clean and very readable.
  • Code style - Go ships with gofmt, which means programmers no longer have to quibble over code style nonsense, the language designers decided on it for you.
  • Sweet tooling - the go compiler suite comes with lots of awesome functionality straight out of the box.
  • Concurrency - I didn’t know what CSP was before I picked up Go, but I would definitely choose goroutines and channels over Java java.util.concurrent / Node callback-soup / Node fibers any day.

Some parts about Go that I’m concerned about, or don’t fully understand:

  • Dependency versioning - while Go has a very opionated appreoach to dependency management, it also sports a very rudimentary package manager of sorts (I consider go get package management, feel free to tell me I’m wrong), it seems that handling dependency versioning is a bit dodgy.
  • Runtime performance - Go site itself mentions that its still early days for its garbage collection and allocator. That said many benchmarks are looking very promising for Go 1.0, and also much better for Go 1.1.
  • Package maturity - I’ve learnt this one from personal experience - you need to be extemely cautious about which 3rd-party packages you pull in for an immature programming language. I started using Node.js as an early-adopter, and while NPM is exploding with amazing packages now, early on there were a lot of poorly written libraries, and worse, some libraries written in C that had all kinds of concurrency issues. Since Go’s recommended approach is to re-implement all of the things in pure Go, I suspect that many people will find themselves re-implementing/porting stuff all over the place.

With that said, I haven’t actually had a chance to use Go in anger on something interesting yet, though I do intend to rectify that ASAP. I look forward to open sourcing something interesting in Go on my github profile in the near future!

Editing Remote Server Content. Like a Boss.

So you have a Unix-based host sitting there. It’s got some files on it. You wanna edit those files. Let’s pretend you’re not editing config files though, because you should be using Git/Puppet/Chef for that. Instead, I’m assuming you’re wanting to maybe make a few adhoc changes somewhere in the filesystem. I’m also assuming it’s no longer 1994 and you don’t use FTP for productivity/development out of principle.

Enter sshfs. Briefly, SSHFS is a FUSE driver that allows you to mount a remote filesystem locally, using SSH as the transport. The benefits of this should be immediately obvious:

  • Instantly secure.
  • Ubiquitous support - only requires OpenSSH server on the remote end.
  • Easy to setup - in most cases there won’t be an iota of configuration required if you’ve already got your pubkey on the remote.

Installing sshfs is straightward enough on Ubuntu:

1
(sudo) apt-get install sshfs

Mounting a remote location is also easy as pie:

1
2
3
4
5
mkdir -p /local/mount/point
sshfs remoteuser@remote.host:/remote/location /local/mount/point
ls /local/mount/point

# zomgawd the remote filesystem location lists on your local mount point how does that even oh my god what

Bonus points

So the above is all well and good for most use cases, but there’s a caveat. What if you want to edit files as a specific user on the remote host, but you either don’t have direct ssh access to that user, or it doesn’t have a shell (the latter would be more likely if you were editing system config files, but we already established you’re not doing that)? Turns out this is something that sshfs can handle, too!

What you can do is override some sshfs settings and escalate your privileges to the other user when connecting to the remote end. That way, whenever you make changes locally, the changes are applied using the correct user on the remote end. This avoids permission issues, and also means new files will be created with the correct user!

In order for this to work, you’ll need to have sudo permissions with a user on the remote host. Specifically, you’ll need:

  • privileges that DON’T require a TTY, or a password.
  • the path to the sftp-server binary on your remote host.

Determining the sftp-server path is up to you, as a hint though, you’ll probably find it in the following locations…

Ubuntu:

/usr/lib/openssh/sftp-server

CentOS 5:

/usr/libexec/openssh/sftp-server

Others

locate sftp-server

Then you’ll need to make sure your sudoers file has something like the following:

# Ensure user "remoteuser" does not require a TTY to run sudo.
Defaults:remoteuser !requiretty

# Ensure user "remoteuser" can execute sftp-server without a password.
remoteuser ALL=(ALL) NOPASSWD: /your/path/to/sftp-server

You can ensure the disabling of requiretty works by running the following in your workstation:

ssh remoteuser@remote.host "sudo -v"

If you don’t get any errors, then you can assume all is peachy.

With that out of the way, you can now execute this on your workstation:

shfs remoteuser@remote.host:/some/path /local/mount -o sftp_server="/usr/bin/sudo -u otheruser /path/to/sftp-server"

Voila! Now when you make changes to /local/mount on your workstation, the changes will be made using the otheruser on the remote host.

Go forth and securely edit remote filesystems with wild abandon!

Octopress!

A couple of months I blogged about my super exciting Cork static site generator, and about how I was now dogfooding it for my personal blog. Well, that was 4 months ago, and not a whole lot has happened since then. It turns out having a full-time job, and several other projects of varying ambitiousness, hampers efforts like Cork for me. The end result was my blog looked ugly as hell, and as a result I neglected it. Again.

So now I’m going for a slightly more pragmatic approach. I’m using Octopress to generate my blog. Jekyll is well and truly the most popular static site generator out there, and Octopress gets you from 0 to sexy-blog-plus-mobile-responsive-and-zomg-sexy-asides in 3 seconds. The plugins are mature, so now I should be able to keep onto myself to write a blog post now and then, without the excuse of a half written personal project to stop me.

I haven’t given up on Cork yet though, as a long-term goal I’d still like to finish implementing it, and use it for my blog again. However by the time I get to it, there may be other exciting new site generators out there that I can look into.