GitHub Social CodingRemember my previous article "I pushed 30 of my projects to GitHub?" Well, I just gathered 20 more projects that I had done (or did recently) and pushed them all to GitHub.

Quick note on GitHub - GitHub is the best invention ever for programmers. Nothing stimulates you more than pushing more and more projects to GitHub and seeing people forking them, following them, finding and fixing bugs for you. I wouldn't be doing so much coding if there wasn't GitHub.

If you like my projects, I'd love if you followed me on github! Oh, and also on twitter! Thank you!

Right, so here are the new projects:


This is a node.js module for recording HTML5 Theora/Ogg videos. It's written in C++ and is the first class citizen of node.js, meaning that it's fully asynchronous. It uses libtheora and libogg libraries for recording.

I wrote this module for my StackVM startup so that anyone could record virtual machine video screencasts. See StackVM demo video #1 at around 1min 23secs, where I demo this module in action.


This is another node.js module for producing PNG images from raw RGB/BGR/RGBA/BGRA buffers. It's also written in C++, is asynchronous, and uses libpng to produce images. I also wrote it for StackVM. I added a concept of stacked-pngs to the library where many virtual machine screen updates get stacked together to produce the final image, but that is a topic for a separate post.


This is also a node.js module for producing JPEG images from raw RGB buffers. It uses libjpeg (or libjpeg-turbo, which is much faster than libjpeg), it's is written in C++ and is asynchronous.

This module was also written for StackVM and it will be used in cases when the client has a really slow connection. In that case the virtual machine screen updates get downsampled to quality and size that the client is able to receive.


This is a module for node.js for producing GIF images. I like this module the most because it can be used to record what I call "gifcasts". Gifcasts are screencasts that get recorded to animated gifs. Here is an example gifcast that I recorded - A gifcast of me plurking from Windows XP.

This module is also written in C++ and uses giflib.


This is a module for node.js that unifies node-png, node-jpeg and node-gif. So instead of requiring all three modules, you just var Image = require('image') and then can do things like:

var png = Image.encodeSync('png', buffer); // or
var jpeg = Image.encodeSync('jpeg', buffer); // etc.
var gif = Image.encodeSync('gif', buffer);


Node-supermarket is like a regular key-value store (hash-table), except greater. It uses node-sqlite as the underlying storage engine that gives it unprecedented stability. This library doesn't end here. The plan is to create an object store, where you can just dump the whole js objects, and then restore them back, map, filter and fold on them, etc.


Supermarket-cart stores connect sessions in supermarket key-value store.


This ia a node.js module for doing base64 encoding/decoding. I wrote it because half a year ago when I started working on StackVM, node.js didn't have base64 encoding functions and all other modules were terribly broken for binary data. So I named this module "base64 module that actually works."


This is a HTTP proxy written in node.js. It has access control and URL black lists. I wrote it for fun.

Perl TCP Proxy

This is a TCP proxy written in Perl. I wrote it as a helper program for my "Turn any Linux computer into SOCKS5 proxy in one command" post.


This is a my, James's and Joshua's node.js knockout entry - an online chess game (half-working). blog engine

I wrote a new engine in Python. I used Werkzeug, SQLAlchemy, Mako, Pygments, Memcached, Sphinx and repoze.profile to make it as awesome as it is.The design followed the "50 ideas for the new website."

Social Scraper

This is an older project from 2007 that I found on my hard drive. It's a social media website scraper (and also some popular news site scraper). It used to scrape data from boingboing,, digg, flickr, furl, reddit, simpy, stumbleupon and wired.

The Little Schemer Book Review

This is a book review of The Little Schemer. The book is a dialogue between you and the authors about interesting examples of Scheme programs and it teaches you to think recursively.

If anyone asks me which book do I recommend for learning basics of Lisp, I recommend this one (and The Seasoned Schemer, see below). It's very fun to read and can be read in one evening.

The Seasoned Schemer Book Review

This is a book review of The Seasoned Schemer. This book continues where The Little Schemer ended and introduces more advanced programming and Scheme concepts such as accumulators, letrec, letcc, call/cc and generators.

Where The Little Schemer can be read in one evening, this book will take one whole day.

The Reasoned Schemer Book Review

This is a book review of The Reasoned Schemer. Though this is not yet a full book review. I currently only had time to go through first few chapters. It's really complicated and takes a lot of effort to understand. One of the authors is Oleg Kiselyov, which instantly makes this book so conceptually difficult that it may take one full week to comprehend some of the topics.

Here is how I summarize this book:

The goal of the book is to show the beauty of relational programming. The authors of the book believe that it is natural to extend functional programming to relational programming. They demonstrate this by extending Scheme with a few new constructs, thereby combining the benefits of both styles. This extension also captures the essence of Prolog, the most well-known logic programming language.

The Little MLer Book Review

This is a book review of The Little MLer. The Little MLer book has two goals. The first and primary goal is to teach you to think recursively about types and programs. The second goal is to expose you to two important topics concerning large programs: dealing with exceptional situations and composing program components.

Having learned the concept of functors in ML, I realized that various programming languages like to call all kinds of unrelated things "functors". So I wrote a post "On Functors".


These are not all the projects that I have pushed to GitHub since last time, but the others are not that interesting. Just for completeness, they are:

  • php2000 - written in 2000, a php routing engine via require().
  • webdev-template - a small webdev template with reset css.
  • node-bufferdiff - compares two node.js buffers fast.
  • node-time - time functions for node.js (had forgotten about Date object).
  • node-jsmin - javascript minification node.js module.
  • node-async - simplest possible asynchronous node.js C++ module (useful as an example).
  • rfb-protocols - implements hextile rfb decoder to RGB buffer in C++.

This is actually more than 20 projects, but not all of them count. :) Anyway, hope you find some of them useful and until the next post!

And just another reminder, I'd love if you followed me on github and twitter! :)

So I participated in the 48 hour Node.js Knockout competition together with James Halliday and Joshua Holbrook. Our team was called Dark Knights and we created an online chess application called Node Chess.

We didn't quite manage to completely finish the game and it has several bugs, like the turns don't alternate and the king can be captured, but it's crazy awesome anyway. If both players follow the rules, it all works correctly. Castling works, pawn promotion works, capture en-passant also. Try it and if you find it awesome, please vote! Oh, and it works only in Chrome. We were under time pressure and at one point it stopped working under Firefox and we did not get to fixing it.

Here is how the game looks,

A chess game between pkrumins and someone. King's Indian Defence.

Joshua did all the awesome vector graphics work. I did the chess engine work, and James used his amazing dnode node.js module to blend client and server code together. James has actually redefined how web development happens. Instead of writing server code and client code, as we are so used to, with his dnode framework it's now possible to use the same code both server and client side! Much less hustle and purely ingenious!

Here is the same game in perspective view, the highlighted squares are the available moves,

The same game in perspective view.

And the moves are animated, too! The pawns shoot the opponent pieces and the queen stabs them. Try it!

Right, so my reflections on the competition.

It was well organized, and we were sent access to a Joyent deployment server and a Heroku server early on together with instructions. It turned out that Heroku's didn't support Socket.IO or websockets. Win for Joyent. Pretty much everyone went with Joyent as far as I know. We had some technical difficulties at the start with deploying our code, but guys at #node.js helped us and we got our app running pretty quickly.

We used 3 Git repositories to push the code to, our own GitHub repositories (pkrumins, substack, jesusabdullah), then the node knockout's private repository for judges, and deployment repository on Joyent. Joyent was configured so that as you push your code to its Git repo, the hooks in it would restart the node.js service and the you'd be instantly running the latest version of your code.

So I'd make changes push to my GitHub repo, James would pull from me. He'd make changes, I'd pull from him, and same for Joshua. It went pretty flawless. We had like 12 merge errors total, but those were all resolved within a minute or two.

Now some numbers. We're actually amazed by our performance. Check out these numbers:

$ git log | grep Author | wc -l

429 commits! Can you believe that? 429 commits in 2 days! That's 9 commits per hour on average! That is what I call hacking!

My commits:

$ git log | grep Author | grep Peteris | wc -l

I did 3.5 commits per hour on average. And funnily, James and Joshua each had 130 commits:

$ git log | grep Author | grep James | wc -l
$ git log | grep Author | grep Joshua | wc -l

That's 2.7 commits per hour on average! Amazing! But we also slept between the competition days. On the both days we did about 4 hours of clean sleep, shrinking our competition time to 40 hours. Then our average becomes 10.7 commits per hour! Wowsers!

Here is a graph, made with Raphael.js, that shows our git commit activity by hour, starting from 3am UTC Aug 28 to 3am UTC Aug 30:

Team "Dark Knights" git commit activity by hour.

Our peak commit intensity was at 9pm the last night, when we did 23 commits in one hour. Our team was also widely spread out. I am in Riga, Latvia, James is in Kenai, Alaska, and Joshua is in Fairbanks, Alaska. Yet we managed to keep the same schedule. I'd go to bed at noon (noon for me is 3pm UTC, see the graph above), while James and Joshua at midnight, and we'd wake up several hours later and keep hacking!

Total number of code lines written:

$ wc -l `find . -name '*.js' -o -name '*.html' -o -name '*.css' | \
  egrep -v '(jquery|raphael)'`
3074 total

So we wrote 3074 lines in two days, which according to git break up into added vs. deleted as following:

$ git log --numstat | grep '^[0-9]' | \
  egrep '(\.js|\.css|\.html|jquery|raphael)' | \
  awk '{a+=$1;d+=$2}END{print "Added: " a, "Deleted: " d}'
Added: 5210 Deleted: 2042

Hmm, 5210-2042 doesn't quite add up to 3074 but is close enough. From these 3074 lines of code non-empty were:

$ cat `find . -name '*.js' -o -name '*.html' -o -name '*.css' | \
  egrep -v '(jquery|raphael)'` | perl -nle 'print if /\S/' | \
  wc -l

So 2659 real lines of code in 2 days! Talk about productivity! And that's just code alone. Joshua also did 50 artworks,

$ find . -name '*.svg' -o -name '*.png' | wc -l

Total number of file changes:

$ git log --shortstat | grep 'files changed' | \
  awk '{t+=$1}END{print t}'  

We communicated in IRC, in our #stackvm startup channel. Here are some statistics on how much stuff went on in our IRC channel:

$ (
  grep -v '^0[012]:' '#stackvm.08-28.log';
  cat '#stackvm.08-29.log';
  grep '^0[012]' '#stackvm.08-30.log'
  ) | wc -l

So 5069 events happened during the challenge. That's 105 events per hour on average. We have a special lulbot in it who tells us when we commit, for example:

05:59 < lulzbot-X> Whoa Nelly! New commits to pkrumins/node-chess (master)!
05:59 < lulzbot-X>     * Peteris Krumins: MoveGenerator stub
05:59 < lulzbot-X>     * Peteris Krumins: abstract pieces
05:59 < lulzbot-X> githubs:

Here lulzbot informed us that I committed MoveGenerator stub and abstracted pieces in node-chess repo.

Out of these 5069 events, we talked this much,

$ (
  grep -v '^0[012]:' '#stackvm.08-28.log';
  cat '#stackvm.08-29.log';
  grep '^0[012]' '#stackvm.08-30.log'
  ) | egrep -i '^< pkrumins|substack|jesus' | wc -l

So we spoke 2682 times or 83.8 times per hour. We also asked quite a lot questions:

$ (
  grep -v '^0[012]:' '#stackvm.08-28.log';
  cat '#stackvm.08-29.log';
  grep '^0[012]' '#stackvm.08-30.log';
  ) | grep '?$' | wc -l

246 questions, for example (random selection):

< pkrumins> wait, are we including a version of in dnode?
< jesusabdullah> but: simplified pieces for thumbs--yea or nay?
< pkrumins> is anyone working ont he problem where the opponent cant make moves?
< SubStack> pkrumins did you see how I just dumped the node EventEmitter code into our lib/?
< SubStack> does S create a row?
< jesusabdullah> pkrumins: You fixing the board?
< pkrumins> how does resizing in raphael happen?

My chess code wasn't the easiest to write and to make sure it works correctly, I wrote 52 expresso tests,

$ expresso 

   100% 52 tests

Without tests I would have never got that chess code right.

That's about it. The competition was awesome, A++ would participate again. Hope they organize node.js knockout the next year, too!

I hope you enjoyed my post and don't forget to vote for our project! Your vote is so important to us. Thank you!

Hey everyone. We at StackVM just finished recording the 2nd demo video. The 2nd video shows all the cool new features we have recently built - user login system, chatting and sharing of virtual machines by just dragging and dropping. Also this time James Halliday joins me from Fairbanks, Alaska!

Here is the video #2,

StackVM brings virtual machines to the web. Join #stackvm on FreeNode to discuss!

If you haven't seen the first video, see my StackVM startup announcement post!

During the past few weeks we have also written two new node.js libraries for use at StackVM:

We did not demo gifcasts in this video but I am going to do a separate video in the next week or two showing just that. They're pretty awesome!

In a few weeks we'll also post the 3rd demo video. In that video we have planned to show virtual network editor that allows to network virtual machines by just dragging and dropping! Be sure to subscribe to catonmat's rss feed and follow me on twitter to know when the video is out!

See you!

Ps. Join #stackvm on FreeNode to discuss StackVM with me and James! We're there 24/7!

Hey everyone,

I have exceptional news - I am doing a startup together with James Halliday!

I met James on FreeNode several years ago and it turned out that we had very similar ideas about doing great hacking. So we teamed up as equal co-founders to do a startup. We're on different continents but we use IRC, GitHub and WeDoist to get stuff done!

Check out an early demo of our software:

StackVM brings virtual machines to the web. Join #stackvm on FreeNode to discuss!

The startup we're doing is called StackVM. StackVM makes virtual machines much more accessible over the web, makes them easier to use, and makes them embeddable in webpages (like putting a virtual machine in a blog post). This is just the first step that we're starting with. Next we're making networking between them very easy, just drag and drop to create any virtual network topology you wish, with firewalls, switches, etc. (fun for hacking competitions and learning networking). Then we're making what we call "vmcasts" - much like a screencasts, except the computation is recorded, meaning that at any point you can break into the playing vmcast and change the course of computation (and return back to it later).

Our plan is to host the virtual machines and our awesome software for you, so that you don't have to worry about anything. We'll make sure it all works! At the moment we're actually already at our third major iteration of the software.

Here are a few use cases:

  • Suppose you're selling software and you want your users to try it before they buy it. Perfect use of StackVM - put your software in the virtual machine and embed it on your products page. The potential customers can try your software before they buy it right from your website!
  • Suppose you're an application developer and have written a program that should work cross-platform. You can easily rent 10 virtual machines with Linux, Windows, MacOS, and other operating systems and test your software. Just drag and drop it into the virtual machines, and you can test your software!
  • Suppose you want to teach someone how to work in Perl in a series of blog posts (like I do), you can embed the terminal with a vmcast in your blog post, and everyone can follow your tutorial, and also try out the examples interactively, in a real shell!
  • You can build a virtual honeypot network and have hackers break into it, then analyse how they did breakins. Or, you can build a huge network and learn routing and networking concepts!
  • Suppose you want to share your work with a group of people. You can easily do it in stackvm! Just send the other people link to your VM and they can connect to it with any web browser. They'll be able to see what you're doing, comment on your work, and if you allow fix your bugs (think pair programming!)

Hosting virtual machines requires a lot of infrastructure, so we plan to start collecting revenue as soon as possible with affordable paid user accounts. We're not going to do the common nonsense among startups of first building up the project for years and only then trying to figure out how to make it profitable. We're going to offer virtual machines together with an awesome interface and features to them as a monthly paid service from the day we launch. We're also going to have an open API to our software, so that you can build on top of it or customize it (for example script virtual machines to do something specific).

We're also testing out the idea of complete openness with this startup. It's going to be 100% open-source and 100% idea-open, meaning that anyone can read what we're up to, and run our code if they want to run stackvm on their own. So far we have shared all the code on github (my stackvm repo, James's stackvm repo) and documented most of the ideas and experiments on stackvm github wiki.

We're applying for YC funding later this year so that we can focus on hacking and not paperwork.

Now a little bit about the tech we use at StackVM. I won't go into much details right now because many of the future posts will detail the tech at StackVM.

At the moment StackVM is in its 3rd major iteration already and is almost entirely built on node.js. Since we're doing everything open-source, we've written a bunch of reusable node.js modules:

  • dnode - Simple asynchronous remote method invocation for node.js.
  • node-bufferlist - Abstraction of node.js's buffers and monadic binary parsing.
  • node-rfb - Implements the client-side of the RFB protocol that VNC uses.
  • node-png - Produces PNG images from RGB or RGBA values.
  • node-jpeg - Produces JPEG images from RGB or RGBA values.
  • node-video - Records Theora/Ogg videos from RGB values.
  • node-base64 - Encodes binary data to base64 to send across websockets and ajax.
  • node-jsmin - Minify JavaScript on the server side, before sending it to the browser.
  • node-bufferdiff - A module to compare two buffers quickly.

I'll keep everyone updated on the progress and technical insight into our development. If you're not yet subscribed to my posts, please do it - catonmat rss feed. Going to make this happen!

And if you wish to talk about our software, come join #stackvm on FreeNode!

A Year of BloggingHey everyone! Another year has passed and it's now 3 years since I've been blogging here on catonmat! In this post I wish to summarize this year's statistics.

See the one year of blogging and two years of blogging for previous year statistics.

First of all traffic statistics,

Traffic statistics for the period 2009-07-01 - 2010-07-01 from Google Analytics.

Alright, so catonmat has received 1.43 million visitors and 2.11 million page views during this year. That's 120k visits and 175k page views per month. Or 4,000 visitors per day and 5,800 page views per day. Good numbers.

Let's look at the whole traffic picture from the day one of blogging,

Traffic statistics for the period 2007-07-01 - 2010-07-01 from Google Analytics.

Looks pretty random but seems to have a hidden linear trend upwards.

Now the FeedBurner subscriber stats,

Feedburner statistics for the period 2009-07-01 - 2010-07-01.

Last year I left off with 7000 subscribers, and now I have around 12,000. That's 5000 new subscribers, or 13 new subscribers per day on average.

And the whole picture of subscriber dynamics since the beginning of blogging,

Feedburner statistics for the period 2007-07-01 - 2010-07-01.

Also a nice positive trend, if it keeps going the same way, I expect to have around 17,000 subscribers the next year.

Now to articles. During this year I have written 43 articles. Here are ten most popular ones:

Here are my personal favorites that didn't make it into top ten:

It's now time for delicious cake:

Let's meet for cake the next year again! See you!