Six Years of BloggingHey everyone! Another year has passed and it's been 7 years since I've been blogging here at catonmat.net! In this post I want to summarize this year's statistics.

See the one year of blogging, two years of blogging, three years of blogging, four years of blogging, five years of blogging, and six years of blogging for the previous years' statistics.

Let's start this year's stats with traffic:



Traffic statistics from Google Analytics for Jul 1, 2013 - Jun 30, 2014.

This year: 784,992 uniques. 983,890 visits. 1,376,933 page views.
Last year: 903,468 uniques. 1,191,596 visits. 1,695,544 page views.
Delta: -118,476 unqiues. -207,706 visits. -318,611 page views.

I haven't been blogging much at all and that explains the negative delta. I've about 200 articles in drafts so stay tuned for some quality articles soon!

Feedburner stats: At 16,500 rss subscribers this year. Last year 16,343. Delta 157. subscribe.

Feedburner is a dead service now, so the rss subscriber count has become a meaningless, random number that changes daily. It fluctuates around 16,500 subscribers.

Github stats: At 1700 followers this year. Last year 1400. Delta +300. follow me.

Twitter stats: At 4130 followers this year. Last year 3774. Delta +356. follow me.

Total articles written: 20. Top 10 articles:

My top 5 favorite articles:

That's all folks! Now let's have some 7 year birthday cake,

And let's meet for the cake next year again! See you!

Someone asked me today about Perl array operations and it turned out he didn't know that all basic array operations, such as push, pop, shift, unshift can be implemented with just splice. So I thought I'd write a blog post that shows how awesome splice is.

splice

Splice has the following prototype:

splice(array, [offset, [length, [list]]])

Splice replaces length elements in array at offset with list. Offset, length, and list can be omitted, which makes it a really powerful function that can do a bunch of array operations.

push via splice

To push to an array via splice, you need to set offset to the end of the array, length to anything, and list to the list of values that you want to add.

Let's say you've array @a:

my @a = ("foo", "bar", "baz", "rms", "lbt", "esr");

Then you can push to it via splice like this:

splice(@a, @a, 0, "val1", "val2", "val3")

The @a array now contains:

("foo", "bar", "baz", "rms", "lbt", "esr", "val1", "val2", "val3")

pop via splice

To pop an element from an array via splice, you need to set offset to the end of the array.

Let's say you've array @a:

my @a = ("foo", "bar", "baz", "rms", "lbt", "esr");

Then you can pop it via splice like this:

splice(@a, -1)

The @a array now contains:

("foo", "bar", "baz", "rms", "lbt")

Here splice replaced the last element of the array with nothing, effectively removing it.

shift via splice

To shift a value off an array via splice, you need to set offset to the beginning of the array, and length to 1.

Let's say you've array @a:

my @a = ("foo", "bar", "baz", "rms", "lbt", "esr");

Then you can shift a value like this:

splice(@a, 0, 1)

The @a array now contains:

("bar", "baz", "rms", "lbt", "esr")

Similarly you can shift more than one value, if you increase length:

splice(@a, 0, 5)

This shifts first 5 values off the original array and it now contains:

("esr")

unshift via splice

To unshift values to an array via splice, you need to set offset to the beginning of the array, length to 0, and list to the list of values that you want to add.

Let's say you've array @a:

my @a = ("foo", "bar", "baz", "rms", "lbt", "esr");

Then you can unshift a new list to this array like this:

splice(@a, 0, 0, "val1", "val2")

The @a array now contains:

("val1", "val2", "bar", "baz", "rms", "lbt", "esr")

replace i-th element in an array via splice

To replace i-th element in an array via splice, you need to set offset to i, length to 1 and list to the list of values.

Let's say you've array @a:

my @a = ("foo", "bar", "baz", "rms", "lbt", "esr");

Then you can replace the 2nd element ("bar") via splice like this:

splice(@a, 1, 1, "ror")

The @a array now contains:

("foo", "ror", "baz", "rms", "lbt", "esr")

Similarly you can replace the i-th element with a list of values:

splice(@a, 1, 2, "ror", "zoz")

The @a array now contains:

("foo", "ror", "zoz", "baz", "rms", "lbt", "esr")

There are more operations that you can do with splice, such as deleting all elements in the array, adding elements at the i-th position. I'll leave these other operations as an exercise to the reader.

I think you can do the same in other languages as well, such as JavaScript, as it also has array.splice operation.

How often have you accidentally pressed ctrl+s and had your terminal frozen? I don't understand this default mapping at all. It should be gone from Linux installations. If you're a newbie you might have even thought that the terminal has died (I did think so, and I'd restart the terminal when that happened). Apparently you need to press ctrl+q to unfreeze the terminal. These keys could be used for something better than freezing and unfreezing the console.

Get rid of these keys by running these commands:

stty stop undef
stty start undef

Instead remap these combinations to something much more useful, such as saving the file in vim with ctrl+s.

Here's another pro tip. Say you wanted to run a command in sudo context but forgot that. Usually you'd hit the up arrow, press ctrl+a, or HOME button, type sudo and hit ENTER.

There is a geekier way to do that. Just type:

$ sudo !!

Bash will expand the two exclamation points to the previous command and it will be run through sudo.

Today is 4 years since I and my co-founder James Halliday have been working at Browserling Inc. I thought I'd share what I think are the top 10 inventions at our company.

The first two choices are obvious. They're our products Browserling and Testling. The next eight choices are useful and popular open source projects - browserify, dnode, ploy, seaport, airport and upnode, bouncy, hyperglue and hyperspace and hyperstream, cipherhub.

1. Browserling

Browserling is our first and most successful commercial product. Browserling lets you interactively test websites in all the browsers - IE, FireFox, Opera, Chrome, and Safari.

We successfully managed to monetize the vnc-to-browser technology that we developed four years ago (see the announcement). Our vnc-to-browser technology don't use Flash or Java applets, it uses HTML5 canvas instead.

Try it out at www.browserling.com! (Use Chrome)

2. Testling

Testling is our second product. It lets you run cross-browser JavaScript tests on every git push. After the tests run, you get a badge that shows their status in all the browsers:

Learn more about Testling at www.testling.com!

3. Browserify

Browserify lets you use node.js modules from npm in your browser. It's a game-changer for front-end development.

Browsers don't have the require method defined, but node.js does. With browserify you can write code that uses require in the same way that you would use it in node.

Here's a tutorial on how to use browserify on the command line to bundle up a simple file called main.js along with all of its dependencies:

var unique = require('uniq');

var data = [1, 2, 2, 3, 4, 5, 5, 5, 6];

console.log(unique(data));

Install the uniq module with npm:

$ npm install uniq

Now recursively bundle up all the required modules starting at main.js into a single file called bundle.js with the browserify command:

$ browserify main.js -o bundle.js

Browserify parses the AST for require calls to traverse the entire dependency graph of your project.

Drop a single <script> tag into your HTML and you're done!

<script src="bundle.js"></script>

We use browserify everywhere at Browserling. Browserling and Testling are built out of hundreds of small modules. They're bundled using browserify. If you look at Browserling's or Testling's source code, you'll see that there is just one source file bundle.js. All your Testling tests are browserified before run in the browsers as well.

Learn more about browserify at www.browserify.org.

4. Dnode

Dnode is an asynchronous RPC system for node.js that lets you call remote functions. Dnode is safe. No code gets passed along when you call a remote function. Only function references and their arguments get passed along.

Here's a tutorial on how to use dnode. First, create server.js that listens on port 5555 and exports transform function that converts its arguments to uppercase:

var dnode = require('dnode');

var server = dnode({
    transform : function (s, cb) {
        cb(s.toUpperCase())
    }
});
server.listen(5555);

Then create client.js that connects to the server over dnode and calls the transform function:

var dnode = require('dnode');

var d = dnode.connect(5555);
d.on('remote', function (remote) {
    remote.transform('test', function (s) {
        console.log('test => ' + s);
        d.end();
    });
});

When you run this, you get test => TEST as output.

We use dnode heavily at Browserling. All the processes communicate between each other using dnode. For example, we've a centralized authentication service that Browserling and Testling use. When someone signs up at Browserling, they can use the same login at Testling. Dnode makes that work behind the scenes.

Learn more about dnode at dnode's github page.

5. Ploy

Ploy is our node.js deployment system. It includes a http(s) router, a git endpoint and a process manager all in one. You just git push your code at ploy, and it deploys it and runs the necessary processes. You can also manage staging branches and monitor process logs with it.

I recently wrote an in-depth article about how we deploy code at Browserling and Testling. It starts as a tutorial and then explains how we deploy our code in-depth. Read it!

Learn more about ploy at ploy's github page.

6. Seaport

Seaport is a service registry. Seaport stores (host, port) combos (and other metadata) for you so you won't need to spend so much effort keeping configuration files current as your architecture grows to span many processes on many machines. Just register your services with seaport and then query seaport to see where your services are running!

Here's a tutorial on how to setup a seaport server, register a web server at seaport and connect to the web server from another service. First spin up a seaport server:

$ seaport listen 9000

Then from your web server, connect to seaport and register your web service at seaport by calling ports.register():

var http = require('http');
var seaport = require('seaport');
var ports = seaport.connect('localhost', 9000);

var server = http.createServer(function (req, res) {
    res.end('hello world\r\n');
});

server.listen(ports.register('web@1.2.3'));

Now that your web server has been registered at seaport, just ports.get() that web service from another program:

var seaport = require('seaport');
var ports = seaport.connect(9000);
var request = require('request');

ports.get('web@1.2.x', function (ps) {
    var url = 'http://' + ps[0].host + ':' + ps[0].port;
    request(url).pipe(process.stdout);
});

This program gets the web server's port, connects to it through request, gets response, and prints hello world. Port 9000 is the only port that you need to know in all of your programs.

We also use seaport heavily at Browserling and Testling. Browserling is built using many small services, such as stripe service for taking payments, stats for usage statistics, monitor for monitoring services, status for status.browserling.com, auth that's a centralized point for authentication, and many others. We don't even know what ports these services run on, seaport takes care of it.

Learn more about seaport at seaport's github page.

7. Airport and Upnode

Airport provides seaport-based port management for upnode. What's upnode? Upnode keeps a dnode (see invention #4 above) connection alive and re-establishes state between reconnects with a transactional message queue.

Here's a tutorial on upnode. First create a server.js that exports the time function:

var upnode = require('upnode');

var server = upnode(function (client, conn) {
    this.time = function (cb) { cb(new Date().toString()) };
});
server.listen(7000);

Now when you want to make a call to the server, guard your connection in the up() function. If the connection is alive the callback fires immediately. If the connection is down the callback is buffered and fires when the connection is ready again.

var upnode = require('upnode');
var up = upnode.connect(7000);

setInterval(function () {
    up(function (remote) {
        remote.time(function (t) {
            console.log('time = ' + t);
        });
    });
}, 1000);

This program will connect to upnode server on port 7000, and keep printing the time every second. If you take the upnode server down, it will buffer the callbacks and they'll fire when the server is available again.

Learn more about upnode at upnode's github page.

Airport provides an upnode-style dnode connections using service names from a seaport server (see invention #6 above). Instead of connecting and listening on hosts and ports, you can .connect() and .listen() on service names.

Here's a tutorial on airport. First start a seaport server on port 7000:

$ seaport listen 7000

Then write a service called fiver that exports the timesFive function that multiplies its argument by five:

var airport = require('airport');
var air = airport('localhost', 7000);

air(function (remote, conn) {
    this.timesFive = function (n, cb) { cb(n * 5) }
}).listen('fiver');

Now write a client that connects to the fiver service and calls the timesFive function:

var airport = require('airport');
var air = airport('localhost', 7000);

var up = air.connect('fiver');

up(function (remote) {
    remote.timesFive(11, function (n) {
        console.log('timesFive(11) : ' + n);
    });
});

This program outputs timesFive(11) : 55. In case the connection between the client and fiver goes down, upnode will buffer the callbacks until the connection is back.

Learn more about airport at airport's github page.

8. Bouncy

Bouncy is a minimalistic, yet powerful http(s) router that supports websockets.

Here's a bouncy tutorial. Let's say you want to route route requests based on the host HTTP header to servers on ports 8001 and 8002. For every http request, bouncy calls function (req, res, bounce) { }, so you can inspect the req.headers.host and then call bounce(8001) or bounce(8002), like this:

var bouncy = require('bouncy');

var server = bouncy(function (req, res, bounce) {
    if (req.headers.host === 'status.browserling.com') {
        bounce(8001);
    }
    else if (req.headers.host === 'browserling.com') {
        bounce(8002);
    }
    else {
        res.statusCode = 404;
        res.end('no such host');
    }
});
server.listen(80);

The bounce(PORT) function redirects the connection to the service running on PORT.

I personally don't see myself using anything but bouncy for routing http(s) requests, including websockets. It's rock solid and has been used in production for many years.

Learn more about bouncy at bouncy's github page.

9. Hyperglue, Hyperspace, and Hyperstream

Hyperglue lets you update HTML elements by mapping query selectors to attributes, text, and hypertext both in the browser and node.js.

Here's a tutorial on hyperglue. Let's say you've an article template in article.html and you want to fill a .name, span .author, and div .body with data:

<div class="article">
  <div class="title">
    <a name="name"></a>
  </div>
  <div class="info">
    <span class="key">Author:</span>
    <span class="value author"></span>
  </div>
  <div class="body">
  </div>
</div>

With hyperglue you can write the following code that will do just that:

function createArticle (doc) {
    return hyperglue(html, {
        '.title a': {
            href: doc.href,
            _text: doc.title
        },
        'span .author': doc.author,
        'span .body': { _html: doc.body }
    });
}

document.body.appendChild(createArticle({
    author: 'James Halliday',
    href: '/robots',
    title: 'robots are pretty great',
    body: '<h1>robots!</h1>\n\n' +
          '<p>Pretty great basically.</p>'
}));

The createArticle function will fill the article HTML template with data at the right selectors. As a result it will produce the following HTML, and it will append it to document.body:

<div class="article">
  <div class="title">
    <a name="name" href="/robots">robots are pretty great</a>
  </div>
  <div class="author">
    <span class="key">Author:</span>
    <span class="value author">James Halliday</span>
  </div>
  <div class="body">
<h1>robots!</h1>

<p>Pretty great basically.</p>
  </div>
</div>

The object returned by hyperglue also has an innerHTML property that contains the generated HTML, so you can also use it on the server side to get the resulting HTML:

console.log(createArticle({
    ...
}).innerHTML);

Learn more about hyperglue at hyperglue's github page.

Hyperspace renders streams of HTML on the client and the server. Here's a tutorial on hyperspace. First pick a stream data source that will give you records and let you subscribe to a changes feed. Let's start with the rendering logic in file render.js that will be used on both the client and the server:

var hyperspace = require('hyperspace');
var fs = require('fs');
var html = fs.readFileSync(__dirname + '/row.html');

module.exports = function () {
    return hyperspace(html, function (row) {
        return {
            '.who': row.who,
            '.message': row.message
        };
    });
};

The return value of hyperspace() is a stream that takes lines of JSON as input and returns HTML strings as its output. Text, the universal interface!

Template row.html used is just a really simple stub thing:

<div class="row">
  <div class="who"></div>
  <div class="message"></div>
</div>

It's easy to pipe some data to the renderer:

var r = require('./render')();
r.pipe(process.stdout);
r.write(JSON.stringify({ who: 'substack', message: 'beep boop' }) + '\n');
r.write(JSON.stringify({ who: 'pkrumins', message: 'h4x' }) + '\n');

Which prints:

<div class="row">
  <div class="who">substack</div>
  <div class="message">beep boop</div>
</div>
<div class="row">
  <div class="who">pkrumins</div>
  <div class="message">h4x</div>
</div>

To make the rendering code work in browsers, you can just require() the shared render.js file and hook that into a stream. In this example we'll use shoe to open a simple streaming websocket connection with fallbacks:

var shoe = require('shoe');
var render = require('./render');

shoe('/sock').pipe(render().appendTo('#rows'));

If you need to do something with each rendered row you can just listen for 'element' events from the render() object to get each element from the data set, including the elements that were rendered server-side.

Learn more about hyperspace at hyperspace's github page and the html streams for the browser and the server section of Stream Handbook.

Hyperstream streams HTML into HTML at css selector keys. Here's a tutorial on hyperstream. Let's say you've the following HTML template in index.html:

<html>
  <body>
    <div id="a"></div>
    <div id="b"></div>
  </body>
</html>

And you've two more files, a.html that contains:

<h1>title</h1>

And b.html that contains:

<b>hello world</b>

And you want to stream HTML from files a.html and b.html into selectors #a and #b. You can write the following code using hyperstream:

var hyperstream = require('hyperstream');
var fs = require('fs');

var hs = hyperstream({
    '#a': fs.createReadStream(__dirname + '/a.html'),
    '#b': fs.createReadStream(__dirname + '/b.html')
});
var rs = fs.createReadStream(__dirname + '/index.html');
rs.pipe(hs).pipe(process.stdout);

And it will do just that! You'll get the following output:

<html>
  <body>
    <div id="a"><h1>title</h1></div>
    <div id="b"><b>hello world</b></div>
  </body>
</html>

Learn more about hyperstream at hyperstream's github page.

10. Cipherhub

Cipherhub is our secure communications tool. It can be frustrating and annoying to communicate with somebody using public key cryptography since setting up PGP/GPG is a hassle, particularly managing key-rings and webs of trust.

Luckily, you can fetch the public ssh keys of anybody on GitHub by going to:

https://github.com/USERNAME.keys

If you just want to send somebody an encrypted message out of the blue and they already have a GitHub account with RSA keys uploaded to it, you can just do:

$ cipherhub USERNAME < secret_message.txt

And it will fetch their public keys from GitHub, storing the key locally for next time, and output the encrypted message. You can now send this message to your friend over IRC and they can decode it by running:

$ cipherhub <<< MESSAGE

Just recently we used cipherhub to send company's credit card information over IRC and I loved how easy that was.

Learn more about cipherhub at cipherhub's github page.

4 Years of Browserling

Happy birthday to Browserling Inc!

You can follow Browserling and Testling on Twitter at @browserling and @testling, and co-founders at @pkrumins and @substack.

We can also be contacted over email at hello@browserling.com.

Hooray!