Y-CombinatorIn this post I'll derive the Y-combinator and explain all the steps taken. The key to understanding the Y-combinator is knowing precisely what it does. I am sure many people never understand Y-combinator because of this very reason, or because the introduction is too lengthy to comprehend. Therefore I'll give the shortest possible explanation of Y-combinator before I derive it so that you know what the end result should be.

The Y-combinator allows an anonymous function to call itself, that is, it allows anonymous function recursion. Since anonymous functions don't have names, they can't refer to themselves easily. The Y-combinator is a clever construct helps them to refer to themselves. That's it. That's all the Y-combinator does. Remember that.

I'll use the Scheme programming language to derive the Y-combinator. My derivation is based on the one in "The Little Schemer" book. I took the examples from the book, filled in the missing steps and explained the steps in more details.

Here it is.

Suppose we want to write the length function, which, given a list, returns the number of elements in it. It's really easy if we can give the function a name:

(define length
  (lambda (list)
    (cond
      ((null? list) 0)
      (else
        (add1 (length (cdr list)))))))

But now suppose that we can't give names - we can only use anonymous functions:

(lambda (list)
  (cond
    ((null? list) 0)
    (else
      (add1 (??? (cdr list)))))))

Suddenly there is no way this anonymous function can refer to itself. What do we put in ???? We can't refer to the name length anymore because it's an anonymous function, which doesn't have any name. One thing we can try is to put the function itself in the place of ???:

(lambda (list)
  (cond
    ((null? list) 0)
    (else
      (add1 (
             
  (lambda (list)                   ; the
    (cond                          ; 
      ((null? list) 0)             ; function
      (else                        ;
       (add1 (??? (cdr list))))))  ; itself
  
  (cdr list))))))

This is not much better, we are still left with ???. But there is a bright side to it, this function can determine lengths of lists with one element and no elements. Let's try inserting the same anonymous function in place of ??? again:

(lambda (list)
  (cond
    ((null? list) 0)
    (else
      (add1 (
  (lambda (list) 
    (cond 
      ((null? list) 0) 
      (else 
       (add1 (
    (lambda (list)
      (cond
        ((null? list) 0)
        (else
         (add1 (??? (cdr list))))))
    (cdr list))))))  
  (cdr list))))))

Now this function can determine lengths of lists with 0, 1 and 2 elements. If we continue this way, we can construct the length function for lists with a certain number of elements. But this is not what we want, we the real length function that works for lists with any number of elements.

As we all know, repeating code is not a good thing. Let's try to factor out the repetitions and rewrite the original anonymous function slightly. Instead of leaving ??? in the code, let's pass it to an anonymous function via an argument called length.

((lambda (length)
   (lambda (list)
     (cond
       ((null? list) 0)
       (else
        (add1 (length (cdr list)))))))
 ???)

Notice that this code invokes the first lambda (length) function and passes it ??? as the length argument. This function in turn returns the original anonymous function:

(lambda (list)
  (cond
    ((null? list) 0)
    (else
     (add1 (??? (cdr list))))))

Now let's try constructing the previous two length functions that work for lists with a maximum of one and two elements. First the function for lists with a maximum of one element:

((lambda (f)
   (lambda (list)
     (cond
       ((null? list) 0)
       (else
        (add1 (f (cdr list)))))))
 ((lambda (g)
   (lambda (list)
     (cond
       ((null? list) 0)
       (else
        (add1 (g (cdr list)))))))
 ???))

Here ??? first gets passed to lambda (g), which returns the original anonymous function, which then gets passed to lambda (f), which in turn returns the function that works for lists with one element:

(lambda (list)
  (cond
    ((null? list) 0)
    (else
      (add1 (
  (lambda (list)
    (cond
      ((null? list) 0)
      (else
       (add1 (??? (cdr list))))))
  (cdr list))))))

Similarly, the following code returns a function that works for lists with a maximum of two elements:

((lambda (f)
   (lambda (list)
     (cond
       ((null? list) 0)
       (else
        (add1 (f (cdr list)))))))
 ((lambda (g)
   (lambda (list)
     (cond
       ((null? list) 0)
       (else
        (add1 (g (cdr list)))))))
 ((lambda (h)
   (lambda (list)
     (cond
       ((null? list) 0)
       (else
        (add1 (h (cdr list)))))))
 ???)))

Since the argument names f, g,h in (lambda (f) ...), (lambda (g) ...), (lambda (h) ...) are independent, we can rename all of them to length, to make it look more similar to the length function:

((lambda (length)
   (lambda (list)
     (cond
       ((null? list) 0)
       (else
        (add1 (length (cdr list)))))))
 ((lambda (length)
   (lambda (list)
     (cond
       ((null? list) 0)
       (else
        (add1 (length (cdr list)))))))
 ((lambda (length)
   (lambda (list)
     (cond
       ((null? list) 0)
       (else
        (add1 (length (cdr list)))))))
 ???)))

We are still repeating code. The obvious thing we can do about it is create an anonymous function called mk-length (make length) that creates this code for us:

((lambda (mk-length)
   (mk-length ???))
 (lambda (length)
   (lambda (list)
     (cond
       ((null? list) 0)
       (else
        (add1 (length (cdr list))))))))

This is pretty tricky. Observe how (lambda (mk-length) ...) gets the (lambda (length) ...) function passed as the mk-length argument, which in turn accepts ??? as an argument and returns our original anonymous function:

(lambda (list)
  (cond
    ((null? list) 0)
    (else
     (add1 (??? (cdr list))))))

Now let's try constructing length functions for lists of length one and two. For the list of length one, we just need to apply mk-length on itself:

((lambda (mk-length)
   (mk-length
     (mk-length ???)))
 (lambda (length)
   (lambda (list)
     (cond
       ((null? list) 0)
       (else
        (add1 (length (cdr list))))))))

Let's go through this code. First (lambda (length) ...) gets passed to the lambda (mk-length) function as the mk-length argument. Then it applies the result of (mk-length ???) (which is the original anonymous function) to the mk-length. This produces our well known function that works on lists with one or none elements:

(lambda (list)
  (cond
    ((null? list) 0)
    (else
      (add1 (
  (lambda (list)
    (cond
      ((null? list) 0)
      (else
       (add1 (??? (cdr list))))))
  (cdr list))))))

Similarly, by adding another call to mk-length, we get the function that works for lists with two or less elements:

((lambda (mk-length)
   (mk-length
     (mk-length
       (mk-length ???)))
 (lambda (length)
   (lambda (list)
     (cond
       ((null? list) 0)
       (else
        (add1 (length (cdr list))))))))

Now, since we don't know what ??? is, let's just call it mk-length and see what happens:

((lambda (mk-length)
   (mk-length mk-length))
 (lambda (length)
   (lambda (list)
     (cond
       ((null? list) 0)
       (else
        (add1 (length (cdr list))))))))

Nothing bad actually happens, we still get the same original anonymous function, except in place of ??? we have the function:

(lambda (length)
  (lambda (list)
    (cond
      ((null? list) 0)
      (else
       (add1 (length (cdr list)))))))

Notice also that argument names mk-length and length in lambda (mk-length) and lambda (length) are independent. Therefore we can rename length to mk-length to remind that the first argument to mk-length is mk-length:

((lambda (mk-length)
   (mk-length mk-length))
 (lambda (mk-length)
   (lambda (list)
     (cond
       ((null? list) 0)
       (else
        (add1 (mk-length (cdr list))))))))

And now comes the key trick. We replace mk-length with a self-application (mk-length mk-length):

((lambda (mk-length)
   (mk-length mk-length))
 (lambda (mk-length)
   (lambda (list)
     (cond
       ((null? list) 0)
       (else
        (add1 ((mk-length mk-length) (cdr list))))))))

If you try this code out, you'll see that it returns the length of a list for any list. Here is an example of applying it to a list of 10 elements:

(((lambda (mk-length)
   (mk-length mk-length))
 (lambda (mk-length)
   (lambda (list)
     (cond
       ((null? list) 0)
       (else
        (add1 ((mk-length mk-length) (cdr list))))))))
 '(a b c d e f g h i j))

The function works because it keeps adding recursive uses by passing mk-length to itself, just as it is about to expire. This is not yet the Y-combinator, but we have successfully managed to recursively call an anonymous function. Now we need to massage the code a bit to separate the anonymous function from the self-applicative code.

The first step is to move the self-applicative code out as much as possible:

((lambda (mk-length)
   (mk-length mk-length))
 (lambda (mk-length)
   ((lambda (length)
      (lambda (list)
        (cond
          ((null? list) 0)
          (else
           (add1 (length (cdr list)))))))
   (lambda (x)
     ((mk-length mk-length) x)))))

Now the code in bold looks very much like the length function we need! Let's move it outside as well:

((lambda (le)
   ((lambda (mk-length)
      (mk-length mk-length))
    (lambda (mk-length)
      (le (lambda (x)
            ((mk-length mk-length) x))))))
 (lambda (length)
    (lambda (list)
      (cond
        ((null? list) 0)
        (else
         (add1 (length (cdr list))))))))

There we have it! The anonymous length function has been separated from the self-applicative call. The code in bold is the length function, and the code not-in-bold is the Y-combinator!

Now let's rename mk-length to f and join the lines to make it more compact:

((lambda (le)
   ((lambda (f) (f f))
    (lambda (f)
      (le (lambda (x) ((f f) x))))))
 (lambda (length)
   (lambda (list)
      (cond
        ((null? list) 0)
        (else
         (add1 (length (cdr list))))))))

And we are done. We can now write down the definition of Y-combinator:

(define Y
  (lambda (le)
    ((lambda (f) (f f))
     (lambda (f)
       (le (lambda (x) ((f f) x)))))))

And we can apply it to our anonymous length function:

((Y (lambda (length)
     (lambda (list)
       (cond
         ((null? list) 0)
         (else
          (add1 (length (cdr list))))))))
 '(a b c d e f g h i j))

; ==> 10

This is one of many derivations of the applicative-order Y-combinator. See the publication "The Why of Y" for another derivation.

Windows 95 LogoI am not just a Linux enthusiast, I also happen to use Windows quite often. In fact, Windows is my primary desktop from which I connect to all the other boxes and do my work on. During the years of Windows usage, I have accumulated a list of must-have Windows programs that I wouldn't be able to work without. Some of them are commercial, some are freeware, but that doesn't matter. What matters is how productive you are with your setup. If you're really productive on Linux with your own set of tools, it's perfectly fine and you have done a great job of finding the best tools of trade.

1. Total Commander.

I can't imagine working on a computer without Total Commander. Absolutely must-have software. Total Commander is what separates boys from men. Total Commander is probably the #1 reason why I don't use other operating system on my desktop. Tabs, a great GUI interface and customize-everything-configuration make it superior to all the other file managers. If I switch the desktop OS, I'll have to write my own clone of Total Commander or I wouldn't be able to work with the computer. Total commander is shareware.

2. TrueCrypt.

Because you don't want anyone to know what you store on your disks. TrueCrypt the best disk encryption software for Windows. TrueCrypt is freeware.

3. SecureCRT and Putty.

SecureCRT is the SSH client to use. Again, the primary reason I use it is because it has tab support. Putty comes second as it doesn't have tabs. SecureCRT is shareware. Putty is freeware.

4. WinSCP.

Forget all your FTP clients. FTP is dead and unsecure. I use WinSCP to send all my files over a secure connection. WinSCP is freeware.

5. Cygwin.

Since Windows has never had a usable shell, the only way to get one is to run a local ssh server via Cygwin and connect to localhost via SecureCRT. This way you get the best possible shell. Cygwin is freeware.

6. VMWare Workstation.

I have been using VMWare Workstation since 2004 and have never had a single problem. It's the best virtualization software ever written and nothing comes close. VMWare Workstation is shareware.

7. Synergy.

Synergy allows you to share your mouse and keyboard via network with other computers (no need for a KVM switch). It just works. Synergy is freeware.

8. Beyond compare.

The greatest visual diff tool ever written. Period. Beyond compare is shareware.

9. Tclock2.

Tclock2 is freeware that allows you to customize the windows clock in the systray. I set it to dddd\nyyyy.mm.dd\nhh:nn:ss.

10. Auto Hotkeys.

With Auto Hotkeys you can remap your keyboard and script shortcuts. For one, the first thing to do is to remap ESC to CAPS LOCK so that you could work in Vim. Auto Hotkeys is freeware.

11. Locate32.

Locate32 is to Windows what locate is to Linux. It helps you find all your files in an instant. Locate32 is freeware.

12. allSnap.

allSnap snaps your windows one next to each other, i.e., it aligns them nicely. allSnap is freeware.

13. Fineprint.

Fineprint is the best printer proxy ever. Allows you to print multiple pages per sheet even much more smartly than your printer does. Plus it has a real print preview. Fineprint is shareware.

14. Launchy.

I can't imagine working without Launchy. It's QuickSilver for Windows. It allows to quickly launch programs by pressing Alt+Space and typing in the first few chars of the program's name. Launchy is freeware.

15. ClipX.

ClipX maintains a paste buffer. Use Winkey+V to paste from it. If you haven't used it, it will change the way you work with your computer. ClipX is freeware.

16. DU Meter.

Du Meter measures your network usage and shows nice plots of it. I like it. Du Meter is shareware.

17. Taskbar Shuffle.

Taskbar Shuffle allows you to rearrange taskbar tabs and systray icons. Must-have. Taskbar Shuffle is freeware.

18. UltraMon.

If you're on a multi-screen setup, like me, UltraMon adds taskbars to your other screens. It's productivity super-booster. UltraMon is shareware.

19. Sumatra PDF.

Adobe Reader got bloated and died 5 years ago. Foxit Reader got bloated died 2 years ago. I hope Sumatra PDF reader doesn't die. Sumatra PDF is freeware.

20. KeePass.

KeePass is a secure password manager. I probably have 10 thousand passwords in it secured by one master password. Keepass is freeware.

21. RoboForm.

While KeePass keeps your passwords secure in a file, RoboForm allows you to dynamically submit them to various website forms. It also stores them under the master password. RoboForm is shareware.

22. WinRAR.

WinRAR is the best archivator for Windows. I have been using it since I was born. 7-Zip doesn't come close. WinRAR is shareware.

23. Inkscape.

Inkscape is a descent vector graphics editor. It's free.

24. FFDShow.

FFDShow is a codec library. If you wish to watch all kinds of videos on your computer, it's a must-have. FFDShow is freeware.

25. Media Player Classic.

MPC is the best video player for windows. It doesn't have skins or any other useless bullshit. MPC is freeware.

26. Real Alternative.

Real Alternative will use Media Player Classic to play RealMedia files. Real Alternative is freeware.

27. Quicktime Alternative.

Quicktime Alternative is to QuickTime files what Real Alternative is to RealMedia files. Freeware.

28. AviSynth.

AviSynth is a video processing programming language. If you wish to speed your videos up, it's a must-have. AviSynth is freeware.

29. FFMpeg.

FFMpeg is a must-have for converting videos and working with AviSynth scripts. FFMpeg is freeware.

30. VirtualDub.

VirtualDub is another must-have software for editing videos on Windows. VirtualDub is freeware.

31. Dependency Walker.

Sometimes programs don't work because of DLL hell. Dependency Walker helps you to diagnose and solve these problems. Dependency Walker is freeware.

32. Most of the Sysinternals tools.

Autoruns, regmon, procmon, procexp, tcpview, to name a few from Sysinternals are just must-have. They are all freeware.

33. File & Folder Unlocker.

UFFunlock is used when you get the nasty "Access denied" errors and you know that no process should be using the resource. FFunlock is freeware.

34. Hijackthis.

Hijackthis is necessary just to make sure you don't have nasty programs on your computer. Hijackthis is freeware.

35. ImgBurn.

ImgBurn is the new Nero. It burns your CDs and DVDs. ImgBurn is freeware.

36. IsoBuster.

IsoBuster knows all the CD/DVD image formats. IsoBuster is shareware.

37. Virtual CloneDrive.

Virtual CloneDrive allows you to mount ISOs right as windows drives. Virtual CloneDrive is freeware.

38. SQLyog.

As crappy as SQLyog is, it's somehow the most usable MySQL front-end. 10 years ago I used mysql-gui-console but the project got discontinued and no one has written a usable MySQL front-end ever since. If I'm ever in a mood, I'll write one that doesn't suck. SQLyog is shareware but I wouldn't pay a cent for it.

39. pgAdmin.

I applaud to pgAdmin dev team for making the best GUI front-end for PostgreSQL. Folks who write MySQL front-ends should learn from these guys. They know what a GUI tool should do. pgAdmin is freeware.

40. XML Notepad.

A notepad.exe for XML files. Allows you to edit XML smartly without destroying its structure accidentally. XML Notepad is freeware.

41. Hex Workshop.

Hex Workshop is a really great and easy to use Hex editor. It's shareware.

42. Vim.

For everything else.

What tools do you use?

This was a list of tools that I can't live without. What tools did I miss?

This article is part of the article series "Perl One-Liners Explained."
<- previous article next article ->

Perl One LinersThis is the fifth part of a nine-part article on famous Perl one-liners. In this part I will create various one-liners for text conversion and substitution. See part one for introduction of the series.

Famous Perl one-liners is my attempt to create "perl1line.txt" that is similar to "awk1line.txt" and "sed1line.txt" that have been so popular among Awk and Sed programmers.

The article on famous Perl one-liners will consist of nine parts:

After I'm done with explaining the one-liners, I'll release an ebook. Subscribe to my blog to know when that happens!

Awesome news: I have written an e-book based on this article series. Check it out:

Alright then, here are today's one-liners:

Text conversion and substitution

62. ROT13 a string.

'y/A-Za-z/N-ZA-Mn-za-m/'

This one-liner uses the y operator (also known as tr operator) to do ROT13. Operators y and tr do string transliteration. Given y/SEARCH/REPLACE/, the operator transliterates all occurrences of the characters found in SEARCH list with the corresponding (position-wise) characters in REPLACE list.

In this one-liner A-Za-z creates the following list of characters:

ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz

And N-ZA-Mn-za-m creates this list:

NOPQRSTUVWXYZABCDEFGHIJKLMnopqrstuvwxyzabcdefghijklm

If you look closely you'll notice that the second list is actually the first list offset by 13 characters. Now the y operator translates each character in the first list to a character in the second list, thus performing the ROT13 operation.

If you wish to ROT13 the whole file then do this:

perl -lpe 'y/A-Za-z/N-ZA-Mn-za-m/' file

The -p argument puts each of file's line in the $_ variable, the y does ROT13, and -p prints the $_ out. The -l appends a newline to the output.

Note: remember that applying ROT13 twice produces the same string, i.e., ROT13(ROT13(string)) == string.

63. Base64 encode a string.

perl -MMIME::Base64 -e 'print encode_base64("string")'

This one-liner uses the MIME::Base64 module that is in the core (no need to install it, it comes with Perl). This module exports the encode_base64 function that takes a string and returns base64 encoded version of it.

To base64 encode the whole file do the following:

perl -MMIME::Base64 -0777 -ne 'print encode_base64($_)' file

Here the -0777 argument together with -n causes Perl to slurp the whole file into the $_ variable. Then the file gets base64 encoded and printed out, just like the string example above.

If we didn't slurp the file and encoded it line-by-line we'd get a mess.

64. Base64 decode a string.

perl -MMIME::Base64 -le 'print decode_base64("base64string")'

The MIME::Base64 module also exports decode_base64 function that takes a base64-encoded string and decodes it.

The whole file can be similarly decoded by:

perl -MMIME::Base64 -ne 'print decode_base64($_)' file

There is no need to slurp the whole file into $_ because each line of a base64 encoded file is exactly 76 characters and decodes nicely.

65. URL-escape a string.

perl -MURI::Escape -le 'print uri_escape($string)'

You'll need to install the URI::Escape module as it doesn't come with Perl. The module exports two functions - uri_escape and uri_unescape. The first one does URL-escaping (sometimes also referred to as URL encoding), and the other does URL-unescaping (URL decoding).

66. URL-unescape a string.

perl -MURI::Escape -le 'print uri_unescape($string)'

This one-liner uses the uri_unescape function from URI::Escape module to do URL-unescaping.

67. HTML-encode a string.

perl -MHTML::Entities -le 'print encode_entities($string)'

This one-liner uses the encode_entities function from HTML::Entities module. This function encodes HTML entities. For example, < and > get turned into &lt; and &gt;.

68. HTML-decode a string.

perl -MHTML::Entities -le 'print decode_entities($string)'

This one-liner uses the decode_entities function from HTML::Entities module.

69. Convert all text to uppercase.

perl -nle 'print uc'

This one-liner uses the uc function, which by default operates on the $_ variable and returns an uppercase version of it.

Another way to do the same is to use -p command line option that enables automatic printing of $_ variable and modify it in-place:

perl -ple '$_=uc'

The same can also be also achieved by applying the \U escape sequence to string interpolation:

perl -nle 'print "\U$_"'

It causes anything after it (or until the first occurrence of \E) to be upper-cased.

70. Convert all text to lowercase.

perl -nle 'print lc'

This one-liner is very similar to the previous. Here the lc function is used that converts the contents of $_ to lowercase.

Or, using escape sequence \L and string interpolation:

perl -nle 'print "\L$_"'

Here \L causes everything after it (until the first occurrence of \E) to be lower-cased.

71. Uppercase only the first word of each line.

perl -nle 'print ucfirst lc'

The one-liner first applies the lc function to the input that makes it lower case and then uses the ucfirst function that upper-cases only the first character.

It can also be done via escape codes and string interpolation:

perl -nle 'print "\u\L$_"'

First the \L lower-cases the whole line, then \u upper-cases the first character.

72. Invert the letter case.

perl -ple 'y/A-Za-z/a-zA-Z/'

This one-liner does transliterates capital letters A-Z to lowercase letters a-z, and lowercase letters to uppercase letters, thus switching the case.

73. Camel case each line.

perl -ple 's/(\w+)/\u$1/g'

This is a lousy Camel Casing one-liner. It takes each word and upper-cases the first letter of it. It fails on possessive forms like "friend's car". It turns them into "Friend'S Car".

An improvement is:

s/(?<!['])(\w+)/\u\1/g

Which checks if the character before the word is not single quote '. But I am sure it still fails on some more exotic examples.

74. Strip leading whitespace (spaces, tabs) from the beginning of each line.

perl -ple 's/^[ \t]+//'

This one-liner deletes all whitespace from the beginning of each line. It uses the substitution operator s. Given s/REGEX/REPLACE/ it replaces the matched REGEX by the REPLACE string. In this case the REGEX is ^[ \t]+, which means "match one or more space or tab at the beginning of the string" and REPLACE is nothing, meaning, replace the matched part with empty string.

The regex class [ \t] can actually be replaced by \s+ that matches any whitespace (including tabs and spaces):

perl -ple 's/^\s+//'

75. Strip trailing whitespace (space, tabs) from the end of each line.

perl -ple 's/[ \t]+$//'

This one-liner deletes all whitespace from the end of each line.

Here the REGEX of the s operator says "match one or more space or tab at the end of the string." The REPLACE part is empty again, which means to erase the matched whitespace.

76. Strip whitespace from the beginning and end of each line.

perl -ple 's/^[ \t]+|[ \t]+$//g'

This one-liner combines the previous two. Notice that it specifies the global /g flag to the s operator. It's necessary because we want it to delete whitespace at the beginning AND end of the string. If we didn't specify it, it would only delete whitespace at the beginning (assuming it exists) and not at the end.

77. Convert UNIX newlines to DOS/Windows newlines.

perl -pe 's|\n|\r\n|'

This one-liner substitutes the Unix newline \n LF with Windows newline \r\n CRLF on each line. Remember that the s operator can use anything for delimiters. In this one-liner it uses vertical pipes to delimit REGEX from REPLACE to improve readibility.

78. Convert DOS/Windows newlines to UNIX newlines.

perl -pe 's|\r\n|\n|'

This one-liner does the opposite of the previous one. It takes Windows newlines CRLF and converts them to Unix newlines LF.

79. Convert UNIX newlines to Mac newlines.

perl -pe 's|\n|\r|'

Apple Macintoshes used to use \r CR as newlines. This one-liner converts UNIX's \n to Mac's \r.

80. Substitute (find and replace) "foo" with "bar" on each line.

perl -pe 's/foo/bar/'

This one-liner uses the s/REGEX/REPLACE/ command to substitute "foo" with "bar" on each line.

To replace all "foos" with "bars", add the global /g flag:

perl -pe 's/foo/bar/g'

81. Substitute (find and replace) "foo" with "bar" on lines that match "baz".

perl -pe '/baz/ && s/foo/bar/'

This one-liner is equivalent to:

while (defined($line = <>)) {
  if ($line =~ /baz/) {
    $line =~ s/foo/bar/
  }
}

It puts each line in variable $line, then checks if line matches "baz", and if it does, it replaces "foo" with "bar" in it.

Perl one-liners explained e-book

I've now written the "Perl One-Liners Explained" e-book based on this article series. I went through all the one-liners, improved explanations, fixed mistakes and typos, added a bunch of new one-liners, added an introduction to Perl one-liners and a new chapter on Perl's special variables. Please take a look:

Have Fun!

Have fun with these one-liners for now. The next part is going to be about selective printing and deleting of certain lines.

Can you think of other text conversion and substitution procedures that I did not include here?

Hey everyone, just wanted to do a quick post on how to keep track of who's talking about you on the net. Nothing really unique, just a list of tools that I use often. Why is it important? Well, it's always interesting to know what people are saying about you and sometimes you want to engage in a conversation or just thank them for linking to your article.

Alright, here are the tools that I use:

Twitter Search

Twitter search is definitely the #1 source for keeping track of who's talking about you right now. But you already knew that.

Twitter Search
Twitter search example for the term "catonmat."

Perhaps what you didn't know is that they have an RSS feed for search queries.

Twitter search RSS feed
Location of RSS feed link for Twitter search results.

Now combined with a service like feedblitz.com you can email the RSS updates to yourself or just read them from your favorite RSS reader.

I am monitoring terms "Peteris Krumins", "pkrumins" and "catonmat".

Google Alerts

Google Alerts automatically notifies you when the Google search engine locates new results for your search terms. You can choose to have your alerts delivered via email or RSS feed.

Google Alerts
Google Alerts email for the term "catonmat."

You can even customize the type of alerts you wish to receive. Google Alerts lets you choose to get notified when a new result appears on web pages, usenet (google groups), blogs, news or videos.

Backtype Comment Alerts

Backtype is Google for comments. Want to find out when someone's mentioned you on Reddit, FriendFeed, Digg or Hacker News? Backtype will alert you.

Backtype Comment Alerts
Backtype Alerts email for the term "peteris."

Backtype also recently launched a service called BackTweets that allows you to find who's linking back to you via shortened URLs.

Have Fun!

Have fun keeping track of yourself!

Btw, let me know in the comments if I missed any other cool tools.

hackers steal moneyI recently watched an interesting video lecture on stealing botnets. A group of researchers at UCSB recently managed to take control over a part of Torpig botnet for 10 days. During this time, they observed 180 thousand infections and recorded almost 70GB of data that bots collected. This data included submitted form information from all the websites the infected person had visited, smtp, ftp, pop3, windows, passwords, credit card numbers and passwords from various password managers.

Here are the most interesting facts from the lecture:

Torpig uses a technique called "domain fluxing" to avoid being shut down by simply blocking the IP or the domain name of control center servers. The idea is simple - depending on date and time the algorithm generates a domain name to connect to. If the domain gets shut down, the bots will simply use a different domain after some time.

The researchers were able to take control over a part of the botnet by cracking the domain name generating algorithm and registering some of the domain names to be used for communication in the future.

The bad guys noticed that a part of botnet has been taken over and issued a software update to all bots to use a new domain flux algorithm, which used Twitter's popular topics for the day to generate domain names. It was no longer possible to predict the domain that would be used tomorrow.

When communicating with command & control server, the bots included a unique id field that was generated from machine's hardware. This allowed researchers to estimate the real number of unique computers infected. Researchers saw 1.2 million unique IP addresses but only 180k unique machines.

The bots would steal financial data from 410 financial institutions (top 5: PayPal, Poste Italiane, Capital One, E*Trade, Chase), they would log credit card information (top 5 cards: Visa, Mastercard, American Express, Maestro, Discover), and they would also steal all the passwords from browser's password manager.

In a 2008 study Symantec estimated that credit card information is valued at $.10 to $25 per card in the underground market. The bank account information is valued at $10.00 to $1,000 per account. Using this study, researchers estimated that during 10 day period the amount of financial data bots collected were worth $83k to $8.3 million.

Using various estimations researchers calculated that if the bots are used for denial of service the total bandwidth would be 17Gbps.

Researchers observed that there was a fraction of people who'd fill out the phishing page and then immediately email the company's security group telling that they may have been victims of identity theft.

Since Torpig was sending all the HTTP POST data and emails to command & control servers, researchers did statistics on emails and found out that 14% of all captured emails were about jobs and resumes, 10% discussed computer security/malware, 7% discussed money, 6% were sports fans, 5% were worried about exams and their grades, 4% were seeking partners online.

Researchers collected 300,000 unique credentials on 370,000 websites. 28% of people reused their password on multiple domains. There were 173,686 unique passwords.

Researchers converted the passwords in Unix format and tried to crack them with John the Ripper. 56,000 were cracked in less than 65 minutes using brute-force. Using a wordlist 14,000 passwords were cracked in the next 10 minutes. And another 30,000 passwords were cracked in the next 24 hours. That's 58% of all passwords cracked in 24 hours.

You're welcome to watch the video lecture. It's 1h 15m long. It's presented by Richard A. Kemmerer.

Here are all the topics in the lecture:

  • [02:00] Botnet terminology - bot, botnet, command & control server, control channel, botmaster.
  • [03:00] Introduction to the Torpig trojan and Mebroot malware platform.
  • [05:00] How Torpig works.
  • [11:30] Torpig HTML injection.
  • [15:00] Domain fluxing.
  • [19:15] Taking over Torpig's c&c server.
  • [24:10] Data collection principles.
  • [26:00] C&c server protocol.
  • [31:10] Botnet's size estimation.
  • [37:00] Botnet's threats: theft of financial information, denial of service, proxy servers, privacy thefts.
  • [37:30] Threat: Theft of financial information.
  • [42:00] Threat: Denial of service.
  • [43:30] Threat: Proxy servers.
  • [44:20] Threat: Privacy theft.
  • [47:00] Password analysis.
  • [50:40] Criminal retribution.
  • [53:00] Law enforcement.
  • [58:00] Repatriating the data.
  • [01:00:00] Ethics.
  • [01:02:00] Conclusions.
  • [01:06:00] Questions and answers.

For more information see the publication "Your Botnet is My Botnet: Analaysis of a Botnet Takeover."