I am not just a Linux enthusiast, I also happen to use Windows quite often. In fact, Windows is my primary desktop from which I connect to all the other boxes and do my work on. During the years of Windows usage, I have accumulated a list of must-have Windows programs that I wouldn’t be able to work without. Some of them are commercial, some are freeware, but that doesn’t matter. What matters is how productive you are with your setup. If you’re really productive on Linux with your own set of tools, it’s perfectly fine and you have done a great job of finding the best tools of trade.

1. Total Commander.

I can’t imagine working on a computer without Total Commander. Absolutely must-have software. Total Commander is what separates boys from men. Total Commander is probably the #1 reason why I don’t use other operating system on my desktop. Tabs, a great GUI interface and customize-everything-configuration make it superior to all the other file managers. If I switch the desktop OS, I’ll have to write my own clone of Total Commander or I wouldn’t be able to work with the computer. Total commander is shareware.

2. TrueCrypt.

Because you don’t want anyone to know what you store on your disks. TrueCrypt the best disk encryption software for Windows. TrueCrypt is freeware.

3. SecureCRT and Putty.

SecureCRT is the SSH client to use. Again, the primary reason I use it is because it has tab support. Putty comes second as it doesn’t have tabs. SecureCRT is shareware. Putty is freeware.

4. WinSCP.

Forget all your FTP clients. FTP is dead and unsecure. I use WinSCP to send all my files over a secure connection. WinSCP is freeware.

5. Cygwin.

Since Windows has never had a usable shell, the only way to get one is to run a local ssh server via Cygwin and connect to localhost via SecureCRT. This way you get the best possible shell. Cygwin is freeware.

6. VMWare Workstation.

I have been using VMWare Workstation since 2004 and have never had a single problem. It’s the best virtualization software ever written and nothing comes close. VMWare Workstation is shareware.

7. Synergy.

Synergy allows you to share your mouse and keyboard via network with other computers (no need for a KVM switch). It just works. Synergy is freeware.

8. Beyond compare.

The greatest visual diff tool ever written. Period. Beyond compare is shareware.

9. Tclock2.

Tclock2 is freeware that allows you to customize the windows clock in the systray. I set it to dddd\nyyyy.mm.dd\nhh:nn:ss.

10. Auto Hotkeys.

With Auto Hotkeys you can remap your keyboard and script shortcuts. For one, the first thing to do is to remap ESC to CAPS LOCK so that you could work in Vim. Auto Hotkeys is freeware.

11. Locate32.

Locate32 is to Windows what locate is to Linux. It helps you find all your files in an instant. Locate32 is freeware.

12. allSnap.

allSnap snaps your windows one next to each other, i.e., it aligns them nicely. allSnap is freeware.

13. Fineprint.

Fineprint is the best printer proxy ever. Allows you to print multiple pages per sheet even much more smartly than your printer does. Plus it has a real print preview. Fineprint is shareware.

14. Launchy.

I can’t imagine working without Launchy. It’s QuickSilver for Windows. It allows to quickly launch programs by pressing Alt+Space and typing in the first few chars of the program’s name. Launchy is freeware.

15. ClipX.

ClipX maintains a paste buffer. Use Winkey+V to paste from it. If you haven’t used it, it will change the way you work with your computer. ClipX is freeware.

16. DU Meter.

Du Meter measures your network usage and shows nice plots of it. I like it. Du Meter is shareware.

17. Taskbar Shuffle.

Taskbar Shuffle allows you to rearrange taskbar tabs and systray icons. Must-have. Taskbar Shuffle is freeware.

18. UltraMon.

If you’re on a multi-screen setup, like me, UltraMon adds taskbars to your other screens. It’s productivity super-booster. UltraMon is shareware.

19. Sumatra PDF.

Adobe Reader got bloated and died 5 years ago. Foxit Reader got bloated died 2 years ago. I hope Sumatra PDF reader doesn’t die. Sumatra PDF is freeware.

20. KeePass.

KeePass is a secure password manager. I probably have 10 thousand passwords in it secured by one master password. Keepass is freeware.

21. RoboForm.

While KeePass keeps your passwords secure in a file, RoboForm allows you to dynamically submit them to various website forms. It also stores them under the master password. RoboForm is shareware.

22. WinRAR.

WinRAR is the best archivator for Windows. I have been using it since I was born. 7-Zip doesn’t come close. WinRAR is shareware.

23. Inkscape.

Inkscape is a descent vector graphics editor. It’s free.

24. FFDShow.

FFDShow is a codec library. If you wish to watch all kinds of videos on your computer, it’s a must-have. FFDShow is freeware.

25. Media Player Classic.

MPC is the best video player for windows. It doesn’t have skins or any other useless bullshit. MPC is freeware.

26. Real Alternative.

Real Alternative will use Media Player Classic to play RealMedia files. Real Alternative is freeware.

27. Quicktime Alternative.

Quicktime Alternative is to QuickTime files what Real Alternative is to RealMedia files. Freeware.

28. AviSynth.

AviSynth is a video processing programming language. If you wish to speed your videos up, it’s a must-have. AviSynth is freeware.

29. FFMpeg.

FFMpeg is a must-have for converting videos and working with AviSynth scripts. FFMpeg is freeware.

30. VirtualDub.

VirtualDub is another must-have software for editing videos on Windows. VirtualDub is freeware.

31. Dependency Walker.

Sometimes programs don’t work because of DLL hell. Dependency Walker helps you to diagnose and solve these problems. Dependency Walker is freeware.

32. Most of the Sysinternals tools.

Autoruns, regmon, procmon, procexp, tcpview, to name a few from Sysinternals are just must-have. They are all freeware.

33. File & Folder Unlocker.

UFFunlock is used when you get the nasty “Access denied” errors and you know that no process should be using the resource. FFunlock is freeware.

34. Hijackthis.

Hijackthis is necessary just to make sure you don’t have nasty programs on your computer. Hijackthis is freeware.

35. ImgBurn.

ImgBurn is the new Nero. It burns your CDs and DVDs. ImgBurn is freeware.

36. IsoBuster.

IsoBuster knows all the CD/DVD image formats. IsoBuster is shareware.

37. Virtual CloneDrive.

Virtual CloneDrive allows you to mount ISOs right as windows drives. Virtual CloneDrive is freeware.

38. SQLyog.

As crappy as SQLyog is, it’s somehow the most usable MySQL front-end. 10 years ago I used mysql-gui-console but the project got discontinued and no one has written a usable MySQL front-end ever since. If I’m ever in a mood, I’ll write one that doesn’t suck. SQLyog is shareware but I wouldn’t pay a cent for it.

39. pgAdmin.

I applaud to pgAdmin dev team for making the best GUI front-end for PostgreSQL. Folks who write MySQL front-ends should learn from these guys. They know what a GUI tool should do. pgAdmin is freeware.

40. XML Notepad.

A notepad.exe for XML files. Allows you to edit XML smartly without destroying its structure accidentally. XML Notepad is freeware.

41. Hex Workshop.

Hex Workshop is a really great and easy to use Hex editor. It’s shareware.

42. Vim.

For everything else.

What tools do you use?

This was a list of tools that I can’t live without. What tools did I miss?

This is the third post in the article series about Unix and Linux utilities that you should know about. In this post I will take you through the useful lsof tool. If netcat was called the Swiss Army Knife of Network Connections, then I’d call lsof the Swiss Army Knife of Unix debugging.

Lsof follows Unix philosophy closely. It does just one task and it does it perfectly — it lists information about files opened by processes. An open file may be a regular file, a directory, a NFS file, a block special file, a character special file, a shared library, a regular pipe, a named pipe, a symbolic link, a socket stream, an Internet socket, a UNIX domain socket, and many others. Since almost everything in Unix is a file, you can imagine how incredibly useful lsof is!

See the first post on pipe viewer for the introduction to this article series. If you are interested in articles like this one, I suggest that you subscribe to my rss feed to receive my future posts automatically!

How to use lsof?

In this article I will try to present lsof based on as many use cases as I can think of. Let’s start with the simplest (that you probably already know) and proceed to more complicated ones.

List all open files.

# lsof

Running lsof without any arguments lists all open files by all processes.

Find who’s using a file.

# lsof /path/to/file

With an argument of a path to a file, lsof lists all the processes, which are using the file in some way.

You may also specify several files, which lists all the processes, which are using all the files:

# lsof /path/to/file1 /path/to/file2

Find all open files in a directory recursively.

# lsof +D /usr/lib

With the +D argument lsof finds all files in the specified directory and all the subdirectories.

Note that it’s slower than the usual version with grep:

# lsof | grep '/usr/lib'

It’s slower because +D first finds all the files and only then does the output.

List all open files by a user.

# lsof -u pkrumins

The -u option (think user) limits output of files opened only by user pkrumins.

You can use comma separated list of values to list files open by several users:

# lsof -u rms,root

This will list all the files that are open by users rms and root.

Another way to do the same is by using the -u option twice:

# lsof -u rms -u root

Find all open files by program’s name.

# lsof -c apache

The -c option selects the listing of files for processes whose name begins with apache.

So instead of writing:

# lsof | grep foo

You can now write the shorter version:

# lsof -c foo

In fact, you can specify just the beginning part of the process name you’re looking for:

# lsof -c apa

This will list all the open files by a processes whose starts with apa.

You can also specify several -c options to output open files by several processes:

# lsof -c apache -c python

This will list all open files by apache and python.

List all open files by a user OR process.

# lsof -u pkrumins -c apache

Lsof options can be combined. The default is to OR between options. It means it will combine outputs of -u pkrumins and -c apache producing a listing of all open files by pkrumins and all open files by apache.

List all open files by a user AND process.

# lsof -a -u pkrumins -c bash

Notice the -a option. It combines the options with AND. The output listing is files opened by bash, which is run under pkrumins user.

List all open files by all users EXCEPT root.

# lsof -u ^root

Notice the ^ character before root username. It negates the match and causes lsof print all open files by all users who are not root.

List all open files by the process with PID.

# lsof -p 1

The -p option (think PID) filters out open files by program’s id.

Remember that you can select multiple PIDs by either comma separating the list or using multiple -p arguments:

# lsof -p 450,980,333

This selects processes with PIDs 450, 980 and 333.

List all open files by all the processes EXCEPT process with PID.

# lsof -p ^1

Here the negation operator ^ is used again. It inverts the list and does not include process with PID 1.

List all network connections.

# lsof -i

Lsof with -i option lists all processes with open Internet sockets (TCP and UDP).

List all TCP network connections.

# lsof -i tcp

The -i argument can take several options, one of them is tcp. The tcp option forces lsof to list only processes with TCP sockets.

List all UDP network connections.

# lsof -i udp

The udp option causes lsof to list processes with UDP sockets.

Find who’s using a port.

# lsof -i :25

The :25 option to -i makes lsof find processes using TCP or UDP port 25.

You may also use service port name (found in /etc/services) rather than port number:

# lsof -i :smtp

Find who’s using a specific UDP port.

# lsof -i udp:53

Similarly, to find who’s using a TCP port, use:

# lsof -i tcp:80

Find all network activity by user.

# lsof -a -u hacker -i

Here the -a option combines -u and -i to produce listing of network file usage by user hacker.

List all NFS (Network File System) files.

# lsof -N

This option is easy to remember because -N is NFS.

List all Unix domain socket files.

# lsof -U

This option is also easy to remember because -U is Unix.

List all files for processes with a specific group id.

# lsof -g 1234

Process groups are used to logically group processes. This example finds all files opened by processes with PGID 1234.

List all files associated with specific file descriptors.

# lsof -d 2

This lists all files that have been opened as file descriptor 2.

You may also specify ranges of file descriptors:

# lsof -d 0-2

This would list all files with file descriptors 0, 1 and 2.

There are also many special values, such as mem, that lists memory-mapped files:

# lsof -d mem

Or txt for programs loaded in memory and executing:

# lsof -d txt

Output PIDs of processes using some resource.

# lsof -t -i

The -t option outputs only PIDs of processes. Used together with -i it outputs PIDs of all processes with network connections. It’s easy to kill all processes that use network:

# kill -9 `lsof -t -i`

Repeat listing files.

# lsof -r 1

The -r option makes lsof repeatedly list files until interrupted. Argument 1 means repeat the listing every 1 second. This option is best combined with a narrower query such as monitoring user network file activity:

# lsof -r 1 -u john -i -a

How to install lsof?

Lsof comes preinstalled on many Unix systems. If your system doesn’t have it, try to install it from the source.

BSD supplies its own utility that does similar things, it’s called fstat.

For the full documentation of lsof see the man lsof page or type lsof -h for a small cheat sheet.

Have fun with lsof!

This is the second post in the article series about Unix utilities that you should know about. In this post I will introduce you to the netcat tool or simply nc.

Netcat is often referred to as a “Swiss Army knife” utility, and for a good reason. Just like the multi-function usefulness of the venerable Swiss Army pocket knife, netcat’s functionality is as helpful. Some of its features include port scanning, transferring files, port listening and it can be used a backdoor.

In 2006 netcat was ranked #4 in “Top 100 Network Security Tools” survey, so it’s definitely a tool to know.

See the first post on pipe viewer for the introduction to this article series. If you feel like you are interested in this stuff, I suggest that you subscribe to my rss feed to receive my future posts automatically.

How to use nc?

Let’s start with a few very simple examples and build up on those.

If you remember, I said that netcat was a Swiss Army knife. What would a Swiss Army knife be if it also wasn’t a regular knife, right? That’s why netcat can be used as a replacement of telnet:

$ nc www.google.com 80

It’s actually much more handy than the regular telnet because you can terminate the connection at any time with ctrl+c, and it handles binary data as regular data (no escape codes, nothing).

You may add “-v” parameter for more verboseness, and two -v’s (-vv) to get statistics of how many bytes were transmitted during the connection.

Netcat can also be used as a server itself. If you start it as following, it will listen on port 12345 (on all interfaces):

$ nc -l -p 12345

If you now connect to port 12345 on that host, everything you type will be sent to the other party, which leads us to using netcat as a chat server. Start the server on one computer:

# On a computer A with IP 10.10.10.10
$ nc -l -p 12345

And connect to it from another:

# On computer B
$ nc 10.10.10.10 12345

Now both parties can chat!

Talking of which, the chat can be turned to make two processes talk to each other, thus making nc do I/O over network! For example, you can send the whole directory from one computer to another by piping tar to nc on the first computer, and redirecting output to another tar process on the second.

Suppose you want to send files in /data from computer A with IP 192.168.1.10 to computer B (with any IP). It’s as simple as this:

# On computer A with IP 192.168.1.10
$ tar -cf - /data | nc -l -p 6666

# On computer B
$ nc 192.168.1.10 6666 | tar -xf -

Don’t forget to combine the pipeline with pipe viewer from previous article in this series to get statistics on how fast the transfer is going!

A single file can be sent even easier:

# On computer A with IP 192.168.1.10
$ cat file | nc -l -p 6666

# On computer B
$ nc 192.168.1.10 6666 > file

You may even copy and restore the whole disk with nc:

# On computer A with IP 192.168.1.10
$ cat /dev/hdb | nc -l -p 6666

# On computer B
$ nc 192.168.1.10 6666 > /dev/hdb

Note: It turns out that “-l” can’t be used together with “-p” on a Mac! The solution is to replace “-l -p 6666″ with just “-l 6666″. Like this:

$ nc -l 6666

# nc now listens on port 6666 on a Mac computer

An uncommon use of netcat is port scanning. Netcat is not the best tool for this job, but it does it ok (the best tool is nmap):

$ nc -v -n -z -w 1 192.168.1.2 1-1000
(UNKNOWN) [192.168.1.2] 445 (microsoft-ds) open
(UNKNOWN) [192.168.1.2] 139 (netbios-ssn) open
(UNKNOWN) [192.168.1.2] 111 (sunrpc) open
(UNKNOWN) [192.168.1.2] 80 (www) open
(UNKNOWN) [192.168.1.2] 25 (smtp) : Connection timed out
(UNKNOWN) [192.168.1.2] 22 (ssh) open

The “-n” parameter here prevents DNS lookup, “-z” makes nc not to receive any data from the server, and “-w 1″ makes the connection timeout after 1 second of inactivity.

Another uncommon behavior is using netcat as a proxy. Both ports and hosts can be redirected. Look at this example:

$ nc -l -p 12345 | nc www.google.com 80

This starts a nc server on port 12345 and all the connections get redirected to google.com:80. If you now connect to that computer on port 12345 and do a request, you will find that no data gets sent back. That’s correct, because we did not set up a bidirectional pipe. If you add another pipe, you can get the data back on another port:

$ nc -l -p 12345 | nc www.google.com 80 | nc -l -p 12346

After you have sent the request on port 12345, connect on port 12346 to get the data.

Probably the most powerful netcat’s feature is making any process a server:

$ nc -l -p 12345 -e /bin/bash

The “-e” option spawns the executable with it’s input and output redirected via network socket. If you now connect to the host on port 12345, you may use bash:

$ nc localhost 12345
ls -las
total 4288
   4 drwxr-xr-x 15 pkrumins users    4096 2009-02-17 07:47 .
   4 drwxr-xr-x  4 pkrumins users    4096 2009-01-18 21:22 ..
   8 -rw-------  1 pkrumins users    8192 2009-02-16 19:30 .bash_history
   4 -rw-r--r--  1 pkrumins users     220 2009-01-18 21:04 .bash_logout
   ...

The consequences are that nc is a popular hacker tool as it is so easy to create a backdoor on any computer. On a Linux computer you may spawn /bin/bash and on a Windows computer cmd.exe to have total control over it.

That’s everything I can think of. Do you know any other netcat uses that I did not include?

How to install nc?

If you’re on Debian or Debian based system such as Ubuntu do the following:

$ sudo aptitude install netcat

If you’re on Fedora or Fedora based system such as CentOS do:

$ sudo yum install netcat

If you’re on Slackware, FreeBSD, NetBSD, Solaris or Mac, download the source code of nc and just:

$ tar -zxf nc-version.tar.gz
$ cd nc-version
$ ./configure && sudo make install

Another way to do it on Mac, if you have MacPorts is:

$ sudo port install netcat

On Slackware you can actually install it as a package from n/ package directory:

$ sudo installpkg nc-1.10-i386-1.tgz

If you’re on Windows, download the Windoze port of it from securityfocus.

The manual of the utility can be found here man nc.

Have fun netcatting, and until next time!

Hi all. I’m starting yet another article series here. This one is going to be about Unix utilities that you should know about. The articles will discuss one Unix program at a time. I’ll try to write a good introduction to the tool and give as many examples as I can think of.

Before I start, I want to clarify one thing - Why am I starting so many article series? The answer is that I want to write about many topics simultaneously and switch between them as I feel inspired.

The first post in this series is going to be about not so well known Unix program called Pipe Viewer or pv for short. Pipe viewer is a terminal-based tool for monitoring the progress of data through a pipeline. It can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion.

Update: French translation available.

Pipe viewer is written by Andrew Wood, an experienced Unix sysadmin. The homepage of pv utility is here: pv utility.

If you feel like you are interested in this stuff, I suggest that you subscribe to my rss feed to receive my future posts automatically.

How to use pv?

Ok, let’s start with some really easy examples and progress to more complicated ones.

Suppose that you had a file “access.log” that is a few gigabytes in size and contains web logs. You want to compress it into a smaller file, let’s say a gunzip archive (.gz). The obvious way would be to do:

$ gzip -c access.log > access.log.gz

As the file is so huge (several gigabytes), you have no idea how long to wait. Will it finish soon? Or will it take another 30 mins?

By using pv you can precisely time how long it will take. Take a look at doing the same through pv:

$ pv access.log | gzip > access.log.gz
611MB 0:00:11 [58.3MB/s] [=>      ] 15% ETA 0:00:59

Pipe viewer acts as “cat” here, except it also adds a progress bar. We can see that gzip processed 611MB of data in 11 seconds. It has processed 15% of all data and it will take 59 more seconds to finish.

You may stick several pv processes in between. For example, you can time how fast the data is being read from the disk and how much data is gzip outputting:

$ pv -cN source access.log | gzip | pv -cN gzip > access.log.gz
source:  760MB 0:00:15 [37.4MB/s] [=>     ] 19% ETA 0:01:02
  gzip: 34.5MB 0:00:15 [1.74MB/s] [  <=>  ]

Here we specified the “-N” parameter to pv to create a named stream. The “-c” parameter makes sure the output is not garbaged by one pv process writing over the other.

This example shows that “access.log” file is being read at a speed of 37.4MB/s but gzip is writing data at only 1.74MB/s. We can immediately calculate the compression rate. It’s 37.4/1.74 = 21x!

Notice how the gzip does not include how much data is left or how fast it will finish. It’s because the pv process after gzip has no idea how much data gzip will produce (it’s just outputting compressed data from input stream). The first pv process, however, knows how much data is left, because it’s reading it.

Another similar example would be to pack the whole directory of files into a compressed tarball:

$ tar -czf - . | pv > out.tgz
 117MB 0:00:55 [2.7MB/s] [>         ]

In this example pv shows just the output rate of “tar -czf” command. Not very interesting and it does not provide information about how much data is left. We need to provide the total size of data we are tarring to pv, it’s done this way:

$ tar -cf - . | pv -s $(du -sb . | awk '{print $1}') | gzip > out.tgz
 253MB 0:00:05 [46.7MB/s] [>     ]  1% ETA 0:04:49

What happens here is we tell tar to create “-c” an archive of all files in current dir “.” (recursively) and output the data to stdout “-f -”. Next we specify the size “-s” to pv of all files in current dir. The “du -sb . | awk ‘{print $1}’” returns number of bytes in current dir, and it gets fed as “-s” parameter to pv. Next we gzip the whole content and output the result to out.tgz file. This way “pv” knows how much data is still left to be processed and shows us that it will take yet another 4 mins 49 secs to finish.

Another fine example is copying large amounts of data over network by using help of “nc” utility that I will write about some other time.

Suppose you have two computers A and B. You want to transfer a directory from A to B very quickly. The fastest way is to use tar and nc, and time the operation with pv.

# on computer A, with IP address 192.168.1.100
$ tar -cf - /path/to/dir | pv | nc -l -p 6666 -q 5

# on computer B
$ nc 192.168.1.100 6666 | pv | tar -xf -

That’s it. All the files in /path/to/dir on computer A will get transferred to computer B, and you’ll be able to see how fast the operation is going.

If you want the progress bar, you have to do the “pv -s $(…)” trick from the previous example (only on computer A).

Another funny example is by my blog reader alexandru. He shows how to time how fast the computer reads from /dev/zero:

$ pv /dev/zero > /dev/null
 157GB 0:00:38 [4,17GB/s]

That’s about it. I hope you enjoyed my examples and learned something new. I love explaining things and teaching! :)

How to install pv?

If you’re on Debian or Debian based system such as Ubuntu do the following:

$ sudo aptitude install pv

If you’re on Fedora or Fedora based system such as CentOS do:

$ sudo yum install pv

If you’re on Slackware, go to pv homepage, download the pv-version.tar.gz archive and do:

$ tar -zxf pv-version.tar.gz
$ cd pv-version
$ ./configure && sudo make install

If you’re a Mac user:

$ sudo port install pv

If you’re OpenSolaris user:

$ pfexec pkg install pv

If you’re a Windows user on Cygwin:

$ ./configure
$ export DESTDIR=/cygdrive/c/cygwin
$ make
$ make install

The manual of the utility can be found here man pv.

Have fun measuring your pipes with pv, and until next time!

A question to my readers: what other not so well known Unix utilities do you use and/or know about?

When I was developing the reddit media: intelligent fun online website, I needed to embed reddit’s up/down voting buttons to allow users to cast votes on media links directly from the site.

I remembered that reddit had decided not to display posts with a submission time less than two hours ago.

This left me thinking, if the scores are not displayed for new posts, what’s the point of having vote boxes on a just posted article page? I thought, it wouldn’t make sense if it wasn’t available. Quickly did I find a link on reddit’s new page which seemed to have received a few votes and added a reddit’s button to an empty HTML document.

A reddit voting button/widget can be embedded on a site by putting the following JavaScript code fragment anywhere in the HTML source:

<script>reddit_url='[URL]'</script>
<script>reddit_title='[TITLE]'</script>
<script language="javascript" src="http://reddit.com/button.js?t=2"></script>

where the URL is the URL to the article and TITLE is the title of the article.

Voilà! I now know something nobody else did - how many votes had the post received!

NOW, let’s create something cool for general public to use so that anyone could reveal the scores for all recently posted links :)

Continue reading ‘Revealing Reddit Score for Just Posted Links with FireFox and GreaseMonkey’