Hey everyone, just wanted to do a quick post on how to keep track of who's talking about you on the net. Nothing really unique, just a list of tools that I use often. Why is it important? Well, it's always interesting to know what people are saying about you and sometimes you want to engage in a conversation or just thank them for linking to your article.

Alright, here are the tools that I use:

Twitter Search

Twitter search is definitely the #1 source for keeping track of who's talking about you right now. But you already knew that.

Twitter Search
Twitter search example for the term "catonmat."

Perhaps what you didn't know is that they have an RSS feed for search queries.

Twitter search RSS feed
Location of RSS feed link for Twitter search results.

Now combined with a service like feedblitz.com you can email the RSS updates to yourself or just read them from your favorite RSS reader.

I am monitoring terms "Peteris Krumins", "pkrumins" and "catonmat".

Google Alerts

Google Alerts automatically notifies you when the Google search engine locates new results for your search terms. You can choose to have your alerts delivered via email or RSS feed.

Google Alerts
Google Alerts email for the term "catonmat."

You can even customize the type of alerts you wish to receive. Google Alerts lets you choose to get notified when a new result appears on web pages, usenet (google groups), blogs, news or videos.

Backtype Comment Alerts

Backtype is Google for comments. Want to find out when someone's mentioned you on Reddit, FriendFeed, Digg or Hacker News? Backtype will alert you.

Backtype Comment Alerts
Backtype Alerts email for the term "peteris."

Backtype also recently launched a service called BackTweets that allows you to find who's linking back to you via shortened URLs.

Have Fun!

Have fun keeping track of yourself!

Btw, let me know in the comments if I missed any other cool tools.

hackers steal moneyI recently watched an interesting video lecture on stealing botnets. A group of researchers at UCSB recently managed to take control over a part of Torpig botnet for 10 days. During this time, they observed 180 thousand infections and recorded almost 70GB of data that bots collected. This data included submitted form information from all the websites the infected person had visited, smtp, ftp, pop3, windows, passwords, credit card numbers and passwords from various password managers.

Here are the most interesting facts from the lecture:

Torpig uses a technique called "domain fluxing" to avoid being shut down by simply blocking the IP or the domain name of control center servers. The idea is simple - depending on date and time the algorithm generates a domain name to connect to. If the domain gets shut down, the bots will simply use a different domain after some time.

The researchers were able to take control over a part of the botnet by cracking the domain name generating algorithm and registering some of the domain names to be used for communication in the future.

The bad guys noticed that a part of botnet has been taken over and issued a software update to all bots to use a new domain flux algorithm, which used Twitter's popular topics for the day to generate domain names. It was no longer possible to predict the domain that would be used tomorrow.

When communicating with command & control server, the bots included a unique id field that was generated from machine's hardware. This allowed researchers to estimate the real number of unique computers infected. Researchers saw 1.2 million unique IP addresses but only 180k unique machines.

The bots would steal financial data from 410 financial institutions (top 5: PayPal, Poste Italiane, Capital One, E*Trade, Chase), they would log credit card information (top 5 cards: Visa, Mastercard, American Express, Maestro, Discover), and they would also steal all the passwords from browser's password manager.

In a 2008 study Symantec estimated that credit card information is valued at $.10 to $25 per card in the underground market. The bank account information is valued at $10.00 to $1,000 per account. Using this study, researchers estimated that during 10 day period the amount of financial data bots collected were worth $83k to $8.3 million.

Using various estimations researchers calculated that if the bots are used for denial of service the total bandwidth would be 17Gbps.

Researchers observed that there was a fraction of people who'd fill out the phishing page and then immediately email the company's security group telling that they may have been victims of identity theft.

Since Torpig was sending all the HTTP POST data and emails to command & control servers, researchers did statistics on emails and found out that 14% of all captured emails were about jobs and resumes, 10% discussed computer security/malware, 7% discussed money, 6% were sports fans, 5% were worried about exams and their grades, 4% were seeking partners online.

Researchers collected 300,000 unique credentials on 370,000 websites. 28% of people reused their password on multiple domains. There were 173,686 unique passwords.

Researchers converted the passwords in Unix format and tried to crack them with John the Ripper. 56,000 were cracked in less than 65 minutes using brute-force. Using a wordlist 14,000 passwords were cracked in the next 10 minutes. And another 30,000 passwords were cracked in the next 24 hours. That's 58% of all passwords cracked in 24 hours.

You're welcome to watch the video lecture. It's 1h 15m long. It's presented by Richard A. Kemmerer.

Here are all the topics in the lecture:

  • [02:00] Botnet terminology - bot, botnet, command & control server, control channel, botmaster.
  • [03:00] Introduction to the Torpig trojan and Mebroot malware platform.
  • [05:00] How Torpig works.
  • [11:30] Torpig HTML injection.
  • [15:00] Domain fluxing.
  • [19:15] Taking over Torpig's c&c server.
  • [24:10] Data collection principles.
  • [26:00] C&c server protocol.
  • [31:10] Botnet's size estimation.
  • [37:00] Botnet's threats: theft of financial information, denial of service, proxy servers, privacy thefts.
  • [37:30] Threat: Theft of financial information.
  • [42:00] Threat: Denial of service.
  • [43:30] Threat: Proxy servers.
  • [44:20] Threat: Privacy theft.
  • [47:00] Password analysis.
  • [50:40] Criminal retribution.
  • [53:00] Law enforcement.
  • [58:00] Repatriating the data.
  • [01:00:00] Ethics.
  • [01:02:00] Conclusions.
  • [01:06:00] Questions and answers.

For more information see the publication "Your Botnet is My Botnet: Analaysis of a Botnet Takeover."

This article is part of the article series "MIT Linear Algebra."
<- previous article next article ->

MIT Introduction to Linear AlgebraThis is the fifth post in an article series about MIT's course "Linear Algebra". In this post I will review lecture five that finally introduces real linear algebra topics such as vector spaces their subspaces and spaces from matrices. But before it does that it closes the topics that were started in the previous lecture on permutations, transposes and symmetric matrices.

Here is a list of the previous posts in this article series:

Lecture 5: Vector Spaces and Subspaces

Lecture starts with reminding some facts about permutation matrices. Remember from the previous lecture that permutation matrices P execute row exchanges and they are identity matrices with reordered rows.

Let's count how many permutation matrices are there for an nxn matrix.

For a matrix of size 1x1, there is just one permutation matrix - the identity matrix.

For a matrix of size 2x2 there are two permutation matrices - the identity matrix and the identity matrix with rows exchanged.

For a matrix of size 3x3 we may have the rows of the identity matrix rearranged in 6 ways - {1,2,3}, {1,3,2}, {2,1,3}, {2,3,1}, {3,1,2}, {3,2,1}.

For a matrix of size 4x4 the number of ways to reorder the rows is the same as the number of ways to rearrange numbers {1,2,3,4}. This is the simplest possible combinatorics problem. The answer is 4! = 24 ways.

In general, for an nxn matrix, there are n! permutation matrices.

Another key fact to remember about permutation matrices is that their inverse P-1 is their transpose PT. Or algebraically PT·P = I.

The lecture proceeds to transpose matrices. The transpose of a matrix exchanges its columns with rows. Another way to think about it that it flips the matrix over its main diagonal. Transpose of matrix A is denoted by AT.

Here is an example of transpose of a 3-by-3 matrix. I color coded the columns to better see how they get exchanged:

Transpose A^T of a 3x3 matrix A

A matrix does not have to be square for its transpose to exist. Here is another example of transpose of a 3-by-2 matrix:

Transpose A^T of a 3x2 matrix A

In algebraic notation transpose is expressed as (AT)ij = Aji, which says that an element aij at position ij get transposed into the position ji.

Here are the rules for matrix transposition:

  • The transpose of A + B is (A + B)T = AT + BT.
  • The transpose of A·B is (A·B)T = BT·AT.
  • The transpose of A·B·C is (A·B·C)T = CT·BT·AT.
  • The transpose of A-1 is (A-1)T = (AT)-1.

Next the lecture continues with symmetric matrices. A symmetric matrix has its transpose equal to itself, i.e., AT = A. It means that we can flip the matrix along the diagonal (transpose it) but it won't change.

Here is an example of a symmetric matrix. Notice that the elements on opposite sides of the diagonal are equal:

Symmetric matrix

Now check this out. If you have a matrix R that is not symmetric and you multiply it with its transpose RT as R·RT, you get a symmetric matrix! Here is an example:

Matrix times its transpose is symmetric matrix

Are you wondering why it's true? The proof is really simple. Remember that matrix is symmetric if its transpose is equal to itself. Now what's the transpose of the product R·RT? It's (R·RT)T = (RT)T·RT = R·RT - it's the same product, which means that R·RT is always symmetric.

Here is another cool fact - the inverse of a symmetric matrix (if it exists) is also symmetric. Here is the proof. Suppose A is symmetric, then the transpose of A-1 is (A-1)T = (AT)-1. But AT = A, therefore (AT)-1 = A-1.

At this point lecture finally reaches the fundamental topic of linear algebra - vector spaces. As usual, it introduces the topic by examples.

Example 1: Vector space R2 - all 2-dimensional vectors. Some of the vectors in this space are (3, 2), (0, 0), (π, e) and infinitely many others. These are all the vectors with two components and they represent the xy plane.

Example 2: Vector space R3 - all vectors with 3 components (all 3-dimensional vectors).

Example 3: Vector space Rn - all vectors with n components (all n-dimensional vectors).

What makes these vectors vector spaces is that they are closed under multiplication by a scalar and addition, i.e., vector space must be closed under linear combination of vectors. What I mean by that is if you take two vectors and add them together or multiply them by a scalar they are still in the same space.

For example, take a vector (1,2,3) in R3. If we multiply it by any number α, it's still in R3 because α·(1,2,3) = (α, 2α, 3α). Similarly, if we take any two vectors (a, b, c) and (d, e, f) and add them together, the result is (a+d, b+e, f+c) and it's still in R3.

There are actually 8 axioms that the vectors must satisfy for them to make a space, but they are not listed in this lecture.

Here is an example of not-a-vector-space. It's 1/4 of R2 (the 1st quadrant). The green vectors are in the 1st quadrant but the red one is not:

Not a vector space
An example of not-a-vector-space.

This is not a vector space because the green vectors in the space are not closed under multiplication by a scalar. If we take the vector (3,1) and multiply it by -1 we get the red vector (-3, -1) but it's not in the 1st quadrant, therefore it's not a vector space.

Next, Gilbert Strang introduces subspaces of vector spaces.

For example, any line in R2 that goes through the origin (0, 0) is a subspace of R2. Why? Because if we take any vector on the line and multiply it by a scalar, it's still on the line. And if we take any two vectors on the line and add them together, they are also still on the line. The requirement for a subspace is that the vectors in it do not go outside when added together or multiplied by a number.

Here is a visualization. The blue line is a subspace of R2 because the red vectors on it can't go outside of line:

Subspace of R2
An example of subspace of R2.

And example of not-a-subspace of R2 is any line that does not go through the origin. If we take any vector on the line and multiply it by 0, we get the zero vector, but it's not on the line. Also if we take two vectors and add them together, they are not on the line. Here is a visualization:

Not a vector subspace
An example of not-a-subspace of R2.

Why not list all the subspaces of R2. They are:

  • the R2 itself,
  • any line through the origin (0, 0),
  • the zero vector (0, 0).

And all the subspaces of R3 are:

  • the R3 itself,
  • any line through the origin (0, 0, 0),
  • any plane through the origin (0, 0, 0),
  • the zero vector.

The last 10 minutes of the lecture are spent on column spaces of matrices.

The column space of a matrix is made out of all the linear combinations of its columns. For example, given this matrix:

Matrix a

The column space C(A) is the set of all vectors {α·(1,2,4) + β·(3,3,1)}. In fact, this column space is a subspace of R3 and it forms a plane through the origin.

More about column spaces in the next lecture.

You're welcome to watch the video lecture five:

Topics covered in lecture five:

  • [01:30] Permutations.
  • [03:00] A=LU elimination without row exchanges.
  • [03:50] How Matlab does A=LU elimination.
  • [04:50] PA=LU elimination with row exchanges
  • [06:40] Permutation matrices.
  • [07:25] How many permutation matrices are there?
  • [08:30] Permutation matrix properties.
  • [10:30] Transpose matrices.
  • [11:50] General formula for transposes: (AT)ij = Aji.
  • [13:06] Symmetric matrices.
  • [13:30] Example of a symmetric matrix.
  • [15:15] R·RT is always symmetric.
  • [18:23] Why is R·RT symmetric?
  • [20:50] Vector spaces.
  • [22:05] Examples of vector spaces.
  • [22:55] Real vector space R2.
  • [23:20] Picture of R2 - xy plane.
  • [26:50] Vector space R3.
  • [28:00] Vector space Rn.
  • [30:00] Example of not a vector space.
  • [32:00] Subspaces of vector spaces.
  • [33:00] A vector space inside R2.
  • [34:35] A line in R2 that is subspace.
  • [34:50] A line in R2 that is not a subspace.
  • [36:30] All subspaces of R2.
  • [39:30] All subspaces of R3.
  • [40:20] Subspaces of matrices.
  • [41:00] Column spaces of matrices C(A).
  • [44:10] Example of column space of matrix with columns in R3.

Here are my notes of lecture five:

MIT Linear Algebra, Lecture 5: Vector Spaces and Subspaces
My notes of linear algebra lecture 5 on vector spaces and subspaces.

Have fun with this lecture! The next post is going to be more about column spaces and null spaces of matrices.

PS. This course is taught from Introduction to Linear Algebra textbook. Get it here:

This article is part of the article series "Vim Plugins You Should Know About."
<- previous article next article ->

Vim Plugins, surround.vimThis is the sixth post in the article series "Vim Plugins You Should Know About". This time I am going to introduce you to a vim plugin called "nerd_tree.vim". It's so useful that I can't imagine working without it in vim.

Nerd Tree is a nifty plugin that allows you to explore the file system and open files and directories directly from vim. It opens the file system tree in a new vim window and you may use keyboard shortcuts and mouse to open files in new tabs, in new horizontal and vertical splits, quickly navigate between directories and create bookmarks for your most important projects.

This plugin was written by Marty Grenfell (also known as scrooloose).

Previous articles in the series:

Ps. Please help me reach 10,000 RSS subscribers. I am almost there. If you enjoy my posts and have not yet subscribed, subscribe here!

How to use nerd_tree.vim?

Nerd Tree plugin can be activated by the :NERDTree vim command. It will open in vim as a new vertical split on the left:

Vim Nerd Tree
A screenshot of Nerd Tree plugin in action.

Here are the basics of how to use the plugin:

  • Use the natural vim navigation keys hjkl to navigate the files.
  • Press o to open the file in a new buffer or open/close directory.
  • Press t to open the file in a new tab.
  • Press i to open the file in a new horizontal split.
  • Press s to open the file in a new vertical split.
  • Press p to go to parent directory.
  • Press r to refresh the current directory.

All other keyboard shortcuts can be found by pressing ?. It will open a special help screen with the shortcut listings. Press ? again to get back to file tree.

To close the plugin execute the :NERDTreeClose command.

Typing :NERDTree and :NERDTreeClose all the time is really inconvenient. Therefore I have mapped the toggle command :NERDTreeToggle to the F2 key. This way I can quickly open and close Nerd Tree whenever I wish. You can also map it to F2 by putting map <F2> :NERDTreeToggle<CR> in your .vimrc file.

How to install nerd_tree.vim?

To get the latest version:

  • 1. Download NERD_tree.zip.
  • 2. Extract NERD_tree.zip to ~/.vim (on Unix/Linux) or ~\vimfiles (on Windows).
  • 3. Run :helptags ~/.vim/doc (on Unix/Linux) or :helptags ~/vimfiles/doc (on Windows) to rebuild the tags file (so that you can read :help NERD_tree.)
  • 4. Restart Vim.

Have Fun!

Have fun exploring your files with this awesome plugin and until next time!

Fibonacci SunflowerI learned an interesting fact about Fibonacci numbers recently while watching a lecture on number theory. Fibonacci numbers can be used to approximately convert from miles to kilometers and back.

Here is how.

Take two consecutive Fibonacci numbers, for example 5 and 8. And you're done converting. No kidding – there are 8 kilometers in 5 miles. To convert back just read the result from the other end - there are 5 miles in 8 km!

Another example. Let's take the consecutive Fibonacci numbers 21 and 34. What this tells us is that there are approximately 34 km in 21 miles and vice versa. (The exact answer is 33.79 km.)

If you need to convert a number that is not a Fibonacci number, just express the original number as a sum of Fibonacci numbers and do the conversion for each Fibonacci number separately.

For example, how many kilometers are there in 100 miles? Number 100 can be expressed as a sum of Fibonacci numbers 89 + 8 + 3. Now, the Fibonacci number following 89 is 144, the Fibonacci number following 8 is 13 and the Fibonacci number following 3 is 5. Therefore the answer is 144 + 13 + 5 = 162 kilometers in 100 miles. This is less than 1% off from the precise answer, which is 160.93 km.

Another example, how many miles are there in 400 km? Well, 400 is 377 + 21 + 2. Since we are going the opposite way now from miles to km, we need the preceding Fibonacci numbers. They are 233, 13 and 1. Therefore there are 233 + 13 + 1 = 247 miles in 400 km. (The correct answer is 248.55 miles.)

Just remember that if you need to convert from km to miles, you need to find the preceding Fibonacci number. But if you need to convert from miles to km, you need the subsequent Fibonacci number.

If the distance you're converting can be expressed as a single Fibonacci number, then for numbers greater than 21 the error is always around 0.5%. However, if the distance needs to be composed as a sum of n Fibonacci numbers, then the error will be around sqrt(n)·0.5%.

Here's why it works.

Fibonacci numbers have a property that the ratio of two consecutive numbers tends to the Golden ratio as numbers get bigger and bigger. The Golden ratio is a number and it happens to be approximately 1.618.

Coincidentally, there are 1.609 kilometers in a mile, which is within 0.5% of the Golden ratio.

Now that we know these two key facts, we can figure out how to do the conversion. If we take two consecutive Fibonacci numbers, Fn+1 and Fn, we know that their ratio Fn+1/Fn is approximately 1.618. Since the ratio is also almost the same as kilometers per mile, we can write Fn+1/Fn = [mile]/[km]. It follows that Fn·[mile] = Fn+1·[km], which translates to English as "n-th Fibonacci number in miles is the same as (n+1)-th Fibonacci number in kilometers".

That's all there is to it. A pure coincidence that the Golden ratio is almost the same as kilometers in a mile.