traffic shaping with iptables

A few years ago I worked as a Linux system administrator at a small (few hundred users) Internet service provider. Among all the regular system administrator duties, I also had the privilege to write various software and tools for Linux. One of my tasks was to write a tool to record how much traffic each of the clients was using.

The network for this provider was laid out in a very simple way. The gateway to the Internet was a single Linux box, which was a router, a firewall and performed traffic shaping. Now it had to be extended to do traffic accounting as well.

isp network diagram
Simplified network diagram, all that matters is that the gateway is a Linux box.

At that time I had already mastered IPTables and I had noticed that when listing the existing rules, iptables would display packet count and total byte count for each rule. I thought, yeah, why not use this for accounting? So I did. I created an empty rule (which gets passed through the firewall) for each IP address of users and a script which extracted the byte count.

Here is a detailed explanation of how I did it exactly.

First, let's see what iptables shows us when we have just booted up.

# iptables -L -n -v -x
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination

At this moment we have no rules added. That's alright, let's just get familiar with the output we will be interested in when we have some rules. Notice the 'pkts' and 'bytes' columns. The 'pkts' stands for packets and displays the total number of packets matched by the rule. The 'bytes' stands for total number of bytes matched by the rule. Notice also three so called "chains" - INPUT, FORWARD and OUTPUT. The INPUT chain is for packets destinated to the Linux box itself, OUTPUT chain is for packets leaving the Linux box (generated by programs running on the Linux box) and FORWARD is for packets passing through the box.

You might also be interested in the command line arguments that I used:

  • -L lists all the rules.
  • -n does not resolve the ip addresses.
  • -v lists the packet and byte count.
  • -x displays the byte count (otherwise it gets abbreviated to 200K, 3M, etc).

A more serious firewall might have the FORWARD chain filled up with various entries already. Not to mess with them, let's create a new traffic accounting chain called TRAFFIC_ACCT:

# iptables -N TRAFFIC_ACCT
# iptables -L -n -v -x
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination

Chain TRAFFIC_ACCT (0 references)
    pkts      bytes target     prot opt in     out     source               destination

Now let's redirect all the traffic going through the machine to match the rules in the TRAFFIC_ACCT chain:

# iptables -I FORWARD -j TRAFFIC_ACCT
# iptables -L -n -v -x
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination
       0        0 TRAFFIC_ACCT  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination

Chain TRAFFIC_ACCT (0 references)
    pkts      bytes target     prot opt in     out     source               destination

A side note: if you had a Linux desktop computer, then you could insert the same rule in the INPUT chain (iptables -I INPUT -j TRAFFIC_ACCT) as all the packets would be destinated for your computer.

IPTables command argument -L can actually take the name of a chain to list the rules from. From now on we will only be interested in rules of TRAFFIC_ACCT chain:

# iptables -L -n -v -x
Chain TRAFFIC_ACCT (1 references)
    pkts      bytes target     prot opt in     out     source               destination

Now to illustrate the main idea, we can play with the rules. For example, let's do the breakdown of traffic by tcp, udp and icmp protocols. To do that we insert three rules in the TRAFFIC_ACCT chain - one to match tcp protocol, one to match udp protocol and the last one to match icmp protocol.

# iptables -A TRAFFIC_ACCT -p tcp
# iptables -A TRAFFIC_ACCT -p udp
# iptables -A TRAFFIC_ACCT -p icmp

After some time has passed, let's look at what we have:

# iptables -L TRAFFIC_ACCT -n -v -x
Chain TRAFFIC_ACCT (1 references)
    pkts      bytes target     prot opt in     out     source               destination
    4356  2151124            tcp  --  *      *       0.0.0.0/0            0.0.0.0/0
     119    15964            udp  --  *      *       0.0.0.0/0            0.0.0.0/0
       3      168            icmp --  *      *       0.0.0.0/0            0.0.0.0/0

We see that 4356 tcp packets totaling 2151124 bytes (2 megabytes) have passed through the firewall, 119 udp packets and 3 icmp packets!

You can zero out the counters with -Z iptables command:

# iptables -Z TRAFFIC_ACCT
# iptables -L TRAFFIC_ACCT -n -v -x
Chain TRAFFIC_ACCT (1 references)
    pkts      bytes target     prot opt in     out     source               destination
       0        0            tcp  --  *      *       0.0.0.0/0            0.0.0.0/0
       0        0            udp  --  *      *       0.0.0.0/0            0.0.0.0/0
       0        0            icmp --  *      *       0.0.0.0/0            0.0.0.0/0

You can remove all the rules from TRAFFIC_ACCT chain with -F iptables command:

# iptables -F TRAFFIC_ACCT
# iptables -L TRAFFIC_ACCT -n -v -x
Chain TRAFFIC_ACCT (1 references)
    pkts      bytes target     prot opt in     out     source               destination

Another fun example you can do is count how many actual connections have been made:

# iptables -A TRAFFIC_ACCT -p tcp --syn
# iptables -L -n -v -x
Chain TRAFFIC_ACCT (1 references)
    pkts      bytes target     prot opt in     out     source               destination
       5      276            tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp flags:0x16/0x02

Shows us that 5 tcp packets which start the connections have been sent. Pretty neat, isn't it?

What I did when I was working as a sysadmin, was add user IP addresses to the TRAFFIC_ACCT chain. Then, I periodically listed and recorded traffic, and zero'ed it out.

You can even create two chains TRAFFIC_ACCT_IN and TRAFFIC_ACCT_OUT to match incoming and outgoing traffic.

# iptables -N TRAFFIC_ACCT_IN
# iptables -N TRAFFIC_ACCT_OUT
# iptables -I FORWARD -i eth0 -j TRAFFIC_ACCT_IN
# iptables -I FORWARD -o eth0 -j TRAFFIC_ACCT_OUT
# iptables -L -n -v -x
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination
       0        0 TRAFFIC_ACCT_OUT  all  --  *      eth0    0.0.0.0/0            0.0.0.0/0
       0        0 TRAFFIC_ACCT_IN  all  --  eth0   *       0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination

Chain TRAFFIC_ACCT_IN (1 references)
    pkts      bytes target     prot opt in     out     source               destination

Chain TRAFFIC_ACCT_OUT (1 references)
    pkts      bytes target     prot opt in     out     source               destination

For example, to record incoming and outgoing traffic usage of IP addresses 192.168.1.2 and 192.168.1.3 you would do:

# iptables -A TRAFFIC_ACCT_IN --dst 192.168.1.2
# iptables -A TRAFFIC_ACCT_IN --dst 192.168.1.3
# iptables -A TRAFFIC_ACCT_OUT --src 192.168.1.2
# iptables -A TRAFFIC_ACCT_OUT --src 192.168.1.2

And to list the rules:

# iptables -L TRAFFIC_ACCT_IN -n -v -x
Chain TRAFFIC_ACCT_IN (1 references)
    pkts      bytes target     prot opt in     out     source               destination
     368   362120            all  --  *      *       0.0.0.0/0            192.168.1.2
      61     9186            all  --  *      *       0.0.0.0/0            192.168.1.3

# iptables -L TRAFFIC_ACCT_OUT -n -v -x
Chain TRAFFIC_ACCT_OUT (1 references)
    pkts      bytes target     prot opt in     out     source               destination
     373    22687            all  --  *      *       192.168.1.2          0.0.0.0/0
     101    44711            all  --  *      *       192.168.1.3          0.0.0.0/0

That concludes it. You see that it is trivial to do accurate traffic accounting on a Linux machine. In a future post, I might publish a program which displays the traffic in a nice, visual, manner.

At the moment you can output the traffic in a nice manner with this combination of iptables and awk commands:

# iptables -L TRAFFIC_ACCT_IN -n -v -x | awk '$1 ~ /^[0-9]+$/ { printf "IP: %s, %d bytes\n", $8, $2 }'
IP: 192.168.1.2, 1437631 bytes
IP: 192.168.1.3, 449554 bytes

# iptables -L TRAFFIC_ACCT_OUT -n -v -x | awk '$1 ~ /^[0-9]+$/ { printf "IP: %s, %d bytes\n", $7, $2 }'
IP: 192.168.1.2, 88202 bytes
IP: 192.168.1.3, 244848 bytes

I first learned IPTables from this tutorial. It's probably the best tutorial one can find on the subject.

If you did not understand any parts of the article, please let me know in the comments. I will update the post and explain those parts.

Comments

Josh Permalink
August 06, 2008, 15:06

I really enjoyed this article. It's rather refreshing when you come across something that is useful in real world situations (and explained very well)! Thanks!

Sudarshan Permalink
August 07, 2008, 08:26

Nice article. We laid out and explained.

/not a linux user
/not a sys admin
/still found your article to be interesting

Sudarshan Permalink
August 07, 2008, 08:26

*Well laid out

Rahul Permalink
August 12, 2008, 20:17

Introduced me to the wonderful iptables command in LINUX.

Thanx

September 29, 2008, 10:39

This is cool except I cannot get this to work properly with my LVS setup.

HTTP requests come in on a unique IP, and I want ot log the traffic by that IP. However, it looks like in the transition (since the traffic is load balanced using ipvsadm/iptables/LVS into internal servers using NAT and then forwarded back out) it either loses that or I can't seem to figure it out. Hrm.

Valmik Roy Permalink
June 22, 2009, 05:43

Thanks for sharing this trick , it works well :)

Sergei Tvorogov Permalink
September 23, 2009, 11:28

Great work!
All simple and clear!

Phil Permalink
February 10, 2010, 10:49

Good stuff. Well written and useful.

hjs Permalink
April 19, 2010, 22:39

Let us do some real world accounting: Given iptable rules from the famous Linux IP-Masquerade-HOWTO [1], i.e.

IPTABLES=/sbin/iptables 
EXTIF="ppp0"
INTIF="eth1"
EXTIP="xxx.xxx.xxx.xxx"
INTNET="192.168.0.0/24"
INTIP="192.168.0.1/32"
UNIVERSE="0.0.0.0/0"
$IPTABLES -A INPUT -i lo -s $UNIVERSE -d $UNIVERSE -j ACCEPT
$IPTABLES -A INPUT -i $INTIF -s $INTNET -d $UNIVERSE -j ACCEPT
$IPTABLES -A INPUT -i $EXTIF -s $INTNET -d $UNIVERSE -j DROP
$IPTABLES -A INPUT -i $EXTIF -s $UNIVERSE -d $EXTIP -m state --state \
 ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A INPUT -s $UNIVERSE -d $UNIVERSE -j DROP
$IPTABLES -A OUTPUT -m state -p icmp --state INVALID -j DROP
$IPTABLES -A OUTPUT -o lo -s $UNIVERSE -d $UNIVERSE -j ACCEPT
$IPTABLES -A OUTPUT -o $INTIF -s $EXTIP -d $INTNET -j ACCEPT
$IPTABLES -A OUTPUT -o $INTIF -s $INTIP -d $INTNET -j ACCEPT
$IPTABLES -A OUTPUT -o $EXTIF -s $UNIVERSE -d $INTNET -j DROP
$IPTABLES -A OUTPUT -o $EXTIF -s $EXTIP -d $UNIVERSE -j ACCEPT
$IPTABLES -A OUTPUT -s $UNIVERSE -d $UNIVERSE -j DROP
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -m state --state \ 
  ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
$IPTABLES -A FORWARD -j DROP
$IPTABLES -t nat -A POSTROUTING -o $EXTIF -j SNAT --to $EXTIP

For this firewall

iptables -L -nvx

returns something like the following

Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts    bytes target   prot opt in     out     source           destination          
    0      (a) ACCEPT   all  --  lo     *       0.0.0.0/0        0.0.0.0/0           
11940      (b) ACCEPT   all  --  eth1   *       192.168.0.0/24   0.0.0.0/0           
    0      (i) REJECT   all  --  ppp0   *       192.168.0.0/24   0.0.0.0/0        reject-with icmp-port-unreachable 
 1147     (ii) ACCEPT   all  --  ppp0   *       0.0.0.0/0        xxx.xxx.xxx.xxx state RELATED,ESTABLISHED 
  174    (iii) REJECT   all  --  *      *       0.0.0.0/0        0.0.0.0/0       reject-with icmp-port-unreachable 

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts    bytes target   prot opt in     out     source           destination         
50119     (iv) ACCEPT   all  --  ppp0   eth1    0.0.0.0/0        0.0.0.0/0       state RELATED,ESTABLISHE 
42148      (c) ACCEPT   all  --  eth1   ppp0    0.0.0.0/0        0.0.0.0/0       
    0       0  REJECT   all  --  *      *       0.0.0.0/0        0.0.0.0/0       reject-with icmp-port-unreachable 

Chain OUTPUT (policy DROP 0 packets, 0 bytes)
 pkts    bytes target   prot opt in     out     source           destination         
   21    1608  DROP     icmp --  *      *       0.0.0.0/0        0.0.0.0/0       state INVALID
    0      (d) ACCEPT   all  --  *      lo      0.0.0.0/0        0.0.0.0/0           
    0       0  ACCEPT   all  --  *      eth1    xxx.xxx.xxx.xxx  192.168.0.0/24      
 7620  655618  ACCEPT   all  --  *      eth1    192.168.0.1      192.168.0.0/24      
    0       0  REJECT   all  --  *      ppp0    0.0.0.0/0        192.168.0.0/24  reject-with icmp-port-unreachable
 1331      (e) ACCEPT   all  --  *      ppp0    xxx.xxx.xxx.xxx  0.0.0.0/0           
    0      (v) REJECT   all  --  *      *       0.0.0.0/0        0.0.0.0/0  

NB.: Some numbers of bytes were replaced by (a), (b),..., (e), (i), (ii),..., (v).

Then

Upload  =(a)+ (b)+...+(e)
Download=(i)+(ii)+...+(v)

Would you confirm?

Regards
HJS

[1] http://tldp.org/HOWTO/IP-Masquerade-HOWTO/stronger-firewall-examples.html#RC.FIREWALL-IPTABLES-STRONGER

boby Permalink
September 14, 2010, 17:28

hi
for my site i need to get usage bytes of a specific IP, which options do that with your command:
iptables -L -nvx|awk...
thanks in advance

efix Permalink
September 16, 2010, 12:32

"# iptables -L TRAFFIC_ACCT_OUT -n -v -x | awk '$1 ~ /^[0-9]+$/ { printf "IP: %s, %d bytes\n", $7, $2 }'"
this command works in console perfectly, but output file is all in one line (not column). Anyone help me make column like in console?

jee Permalink
June 12, 2011, 09:30

output file is in tabular (column) mode too. The printf command uses '\n' to add new lines.

Maybe you are trying to open this file on a windows machine? If so, use command "unix2dos" or sed equivalent.

Felipe Echeverria Permalink
October 01, 2010, 16:50

Thanks a lot for the tutorial, this is the best for IP tables introduction. Now i can handle accounting for my so expensive 3g plan.

Regards.

Felipe

Rashid Iqbal Permalink
December 23, 2011, 17:03

I am trying to configure the remote desktop of windows machine behind the squid proxy

#iptables -nvL
222K 10M ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3389
431K 179M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
#iptables -t nat -nvL
[root@2share-proxy ~]# iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 2861K packets, 248M bytes)
pkts bytes target prot opt in out source destination
1 48 DNAT tcp -- * * 0.0.0.0/0 192.168.1.100 tcp dpt:3389 to:192.168.3.61:3389

#cat /proc/net/ip_conntract | grep 3389
tcp 6 76 SYN_SENT src=212.100.219.15 dst=192.168.3.61 sport=3275 dport=3389 packets=3 bytes=144 [UNREPLIED] src=192.168.3.61 dst=212.100.219.15 sport=3389 dport=3275 packets=0 bytes=0 mark=0 secmark=0 use=1

Can you please help me to solve this issue

Peter Permalink
August 29, 2012, 19:02

the IPTables tutorial mentioned at the end of the article is now at http://www.frozentux.net/documents/iptables-tutorial/.

November 28, 2013, 08:52

Note that not all your IP traffic will be accounted that way but only your IPv4 traffic.

If you want to monitor IPv6 traffic as well you'd have to use the same commands with "ip6tables" and sum up the results.

User Name Permalink
August 21, 2014, 17:45

Hi, is there any chance to count traffic per IP automatically? I want to have traffic usage per user on my server. Some of them cause high load and I need the way to block some greedy ones. Theoretically I can parse access.log file of nginx and add them all as separate rules but will not it be a problem? (since I will have over 5000 rules)

Jake Permalink
December 11, 2014, 16:41

The link you have for learning iptables no longer exists. I would advise using www.iptables.info.

Leave a new comment

(why do I need your e-mail?)

(Your twitter name, if you have one. (I'm @pkrumins, btw.))

Type the word "halflife3_81": (just to make sure you're a human)

Please preview the comment before submitting to make sure it's OK.

Advertisements