Dual Wan DHCP Tomato with Round Robbin Loadbalancing

I have been working on a dual wan setup for Tomato for a while now and have compiled and modified a few scripts from various sources. This setup enables a second wan on port 4 of the router configured for DHCP (PPPOE is possible with modifications to the script). QOS is only applied to the primary WAN (i.e. WAN2 no QOS). I haven't tested to see what effect port forwarding has as that has not been a priority of the project. This works on a Asus WL-500GPv2 running the following flavor of Tomato:

tomato-K26USB-1.27.9047MIPSR1-beta16-Ext available from: http://tomatousb.org/ (Thank you teddy_bear.)

Of course you are modifying your router so the standard disclaimer applies... it is possible you can brick your router, use at your own risk, however I haven't bricked one beyond recovery...yet. I recommend running the w2.init script from the command line first, before you add any lines to the Administration, Scripts web interface, this way a simple power cycle will restore the routing table and firewall if you run into problems, if you mess up the nvram port settings then look into router reset / recovery, unless of course you implemented the safeguard mentioned in the Removal section. This is a multi step process where first the new interface is created and then scripts are added to the 'Firewall' and 'WAN Up' scripts. Since the firewall and routing table is flushed and rebuilt every time the WAN comes up.

Prerequisites
2 wan connections jffs enabled and formatted. open port 4 on the router for wan 2 (you can change this) wan 1 is connected to WAN port Console access to router (ssh, telnet, etc...)

Removal
(This is first just in case...) Remove the w2.init and w2.evt scripts from Firewall and WAN Up respectively and run the following in a console:
 * as a safety precaution you may want to put this into the 12 second plus button script in case something goes wrong.

nvram set vlan0ports="0 1 2 3 5*" nvram set vlan1ports="4 5" nvram unset vlan2ports nvram unset vlan2hwname nvram unset wan2_iface nvram unset wan_weight nvram unset wan2_weight

nvram set script_fire="" nvram set script_wanup=""
 * 1) the following is optional and will remove all scripts from the Firewall and WAN Up... you are warned.

nvram set script_fire=`nvram get script_fire | grep -v jffs/wan2/w2.evt` nvram set script_wanup=`nvram get script_wanup | grep -v /jffs/wan2/w2.init`
 * 1) or if you stored your scripts as illustrated this will preserve your other scripts... I like this one better

nvram commit

rm -r /jffs/wan2
 * 1) for completeness delete the scripts folder

Setup Dual Wan Round Robbin Loadbalancing
First Create your 2nd wan interface: vlan2 on port 4 Box label: WAN 1 2 3 4 Nvram label: 4 3 2 1 0
 * Ports are labeled in reverse order i.e.

some of the newer routers may have different internal assignments, so you may want to run nvram show | grep vlan.ports and examine the output... before modification mine looks like: vlan0ports=1 2 3 4 5* vlan1ports=0 5

Using the console
nvram set vlan0ports="3 2 1 5*" nvram set vlan1ports="4 5" nvram set vlan2ports="0 5" nvram set vlan2hwname=et0 nvram set wan2_iface=vlan2

nvram set wan_weight=1 nvram set wan2_weight=1
 * 1) the following adjust the Round Robbin weighting of each respective interface

nvram commit

mkdir /jffs/wan2

Using the console and your favorite text editor create the following script and save it to
/jffs/wan2/w2.init


 * 1) !/bin/sh
 * 2) script to bring up the secondary wan interface

ModName="WAN2 init: "

runscr="/jffs/wan2/w2.evt" iface=`nvram get wan2_iface` lanhost=`nvram get lan_hostname`

kill `ps|grep $iface|grep -v grep|awk '{print $1}'` ifconfig $iface down sleep 5 logger $ModName starting... $iface ifconfig $iface up ip=`udhcpc -i $iface -s $runscr -H $lanhost | grep Lease | awk '{print $3}'`
 * 1) logger $ModName killing old: `ps|grep $iface|grep -v grep|awk '{print $1}'`
 * 1) ifconfig $iface up $ip

logger $ModName complete: $iface $ip

Using the console and your favorite text editor create the following script and save it to
/jffs/wan2/w2.evt


 * 1) !/bin/sh
 * 2) Dual Wan Script for Tomato
 * 3) Purpose: insert the firewall and route entries, update DNS for wan2
 * 4) ver. 0.5
 * 5) secondary wan comes up, or the firewall is rebuilt


 * 1) Assumptions:
 * 2) Wan 2 is DHCP
 * 3) the following entries exist in nvram and are configured properly ( name(default value) ):
 * 4) lan_ifname(br0)
 * 5) wan_iface(vlan1)
 * 6) wan2_iface(vlan2)
 * 7) *use these values to adjust the round robbin weighting for each wan:
 * 8) wan_weight(1)
 * 9) wan2_weight(1)
 * 10) vlan0ports(3 2 1 5*)
 * 11) vlan1ports(4 5)
 * 12) vlan2ports(0 5)
 * 13) vlan2hwname(et0)
 * 1) vlan2ports(0 5)
 * 2) vlan2hwname(et0)


 * 1) IF0= local interface
 * 2) P0_NET= local network


 * 1) IFx= name of the interface on WANx
 * 2) IPx= IP address associated with $IFx
 * 3) Px= be the IP address of the gateway at Provider x
 * 4) Px_NET= be the IP network $Px is in
 * 5) IFx_W= Weight to assign the interface #IFx for round robbin load balancing


 * 1) Example values for variables:


 * 1) Local lan: (this is for understanding only as most values are set using either nvram or dynamically determined)
 * 2) IF0=br0
 * 3) IP0=192.168.108.1
 * 4) P0_NET=192.168.108.0/24


 * 1) Wan #1: (this is for understanding only as most values are set using either nvram or dynamically determined)
 * 2) IF1=vlan1
 * 3) IF1_W=1
 * 4) IP1=192.168.8.115
 * 5) P1=192.168.8.1
 * 6) P1_NET=192.168.8.0/24


 * 1) Wan #2: (this is for understanding only as most values are set using either nvram or dynamically determined)
 * 2) IF2=vlan2
 * 3) IF2_W=1
 * 4) IP2=192.168.0.14
 * 5) P2=192.168.0.1
 * 6) P2_NET=192.168.0.0/24

RESOLV_CONF="/tmp/etc/resolv.dnsmasq"
 * 1) dnamasq DNS table path

WANname="WAN2 evt: "

T1=100 T2=200
 * 1) must use numbers for table names

Mrk1=0x100 Mrk2=0x200
 * 1) marks for rules/routes


 * 1) logger $WANname start $0 $1

case "$1" in	deconfig)		ifconfig $interface 0.0.0.0		nvram set wan2_get_dns=""		;;	renew|bound)

ifconfig $interface $ip ${broadcast:+broadcast $broadcast} ${subnet:+netmask $subnet}

if [ -n "$router" ] ; then

#echo "deleting routers" #while route del default gw 0.0.0.0 dev $interface ; do			#       : #done

# record the gateway for i in $router ; do				#route add default gw $i dev $interface #logger $WANname gateway $i nvram set wan2_gateway=$i done fi

# i don't think I want to lose my existing dns servers. ${domain:+echo search $domain >> $RESOLV_CONF}

for i in $dns ; do grep $i $RESOLV_CONF if [ $? -ne 0 ]; then logger $WANname adding dns $i echo nameserver $i >> $RESOLV_CONF nvram set wan2_get_dns=$i fi #if you use dyndns you may wan to use this. #wget -O /dev/null http://USER:PASSWORD@members.dyndns.org/nic/update?hostname=DOMAIN done

;; esac


 * 1) logger $WANname variable init

IF0=`nvram get lan_ifname_old` IP0=`nvram get lan_ip_old` P0_NET=`nvram get lan_net_old`
 * 1) retrieve all the old settings to use for iptables/route removal

IF1=`nvram get wan_iface_old` IP1=`nvram get wan_ip_old` P1=`nvram get wan_gateway_old` P1_NET=`nvram get wan_net_old` IF1_W=`nvram get wan_weight_old`

IF2=`nvram get wan2_iface_old` IP2=`nvram get wan2_ip_old` P2=`nvram get wan2_gateway_old` P2_NET=`nvram get wan2_net_old` IF2_W=`nvram get wan2_weight_old` IF2_DNS=`nvram get wan2_get_dns_old`


 * 1) since we want to preserve the existing routing table
 * 2) just remove what we are going to add

ip route del default scope global nexthop via $P1 dev $IF1 weight $IF1_W nexthop via $P2 dev $IF2 weight $IF2_W
 * 1) logger $WANname removing firewall and routes


 * 1) ip route del $P1_NET dev $IF1 src $IP1 table $T1
 * 2) ip route del default via $P1 table $T1
 * 3) ip route del $P2_NET dev $IF2 src $IP2 table $T2
 * 4) ip route del default via $P2 table $T2

ip route del $P1_NET dev $IF1 src $IP1 ip route del $P2_NET dev $IF2 src $IP2

ip route del default via $P1 ip route del default via $P2

ip route flush table $T1 ip route flush table $T2

ip rule flush ip rule add lookup main prio 32766 ip rule add lookup default prio 32767


 * 1) logger $WANname routes removed

iptables -D FORWARD $NDX -o $IF2 -j wanout iptables -D FORWARD $NDX -i $IF2 -j wanin iptables -D INPUT -j DROP -i $IF0 -d $IP2 iptables -t nat -D PREROUTING -j DROP -i $IF2 -d $P0_NET iptables -t nat -D PREROUTING -j DNAT -p icmp -d $P2 --to-destination $IP0 iptables -t mangle -D PREROUTING -i $IF1 -m state --state NEW -j CONNMARK --set-mark $Mrk1 iptables -t mangle -D PREROUTING -i $IF2 -m state --state NEW -j CONNMARK --set-mark $Mrk2 iptables -t mangle -D POSTROUTING -o $IF1 -m state --state NEW -j CONNMARK --set-mark $Mrk1 iptables -t mangle -D POSTROUTING -o $IF2 -m state --state NEW -j CONNMARK --set-mark $Mrk2 iptables -t mangle -D PREROUTING -i $IF0 -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark iptables -t mangle -D OUTPUT -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark iptables -D POSTROUTING -t nat -m mark --mark $Mrk1 -j SNAT --to-source $IP1 iptables -D POSTROUTING -t nat -m mark --mark $Mrk2 -j SNAT --to-source $IP2 iptables -t nat -D POSTROUTING -j MASQUERADE -o $IF2
 * 1) remove iptables rules


 * 1) logger firewall rules removed

IF0=`nvram get lan_ifname` IP0=`ifconfig $IF0 | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | awk -F. '$1 != 127 && $1 !=255 && $4 < 255{print $1 "." $2 "." $3 "." $4}'` P0_NET=`echo $IP0 | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | awk -F. '$1 != 127 && $1 !=255 && $4 < 255{print $1 "." $2 "." $3 ".0/24"}'`
 * 1) Fill Local lan variables

IF1=`nvram get wan_iface` IP1=`ifconfig $IF1 | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | awk -F. '$1 != 127 && $1 !=255 && $4 < 255{print $1 "." $2 "." $3 "." $4}'` P1=`nvram get wan_gateway` P1_NET=`echo $IP1 | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | awk -F. '$1 != 127 && $1 !=255 && $4 < 255{print $1 "." $2 "." $3 ".0/24"}'` IF1_W=`nvram get wan_weight`
 * 1) Fill Wan #1 variables

IF2=`nvram get wan2_iface` IP2=`ifconfig $IF2 | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | awk -F. '$1 != 127 && $1 !=255 && $4 < 255{print $1 "." $2 "." $3 "." $4}'` P2=`nvram get wan2_gateway` P2_NET=`echo $IP2 | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | awk -F. '$1 != 127 && $1 !=255 && $4 < 255{print $1 "." $2 "." $3 ".0/24"}'` IF2_W=`nvram get wan2_weight` IF2_DNS=`nvram get wan2_get_dns`
 * 1) Fill  Wan #2 variables

if [ "$1" != "deconfig" ]; then #logger $WANname appending firewall and routes
 * 1) add the routes and firewall rules

if [ "$IF2_DNS" != "" ]; then #Add the wan2 dns route if it exists nvram set wan2_get_dns_old=$IF2_DNS nvram set wan2_iface_old=$IF2 ip route add $IF2_DNS dev $IF2 fi if [ "$IP1" != "" ]; then nvram set wan_iface_old=$IF1 nvram set wan_ip_old=$IP1 nvram set wan_gateway_old=$P1 nvram set wan_net_old=$P1_NET nvram set wan_weight_old=$IF1_W # add rules first ip rule add from $IP1 table $T1 prio 100 ip rule add fwmark $Mrk1 table $T1 prio 101 # now add the routes... ip route add $P1_NET dev $IF1 src $IP1 table $T1 ip route add $P1_NET dev $IF1 src $IP1 ip route add default via $P1 table $T1 ip route add $P0_NET    dev $IF0 table $T1 ip route add 127.0.0.0/8 dev lo  table $T1 fi

if [ "$IP2" != "" ]; then # save values nvram set wan2_iface_old=$IF2 nvram set wan2_ip_old=$IP2 nvram set wan2_gateway_old=$P2 nvram set wan2_net_old=$P2_NET nvram set wan2_weight_old=$IF2_W # add rules first ip rule add from $IP2 table $T2 prio 200 ip rule add fwmark $Mrk2 table $T2 prio 201 # now add the routes... ip route add $P2_NET dev $IF2 src $IP2 table $T2 ip route add $P2_NET dev $IF2 src $IP2 ip route add default via $P2 table $T2 ip route add $P0_NET    dev $IF0 table $T2 ip route add 127.0.0.0/8 dev lo  table $T2 fi

# if both interfaces are up load balance if "$IP1" != "" && "$IP2" != "" ; then logger $WANname LoadBalance enabled ip route add $P2_NET    dev $IF2 table $T1 ip route add $P1_NET    dev $IF1 table $T2 # now apply weighted round robbin load balancing: ip route add default scope global nexthop via $P1 dev $IF1 weight $IF1_W nexthop via $P2 dev $IF2 weight $IF2_W else # see if we can add a default route if [ "$IP1" != "" ]; then logger $WANname LoadBalance disabled Default route via $IF1 ip route add default via $P1 fi

if [ "$IP2" != "" ]; then logger $WANname LoadBalance disabled Default route via $IF2 ip route add default via $P2 fi fi

ip route flush cache if [ "$IP2" != "" ]; then

#get the line number after wanout to insert the interface at the correct position NDX=`iptables -L FORWARD -v --line-numbers | sed -n "/wanout/h;$ {x;p;}" | awk '{print $1}'` NDX=`expr $NDX + 1`

iptables -I FORWARD $NDX -o $IF2 -j wanout iptables -I FORWARD $NDX -i $IF2 -j wanin iptables -I INPUT -j DROP -i $IF0 -d $IP2

iptables -t nat -A PREROUTING -j DROP -i $IF2 -d $P0_NET iptables -t nat -A PREROUTING -j DNAT -p icmp -d $P2 --to-destination $IP0 # Setup rules for preserving routes across connections (conversations) iptables -t mangle -A PREROUTING         -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark iptables -t mangle -A OUTPUT             -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark iptables -t mangle -A PREROUTING -i $IF1 -m state --state NEW                 -j CONNMARK --set-mark $Mrk1 iptables -t mangle -A PREROUTING -i $IF2 -m state --state NEW                 -j CONNMARK --set-mark $Mrk2 iptables -t mangle -A PREROUTING -m connmark --mark $Mrk1                         -j MARK --set-mark $Mrk1 iptables -t mangle -A PREROUTING -m connmark --mark $Mrk2                         -j MARK --set-mark $Mrk2 iptables -t mangle -A PREROUTING -m state --state NEW -m connmark ! --mark 0  -j CONNMARK --save-mark

# Selective routing example: send all http traffic to a specific interface (WAN 1) #iptables -t mangle -A PREROUTING -i $IF1 -m state --state NEW -p tcp --dport 80 -j CONNMARK --set-mark $Mrk1 #iptables -t mangle -A PREROUTING -i $IF1 -m state --state NEW -p tcp --dport 443 -j CONNMARK --set-mark $Mrk1

# enable masquerading on the interface iptables -t nat -A POSTROUTING -j MASQUERADE -o $IF2

# turn off Reverse Path Filtering as this is a multi-homed router. RP_PATH=/proc/sys/net/ipv4/conf for IFACE in `ls $RP_PATH`; do		  echo 0 > $RP_PATH/$IFACE/rp_filter done fi fi

When testing out the scripts a bunch of "RTNETLINK answers" may occur as the script is trying to remove routes that don't yet exist, this is ok. Once you have tested out your connection using the console add the following to the Firewall and WAN Up scripts and your done.

Web Interface:Administration:Scripts:Firewall
/jffs/wan2/w2.evt

Web Interface:Administration:Scripts:WAN Up

/jffs/wan2/w2.init

TODO
Implement QOS on WAN2 (help?) WAN Failover DNS Failover (server doesn't appear to switch when an interface is down.) VPN testing

Change Log

 * ver. 0.1 - Modified w2.evt script to account for DNS route and disabled interfaces.
 * ver. 0.5 - Added iptables rules to preserve routes across connections, saved previous settings to properly remove rules/routes.

Useful info
"RTNETLINK answers: No such process" means ip route cant find the interface "RTNETLINK answers: file exists" means there is already an overlapping entry in the routing table

I use the following at the beginning of my scripts to paste them into putty:

scrName=fullPathToYourScript echo "" > $scrName vi $scrName :set noautoindent i

Then just type: `Esc` :wq `Enter`