Per-Traffic Load Balancing

Introduction

Bandwidth management is an essential part of every day operation for typical ISP's, business, and even everyday home users. There are many different types of management tools available to RouterOS users, QoS, rate-limiting, packet-limiting, to name few
I personally operate a wireless ISP in an area that has no other type of conventional high-speed internet conection (ie. cable, fibre or DSL). Not having access to fibre myself, I am in a situation where the single fastest backbone connection I can get does not provide enough bandwidth for me to have only one connection. As a result of this limitation load-balancing multiple internet backbone connections is very important.
In the past I have used ECMP, persistent per connection styled load-balancing (see Routing), as well as various other methods. However, I found all of then lacking in various different areas (not load-balancing correctly, broken large HTTP downloads, IM problems, to name a few issues). I then investigated a way to give me more control over my bandwidth while minimizing the potential problems. The end result was a per-traffic type of load-balancing. This tutorial is designed address that one specific area in depth, at a later date, I may expand/add additional info on fail-over, and other topics

Functions of RouterOS used

• Firewall mangle rules
• Firewall address-lists
• Routing

Step 1 - How to break up to traffic

Before you even log onto your RouterOS box, you should have an idea of how you want to divide your traffic, and have an understanding of what traffic can and what traffic can't be broken up.
Here is an example of what you may want to attempt to separate (in no particular order)

• HTTP traffic (port 80)
• SSL traffic (port 443)
• POP3 traffic (port 110)
• SMTP traffic (port 25)
• P2P traffic (various port)
• Unknown traffic (various port)

After making a list of the traffic type, and the ports they operate on, you need to look at the list and decide if it is at all even capable of being forced out a certain internet connection.
Using the above list as an example here is what I came up with

• HTTP traffic (no problems found yet)
• SSL traffic (some issues, for normal SSL websites and 90% of all software it is not a problem, I'll explain why issues can arise later)
• POP3 traffic (no problems found yet)
• SMTP traffic (no problems found yet)
• P2P traffic (must go out same internet connection as Unknown Traffic, I'll explain why this is later)
• Unknown traffic (must go out same internet connection as P2P traffic, I'll explain why this is later)

Now for a quick explanation of why there can be some issues depending on the traffic type. Some websites/programs do not play friendly with multiple requests from different IP addresses, this is the reason why ECMP has so many problems. I will provide a simple solution to the rare conditions where load-balancing can't be completed.
Again using the above as an example lets explain where and why issues can be arise. SSL - The beauty of a website is that are separate requests for different data, ie. loading pictures from 3 different sources would be a request to each respective server. The result is if a we have a website that uses both SSL and HTTP traffic we know that in most cases the website will just answer the request without care of the originating IP address. However, and this is a special case, if the website/program developer checks where the requests are originating and they find that the IP are different, they may not succesfully answer those requests (this can be by design or by accident). I had 2 cases of this, in both cases it was secured medical websites that where using it as a method of protecting their data.
P2P and Unknown - I am addressing these together because the issue is one and the same. RouterOS doesn't identify P2P based on any single condition, but instead analyzes the packets! This means RouterOS needs time to watch the data before it realizes that is in fact P2P traffic. As a result RouterOS doesn't know the data is P2P until AFTER the connection is made. This is important because the only way to send data out a specific internet connection you must know that traffic is BEFORE the connection is made. So as with P2P traffic, unknown traffic is just that, it is unknown. By marking the unknown traffic though you can control what internet connection is used for both P2P and the left over unknown traffic (very useful!)

Step 2 - Setting Up the Network

Before we get to the real point of this tutorial we need a fictious network we can use as an example.

• client computers (172.18.1.0/24)
• Internet Gateways (10.0.1.1/24, 10 0.2.1/24)
• RouterOS IPs (10.0.1.2/24, 10.0.2.2/24)

Asuming that the IPs, default routes, and DNS settings are already in place the following allow users to get internet access.
Create Address-List for permitted use of internet

/ ip firewall address-list
add list="Allowed-Internet" address=172.18.1.0/24 comment="" disabled=no

Create Address-List to bypass load-balancing

/ ip firewall address-list
add list="WAN-01" address=172.18.1.24/32 comment="" disabled=no
add list="WAN-02" address=172.18.1.76/32 comment="" disabled=no

Apply 'Masquerading'to the traffic leaving the WAN interfaces

/ ip firewall nat 
add chain=srcnat action=masquerade out-interface="WAN - 01" src-address-list="Allowed-Internet" comment="Gateway 10.0.1.1/24" disabled=no 
add chain=srcnat action=masquerade out-interface="WAN - 02" src-address-list="Allowed-Internet" comment="Gateway 10.0.2.1/24" disabled=no

Clients should be able to browse the internet, however only one internet connection would be used (the current default route in RouterOS)

Step 3 - Using RouterOS's Mangle Tool to mark specific traffic

The following is the necessary RouterOS commands to mark specific traffic for a specific route. We will use the same example traffic types as in the beginning,i.e. HTTP, SSL, POP3, SMTP, P2P, and Unknown

/ ip firewall mangle 
add chain=prerouting action=mark-routing new-routing-mark="WAN-01" src-address-list="WAN-01" passthrough=no comment="" disabled=no 
add chain=prerouting action=mark-routing new-routing-mark="WAN-02" src-address-list="WAN-02" passthrough=no comment="" disabled=no 
add chain=prerouting action=mark-routing new-routing-mark="HTTP traffic" passthrough=no dst-port=80 protocol=tcp comment="" disabled=no 
add chain=prerouting action=mark-routing new-routing-mark="SSL traffic" passthrough=no dst-port=443 protocol=tcp comment="" disabled=no 
add chain=prerouting action=mark-routing new-routing-mark="POP3 traffic" passthrough=no dst-port=110 protocol=tcp comment="" disabled=no 
add chain=prerouting action=mark-routing new-routing-mark="SMTP traffic" passthrough=no dst-port=25 protocol=tcp comment="" disabled=no 
add chain=prerouting action=mark-routing new-routing-mark="P2P traffic" passthrough=no p2p=all-p2p comment="" disabled=no 
add chain=prerouting action=mark-routing new-routing-mark="Unknown traffic" passthrough=no comment="" disabled=no 

The first two lines provide a method of marking certain clients to bypass the load-balancing and use only a specific internet connection for all of their traffic.
The next lines then will mark traffic based on the dst-port, notice how we are not passing through, also notice that we are marking all traffic, even if not known and that there is always a different internet connection to be specified for the Unknown and P2P instead of using the router's default route. I also mark P2P separate, even though it must go out the same internet connection as Unknown traffic.
I do this for a couple of reasons, one is that I could easily stop all traffic by simply disabling a route, and the other is in my RouterOS configuration I use a lot of QoS, it's very easy to remember how everything is configured if the QoS mirrors the load-balancing.
So now that we are marking traffic for their respective routes, we next have to add those actual routes.

Step 4 - Using the routing functions of RouterOS to force traffic out to certain internet connections

This following is the necessary RouterOS commands to provide routes for the marked HTTP, SSL, POP3, SMTP, P2P, and Unknown Traffic

/ ip route 
add dst-address=0.0.0.0/0 gateway=10.0.1.1 scope=255 target-scope=10 routing-mark="WAN - 01" comment="" disabled=no
add dst-address=0.0.0.0/0 gateway=10.0.2.1 scope=255 target-scope=10 routing-mark="WAN - 02" comment="" disabled=no
add dst-address=0.0.0.0/0 gateway=10.0.2.1 scope=255 target-scope=10 routing-mark="HTTP traffic" comment="" disabled=no 
add dst-address=0.0.0.0/0 gateway=10.0.2.1 scope=255 target-scope=10 routing-mark="SSL traffic" comment="" disabled=no 
add dst-address=0.0.0.0/0 gateway=10.0.2.1 scope=255 target-scope=10 routing-mark="POP3 traffic" comment="" disabled=no 
add dst-address=0.0.0.0/0 gateway=10.0.1.1 scope=255 target-scope=10 routing-mark="SMTP traffic" comment="" disabled=no 
add dst-address=0.0.0.0/0 gateway=10.0.1.1 scope=255 target-scope=10 routing-mark="P2P traffic" comment="" disabled=no 
add dst-address=0.0.0.0/0 gateway=10.0.1.1 scope=255 target-scope=10 routing-mark="Unknown traffic" comment="" disabled=no

The first two lines provide the routes necessary to give the clients that are not being load-balanced via traffic type the correct gateway to the internet. The remaining lines are the routes necessary to provide the appropiate gateway based on traffic type

Step 5 - Review what you've created

What did you do: * You intelligently broke up your internet traffic into different types * Your marked that traffic using the Mangle Tool of RouterOS * You created a bypass list to allow certain IP to bypass the Per-Traffic Load-Balancing * You assigned particular routes based on the traffic you marked with Mangle

Conclusion

What you have done is very powerful and this tutorial provides just the tip of the iceberg so to say. The traffic types I have listed here are only a small amout of the total and you may want to add many others (ie. DNS, Terminal Services, ICMP, etc). This can be used is a great deal of different ways be it in a small offfice enviroment (separating your email usage from your browsing) or a large wireless ISP (for load-balancing). Some tips and final thougths
1. Think though what you want to achieve before you start 2. Pay attention to the traffic of type on each line (is it incoming intensive or outgoing intensive) 3. Finally experiment, nothing better then finding better, faster, and more intelligent ways to improve your services with a little capital costs

source: http://wiki.mikrotik.com/wiki/Per-Traffic_Load_Balancing