Table of Contents

Использование нескольких WAN интерфейсов с пакетом mwan3

Последний релиз:

Пакет mwan3 был обновлен 2015-12-17 и находится:

См. процедуру загрузки и настройки ниже.

Описание

Возможности

Пакет mwan3 предоставляет следующие возможности:

Авторы

Огромное им человеческое СПАСИБО!!!

Зачем использовать mwan3?

Как mwan3 производит баллансировку трафика

Mwan3 архитектура

Mwan3 Роутинг

The following steps are taken to route a packet with mwan3 (version 2.0 or higher):

Every incoming packet (this includes router originated traffic) is handled by the iptables mwan3_hook. This hook takes 5 steps:

  1. Restore mark if previous set. If successful marked, goto step 5.
  2. Check if the packet arrives on a wan interface. If originated from a local connected ip network, then mark packet with iface_id 255 (default). If the packet is from another (non-local) network and arrives on wan interface, then mark it with iface_id. If successful marked, goto step 5.
  3. Check if packet destined for a known ip network (has a route for it other than default). If so then mark packet with iface_id 255 (default) and goto step 5.
  4. Check if packet source address is that of a wan interface. If so use that wan interface for routing regardless of user defined rules and mark packet with iface_id of corresponding wan.
  5. Apply user rules and mark with configured iface_id. If no match leave unmarked.
  6. If marked then save mark.

Remember that iptables only marks the packet, it does not make routing decisions. Next in line are the ip rules. In following order they are:

  1. Ip rules 1001 till 1250 are for wan interface 1 till 250 respectively. This rule says: If packet is incoming from wan interface use main routing table, regardless of mark.
  2. Ip rules 2001 till 2250 are for wan interface 1 till 250 respectively. This rule says: If packet is marked with iface_id [1-250], use the corresponding wan interface routing table.
  3. Ip rule 2253 is a blackhole rule. This rule states: If packet is marked with iface_id 253 (blackhole), silently drop packet.
  4. Ip rule 2254 is a blackhole/unreachable rule. This rule states: If packet is marked with iface_id 254 (unreachable), drop packet and return icmp unreachable.

Next up are the routing tables. These are really simple. There is just the standard main routing table and one routing table containing one gateway for each wan interface. Route table 1 for the first wan, route table 2 for the second and so on. Hopes this make troubleshooting easier.

Mwan3 and IPv6

It is ok to have ipv6 and mwan3 running on the same router. Up until version 2.0 ipv6 is ignored by mwan3. Ipv6 routing is done without intervention of any mwan3 rule/route. (Source: Adze's post at https://forum.openwrt.org/viewtopic.php?pid=243603#p243603) Starting from version 2.0, you can steer ipv6 traffic if you want.

Mwan3 опции командной строки

Здесь описаны некоторые cli команды для выяснения проблем или просмотра статуса Ваших соединений: 

root@OpenWrt:~# mwan3
Syntax: mwan3 [command]

Присутствующие комманды:
	start           Загрузка правил для iptables , ip правил и ip маршрутов
	stop            Выгрузка правил для iptables , ip правил и ip маршрутов
	restart         Перезагрузка правил для iptables , ip правил и ip маршрутов
        ifup <iface>    Загрузка правил им маршрутов для выбраного интерфейса
        ifdown <iface>  Выгрузка правил им маршрутов для выбраного интерфейса
        interfaces      Показать статус интерфейсов
        policies        Показать текущую активную политику соединений
	connected	Показать существующие подключенные соединения
        rules           Показать активные правила
        status          Показать все статусы
root@OpenWrt:~# mwan3 interfaces
Interface status:
Interface wan is online (tracking active)
Interface wan2 is online (tracking active)

Требования

версии OpenWrt

Оборудование

Any router configured with multiple WAN interfaces running OpenWrt 12.09 or later should work. Just pick a device with good OpenWrt support, preferably one with VLAN support for the additional interface flexibility VLAN support provides.

At least three interfaces need to exist for the minimal configuration: inside LAN, WAN1 and WAN2. The simplest way to do this is use VLANs to put individual switch ports into their own VLANs, thus each becoming separate interfaces.

Зависимости пакетов

Конфликты пакетов

Test external DNS/mail/etc. servers for access from each WAN interface

Пре-конфигурация

до установки рекомендуется провести пре-конфигурацию и тестирование ваших соединений. Это не обязательно, но способно упростить инсталляцию и избежать многих подводных камней после нее.

Сперва Вы должны открыть несколько WAN интерфейсов и проверить их работоспособность. Это облегчит Ваши действия после установки даного пакета. Больше информации Вы сможете прочитать сдесь позже после дополнения даной статьи..

Первый WAN интерфейс (wan)

The mwan3 default configuration file assumes two WAN interfaces named “wan” and “wan2”. If this is not the case, edit the file /etc/config/mwan3 to configure the “interface” definitions with the same WAN name as defined in /etc/config/network configuration. Definitions are case sensitive!

Второй WAN интерфейс (wan2)

Самый простой способ создать несколько интерфейсов WAN использовать VLAN маршрутизатора. Это позволяет каждому отдельному порту назначить его собственный отдельный интерфейс в OpenWrt.

Create additional WAN interfaces (e.g. wan3, ...) as desired if more than two WAN connections will be used. More information on howto create interfaces can be found at https://wiki.openwrt.org/doc/uci/network.

The routable loopback (self)

On routers with just one wan interface (and one default route), there is no issue on which source address to use for new initiated sessions. But with two or more wan interfaces you may wish to have control over this. Up until version 2.0, mwan3 did not respect the already set source address of router originated packets. Packets were load-balanced regardless of source address, based on configured user rules.

As of version 2.0 mwan3 does respect the already set source address. The advantage of this is that an applications can have control over which wan interface to use. The downside of this is that when an application does not specify which source address to use (most of the time) the kernel will pick a source address based on the routing table. In practice this means the default route with the lowest metric is used to determine which source address to use. So if you don't configure a routable loopback address with corresponding more preferred default route, all traffic originating from the router itself will leave the primary wan with the source address of that wan interface, regardless of configured user mwan3 rules.

This however only effects router initiated traffic. Traffic from LAN clients will always be balanced based on mwan3 configured rules even if no routable loopback address is configured.

Configuring a routable loopback with default route is easy: 

config interface 'self'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '192.168.1.1'
	option netmask '255.255.255.255'
	option gateway '192.168.1.1'

After this all traffic originating from router itself (if no more specific route is found) will have source address of 192.168.1.1 (before NAT). Extra advantage is that configuring mwan3 rules for router only traffic is much easier.

Prepare and the check the default routing table for the wan interfaces

Step 1: Configure a different metric for each WAN interface

Note: PPPoE connections only show the “Use gateway metric” option if “Use default gateway” option is enabled.

wan setting

wan is the default wan interface in this example, and so will get a metric of 10.

wan2 setting

wan2 is the second wan interface in this example, and so will get the a metric of 20.

Verify the routing table

If configured correctly, you should have a default gateway (the lines with a target address of 0.0.0.0/0) with a unique metric set for each WAN interface. For example: 

default via 10.0.3.2 dev eth1  proto static  src 10.0.3.15  metric 10 
default via 10.0.4.2 dev eth2  proto static  src 10.0.4.15  metric 20

Troubleshooting

Interfaces are missing a metric value

Verify outbound traffic on each WAN interface

Check that each WAN interfaces works by trying to ping www.google.com out from each interface. Ensure all interfaces are correctly sending and receiving traffic before proceeding.

Проверка первого интернет (first WAN) соединения

root@openwrt:~# ping -c 1 -I eth0.1 www.google.com
PING www.google.com (209.85.148.103): 56 data bytes
64 bytes from 209.85.148.103: seq=0 ttl=54 time=19.637 ms

--- www.google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 19.637/19.637/19.637 ms

Проверка wan2 соединения

root@openwrt:~# ping -c 1 -I eth0.2 www.google.com
PING www.google.com (209.85.148.99): 56 data bytes
64 bytes from 209.85.148.99: seq=0 ttl=56 time=25.552 ms

--- www.google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 25.552/25.552/25.552 ms

Проверка всех остальных WAN соединений

Ensure the CONNTRACK module is enabled in OpenWrt

mwan3 requires that the CONNTRACK module is enabled and active on its WAN interfaces. 

option 'conntrack' '1'

Ручная загрузка пакетов

This step is only required for OpenWrt 12.09. In OpenWrt 14.07 “Barrier Breaker” and later, the mwan3 packages are in the standard package repositories and no manual download is required.

Adze and Arfett keep the OpenWrt package repositories up to date and you are more likely to download the recommended latest “stable” version from there.

Рекомендованый метод загрузки заключается в подключении к роутеру посредством telnet или SSH и загрузки пакетов в директорию /tmp.

для версии 1.4 

root@OpenWrt: cd /tmp
root@OpenWrt:/tmp# rm mwan3_1.4-25_all.ipk
root@OpenWrt:/tmp# wget https://www.dropbox.com/s/goolfkiu0u49fqe/mwan3_1.4-25_all.ipk
root@OpenWrt:/tmp# rm luci-app-mwan3_1.4-2_all.ipk
root@OpenWrt:/tmp# wget https://www.dropbox.com/s/eekcbg58f6d6lsi/luci-app-mwan3_1.4-2_all.ipk

или для версии 2.0 

root@OpenWrt: cd /tmp
root@OpenWrt:/tmp# wget https://www.dropbox.com/s/2e1qkxf3dqdgkpa/mwan3_2.0-2_all.ipk
root@OpenWrt:/tmp# wget https://www.dropbox.com/s/t6wn0ijltbbn76a/luci-app-mwan3_1.4-4_all.ipk

Установка

OpenWrt 14.07 and later

Метод использующий веб интерфейс LuCi

SSH method

# обновите список пакетов для подготовки их к загрузке
opkg update

# сделайте back up текущей конфигурации mwan3 в случае не правильного сохранения их автоматически
cp -a /etc/config/mwan3 /etc/config/mwan3-tempbackup

# установите luci-app-mwan3, mwan3 и все необходимые зависимости
opkg install luci-app-mwan3

OpenWrt 12.09 и версии старше

cd /tmp

# обновите список пакетов для подготовки их к загрузке
opkg update

# сделайте back up текущей конфигурации mwan3 в случае не правильного сохранения их автоматически
cp -a /etc/config/mwan3 /etc/config/mwan3-tempbackup

# установите mwan3 и все необходимые зависимости
opkg install mwan3_*.ipk

# установите luci-app-mwan3
opkg install luci-app-mwan3_*.ipk

Перезагрузите роутер через веб интерфес LuCI или другим доступным методом если это необходимо

To ensure the new menu item for mwan3 appears, restart the web server hosting the LuCI interface (or just reboot the router).

В меню веб интерфейса LuCi должна появится новая запись “Network > Load Balancing” .

Обновление

Обновление производится только установкой пакетов со старшими версиями. FIXME

MWAN3 configuration

A mwan3 configuration consists of four section elements, namely:

Interface configuration

For each wan interface configure an interface section and define how each WAN interface is tested for up/down status. Each interface section must have a name that corresponds with the interface name in your network config. The settings are described below:

Name Type Required Default Description
interface name string yes (none) The OpenWrt interface name as shown in OpenWrt's Network > Interfaces list (if using a PPPoE interface, the interface name specified here should be the underlying OpenWrt interface name, not the “pppoe-…” interface name)
enabled boolean yes (none) Should mwan3 run on this interface
track_ip list of ip addresses no (none) The hosts to test if interface is still alive. If this value is missing the interface is always considered up
reliability number no 1 Number of track_ip hosts that must reply for the test to be considered as successful. Ensure there are at least this many track_ip hosts defined or the interface will always be considered down
count number no 1 Number of pings to send to each host with each test
timeout seconds no 4 Number of seconds to wait for an echo-reply after an echo-request
interval seconds no 10 Number of seconds between each test
up number no 5 Number of successful tests to considered link as alive
down number no 5 Number of failed tests to considered link as dead

The primary reason to change the default settings is to shorten the time before an interface is failed-over (by reducing the ping interval and number of pings before the interface is down) or lengthen the time to avoid a false link failure report. Please note that if you change the timeout value on low bandwidth interfaces (e.g. 3g) or busy interfaces, that false time-outs can occur. A timeout value of less then 2 seconds is not recommended.

A typical interface section looks like this: 

config interface 'wan'
        option enabled '1'
        list track_ip '8.8.4.4'
        list track_ip '8.8.8.8'
        list track_ip '208.67.222.222'
        list track_ip '208.67.220.220'
        option reliability '2'
        option count '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '8'

Member configuration

Each member represents an interface with a metric and a weight value. Members are referenced in policies to define a pool of interfaces with corresponding metric and load-balancing weight. Members can't be used for rules directly. The default settings are described below:

Name Type Required Default Description
interface string yes (none) Member applies to this interface (use the same interface name as used in the mwan3 interface section, above)
metric number no 1 Members within one policy with a lower metric have precedence over higher metric members
weight number no 1 Members with same metric will distribute load based on this weight value

A typical member section looks like this: 

config 'member' 'wan1_m1_w3'
	option 'interface' 'wan1'
	option 'metric' '1'
	option 'weight' '3'

Policy configuration

Policies define how traffic is routed through the different WAN interface(s). Every policy has to have one or more members assigned to it, which defines the policy's traffic behavior. If a policy has a single member, traffic will only go out that member. If a policy has more than one member, it will either load-balance among members or use one member but fail-over to another, depending on how the members are configured.

If there is more than one member assigned to a policy, members within the policy with a lower metric have precedence over higher metric members. Members with the same metric will load-balance. Load-balancing members (with same metric) will distribute load based on assigned weights values.

A typical policy section looks like this: 

config policy 'balanced'
        list use_member 'wan_m1_w3'
        list use_member 'wan2_m1_w2'

Rule configuration

A rule describes what traffic to match and what policy to assign for that traffic. These are the available options:

Name Type Required Default Description
use_policy string yes (none) Use this policy for traffic that matches or set to default to use the default routing table to lookup
src_ip ip address no any Match traffic from the specified source ip address
src_port port or range no any Match traffic from the specified source port or port range, if relevant proto is specified
proto protocol no all Match traffic using the given protocol. Can be one of tcp, udp, icmp or all or it can be a numeric value, representing one of these protocols or a different one
dest_ip ip address no any Match traffic directed to the specified destination ip address
dest_port port or range no any Match traffic directed at the given destination port or port range, if relevant proto is specified
ipset string no (none) Match traffic directed at the given destination ip address to an ipset set
sticky boolean no 0 Allow traffic from the same source ip address within the timeout limit to use same wan interface as prior session
timeout number no 600 Stickiness timeout value in seconds

A typical rule section looks like this: 

config rule 'default_rule'
        option dest_ip '0.0.0.0/0'
        option use_policy 'wan_wan2_wan3'

Stickiness and ipset

Mwan3 version 1.6 has sticky and ipset support. Stickiness lets you route new session over the same wan interface as the previous session, as long as the time between the new and the previous session is shorter then the timeout value (default 600s). This can solve some problems with https sites, which don't allow a new source address within the same cookie/https session. Ipset lets you route traffic over wan interfaces based on set of ip addresses. A set can be created by hand, by dnsmasq based on domain names, or your own script. Mwan3 rules with ipset option will try to match destination ip address to the configured ipset. 

config rule 'youtube'
    option sticky ‘1'
    option timeout ‘300'
    option ipset 'youtube'
    option dest_port '80,443'
    option proto 'tcp'
    option use_policy 'balanced'

With sticky set to 1, this rule has now sticky enabled. When a packet for a new session matches this rule, its source ip address and interface mark are stored in an ipmark set with a timeout of 300 seconds (default 600). When packet for a second new session from the same lan host within the timeout period matches this rule, it will use the same wan interface as the first packet and the timeout counter is reset back to 300 again.

Stickiness is on a per rule basis. With this example, all traffic from lan hosts will use the same wan interface for all youtube hosts, even if the source or destination ip address differs.

The option ipset matches only destination ip addresses. This example will only work if your lan clients use the dnsmasq server as their one and only dns server. Mwan3 will create the ipset set for you if it does not exist already. For this to work you need to configure a rule in your /etc/dnsmasq.conf file: 

ipset=/youtube.com/youtube

Example configuration

config interface 'wan'
        option enabled '1'
        list track_ip '8.8.4.4'
        list track_ip '8.8.8.8'
        list track_ip '208.67.222.222'
        list track_ip '208.67.220.220'
        option reliability '2'
        option count '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '8'

config interface 'wan2'
        option enabled '1'
        list track_ip '8.8.8.8'
        list track_ip '208.67.220.220'
        option reliability '1'
        option count '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '8'

config member 'wan_m1_w3'
        option interface 'wan'
        option metric '1'
        option weight '3'

config member 'wan_m2_w3'
        option interface 'wan'
        option metric '2'
        option weight '3'

config member 'wan2_m1_w2'
        option interface 'wan2'
        option metric '1'
        option weight '2'

config member 'wan2_m2_w2'
        option interface 'wan2'
        option metric '2'
        option weight '2'

config policy 'wan_only'
        list use_member 'wan_m1_w3'

config policy 'wan2_only'
        list use_member 'wan2_m1_w2'

config policy 'balanced'
        list use_member 'wan_m1_w3'
        list use_member 'wan2_m1_w2'

config policy 'wan_wan2'
        list use_member 'wan_m1_w3'
        list use_member 'wan2_m2_w2'

config policy 'wan2_wan'
        list use_member 'wan_m2_w3'
        list use_member 'wan2_m1_w2'

config rule 'sticky_even'
        option src_ip '0.0.0.0/0.0.0.1'
        option dest_port '443'
        option proto 'tcp'
        option use_policy 'wan_wan2'

config rule 'sticky_odd'
        option src_ip '0.0.0.1/0.0.0.1'
        option dest_port '443'
        option proto 'tcp'
        option use_policy 'wan2_wan'

config rule 'default_rule'
        option dest_ip '0.0.0.0/0'
        option use_policy 'balanced'

Further configuration tips

Configuring rules with large numbers of destination IPs or subnets using ipsets

According to forum posts, there is a limit of around 300 specific destination IP addresses or subnets in mwan3 rules. Starting with Mwan3 version 1.6, ipset address groups can be used to bypass this limit. See the forum post at https://forum.openwrt.org/viewtopic.php?pid=301009#p301009 for more information.

OpenWrt hotplug script fix (OpenWrt 12.09 only)

This is for OpenWrt 12.09 only. The OpenWrt 14.07 hotplug scripts were substantially re-written and there is no evidence yet that the workaround below is needed on OpenWrt 14.07. 

vi /sbin/hotplug-call
...

[ \! -z "$1" -a -d /etc/hotplug.d/$1 ] && {
        for script in $(ls /etc/hotplug.d/$1/* 2>&-); do (
                [ -f $script ] && . $script
                ## customization: add a 1 second delay to prevent segmentation faults
                sleep 1
        ); done
}

Start mwan3

Preserve configuration files in an OpenWrt upgrade

Verification of basic operation

Check MWAN3 status in cli

root@OpenWrt:~# mwan3 status
Interface status:
Interface wan is online (tracking active)
Interface wan2 is online (tracking active)

Policy balanced:
 wan2 (40%)
 wan (60%)

Policy wan1_only:
 wan (100%)

Policy wan2_only:
 wan2 (100%)

Policy wan2_wan:
 wan2 (100%)

Policy wan_wan2:
 wan (100%)

Local connected networks:
destination        policy             hits     
------------------------------------------------
127.0.0.0/8        default            22       
224.0.0.0/3        default            0        
192.168.1.0/24     default            0        
192.168.33.0/24    default            0        
213.154.232.8/29   default            0        

Active rules:
source             destination        proto  src-port      dest-port     policy          hits     
---------------------------------------------------------------------------------------------------
0.0.0.0/0          213.136.223.128/25 tcp    0:65535       80            wan_wan2        0        
1.2.3.4            5.6.7.8            udp    12345:54321   12345:54321   wan2_wan        0        
0.0.0.0/0          0.0.0.0/0          all                                balanced        2862

Check status in the MWAN3 overview page

Check kernel routing tables

Verification of WAN interface load-balancing

Verification of WAN interface failover

Test interface failover

Test interface fail-back

Управление

Остановка mwan3

LuCI

SSH

mwan3 stop

Запуск mwan3

LuCI

SSH

mwan3 start

Monitoring mwan3

Ручная проверка статуса

Automated status check

It would be good to be able to automatically monitor interface status through mwan3 using a monitoring system (for example, OpenNMS or monit).

Alerting

It would be good for mwan3 to send some kind of alert (e.g. e-mail, SNMP trap) if it detects a failed interface and performs a failover, or if it performs a fail-back.

If you install the luci-app-mwan3 (mwan3-luci) package it comes with /etc/hotplug.d/iface/16-mwancustom hotplug shell script which may be modified to perform custom actions on interface ifup/ifdown events. It is already filled with a basic template for gathering information on the current state of WAN connections and just needs your changes in the send_alert() function as well as uncommenting the case statement at the bottom. The mwan3 LuCI package provides a page for editing this hotplug shell script or you may edit via command line.

Controlling the mapping between internal IP sources and external IPs and interfaces

When using multiple WAN connections, there will be multiple external IPs which can be used as the external IP for outgoing NATed traffic. In particular, an external interface might have a block of external IPs that should be mapped in a particular way to specified internal servers. For example, the internal mail server should send out traffic on the same external IP identified in its MX record. This is the procedure to do this.

Step 1: Set mwan3 rules to send traffic out the right interface

Add an mwan3 traffic rule that directs the specific desired source IP out the correct WAN interface. Rules are processed in top-down order, so be sure this specific rule is higher in the list (thus higher priority) than more general rules below that implement load-balancing or failover in the default case. 

config policy 'wan3_only'
        list use_member 'wan3_m1_w10'
config rule 'Mailserver_uses_wan3_only'
        option src_ip '172.16.1.20'
        option proto 'all'
        option use_policy 'wan3_only'

If the external WAN interface only has a single external IP, this is all that is needed. If the interface has multiple external IPs, both the next two steps are also needed.

Step 2: Assign multiple external IP addresses selected interface (optional)

This step is only needed if the desired external interface needs to have multiple external IP addresses assigned to it.

The specified external interface may have multiple IPs assigned to it. For OpenWrt 12.09, the preferred way to do this is using multiple interface definitions -- see reference.

Step 3: Set OpenWrt NAT rules to send traffic out the right IP on the selected interface (optional)

This step is only needed if the desired external interface has multiple external IP addresses assigned to it.

With multiple external IP addresses, we want to control which address is used when sending out traffic from particular servers. This is configured using a source NAT rule in OpenWrt.

Note that both a mwan3 rule to select the interface and an SNAT rule to select the specific IP on that interface are needed to correctly send traffic out a specific external IP. 

config redirect
        option target 'SNAT'
        option name 'Mail server goes out 170.53.100.25'
        option src 'lan'
        option dest 'wan'
        option src_ip '172.16.1.20'
        option src_dip '170.53.100.25'
        option proto 'all'

mwan3 and other programs

ddns-scripts

Example 1: Register the external IP of the active WAN interface

This is the case where you want external clients using a DDNS name to automatically reconnect to the alternate WAN interface if the primary WAN interface fails.

Example 2: Register the external IP of a specific WAN interface using the "interface" source

This is the case where you want each specific WAN interface to register its own DDNS name and the WAN interface in question has an external IP directly assigned to it.

Example 3: Register the external IP of a specific WAN interface using the "web" source

This is the case where you want each specific WAN interface to register its own DDNS name but the WAN interface in question is behind a NAT device and so does not directly have the correct external IP.

This is tricky when the WAN interface is not the default WAN interface, as ddns-scripts cannot be configured to use a specific interface to check its IP.

Option 1: use a static route
Idea for option 2: use curl

OpenVPN

Possible problems

If the openwrt system is an openvpn client and a zone 'vpn' is defined on the vpn interface and this zone has the masquerading active, for reasons (yet) unknown the traffic from the internal lan to the vpn will be able to reach the destination and go back to the router but then will be not dispatched back to the lan clients. Disabling mwan3, instead, let the traffic be dispatched properly.

It could be a misconfiguration, more testing is needed.

Example 1: Have OpenVPN Server be accessible through multiple WAN interfaces (server mode)

If load-balancing between multiple WAN interfaces, it is desirable to have OpenVPN clients be able to connect through all active WAN interfaces.

In a multiple WAN interface failover scenario, OpenVPN will not accept client connections on the secondary WAN interface after a failover, as it started listening on the primary WAN interface when it was started.

The following configuration will allow multiple WAN interface to be used with OpenVPN Server.

Step 1: Listen only on the internal LAN interface
vi /etc/openvpn/my-vpn.conf
...
# Which local IP address should OpenVPN
# listen on? (optional)
;local a.b.c.d
## Customization: have OpenVPN listen on the internal LAN interface IP only to allow client re-connections after a WAN interface failover
local 192.168.1.1

...
Step 2: Set up port-forward(s)
Step 3: OpenVPN client and DNS configuration

Example 2: Use OpenVPN tunnels as virtual wan(s) (client mode)

If you want to use your OpenVPN client tunnels as virtual wan interfaces in mwan3, you have to make sure that you set a default route with different metric for each tunnel interface. Also most commercial VPN solutions push two static routes to override the standard default gateway. In most cases you don't want this override when using OpenVPN client tunnels in conjunction with mwan3.

As a solution you can add the following lines to your OpenVPN client config: 

route-nopull
route 0.0.0.0 0.0.0.0 vpn_gateway 20

This example will ignore the routes pushed from the OpenVPN server and will add a default route with metric 20 over the OpenVPN tunnel interface.

privoxy transparent HTTP proxy

Transparent HTTP proxying relies on using iptables rules to transparently redirect outgoing traffic to port 80 first through the local proxy at another port number.

For example, here is a OpenWrt redirect rule to redirect outgoing traffic to TCP 80 port and re-send it to the local proxy listening on TCP port 8118. This will go into iptables NAT table rules. 

config redirect
    option target 'DNAT'
    option dest 'lan'
    option proto 'tcp'
    option src 'lan'
    option src_dip '!10.0.2.1'
    option src_dport '80'
    option dest_ip '10.0.2.1'
    option dest_port '8118'
    option name 'Transparent Proxy [privoxy]'
    option enabled '1'

The problem is that mwan3 adds rules to the iptables's MANGLE table, and this is handled before the NAT table. So when a client makes a request to fetch a web page, it is first marked by mwan3. Mwan3 decides based on your mwan3 rules which wan interface to exit and marks the session accordingly.

Next, iptable nat rule handling takes place and diverts the web page request to privoxy. The reply from privoxy however is part of the same session and is already marked to leave a wan interface. The reply from privoxy is then send over the internet, which is obviously incorrect.

To fix this add the following rules to your mwan3 config: 

config 'rule' 'rule1'
    option 'proto' 'tcp'
    option 'dest_port' '80'
    option 'src_ip' '10.0.2.1'
    option 'dest_ip' '0.0.0.0/0'
    option 'use_policy' 'wan1_wan2_loadbalanced'

config 'rule' 'rule2'
    option 'proto' 'tcp'
    option 'dest_port' '80'
    option 'src_ip' '10.0.2.0/24'
    option 'dest_ip' '0.0.0.0/0'
    option 'use_policy' 'default'

config 'rule' 'rule3'
    option 'dest_ip' '0.0.0.0/0'
    option 'use_policy' 'wan1_wan2_loadbalanced'

The policy “wan1_wan2_loadbalanced” is just an example. Change it to whatever policy you like.

nodogsplash

Out of the box, mwan3 does not work with nodogsplash. The problem is that both mwan3 and nodogsplash use the same iptables mark bits. A common symptom of this is the nodogsplash splash page appearing for every page even as an authenticated client.

However, it is possible to fix this with a minor change! Fortunately, it is simple to change the mark bits that nodogsplash uses. Simply add the following lines to '/etc/nodogsplash/nodogsplash.conf' to override the marking bits. 

# Change the default marking flags to work with mwan3 and qos-scripts
FW_MARK_AUTHENTICATED 262144
FW_MARK_TRUSTED 131072
FW_MARK_BLOCKED 65536

These values let nodogsplash work together with mwan3 and also work with standard Openwrt qos-scipts.

Usage reports

12.09, tplink wdr3600, mwan3 - 1.4-24, two wan connections

Premise: very nice piece of work given to the internet community, congrats to the contributors. We, users, can only give back a bit of experience and documentation, still somehow useful.

In a simulated test environment described as follows: 

external network  <---> (ip a.b.c.118) router1 (ip 192.0.2.1)  <---> (ip 192.0.2.166 - wan) tplink (ip 192.168.1.1)  <---> (ip 192.168.1.50) pcA
                  <---> (ip a.b.c.224) router2 (ip 192.0.10.1) <---> (ip 192.0.10.166 - wan2)       (ip 192.168.10.1) <---> (ip 192.168.10.101)pcB

The mwan3 was installed on the tplink with the following configuration (apart from wan and wan2) in /etc/config/mwan3: 

...lines....

config member 'wan_m10'
        option interface 'wan'
        option metric '10'

config member 'wan_m20'
        option interface 'wan'
        option metric '20'

config member 'wan2_m10'
        option interface 'wan2'
        option metric '10'

config member 'wan2_m20'
        option interface 'wan2'
        option metric '20'

config policy 'wan_wan2'
        list use_member 'wan_m10'
        list use_member 'wan2_m20'

config policy 'wan2_wan'
        list use_member 'wan_m20'
        list use_member 'wan2_m10'

config rule 'rule1'
        list comment    'from 192.168.1.50 to wan_wan2'
        option src_ip   '192.168.1.0/24'
        option dest_ip  '0.0.0.0/0'
        option use_policy 'wan_wan2'

config rule 'rule2'
        list comment    'from 192.168.10.101 to wan2_wan'
        option src_ip   '192.168.10.0/24'
        option dest_ip  '0.0.0.0/0'
        option use_policy 'wan2_wan'

Using different wan connections from different Pc-s

Then with the pcA and pcB two different download were started, and every pc used a different connection. Great.

Line fail

When one wan connection was physically disconnected, one pc lost the tcp active connections, but after 'restarting' them no problem, the pc was using the other wan connection configured as line backup. Great.

Line recovered

If a wan line recovers (let's say A), then the pc that was using the other wan line (configured as backup, let'S say B) is switched back to the wan line A, and this cause another disruption of tcp connections.

Edit: On recovery, connections already established over backup links will not be terminated and continue to traverse over backup wan. Only new connections will be routed over preferred wan.

Incoming connections routed behind the router

Rdp connections to the pc behind the tplink are stable, this means that a service behind the tplink with mwan3 is reachable in a reliable way.

Therefore does not happen that an external request coming on one wan connection gets the replies through the other wan connection (at least for failover policies)

Incoming connections to services on the router

At least testing with ssh, does not happen that one connection through a wan line is router on the other wan line in case of line failover. Therefore ssh is stable.

mwan3 status hints

mwan3 status shows often 'hits' only for the last rule if this one is generic (source 0.0.0.0/24 dest 0.0.0.0/24 ) like for web traffic. This is a bit misleading if in reality the traffic was split by other rules. I do not know why iptables reports this (mwan3 status just reports what iptables says) but analyzing it with iftop and bmon there is better to asses if the traffic is directed properly using the wan connections.

Reliable public ip addresses to ping

After some weeks of continous pinging to opendns Ips (208.67.222.222) and (208.67.220.220), seems that opendns does not like this and interrupt the icmp replies for a while, therefore the mwan3 thinks that the lines are down. Could be useful to collect stable internet ip addresses? (possibly from backbone providers).

Some possible choices: 

# Level3 communications (large network carrier)
4.2.2.2

# google dnses
8.8.4.4
8.8.8.8

# facebook.com
173.252.120.6

# Opendns (with limits after a certain amount of days?)
208.67.220.220
208.67.222.222

Possible work arounds to test connection not only by ping ?

Could be possible to implement other ways to test connections, maybe through testing ip/ports (with additional packages, of course), like: netcat -z -w 2 208.67.222.222 53 ; echo $?