Flobbe.net tech corner

Lab setup

While doing performance tuning of my new NAS, I was using the following setup:

Configuration of the Synology NAS:

No LAG, no LACP
NIC 1 = 192.168.1.15
NIC 2 = 192.168.1.16
NIC 3 = 192.168.1.17
NIC 4 = 192.168.1.18
four iSCSI targets, each using two dedicated disks in RAID-0
four linux test machines in total, having one iSCSI target mounted
each of IP addresses is only being used by one linux machine

This should give me a maximum network bandwith of about 480 MB/s. However I was seeing only 142 MB/s max. Which triggered me to check out the ARP tables (only showing two machines here):

[root@brik1 ~]# arp -i eth0 -n | grep c4
192.168.1.15 ether 00:11:32:1d:c4:31 C eth0
192.168.1.16 ether 00:11:32:1d:c4:32 C eth0
192.168.1.17 ether 00:11:32:1d:c4:32 C eth0
192.168.1.18 ether 00:11:32:1d:c4:32 C eth0

[root@brik2 ~]# arp -i eth0 -n | grep c4
192.168.1.15 ether 00:11:32:1d:c4:32 C eth0
192.168.1.16 ether 00:11:32:1d:c4:32 C eth0
192.168.1.17 ether 00:11:32:1d:c4:32 C eth0
192.168.1.18 ether 00:11:32:1d:c4:32 C eth0

On the Synology itself (ifconfig -a):

eth0 Link encap:Ethernet HWaddr 00:11:32:1D:C4:31
inet addr:192.168.1.15 Bcast:192.168.1.255 Mask:255.255.255.0

eth1 Link encap:Ethernet HWaddr 00:11:32:1D:C4:32
inet addr:192.168.1.16 Bcast:192.168.1.255 Mask:255.255.255.0

eth2 Link encap:Ethernet HWaddr 00:11:32:1D:C4:33
inet addr:192.168.1.17 Bcast:192.168.1.255 Mask:255.255.255.0

eth3 Link encap:Ethernet HWaddr 00:11:32:1D:C4:34
inet addr:192.168.1.18 Bcast:192.168.1.255 Mask:255.255.255.0

On the Cisco SG300:

101 00:11:32:1d:c4:31 gi7 dynamic
101 00:11:32:1d:c4:32 gi9 dynamic
101 00:11:32:1d:c4:33 gi8 dynamic
101 00:11:32:1d:c4:34 gi10 dynamic

With manual ARP entries I was able to get 247 MB/s dd write performance. Without the manual ARP entries this was only 142 MB/s. This clearly shows that there is a performance impact due to this ARP problem. So I asked Synology tech support to comment on this setup. They came up with a solution, there is a special option in the Control Panel / Network section, for this setup:

[x] Reply to ARP requests if the target IP address is a local addres configured on the incoming interface

On the Synology, only one option of the linux kernel is changed (to 1) when you select this:

/proc/sys/net/ipv4/conf/all/arp_ignore

Linux kernel configuration

I did some more lab tests, to find out which kernel parameters are of relevance here. On a plain Centos 6.5 machine, the effect of setting arp_ignore to 1 is that:

on eth0 (first interface that's up) the ARP reply is only given for its own IP address
no ARP reply is given, on any interface, for any other IP address

With arp_ignore set to 0:

on eth0 (first interface that's up) the ARP reply is given for any IP address on the host
no ARP reply is given, on any other interface

So effectively you can't ping to the other IP addresses if you set arp_ignore to 1.

RTFM

echo -n 1 > /proc/sys/net/ipv4/conf/all/arp_ignore

arp_ignore - INTEGER
	Define different modes for sending replies in response to
	received ARP requests that resolve local target IP addresses:
	0 - (default): reply for any local target IP address, configured
	on any interface
	1 - reply only if the target IP address is local address
	configured on the incoming interface

With another change in kernel parameters, results are better:

for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do echo -n 0 > $i ; done

rp_filter - INTEGER
	0 - No source validation.
	1 - Strict mode as defined in RFC3704 Strict Reverse Path
	    Each incoming packet is tested against the FIB and if the interface
	    is not the best reverse path the packet check will fail.
	    By default failed packets are discarded.

Configure ARP on linux

Lab setup

Linux kernel configuration