I have a pc and two embedded linux based systems that I am trying to chain together through IP forwarding. They span two separate local networks. Both networks are wired. I can ping 10.10.10.6 from the PC and ping 10.10.30.1 from target 1. the problem occurs when I run an application on target 1 that generates a healthy(< 3MBps) amount of UDP traffic, directed toward the PC. The performance of the system can not keep up and I see the RX dropped packet count continuously increase on target 0's 10.10.10.4 interface.
There is plenty of CPU cycles on both targets. So the question is, why is target 0 dropping so many packets?
Is there something I can do with the configuration to get the through put up? Increase buffer sizes?
+-----------+ +--------------------------------------+ +-------------------+
| PC |-->| target 0 |-->| target 1 |
|10.10.30.1 | | eth1 =10.10.30.2 eth0 =10.10.10.4 | | eth0 = 10.10.10.6 |
+-----------+ +--------------------------------------+ +-------------------+
target 0 has two ethernet interfaces and ip_forwarding = 1
And I have the following route's setup: on the PC:
Active Routes:
Network Destination Netmask Gateway Interface Metric
10.10.10.6 255.255.255.255 10.10.30.2 10.10.30.1 1
10.10.30.0 255.255.255.0 10.10.30.1 10.10.30.1 20
and on target 0:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.10.30.0 * 255.255.255.0 U 0 0 0 eth1
10.10.10.0 * 255.255.255.0 U 0 0 0 eth0
default 10.10.10.6 0.0.0.0 UG 0 0 0 eth0
default 10.10.30.1 0.0.0.0 UG 0 0 0 eth1
and on target 1:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.10.30.1 10.10.10.4 255.255.255.255 UGH 0 0 0 eth0
10.10.10.0 * 255.255.255.0 U 0 0 0 eth0
default 10.10.10.4 0.0.0.0 UG 0 0 0 eth0
Edit: target 0
# ip link list | grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
target 1
# ip link list | grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
Both connections have been confirmed to be set at 100Mbps Full duplex.
Another thought:
As it appears the input to target 0 is getting flooded, can the switch between the two targets, buffer the packets and then flood target 0 with a big burst of data?
Update:
By replacing the 10/100/1000 switch with an old 10M half duplex hub the system is working fine, no dropped packets.
In the bad condition, ethtool register dump indicated RX_DROP count continuously increasing. This is indicating the Processor/Linux is not able to remove the data quickly enough from the ethernet chip. With the 10M hub this is no longer a problem.
So this sounds to me like Linux is not configured correctly to keep up the higher(but not that high) data rate. I would like to understand why the OS/processor can't keep up?
-
What is the MTU size set to on target0 and target1?
As this is L3 forwarding a packet larger than the MTU on Target0 would need to be fragmented before being passed to the Target0 eth1 interface, it is possible this is cause issues (However have no proof this would lead to dropped RX packets).
Can you do a 'ip link list | grep mtu' on both targets?
Have you enabled any IPtables filtering or anything IP policy/QoS related?
: All the interfaces are set to MTU size of 1500. So unless the ip forwarding needs to make the packets slightly bigger, it shouldn't have to fragment. I do not have any IPTables filtering or QoS.: target0 # ip link list | grep mtu 1: lo:: target 1 # ip link list | grep mtu 1: lo:From TrXuk -
Have you verified that all links are running at full duplex?
dmesg | grep -i duplexShould give you this information on Linux. If that isn't successful, try:
mii-tool <interface>(Note the latter needs to be run as root.)
What is the size distribution of the UDP packets you are sending? 3MBps can mean 2,000 packets per second or 50,000 packets per second, depending on how big your datagrams are.
3MBytes/second should not be troubling most modern hardware, which is why I am interested in the duplex of all the links. A half duplex connection would lead to collisions, as well as increased buffering.
Regarding your question about the switch being able to buffer frames, this is possible, but is normally not tunable. Are you able to provide make/model information on the switches?
: dmesg | grep -i duplex doesn't return anything?: the full packets are 1500 bytes. so roughly 2000 packets/sec.Murali Suriar : Try 'mii-tool' to discover speed/duplex? (Needs to be run as root): The switch in question is a consumer 10/100/1000 netgear. I'll have to get the model number on Monday.: where do I get the mii-tool? I assume I would have to cross compile it for the ARM targets?Murali Suriar : A consumer switch is unlikely to have much in the way of buffers; in fact, many cannot sustain line rate gigabit throughput. Do all three switch ports show as connected at full duplex? (Normally the indicator lights on the switch will turn a different colour for full/half duplex). Regarding mii-tool, it's included in many distributions, but if you don't have it installed, there may be another tool you can use. Perhaps ask a separate question about determining duplex settings on the Linux distro you are using?: So the switch is a netgear gs608, and the LED for the port in question is lighting up Amber. I have only been able to find speculation that this may indicate 10M half duplex(?); but as this is a 10/100/1000 switch it may also be 10/100M.: I swapped out the gs608 with a linksys router(utilizing the switching capabilities). Unfortunately this router is even less descriptive about the connection status and it to resulted in Rx dropped packets.: I have confirmed the connections are at 100M Full duplex with ethtool.From Murali Suriar
0 comments:
Post a Comment