Using 2 channels with 2 cards to improve throughput

[tim]

I want to create a radio link between 2 Soekris boards.

If I use 2 WiFi cards in each Soekris can I set up the IP routing to use two radio channels between the SBCs and :

1) double throughput
2) improve resilience

Similiarly if I use 2 Soekris boards at each end can I run 4 radio links PtP.

Tim

[tony]

That would be a neat idea, however we have not implemented Wireless channel bonding in the current release of StarOS. This is something I will look into however.

Thanks!

[redfeaag]

My understanding of TCP/IP is that if you have two routes with the same metric, it will automatically provide load balancing & failover between routes. I haven't seen anywhere to add a metric by hand in staros (OK, prove me wrong here....) so I assume that when you add a route the metric is the same every time.

Try this:

Add two routes (NEITHER OF THEM DEFAULT) for 0.0.0.0 on each card back to the other end. The key here is the metric. If the metric is the same it should treat each route the same and split the traffic equally. If the metrics are different, and I am assuming a default gateway will have a different metric than a straight route, it will use one in preference to the other.

This can be proved simply by a series of speed/download tests. Try with one card only, then with two cards. You won't see a doubling but you should see something like it.

I have tried this on 64k leased lines in the UK with great sucess. Actual channel bonding can be a waste of time. There is a processor overhead in aggregating the channels (Soekris may not be able to cope?), plus a loss of channel capacity as some bandwidth is taken up by the aggregation processes. My experience has been that aggregation can be 10 - 20% faster but by using IP routing - and that is basically what it is there for - you will get about a 180% improvement on a single channel.

My rule of thumb - aggregation OK for bridged links, IP routing vastly preferable.

[tony]

Yes, the metrics are the same. Please let us know how it works, if you are able to test this.

As for the aggregation, we have our own technique for this that should provide low overhead and some resilience if one of the links drops out.

Thanks!

[georgew]

My understanding of TCP/IP is that if you have two routes with the same metric, it will automatically provide load balancing & failover between routes.

Actually, tcp's specification is actually very simplistic, in this regard, it does none of this at all. Load spreading is not automatic in any sense of the word. In fact this is difficult to do, it has taken me many years before I saw a load-balanced T1's that worked well. The *only* load balancer that has worked well was one using proprietary features, with two identical devices at each end running magical code that makes it all work... just barely.

T1's (as well as radios, modems, and most other media) are asyncronous to each other. When bonded on a DS3, it is assumed that the T1's will all drift with different clocks, so the ds3 equipment does not even try to allign t1 circuits. You can try to set the T1's up in allignment with a syncronized bit clock, but I do not know how to sync the frames. As far as I know all equipment doing this uses a unique frame clock for each T1, with no way to sync them.

This is an important issue, as the load ballancer will get packets on the multiple paths in an out of order condition frequently. Most load balancers do nothing about this. Working load balancers detect the packet order, and either maintain all of it, or particular protocols that demand it. In other words, load balancing breaks many protocols, unless it is done well. If you slow down the traffic to make frame alignment easy, you end up needing more memory and processor time, as well as adding to the latency. It is a juggling act that is not well defined by the TCP standard... or any standard I am aware of.

I would suggest that 5ghz support is more important, and that perhaps a 2.4ghz link could serve as a failover, using a route with a longer metric.

However here is another problem... How does the router know it needs to use a different route? There are conditions where a router running a nice high-power routing protocol like BGP will think a link is up when it is not, and be unwilling to use a failover link. Most routing protocols have dark alleys preventing them from working well in every situation. In my experience, routing protocols cause more outtages than they resolve. (I thought about the previous statement long and hard, and while my business partner would tell me that routing protocols are essential, I still stand behind that statement).

So I would think that the programmers would be developing something semi-proprietary if it is going to work well. Proprietary means lots of time. Probably won't happen soon.

George

[lonnie]

Out of order will only be a factor for udp, as TCP is supposed to bale to re-assemble at the receiver. The intention when it was designed was to allow the packets to take differing paths.

We have tested load balancing and it is merely OK. The channel usage put it in the dedicated ptp with no other service planned.

Our efforts are now turning to 5.8 GHz and some more hotspot support.

[Anonymous]

There's no easy way to bond 2 connections... BUT you could improve throughput, especially if there's lots of traffic in both directions, by putting up parallel links, and routing in one and out the other.

802.11b hardware doesn't cope well with 2 way traffic. It will put considerably more through, if it's all one direction. Thus, if you have 2 unidirectional links, you can exceed the capacity of 1, if your data is heavy in both directions.

If memory serves, StarOS implments some version of OSPF (maybe?), which will fail over to the remaining channel should one of them go down.

I'm sure lonnie and/or tony c an be more enlightening, but I hope this idea is helpful...

[lonnie]

Routing bi-directionally will simulate full duplex, and will be optimum for mixed 2 way traffic. If you can get a lot of traffic in one direction then the load sharing will provide twice the throughput.

With 2.4 GHz and its overlapping channels you'll have a hard time to really make this work and then have other units to rebroadcast to other users.

[BNDComm]

in my experience, 802.11b wireless performance suffers heavily when there is bi-directions traffic going through 1 link (the nature of half duplex, rapidly switching from transmit mode to recieve mode). To me this means packet loss.

I have done tests with 2 lucent cards at point A and 2 lucent cards at point b with bi-directional routing, to simulate full duplex and the results were very pleasing.

snappyness(low pings no packet loss) of the network was better even when there was 300k/sec going in both directions in both of my tests I have done

Anyone else done similar real world tests?

Brian

[BNDComm]

I found this out a few years ago when a problem arose. We feed our isp from a wireless backbone to the next town over. One day we experienced about 80% packetloss on our wireless backbone, but it semed to be in 1 direction only. Sita A was getting packets to site B no problem, but B was having a hard time talking to A.

B to A is our upload direction. I hooked an isdn modem to our linux-based isp router, and dialed up into my upstream provider, and sent all outgoing traffic over this isdn modem.

This traffic then came back from the internet through the normal path, back over the wireless babkbone into out network.

In this scenario, the wireless backbone only ever sent traffic in 1 direction, our download direction.

After doing some tests, I was suprised to see how fast the network had become- using an isdn modem to transmit and wireless to recieve was actually faster and no packet loss than using the wireless link to send and recievce.

Interesting....

-Brian

[tim]

I like the hdx to fdx solution, nice and simple, but in practice my traffic is asymetric so although it may give 180% with symetric test traffic, with real world traffic I am not convinced that it will give me that. + I also wanted it for resilience for when links fail + I wanted more than 2 links.

I have been looking into iptables and that seems to offer kernel level support APIs that can be used for load balancing over multiple links. I'm not sure whether this can be supported under staros but if it can that could open up some interesting possibilities.

I understand the importance of getting the 5.8 stuff operational. Would nt it be really cool though to have dual links going with 5.8<g>.

Tim

[angilberto]

in my experience, 802.11b wireless performance suffers heavily when there is bi-directions traffic going through 1 link (the nature of half duplex, rapidly switching from transmit mode to recieve mode). To me this means packet loss.

I have done tests with 2 lucent cards at point A and 2 lucent cards at point b with bi-directional routing, to simulate full duplex and the results were very pleasing.

snappyness(low pings no packet loss) of the network was better even when there was 300k/sec going in both directions in both of my tests I have done

Anyone else done similar real world tests?

Brian

Would you elaborate on this "bi-directional routing"? How would you acomplish this sort of thing? I'd like to give it try, but can't understand how to setup independet routes for each direction...

Thanks,

Angilberto.

[ddvzlnz]

I don't know if star-os is compiled with the required flags, ie:
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y

but it should be easy to do with Linux. The article below goes into some detail on setting this up in the wired world.

http://www.samag.com/documents/s=1824/sam0201h/0201h.htm

It may not be exactly what you desire, but it is a quick way of getting a Linux OS to use multuiple connections.

It may also be another way to build a self healing IP network with redundnat backhaul without using BGP.

gt

[nuclearcat]

Multipath is make load balance route-based, and by this load balance not very good (routes will be cached, and for example if user upload 1 file to kernel.org, he will use only one channel (route) selected first and cached On biffer amount connections/users works better :P

But for bonding as i know, you must have bonding server or supported devices on other side of link :P