General expectations for number of clients, and latency.

[webwave]

I would like to know what the general expectations are for number of clients, speed for each client, and latency under heavy loads.

Currently I am using a WAR2 with a CM9 on the the backhaul link in A mode, and an XR2 in B mode. These interfaces are all bridged, and I'm running approx. 30 clients. I noticed that when a couple of people are downloading, that the latency fluctuates to the other customers that are not downloading. However, the backhaul link remains constant at around 2ms.

At the present time I am throttling the clients to 2.2 mbps download, and 768kbps upload.


Is this normal? If I ran nothing but Atheros based clients, would I see better overall latency even when a couple of clients are downloading? Would routing increase the XR2s capacity without compromising the ping times?

I read these posts, and have to wonder how some of you are running 100s of clients without complaints of latency from the gamers, and voipers.

For this business case to be viable, it seems like you would need to be able to run atleast 100 clients on one AP, without sacrificing anything.

Can anyone pipe in about how many clients they run, what hardware they are running on, and if they have these varying latency issues under a load?

Thanks

[go.fast]

For this business case to be viable, it seems like you would need to be able to run atleast 100 clients on one AP, without sacrificing anything.



For the price point that star offers per ap, you can do 10 to 20 per ap and be successful.

[kbldawg]

Our typical setup is...

5GHz Backhauls
2.4GHz APs X2 cloaking
Everything is routed

I limit our APs to 29 customers because that is what a /27 will allow. Anything over that and I have customers calling complaining of slow speeds.

P2P is a problem for everybody. It doesn't seem to matter if you have 10 or 100 customers per AP. We are building a StarV3 server that will handle all the firewalling/CBQ for P2P rules. We hope to find a working solution for the bandwidth hogs soon. Right now, we are calling the problem customers and dealing with them on a case-by-case basis.

Our customers usually run between 5 - 20ms latency on our network. Our backhauls average about 2ms.

We have about 350 wireless subscribers.

Our services are...

512kbps
1024kbps
2048kbps
3072kbps

synchronous speeds

[tog]

I recommend offering async speeds with wireless for two reasons:

1) Transmit from clients to AP is prone to far more limitation and performance problems than transmit from AP to clients.

2) If you are offering sub-$100 DSL/cable competitive services, the few people who have special business-related needs (running their own server(s), VPN, etc.) are much more prone to negotiate an over-$100 type of service with you.

If a business needs good upstream speeds, I generally call it "dedicated" and do it at $100/meg/mo and put them on my pager.

[DrLove73]

The key IS routing. Been there, done that. NO MORE bridging for me.
I had latency go to 600ms or even more with bridging, with only 500-600 Kbit/s/ap. Now latency is as it should be, excelent.
Also, I do not like even SR2, not to mention XR2! More power means more interference. I had 1W and 2W boosters (I still do, but I am only going to sell them, not to use them). You do not get quality with card power, only with antenna gain.

[DrLove73]

Concerning card power vs. antenna gain.

There is excelent text by Dragoslav Dobricic, YU1AW who published papers about antennas, but it is on serbian. His page with home made but quality antennas is:
http://yu1aw.ba-karlsruhe.de/engl.htm

His point in his calculations is:

freq f= 2400 MHz, distance AP i clienta d= 1 km,
free space loss Ls=32.4+20*log(f)+20*log(d)=32.4+67.6+0= 100dB.
sending card power Pt=13 dBm (20 mW), reciver sens. Np= -85dBm, cable loss 1 dB on each side Lc=1+1= 2 dB and both antena boost is Gt=Gr= 10dBi.
Calculation gives recive signal power
Pr=Gt+Gr+Pt-Ls-Lc=10+10+13-100-2= -69dBm,
and signal/noise ratio is S/N=Pr-Np=-65-(-85)= 16dBm,
witch is enough for connection.

More power, lesser antenna scenario:
AP side has +10dB card power, tj. Pt=13dBm+10dB= 23dBm (200 mW), and antena is -10dB, Gt=10dBm-10dB= 0dBi (izotropic antena).
Client side is the same (10 dB of antenna on Ap side is compasated
with 10 dB more card power.
AP S/N is same, 16 dB.

But, AP's reciving is diferent:
S/N= Gt+Gr+Pt-Ls-Lc-Np=10+0+13-100-2-(-85)= 6dB,
that is -10dB then in previous case, and it is below good connection margin (S/N>10dB)!
So, client has good connection, but AP does not! NO LINK!

Lesser power, biger antenna gain scenario:
AP card power 10dB lesser then client Pt=13-10=3dBm (2 mW), but we put AP antenna with 10dB higher gain. Gt=10+10=20dBi.
Client is same becouse AP-a is sam (+10-10=0):
S/N= Gt+Gr+Pt-Ls-Lc-Np=20+10+3-100-2-(-85)= 16 dB.
But, AP has improved reciving signal:
S/N= Gt+Gr+Pt-Ls-Lc-Np=10+20+13-100-2-(-85)= 26 dB.

SO, lesser card power, higher antenna gain, beter AP performance!

[rbolduc]

The key IS routing. Been there, done that. NO MORE bridging for me.
I had latency go to 600ms or even more with bridging, with only 500-600 Kbit/s/ap. Now latency is as it should be, excelent.
Also, I do not like even SR2, not to mention XR2! More power means more interference. I had 1W and 2W boosters (I still do, but I am only going to sell them, not to use them). You do not get quality with card power, only with antenna gain.

I replaced a WLM with a XR2 and see much better throughput and quality, I am though running the card powered down to 8 (18 real) DB output as I had no need for the power I just wanted the better filter/grounding and the mmcx connector the xr2 has to offer. Power I just don't care about that much any more! FWIW

Reed

[Stratolinks]

I replaced a WLM with a XR2 and see much better throughput and quality, I am though running the card powered down to 8 (18 real) DB output as I had no need for the power I just wanted the better filter/grounding and the mmcx connector the xr2 has to offer. Power I just don't care about that much any more! FWIW

Reed

It generally take actual experience before people will get to that conclusion. I have clients on our system that are turned as low a 3dB for tx power and they still get back to the AP at -61dB, right where I like to keep them. Sure the more distant clients have lower signals, but at least they can get through without being drowned out by the local clients.

Terry

WAR1 Clients ROCK !

[Beebe]

Do XR2s have super features and compression, fast frames all the good stuff?

[webwave]

Thanks for all of the input..

I'm currently bridging everywhere. The wierd thing is that I only see latency issues going from the AP to the customer.. I never see ping latency go higher than about 5ms on the backhaul..... I simply don't understand this.. if there is as much traffic going through the XR2 to the client as going through the CM9 to the backhaul link.... Then logic states that the backhaul would see just as much latency as the XR2. I'm wondering if perhaps it is my cruddy clients... (Realtek 8186) that cannot handle the contention in the air, and that is why I see the latency.. This is why I'm wondering if others see this same phenomenon with an all atheros network.

here is a sample...

ping to the client:
--- 192.168.110.126 ping statistics ---
1392 packets transmitted, 1390 received, 0% packet loss, time 1398739ms
rtt min/avg/max/mdev = 4.409/24.428/524.945/42.424 ms

ping to the AP that client is connected to:
--- 192.168.110.2 ping statistics ---
1395 packets transmitted, 1395 received, 0% packet loss, time 1398681ms
rtt min/avg/max/mdev = 2.379/3.455/18.955/1.658 ms

You can clearly see that the latency can spike to MUCH higher values.. This happens when even ONE subscriber is downloading at 2.2 mbps.. YET, the backhaul stays right at 2, or 3 ms..

Also, I grow tired of hearing... "You don't need that much power....." The XR2 was touted to be a true carrier class solution with superior ESD protection, 1900 mHz rejection, higher recieve sensitivity, and 600 milliwatts of power... We don't buy it JUST for the power. Most of us, including myself, do not turn the power all the way up.. Most of us know that turning up the power past a certain point is not good due to noise/performance ratio.. Besides.. I really was under the impression that when these radios were running at the 54 mbps rating.. that the radio is not running at NEAR the power as it is when in 11mbps mode.... right??? So the thought was... "Good power at higher ratings...."

Also, a thought about StarOS in general.. I bought into this solution because of the DIY aspect of it.... I kinda felt that it was the "best kept secret" in wireless.... And if one were so inclined, that he/she could achieve carrier class service comparable to a Canopy type system for a fraction of the cost.. Simply for taking the time to DIY... Meanwhile, I see my competitor (who is running all canopy) taking on 1000s of customers, and giving a guaranty that their VOIP service will work flawlessly.
I feel that if I use nothing but high quality Atheros clients, cloaking, super AG features, and prioritization techniques on the radio... That I should be able to do the same thing... Am I wrong? If I am.. then I suppose it's time to mortgage the house, for a Canopy startup kit.... LOL

Overall, I am happy with the service it delivers for the price point. But still don't see how I can truly give good service to gamers, voipers, etc.. and still give something over 2 megabits to sustain my company in this market..

I'm not sure if any of you have heard of Utopia.. but its basically a fiber network to the home. (FTTH). It is funded by several cities, and is delivering 15mbps symmetrical bandwidth for 40 dollars per month.. It's theoretical max is 1 Gbps... Dubbing us WISPers as the next "dialup". http://www.utopianet.org/

Bottom line is that I expect the clients that are atheros, and that support cloaking to have the same performance that I see in the back haul since it is basically the same thing.. (client connecting to AP). Just wondering if anyone else has this setup, and can tell me if this is truly the case.. And also how many clients really can be run while give good QOS to ALL clients when starV3 is ran across the entire network on good hardware..

The question again is...... Why the high latency on the broadcast, but not even as much as a flutter on the backhaul..

Thanks

[webwave]

Yes XR2 offers all of the atheros features that we have all come to love.

Also, I failed to mention that I will be implementing routing next since several people seem to agree that routing reduces latency.

[webwave]

P2P is a problem for everybody. It doesn't seem to matter if you have 10 or 100 customers per AP. We are building a StarV3 server that will handle all the firewalling/CBQ for P2P rules. We hope to find a working solution for the bandwidth hogs soon. Right now, we are calling the problem customers and dealing with them on a case-by-case basis.


I would recommend setting up a bridging firewall with pfsense http://www.pfsense.org (http://www.pfsense.org/) It has a traffic shaping wizard that is outta this world, and will give you complete control over that nasty lil P2P problem.

Atleast until you get that StarOS box configured.. ;)

[go.fast]

I would recommend setting up a bridging firewall with pfsense http://www.pfsense.org (http://www.pfsense.org/) It has a traffic shaping wizard that is outta this world, and will give you complete control over that nasty lil P2P problem.

;)

Does it handle the latest encrypted torrents?

[tony]

pfsense does not have any layer-7 filtering abilities.

[DrLove73]

As soon as you upgrade your network from bridged to routed, you will forget all about latency problem.

Lets say you have 3 POP's with 3 sectors each, 20 clients per sector, all bridged. That would be 180 clients on the same subnet.

Now, one of the clients connects to the network after some inactivity period. His MAC address says "Hi, who also has my subnet?"

And 180 of them say "WE DO" as load as they can! But that new client does not know where physicaly those client are.

You must had your computer freeze when you try to read out damaged CD/DVD disk. Whole computer just holds its breath waiting for data to arive. Until it does, you pound on the table or even keyboard with your fist.

Now, when your clients computers try to connect to some other clients computer for any reason, and that computer went to power saving just seconds ago, it sends "Where are you ^$#@&!!!" across the network
, to all 180 of them. Until reciving client answers, it holds back its breath. When enough clients do this all the time..... large latency makes your hair stand out.

This was not very scientific explanation, probably not exactly accurate, but rather a colourfull aproach. The exact couse is somewere in this forums, STF for it, but the punch line is close enough. You will se as soon as you route your network.

P.S. Every single one client that I installed is Ovislink CPE based on Realtek 8181 (11b) and 8186 (11b/g), and they are not cousing this problem, bridging does. And of course, turn OFF InterBSS Relay, it is also pan in the a..

[webwave]

Does it handle the latest encrypted torrents?Unfortunately no, it is only layer 3.. The only one that does at this point I believe is NETFILTER within IPTABLES... I didn't realize you had torrents using standard ports....

You have my condolences... :mad:

I suppose I will start subnetting my class C this week.. I appreciate anyone who can offer suggestions, and point out any pit falls before I begin....

[DrLove73]

Put plan of your network on paper first, with ALL interfaces. Then write down subnets for every subnet, and then start putting routes first. Start with static routes, then when as you learn more, you can stepup to RIP, OLSR,... . when you are done with that, only then start changing.

Do periferals (fartherest AP's) first. then go inward to the core.
As you come to the core, your units will have more and more routes to insert. It would be smart do use 172.16.0.0/12 or 10.0.0.0/8 and branch (group) subnets, brake them in smaller groups.

Parts of 172.x.x.x range are PUBLIC IP's!!! I made mistake!!! I changed examples from 172.x.x.x to 10.x.x.x range.

Table 2-1. IP Address Ranges Reserved for Private Use
Class Networks
A 10.0.0.0 through 10.255.255.255
B 172.16.0.0 through 172.31.0.0
C 192.168.0.0 through 192.168.255.0

If you choose to use 10.0.0.0/8,
then backhaull 1 over first interface should be 10.0.0.0/11,
over second put 10.32.0.0/11, etc.

That way, you will get 8 main backhauls. Link to online IP Calc is here (http://jodies.de/ipcalc?host=10.0.0.0&mask1=8&mask2=11)
Using 10.x.x.x/10 will give you 4 of them.

Then on next hop on backhaull 1, to branch 4 backaulls:
on interface 1 put for example10.0.0.0/13
on interface 2 put 10.8.0.0/13 , etc..

braking larger subnets to parts gives following effects:
From /x to /x+1 brakes subnet to 2 parts.
From /x to /x+2 brakes subnet to 4 parts.
From /x to /x+3 brakes subnet to 8 parts.
From /x to /x+4 brakes subnet to 16 parts.

Then you can write only one static route entry for all subnets on that part of the backhaull. So entry:
10.0.0.0 255.224.0.0 172.0.1.x (10.0.0.0/11 (http://jodies.de/ipcalc?host=172.1.0.0&mask1=8&mask2=11))
in root (main) backhaull unit will cover all IP's on that backhaull in range 10.0.0.1 - 10.31.255.254, with 32 x 256 subnets, instead writing them all separate.

Chose /x wisely, there is no turning back once you implement them (all you will have major headache :D ). When u brake larger subnet to larger number of smaller subnets, on all smaller ones you have lesser number of available subnets to play with.

If you use cheap CPE like realtek 8181 or 8186 based, that do not posses NAT-ing, consider to use IP x.x.x.100 for gateway (AP's IP). In doing so, you use x.x.x.1-99 for client CPE IP's, and x.x.x.101-199 for client IP's respectively.
Example:
AP's IP = 192.168.212.100
CPE = 192.168.212.22
LAN on PC = 192.168.212.122 with gateway set to 192.168.212.100
I found that it is extreamly easy to remember with ones to CBQ, any easily remember CPE's IP's, for any of your workers, no mater the IQ :p.

Of course, you will have to change IP's on every client also, so do only one antenna/interface at the time. First you put new IP on the interface (after ALL routes for ALL POP's have been entered), then switch clients IP's, and then disengage bridging for that interface, one at the time.

[oscarBravo]

Bear in mind that 172/8 isn't a private address range. The RFC1918 range is 172.16/12, which means you can use from 172.16.0.0 to 172.31.255.255.

[webwave]

Of course, you will have to change IP's on every client also, so do only one antenna/interface at the time. First you put new IP on the interface (after ALL routes for ALL POP's have been entered), then switch clients IP's, and then disengage bridging for that interface, one at the time.

Good tip.. thanks..

I still don't know how ARP broadcasts, and replies could be causing that much latency, and why it doesn't happen on the backhaul link.. especially since my network is very very small... Routing is going to add a complexity that I'm really not looking forward to.. (port forwards, 1:1 nats.. etc...)

[DrLove73]

lol. I thought there was something I mised with that range, but could not remember what it was. Thanks.
When I used 172.x.x.x range, I (for some wierd reason :-) ) chose 172.20.x.x - 172.31.x.x. I guess I now know why.

Bear in mind that 172/8 isn't a private address range. The RFC1918 range is 172.16/12, which means you can use from 172.16.0.0 to 172.31.255.255.

Search The Forums! I belive tog knows the way to assign public IP's to client units, if they have NAT-ing. My way is NAT-ing only on my main gateway (RHEL Linux box - Clarkconnect (http://www.clarkconnect.com), very easy to use), not anyware else, just on 1 place.

I wrote that it is close aprox of real reason, I forgot what it was exactly. STF also on that, posts of guys here convinced me to route my network, I did not think of it my self.

There is also online books on networking (http://forums.star-os.com/forumdisplay.php?f=32). Read it, AND STF!!! You will learn so much on these forums.

[rafamous]

webwave > Route NOW!

If you want an easy business then your in the wrong business. Route or it only gets worse.

[webwave]

I will start routing.

I didn't expect an "easy business"... I expected a slightly better business case.. There's a diff....

[rafamous]

Routing is a pain in the rear at first, but once you get it you'll pat yourself on the back for taking the effort.

Have a super day!

[ripv]

webwave,

here is a good concise case for routing - posted a while back here
http://forums.star-os.com/showthread.php?t=6516

==========================

Hello,

I was asked to post a write-up I just did in the isp-wireless.com mailing list concerning routing and bridging- here it is reprinted- I hope you find it useful.

Advantages and Disadvantages of a Routed Topology-

Routing has a number of Advantages

- Increased visibility into your network. For graphing, monitoring, and troubleshooting. For instance, where's the packet loss occurring? Discerning that kind of information from L2 frame statistics can be, many times, challenging. Routed hops are easier to track that down on.

- Ability to properly enable QOS. This helps guarantee communications quality for select services (like VOIP.) CBQ fits here, too.

- Ability to properly inact security filters. Especially if the routers in question are linux/BSD boxes, which have the ability to load various security software apps to further secure a network (read- snort and other IDS/IPS solutions.)

- Better redundancy and re-routing. As a network grows in inter-connected locations and complexity, IGP routing suites (RIP2, IS-IS, OSPF, IBGP) will be a greater benefit to it than (R)STP.

- Scalability. A Layer2 switched network doesn't scale well for wireless applications. Routing solves the network scalability problem. This is related mainly to broadcast domains that a network's devices exist in- it is often desirable to segment the broadcast domains (which routing does) as opposed to sharing one big broadcast domain (which switching does.) As the network grows, routing will help interconnect those portions of the network in ways that switching will be deficient.


Routing has a few Disadvantages

- Increased configuration complexity. Switches are really easy to work with compared to routers, and considerably easier than Linux/BSD boxes acting as routers and security/monitoring devices. Many switches today come with a web-gui, and while many routers have some web-gui functionality, to truly take advantage of them, operators normally have to be proficient with their CLI (command line interface.) Even that isn't so bad for a single vendor implementation, but if hardware or operating systems from multiple vendors are used in a mixed environment (which is always the case to some degree), their individual web-gui or CLI systems will all have to be learned to some level of proficiency.

- Increased support costs.Hiring someone who knows routing is more expensive than hiring someone who knows nothing that will be taught switching. Pretrained, expert-level consultants will likely spend more time working on a routed network than a switched network, meaning increased consulting fees.

- Increased points of failure. While routing provides redundancy to help eliminate single points of failure (mainly of the hardware and circuit variety), because of their increased responsibilities and complexity, they present more options and services that can fail, any of which may ultimately result in an outage on the network. For instance, dumb switches talk to each other out of the box and just starting exchanging L2 frames, but routers each need their own routing configuration, and each router has a chance of being misconfigured. Add the additional services to configure (QOS, security) and there will be an increase in the likelihood of creating problems... basically the opposite of KISS (Keep It Simple Stupid.)


In the long run, I think the network exampled by the original poster could greatly benefit from a routed topology. But if you start now, be prepared to question that logic for a good 3 to 6 months. You won't see too much of a change in service at first, and you will definitely be sacked with lots of implementation challenges (especially if you aren't already a routing expert.) But in the end, I believe it is worth it, especially if your network is expected to grow.

If you have any specific questions about this or how it fits into your current network challenges, I'd be happy to discuss. Again, I hope you found this breakdown useful.


Danno

http://danno.appliedi.net/drupal/

[webwave]

Yeah actually I was kinda hoping for an immediate improvement in latency on the customers side.. But I see what you mean..

Also, I was under the impression that addresses like 172.x.x.x and 192.168.x.x. would not route... they are always referred to as "non routables"

can I truly route with these "internal" address sets?

also, would someone please look at my current topology..


Internet <--nat--> internal network <--bridge--> wireless backhaul <--bridge--> customer public side <--nat-->customer internal side

Do I need to route every single hop on the network? even though I don't even see as much as a 2 ms jitter on the backhaul?

Also, cheap 8186 clients DO nat.. just FYI..

Again I want to re-iterate that I only see latency on the broadcast side of my AP when there is about 2 or more people downloading at around 2.2 megabits.. I want someone AGAIN to give me assurity that if I route, that the one customer that games will see 5ms ping times even if another customer is doing a download..

I'm assuming that the reason everyone keeps telling me that I will have better latency is because I will be able to implement QOS features on the radio.. but I still don't get why I have NO latency issue across the backhaul...

[butchkemper]

Also, I was under the impression that addresses like 172.x.x.x and 192.168.x.x. would not route... they are always referred to as "non routables"


"non routable" means these IP blocks will not be routed in the Internet. You may route them within your network.

Butch
.

[DrLove73]

Yeah actually I was kinda hoping for an immediate improvement in latency on the customers side.. But I see what you mean..
You will see immediate improvement if you already have a problem.

Also, I was under the impression that addresses like 172.x.x.x and 192.168.x.x. would not route... they are always referred to as "non routables"

can I truly route with these "internal" address sets?

YES, but only inside your network. For exit to public IP space, NAT is necessary.

also, would someone please look at my current topology..

Internet <--nat--> internal network <--bridge--> wireless backhaul <--bridge--> customer public side <--nat-->customer internal side

Do I need to route every single hop on the network? even though I don't even see as much as a 2 ms jitter on the backhaul?

You should. If you have only one hop, you are immune until your appetite for income outgrows your bridged network, and your fall in to despair imagining simultaneous change of 50-100-300 customers IP's.

Also, cheap 8186 clients DO nat.. just FYI.. NAT at the clients unit is a must if you want security for the client, and also, this significantly buffers your bridging problem.

Again I want to re-iterate that I only see latency on the broadcast side of my AP when there is about 2 or more people downloading at around 2.2 megabits.. I want someone AGAIN to give me assurity that if I route, that the one customer that games will see 5ms ping times even if another customer is doing a download..
IF the problem is ONLY bridging, that problem will be solved. Beside routing, if your clients are using weak antennas (<20dB on 2-3 miles), or you have great interference, problem might not go away completely.

Look my old posts for bridging/routing, and will see what I had (2 PoP's connected to 3-rd, all bridged, and what problems I had. 18 months
latter, I just waggly remember that I even had some latency issues ;-)

Bottom line, it can only be better, and you can only start to have less phone calls. Boy when I just remember those days.... brrrr... chill goes true my veins.

I'm assuming that the reason everyone keeps telling me that I will have better latency is because I will be able to implement QOS features on the radio.. but I still don't get why I have NO latency issue across the backhaul...

NO, Qos is not a real reason. Disabling people to (by bridged network) shout in each others ears, is the main reason. This is just colorful representation, and shouting is meant as IP babbling, not an RF issue.

Even InterBSS can be BIG problem. Just chokes your radio card.

Decision is your, upgrade your network or not.

[gunther_01]

Webwave,

I don't think it is mentioned in here what firmwares or AP device in use. I fully agree with everyone about not bridging.

But if any of your clients have poor signal strengths or qualities and your AP is switching data rates while you have 2.2 Mb downloading as well as the potential to have 768K uploading at the same time. There is most likely your spikes in latency from AP to client. Just about anytime I have a client max out his upload, I have problems with qualities going down across the board.

The joys of a half duplex system. There is only so much air time

[webwave]

I believe that the interference is actually the issue.. The other evening I got a call from a gamer who was seeing 200ms latency. When I looked at the radio I saw ONE sub doing a download. He was getting about 1700 kbps down, but losing 60 percent of the packets!

I think this is the real issue. You have to remember that this topology is not like dsl where every sub plugs into their own line card.. it's one radio taking care of 20 plus people!!

but I also feel that there IS latency caused by bridging. And I still plan on re-designing, and re-implementing the network.

One more questions..

Someone mentioned "interbss relay".. does it reak havoc when it's on or off? Right now I have it off, but I really could keep it on since every client is natted..

Thanks

[DrLove73]

If you turn it on, clients will be able to see each other, but they will use your wifi card as the relay, but without ANY control, firewall or bandwidth. If they start file sharing, you are done for, latency will go sky high.

So keep it OFF.