Routing design - help!

[valenti]

I've been trying to figure out how to switch my little net from bridging to routing. While I have worked with networks for many years, I have never needed to design one. I've looked over the manual, researched on the web, etc but still don't have much confidence in this "design".

The following is what I have come up with, please let me know how it looks. If you can point me to a sample layout documented elsewhere, I'm more than willing to use that if it meets my needs.

(if this isn't torn to shreds, I'll try to do a diagram later this weekend)

Routing design goals

Switch from bridging to routing.
Use my /26 of public IPs conservatively.
Connect customers behind NAT, unless they specifically need a public/static IP.
Allow for growth to many POPs (microcells).
Move customer packets around my net on a known IP until the egress point where they are NATTED (CALEA).
Use 10. net for less typing.
Allow for easy access (ssh) right down to customer radio.
Allow for multiple static IPs at customer, if needed.
Customer radios could do NAT for them if needed (but typical setup connects them to a wifi router with firewall that would handle their household NAT)
Use OSLR for internal network routing.
Minimize amount of static routes and routing config on each WAR.
Allow for multiple connections to the public internet.
All net traffic to be encrypted.
Most POPs would link to at least two others.

Current situation
I have about six POPs and one backhaul site. Everything is bridged. RF connections seem pretty solid (except for a few that are waiting for antenna upgrades). I have at least five more POPs lined up.

Design
Use second octet for POP #, example 10.3.1.1 would be an interface on the WAR board at my #3 POP.
Use third octet for each net (so a particular AP's radio might be 10.3.6.1
(reserve .0 thru .5 for ethernet ports, so radios would usually be .6 thru .9)
DHCP would be turned on for each radio, customers would eventually get a static entry for DHCP.
"Core router(s)" at egress point(s) would need to do NAT for the whole net.

Practice
(much hand waving here) What would I actually have to configure on each WAR?

Questions
Can StarV3 handle the "core routing"? I'm pretty sure I could do that with openbsd's pf if not. It would need to do the one to one NAT for customers getting a static IP and masq for the others.

Help!

[rafamous]

RIP works great on V3 to do most of the routing work.

[gunther_01]

I've been trying to figure out how to switch my little net from bridging to routing. While I have worked with networks for many years, I have never needed to design one. I've looked over the manual, researched on the web, etc but still don't have much confidence in this "design".

The following is what I have come up with, please let me know how it looks. If you can point me to a sample layout documented elsewhere, I'm more than willing to use that if it meets my needs.

(if this isn't torn to shreds, I'll try to do a diagram later this weekend)

Routing design goals

Switch from bridging to routing.
Use my /26 of public IPs conservatively.OK

Connect customers behind NAT, unless they specifically need a public/static IP.Done at edge

Allow for growth to many POPs (microcells).That's what the routing does

Move customer packets around my net on a known IP until the egress point where they are NATTED (CALEA).Once again routing helps here
Use 10. net for less typing.OK
Allow for easy access (ssh) right down to customer radio.Routing again takes care of this when done completly

Allow for multiple static IPs at customer, if needed.Routing again
Customer radios could do NAT for them if needed (but typical setup connects them to a wifi router with firewall that would handle their household NAT)Only issue here is double NAT

Use OSLR for internal network routing.Your main back huals and AP's should be Star to do this

Minimize amount of static routes and routing config on each WAR.Close to none if you use a dynamic routing protocol like OSLR, but you will have to configure them at least once

Allow for multiple connections to the public internet.Ticky and depends on what you mean, like redundant paths or just multiple feeds to aid in bandwidth

All net traffic to be encrypted.No WPA yet in stable V3, but it's coming

Most POPs would link to at least two others.As long as you have enough radio slots this is not a problem



Design
Use second octet for POP #, example 10.3.1.1 would be an interface on the WAR board at my #3 POP.
Use third octet for each net (so a particular AP's radio might be 10.3.6.1
(reserve .0 thru .5 for ethernet ports, so radios would usually be .6 thru .9) As long as clients can route, this should be fine. If they are going to be Star, no problem

DHCP would be turned on for each radio, customers would eventually get a static entry for DHCP.OK

"Core router(s)" at egress point(s) would need to do NAT for the whole net.OK

Practice
(much hand waving here) What would I actually have to configure on each WAR?IP's addresses, MASQ staements for customers, OSLR, Gateways maybe (have not used OSLR yet), general AP settings and other I am sure

Questions
Can StarV3 handle the "core routing"? I'm pretty sure I could do that with openbsd's pf if not. It would need to do the one to one NAT for customers getting a static IP and masq for the others.Yes but you don't have BGP for redundant ISP based Internet feeds yet

Help!

Give us your current layout and devices across your network, I for one am happy to help. But keep in mind it's up to you to learn it (routing).
Quote edited for replies

[valenti]

Gunther_01, thanks for the offer....

I'm currently fighting with the diagramming software, but verbally, here is what I have:

Backhaul site - TrangoLink 10 remote

Main POP - TrangoLink 10 master
Trango 900 AP
WAR2 AP for 2.4 clients, spare radio
WAR2 backhaul to POPs 2 & 3
OpenBSD computer

POP2 Trango 900 SU (current backhaul)
WAR2 AP for 2.4 clients, spare radio
WAR2 for backhaul to main POP & POP 3

POP3 Trango 900 SU (current backhaul)
WAR1 AP for 2.4 clients
Trango 900 AP (few clients & backhaul for POP4)
WAR1 backhaul to main POP -- this needs to be a WAR2 for OLSR

POP4 Trango 900 SU (backhaul)
WAR1 AP for 2.4 clients

POP5 WAR2
first radio connects back to main POP
second radio has omni for 2.4 clients

I plan to use StarV3 (either WAR2 or WAR4) for all radio links eventually. The TrangoLink10 has been perfect, so I won't replace it immediately.
My POP 2 listed above is the one that I hope to get a second Internet feed at later this year. That would probably be my primary, the other would just be a backup.

If I need to use BGP to connect to the two ISPs I would probably use OpenBSD+OpenBGP.

[gunther_01]

OK, so add an edge router in there at the begining of it all. Add your IP's (to your list on the forum) that you would like to have in there for the layout you had mentioned before. We can go from there.

Only issue as of right now that I see, is that you are using an AP (900) with multiple clients for backhauls. That's fine for now, but you should get away from that as soon as possible. Once we get the WARs all linked up and routed that should not be a problem.

Question, are any of the WARs connecting with each other at the moment? Without the 900 doing it.

[valenti]

Gunther, to answer your questions (belatedly):

I've worked on the edge router, I think that part is OK.
Yes, I know the AP combining clients & backhauls is bad.
Yes, most of the WARs are linked up (and getting much better connections than the 900)

I have a new design sketched out, I will post in a minute.

[valenti]

Ok, trying to get back to this. My first attempt using 10.x addresses didn't work out, I had the subnet masks all messed up. Here is what I have for a ring including my three primary POPs.

Routing sample for a ring with 3 POPs

POP2 ===========
WAR2
eth1 10.2.2.1 / 16 to bsd Internet gateway
eth2
wpci1 10.3.6.2 / 24 link to POP3
wpci2 10.4.6.2 link to POP4
default gateway is 10.2.1.1 thru out network

BSD box doing NAT
eth1 69.41.x.y - Internet connection
eth2 10.2.1.1 / 16

Other WAR2's here would use 10.2.6 - 15.1 for client AP radios

POP3 ===========
WAR2
eth1 10.3.1.1 to switch connected to other WAR2s
eth2
wpci1 10.3.6.1 link to POP2
wpci2 10.4.7.2 link to POP4

Other WAR2's here would use 10.3.7 -15.1 for client AP radios

POP4 ===========
WAR2
eth1 10.4.1.1 to switch connected to other WAR2s (or direct connection if only one)
eth2
wpci1 10.4.6.1
wpci2 10.4.7.1

Other WAR2's here would use 10.4.8 -15.1 for client AP radios

[gunther_01]

Any particular reason you are using such a huge block of IP's for your interfaces? (/16 on the BSD box)

I would also renumber them so the AP for your bachauls is number 1 instead of 2. like pop4 I look at as having two backhaul AP's with now where to connect to on paper.

[valenti]

Any particular reason you are using such a huge block of IP's for your interfaces? (/16 on the BSD box)


I wanted to have the second octet vary for each POP. (And the third octet would vary for each radio. This is probably overkill, since I can't imagine more than a dozen radios at one pop)

Given that all the other networks would be flowing thru this point, I thought I needed a /16.

If that causes problems, I'm open to other ideas.

[valenti]

I would also renumber them so the AP for your bachauls is number 1 instead of 2. like pop4 I look at as having two backhaul AP's with now where to connect to on paper.

I was trying to be consistent and have the second octet show whatever POP that subnet starts out on. But for a backhaul, the subnet is on two POPs.

Network-wise, it doesn't matter much which end of a backhaul has the AP, right? One end would be .1, the other .2. I would probably setup the easiest end to climb as the AP.

[gunther_01]

The joy of routing is that your subnets can be as large or as small as needed. Personally I would use a /24 for each Ap and client set, and even a /24 for the BSD box and backhauls links (/24 for each subnet/radio, client set). I think your thinking of the /16 would follow suit with a bridged network. But with your eventual ring I am not sure if you do or not need it with failover in place. I wouldn't think so.

My thoughts on AP placement are the same as yours. You want the AP to be at the most easily accessable place. But also, I think that you need to have the AP closer to your gateway for things to work/flow properly. Again not sure how dynamic failovers come in to play there.

I personally choose to have my AP numbered the first available IP out of an IP subnet. Others use the last available and work backwards for IP allocation.

[rbolduc]

You can use any size at an AP I start with a /27 on each ap, thats good for 30 ips -1 for the radio. All my backhauls are using a /30 you can always add another block as needed.. after all its routing ;)

Reed

[valenti]

Thanks for the comments (and keep them coming).

I did get a message from someone that is using a similar system to what I proposed and they are happy with it. They are using static routing, but I'm too lazy for that so I hope I can figure out the olsr. (Plus static can't support redundant links or the ring, can it?)

I put the edge router/NAT into place this morning, and have started switching WARs over to the 10.x addresses. Hope to get a few customers on it this weekend.

[rbolduc]

Thanks for the comments (and keep them coming).

I did get a message from someone that is using a similar system to what I proposed and they are happy with it. They are using static routing, but I'm too lazy for that so I hope I can figure out the olsr. (Plus static can't support redundant links or the ring, can it?)

I put the edge router/NAT into place this morning, and have started switching WARs over to the 10.x addresses. Hope to get a few customers on it this weekend.

I am running rip, it is very easy to get going. But I will be switching to OSLR soon once I get a little more time to play with it and figure out some of the basics.

Reed

[gunther_01]

RIP for me as well. I just made sure to use the "neighbor" statements and have never had a problem. Do a search on that for why and where.

[valenti]

I'm supposed to remove the interfaces not using OLSR in the config file, right? I'm only using wpci2 (as displayed in StarOS), is that still wpci2 in this config file? It seems to start numbering them at zero....

[tog]

You have to use the system devices which do start at wpci0, wpci1, wpci2, wpci3.

Yes, when crafting your OLSR configuration you should only specify interfaces which need to be running OLSR.

[valenti]

You have to use the system devices which do start at wpci0, wpci1, wpci2, wpci3.

Yes, when crafting your OLSR configuration you should only specify interfaces which need to be running OLSR.

thanks, tog!

hmmm, I'm still missing something:

POP2: bsd 10.2.1.1/16 <> war2 10.2.2.1/16 10.3.6.2/24

POP3: war2 10.3.1.1/24 10.3.6.1/24

I have OLSR turned on for both WAR2s. No default route on the one at POP3. The wireless link between 10.3.6.1 & .2 is good. HNA4 0.0.0.0 at POP2.

From the war2 at POP2 I can ping out to the internet and to 10.3.6.1
(but not 10.3.1.1)
Do I need an HNA4 {10.3.1.0 255.255.255.0} at POP3?

From the war2 at POP3 I can ping to 10.2.2.1, but not 10.2.1.1.
Not sure how to fix that one.

[tog]

Do I need an HNA4 {10.3.1.0 255.255.255.0} at POP3?

Yes.

In order for routing to work, you must have two-way reachability.

POP2 must know how to reach POP3's IPs, you do that by having OLSR over at POP3 announce its IP blocks. POP2 then "intelligently" figures out via OLSR how to reach those IP blocks you're announcing at POP3. Only then do POP3's IPs become reachable.

Not that it truly matters, but why are you using a /16? That seems kind of unnecessary and kind of throws a bit of unnecessary confusion in. How about a nice /30 or at least a more simple /24? You could have 10.0.0.1/30 at one end and 10.0.0.2/30 at the other end of a PtP link for example.

[valenti]

Yes.

In order for routing to work, you must have two-way reachability.

POP2 must know how to reach POP3's IPs, you do that by having OLSR over at POP3 announce its IP blocks. POP2 then "intelligently" figures out via OLSR how to reach those IP blocks you're announcing at POP3. Only then do POP3's IPs become reachable.


Ok, I was thinking olsr was more intelligent/powerful, that it would automatically figure that out.

I only need to put the extra HNA4 routes in at the closest router, right? Then olsr will pass that info along to the other routers automatically.

So, for instance, if I extend this to another hop:
POP4: war2 10.7.6.1 10.7.7.1
Passing out client addresses such as 10.7.7.x, with the 6.1 linked back to POP3. I would just put in a HNA4 10.7.7.1 255.255.255.0


Not that it truly matters, but why are you using a /16? That seems kind of unnecessary and kind of throws a bit of unnecessary confusion in. How about a nice /30 or at least a more simple /24? You could have 10.0.0.1/30 at one end and 10.0.0.2/30 at the other end of a PtP link for example.

I was just trying to simplify it (in my mind). The second octet would always tell me what POP was involved. Using the /30 is Ok the first time, but then what consistent scheme would I use for the next dozen backhauls?

I'm just trying to get to a system that I can drive up to a grain leg, call it POP27 or whatever and know how to add it into my network without spending hours planning out routing.

[tog]

You have to announce the network via Hna4 at the router that has the network attached to it.
If the router doesn't have the network attached to it, it shouldn't be announcing it via Hna4.

With this setup you can in fact light up a new board, call it POP27 and announce your block via Hna4 and the rest of the network will figure out how to reach the network you just started announcing.

As for a consistent scheme, you could use 10.0.0.1/24, 10.0.0.2/24, 10.0.0.3/24 assuming you're connecting these all to the same AP interface for backhaul purposes. If they're all going on their own PtP AP interface, then it would be more like 10.0.0.1/30 and 10.0.0.2/30... 10.0.0.5/30 and 10.0.0.6/30, etc.

[valenti]

I'm still fighting with my routing setup. This has gone on too long. I'm looking for a consultant$, if anyone wants to recommend someone or volunteer.

I still think that what I've suggested above should work, but I just can't figure it out. (actually someone emailed me that they use this system and like it, but they are using static routes and not olsr)

[DrLove73]

I can not help you with this, but just a question. Have you used new OLSR config that tog posted on his wiki (http://staros.tog.net/wiki/Main_Page)? I do not know when this was put up.

[valenti]

I can not help you with this, but just a question. Have you used new OLSR config that tog posted on his wiki (http://staros.tog.net/wiki/Main_Page)? I do not know when this was put up.


Good question. I thought I was, but the one on the wiki now seems different. I'll try to switch them over.

I don't see a way to move olsr scripts with starutil - am I missing something?

[ninedd]

I'm still fighting with my routing setup. This has gone on too long. I'm looking for a consultant$, if anyone wants to recommend someone or volunteer.

I still think that what I've suggested above should work, but I just can't figure it out. (actually someone emailed me that they use this system and like it, but they are using static routes and not olsr)What we do is this...

We use the 10. network and we NAT at our border StarOS machine.

We number each AP on the second octet, so 10.1.x.x is the store, 10.2.x.x is the next machine, 10.3.x.x is the next hop and so on. This makes (for us) very clear to visualize the network... when we refer to '21', everyone knows that is 10.21.x.x and knows physically what board that is.

Of course, the exception to the above is that the interface on '21' that connects back to the previous hop will have to be numbered with the previous machines' net, so in this particular case, the WCPI1 card is 10.2.3.2 since it's connected back to 10.2.3.1 in '2' - but other than that, all the other cards in '21' are numbered 10.21.1.x and 10.21.2.x and 10.21.3.x and so on.

So, we basically use 10.21.2.43 kind of numbering to mean network.machine.card.customer and for us, it's easy to keep straignt in our minds.

Then, since 10.21.1.x and 10.21.2.x and 10.21.3.x and 10.21.4.x and 10.21.5.x are all interfaces on '21', the routing table on '2' (meaning on 10.2.x.x) is pretty simple - we just know that 10.21.0.0/16 need to go to 10.23.3.2 and we don't have to have 5 routes for those 5 interfaces / subnets - one /16 route will handle it. So, for us, static routing is still OK.

At any particular machine along the way, it'll have /16 routes for the 'upstream' machines and a default 0.0.0.0/0 route for everything else. It certainly is getting larger the more we build out, but with 23 machines, the farthest away one is currently 8 hops away, so when we add a new machine, it's only 8 routing tables to edit with one single /16 line added to it - so not too bad.

We are not opposed to using an automatic routing protocal at all, we just started with static so that we would understand what we were doing and so far, it's been simple enough to expand that way.

The only thing we don't like is that we're NATing on the customers's CPE as well, and we hear that double NAT is evil. Plus, most customers then put in a DLink/LinkSys routed and NAT a 3rd time.

[DrLove73]

I've posted similar design logic on other thread (http://forums.star-os.com/showpost.php?p=48381&postcount=17), but with star-like network design.

P.S. I do not think that double or triple NAT-ing an the CPE level is BAD, but is pain in the ass if you have to open ports to them. NAT-ing inside your network is evil. :-D

[valenti]

ninedd, thanks for the additional comments

I think I understand how it works with static routes now. I'm trying to switch things over to that and see what happens, but I seem to have shot myself in the foot tonight. Should have started at the far end, guess I have to do some driving tomorrow.

I was hoping to jump directly to olsr because I have a spot for a ring in my backbone, and I don't think the static routes could account for that. Also, was hoping for two connections to the Internet pretty soon - trying to avoid the concept of upstream/downstream.

RE the NAT: I was thinking of having the customer radio just bridge, have the AP do dhcp for all the devices at the customer, and turn off dhcp on their wifi routers. But I haven't gotten far enough to see if that is a good solution. Solves some problems, but creates others.

[valenti]

I've posted similar design logic on other thread (http://forums.star-os.com/showpost.php?p=48381&postcount=17), but with star-like network design.

P.S. I do not think that double or triple NAT-ing an the CPE level is BAD, but is pain in the ass if you have to open ports to them. NAT-ing inside your network is evil. :-D

Thanks for that design. I did study it for a while, but I was leery of picking the size of the subnets for the backhauls. I expect to use a lot of microcells, and don't really know which way I might expand next. I was hoping for a design that allowed for more of a mesh, so someday 10.4.x.y might link to 10.19.x.y and the routing protocol would handle all the details. (Maybe your design does this too, as I said, I don't understand routing very well)

=======
Oh, you were talking about your DrLove73 name in another thread. When I typed the previous response to you I was in the living room with my wife (both of us on our computers). She wanted to know who I was chatting with, since she heard all the typing. I replied "Dr Love in Serbia" ... things went downhill from there. :)

[DrLove73]

Design I proposed gives you less trouble when starting to route, becouse cuts down on the typing, and is easy to learn and understand. Later on, you can do what ever you want, and enter what ever routes you need.

My network for example, small that it is (9 PoP's), grew without control, and I had to put 22 routes on the main router. Later on I started designing and administrating another network and made preaty big routing chaos. That is when I understood what is to be done next time.

Downhill? I hope it was couse my nick, not the location. We had and still have problem with western propaganda pounding on our dignity couse there own military and strategic goals. When I listen to VoA or CNN about us, I often (no, allways) wonder "who they are talking about?". But that is entirely another matter :).

[valenti]

Well, la de da:
# traceroute 10.11.7.1
traceroute to 10.11.7.1 (10.11.7.1), 64 hops max, 40 byte packets
1 10.2.2.1 (10.2.2.1) 0.487 ms 0.328 ms 0.152 ms
2 10.3.6.1 (10.3.6.1) 2.709 ms 2.546 ms 1.309 ms
3 10.3.1.2 (10.3.1.2) 1.978 ms 1.806 ms 1.584 ms
4 10.11.7.1 (10.11.7.1) 2.591 ms 2.518 ms 2.275 ms
#

I spent several hours yesterday plugging in static routes and pinging until packets were flowing. This should keep me running until I get time to figure out olsr.

Generally my mistake was using the IP address on that machine for the gateway, instead of the next machine over.

[DrLove73]

When you have problem with routing, just log on the directly connected neighbor, and ssh to the troubled one from there.

menu system\system console,
then type shell <ENTER>
then type ssh x.x.x.x <ENTER>
when you are done, type exit 2 times.