OSPF bugs are showstoppers again

[oscarBravo]

As much of an improvement as Quagga 0.98.5 was over the previous Zebra version, it still has bugs. These have become our single biggest administrative headache. Every time we activate changes on an OSPF-enabled router, it's pretty much a certainty that routing will promptly collapse in a heap.

The problem is that after activating changes, OSPF fails to list the route for at least one directly-connected interface. This means that every router and every connected customer downstream of that interface falls off the network. The only workaround is to activate changes on the neighbouring router, which - you guessed it - triggers a similar problem in its OSPF process. This cascades continually, and potentially even in circles because of the routing redundancy we're relying on OSPF to provide. It only seems to stop when we're lucky enough to activate changes and not trigger the problem.

This leads to hours of tearing hair out, and very unhappy customers. It's impossible to administer a StarOS network without occasionally activating changes on a router, so this is a pervasive problem.

Lonnie, I know you're prioritising v3 development, but I have to question that priority. Your existing v2 customers are suffering from a bug that you're refusing to fix. You've said that it's a big deal to do a build with Quagga 0.99.2 instead of 0.98.5 - I don't understand why that should be.

We need this fixed, and there's no way we can afford to wait for an x86 build of v3.

[lonnie]

First off, you do even know if the new release will fix this, and we do not have 2 days to devote a new v2 release for a "maybe" fix. v3 is using the latest quagga and it is coming. We have it on 11 WAR systems at the moment.

So what happened all of a sudden? The reports from you and everyone else were that your networks settled right down and were no trouble. Now you tell me they could never have worked.

[oscarBravo]

First off, you do even know if the new release will fix this We've implemented 0.99.2 on our (Red Hat) network edge machine. It's completely stable, and has addressed bugs we were seeing with 0.98.5.

If you recall, the last time we asked you to implement an updated software version it was because we put a lot of time and effort into testing it in side-by-side comparisons with the older Zebra version. In this case, we're basing our expectation of a fix on close watching of the Quagga developers list and on experience with our network edge router. So what happened all of a sudden? The reports from you and everyone else were that your networks settled right down and were no trouble. Our network grew, that's what. There's a difference between a 12-node OSPF network and a 45-node network with multiple overlapping redundant routes. The bugs we experienced with Zebra are gone. These are new bugs, and they're causing us as much grief as the older ones. Now you tell me they could never have worked. No, Lonnie, I didn't tell you that.

[lonnie]

I have said in another thread this is not a topic for debate. Threats and pleas and not going to change our development schedule.

99.2 is still experimental and could easily break something else and might not even fix the issue.

When we get a lull in the v3 development I will get the new 99.x imported and we'll get a release. Until that happens just be patient.

We've implemented 0.99.2 on our (Red Hat) network edge machine. It's completely stable, and has addressed bugs we were seeing with 0.98.5.

If you recall, the last time we asked you to implement an updated software version it was because we put a lot of time and effort into testing it in side-by-side comparisons with the older Zebra version. In this case, we're basing our expectation of a fix on close watching of the Quagga developers list and on experience with our network edge router. Our network grew, that's what. There's a difference between a 12-node OSPF network and a 45-node network with multiple overlapping redundant routes. The bugs we experienced with Zebra are gone. These are new bugs, and they're causing us as much grief as the older ones. No, Lonnie, I didn't tell you that.

[cal361]

Lonnie,

I notice you are getting more and more agitated with questions like, "When will v3 be out?" and "We have found a problem in OSPF and have tested a solution that appears to work, can you implement it?"

Personally, I am a horrible support guy. I get angry, offended and irritated because my hard work and hours seem to go unnoticed. I, like possibly you, sometimes must focus on finishing one project at a time to feel a sense of completion. Often times, personal and family issues will come up that need attention and I feel really overwhelmed. So I get where you are coming from. Don't think people are unsympathetic.

That being said, I can really understand where these OSPF problem guys are coming from as well. They trusted your software to build their networks. They put their faith and financial well-being in your hands. By doing so they also put the faith and financial well-being of their clients in your hands by proxy. Their customers may have businesses that rely on Internet access to service their customers customers and so on. These guys are under as much pressure as you, and maybe even more depending on the size of their networks and personal issues.

When you say something like, "No bug fixes until a lull in version 3" I don't think you realize how frightening a concept that is for people, like me, who have been with you from the beginning. You have been toutining version 3 since mid-2004. On 09-28-2004 you told us about a production beta. So to say on January 20th, 2006 that someone should wait until v3 is released or a lull in v3 production is incredibly frightening. Especially considering you have told people not to bother asking for time frames anymore.

This is no threat or anger coming your direction, but you maybe causing alot of hurt with the pointed phrasing of your intentions to "not be threatened" for a faster release. No one is doing that. In fact, they have been incredibly supportive, but have reached the end of their rope. They are emotional because they want to keep using your software, but it is getting difficult for them.

Think about it. These are guys runnning 45-node ospf networks and implementing quagga on their own linux boxes to test the fix. They know there is other software that does what they want, they just want to keep using the software made by the person they trust the most. You.

Personally, I stopped buying new licenses a while back because of what I felt were one to many angry replies to people asking about v3. It wasn't even a reply to me that did it. I went as far as to develop my own simple platform for CPE. Recently, due to my wife's health issues I have had to hire new people to do parts of my job. This required that a simple gui be created for my CPE, with no time to code I started buying star-os licenses again. I like star-os. I perfer star-os to my own platform.

All I am trying to say is that you have a rabid customer base that even in the face of knowing there are more functional platforms out there, they want yours. You are respected and trusted. Try not to lose sight of that.

[lonnie]

We are aware of the trust people have given us but we are don't have the time to do everything. We are getting blasted from all sides because everybody thinks their problem is the end of the world until we fix it for them.

In the past we had one image that we constantly updated and improved. When someone came up with a new issue or feature that made sense, we responded immediately and had new code being posted that very day sometimes.

Well, things have changed. We now have many different images that must all be maintained. The new v3 is all new code and thus it is a different envirioment than the v2 branch. When we do a new release it does literally take 2 days to code it, build it, test it and upload it. We cannot do new releases on a whim. We do not have the resources and when we do v2 work, v3 development suffers. You have pointed out that v3 is much delayed. We are very aware of that and we want it as much as everyone else. We want the new features too, for our own WISP.

The Quagga that will "fix" this problem is marked as unstable. We have had very bad experiences with Quagga unstable releases in the past and thus we are very reluctant to devote that two days to something that is not proven to fix anything and that might in fact break something else, with resulting demands for 2 more days of our time to fix that new issue.

It is a no-win situation for us, and I have to choose the actions that I feel are the most productive.

Having said that, we will take a half day and spin a WRAP only release with absolutely ZERO testing other than it boots up and runs with the new Quagga. If it whacks out a remote site and you go bankrupt because of it, too bad. It will be marked "Use at your own risk".

By the way, I am glad you came back.

[bminish]

The Quagga that will "fix" this problem is marked as unstable. We have had very bad experiences with Quagga unstable releases in the past and thus we are very reluctant to devote that two days to something that is not proven to fix anything and that might in fact break something else, with resulting demands for 2 more days of our time to fix that new issue.

The Quagga version in question is the current development build, yes it's marked unstable however I have tested it to the extent that I can. We are also on the Quagga development list and will interact as required with the Quagga developers.
This whole interaction between ourselves and the quagga developers is made much harder because Staros is closed to us and we cannot bring Quagga issues to them unless staros is on the latest version.

To replicate the previous Zebra Bugs we had to construct an entire virtual network of 10 Debain machines in VMware and duplicate the bug in that environment before we could conclusively prove that the issue we were seeing with Zebra could be fixed by moving to Quagga. That took us about 1 man Week of our time. This time would have been considerably reduced if Staros was open-source since the testing could have been done in staros directly.

This time around I cannot do anything to duplicate the curent bug in a VMware environment becase it's only affecting radio interfaces and there is no way for me to model these in VMware. I have however had the 98.2 version running on my network edge router for more than 1 month now with zero issues, Our entire network is routed by this box.

Personally I would feel a lot happier about Staros if we were on an open source platform but paying (the same!) for the proprietary radio drivers. That way we could deal with managing our own bug fixes to a greater extent. It scares the crap out of me that issues will only get dealt with on Valemont's schedule, however good the reasons may be for Valemont to stick to the schedule
I don't suppose valemont might consider this open route for version 2 once version 3 gets released ?
V2 is built on the back of primarily open source software anyway and opening things up would enable valemont to focus on current projects.



Having said that, we will take a half day and spin a WRAP only release with absolutely ZERO testing other than it boots up and runs with the new Quagga. If it whacks out a remote site and you go bankrupt because of it, too bad. It will be marked "Use at your own risk".


Thank you, any idea of a timeframe for this?
As it currently stands we seem to be on 'Use at your own risk anyway so I'll take my chances.
A PC build would be helpful if time allows so that we can stick it in the VMware network for testing.

.Brendan

[cal361]

That's pretty cool of you. I could care less about quagga since I don't use it.

Sorry, in my droning on I think the point was missed. I think it could have been more easily stated in "We like your software. Please don't yell at us.", but I didn't want to say it that simply because I know I would get defensive with a sentence like that directed at me.

I've been in your situtation and totally understand what you are going through. I also react very similarly. The pressure can easily get to be too much and tempers flare, blood boils, all that stuff. I finally had to hire a tech support and sales staff just so I could put a buffer between me and clients. After that my attitude totally changed and I freed myself up to handle what needs to be done.

I would have private messaged all of this to you, but your system won't let me. So please don't misconstrue this as a public rib or anything like that. Hope your weekend goes well sir.

[bminish]

Lonnie
Any update on the new build with the new quagga version yet?
.brendan

[lonnie]

99.3 is out. We are right in the middle of some important code for the driver. We have performance up to VxWorks levels but we now seem to have some compiler bugs. Focus is everything and to tear away and change focus to v2 is a bad thing. Once we get our third alpha test going we'll spin a new v3 release with the latest quagga.

I ask for patience as we try and satisfy the three groups of customers (WAR wanting v3, WRAP wanting v3, and v2 guys wanting updates). And of course there is ourselves, desperate for v3.

[bminish]

I ask for patience as we try and satisfy the three groups of customers (WAR wanting v3, WRAP wanting v3, and v2 guys wanting updates). And of course there is ourselves, desperate for v3.

How long do I have to keep asking my customers for patience Lonnie?

.Brendan

[bminish]

Lonnie any idea of a time-frame for a new version of quagga in V2 please.
This is on going problem for us. Surely part of the 'deal ' when we buy staros licences is that we get some support.
We are running our business on this product and the level of support that is forthcoming on this issue has been appalling to say the least.

Because Staros is closed source I cannot fix things myself. I am utterly dependant on Valemont to do it for me and the fact of the matter is that currently valemont are not fixing things in V2, just making some vague promises about doing so at some stage in the future.

Lonnie, how about opening up V2 and licencing the atheros drivers? that way WE the users could implement our own fixes and valemont would still get paid for the bits that they developed outside of GPL.

.brendan

[Marlow]

A new v2 release definatly is needed, because the issues with star-os currently can be found 2 ways:

- if a hermes/ruby card is in the system, no matter if that is used for ospf or not, it can (not necessary will) stop the system for distributing routes to it's neighbors. Sometimes setting the interfaces to non-broadcast doesn't even solve this.

- on systems with atheros cards only either multiple powercuts, reboots or in some cases systems that have been online for some time, but never run ospf, now are enabled for it, won't redistribute their routes at all. This only is the ospf failing. The rest of the the star-os cf seems working perfectly.

Problem 1 has been discussed earlier and as hermes/ruby cards not even are supported in v3 anymore probably won't be fixed.

Problem 2 however can in most cases has to be solved with reflashing the CF and completly configuring it from scratch by hand again, but that is not really a viable solution.

The problems have been discussed earlier and they have never settled down, but to be honest we've mainly instead of been looking at finding a replacement for star-os because of the issues. We've been for the last months faced with driving out and reflashing the CF's in those cases or simply using static routes injected at the nearest working router until we could get out there. So another v2 release is quite important.

Martin List-Petersen

[lonnie]

Check the beta we released on Sunday night.

Where do you buy your flash disks? What sort of system are you running? My feeling is that if you ever have an image go bad, that flash is destined for the garbage. If simply reuse that flash you are merely waiting for it to happen again.

[tog]

I've had an inordinately large percentage of Lexar compact flash cards die, can't say I've had any problems with anything else but there are many manufacturers and I've only tried around 10 of them. If you're having problems that are solved by re-flashing the card, I'd ask if you have looked in the system log for any messages about I/O errors before you re-flashed the card.

To make things easier and faster for you to replace you could do the following:
Copy your current system key from the running system
Use starutil to download your config from the current running system
Flash a new card (and try a different compact flash card manufacturer)
Put the new flash card in your system
Login to it, put the system key in, use starutil to restore the config and activate changes

[Marlow]

Check the beta we released on Sunday night.

Where do you buy your flash disks? What sort of system are you running? My feeling is that if you ever have an image go bad, that flash is destined for the garbage. If simply reuse that flash you are merely waiting for it to happen again.

The flashes are not the problem. These are usually brand new flashed and can be Bi-Win, Transcend, Sandisk or Canon, think that wraps most of the brands up we have. It's not particularly one brand and we always replace with a fresh cf card. The story is simply that in 1 out of 10 cases problems like these occur, on top of the problems that Brendan describes. And they are unlikely to be down to flash, when everything else works and a change to non-broadcast can make things work, however a real pain in a otherwise Atheros only based system. Also, the problem that Brendan describes especially occours if you are running non-broadcast setup.

Martin List-Petersen

[lonnie]

It cannot be software that is hurting the flash. We do not even have the flash mounted after it has done its job for bootup. I have never had a WRAP board mangle a flash and need recopying, but I have had them go bad. It is a pain, but please, you can't blame the software for a hardware failure.

The Orinoco and Hermes are end of life. Ever since Inersil got out there has been no new firmware development, and thus no driver work. They do what they do and if they cause trouble you have no option but to replace them with an Atheros based system.

[lonnie]

It would appear that the new release has fixed the OSPF issue. No complaints in 3 days, so since this was a show stopper, and we tore ourselves away from v3 so the image should have been immediately used, we can say it must have fixed the issue.

[oscarBravo]

See my response on the beta thread. As long as clients can't connect, there's no way we can usefully test OSPF.

[lonnie]

What kind of clients? What kind of AP?

[oscarBravo]

Tranzeo TR-CPQ15, CM9 radio in a WRAP.

[bminish]

So Lonnie

the long awaited 'fix' came complete with another bug that you already knew about (and that had been publicly reported to you) from an older beta that you haven't rolled back or fixed.

this means that we still have no useful solution to the OSPF issue. What are you going to do about it for us Lonnie ?

We ARE your paying customers, WE are in danger of loosing OUR paying customers if this 'level of support' continues for much longer.

can we have a 99.3 Quagga built on the last WORKING version 2 release 2.10.0. 4693 , not the known NON working b2 version (2.10.1b2 build 4714)

we are trying to get a fix to a problem here not introduce new ones. this 'fixed' version may indeed resolve the quagga issue but since I can't connect my clients AT ALL using this version I simply cannot use it

Lonnie this is a dreadful mess, what do you propose I do?
.brendan

[lonnie]

We'll look into it again, but the issue was one of the preamble having to be set for certain client systems. We use the release as-is and it works for all we have tested with, which is StarOS clients using Orinoco, prism2, prism2.5 and Atheros. We also have a few discrete cards in customer machines that are Orinoco or prism2.

It sounds like this issue hit you right away, so why did it take so long for you to do the test? We seriously took time away from v3 to build this because it was very serious for you and yet you waited nearly 3 days to get around to testing it.

When you had it running did you send me an IP and password so I could take a look? I did not receive it, so how am I supposed to "just fix it". The trouble is not visible on our network with our equipment choices, so it up to you to get me access so I can at least see what the problem is on your network.

[bminish]

It sounds like this issue hit you right away, so why did it take so long for you to do the test? We seriously took time away from v3 to build this because it was very serious for you and yet you waited nearly 3 days to get around to testing it.


No, not on the live net. we wanted to test it first in our vmware environment before deploying it more widely on a live network with customers connected to it, Of course there are no radios on the Vmware net :-( and it looked ok in the VM world


When you had it running did you send me an IP and password so I could take a look? I did not receive it, so how am I supposed to "just fix it".

We can probably arrange this but as soon as it was deployed all hell broke loose and we had to roll back.
We didn't deploy Beta2 in the wild because of the already reported bug that caused problems with the client kit we are using.
Did you not look into that bug at the time it was reported?

.brendan

[lonnie]

It is not a bug from what we can tell. It works as it is supposed to and if you have the wrong preamble you are dead in the water, which sounds like what happened to you.

# @@ AP or Client non-security related options @@
# set preamble (long or shor/long)
# sets the card's preamble to short/long, or long.
set preamble long

This is likely what you required for the Atheros and the units that did not pass any traffic.

[oscarBravo]

Given that it works in 2.10.0, does that mean 2.10.0 uses long preamble only?

[lonnie]

It should be short/long. That is the default since it gives the best performance. With the addition of the preamble option it might have changed the default. Try it and see. I really don't know what your cards are expecting and this release was NOT tested at all, as it was requested.

[bminish]

I really don't know what your cards are expecting and this release was NOT tested at all, as it was requested.

yes but we did expect it to be built on a working version, not a Beta that had unresolved bug reports against it.


.brendan

[David L. Vrablic]

yes but we did expect it to be built on a working version, not a Beta that had unresolved bug reports against it.


.brendan

Brendan,
You have posted several times now about "a Beta that had unresolved bug reports against it."
I haven't seen any message regarding a bug of any kind that stops associations or impedes throughput.
I believe the preamble problem exhibited itself in that manner.
As code does not change by itself problems usually come about when one goes in and makes changes to specific modules of code. It is very easy to "Mess up existing defaults" (which might be the case here.)
They told us this was a rush job with no testing and use it at your own risk.
This was just another time they dropped everything to try to help someone in trouble out.
It is obvious that you are very distressed by this.
Would you please share the posting with us so we can all work in unison to get this issue resolved?
As you shurly know it might be a special set of implementation of features that cause a problem. Some that I may not even use.
Input is what the authors need at this time, not finger pointing and criticism.
Work with us here my friend, we all want our needs met. :-)

[lonnie]

For the LAST time, it was NOT an unresolved bug. It was a configuration setting that was not set correctly. The beta notes described how to set it up, but you must not have read them.

I was asked, pleaded, and threatened to get that release out "without any testing whatsoever", so we did it.

Normally, as I explained a release takes us TWO days to build, test, and upload. That was not possible because of the v3 work in progress.
Have you figured out how to set the preamble? Are you going to try the beta we prepared? There is nothing wrong with the driver, because, and I know you'll hate this, we use that release here. It is solid.

yes but we did expect it to be built on a working version, not a Beta that had unresolved bug reports against it.


.brendan

[oscarBravo]

Testing progress report: I've rolled out the new beta on several backbone routers. (I'm avoiding AP routers for the moment until I've sorted out any preamble issues.) Seems pretty stable so far - I'll roll it out further and continue to test.

btw: congrats on the release of v3 - big milestone. Don't forget to have a few drinks!

[bminish]

For the LAST time, it was NOT an unresolved bug. It was a configuration setting that was not set correctly. The beta notes described how to set it up, but you must not have read them.

the beta notes for beta2 make mention of how to change from short/long (default) to long
I was not aware that everything was by default long until beta2, is this what you are saying here?

in addition the bottom postings on this thread
http://forums.star-os.com/showthread.php?t=4560
match the problems that we had when we tried the new beta on client facing AP's unfortunately with client facing IP's it's hard to leave things in a non-working config for long (the phone starts to ring..)
I'll get a test node and a test CPE up for you if that's what is required


I was asked, pleaded, and threatened to get that release out "without any testing whatsoever", so we did it.


I had hoped that you might have built it against the working release version instead of the B2 that still appeared to have issues against it.
Threatened is a bit heavy, all we are looking for is support on what is the current release version of your product.
Also bear in mind please that it was us who worked out how to get OSPF working on your product in the first place.



That was not possible because of the v3 work in progress.

Congratulations on V3 beta release. Any idea of a time-frame for x86 betas?

Have you figured out how to set the preamble? Are you going to try the beta we prepared? There is nothing wrong with the driver, because, and I know you'll hate this, we use that release here. It is solid.

So did the default preamble setting change from that last release version to the beta with the preamble option?
Did the other testers of beta2 get their CPE issues resolved? the thread would indicate not
We are testing this beta, carefully and will report back what we find. You should already know that when we report back to you it will be documented and as accurate as we can make it within the confines of a closed OS.

.brendan

[David L. Vrablic]

Brendan,
Your post:
"I had hoped that you might have built it against the working release version instead of the B2 that still appeared to have issues against it."

I am using b2 on many of our boxes and I want to know what issues you keep referring to.
If it involves OSPF it doesn't effect us, But if there is something else I would sure like to know about it.
If this exchange is just about OSPF I will stop reading what you have to say.
But please remember that a search for bugs shows everything.

[bminish]

Brendan,
Your post:
"I had hoped that you might have built it against the working release version instead of the B2 that still appeared to have issues against it."

Just follow the B2 thread. Issues with certain clients (which we happen to be using almost exclusively) not being able to pass traffic.
follow this thread
http://forums.star-os.com/showthread.php?t=4560


I am using b2 on many of our boxes and I want to know what issues you keep referring to.
If it involves OSPF it doesn't effect us, But if there is something else I would sure like to know about it.
If this exchange is just about OSPF I will stop reading what you have to say.
But please remember that a search for bugs shows everything.

The B5 is just to address issues with OSPF.
Does it do this? We don't know yet as it's proving difficult to roll this out because of the clients issue. We are slowly putting it out on all the backbone and we will look in more detail at the client issue but it would be really helpful if we had some feedback from Valemont about exactly what changed with the radio drivers in B2 and above.
Did the default preamble settings change?
Did the other users that reported issues with B2 find a solution to their issues ?
The other issue for us right now is time, we are in the middle of a period of good weather and the middle of a big network expansion. It's harder for us to test stuff now that the AP's have paying customers connected instead of a few 'crash test dummies' This is one of the reasons we were pushing so hard for this quagga update sooner.

In summary this beta is about OSPF if you are not using OSPF then you should probably stick with what you are using as those versions have more testing at this stage.

.brendan

[lonnie]

In summary, this b5 was an extension of b2. Nobody generates unique betas. They are all in a chain leading to an eventual release.

To repeat myself, there is no bug in bv2 that causes certain clients to drop. It is likely an issue with preamble and I explained how to set preamble, so just go and do it. Who cares what the setting WAS if you can set it any which way you please? It might have changed while testing, I have no idea and really, it does not matter.

The other user was not able to get it working, but then it did not work before b2 either.

[bminish]

I have rolled out beta5 on all our backbone but I have not done client facing APs. I will look at those over the next few days and report back on or findings with CPE.

The problem with OSPF loosing wireless interfaces on occasion persists. On the upside convergence times are much better and 99.3 seems at as stable as 98.5 in other respects.

Since I cannot duplicate this missing interface problem in a Staros or Debian test environment under Vmware and it only seems to affect radio interfaces there is little more debugging I can do on this issue, nor is there the makings of a bug report I can bring to the quagga developers.

Is there a possibility that there is something slightly broken with the way that the Atheros driver notifies the kernel about the current link state ?
This bug remains a showstopper for us.
Valemont, What is our next step towards resolving this issue?
We remain willing to work with you towards resolving this.

.brendan

[lonnie]

I guess it will have to await v3 for any further fixes. The WAR boards have the new driver and new quagga if you are interested. They'll also give you better speeds.

[bminish]

I guess it will have to await v3 for any further fixes. The WAR boards have the new driver and new quagga if you are interested. They'll also give you better speeds.

Lonnie,
Since we have not successfully fixed it or even identified with certainty where exactly in the OS the problem lies in V2 how can you possibly tell me with certainty that it's fixed in V3.
I'll have to spend a ball of my money to upgrade all my routers to V3 (when it becomes available for x86) and there's still no guarantees that this issue won't remain in V3.
In effect I get to beta test more software and pay handsomely for the privilege.

We need a working OSPF. Version 2 is still the curent release version, It's sold as having a working OSPF. Now how about continuing to work with us to resolve the issue?

.Brendan

[lonnie]

Of course I cannot tell you that with any certainty, but I am not about to go searching with a fine tooth comb in a driver that is end of life. What am I supposed to be looking for? You can't tell me that this is even an Atheros problem. I can't duplicate your system and therefore I can't see the problem.

The new v3 is a new driver, new kernel, new framework, pretty much new everything. I am hopeful it fixes your problem. If it does not then perhaps it means it is a quagga problem and we'll have to await their fixes. You cannot rule out your configuration either.

For instance, Quagga RIP just hums along until something derails it. If you have manual static routes in some parts of the system those routes can throw everything off for an hour or more until things synchronize again. If the manual statics are not there things come together pretty swiftly upon restoration of the down unit. Perhaps Quagga OSPF has the same sort of issues. Do you have any manual static routes?

[bminish]

Of course I cannot tell you that with any certainty, but I am not about to go searching with a fine tooth comb in a driver that is end of life. What am I supposed to be looking for? You can't tell me that this is even an Atheros problem. I can't duplicate your system and therefore I can't see the problem.


Everyone who is running largish ospf networks using staros is seeing this problem, Lonnie.
(guys could we have some me toos here please !)

Staros is closed source to me, this prevents me from doing any more debugging on this issue. With the earlier Zebra Bugs I was able to duplicate them in a non-staros test environment.
I can't do this with the current bug, This is why we need help from you guys, not to be told 'tough shit' spend a pile more money with us at some unspecified date in the future to upgrade 50 or so nodes and hopefully the problem will go away.


The new v3 is a new driver, new kernel, new framework, pretty much new everything. I am hopeful it fixes your problem. If it does not then perhaps it means it is a quagga problem and we'll have to await their fixes.


Without your HELP we can't even confirm or rule out a Quagga problem. If we Could we would interact with the Quagga developers ourselves to get it resolved. At least them we would know what the issue is and have more confidence that it's not lurking in V3 as well.


For instance, Quagga RIP just hums along until something derails it. If you have manual static routes in some parts of the system those routes can throw everything off for an hour or more until things synchronize again. If the manual statics are not there things come together pretty swiftly upon restoration of the down unit. Perhaps Quagga OSPF has the same sort of issues. Do you have any manual static routes?

Yes I now have some static routes and I can confirm with absolute certainty that static routes do not derail OSPF. I now have some static routes in place because I have clients I will loose if this shit keeps up much longer. Static and OSPF routes can coexist and it's within the spec that they can do so.

V2 may indeed be reaching the end of it's life, however Valemont do not currently sell V3 for x86 nor are you providing a date for a stable x86 release.
V2 is still your only currently available retail product for x86. Why is it so unreasonable to expect support for bugs, I an not asking for features, just bug fixes.

We did a lot of work for you guys working out how to get OSPF working. we also diagnosed several Zebra bugs and identified that Quagga was a big improvement over Zebra, We are still willing to work with you. Is this how you replay us for working with you when we need your help?

Franky I am disgusted with valemont's behaviour on this issue.

.Brendan

[aldo]

I guess it will have to await v3 for any further fixes. The WAR boards have the new driver and new quagga if you are interested. They'll also give you better speeds.

i completley disagree with this lonnie. you still sell licences for v2. this product is sold with ABC features. your last post almost accepts that there might be problems with your product, but you are still happy to sell it as a release with all the featuers of which some don't work.

we purchused staros for testing for the following features.
VLAN(no mention anywhere but it does not support large frames 1024 is the largest.
OSPF which works inconsistanly with atheos cards.
BRIDGING SUPPORT which has show to not support large frame sizes as well and is less than stable.
WDS which it says out your website is in the V2 product but there is only VDS.

this just plain and simply sucks. Any your solution is to pay more money for another product that you haven't finished releasing, when we have got no value out of the exsisting product.

there have been great detailed reports in your forums about these problems and it is difficut to diagnose them effectively because your source is closed. yet you still want them to clarify the issue for you.

GO FISH MAN.
I think that consumer guareentees needs to here a little about your proactices.

[ninedd]

Staros is closed source to me, this prevents me from doing any more debugging on this issue.Forgive me for asking, but you keep mentioning the closed souce thing - why not just use an open source product and be done with StarOS then?

I'm guessing the reason the rest of us use StarOS is that it 'protects' us from having to assemble the pieces ourselves, plus they have great wireless drivers and good script parsers and a nicely integrated (text) GIU and so on.

However, some people then complain that the whole Squid config isn't there or that their choice of some library isn't what you would have compiled in. I guess my question is for the guy's that don't want to be 'protected', I'm confused as to why you'd use StarOS in the first place?

[simcor23]

No doubt.

[lonnie]

I did not make any such concession. I provided a b5 that the customer does not wish to try since the first attempt did not work due to not knowing how to use preamble. They are reluctant to try again, even though I have explained what they must do. That is not my fault. I am saying that with a new driver, new kernel and new quagga, the customer has a reasonable chance that the problem is fixed. I also said that if it is not then we have pretty much eliminated everything but quagga.

WDS is available for prism and Orinoco. Atheros has never had WDS as a feature.

VLAN must subtract the VLAN tag from the packet size. Not very many systems can handle jumbo frames and we have never claimed to be able to do that.

Where did you get the idea that bridging has any packet size limits? Surely not from using it. Standard rukes apply and it fragments if you exceed 1472.

I am not sure where you got the idea that OSPF is inconsistent with Atheros cards. We have not said that, and it is only speculation by someone who could easily have other problems, or it could be OSPF in general.

I guess that is why we provide demo mode, so you can see if it meets your needs.

StarOS is partially closed source due to the drivers we use and it will remain that way. If you want fully open source, there is no shortage of distributions that will satisfy you. The Atheros driver from madwifi is getting better all the time. Make the move and you can fix any problems as they arise and you do not even have to wait for us, and there are no license fees.

Imagine, getting all those benefits of fully open source and no fees. Does it get any better than that? Why would anybody continue to use us?

i completley disagree with this lonnie. you still sell licences for v2. this product is sold with ABC features. your last post almost accepts that there might be problems with your product, but you are still happy to sell it as a release with all the featuers of which some don't work.

we purchused staros for testing for the following features.
VLAN(no mention anywhere but it does not support large frames 1024 is the largest.
OSPF which works inconsistanly with atheos cards.
BRIDGING SUPPORT which has show to not support large frame sizes as well and is less than stable.
WDS which it says out your website is in the V2 product but there is only VDS.

this just plain and simply sucks. Any your solution is to pay more money for another product that you haven't finished releasing, when we have got no value out of the exsisting product.

there have been great detailed reports in your forums about these problems and it is difficut to diagnose them effectively because your source is closed. yet you still want them to clarify the issue for you.

GO FISH MAN.
I think that consumer guareentees needs to here a little about your proactices.

[tog]

Imagine, getting all those benefits of fully open source and no fees. Does it get any better than that? Why would anybody continue to use us?

Because even your v2 atheros driver sucks less than madwifi's and Sam's ath do right now. (This is actually praise in the world of software.)

[lonnie]

RIP and static routes are also supposed to be part of the spec, but I can assure you they cause issues if they are not right at the edge. If you have any statics in the middle it really messes things up on any sort of network change. Spec or no spec.

What is the status of b5 with quagga 99.3? Have you even tested it since I explained how to adjust the preamble?

I appreciate your effort, but I do not appreciate your demands that we drop everything and attend to your problem. The only thing we can do is install the latest quagga and let you try it. If that does not fix the issue, then it is back to waiting for the next release of quagga. We did that but you want us to magically take over quagga and fix it too. Sorry, not in the plans. We do drivers and user interfaces and general framework.

I cannot duplicate your LAN and I am not an OSPF expert --> and I never claimed to be one. In fact I have always said we include OSPF because it is part of a package, and that we have no great knowledge of it.

As for our behaviour, we have addressed your concerns. Not as quickly as you wanted, but quicker than some others react. We are faced with limited resources and a large requirement to finish v3. It is coming rather nicely and from all reports it is stable and fast. We have a few more features to add, then we port to x86.

This all started as a request that we try 99.2 because you had some sites that gave trouble when they lost power. My response to fix the power met with an ever increasing set of issues. Now it seems it does not even work and it is very serious stuff. I provided you with a new beta and you did not even bother to check the beta trail on the Forums. You put it in and a b2 change bites you. I have explained what you have to do for preamble in b5, but you have not even tried it. You demand that I provide a pre-beta with quagga so you can simply pop it into your system. Sorry, but b2 changed the release and it no longer exists pre b2. b2 was the code to build upon and add the new quagga, and that became b5. The next beta will build upon b5 and eventually we will have a release of the next version.

If you are serious about working with us, why not start by simply trying b5 again? You now know what to do to tweak for those client bridges you use.

If you think you are better off with a fully open source package, go for it. There are many packages available and the madwifi driver is getting better all the time.

Until you use b5 I will not be interested in having any further discussion about this. I'm not sure what you expect to happen if you won't even use the fixes we provide.

Everyone who is running largish ospf networks using staros is seeing this problem, Lonnie.
(guys could we have some me toos here please !)

Staros is closed source to me, this prevents me from doing any more debugging on this issue. With the earlier Zebra Bugs I was able to duplicate them in a non-staros test environment.
I can't do this with the current bug, This is why we need help from you guys, not to be told 'tough shit' spend a pile more money with us at some unspecified date in the future to upgrade 50 or so nodes and hopefully the problem will go away.



Without your HELP we can't even confirm or rule out a Quagga problem. If we Could we would interact with the Quagga developers ourselves to get it resolved. At least them we would know what the issue is and have more confidence that it's not lurking in V3 as well.



Yes I now have some static routes and I can confirm with absolute certainty that static routes do not derail OSPF. I now have some static routes in place because I have clients I will loose if this shit keeps up much longer. Static and OSPF routes can coexist and it's within the spec that they can do so.

V2 may indeed be reaching the end of it's life, however Valemont do not currently sell V3 for x86 nor are you providing a date for a stable x86 release.
V2 is still your only currently available retail product for x86. Why is it so unreasonable to expect support for bugs, I an not asking for features, just bug fixes.

We did a lot of work for you guys working out how to get OSPF working. we also diagnosed several Zebra bugs and identified that Quagga was a big improvement over Zebra, We are still willing to work with you. Is this how you replay us for working with you when we need your help?

Franky I am disgusted with valemont's behaviour on this issue.

.Brendan

[lonnie]

The reports are that the v3 driver sucks even less again, which in software is another leap forward. It has been a rough road to get here, but we made it. A few features are left to implement, but the pace in the last week was pretty cool, and we added a lot of things that had been on the todo list.

Because even your v2 atheros driver sucks less than madwifi's and Sam's ath do right now. (This is actually praise in the world of software.)

[tog]

Yes, good job so far, although I haven't been able to test it in the field yet.

[aldo]

[QUOTE=Inet2000]Forgive me for asking, but you keep mentioning the closed souce thing - why not just use an open source product and be done with StarOS then? QUOTE]

for us anyway the closed source open souce debate is only to do with lonnie continuing to ask for debugging. this seems less than possible with staros without a shell for clear direction on where to start.

[bminish]

I did not make any such concession. I provided a b5 that the customer does not wish to try since the first attempt did not work due to
not knowing how to use preamble.

Lonnie, as I have already explained I have at this stage put B5 on almost all my network, it's on about 35 nodes at this stage. I didn't rush into it as soon as it was released as this is a live net with paying customers connected to it and initial tests threw up a problem with CPE facing AP's that use Atheros.


They are reluctant to try again, even though I have explained what they must do. That is not my fault.

WTF are you saying here Lonnie? That we didn't test it?


I am saying that with a new driver, new kernel and new quagga, the customer has a reasonable chance that the problem is fixed. I also said that if it is not then we have pretty much eliminated everything but quagga.


I have been running Quagga 99.3 on my core router since it came out.
I cannot duplicate this issue in a vmware test network with staros or Debian. This vmware virtual net duplicates exactly our network topology (down to IP addressing and interconnectivity)
This bug ONLY happens in conjunction with staros AND Atheros radio links, it also only affects atheros interfaces. it's by no means certain that it's a 'quagga issue'
Since staros is Closed it's impossible for me to do anything more to do further diagnose the problem unless Valemont are willing to work further with me, It is now pretty clear that Valemont are unwilling to work further with me.
Thanks a bunch, I am up shit creek here


StarOS is partially closed source due to the drivers we use and it will remain that way.

partially??
Howe about sharing your configuration files, even?
VDS uses VTUN how have you configured VTUN?

Come on Lonnie, you are sailing about as close to the wind as you can get away with when it come to the GPL, You have built a commercial product that's as closed as you can make it on the back of the GPL.

If you were serious about this, how about opening your software and just licencing the atheros support. I would happily pay the same as I currently have to pay for a truly open product with licenced radio drivers, your atheros drivers are excellent.
In this licencing scenario with the current OSPF issue I would now be at one of three places.

1/ back to the Quagga developers with a bug report and possibly a patch to fix it.
2/ Back to Valemont with a full and accurate bug report identifying where the Atheros driver breaks Quagga.
3/ Have identified and fixed our problem and reporting back how to fix it.

Since Staros is completely closed we are stuck where we are trying to get Valemont to provide meaningful support on their current retail product.

.Brendan

[lonnie]

Brendan, did you put the new image on the Atheros systems and use preamble to solve the client connect issues? I suspect that the new release has to be on 100% of the systems to have any sort of fix for network wide issues.

Even one bad node can cause network issues.

As for the issues with losing kernel link state in the Atheros, there is notan issue, at least not one we have seen. The radio being activated properly tears down the network and rebuilds it. We have the quagga watchdog in case it quits, but I doubt that it has ever really been called upon.

If you want to work this out you'll have to quit blasting us and switch to work it out mode. As long as you keep making accusations and trying to assess blame I am in defense mode, and you want me in help you out mode.

Get your network 100% using 99.3 and then let me know what you find. If 99.4 comes out soon we'll add it in right away for further testing

[lonnie]

Aldo, are you associated with bminish on this project?

[QUOTE=Inet2000]Forgive me for asking, but you keep mentioning the closed souce thing - why not just use an open source product and be done with StarOS then? QUOTE]

for us anyway the closed source open souce debate is only to do with lonnie continuing to ask for debugging. this seems less than possible with staros without a shell for clear direction on where to start.

[bminish]

Brendan, did you put the new image on the Atheros systems and use preamble to solve the client connect issues? I suspect that the new release has to be on 100% of the systems to have any sort of fix for network wide issues.

Just finished testing the b5 release on APs with atheros, Specifically the preamble options.
My clients can't pass data with either preamble setting with the AP set to B or G mode so it's not possible for us to use b5 on AP's with connected clients.


Even one bad node can cause network issues.


but not more than one hop away unless there is something seriously broken in the config.


As for the issues with losing kernel link state in the Atheros, there is notan issue, at least not one we have seen. The radio being activated properly tears down the network and rebuilds it. We have the quagga watchdog in case it quits, but I doubt that it has ever really been called upon.

good to know about the radio driver.
I have however seen zebdog messages in the system logs on occasion, presumably this is the zebra watchdog. Unfortunately due to the routing failing at that point remote syslogs don't show zebdog events


If you want to work this out you'll have to quit blasting us and switch to work it out mode. As long as you keep making accusations and trying to assess blame I am in defense mode, and you want me in help you out mode.

I have always been more than willing to work with you to resolve this issue, it's just that you keep coming out and telling us to wait for ver3.


Get your network 100% using 99.3 and then let me know what you find. If 99.4 comes out soon we'll add it in right away for further testing
I can't get my network 100% running b5 due to the clients issue. How can we best assist you in resolving that issue?

.brendan

PS aldo is unknown to me outside of this forum and as not associated with our company.

[lonnie]

Brendan,
Do you have any WAR boards there? I'd be curious to know if the new driver talks with your clients. That would be a huge plus for the new driver. Skaught --> can you test a WAR with the new 1.06 beta to see if it talks where the v2 driver does not?

[bminish]

Brendan,
Do you have any WAR boards there? I'd be curious to know if the new driver talks with your clients. That would be a huge plus for the new driver. Skaught --> can you test a WAR with the new 1.06 beta to see if it talks where the v2 driver does not?

Hi Lonnie,
No warboards here yet, sorry.
.brendan

[aldo]

Aldo, are you associated with bminish on this project?

[QUOTE=aldo]

Not at all assciated I think I meantioned we use BSD and brenden seems to use redhat. no relationship. The issue with ospf does not seem to apply to many as not many seem to us it.

[bminish]

Not at all assciated I think I meantioned we use BSD and brenden seems to use redhat.

Well a mixture of stuff actually, Centos, gento, fedora, debian and a little bit of BSD. Each has it's place
.brendan

[bminish]

What do you want us to do next Lonnie?

We have 2 issues here, the OSPF issue is the more serious one for us, OSLR in V3 is not ever going to be a good replacement for OSPF for us.

The other issue is the 'not talking to clients' bug in b2 and b5 i am certan that the presence of a few nodes running the older version of Quagga is not the root cause of our issue but the 'not talking to clients' problem will also need to be worked out


.brendan

[ninedd]

the 'not talking to clients' bug in b2 and b5Hey Brendan, what types of clients are these? I've read back through the treads and I don't see it mentioned - I probably just missed it. Perhaps some of us could try help track down what that is if we have the same clients and can try to duplicate the problem?

[bminish]

Hey Brendan, what types of clients are these? I've read back through the treads and I don't see it mentioned - I probably just missed it. Perhaps some of us could try help track down what that is if we have the same clients and can try to duplicate the problem?

Tranzeo TR-CPQ, these are atheros based and worked just fine up untill beta2. We have several hundred of these.

.brendan

[tony]

Thank you brendan.

[Skaught]

What is the current status of OSPF on V2? Is it running stable for everyone? My static routing tables are getting silly(over 100 routes) and I really need to get OSPF running.

[bminish]

What is the current status of OSPF on V2? Is it running stable for everyone? My static routing tables are getting silly(over 100 routes) and I really need to get OSPF running.

No it's not stable, it's broken in that on occasion OSPF will just loose an attached wireless interface from time to time. The only way to get things running again is to apply changes on the neighbouring node, this may in turn break OSPF on that node etc. Stick with your static routes if you can for now.

We need OSPF because we have engineered in quite a lot of redundancy into our backbone (over 55 nodes and growing) but yet again we are waiting for Lonnie to come back and work some more with us

.brendan

[lonnie]

We are waiting for the next quagga release.

No it's not stable, it's broken in that on occasion OSPF will just loose an attached wireless interface from time to time. The only way to get things running again is to apply changes on the neighbouring node, this may in turn break OSPF on that node etc. Stick with your static routes if you can for now.

We need OSPF because we have engineered in quite a lot of redundancy into our backbone (over 55 nodes and growing) but yet again we are waiting for Lonnie to come back and work some more with us

.brendan

[lonnie]

By the way, did you try the NEW beta for v2? It has some tweaks to the preamble code and a few others. It was released yesterday and was posted on the Forums in the Beta section.

Even though v2 is not supported (according to what some people think) we seem to be releasing fixes. Curious, eh?

[bminish]

We are waiting for the next quagga release.


Rumour has it that Mikrotik has a working OSPF. It appears that they use Quagga too.
see here for GPl list http://www.mcda.org/help/license.html
In addition to this I have never seen this issue outside of staros (we are running quagga on our linux boxes too)

If you really think it's a quagga issue how about we do some diagnosing so that we can submit a bug report to the quagga developers. A reported bug has a much better probability of getting fixed.

On the new version yes we will roll that out and also test it with our client kit. Do you still have to explicitly set the preamble in this version

.brendan

[lonnie]

You will want to set preamble to work with the client mix you have. Since we have no way of knowing that, I can't say for sure whether you will have to or not.

You are free to diagnose and submit any bugs you want. Why would you ask my permission for that? You have a working knowledge of OSPF and you can report the symptoms and describe what is happening.

[bminish]

You are free to diagnose and submit any bugs you want. Why would you ask my permission for that? You have a working knowledge of OSPF and you can report the symptoms and describe what is happening.

The previous OSPF issues we were able to duplicate in a non-staros environment. This time around I have been unable to duplicate this bug on an open platform.

Since I cannot recreate the bug outside of staros and staros is closed to me I can do little more to debug this issue without your help.

Interesting that the 'other guys' have a working OSPF using quagga over atheros links. Surely this implies that this issue may not be a quagga issue but perhaps a radio driver, kernel config issue or the like?

.brendan

[lonnie]

Just try the release we provided. When all units have been upgraded then please see how it works. Until all old versions of Quagga are gone you cannot say anything about anything.

Will you be able to let us log in when you get all of the upgrades done?

[bminish]

Just try the release we provided. When all units have been upgraded then please see how it works. Until all old versions of Quagga are gone you cannot say anything about anything.

That's simply not true, Quagga only cares about neighbours so any version related issues will not propagate more than one hop. it's certainly possible to poison the routing across a larger number of hops but this is quite different from quagga loosing awareness of directly connected interfaces

Over the next few days we will roll out the new version, providing it works with our CPE units we will roll it out on all nodes, generally however we do not run quagga on the AP facing nodes, just the backbone.


Will you be able to let us log in when you get all of the upgrades done?
we can probably arrange something if this is required but bear in mind that this is live net with paying customers so any quagga related issues which might be interesting to diagnose are also broken links that I need to return to a working state as quickly as possible.
Can starutil or similar be used to grab relevant debugging info from nodes?
Another issue with logging into the net is that you would need to shell node to node since the quagga bug you are looking for also of course breaks routing. For this you would need an IP map of our net.

This issue should be pretty easy to diagnose in the lab. take a bunch of v2 boxes on wrap with atheros cards in them, configure OSPF, wait till it settles, then go around applying changes, you will soon find that quagga will loose interfaces from time to time
.brendan

[lonnie]

But I thought you said it was the neighbours that lost the unit that had been activated. That would seem to be the quagga running on the neighbour.

[bminish]

But I thought you said it was the neighbours that lost the unit that had been activated. That would seem to be the quagga running on the neighbour.

No the unit activated looses an interface including any awareness of it's own local IP's on that interface.
Applying changes on the neighbour will usually bring it back but may in turn cause the same problem on that node.
Applying changes does not clear the quagga database and this may have some bearing as to why applying changes on the affected node does not help on many occasions whereas applying changes on the neighbour will usually do the trick.
If this was version dependant like previous bugs with OSPF I would expect to see problems mostly on nodes closer to older versions and this is simply not the case

.brendan

[lonnie]

Sorry but I just do not understand. You say that an activate breaks it to the point it does not even know its own IP and that another activate will sometimes fix it but may cause issues on other nodes, at which point you have to activate machines on that node.

You also say "I would expect to see problems mostly on nodes closer to older versions and this is simply not the case". This tells me you are running a mix of versions and I would really like to see you bring the whole system to 99.3. It is just not productive to even troubleshoot with older buggy releases still in the system. Why waste time chasing issues that might not even be a factor in 99.3? Get rid of them and let's test with 99.3.

[bminish]

Sorry but I just do not understand. You say that an activate breaks it to the point it does not even know its own IP and that another activate will sometimes fix it but may cause issues on other nodes, at which point you have to activate machines on that node.

OSPF looses all awareness of the interface and any associated IP's assigned to those interfaces.


You also say "I would expect to see problems mostly on nodes closer to older versions and this is simply not the case". This tells me you are running a mix of versions and I would really like to see you bring the whole system to 99.3. It is just not productive to even troubleshoot with older buggy releases still in the system. Why waste time chasing issues that might not even be a factor in 99.3? Get rid of them and let's test with 99.3.

Lonnie, I will bring my remaining 1 or 2 nodes that are running quagga 98.5 to 99.3 as soon as I have had a chance to test if the new version of staros will work with My deployed CPE units.
Most of our AP nodes do not run any routing protocols since vtund* (VDS) is used to drop public IP addresses directly to the CPEs.

I WANT to run 99.3 everywhere it reconverges faster.
In in any case 98.5 works just fine (even in mixed 98.5 / 99.3 environments ) in my test setup.
The Quagga developers call 98.5 the stable release. 99.3 is a beta
describing 98.5 as a buggy version is not correct

As I have said earlier, OSPF only cares about it's neighbours. If an node and all it's neighbours are running the same version but that node goes on to loose awareness of it's own directly connected interface(s) (which is something that should never happen IF that interface is up) than that effectively rules out issues due to differing versions. I have to date been unable to duplicate this behaviour outside of staros.

quagga (OSPF) has an awareness of interface state (up or Down ) it seems that this bug may be directly related to detection of link state on wireless (atheros only?) links. this would imply that the bug is in one of 3 general areas.
1/ Quagga, broken in some way that only gets seen with wireless links
2/ Atheros driver
3/ Kernel configuration (it's involved in link state tracking of course)

I have access to 1/ and am simply unable to replicate this issue outside of staros.
What does your kernel config look like?


*BTW vtund is a GPL'ed application. How have you configured it? I need to be able to connect directly to my Linux router. It's more than just inconvenient to have to break OSPF and all the tunnels by applying changes on my tunnel end point box every time I need to add an extra tunnel

.brendan

[oscarBravo]

This issue might need some clarification.

Under normal circumstances, OSPF is always aware of networks to which it is directly connected (and for which it has a "network" statement - take that as a given). In other words, let's say I execute "show ip ospf interface", and see that OSPF is running on eth0, wpci0 and wpci1, then when I execute "show ip ospf route" I'd expect to see (among others) the networks attached to these interfaces listed as "directly attached to eth0" (or wpci0, or whatever).

When this problem manifests, "show ip ospf interface" shows all the interfaces running, but "show ip ospf route" is missing one or more of the directly attached networks. Because each router is responsible for propagating its directly attached network routes, this means that the attached network disappears from the rest of the network. It also means that all networks beyond that network disappear also.

Activating changes can fix this problem, but it can also trigger the problem on connected routers, leading to a cascade of disconnected networks.

[lonnie]

We can take a look at the up/down states of the driver. In general the 2.4 kernel does not deal with those issues so this is something that OSPF should not be monitoring. We can also take a look at the timing of when we start OSPF. It should be coming up after all drivers as loaded and controlling cards. A long association time might cause it to miss a device so we will check into that.

We are not disclosing any information about vtun since we will be moving away from that code base and we simply cannot have a situation where we would be forced to keep the previous mechanism due to people using other platforms.

I can assure you vtun is totally unmodified and that is all I will say. I consider any other information to be proprietary. If someone does figure it out they have to be advised that we are not obligated to continue support for them and we WILL change the whole VDS system when it suits us.

OSPF looses all awareness of the interface and any associated IP's assigned to those interfaces.



Lonnie, I will bring my remaining 1 or 2 nodes that are running quagga 98.5 to 99.3 as soon as I have had a chance to test if the new version of staros will work with My deployed CPE units.
Most of our AP nodes do not run any routing protocols since vtund* (VDS) is used to drop public IP addresses directly to the CPEs.

I WANT to run 99.3 everywhere it reconverges faster.
In in any case 98.5 works just fine (even in mixed 98.5 / 99.3 environments ) in my test setup.
The Quagga developers call 98.5 the stable release. 99.3 is a beta
describing 98.5 as a buggy version is not correct

As I have said earlier, OSPF only cares about it's neighbours. If an node and all it's neighbours are running the same version but that node goes on to loose awareness of it's own directly connected interface(s) (which is something that should never happen IF that interface is up) than that effectively rules out issues due to differing versions. I have to date been unable to duplicate this behaviour outside of staros.

quagga (OSPF) has an awareness of interface state (up or Down ) it seems that this bug may be directly related to detection of link state on wireless (atheros only?) links. this would imply that the bug is in one of 3 general areas.
1/ Quagga, broken in some way that only gets seen with wireless links
2/ Atheros driver
3/ Kernel configuration (it's involved in link state tracking of course)

I have access to 1/ and am simply unable to replicate this issue outside of staros.
What does your kernel config look like?


*BTW vtund is a GPL'ed application. How have you configured it? I need to be able to connect directly to my Linux router. It's more than just inconvenient to have to break OSPF and all the tunnels by applying changes on my tunnel end point box every time I need to add an extra tunnel

.brendan

[oscarBravo]

We can take a look at the up/down states of the driver. In general the 2.4 kernel does not deal with those issues so this is something that OSPF should not be monitoring. We can also take a look at the timing of when we start OSPF. It should be coming up after all drivers as loaded and controlling cards. A long association time might cause it to miss a device so we will check into that. There's a "link-detect" configuration option in Zebra that we're going to do some testing with. I'm not 100% clear on what it does or what bearing it may have on these problems, but we'll keep you posted. We are not disclosing any information about vtun since we will be moving away from that code base and we simply cannot have a situation where we would be forced to keep the previous mechanism due to people using other platforms. Wouldn't you be forced to keep the current mechanism to allow for interoperability with existing v2 tunnel endpoints?

[lonnie]

I have other reasons and we are not going to modify our plans for vtun. We said right from the very beginning that it would not be compatible. There has been no change.

[aldo]

I have other reasons and we are not going to modify our plans for vtun. We said right from the very beginning that it would not be compatible. There has been no change.

can we move this vtun issue elsewhere this thread is progressing ok on ospf.
can we keep it there please

[bminish]

I have most of my nodes on the latest build, I'll finish off the rest tomorrow night.
OSPF still losses interfaces from time to time.

I have also started testing the zebra 'link detect' option. Early indications are that it does not make any difference whatsoever to this problem.
.brendan

[bminish]

I have most of my nodes on the latest build, I'll finish off the rest tomorrow night.
.brendan

Unfortunately we have run into a problem with wpa-psk that prevents us rolling out onto most AP nodes. As outlined elsewhere AP nodes do not as a rule run quagga on our network.

.brendan

[aldo]

I have most of my nodes on the latest build, I'll finish off the rest tomorrow night.
OSPF still losses interfaces from time to time.

I have also started testing the zebra 'link detect' option. Early indications are that it does not make any difference whatsoever to this problem.
.brendan

concur, don't seem to see any changes here

[aldo]

just wondering if there has been any progress on this it is really causing an issue on one edge of our network now.
this whole network is running latest configs.

so bridge mode does not pass packets large that 1028
vlan mode does not pass anything of any resonable size
ntp time clock seems to have issues on reboot
ospf does not work correctly on atheros card interfaces
wds mode does not exsist apart from vds

but all these features are sold as features.

[lonnie]

Bridge mode works with full size packets.
VLAN passes full size minus the VLAN header so mtu should be set for 1496
ntp requires Internet when it starts. If RIP or OSP is settling it could be an issue.
ospf is a quagga issue. We are awaiting their next release.
wds works just fine with prism2.x and awesome with Orinoco. We have never said we have WDS for Atheros.

We are NOT responsible for YOUR network. I find it odd that you are saying that bridging does not work with large packets and that ospf also does not work. This leads me to think you have a real mixture of equipment and designs and you are expecting StarOS to magically make it all work. Sorry. It is your mess, you clean it up.

If you want some advice you should fix your network to remove ALL bridging unless they are true transparent bridges. If they are pseudo client bridges then they are nothing but trouble. I have been saying this for years now, and I am actually getting tired of giving that advice and having it ignored.

just wondering if there has been any progress on this it is really causing an issue on one edge of our network now.
this whole network is running latest configs.

so bridge mode does not pass packets large that 1028
vlan mode does not pass anything of any resonable size
ntp time clock seems to have issues on reboot
ospf does not work correctly on atheros card interfaces
wds mode does not exsist apart from vds

but all these features are sold as features.

[bminish]

ospf is a quagga issue. We are awaiting their next release.


So why is is this version ONLY a problem under staros then? how come microtik have a working OSPF using quagga?

.brendan

[tony]

I am unsure as to what OSPF they use as I've never used their software in the past. StarOS and V3 both contain the standard, available Quagga.

During the development of V3, the Atheros support has been totally rewritten, and geared towards OSPF, and other multicast-heavy protocols. To date, it seems to work very well with Quagga, outside of a few Quagga-isms that are not Atheros related.

We would like to hear if the new V3 OSPF & Atheros work in places where StarOS would not.

[bminish]

I am unsure as to what OSPF they use as I've never used their software in the past. StarOS and V3 both contain the standard, available Quagga.

During the development of V3, the Atheros support has been totally rewritten, and geared towards OSPF, and other multicast-heavy protocols. To date, it seems to work very well with Quagga, outside of a few Quagga-isms that are not Atheros related.

We would like to hear if the new V3 OSPF & Atheros work in places where StarOS would not.

I can't test version 3, I have 50 or so nodes of wrap (x86) and am currently confined to V2 as a result.

.brendan

[tony]

Thanks. When V3 x86 is released, it will borrow from the WAR release, so all the enhancements will be available at that time.

[aldo]

Bridge mode works with full size packets.
VLAN passes full size minus the VLAN header so mtu should be set for 1496
ntp requires Internet when it starts. If RIP or OSP is settling it could be an issue..

right ospf in our example. and dns timings ntp start sequense on fisrt check is the issue it can talk 10 sec to reassicate was ntp built with this delay in mind.


ospf is a quagga issue. We are awaiting their next release.
not one our other boxes in our other rings it is not only on atheos interfaces on staros


wds works just fine with prism2.x and awesome with Orinoco. We have never said we have WDS for Atheros.


you never said you did not either. you say on your website wds support


We are NOT responsible for YOUR network. I find it odd that you are saying that bridging does not work with large packets and that ospf also does not work. This leads me to think you have a real mixture of equipment and designs and you are expecting StarOS to magically make it all work. Sorry. It is your mess, you clean it up.
.

mess. what mess, you dont even know our kit In fact we dont use staros in AP mode be cause we can't bridge them


If you want some advice you should fix your network to remove ALL bridging unless they are true transparent bridges. If they are pseudo client bridges then they are nothing but trouble. I have been saying this for years now, and I am actually getting tired of giving that advice and having it ignored.

that is what we have and they work great we tryed staros becuase it said it could bridge it does not well.

I am agitated now. We woutline the issues, you attack our network.(NOTE) you haven't seen it you have no idea of anything about me think before you speak.

we have spend aboput 1000 euro in testing your kit and we feel it is wasted. more so now with you attitude to not believing these things are problems.
If the things don't work don't advertise them. that is our only point.

we worked through tests and these are the results. We dont even run staros on a live network only a new test ring so stick you attitude and you sortware. this is constructive critisuim for some we have paid for what entitles you to be my critic.

think about it man. things are not always what they seem

[bminish]

Thanks. When V3 x86 is released, it will borrow from the WAR release, so all the enhancements will be available at that time.

So you are asking me to wait for another unspecified period of time for a fix that might be in a new version for which I will have to pay to upgrade my 50 or so nodes just for the privilege of testing.

V2 is still your current release

Get real!

.brendan

[tony]

Unfortunately I'm not in a place to change policy, or perform magic. All I can do is tell you how it's going to be done, in as clear of manner as possible. I am sorry that you will have to wait a little longer, but it won't be long until V3 is released.

We have taken all your complaints and requests seriously, and have done all we can to ensure these problems will be a non-issue with V3.

[lonnie]

Anytime sometime attacks me I will defend myself and the software. You have gone WAY past simply reporting an issue. By listing all the things you seem to be having trouble with (on more than one occasion) and implying false advertising you cross a line. I won't stand for that.

Our AP units bridge perfectly to the Ethernet and also very simply. ntp is a standard GPL piece of software and was compiled with the standard settings. We created the wrapper to activate it.

You complain about bridging not working for packets over a very small size. Sorry, but I know for a fact that bridging has no issue all the way to full size 1500 byte packets. The fact you have trouble with that tells me that you have other things going on. Therefore you have no right to expect that my software will correct for your troubled network.

You can't be having issues like that and expect OSPF or anything else to work correctly. And I get quite annoyed when guys with issues like you are reporting try and tell me the software is next to useless. You need to get control of your network and clean it up. How do I know this? We use it also for a very large, non bridged, fully routed network that spans over 100 km, using 18 remote repeaters, 4 of which have Solar Power only. We have almost 300 customers and we ONLY use our software configured the way I am trying to tell everyone to do it. By all means try it your own way, but please, don't expect the same results I am getting.

I have been helping people for a very LONG time and the sorts of issues you are facing are not new. They are not the fault of our software either. It hurts to be told you are doing it wrong but that is what you have to accept. Only then can you begin to make the changes you need. As long as you hold onto the thought that your design is OK you will keep making the same decisions and it will only get worse.

Now, as for Quagga, there is really nothing I can do. When they release another fix we will of course add it. Until then, we have to be patient.

right ospf in our example. and dns timings ntp start sequense on fisrt check is the issue it can talk 10 sec to reassicate was ntp built with this delay in mind.


not one our other boxes in our other rings it is not only on atheos interfaces on staros




you never said you did not either. you say on your website wds support



mess. what mess, you dont even know our kit In fact we dont use staros in AP mode be cause we can't bridge them



that is what we have and they work great we tryed staros becuase it said it could bridge it does not well.

I am agitated now. We woutline the issues, you attack our network.(NOTE) you haven't seen it you have no idea of anything about me think before you speak.

we have spend aboput 1000 euro in testing your kit and we feel it is wasted. more so now with you attitude to not believing these things are problems.
If the things don't work don't advertise them. that is our only point.

we worked through tests and these are the results. We dont even run staros on a live network only a new test ring so stick you attitude and you sortware. this is constructive critisuim for some we have paid for what entitles you to be my critic.

think about it man. things are not always what they seem

[lonnie]

When Quagga releases a new version we will of course add it. We don't make all the software and we are at the mercy of the guys who are in charge of the parts you are having trouble with.

So you are asking me to wait for another unspecified period of time for a fix that might be in a new version for which I will have to pay to upgrade my 50 or so nodes just for the privilege of testing.

V2 is still your current release

Get real!

.brendan

[aldo]

well thanks for the flame i can appreciate the complication but on reviewing you forums there has been reacurring ospf issues on atheros cards.

the continued flame is unnecerssary as we are just pointing out the flaws that we find in what we pay for in have seen threads back to early 2005 about these problems and it seems all there is to show for it is v3 on VXworks.

and a it is not our problem. as has been explained by others this quagga issue is not a problem with other atheros driver just the staros one. well i can prove that our other rings on bsd and propriatry kit are fine with ospf and atheros card.

the disappointment is that for all the troubleshooting that has been done on this it still does not seem to be a staros issue. can is ask lonnie how many point to mutli point nodes does your company run with ospf.

this seems to be one of the main areas of concern
the problem is shown with a neighbour hanging in init state
tcp dump shows no helo from the neighbour thus the above
some multicast problem that is ittermittent
or else the whole interface will not show up as connected in ospf but works fine

this has been noted in many thread over the past year. this is the best consolidation i can get on the issue. there is still a sometimes it works and somes not bit. personally i would feel there is a firm need to approach these sort of probelms with logic and pragmatism.

as i said above this has been around for a while i am sure a bug track system, would minimise some of the above 9 pages of reiteration.

[bminish]

When Quagga releases a new version we will of course add it. We don't make all the software and we are at the mercy of the guys who are in charge of the parts you are having trouble with.

Quagga is open source development project under the GPL, if you can identify a bug in Quagga then we can bring it to the developers &/or fix it ourselves.
If there's a bug in Quagga then presumably what Tony said a few posts ago about OSPF working ok in V3 is not true? But I guess I only get to find out after I have paid to upgrade all my nodes When/IF valemont see fit to release a 'stable' V3 for x86

I have seen little evidence that this particular issue is simply a quagga issue.
I cannot duplicate this issue outside of staros, In any configuration I have tried.
You know how hard we tested Zebra when we were identifying the original Zebra bugs, including the use of a 13+ node vmware team with staros, then the same team configuration with 13 nodes of 2.4 Kernel debian.
That's how hard we have tested quagga and in a number of different scenarios, I cannot duplicate this issue outside of atheros links running on staros.
In addition to this I have quagga 99.3 running on lots of other routers with zero problems.
Microtik supposedly have a working OSPF based on quagga.

I remain unconvinced to say the least that in this case we are looking at an underlying problem with the quagga daemon.
If you know otherwise let me bring it to the Quagga developers and lets get it resolved but to date your lack of supporting evidence for this claim does nothing to strengthen your case OR inspire confidence that V3 will be materially better in this area. Stop making excuses and put a little more effort in to supporting us, your customers with a fix for this issue

Lonnie, we have tried to work with you on resolving this issue, you are not making it easy.


.brendan

[tony]

To help us identify the problem regarding OSPF and Atheros, is it possible for you to set up an isolated StarOS v2 test environment that we can duplicate exactly, that shows the problem you are encountering? With this information, we can more easily resolve the issue at hand.

Thanks.

[bminish]

To help us identify the problem regarding OSPF and Atheros, is it possible for you to set up an isolated StarOS v2 test environment that we can duplicate exactly, that shows the problem you are encountering? With this information, we can more easily resolve the issue at hand.

Thanks.


On our network, currently no. I have no nodes that are not key infrastructure.
If you can set up a few nodes, say 6 or more wraps each with atheros links with a couple of redundant links I will be glad to configure them up in such a way that you can observe the issue.

.brendan

[lonnie]

Just to make sure, are you guys using neighbour statements? There is no multicast support in Atheros for v2. That means you need to declare the Atheros as non broadcast and use neighbour statements to declare neighbours to explicitly talk with.

[bminish]

Just to make sure, are you guys using neighbour statements? There is no multicast support in Atheros for v2. That means you need to declare the Atheros as non broadcast and use neighbour statements to declare neighbours to explicitly talk with.

Yes I use neighbour statements along with non-broadcast statements on all my wireless links
BTW I think you can get away with not using neighbour statements on atheros to atheros links, just not prism links.

.brendan

[tony]

On our network, currently no. I have no nodes that are not key infrastructure.
If you can set up a few nodes, say 6 or more wraps each with atheros links with a couple of redundant links I will be glad to configure them up in such a way that you can observe the issue.

.brendan
I'm not sure we have that many wraps kicking about, however I will see what I can dig up.

I understand you have 40 or so systems deployed, which means it must work to some extent, however there must be something that causes it to stop working. Is it random, or is it provoked? (eg, activate changes, etc.). When it's no longer working, what is it not doing?

My statements stating that V3's OSPF seems to be functional is based on user feedback, however I am unsure as to the number of systems they have deployed.

Thanks!

[bairdc]

I understand you have 40 or so systems deployed, which means it must work to some extent, however there must be something that causes it to stop working. Is it random, or is it provoked? (eg, activate changes, etc.). When it's no longer working, what is it not doing?


I'll chime in here, as I've seen the OSPF problems on my network from time to time. It's *much* better with Quagga than it was with Zebra, BTW. Anyway, the problem is almost always triggered by an activate changes.

When it happens, OSPF basically "loses" one or more neighbors. Doing a "show ip ospf neighbor" lists all neighbors, except the one that is lost. I have found that stopping/restarting OSPF usually makes it comes back. However, in doing this, it often causes one of the neighbor routers to then go nuts, requiring restarting OSPF on that router. At that point, a neighbor of that router will often do it, so you have to restart OSPF there as well. You often have to restart OSPF on three or four boxes before the network finally stabilizes and behaves itself. Without intervention, I've found that these problems tend to resolve themselves after about 30 minutes or so.

Generally, my OSPF network is relatively stable so long as I don't activate changes on any machines. To say the least, I have made it a point to avoid doing an activate changes whenever possible.

I also run RIPv2 alongside OSPF, which I hate doing, but I have found that it helps to keep things stable when OSPF does go weird. Obviously, this won't work with all networks due to the hop count limitation of RIP.

As mentioned before, this phenomenon seems to be related to machines with Atheros cards. I do have all my atheros links running unicast with neighbors declared. Despite what others have said regarding not needing unicast on Atheros links, I have found that it is necessary. If you don't declare neighbors and set the Atheros interface to non-broadcast, you end up with intermittent problems.

My personal opinion (as uneducated as it might be) is that this whole problem might have something to do with OSPF coming up before Atheros links are associated. Again, this may be a stupid assumption. I'm just trying to think about what happens during an "activate changes" that might cause OSPF to "lose" a neighbor, and reassociation is obviously something that happens.

Anyway, that's my take on the issue. It would be nice to find a resolution to the problem. I wish someone running v3 on WARs would chime in and tell us if the problems are resolved there. However, I suspect that most OSPF users are in the same boat as us, where we don't want to invest any money in WARs without knowing first that this problem is really fixed...

Craig

[tony]

Thank you for your valuable feedback. OSPF as a whole is restarted during an activate changes, as are the Atheros cards. If the Atheros cards take a few seconds to reconnect, OSPF may go into limbo for a while before trying to discover it's neighbors again, causing network problems.

I will see what can be done to help in situations like this.

[lonnie]

No, you need the neighbour links for Atheros.
Yes I use neighbour statements along with non-broadcast statements on all my wireless links
BTW I think you can get away with not using neighbour statements on atheros to atheros links, just not prism links.

.brendan

[lonnie]

The thing with the WAR systems is that they use the latest quagga, something that Brendan has said he cannot get to work. I am fairly confident that it is fixed with the new quagga since Brendan had assured me that 0.99.2 fixed the problem and that 0.99.3 fixed it as well. They are aware of the issue and seem to have fixed it but now he cannot use the new release because of some incompatibilty with his CPE systems.

[bminish]

No, you need the neighbour links for Atheros.
No Lonnie you can run without the neighbour statements, It can be flaky though and I would recommend that neighbour statements be used for all wireless links.

I had a few 'broadcast' Atheros links running when we were first getting started with OSPF.

try it yourself seeing as you don't believe me

.brendan

[bminish]

The thing with the WAR systems is that they use the latest quagga, something that Brendan has said he cannot get to work. I am fairly confident that it is fixed with the new quagga since Brendan had assured me that 0.99.2 fixed the problem and that 0.99.3 fixed it as well.


No lonnie,
I said that 99.2 was our best CHANCE of a fix for the issues we were seeing as quagga had undergone a considerable number of changes in the area of link detection.
Since I have never, with any quagga version been able to replicate this issue outside of staros with atheros links, how could I possibly say that 99.2 (or.3) was going, with certainty, resolve the problem.

I also pointed out that by, being on the current devel build of quagga that it would be possible to bring any identified quagga bugs to the quagga developers


They are aware of the issue and seem to have fixed it but now he cannot use the new release because of some incompatibilty with his CPE systems.

who IS 'THEY'

Your newer builds broke wpa-psk keying for us

THIS IS NOT AN ISSUE CAUSED BY A COUPE OF LEGACY 98.5 NODES 'POISING' THE ROUTING TABLE.

.brendan

[bminish]

I'm not sure we have that many wraps kicking about, however I will see what I can dig up.

the more you have the more often you will see the issue but more than 3 should be enough.
Once we set it up we will have to keep applying changes until we hit the bug. can we also have an IM session open at the same time


I understand you have 40 or so systems deployed, which means it must work to some extent, however there must be something that causes it to stop working. Is it random, or is it provoked? (eg, activate changes, etc.). When it's no longer working, what is it not doing?


activating changes or loss and resumption of link cause the issue. It is, as described earlier in this thread by ourselves and bradg.
OSPF looses all awareness of one or more directly attached radio interfaces



My statements stating that V3's OSPF seems to be functional is based on user feedback, however I am unsure as to the number of systems they have deployed.
Thanks!

in V2 it's generally ok if you leave it alone but applying changes can break OSPF on neighbouring nodes as can loss of link

.brendan

[lonnie]

Sure we tried it as well, and found as you did, that it is flakey. I thought the point was to not be flakey, and therefore you must use neighbour statements for OSPF when using Atheros.
No Lonnie you can run without the neighbour statements, It can be flaky though and I would recommend that neighbour statements be used for all wireless links.

I had a few 'broadcast' Atheros links running when we were first getting started with OSPF.

try it yourself seeing as you don't believe me

.brendan

[lonnie]

This thread will be removed. Please start a new thread simply stating the problem. Keep it free of speculation as to our motives and no accusations or I will get that one pulled as well.

This board is for support related questions and discussions and I will no longer tolerate any attacks. If you do not like the pace we work at, that is too bad. We do the best we can. Attacking and trying to embarrass to get your way is no longer allowed here.