495071 - Toronto's wireless networks flaky

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations: Projects, task)

Description

[Johnathan Nightingale [:johnath]]

We've got Dave Humphrey in the office today on the Mozilla-Canada-Guest network, and he's seeing very intermittent service.  Sometimes his pings to, e.g., yahoo.com are in the 80-300ms range, sometimes they're at 9000 and he can't resolve anything.

pkim's on Mozilla-Canada and seeing similar pokey.

Any indications on the Cisco that there's a problem?

Comment 1

[matthew zeier [:mrz]]

But wired is fine?  Might need to reset "something" - is there something like a lunch time when folks are mostly away?

Comment 2

[Johnathan Nightingale [:johnath]]

Haven't noticed any problems on wired, nor on the 802.11n airport extreme running off wired.

We'll probably be safely at lunch around 9:30am PDT if it's not a long outage and you don't need any local representation?

Comment 3

[matthew zeier [:mrz]]

Missed the daytime window... will look at it tonight.

Comment 4

[matthew zeier [:mrz]]

I now the router's been moved - all the cables are connected?  

I don't single a single association right now. The Mac Mini you have there isn't responding on the address I have for it (10.240.2.234).

Comment 5

[:Gavin Sharp [email: [e-mail]]]

10.240.2.234 was repurposed and probably isn't on right now. There are a couple of other minis I can probably give you access to, though. I'll look into it and email you the details.

Comment 6

[:Gavin Sharp [email: [e-mail]]]

Oh, and we were pretty careful to reconnect all the cables after moving the router. One of the antenna cable's connectors was bent slightly as we were trying to reattach it, and we had to bend it back. Not the sort of thing that I would expect to cause these issues (it wasn't visibly damaged), but it's always a possibility, I guess...

I haven't had issues with the wireless myself, so I'll check tomorrow (and ask the people who were having issues to see whether things are any better).

Comment 7

[matthew zeier [:mrz]]

Can you guys re-do your connectivity test and this time ping the router?

It's either 10.240.2.1 or 10.240.3.1 depending on which network you're on.  I'm not seeing anything in that range that you were seeing.  

Could be RF interference which would be hard to troubleshoot remotely.  That Mac has iStumbler running.  There are a lot of SSIDs but you're not overlapping with any other channels right now.  

I'll leave it up and you can take a look at it in the morning.

Comment 8

[Johnathan Nightingale [:johnath]]

(In reply to comment #7)
> Can you guys re-do your connectivity test and this time ping the router?
> 
> It's either 10.240.2.1 or 10.240.3.1 depending on which network you're on.  I'm
> not seeing anything in that range that you were seeing.  
> 
> Could be RF interference which would be hard to troubleshoot remotely.  That
> Mac has iStumbler running.  There are a lot of SSIDs but you're not overlapping
> with any other channels right now.  
> 
> I'll leave it up and you can take a look at it in the morning.

Just ran mtr for a bit on Mozilla-Canada, with these results:

                                         Packets               Pings
 Host                                  Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. ca-gw1.office.mozilla.org           0.1%  1535    7.8  41.2   1.4 1597. 128.4

Avg ping of 41.2ms to the router doesn't feel all that great, and there's a fair bit of variability there too.  I'll do a similar-length run on Mozilla-Canada-N, the airport express, for comparison purposes.

Comment 9

[Johnathan Nightingale [:johnath]]

Okay, so this is not quite the same sample size, but I think the difference is reasonably clear:

                                  Packets               Pings
 Host                           Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. ca-gw1.office.mozilla.org    0.0%   959    4.1   3.7   1.5  83.9   6.8

Avg ping of 3.7ms is more in line with what I'd expect. Variability is also much lower.

Both of these were in a reasonably empty office (2 of us for the Mozilla-Canada test, 5 of us for the Mozilla-Canada-N run)

Comment 10

[matthew zeier [:mrz]]

Very different that the results I saw lastnight when no one was around but I'm not sure that's the fix.

I talked with Derek - you might be hitting the same problems Building S has with RF noise.  Solution there was to use A/N (5GHz) instead of B/G (2.4Ghz).  That might be why your N radio works fine.

We might be able to re-use the wifi gear from Landings in Toronto or look at using Aruba up there as well.  Both depend on 650 Castro, so probably 2-3 weeks out.  

Can I drop the severity on this until we have alternate hardware to experiment with?

Comment 11

[Mike Shaver (:shaver emeritus)]

I use N, because I like fast, so that's fine with me.

Comment 12

[Mike Beltzner [:beltzner, not reading bugmail]]

Yeah, fine with me. The only people this hits are guests to our office, really.

Comment 13

[Derek Moore [:dmoore]]

Scheduled for further onsite debugging in July

Comment 14

[matthew zeier [:mrz]]

Did the Aruba stuff arrive yet?

Comment 15

[Johnathan Nightingale [:johnath]]

I was away for a week recently so it may have come then, but I don't see it over by the network rack where I'd expect to find it. Do we have package tracking?

Comment 16

[Derek Moore [:dmoore]]

Closing out this implementation bug. Tracking performance issues in Bug 519469.