499334 - stabilize n810s

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Aki Sasaki (not active)]

I've been regularly losing up to 5 Nokias a night, for whatever reason(s).
They're now in the machine room, so we can rule out heat.
Lost 4 last night.

Try:

- Update start.sh to attempt to perform a noop network action to force connection to Mozilla Guest.  Loop this, and if it passes MAX_WAIT without success, reboot.
- Spread out n810s a bit, in case interference is the issue.
- Static IPs, mobile-specific WIFI network?
- Make my monitor/auto-reboot script more noisy so I can know [more granularly] when this happens and possibly catch them before they hibernate.

Comment 1

[Aki Sasaki (not active)]

It's also possible they're falling over when there aren't enough jobs to keep them busy.  Maybe decrease my idle scheduler retrigger from 2 hours?

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

1) Are we losing more since we moved office? Maybe there's something different about the wifi network they are on? For example, I know the access points used in the new office are different to what we used in old office.

2) You dont know why they are failing. But what symptoms do these Nokias show once they fail?

3) Just curious - what do you have to do to get these working again? Earlier this week you mentioned manually re-imaging some Nokias - was that the same issue?

Comment 3

[Aki Sasaki (not active)]

(In reply to comment #2)
> 1) Are we losing more since we moved office?

Sadly, about the same.  I think what we gained in A/C we lost due to wifi.
And I don't think it's access points necessarily. I do think the re-routing of DHCP has something to do with it.

> 2) You dont know why they are failing. But what symptoms do these Nokias show
> once they fail?

1) Hibernating, and once you hold the power button they boot up just fine.  They're best off when you reboot them, however, and sometimes they end up as #2 afterwards.
2) Infinite reboot.  This requires reimaging.

> 3) Just curious - what do you have to do to get these working again? Earlier
> this week you mentioned manually re-imaging some Nokias - was that the same
> issue?

Everything on here, from flashing the ROM on down.
https://wiki.mozilla.org/ReferencePlatforms/Test/Maemo

Comment 4

[Aki Sasaki (not active)]

Hacks to keep these guys up:
https://bugzilla.mozilla.org/show_bug.cgi?id=499729#c4

Comment 5

[Aki Sasaki (not active)]

These seem to be major culprits:

https://bugs.maemo.org/show_bug.cgi?id=3415 -- dies after standby
https://bugs.maemo.org/show_bug.cgi?id=3144 -- reboot loop when charging (fixed, but not released)

Comment 6

[Aki Sasaki (not active)]

https://bugs.maemo.org/show_bug.cgi?id=3300 -- n810 display doesn't stay lit if it boots while charging

Comment 7

[Aki Sasaki (not active)]

New hypothesis: Since the n810s are booting thinking they're not powered, they only stay up for a certain amount of time.  If I split the test suites enough (to where they finish before Diablo decides to hibernate), they may stay up long enough to reboot again.

Comment 8

[Aki Sasaki (not active)]

Run terminal with while loop:

https://bugs.maemo.org/show_bug.cgi?id=3415#c54
https://wiki.mozilla.org/ReferencePlatforms/Test/Maemo#Set_up_stupid

Comment 9

[Aki Sasaki (not active)]

(In reply to comment #8)
> Run terminal with while loop:
> 
> https://bugs.maemo.org/show_bug.cgi?id=3415#c54
> https://wiki.mozilla.org/ReferencePlatforms/Test/Maemo#Set_up_stupid

This doesn't really help and it effectively hides the homeip widget, which means we can't tell when a device isn't networked.

Comment 10

[Aki Sasaki (not active)]

We had fewer than 10 to reimage today, after maybe a week of running (some of these were imaged with image 3, or manually imaged weeks before).

I think there's still a lot of room for improvement, but this seems sustainable for the number of devices we have.

I'm going to resolve since that's down considerably from losing ~5 a night.  I'll reopen if this starts happening again.