I've been regularly losing up to 5 Nokias a night, for whatever reason(s). They're now in the machine room, so we can rule out heat. Lost 4 last night. Try: - Update start.sh to attempt to perform a noop network action to force connection to Mozilla Guest. Loop this, and if it passes MAX_WAIT without success, reboot. - Spread out n810s a bit, in case interference is the issue. - Static IPs, mobile-specific WIFI network? - Make my monitor/auto-reboot script more noisy so I can know [more granularly] when this happens and possibly catch them before they hibernate.
It's also possible they're falling over when there aren't enough jobs to keep them busy. Maybe decrease my idle scheduler retrigger from 2 hours?
1) Are we losing more since we moved office? Maybe there's something different about the wifi network they are on? For example, I know the access points used in the new office are different to what we used in old office. 2) You dont know why they are failing. But what symptoms do these Nokias show once they fail? 3) Just curious - what do you have to do to get these working again? Earlier this week you mentioned manually re-imaging some Nokias - was that the same issue?
(In reply to comment #2) > 1) Are we losing more since we moved office? Sadly, about the same. I think what we gained in A/C we lost due to wifi. And I don't think it's access points necessarily. I do think the re-routing of DHCP has something to do with it. > 2) You dont know why they are failing. But what symptoms do these Nokias show > once they fail? 1) Hibernating, and once you hold the power button they boot up just fine. They're best off when you reboot them, however, and sometimes they end up as #2 afterwards. 2) Infinite reboot. This requires reimaging. > 3) Just curious - what do you have to do to get these working again? Earlier > this week you mentioned manually re-imaging some Nokias - was that the same > issue? Everything on here, from flashing the ROM on down. https://wiki.mozilla.org/ReferencePlatforms/Test/Maemo
Hacks to keep these guys up: https://bugzilla.mozilla.org/show_bug.cgi?id=499729#c4
These seem to be major culprits: https://bugs.maemo.org/show_bug.cgi?id=3415 -- dies after standby https://bugs.maemo.org/show_bug.cgi?id=3144 -- reboot loop when charging (fixed, but not released)
https://bugs.maemo.org/show_bug.cgi?id=3300 -- n810 display doesn't stay lit if it boots while charging
New hypothesis: Since the n810s are booting thinking they're not powered, they only stay up for a certain amount of time. If I split the test suites enough (to where they finish before Diablo decides to hibernate), they may stay up long enough to reboot again.
Run terminal with while loop: https://bugs.maemo.org/show_bug.cgi?id=3415#c54 https://wiki.mozilla.org/ReferencePlatforms/Test/Maemo#Set_up_stupid
(In reply to comment #8) > Run terminal with while loop: > > https://bugs.maemo.org/show_bug.cgi?id=3415#c54 > https://wiki.mozilla.org/ReferencePlatforms/Test/Maemo#Set_up_stupid This doesn't really help and it effectively hides the homeip widget, which means we can't tell when a device isn't networked.
We had fewer than 10 to reimage today, after maybe a week of running (some of these were imaged with image 3, or manually imaged weeks before). I think there's still a lot of room for improvement, but this seems sustainable for the number of devices we have. I'm going to resolve since that's down considerably from losing ~5 a night. I'll reopen if this starts happening again.
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.