743465 - (talos-r4-lion-069) talos-r4-lion-069 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Phil Ringnalda (:philor)]

Fairly recently reimaged (end of December, in bug 713883), but it's having trouble hanging during shutdown, and the screenshot we take because of that shows a "your computer was restarted because of a problem" dialog along with the iCal window that mochitest-other opens, even when the current run was something else, like a reftest in the case of https://tbpl.mozilla.org/php/getParsedLog.php?id=10727362&tree=Mozilla-Inbound (which, to play to my prejudices about Bluetooth, also has a very odd looking Bluetooth logo peeking up from the bottom of the screen).

Comment 1

[Mike Taylor [:bear]]

disabled

Comment 2

[Jake Watkins [:dividehex]]

Attachment: Tarbal of Logs and Diag Reports

Comment 3

[Jake Watkins [:dividehex]]

I did find multiple error reports in the logs that indicated a problem during the shutdown process eg.  shutdownStall reports.  I have attached a tarball of the /Library/Logs for further investigation.

Lets give this a re-image and see if it solves our problem (at least temporarily.)

Comment 4

[Jake Watkins [:dividehex]]

Re-image is completed

Comment 5

[Chris AtLee [:catlee]]

back to the production party

Comment 6

[Phil Ringnalda (:philor)]

Smells like bad disk to me.

https://tbpl.mozilla.org/php/getParsedLog.php?id=11739390&tree=Mozilla-Aurora is it hitting

warning:  cannot set modification, access times for tp5/filestube.com^Ostatic.filestube.com/
warning:  cannot set permissions for tp5/filestube.com^Ostatic.filestube.com/

(the directory is tp5/filestube.com/static.filestube.com/), and I think it was this slave that last week had unzip complaining about different path/filename between the central directory and local directory of a zip file.

Bug 706492 was when it "started failing to mount and failed to reboot" in November, bug 713883 was a reimage for an unspecified reason in December, and bug 723282 was a corrupt filesystem in February.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Disabled in slavealloc.

Comment 8

[Aki Sasaki (not active)]

Reimaged + put back into the pool.

Comment 9

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=13490852&tree=Ionmonkey ("program finished with exit code 2" unzipping tp5n.zip)
https://tbpl.mozilla.org/php/getParsedLog.php?id=13496219&tree=Mozilla-Inbound ("inflating: places_generated_max/places.sqlite   bad CRC 75d4361c  (should be 66380fad)" unzipping dirtyMaxDBs.zip

That boy ain't right.

Comment 10

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14642467&tree=Mozilla-Inbound

Comment 11

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14732300&tree=Firefox

inflating: places_generated_max/places.sqlite   bad CRC f050dba8  (should be 3d90687f)

Hardware diagnostics, por favor.

Comment 12

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14757138&tree=Mozilla-Inbound

  inflating: talos/page_load_test/kraken/test-contents.js   bad CRC e19a2966  (should be 9ee37dd7)

Don't think this one got disabled yet, which is going to surprise DCOps when they go to run diagnostics on it :)

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Disabled now.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Back into production now that the RAM has been replaced.

Comment 15

[[e-mail] (:bhearsum)]

Seems to be running fine now in production.