824754 - (t-snow-r4-0044) t-snow-r4-0044 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[u429623]

talos-r4-snow-046 is reported as "seeing a rash of mysterious crashes in debug tests" - see bug 824498 for details

Please run diagnostics to see if there are any hardware issues, and resolve as needed.

Regardless of hardware issues, please reimage host before returning to releng.

Comment 1

[Salvador Espinoza [:sal]]

I did a regular reboot on this host and it came up with no problems, I didn't read the diags part. Will get to this Monday

Comment 2

[Justin Wood (:Callek)]

This is now re-enabled in prod

Comment 3

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=19457159&tree=Mozilla-Inbound is exactly the same sort of mysterious crash as bug 824498

Comment 4

[Justin Wood (:Callek)]

disabled in slavealloc again.

coop ideas on what our next step is?

Comment 5

[Chris Cooper [:coop] (he/him)]

(In reply to Justin Wood (:Callek) from comment #4)
> disabled in slavealloc again.
> 
> coop ideas on what our next step is?

This is the point where we usually need to replace the logic board. Please open an IT bug with dcops to start that process. Bonus points if you can batch it with other slaves that need the same attention.

Comment 6

[Chris Cooper [:coop] (he/him)]

Slave has been repaired, reimaged, and is back in service.

Comment 7

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=20162205&tree=Mozilla-Inbound

Comment 8

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=20172738&tree=Mozilla-Inbound

Comment 9

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=20217285&tree=Mozilla-Inbound

Comment 10

[Ryan VanderMeulen [:RyanVM]]

See also bug 824498 - it looks like we're still getting intermittent crashes on this machine.

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=20278709&tree=Mozilla-Inbound

Comment 12

[Ryan VanderMeulen [:RyanVM]]

Also, probably dupes of this - bug 845294, bug 847108, bug 847196

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=20251647&tree=Mozilla-Aurora

Comment 14

[Chris Cooper [:coop] (he/him)]

I'm moving this slave to staging permanently and will mark it as such in slavealloc.

Comment 15

[Chris Cooper [:coop] (he/him)]

Attachment: Move snow-010 to production and snow-046 to staging

Swapping 1-for-1 slaves between staging and production. See previous comments in this bug for reasons why snow-046 is unsuitable for production.

Comment 16

[Armen [:armenzg]]

Can you please check if it exists on the graphs production DB?

Comment 17

[Chris Cooper [:coop] (he/him)]

Comment on attachment 720729 [details] [diff] [review]
Move snow-010 to production and snow-046 to staging

Review of attachment 720729 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/buildbot-configs/rev/8f4dfb408cd0

Comment 18

[Armen [:armenzg]]

I've rebooted both slaves into their own respective pools.

Comment 19

[Armen [:armenzg]]

Merged and reconfiguration completed.

Comment 20

[[e-mail] (:bhearsum)]

not responding to pdu reboots

Comment 21

[[e-mail] (:bhearsum)]

Back in production after an HD replacement.

Comment 22

[Armen [:armenzg]]

Attachment: add snow-046 back to the production pool

Disabled in slavealloc meanwhile.

Comment 23

[Armen [:armenzg]]

Comment on attachment 814148 [details] [diff] [review]
add snow-046 back to the production pool

https://hg.mozilla.org/build/buildbot-configs/rev/1952f6d18716

Comment 24

[Aki Sasaki (not active)]

    test_stag_not_in_prod ...                                            [FAIL]

===============================================================================
[FAIL]: test_slave_allocation.SlaveCheck.test_stag_not_in_prod

Traceback (most recent call last):
  File "test/test_slave_allocation.py", line 33, in test_stag_not_in_prod
    'declared as staging-only:\n%s' % '\n'.join(sorted(common_slaves))
twisted.trial.unittest.FailTest: Staging-only slaves should not be declared as production and vice versa. However, the following production slaves declared as staging-only:
talos-r4-snow-046
not equal:
a = set()
b = set(['talos-r4-snow-046'])

Comment 25

[Armen [:armenzg]]

Attachment: add snow-046 back to the production pool and remove it from staging

Backed it out and here's the good patch after testing it with test-masters.sh

Comment 26

[Armen [:armenzg]]

Comment on attachment 814166 [details] [diff] [review]
add snow-046 back to the production pool and remove it from staging

https://hg.mozilla.org/build/buildbot-configs/rev/1ec4a515adcf

Comment 27

[John Hopkins (:jhopkins)]

in production

Comment 28

[Armen [:armenzg]]

Rebooted the machine into production and enabled it on slavealloc.

Comment 29

[Phil Ringnalda (:philor)]

I'll want to be able to find the GC crash in https://tbpl.mozilla.org/php/getParsedLog.php?id=28895921&tree=Try again.

Comment 30

[Phil Ringnalda (:philor)]

And the GC crash in https://tbpl.mozilla.org/php/getParsedLog.php?id=28906391&tree=Try

Comment 31

[Armen [:armenzg]]

What do you mean? Is the slave doing something unexpected?
I don't know much about "finding the GC crash"

Comment 32

[Phil Ringnalda (:philor)]

Garbage collection and cycle collection apparently do a good job of exercising RAM, and so typically a machine with bad RAM (or a CPU that's bad about talking to its RAM, or a single trace with a hairline crack in it between the CPU and the RAM, or whatever it may really be) will, along with hitting PPoD failures in reftests, hit a lot of GC crashes.

That's what this slave did, and the reason we had multiple bugs filed about crashes in tests that only happened on this slave, and that's the reason it was in staging rather than production with a slavealloc note saying not to put it in production.

Disabled in slavealloc, please do not bring it back to production without diagnosing what's actually wrong with the memory, fixing it, and running at least a hundred test runs in staging without a single unexplained unexpected not-seen-on-other-slaves crash.

Comment 34

[Armen [:armenzg]]

philor: any ideas if we could word this for developers to try to find an easy test case?

(In reply to Phil Ringnalda (:philor) from comment #32)
> Garbage collection and cycle collection apparently do a good job of
> exercising RAM, and so typically a machine with bad RAM (or a CPU that's bad
> about talking to its RAM, or a single trace with a hairline crack in it
> between the CPU and the RAM, or whatever it may really be) will, along with
> hitting PPoD failures in reftests, hit a lot of GC crashes.
> 
> That's what this slave did, and the reason we had multiple bugs filed about
> crashes in tests that only happened on this slave, and that's the reason it
> was in staging rather than production with a slavealloc note saying not to
> put it in production.
> 
> Disabled in slavealloc, please do not bring it back to production without
> diagnosing what's actually wrong with the memory, fixing it, and running at
> least a hundred test runs in staging without a single unexplained unexpected
> not-seen-on-other-slaves crash.

Comment 35

[Phil Ringnalda (:philor)]

Sorry, that was casual phrasing on my part. I have no reason to believe that GC/CC are *better* at detecting bad RAM than memtest86 (which has been under development for almost 20 years, focusing on just that one task), they are simply the most likely part of our tests to wind up crashing. Running tests for 24 hours with me looking at the results will indeed show intermittent memory failures, along with lots of noise which is not from intermittent memory failures, but it's not a better way.

I think it would be a far better thing to focus on first running memtest86 on this slave once, since I think any diagnostics on it would have been before we started using that rather than Apple's memory diagnostics, and second on having a plan to run it long enough to detect intermittent failures that don't happen on just one quick run.

Comment 36

[Armen [:armenzg]]

Thanks philor :)

Comment 37

[Armen [:armenzg]]

Memory analysis requested in bug 933886.

Comment 38

[Armen [:armenzg]]

2013-01-03 - "Ran hardware diagnostic three times but did not find any issues.  All hardware passed."
2013-01-15 - "Host has been reimaged."
2013-02-05 - "mysterious crashes"
2013-02-15 - logic board replaced
2013-02-17 - same issues
2013-10-02 - back from HD replacement
2013-10-09 - reported more issues
2013-11-11 - memtest did not find any issues "After running memtest86+ multiples times"

I can only see ourselves replacing the memory and give it one more shot. After that we should decommission it.

Comment 39

[Armen [:armenzg]]

RAM has been replaced.
Putting back into production.
I will check tomorrow.

Comment 40

[Armen [:armenzg]]

It looks good.