881501 - (bld-lion-r5-035) bld-lion-r5-035 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Phil Ringnalda (:philor)]

Can't create directories in tmpdir, fails make check, needs reimage.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

disabled in slavealloc

Comment 2

[Chris Cooper [:coop] (he/him)]

Back in production.

Comment 3

[Phil Ringnalda (:philor)]

Looks like busted disk, failing builds with all sorts of yummy "bus error" and "I/O error" messages.

Disabled in slavealloc.

Comment 4

[[e-mail] (:bhearsum)]

Back in production after an hd replacement.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=45154287&tree=Try

Disabled.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Hasn't rebooted in 6 days, probably got into a faily state and couldn't get out of it. Going to get a reimage as part of bug 1048866.

Comment 7

[Phil Ringnalda (:philor)]

Disabled, has the new brand of mozinstall bustage where there's a /Volumes/Stub owned by root.

Unfortunately, we're desperately, horribly short of Mac try slaves, so we can't afford to leave this disabled for diagnosis for very long.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Can someone please check syslog or where ever it get logged, which process has created this folder under /Volumes?

Comment 9

[Chris Cooper [:coop] (he/him)]

[[e-mail] Volumes]# ls -la /Volumes/Stub
total 0
d--x--x--x  3 cltbld  admin  102 12 Feb 02:36 .
drwxrwxrwt@ 4 root    admin  136 13 Feb 09:27 ..
drwx------  5 root    admin  170 12 Feb 02:36 .Spotlight-V100

Comment 10

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #9)
> [[e-mail] Volumes]# ls -la
> /Volumes/Stub
> total 0
> d--x--x--x  3 cltbld  admin  102 12 Feb 02:36 .
> drwxrwxrwt@ 4 root    admin  136 13 Feb 09:27 ..
> drwx------  5 root    admin  170 12 Feb 02:36 .Spotlight-V100

...which, I should say, is not noticeably different than other machines in this class.

Comment 11

[Chris Cooper [:coop] (he/him)]

Nothing stands out in the system.logs. I've kicked off a re-image.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Wait... could this be spotlight which is blocking us from unmounting this DMG? So the above folder is in use, and later we fail to remount die DMG because the folder already exists. This would be bug 969334.

Comment 13

[Chris Cooper [:coop] (he/him)]

Currently running a job: https://treeherder.mozilla.org/#/jobs?repo=try&revision=df7ad6b35b45&filter-searchStr=OS%20X%2010.7%2064-bit%20try%20leak%20test%20build

Comment 14

[Chris Cooper [:coop] (he/him)]

(In reply to Henrik Skupin (:whimboo) from comment #12)
> Wait... could this be spotlight which is blocking us from unmounting this
> DMG? So the above folder is in use, and later we fail to remount die DMG
> because the folder already exists. This would be bug 969334.

That seems consistent with what I see on disk.

Comment 15

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #13)
> Currently running a job:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=df7ad6b35b45&filter-
> searchStr=OS%20X%2010.7%2064-bit%20try%20leak%20test%20build

Build was green.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Chris Cooper [:coop] from comment #14)
> > Wait... could this be spotlight which is blocking us from unmounting this
> > DMG? So the above folder is in use, and later we fail to remount die DMG
> > because the folder already exists. This would be bug 969334.
> 
> That seems consistent with what I see on disk.

So all of the build machines have spotlight enabled? Doesn't it cost performance?