1005185 - nuke lockfiles in .repo dir in b2g_build.py

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Aki Sasaki (not active)]

[09:58]	<philor>	[14:59:42] "fatal: Unable to create '/builds/slave/b2g_m-in_wsb_dep-0000000000000/build/.repo/projects/external/fdlibm.git/refs/remotes/caf/kk_2.7_rb1.37.lock': File exists"

Evidently build/.repo is a softlink to the git share dir, so a clobber doesn't help here.  We could do a find build/.repo -name \*.lock -exec rm -f {} \; (or something) to clear those up.

Comment 2

[Aki Sasaki (not active)]

[[e-mail] repo]$ find . -name \*.lock -mmin +30 -print -exec
rm -f {} \;
./projects/external/ipsec-tools.git/refs/remotes/m/master.lock
[[e-mail] repo]$

[[e-mail] repo]$ find . -name \*.lock -mmin +30 -print -exec rm -f {} \;
./projects/external/protobuf.git/refs/remotes/caf/LNX.LA.3.2.7_rb1.lock
[[e-mail] repo]$

[[e-mail] repo]$ find . -name \*.lock -mmin +30 -print -exec rm -f {} \;
./projects/libcore.git/refs/tags/AU_LINUX_ANDROID_JB_3.2.1.4.04.03.00.183.098.lock
./projects/ndk.git/refs/remotes/m/master.lock
[[e-mail] repo]$

Comment 3

[Chris AtLee [:catlee]]

Attachment: delete repo lock files

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Transplanted to production, because I selfishly need to get an hp box and can resuscitate 4 from prod with this:
  https://hg.mozilla.org/build/mozharness/rev/1d5a8147d4eb

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Working in production in a b2g_mozilla-inbound_hamachi_eng_dep build:

17:36:06     INFO - Deleting stale lock files
17:36:06     INFO - Running command: ['find', '.repo', '-name', '*.lock', '-print', '-delete'] in /builds/slave/b2g_m-in_ham_eng_dep-000000000/build
17:36:06     INFO - Copy/paste: find .repo -name *.lock -print -delete
17:36:06     INFO - Return code: 0
17:36:06     INFO - Running command: ['./config.sh', '-q', 'hamachi', '/builds/slave/b2g_m-in_ham_eng_dep-000000000/build/tmp_manifest/hamachi.xml'] in /builds/slave/b2g_m-in_ham_eng_dep-000000000/build

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

aki points out that the find was instantaneous, and a trailing / is needed on symlinks. With that /, it was 5 min expensive to do the find on bld-centos6-hp-008 (it was running repo at the same time so probably some thrashing). catlee has a patch that moves the find to the error case where updating git has failed.

Comment 7

[Chris AtLee [:catlee]]

Attachment: delete lock files if we fail config

Comment 8

[Chris AtLee [:catlee]]

This is in production now (as of http://hg.mozilla.org/build/mozharness/rev/fd1776381421)

I haven't seen any cases of this finding/deleting lock files, but there are a few cases where it's run. It takes about 3 minutes to run find on each of the 3 instances I've found of this.