1013705 - release l10n repacks failed due to failed "rm"

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect, P2)

Description

[Rail Aliiev [:rail]]

Smells like a winrm issue to me.

It runs "rm -rf current" and then tries to "mkdir current", but the directory is still there.

command: START
command: rm -rf /c/builds/moz2_slave/rel-m-beta-w32_rpk_5-000000000/mozilla-beta/obj-l10n/dist/current
command: cwd: c:\builds\moz2_slave\rel-m-beta-w32_rpk_5-000000000
command: output:
command: END (0.00s elapsed)

command: START
command: mkdir /c/builds/moz2_slave/rel-m-beta-w32_rpk_5-000000000/mozilla-beta/obj-l10n/dist/current
command: cwd: c:\builds\moz2_slave\rel-m-beta-w32_rpk_5-000000000
command: output:
command: ERROR
Traceback (most recent call last):
  File "c:\builds\moz2_slave\rel-m-beta-w32_rpk_5-000000000\scripts\lib\python\util\commands.py", line 47, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['mkdir', '/c/builds/moz2_slave/rel-m-beta-w32_rpk_5-000000000/mozilla-beta/obj-l10n/dist/current']' returned non-zero exit status 1
mkdir: cannot create directory `/c/builds/moz2_slave/rel-m-beta-w32_rpk_5-000000000/mozilla-beta/obj-l10n/dist/current': File exists
command: END (0.01s elapsed)

Comment 1

[Chris Cooper [:coop] (he/him)]

Are we using a jacuzzi for this? We could roll back just the slaves in that jacuzzi temporarily to unblock while I debug.

 cp /bin/rm-msys.exe /bin/rm.exe

Comment 2

[Chris AtLee [:catlee]]

Not currently, no.

Comment 3

[Chris Cooper [:coop] (he/him)]

The release isn't blocked on this any more, so downgrading to critical.

Comment 4

[Chris Cooper [:coop] (he/him)]

Attachment: Convert paths to Windows-style prior to removal

We still need a platform-specific switch in the code, but at least we're using a single tool again.

I was able to successfully repack Windows beta builds in staging with this patch. Log is here:

http://dev-master1.srv.releng.scl3.mozilla.com:8044/builders/release-mozilla-beta-win32_standalone_repack/builds/14/steps/run_script/logs/stdio

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 8428770 [details] [diff] [review]
Convert paths to Windows-style prior to removal

Review of attachment 8428770 [details] [diff] [review]:
-----------------------------------------------------------------

You can also call msys2windows() for all platforms, it has that check internally.

Comment 6

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8428770 [details] [diff] [review]
Convert paths to Windows-style prior to removal

Review of attachment 8428770 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Rail Aliiev [:rail] from comment #5) 
> You can also call msys2windows() for all platforms, it has that check
> internally.

Hawt. Landed with simplified logic.

https://hg.mozilla.org/build/tools/rev/2ee071e99643

Comment 7

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

This is pretty crappy -- applying msys2windows() just for this call and not elsewhere seems wrong.

How about I make winrm check for something in the environment.. maybe OSTYPE=msys (that seems to be set), and then have it handle msys style paths?

Comment 8

[Chris Cooper [:coop] (he/him)]

(In reply to Vladimir Vukicevic [:vlad] [:vladv] from comment #7)
> This is pretty crappy -- applying msys2windows() just for this call and not
> elsewhere seems wrong.
> 
> How about I make winrm check for something in the environment.. maybe
> OSTYPE=msys (that seems to be set), and then have it handle msys style paths?

Sure, if you want. I don't want to sign you up for more work.