729998 - Check permissions failed because setgid

Status: RESOLVED WORKSFORME

(Release Engineering :: Release Automation: Other, defect, P2)

Description

[Armen [:armenzg]]

This is the output from the builder:
Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-chk-prms/scripts/scripts/release/push-to-mirrors.py", line 172, in <module>
    buildNumber=buildNumber)
  File "/builds/slave/rel-m-beta-chk-prms/scripts/scripts/release/push-to-mirrors.py", line 85, in checkStagePermissions
    username=stageUsername, sshKey=stageSshKey)
  File "/builds/slave/rel-m-beta-chk-prms/scripts/scripts/release/../../lib/python/util/commands.py", line 57, in run_remote_cmd
    run_cmd(cmd_prefix + cmd, **kwargs)
  File "/builds/slave/rel-m-beta-chk-prms/scripts/scripts/release/../../lib/python/util/commands.py", line 42, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python-2.5.2/lib/python2.5/subprocess.py", line 462, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ssh', '-l', 'ffxbld', '-i', '/Users/cltbld/.ssh/ffxbld_dsa', 'stage.mozilla.org', 'test "0" = "$(find /pub/mozilla.org/firefox/nightly/11.0b4-candidates/build1/ -mindepth 1 -type d ! -perm 755 ! -path \'*/contrib*\' ! -path \'*/partner-repacks*\' | wc -l)"']' returned non-zero exit status 1

Running these commands allowed me to notice that we're having different permissions than beta3 (notice the 's' instead of 'x').
"drwxr-xr-x" vs "drwxr-sr-x"

[ffxbld@surf ~]$ test "0" = "$(find /pub/mozilla.org/firefox/nightly/11.0b4-candidates/build1/ -mindepth 1 -type d ! -perm 755 ! -path \'*/contrib*\' ! -path \'*/partner-repacks*\' | wc -l)"
[ffxbld@surf ~]$ echo $?
1
[ffxbld@surf ~]$ test "0" = "$(find /pub/mozilla.org/firefox/nightly/11.0b3-candidates/build2/ -mindepth 1 -type d ! -perm 755 ! -path \'*/contrib*\' ! -path \'*/partner-repacks*\' | wc -l)"
[ffxbld@surf ~]$ echo $?1
[ffxbld@surf ~]$ find /pub/mozilla.org/firefox/nightly/11.0b4-candidates/build1/ -mindepth 1 -type d ! -perm 755 ! -path \'*/contrib*\' ! -path \'*/partner-repacks*\' | wc -l
723
[ffxbld@surf ~]$ find /pub/mozilla.org/firefox/nightly/11.0b3-candidates/build2/ -mindepth 1 -type d ! -perm 755 ! -path \'*/contrib*\' ! -path \'*/partner-repacks*\' | wc -l
4
[ffxbld@surf ~]$ find /pub/mozilla.org/firefox/nightly/11.0b4-candidates/build1/ -mindepth 1 -type d ! -perm 755 ! -path \'*/contrib*\' ! -path \'*/partner-repacks*\' | tail -n 1
/pub/mozilla.org/firefox/nightly/11.0b4-candidates/build1/win32/zu
[ffxbld@surf ~]$ ls -l /pub/mozilla.org/firefox/nightly/11.0b3-candidates/build2/win32 | grep zu
drwxr-xr-x 2 ffxbld firefox 4096 Feb 16 05:20 zu
[ffxbld@surf ~]$ ls -l /pub/mozilla.org/firefox/nightly/11.0b4-candidates/build1/win32 | grep zu
drwxr-sr-x 2 ffxbld firefox 4096 Feb 22 15:41 zu

Comment 1

[Armen [:armenzg]]

It could be a fallout of:
http://hg.mozilla.org/build/tools/rev/4988e5cda458

Comment 2

[Armen [:armenzg]]

rail executed this:
find /pub/mozilla.org/firefox/nightly/11.0b4-candidates/build1/ -mindepth 1 -type d ! -perm 755 ! -path '*/contrib*' ! -path '*/partner-repacks*' -exec chmod 755 {} \;
and suggested me to fix it for releases/ as well.

[ffxbld@surf ~]$ find /pub/mozilla.org/firefox/nightly/11.0b4-candidates/build1/ -mindepth 1 -type d ! -perm 755 ! -path '*/contrib*' ! -path '*/partner-repacks*' | wc -l
0
[ffxbld@surf ~]$ find /pub/mozilla.org/firefox/releases/11.0b4 -mindepth 1 -type d ! -perm 755 ! -path '*/contrib*' ! -path '*/partner-repacks*' | wc -l
718
[ffxbld@surf ~]$ find /pub/mozilla.org/firefox/releases/11.0b4 -mindepth 1 -type d ! -perm 755 ! -path '*/contrib*' ! -path '*/partner-repacks*' -exec chmod 755 {} \;
[ffxbld@surf ~]$ find /pub/mozilla.org/fireforeleases/11.0b4 -mindepth 1 -type d ! -perm 755 ! -path '*/contrib*' ! -path '*/partner-repacks*' | wc -l
0

Comment 3

[[e-mail] (:bhearsum)]

Mass move of bugs to Release Automation component.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

We've done many releases without issue since this bug was filed, so *magic happened*.