725839 - adjust post_upload.py to use new candidates mount

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Chris AtLee [:catlee]]

post_upload.py needs to be taught how to use the new /pub/mozilla.org/firefox/candidates directory.

I think the best way forward here is to have post_upload.py copy files into the new candidates directory (e.g. /pub/mozilla.org/firefox/candidates/11.0b2-candidates/build1/...) and make sure a symlink exists from the old candidates directory to the new one (/pub/mozilla.org/firefox/nightly/11.0b2-candidates -> /pub/mozilla.org/firefox/candidates/11.0b2-candidates)

Comment 1

[Aki Sasaki (not active)]

Mobile already does this by --nightly-dir=candidates in release uploads, I believe (sans softlink).

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [puppet-manifests] create {firefox,thunderbird}/candidates on our file servers

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Hmm, might be a bunch of buildbot factory work for this so I've pre-created a symlink for 11.0b3 in case I don't get that in time. Did a quick verify that post_upload follows that properly too.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 596843 [details] [diff] [review]
[puppet-manifests] create {firefox,thunderbird}/candidates on our file servers

http://hg.mozilla.org/build/puppet-manifests/rev/fdda5d64c8d0

master-puppet1 updated from f74e9363affa to fdda5d64c8d0.

Comment 5

[Armen [:armenzg]]

Is this bug done?

I run the following steps because the documentation [1] indicates to do so:
 # ffxbld@stage
 [ffxbld@surf ~]$ cd /pub/mozilla.org/firefox/nightly
 [ffxbld@surf nightly]$ mkdir ../candidates/12.0b5-candidates
 [ffxbld@surf nightly]$ ln -s ../candidates/12.0b5-candidates 12.0b5-candidates

[1]
https://wiki.mozilla.org/Releases/BuildNotesTemplate#Preparing_to_start_Automation

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

It's not, because of suckage. Those steps are the workaround.

Comment 7

[[e-mail] (:bhearsum)]

We need this for rapid betas.

Comment 8

[John Hopkins (:jhopkins)]

Attachment: create nightly->candidates symlink if not already present (needs testing)

Comment 9

[Chris AtLee [:catlee]]

Comment on attachment 631103 [details] [diff] [review]
create nightly->candidates symlink if not already present (needs testing)

Review of attachment 631103 [details] [diff] [review]:
-----------------------------------------------------------------

::: stage/post_upload.py
@@ +213,5 @@
>  def ReleaseToTinderboxBuildsOverwrite(options, upload_dir, files):
>      ReleaseToTinderboxBuilds(options, upload_dir, files, dated=False)
>  
> +def symlink_nightly_to_candidates(nightly_path, candidates_base_path, version):
> +    _from = "%s/%s-candidates" % (nightly_path, version)

Can you use CANDIDATES_BASE_DIR here?

@@ +219,5 @@
> +    if not os.path.exists(_from):
> +        print("ln -s %s %s" % (_to, _from))
> +        os.symlink(_to, _from)
> +    else:
> +        print("symlink already exists: %s <- %s" % (_to, _from))

we don't need this else: clause in production.

Comment 10

[John Hopkins (:jhopkins)]

If someone is doing test builds on dev-stage01, it'd be nice to piggyback some testing of this post_upload.py patch.

Comment 11

[John Hopkins (:jhopkins)]

Attachment: create nightly->candidates symlink if not already present (needs testing)

updated patch addressing catlee's feedback

Comment 12

[John Hopkins (:jhopkins)]

Attachment: updated post_upload.py patch and .ini changes

Bugfix + change to accommodate ini-file-based configuration.

Test-build logs (note that the second build detects that directory/symlink are present and does nothing):
 http://dev-master01.build.scl1.mozilla.com:8900/builders/release-comm-beta-win32_build/builds/1/steps/make_upload/logs/stdio
 http://dev-master01.build.scl1.mozilla.com:8900/builders/release-comm-beta-linux_build/builds/4/steps/make_upload/logs/stdio

Comment 13

[Chris AtLee [:catlee]]

Comment on attachment 635811 [details] [diff] [review]
updated post_upload.py patch and .ini changes

Review of attachment 635811 [details] [diff] [review]:
-----------------------------------------------------------------

::: stage/post_upload.py
@@ +228,5 @@
> +        print >> sys.stderr, "mkdir %s" % _to
> +        os.mkdir(_to)
> +    if not os.path.exists(_from):
> +        print >> sys.stderr, "ln -s %s %s" % (_to, _from)
> +        os.symlink(_to, _from)

sorry, I didn't catch this before, but I believe there's a possible race condition here if two release builds are trying to upload at the same time, they could both get False for os.path.exists() and then fail when trying to create either the directory or symlink.

Can you add some code to handle the case where the directory/symlink has been created between the test for existence and their creation?

Comment 14

[John Hopkins (:jhopkins)]

Attachment: updated post_upload.py patch and .ini changes

I've added try/except blocks to address your comments.  I've also made the directory/symlink checks specifically check the existence of a directory or symlink instead of just "something by that name exists".

Comment 15

[Chris AtLee [:catlee]]

Comment on attachment 637620 [details] [diff] [review]
updated post_upload.py patch and .ini changes

Review of attachment 637620 [details] [diff] [review]:
-----------------------------------------------------------------

looks good!

::: stage/post_upload.py
@@ +239,5 @@
> +            os.symlink(_to, _from)
> +        except OSError, e:
> +            if e.errno == errno.EEXIST:
> +                print "%s already exists, continuing anyways" % _from
> +                pass

nit: pass is unnecessary here

Comment 16

[John Hopkins (:jhopkins)]

Comment on attachment 637620 [details] [diff] [review]
updated post_upload.py patch and .ini changes

Landed in https://hg.mozilla.org/build/tools/rev/9473d51af6cb
(addressed nit)

Comment 17

[John Hopkins (:jhopkins)]

Following instructions at https://wiki.mozilla.org/ReleaseEngineering/How_To/Modify_scripts_on_stage

$ svn commit
Sending        files/bin/post_upload.py
Sending        files/etc/post_upload.ini
Transmitting file data ..
Committed revision 42139.

Comment 18

[Justin Wood (:Callek)]

Comment on attachment 637620 [details] [diff] [review]
updated post_upload.py patch and .ini changes

Review of attachment 637620 [details] [diff] [review]:
-----------------------------------------------------------------

So this change broke SeaMonkey, and with the bustage I noticed an (unrelated) error.

I solved this by just creating /home/ftp/pub/seamonkey/candidates  but would have been nice if this code did not require that (the code and ini absolutely require me to have candidates somewhere other than nightly/ so it can symlink, which is really unnecessary for SeaMonkey) that said, if the symlink ends up working through our beta without an issue, I won't argue for more than the actual error to be fixed.

::: stage/post_upload.py
@@ +228,5 @@
> +        print >> sys.stderr, "mkdir %s" % _to
> +        try:
> +            os.mkdir(_to)
> +        except OSError, e:
> +            if e.errno == errno.EEXIST:

So, here (and below) yield errors, due to errno being undefined (I suspect you wanted to import errno above for this)

(Here and below)

Comment 19

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #18)
> So this change broke SeaMonkey, ...

example of said bustage:

python comm-beta/mozilla/build/upload.py --base-path . source/seamonkey-2.11b4.source.tar.bz2
 in dir /builds/slave/rel-comm-beta-source/. (timeout 1200 secs)
...
 using PTY: False
sys.argv: ['/usr/local/bin/post_upload.py', '-p', 'seamonkey', '-v', '2.11b4', '-n', '1', '-c', '/tmp/tmp.5EQ41L0Njy/', '/tmp/tmp.5EQ41L0Njy/source/seamonkey-2.11b4.source.tar.bz2']
mkdir /home/ftp/pub/seamonkey/candidates/2.11b4-candidates
Traceback (most recent call last):
  File "/usr/local/bin/post_upload.py", line 500, in <module>
    func(options, upload_dir, files)
  File "/usr/local/bin/post_upload.py", line 259, in ReleaseToCandidatesDir
    symlink_nightly_to_candidates(NIGHTLY_PATH, candidatesFullPath, options.version)
  File "/usr/local/bin/post_upload.py", line 232, in symlink_nightly_to_candidates
    if e.errno == errno.EEXIST:
NameError: global name 'errno' is not defined
Uploading /builds/slave/rel-comm-beta-source/source/seamonkey-2.11b4.source.tar.bz2
Running post-upload command: post_upload.py -p seamonkey -v 2.11b4 -n 1 -c
Command ['ssh', '-o', 'IdentityFile=~/.ssh/seabld_dsa', '[e-mail]', 'post_upload.py -p seamonkey -v 2.11b4 -n 1 -c "/tmp/tmp.5EQ41L0Njy/" "/tmp/tmp.5EQ41L0Njy/source/seamonkey-2.11b4.source.tar.bz2"'] returned non-zero exit code: 1
program finished with exit code 2
elapsedTime=2.262517

Comment 20

[Justin Wood (:Callek)]

Of course, now, after the build itself finished fine, I'm hitting (on repacks, trying to download the binaries):

--02:34:29--  http://stage.mozilla.org/pub/mozilla.org/seamonkey/nightly/2.11b4-candidates/build1/linux-i686/en-US/seamonkey-2.11b4.tar.bz2
Resolving stage.mozilla.org... 63.245.215.47
Connecting to stage.mozilla.org|63.245.215.47|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
02:34:29 ERROR 403: Forbidden.

Comment 21

[Robert Kaiser]

(In reply to Justin Wood (:Callek) from comment #19)
> NameError: global name 'errno' is not defined


That's the actual problem in the patch jhopkins did there. If it would find errno.EEXIST correctly, it should just continue fine.

Comment 22

[Robert Kaiser]

Also, another problem we did run into is that if we put thinks into candidates/ and symlink from nightly/, then repacks choke because http access on stage forbids access to symlinked files/directories and the steps cannot download the builds for repacks. This might be SeaMonkey-only due to using some old release stuff, but not 100% sure. Better check FF/TB for this as well.

Comment 23

[Justin Wood (:Callek)]

$ svn ci -m "Bug 725839 Backout r42139, due to bustage"
Sending        files\bin\post_upload.py
Sending        files\etc\post_upload.ini
Transmitting file data ..
Committed revision 42383.

#...

$ hg out
comparing with ssh://hg.mozilla.org/build/tools
searching for changes
changeset:   2736:cbe3e3905ccf
parent:      2733:9473d51af6cb
user:        Justin Wood <[e-mail]>
date:        Tue Jul 03 11:33:55 2012 -0400
summary:     Bug 725839 - Backed out changeset 9473d51af6cb due to bustage

changeset:   2737:b5beba297c82
tag:         tip
parent:      2735:979ae81a376c
parent:      2736:cbe3e3905ccf
user:        Justin Wood <[e-mail]>
date:        Tue Jul 03 11:34:25 2012 -0400
summary:     Merge Bug 725839 backout

$ hg push
pushing to ssh://hg.mozilla.org/build/tools
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 2 changesets with 4 changes to 4 files
remote: Trying to insert into pushlog.
remote: Please do not interrupt...
remote: Inserted into the pushlog db successfully.

Comment 24

[John Hopkins (:jhopkins)]

Attachment: fix EEXIST usage, create relative (not absolute) symlink

Comment 25

[Justin Wood (:Callek)]

Comment on attachment 640254 [details] [diff] [review]
fix EEXIST usage, create relative (not absolute) symlink

This looks alright to me, I got confused at first on why you needed to do the path.split there, and I'm still not certain on why it is required, but it looks like it will work anyway.

I'd like to allow catlee to take a stab at the review though.

Comment 26

[John Hopkins (:jhopkins)]

Comment on attachment 640254 [details] [diff] [review]
fix EEXIST usage, create relative (not absolute) symlink

Landed in Mercurial at https://hg.mozilla.org/build/tools/rev/5931465e0903

Subversion productiondelivery commit:

$ svn commit
Sending        files/bin/post_upload.py
Sending        files/etc/post_upload.ini
Transmitting file data ..
Committed revision 43187.

Comment 27

[John Hopkins (:jhopkins)]

Things are looking good.

Comment 28

[Nick Thomas [:nthomas] (UTC+12)]

Please update https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Starting_a_Release#Starting_the_automation for the changes on this bug.