1181331 - archiver client should support default or a tag as a revision and fall back to hgmo if archiver fails

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Jordan Lund (:jlund)]

this will help:

- release based factories that use mozharness but are not tied to a particular gecko repo/rev

- releng services (vcs-sync, mapper, etc) that just care about deploying the latest and greatest of mozharness

Comment 1

[Jordan Lund (:jlund)]

Attachment: 150707_bug_1181331_archiver_client_accepts_default_as_rev-tools.patch

I tested this in buildbot staging on a linux and windows slave.

you can verify locally with:

# where $SOME_WORK_DIR can be relative or abs path
python $BUD/tools_jlund/buildfarm/utils/archiver_client.py mozharness --repo mozilla-central --rev default --destination $SOME_WORK_DIR --debug --staging

# negative test. should behave as before
python $BUD/tools_jlund/buildfarm/utils/archiver_client.py mozharness --repo mozilla-central --rev ffa83d153080 --destination $SOME_WORK_DIR2 --debug

Comment 2

[Jordan Lund (:jlund)]

Comment on attachment 8630715 [details] [diff] [review]
150707_bug_1181331_archiver_client_accepts_default_as_rev-tools.patch

I am going to change this slightly first to support release specific jobs too

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8630715 [details] [diff] [review]
150707_bug_1181331_archiver_client_accepts_default_as_rev-tools.patch

Review of attachment 8630715 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildfarm/utils/archiver_client.py
@@ +257,5 @@
> +        cmd = ['hg', 'id', '-r', 'default', 'https://hg.mozilla.org/%s' % (options.repo,)]
> +        log.info('"default" was passed as the revision. Querying remote repository for '
> +                 'corresponding rev hash of current default tip with cmd: %s', cmd)
> +        options.rev = subprocess.Popen(cmd, stdout=subprocess.PIPE).communicate()[0].strip()
> +        log.info('default revision being used: %s', options.rev)

I like this technique!

Comment 4

[Jordan Lund (:jlund)]

Attachment: 150708_bug_1181331_archiver_client_accepts_default_and_tag_as_rev-tools.patch

interdiff from last patch http://people.mozilla.org/~jlund/150708_bug_1181331_archiver_client_accepts_default_and_tag_as_rev-tools-interdiff.patch

what's changed:
 - you can now pass a tag instead of a rev. This will be required for releases.

how to use:
  # same as before for default rev
  python $BUD/tools_jlund/buildfarm/utils/archiver_client.py mozharness --repo mozilla-central --rev default --destination $SOME_WORK_DIR --debug --staging
  
  # and for tag
  > python $BUD/tools_jlund/buildfarm/utils/archiver_client.py mozharness --repo releases/mozilla-beta --tag FIREFOX_40_0b2_RELEASE --destination $SOME_WORK_DIR --debug --staging

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8631077 [details] [diff] [review]
150708_bug_1181331_archiver_client_accepts_default_and_tag_as_rev-tools.patch

Review of attachment 8631077 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildfarm/utils/archiver_client.py
@@ +208,5 @@
>      parser = OptionParser(__doc__)
>      parser.add_option("--repo", dest="repo", default='mozilla-central',
>                        help="The repository the archive is based on.")
> +    parser.add_option("--rev", dest="rev", help="The revision the archive is based on.")
> +    parser.add_option("--tag", dest="tag", help="The tag the archive is based on.")

Might want to mention that this only works for mozharness

Comment 6

[Jordan Lund (:jlund)]

Comment on attachment 8631077 [details] [diff] [review]
150708_bug_1181331_archiver_client_accepts_default_and_tag_as_rev-tools.patch

thanks http://hg.mozilla.org/build/tools/rev/cceda7e95687

Comment 7

[Jordan Lund (:jlund)]

Attachment: 150717_archiver_client_fallsback_to_hgmo_directly.patch

Comment 8

[Rail Aliiev [:rail]]

Comment on attachment 8635617 [details] [diff] [review]
150717_archiver_client_fallsback_to_hgmo_directly.patch

A nit. Can you also pass timeout=NN (30-60s should be ok) to urlopen()?

Comment 9

[Jordan Lund (:jlund)]

Attachment: 150717_archiver_client_fallsback_to_hgmo_directly-2.patch

r=rail - carrying forward. adding timeouts to urlopen