1162275 - Thunderbird is missing mozharness pinning

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Thunderbird is not configured for mozharness pinning, therefore we always use production, which started failing recently.

Comment 1

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Attachment: Fix - v1

This might fix it. Unfortunately its seems there is no buildbot property with the mozilla revision, so I will use the default branch for getting the file.

Comment 2

[Rail Aliiev [:rail]]

Comment on attachment 8602355 [details] [diff] [review]
Fix - v1

lgtm

Comment 3

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

https://hg.mozilla.org/build/buildbot-configs/rev/285aa773a8aa

Comment 4

[Michael Shal [:mshal]]

Attachment: thunderbird-import

This should fix the test failures:

Traceback (most recent call last):
  File "/home/travis/build/mozilla/build-buildbot-configs/.tox/braindump/buildbot-related/dump_master_json.py", line 111, in dump_master
    c = loadMaster(path)
  File "/home/travis/build/mozilla/build-buildbot-configs/.tox/braindump/buildbot-related/dump_master_json.py", line 26, in loadMaster
    execfile(path, g, g)
  File "/home/travis/build/mozilla/build-buildbot-configs/test-output/bm81-build_scheduler/master.cfg", line 6, in <module>
    import thunderbird_config
  File "/home/travis/build/mozilla/build-buildbot-configs/test-output/bm81-build_scheduler/thunderbird_config.py", line 1066, in <module>
    for _, branch in items_at_least(BRANCHES, 'gecko_version', 30):
NameError: name 'items_at_least' is not defined

Comment 5

[Michael Shal [:mshal]]

Comment on attachment 8602828 [details] [diff] [review]
thunderbird-import

https://hg.mozilla.org/build/buildbot-configs/rev/171dd9aa58fd

Comment 6

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/171dd9aa58fd

Comment 7

[Kent James (:rkent)]

I don't see any evidence this is working in current comm-esr38 or comm-beta builds.

Comment 8

[Armen [:armenzg]]

I've lost my initial analysis.

I think we don't use ScriptFactory for TB builds (bug NightlyScriptFactory), hence, pinning cannot be used.
For test jobs we should be able to fix it by adding that same block to mozilla-tests/thunderbid_config.py

rail: does this analysis make sense to you?

misc.py changes => http://hg.mozilla.org/build/buildbotcustom/diff/cb6ce28989c2/misc.py

Original bug: bug 1110286

armenzg@armenzg-thinkpad:~/repos/hg/buildbotcustom$ grep -r "script_repo_manifest" .
./process/factory.py:    def __init__(self, scriptRepo, scriptName, script_repo_manifest=None,
./process/factory.py:            if script_repo_manifest:
./process/factory.py:                        '--manifest-url %s' % script_repo_manifest)],
./process/factory.py:            if script_repo_manifest:
./process/factory.py:                        '--manifest-url %s' % script_repo_manifest)],
./misc.py:        script_repo_manifest=config.get('script_repo_manifest'),
./misc.py:                        script_repo_manifest=None, is_debug=None):
./misc.py:            script_repo_manifest=script_repo_manifest,
./misc.py:                                   script_repo_manifest=None):
./misc.py:        script_repo_manifest=script_repo_manifest,
./misc.py:                            'script_repo_manifest': branch_config.get(
./misc.py:                                 'script_repo_manifest'),
./misc.py:                                test_builder_kwargs['script_repo_manifest'] = branch_config.get('script_repo_manifest')

Comment 9

[Rail Aliiev [:rail]]

(In reply to Armen Zambrano G. (:armenzg - Toronto) from comment #8)
> I've lost my initial analysis.
> 
> I think we don't use ScriptFactory for TB builds (bug NightlyScriptFactory),
> hence, pinning cannot be used.

Sounds correct to me. We have not switched to mozharness based builds for TB.

Comment 10

[Kent James (:rkent)]

So what is the next step forward, and who is going to do it?

Comment 11

[Kent James (:rkent)]

(In reply to Armen Zambrano G. (:armenzg - Toronto) from comment #8)

> I think we don't use ScriptFactory for TB builds (bug NightlyScriptFactory),
> hence, pinning cannot be used.

I'm not an expert here, but I don't understand that conclusion from my reading of the code.

yes 'mozharness_desktop_build' is not set for Thunderbird builds, but 'mozharness_config' is (for example http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/thunderbird_config.py#66)

If 'mozharness_config' is set, then the factory we use is:

            factory = makeMHFactory(config, pf,
                                    signingServers=secrets.get(pf.get('dep_signing_servers')),
                                    use_credentials_file=True)

which uses ScriptFactory.

Or is there something I am missing here?

Comment 12

[Armen [:armenzg]]

The change landed on this bug was for *mozilla*/thunderbid_config.py (that entails builds) and *mozilla-tests*/thunderbid_config.py does not contain for/loop, hence, not activating the pinning for *test* jobs.

Steps:
* Back out previous change
* Land the same change but for mozilla-tests

Outcome:
* Pinning for test jobs will be activated

Comment 13

[Kent James (:rkent)]

The original change was done IIRC to solve some issues with the Lightning repacks. Do we really need to back out the original change, or can the code be in both mozilla/ and mozilla-tests/ ?

Comment 14

[Armen [:armenzg]]

It can be in both.
We should, however, add a comment saying that the pinning block there does not affect TB builds.

Comment 15

[Kent James (:rkent)]

Sorry to keep bugging you, but one more question.

The original patch only uses the default mozilla-* rev, while in practice we typically build from a branch. Is there any way from within the thunderbird_config.py code to get access to current mozilla-* rev used for the build?

Comment 16

[Kent James (:rkent)]

Attachment: add mozharness pinning in mozilla-test thunderbird configurations

I don't believe that we need to deal with the issue of the mozilla branch at the moment, so let's try to get this in so that we can proceed with our release.

Thanks for all of your assistance!

Comment 17

[Armen [:armenzg]]

I have not touched these code often in the last year so my answers can be very limited.

In Firefox, we substitute both the repo_name and gecko revision:
https://hg.mozilla.org/%(repo_path)s/raw-file/%(revision)s/testing/mozharness/mozharness.json'

I don't know if it would be substituted for the TB jobs.

Let's start with default and see.

Comment 18

[Kent James (:rkent)]

Comment on attachment 8615514 [details] [diff] [review]
add mozharness pinning in mozilla-test thunderbird configurations

https://hg.mozilla.org/build/buildbot-configs/rev/664bea6866b7

Comment 19

[Kent James (:rkent)]

Attachment: define moz_repo_path in testing

xpcshell test runs failed because of missing moz_repo_path  I believe this is what is needed to define them.

Comment 20

[Kent James (:rkent)]

Comment on attachment 8615775 [details] [diff] [review]
define moz_repo_path in testing

https://hg.mozilla.org/build/buildbot-configs/rev/57f0c1c2a0d4

Comment 21

[Armen [:armenzg]]

Thanks rail for taking the review! I was 1/2 day out.

Comment 22

[Armen [:armenzg]]

Attachment: moz_repo_path.diff

Maybe?

Comment 23

[Rail Aliiev [:rail]]

Comment on attachment 8616218 [details] [diff] [review]
moz_repo_path.diff

Yeah, something like that. Not sure if that's the right place though. There are some other places where the properties set.

Comment 24

[Kent James (:rkent)]

Comment on attachment 8616218 [details] [diff] [review]
moz_repo_path.diff

https://hg.mozilla.org/build/buildbotcustom/rev/f4b84fac7878

Comment 25

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/f4b84fac7878

Comment 26

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Has been in production for a while, looks like this is good. Marking fixed.