1153802 - b2g-manifest: travis tests need to be smarter about which b2g bumper configs to test

Status: RESOLVED FIXED

(Firefox OS Graveyard :: General, defect)

Description

[Pete Moore [:pmoore][:pete]]

Forked from https://bugzilla.mozilla.org/show_bug.cgi?id=1134722#c18:

B2G Bumper is running against b2g-inbound (master.py), plus the various b2g release branches (https://hg.mozilla.org/releases/mozilla-b2g*_v*), each for multiple devices, plus against cypress for aries only (aries.py).

B2G Bumper scans this mozharness directory for all configs, and runs all of them.

Each config specifies the b2g-manifests repo and branch to check out (e.g. https://github.com/mozilla/build-mozharness/blob/master/configs/b2g_bumper/v2.2.py#L14-L15) and this is how b2g bumper knows which branch and which source repository of b2g-manifest to use.

Now going back to the travis problem on b2g-manifest... When a change lands on any given b2g-manifest branch, in order to test it, in theory we should "look up" which b2g bumper mozharness config(s) refer to the branch we are on, and run b2g bumper tests using a sanitised version of that config. For example, if a change lands on the v2.2 branch of b2g-manifest, the travis tests for b2g-manifest should detect that the change is on the v2.2 branch, scan the mozharness configs to find https://github.com/mozilla/build-mozharness/blob/master/configs/b2g_bumper/v2.2.py#L14-L15 and thereby ascertain that the v2.2.py b2g bumper config should be used in the travis tests.

At the moment, we simply run across all configs (e.g. see https://github.com/mozilla-b2g/b2g-manifest/blob/master/run_travis_tests.sh#L48).

This causes problems, since we are essentially running b2g bumper tests for a different branch of b2g-manifest than we are currently on, which means we can get invalid failures.

Detecting the branch name automatically in the run_travis_tests.sh script is also not enough, since when somebody forks b2g-manifest to make a change, they will potentially create a different branch name (e.g. including the bug number) which will then make it hard to know which branch it was originally forked from.

The available options I see are:
1) explicitly include the branch name in a config of b2g-manifest
2) explicitly refer to the correct mozharness config(s) in each branch of b2g-manifest (e.g. instead of *.py on line https://github.com/mozilla-b2g/b2g-manifest/blob/master/run_travis_tests.sh#L48, include the config which pertains to the branch you have currently checked out - so the content of the file would vary per branch)

The problem with both solutions is that when a new branch gets added to b2g-manifest, you need to make sure you update it. I don't see any obvious way to enforce this. At best, maybe we could log a warning if we detect the currently checked out git branch name of the b2g-manifest repo doesn't match the value found in the b2g bumper config manifests_revision property of the explicitly specified config file referenced in the run_travis_tests.sh script.

Comment 1

[Pete Moore [:pmoore][:pete]]

Attachment: v2.1 branch Pull Request

Comment 2

[Pete Moore [:pmoore][:pete]]

Attachment: v2.0 branch Pull Request

Comment 3

[Pete Moore [:pmoore][:pete]]

Attachment: v1.4 branch Pull Request

Comment 4

[Pete Moore [:pmoore][:pete]]

Attachment: v2.1s branch Pull Request

Comment 5

[Pete Moore [:pmoore][:pete]]

Attachment: v2.2 branch Pull Request

Comment 6

[Pete Moore [:pmoore][:pete]]

Attachment: master branch Pull Request

Comment 7

[Pete Moore [:pmoore][:pete]]

Merged all Pull Requests.