899969 - mozilla-central should contain a pointer to the revision of all external B2G repos, not just gaia

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Ed Morley [:emorley]]

Today, bug 895299 caused B2G device image builds to fail on multiple trees, which required sheriffs to manually pick through the github mozilla-b2g repos looking for recent commits, in order to find the culprits (in the end we needed backouts out of three repos). This isn't the first time this kind of thing has happened - see also bug 897611.

This is pretty frustrating for sheriffs, particularly since we don't know much of the B2G build process well, nor are we as familiar with what's recently landed as say gecko changes. In addition, the non-deterministic nature of these failures means that the B2G device builds should really not be shown by default on TBPL:
https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#8.29_Must_avoid_patterns_known_to_cause_non_deterministic_failures

As such, we should peg the version of the various b2g device repos in a manfiest and have a bot automatically update them (ie bug 868598 except for more than just gaia).

Thanks! :-)

Comment 1

[Jonathan Griffin (:jgriffin)]

Although this isn't directly a fix to this problem, rel-eng has recently started including links to the sources.xml produced by each build in TBPL.  I've been working on a little Python utility at https://github.com/jonallengriffin/sourcesdiff that can be used to produce a list of commits that were made across all repos, given two arbitrary sources.xml files.

Using this tool will at least mean you won't have to pick through large numbers of B2G repos to figure out what has changed between two builds.

I'll document this a bit better and make a post to dev.b2g about it.

Comment 2

[Ed Morley [:emorley]]

Ah indeed, I'd forgotten about that bug. Thank you :-)

Comment 3

[Ed Morley [:emorley]]

Bug 910662 is an example of why we really need this - I'm currently having to search through the log, manually copy/paste/diff the sources manifest & even then  it's not clear where the repos are in question, so I still can't do anything to fix the breakage :-(

Comment 4

[Ed Morley [:emorley]]

Plus it's affecting multiple trees (I really should have closed all of them, or else just hidden the Leo device image builds; but either way I'll get moaned at) - whereas if the revision were pinned, we'd only be breaking one tree.

Comment 5

[Ed Morley [:emorley]]

Rough proposal:
* Add a manifest to the tree (json or xml don't mind, we're using both at the moment in various places) at https://hg.mozilla.org/mozilla-central/file/tip/b2g/config/manifest.json
* This manifest replaces gaia.json and lists not only the gaia repo + revision, but *all* other non-mozilla-central repositories that make up the B2G build. eg: device-{inari,leo,...}, repos from codeaurora.org... basically anything currently listed in sources.xml.
* The build would then only pull the specified revisions of those repos, and not tip.
* A bot would periodically update manifest.json on b2g-inbound with updated revisions (just as we already do for gaia.json). These commits would be visible to sheriffs (massive win) and also mean that if something broke, the bustage is constrained to just one tree (b2g-inbound) and not all trees as we have at present.
* In times of breakage, the sheriffs would use these manifest.json commits to easily figure out which repo needs a backout, perform that backout upstream, and then the bot would auto-commit the updated rev into b2g-inbound = profit.

Comment 6

[Jonathan Griffin (:jgriffin)]

To make this solution complete, the sheriffs would also need the ability to disable the bot from updating specific repos in the manifest, since there are many repos that the sheriffs cannot back commits out from.

Comment 7

[Chris AtLee [:catlee]]

Rather than inventing a few format, I would put the XML manifest used by repo in the tree. Perhaps already transformed from scary internet urls to friendly git.m.o urls?

Comment 8

[Ed Morley [:emorley]]

More multi-tree breakage today (eg https://tbpl.mozilla.org/php/getParsedLog.php?id=27223986&tree=Mozilla-Inbound) from the staggered landing of bug 907745 / bug 910928.

Comment 9

[Chris AtLee [:catlee]]

I think we can do this in b2g_build.py and some kind of pushbot to keep in-tree manifests up to date.

Comment 11

[Chris AtLee [:catlee]]

Attachment: Support in-tree sources.xml

A few things going on here:
- b2g_make_manifest.py is what we can use to generate the sources.xml that would go in-tree from the manifests in b2g-manifests.git. I think it will end up running together with the gaia.json updater so we only end up with one push to b2g-inbound per manifests update.

- b2g_build.py supports reading in-tree manifests. It looks for 'b2g_intree_manifest' in gecko's config.json for that device. If set, it assumes that sources.xml is present, and uses that manifest instead. We end up checking gecko out first so we can grab the manifest.

- b2g_build.py stops using repo's mirror functionality and instead symlinks the local .repo/ directory to /builds/git-shared/repo. This looks like it speeds up 'repo sync' quite a bit since we don't have to fetch tags into two sets of git repos.

Comment 12

[Aki Sasaki (not active)]

Comment on attachment 804423 [details] [diff] [review]
Support in-tree sources.xml

We may want a way to run the script across multiple manifests (manifest_name/manifest_file as an extend array rather than a store string).  That would probably make rewrite_manifests as an action that calls the various manifest-specific methods per-manifest.  Until then it looks like you're planning on using a wrapper, which is a fine approach.

Comment 13

[Ed Morley [:emorley]]

\o/ :-)

Comment 14

[Chris AtLee [:catlee]]

merged to production...again...

Comment 15

[Chris AtLee [:catlee]]

not working on this currently

to finish up, we need to run b2g_make_manifest.py for each device supported on various branches. that's probably b2g-inbound and b2g26 right now. This should run at the same time as bump_gaia_json.py so we end up making only a single commit or push to each repo.

Comment 16

[Chris AtLee [:catlee]]

Attachment: b2g bumper script

This is very similar to b2g_make_manifest.py, but I wanted a clean start, hence a new script. We can remove b2g_make_manifest any time, it's currently unused.

One question is whether a single invocation and config file limit itself to one branch, or if we should try and handle multiple branches at once? The 1.2 b2g branch needs to get stuff landed on the b2g26_v1.2 gecko branch. I lean towards having the script and configs handling one branch at a time so we can run them in parallel if required.

Comment 17

[Chris AtLee [:catlee]]

Forgot to mention that I had this running against b2g-inbound in my user repo:
http://hg.mozilla.org/users/catlee_mozilla.com/b2g-inbound

Comment 18

[Aki Sasaki (not active)]

Comment on attachment 8344836 [details] [diff] [review]
b2g bumper script

(In reply to Chris AtLee [:catlee] from comment #16)
> This is very similar to b2g_make_manifest.py, but I wanted a clean start,
> hence a new script. We can remove b2g_make_manifest any time, it's currently
> unused.

Do you want to? Want me to?

> One question is whether a single invocation and config file limit itself to
> one branch, or if we should try and handle multiple branches at once? The
> 1.2 b2g branch needs to get stuff landed on the b2g26_v1.2 gecko branch. I
> lean towards having the script and configs handling one branch at a time so
> we can run them in parallel if required.

I dealt with this in gaia bumper by having a repo_list:
http://hg.mozilla.org/build/mozharness/file/75801cf30d51/configs/gaia_bumper/gaia_json.py#l15

That lets us choose to run serially or parallel or both.
That may add significant complexity here, though, so if you really don't see us wanting to run this serially, we can skip that work.

>+class B2GBumper(VCSScript):
>+
>+    def __init__(self, require_config_file=True):
>+        super(B2GBumper, self).__init__(
>+            all_actions=[
>+                'clobber',
>+                'checkout-gecko',
>+                'checkout-manifests',
>+                'massage-manifests',
>+                'commit',
>+                'push',
>+                'push-loop',
>+            ],
>+            default_actions=[
>+                'push-loop',
>+            ],

Another use case for https://bugzilla.mozilla.org/show_bug.cgi?id=873273 !
Which will probably sit on the back burner for a while longer.

>+    def massage_manifests(self):
>+        """
>+        For each device in config['devices'], we'll strip projects mentioned in
>+        'ignore_projects', or that have group attribute mentioned in
>+        'filter_groups'.
>+        We'll also map remote urls
>+        Finally, we'll resolve absolute refs for projects that aren't fully
>+        specified.
>+        """
>+        for device, device_config in self.config['devices'].items():
>+            self.info("Massaging manifests for %s" % device)
>+            manifest = self.query_manifest(device)
>+            self.filter_projects(device_config, manifest)
>+            self.filter_groups(device_config, manifest)
>+            self.map_remotes(manifest)
>+            self.resolve_refs(manifest)
>+            repo_manifest.cleanup(manifest)
>+            self.device_manifests[device] = manifest
>+
>+            manifest_path = self.query_manifest_path(device)
>+            self.write_to_file(manifest_path, manifest.toxml())

Bonus points for looking for git.m.o repos that don't exist, and notifying!
What happens in that case here? We can't figure out the ref for the repo?

>+    def push_loop(self):
>+        while True:
>+            self.checkout_gecko()
>+            self.checkout_manifests()
>+            self.massage_manifests()
>+            if not self.commit():
>+                # Nothing changed, we're all done
>+                self.info("No changes - all done")
>+                break
>+
>+            if self.push():
>+                # We did it! Hurray!
>+                self.info("Great success!")
>+                break

I suppose that works, rather than trying to rebase+retry the push itself over and over.
Do you want a max_retries?  Or a sleep between retries to avoid potentially hammering the servers?

Comment 19

[Chris AtLee [:catlee]]

Attachment: hg hook change

The manifest bumper is going to be using a different username than the gaia pushbot.

Comment 20

[Chris AtLee [:catlee]]

Comment on attachment 8344836 [details] [diff] [review]
b2g bumper script

r+ IRL last week

Comment 21

[Chris AtLee [:catlee]]

Attachment: add a=b2g-bump to manifests commit

I assume we'll need this to land on non-trunk branches.

Comment 22

[Chris AtLee [:catlee]]

Attachment: b2g_bumper module

mostly copy/paste from gaia_bumper. Added gittool since that's required by the bumper.

Comment 23

[Chris AtLee [:catlee]]

Attachment: migrate b2g-inbound bumping to b2g_bumper

Comment 24

[Dustin J. Mitchell [:dustin] (he/him)]

From what I remember, gaia_bumper will eventually be removed?

Comment 25

[Chris AtLee [:catlee]]

Yup. I'll remove the manifests from puppet and clean up bm66 once everything is switched over.

Comment 26

[Chris AtLee [:catlee]]

Attachment: make device builds use in-tree manifests

Comment 27

[Chris AtLee [:catlee]]

Attachment: add wasabi to bumper

Comment 28

[Aki Sasaki (not active)]

Comment on attachment 8361239 [details] [diff] [review]
add wasabi to bumper

hgtool may still have issues finding hg unless PATH is set somewhere.

Comment 29

[Chris AtLee [:catlee]]

Attachment: make device builds use in-tree manifests

Comment 30

[Chris AtLee [:catlee]]

Attachment: use mapper to ensure gaia revisions are consistent

haven't tested this doesn't break b2g builds...but it ShouldJustWork™!

Comment 31

[Aki Sasaki (not active)]

Comment on attachment 8361317 [details] [diff] [review]
use mapper to ensure gaia revisions are consistent

We at least have emulator builds on cypress pointing at mozharness default, so we have a way to test before this rolls out to production.

Comment 32

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/170b15394142

Comment 33

[Chris AtLee [:catlee]]

not quite done yet!

Comment 34

[Chris AtLee [:catlee]]

Attachment: fix bumper configs to really ignore darwin projects

This strips projects with "darwin" in their list of groups, which means we stop cloning gcc binaries for OSX on our linux build slaves.

Comment 35

[Chris AtLee [:catlee]]

Attachment: make sure manifests end with a newline

Comment 36

[Rail Aliiev [:rail]]

Comment on attachment 8362664 [details] [diff] [review]
make sure manifests end with a newline

lgtm

Comment 37

[John Hopkins (:jhopkins)]

in production

Comment 38

[Ed Morley [:emorley]]

The Nexus 4 builds on b2g-inbound are currently broken (bug 965173).
The manifest checkins do not list anything that could be the cause, so I'm presuming at least one of the repos isn't accounted for by the manifests?

Comment 39

[Ryan VanderMeulen [:RyanVM]]

It's there:
https://hg.mozilla.org/integration/b2g-inbound/diff/e0d2a1828aed/b2g/config/mako/sources.xml

And sure enough, that's the one it busted on.

Comment 40

[Ed Morley [:emorley]]

Eugh missed it in the skim read earlier, think I read the next hunk as being the next file, so stopped reading after the gaia line change, since I didn't know they'd be so far apart as to need to be in two hunks for the same file (if that makes sense lol).

Guess bug 872628 would make this easier to spot.

Comment 41

[Chris AtLee [:catlee]]

Still TODO - other b2g release branches

Comment 42

[Aki Sasaki (not active)]

(In reply to Chris AtLee [:catlee] from comment #41)
> Still TODO - other b2g release branches

Please update https://wiki.mozilla.org/ReleaseEngineering/Merge_Duty/Steps when this happens.  The steps for odd-numbered mozilla-central gecko versions will be different from even-numbered (in odd#, m-c will ride the trains to m-a; in even#, m-a will ride the trains to m-b2gXX_vX_X)
https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AmStZDZgJbV7dDhtMDZlQmRtdDB4a1plZXRwNXIzYWc#gid=0

Comment 43

[Chris AtLee [:catlee]]

Attachment: mozharness config changes for v1.3 bumping

Comment 44

[Chris AtLee [:catlee]]

Attachment: puppet changes to bump master + v1.3

Comment 45

[[e-mail] (:bhearsum)]

Comment on attachment 8379031 [details] [diff] [review]
puppet changes to bump master + v1.3

Review of attachment 8379031 [details] [diff] [review]:
-----------------------------------------------------------------

Seems reasonable

Comment 46

[[e-mail] (:bhearsum)]

something is in production

Comment 47

[Chris AtLee [:catlee]]

Attachment: enable using in-tree manifests for mozilla-b2g28

Comment 48

[Chris AtLee [:catlee]]

(In reply to Chris AtLee [:catlee] from comment #47)
> Created attachment 8379794 [details] [diff] [review]
> enable using in-tree manifests for mozilla-b2g26

err, mozilla-b2g28

Comment 49

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8379794 [details] [diff] [review]
enable using in-tree manifests for mozilla-b2g28

Review of attachment 8379794 [details] [diff] [review]:
-----------------------------------------------------------------

Not for this bug, but we should probably remove the Unagi configs to avoid confusion.

Comment 50

[Chris AtLee [:catlee]]

Comment on attachment 8379794 [details] [diff] [review]
enable using in-tree manifests for mozilla-b2g28

Fixed up some trailing comma issues and landed

https://hg.mozilla.org/releases/mozilla-b2g28_v1_3/rev/d57a7f36eb72