mozilla-central should contain a pointer to the revision of all external B2G repos, not just gaia

RESOLVED FIXED in B2G C4 (2jan on)

Status

Release Engineering
General Automation
P2
major
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: emorley, Assigned: catlee)

Tracking

(Depends on: 1 bug)

unspecified
B2G C4 (2jan on)
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(14 attachments, 1 obsolete attachment)

23.82 KB, patch
aki
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
14.58 KB, patch
catlee
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
2.24 KB, patch
emorley
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
895 bytes, patch
aki
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
4.51 KB, patch
rail
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
2.97 KB, patch
bhearsum
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
1.71 KB, patch
aki
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
5.38 KB, patch
aki
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
16.96 KB, patch
aki
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
2.45 KB, patch
aki
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
2.26 KB, patch
rail
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
4.75 KB, patch
bhearsum
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
1.94 KB, patch
bhearsum
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
5.29 KB, patch
RyanVM
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
(Reporter)

Description

4 years ago
Today, bug 895299 caused B2G device image builds to fail on multiple trees, which required sheriffs to manually pick through the github mozilla-b2g repos looking for recent commits, in order to find the culprits (in the end we needed backouts out of three repos). This isn't the first time this kind of thing has happened - see also bug 897611.

This is pretty frustrating for sheriffs, particularly since we don't know much of the B2G build process well, nor are we as familiar with what's recently landed as say gecko changes. In addition, the non-deterministic nature of these failures means that the B2G device builds should really not be shown by default on TBPL:
https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#8.29_Must_avoid_patterns_known_to_cause_non_deterministic_failures

As such, we should peg the version of the various b2g device repos in a manfiest and have a bot automatically update them (ie bug 868598 except for more than just gaia).

Thanks! :-)
Although this isn't directly a fix to this problem, rel-eng has recently started including links to the sources.xml produced by each build in TBPL.  I've been working on a little Python utility at https://github.com/jonallengriffin/sourcesdiff that can be used to produce a list of commits that were made across all repos, given two arbitrary sources.xml files.

Using this tool will at least mean you won't have to pick through large numbers of B2G repos to figure out what has changed between two builds.

I'll document this a bit better and make a post to dev.b2g about it.
(Reporter)

Comment 2

4 years ago
Ah indeed, I'd forgotten about that bug. Thank you :-)
Depends on: 872628
(Reporter)

Updated

4 years ago
Blocks: 910199
(Reporter)

Comment 3

4 years ago
Bug 910662 is an example of why we really need this - I'm currently having to search through the log, manually copy/paste/diff the sources manifest & even then  it's not clear where the repos are in question, so I still can't do anything to fix the breakage :-(
(Reporter)

Comment 4

4 years ago
Plus it's affecting multiple trees (I really should have closed all of them, or else just hidden the Leo device image builds; but either way I'll get moaned at) - whereas if the revision were pinned, we'd only be breaking one tree.
(Reporter)

Comment 5

4 years ago
Rough proposal:
* Add a manifest to the tree (json or xml don't mind, we're using both at the moment in various places) at https://hg.mozilla.org/mozilla-central/file/tip/b2g/config/manifest.json
* This manifest replaces gaia.json and lists not only the gaia repo + revision, but *all* other non-mozilla-central repositories that make up the B2G build. eg: device-{inari,leo,...}, repos from codeaurora.org... basically anything currently listed in sources.xml.
* The build would then only pull the specified revisions of those repos, and not tip.
* A bot would periodically update manifest.json on b2g-inbound with updated revisions (just as we already do for gaia.json). These commits would be visible to sheriffs (massive win) and also mean that if something broke, the bustage is constrained to just one tree (b2g-inbound) and not all trees as we have at present.
* In times of breakage, the sheriffs would use these manifest.json commits to easily figure out which repo needs a backout, perform that backout upstream, and then the bot would auto-commit the updated rev into b2g-inbound = profit.
To make this solution complete, the sheriffs would also need the ability to disable the bot from updating specific repos in the manifest, since there are many repos that the sheriffs cannot back commits out from.
(Assignee)

Comment 7

4 years ago
Rather than inventing a few format, I would put the XML manifest used by repo in the tree. Perhaps already transformed from scary internet urls to friendly git.m.o urls?
(Reporter)

Updated

4 years ago
Blocks: 910685
(Reporter)

Updated

4 years ago
Blocks: 910745
(Reporter)

Comment 8

4 years ago
More multi-tree breakage today (eg https://tbpl.mozilla.org/php/getParsedLog.php?id=27223986&tree=Mozilla-Inbound) from the staggered landing of bug 907745 / bug 910928.
(Assignee)

Comment 9

4 years ago
I think we can do this in b2g_build.py and some kind of pushbot to keep in-tree manifests up to date.
Assignee: nobody → catlee
Component: Builds → General Automation
Product: Boot2Gecko → Release Engineering
QA Contact: catlee
(Assignee)

Updated

4 years ago
Duplicate of this bug: 910201
(Assignee)

Comment 11

4 years ago
Created attachment 804423 [details] [diff] [review]
Support in-tree sources.xml

A few things going on here:
- b2g_make_manifest.py is what we can use to generate the sources.xml that would go in-tree from the manifests in b2g-manifests.git. I think it will end up running together with the gaia.json updater so we only end up with one push to b2g-inbound per manifests update.

- b2g_build.py supports reading in-tree manifests. It looks for 'b2g_intree_manifest' in gecko's config.json for that device. If set, it assumes that sources.xml is present, and uses that manifest instead. We end up checking gecko out first so we can grab the manifest.

- b2g_build.py stops using repo's mirror functionality and instead symlinks the local .repo/ directory to /builds/git-shared/repo. This looks like it speeds up 'repo sync' quite a bit since we don't have to fetch tags into two sets of git repos.
Attachment #804423 - Flags: review?(aki)

Comment 12

4 years ago
Comment on attachment 804423 [details] [diff] [review]
Support in-tree sources.xml

We may want a way to run the script across multiple manifests (manifest_name/manifest_file as an extend array rather than a store string).  That would probably make rewrite_manifests as an action that calls the various manifest-specific methods per-manifest.  Until then it looks like you're planning on using a wrapper, which is a fine approach.
Attachment #804423 - Flags: review?(aki) → review+
(Reporter)

Comment 13

4 years ago
\o/ :-)
(Assignee)

Updated

4 years ago
Attachment #804423 - Flags: checked-in+
(Assignee)

Comment 14

4 years ago
merged to production...again...
(Assignee)

Comment 15

4 years ago
not working on this currently

to finish up, we need to run b2g_make_manifest.py for each device supported on various branches. that's probably b2g-inbound and b2g26 right now. This should run at the same time as bump_gaia_json.py so we end up making only a single commit or push to each repo.
Assignee: catlee → nobody
Priority: -- → P2
(Assignee)

Comment 16

3 years ago
Created attachment 8344836 [details] [diff] [review]
b2g bumper script

This is very similar to b2g_make_manifest.py, but I wanted a clean start, hence a new script. We can remove b2g_make_manifest any time, it's currently unused.

One question is whether a single invocation and config file limit itself to one branch, or if we should try and handle multiple branches at once? The 1.2 b2g branch needs to get stuff landed on the b2g26_v1.2 gecko branch. I lean towards having the script and configs handling one branch at a time so we can run them in parallel if required.
Attachment #8344836 - Flags: feedback?(aki)
(Assignee)

Comment 17

3 years ago
Forgot to mention that I had this running against b2g-inbound in my user repo:
http://hg.mozilla.org/users/catlee_mozilla.com/b2g-inbound

Comment 18

3 years ago
Comment on attachment 8344836 [details] [diff] [review]
b2g bumper script

(In reply to Chris AtLee [:catlee] from comment #16)
> This is very similar to b2g_make_manifest.py, but I wanted a clean start,
> hence a new script. We can remove b2g_make_manifest any time, it's currently
> unused.

Do you want to? Want me to?

> One question is whether a single invocation and config file limit itself to
> one branch, or if we should try and handle multiple branches at once? The
> 1.2 b2g branch needs to get stuff landed on the b2g26_v1.2 gecko branch. I
> lean towards having the script and configs handling one branch at a time so
> we can run them in parallel if required.

I dealt with this in gaia bumper by having a repo_list:
http://hg.mozilla.org/build/mozharness/file/75801cf30d51/configs/gaia_bumper/gaia_json.py#l15

That lets us choose to run serially or parallel or both.
That may add significant complexity here, though, so if you really don't see us wanting to run this serially, we can skip that work.

>+class B2GBumper(VCSScript):
>+
>+    def __init__(self, require_config_file=True):
>+        super(B2GBumper, self).__init__(
>+            all_actions=[
>+                'clobber',
>+                'checkout-gecko',
>+                'checkout-manifests',
>+                'massage-manifests',
>+                'commit',
>+                'push',
>+                'push-loop',
>+            ],
>+            default_actions=[
>+                'push-loop',
>+            ],

Another use case for https://bugzilla.mozilla.org/show_bug.cgi?id=873273 !
Which will probably sit on the back burner for a while longer.

>+    def massage_manifests(self):
>+        """
>+        For each device in config['devices'], we'll strip projects mentioned in
>+        'ignore_projects', or that have group attribute mentioned in
>+        'filter_groups'.
>+        We'll also map remote urls
>+        Finally, we'll resolve absolute refs for projects that aren't fully
>+        specified.
>+        """
>+        for device, device_config in self.config['devices'].items():
>+            self.info("Massaging manifests for %s" % device)
>+            manifest = self.query_manifest(device)
>+            self.filter_projects(device_config, manifest)
>+            self.filter_groups(device_config, manifest)
>+            self.map_remotes(manifest)
>+            self.resolve_refs(manifest)
>+            repo_manifest.cleanup(manifest)
>+            self.device_manifests[device] = manifest
>+
>+            manifest_path = self.query_manifest_path(device)
>+            self.write_to_file(manifest_path, manifest.toxml())

Bonus points for looking for git.m.o repos that don't exist, and notifying!
What happens in that case here? We can't figure out the ref for the repo?

>+    def push_loop(self):
>+        while True:
>+            self.checkout_gecko()
>+            self.checkout_manifests()
>+            self.massage_manifests()
>+            if not self.commit():
>+                # Nothing changed, we're all done
>+                self.info("No changes - all done")
>+                break
>+
>+            if self.push():
>+                # We did it! Hurray!
>+                self.info("Great success!")
>+                break

I suppose that works, rather than trying to rebase+retry the push itself over and over.
Do you want a max_retries?  Or a sleep between retries to avoid potentially hammering the servers?
Attachment #8344836 - Flags: feedback?(aki) → feedback+
(Assignee)

Comment 19

3 years ago
Created attachment 8359266 [details] [diff] [review]
hg hook change

The manifest bumper is going to be using a different username than the gaia pushbot.
Attachment #8359266 - Flags: review?(emorley)
(Assignee)

Comment 20

3 years ago
Comment on attachment 8344836 [details] [diff] [review]
b2g bumper script

r+ IRL last week
Attachment #8344836 - Flags: review+
Attachment #8344836 - Flags: checked-in+
(Assignee)

Comment 21

3 years ago
Created attachment 8359275 [details] [diff] [review]
add a=b2g-bump to manifests commit

I assume we'll need this to land on non-trunk branches.
Attachment #8359275 - Flags: review?(aki)
(Reporter)

Updated

3 years ago
Attachment #8359266 - Flags: review?(emorley) → review+

Updated

3 years ago
Attachment #8359275 - Flags: review?(aki) → review+
(Assignee)

Updated

3 years ago
Attachment #8359266 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Attachment #8359275 - Flags: checked-in+
(Assignee)

Comment 22

3 years ago
Created attachment 8359935 [details] [diff] [review]
b2g_bumper module

mostly copy/paste from gaia_bumper. Added gittool since that's required by the bumper.
Attachment #8359935 - Flags: review?(rail)
Attachment #8359935 - Flags: review?(rail) → review+
(Assignee)

Updated

3 years ago
Depends on: 959753
(Assignee)

Comment 23

3 years ago
Created attachment 8361091 [details] [diff] [review]
migrate b2g-inbound bumping to b2g_bumper
Attachment #8361091 - Flags: review?
Attachment #8361091 - Flags: review? → review+
(Assignee)

Updated

3 years ago
Attachment #8361091 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Attachment #8359935 - Flags: checked-in+
From what I remember, gaia_bumper will eventually be removed?
(Assignee)

Comment 25

3 years ago
Yup. I'll remove the manifests from puppet and clean up bm66 once everything is switched over.
(Assignee)

Comment 26

3 years ago
Created attachment 8361238 [details] [diff] [review]
make device builds use in-tree manifests
Attachment #8361238 - Flags: review?(aki)
(Assignee)

Comment 27

3 years ago
Created attachment 8361239 [details] [diff] [review]
add wasabi to bumper
Attachment #8361239 - Flags: review?(aki)

Comment 28

3 years ago
Comment on attachment 8361239 [details] [diff] [review]
add wasabi to bumper

hgtool may still have issues finding hg unless PATH is set somewhere.
Attachment #8361239 - Flags: review?(aki) → review+
(Assignee)

Comment 29

3 years ago
Created attachment 8361241 [details] [diff] [review]
make device builds use in-tree manifests
Attachment #8361238 - Attachment is obsolete: true
Attachment #8361238 - Flags: review?(aki)
Attachment #8361241 - Flags: review?(aki)

Updated

3 years ago
Attachment #8361241 - Flags: review?(aki) → review+
(Assignee)

Updated

3 years ago
Attachment #8361239 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Attachment #8361241 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Depends on: 960696
(Assignee)

Comment 30

3 years ago
Created attachment 8361317 [details] [diff] [review]
use mapper to ensure gaia revisions are consistent

haven't tested this doesn't break b2g builds...but it ShouldJustWork™!
Attachment #8361317 - Flags: review?(aki)

Comment 31

3 years ago
Comment on attachment 8361317 [details] [diff] [review]
use mapper to ensure gaia revisions are consistent

We at least have emulator builds on cypress pointing at mozharness default, so we have a way to test before this rolls out to production.
Attachment #8361317 - Flags: review?(aki) → review+
(Assignee)

Updated

3 years ago
Attachment #8361317 - Flags: checked-in+
https://hg.mozilla.org/mozilla-central/rev/170b15394142
Assignee: nobody → catlee
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → B2G C4 (2jan on)
(Assignee)

Comment 33

3 years ago
not quite done yet!
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 34

3 years ago
Created attachment 8361690 [details] [diff] [review]
fix bumper configs to really ignore darwin projects

This strips projects with "darwin" in their list of groups, which means we stop cloning gcc binaries for OSX on our linux build slaves.
Attachment #8361690 - Flags: review?(aki)

Updated

3 years ago
Attachment #8361690 - Flags: review?(aki) → review+
(Assignee)

Updated

3 years ago
Attachment #8361690 - Flags: checked-in+
(Assignee)

Comment 35

3 years ago
Created attachment 8362664 [details] [diff] [review]
make sure manifests end with a newline
Attachment #8362664 - Flags: review?(rail)
Comment on attachment 8362664 [details] [diff] [review]
make sure manifests end with a newline

lgtm
Attachment #8362664 - Flags: review?(rail) → review+
(Assignee)

Updated

3 years ago
Attachment #8362664 - Flags: checked-in+
in production
(Reporter)

Comment 38

3 years ago
The Nexus 4 builds on b2g-inbound are currently broken (bug 965173).
The manifest checkins do not list anything that could be the cause, so I'm presuming at least one of the repos isn't accounted for by the manifests?
(Reporter)

Updated

3 years ago
Blocks: 965173
(Reporter)

Updated

3 years ago
Blocks: 965174
It's there:
https://hg.mozilla.org/integration/b2g-inbound/diff/e0d2a1828aed/b2g/config/mako/sources.xml

And sure enough, that's the one it busted on.
(Reporter)

Comment 40

3 years ago
Eugh missed it in the skim read earlier, think I read the next hunk as being the next file, so stopped reading after the gaia line change, since I didn't know they'd be so far apart as to need to be in two hunks for the same file (if that makes sense lol).

Guess bug 872628 would make this easier to spot.
(Reporter)

Updated

3 years ago
No longer blocks: 965173
(Reporter)

Updated

3 years ago
No longer blocks: 965174
(Assignee)

Comment 41

3 years ago
Still TODO - other b2g release branches

Comment 42

3 years ago
(In reply to Chris AtLee [:catlee] from comment #41)
> Still TODO - other b2g release branches

Please update https://wiki.mozilla.org/ReleaseEngineering/Merge_Duty/Steps when this happens.  The steps for odd-numbered mozilla-central gecko versions will be different from even-numbered (in odd#, m-c will ride the trains to m-a; in even#, m-a will ride the trains to m-b2gXX_vX_X)
https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AmStZDZgJbV7dDhtMDZlQmRtdDB4a1plZXRwNXIzYWc#gid=0
(Assignee)

Comment 43

3 years ago
Created attachment 8379030 [details] [diff] [review]
mozharness config changes for v1.3 bumping
Attachment #8379030 - Flags: review?(bhearsum)
(Assignee)

Comment 44

3 years ago
Created attachment 8379031 [details] [diff] [review]
puppet changes to bump master + v1.3
Attachment #8379031 - Flags: review?(bhearsum)
Comment on attachment 8379031 [details] [diff] [review]
puppet changes to bump master + v1.3

Review of attachment 8379031 [details] [diff] [review]:
-----------------------------------------------------------------

Seems reasonable
Attachment #8379031 - Flags: review?(bhearsum) → review+
Attachment #8379030 - Flags: review?(bhearsum) → review+
(Assignee)

Updated

3 years ago
Attachment #8379030 - Flags: checked-in+
something is in production
(Assignee)

Updated

3 years ago
Attachment #8379031 - Flags: checked-in+
(Assignee)

Comment 47

3 years ago
Created attachment 8379794 [details] [diff] [review]
enable using in-tree manifests for mozilla-b2g28
Attachment #8379794 - Flags: review?(ryanvm)
(Assignee)

Comment 48

3 years ago
(In reply to Chris AtLee [:catlee] from comment #47)
> Created attachment 8379794 [details] [diff] [review]
> enable using in-tree manifests for mozilla-b2g26

err, mozilla-b2g28
Comment on attachment 8379794 [details] [diff] [review]
enable using in-tree manifests for mozilla-b2g28

Review of attachment 8379794 [details] [diff] [review]:
-----------------------------------------------------------------

Not for this bug, but we should probably remove the Unagi configs to avoid confusion.
Attachment #8379794 - Flags: review?(ryanvm) → review+
Attachment #8379794 - Attachment description: enable using in-tree manifests for mozilla-b2g26 → enable using in-tree manifests for mozilla-b2g28
(Assignee)

Comment 50

3 years ago
Comment on attachment 8379794 [details] [diff] [review]
enable using in-tree manifests for mozilla-b2g28

Fixed up some trailing comma issues and landed

https://hg.mozilla.org/releases/mozilla-b2g28_v1_3/rev/d57a7f36eb72
Attachment #8379794 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
(Reporter)

Updated

3 years ago
Blocks: 1028111
You need to log in before you can comment on or make changes to this bug.