903654 - B2G Manifest annotator seems to be broken

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

The sources.xml file for last night's build seems to be missing information about tags, as well as not having Gecko or Gaia project nodes.  It is lacking the usual Mercurial-information comments that list the HG sourced repositories.

This points to breakage in the releng system that annotates the manifests given that:

1. the missing repositories are the exact ones that are in Mercurial, which Releng special cases

2. The manifests are clearly being modified post gonk-misc/add-revision.py, but lack all comment tags and do not have anything that Releng normally adds about non-git accessible repositories in Mercurial-Information comments.


There was a change to the add-revision.py script that was landed around the time that the bustage happened, but I've checked locally that the XML that is generated by that change is the same, aside from node ordering.  This shouldn't cause the problem, but we should consider backing it out if it's the cause.

Thanks Dave for finding this!

https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-inari-eng/2013/08/2013-08-08-04-02-03/sources.xml

https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-inari-eng/2013/08/2013-08-09-04-02-01/sources.xml

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This is preventing the B2G performance tests from pushing results to DataZilla.

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

Hal and I met on Vidyo to discuss this.  We've decided to back out these changes for the meantime to get things back into a working state.  We agreed that there should be checks so that the manifests don't break this way again.

Backed out with:

[master 828ca72] bug 903654 - There is something wrong with a downstream consumer of the sources.xml

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: with-patch-sources.xml

Here is an example Unagi master branch sources.xml file that was generated with repo manifest and annotated by add-revisions.py to be used to test the annotator

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

Dave, I don't think this bug is blocking you any more, is it?

Diego, can you please help us get an idea of the priority of this fix landing?

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

We've re-verified and are not aware of any changes here to how these source.xml files are generated. However, I note that there were land/backout/reland changes to add-revision.py between "last known good" manifest and "broken" manifest. dhyland, dwilson: as you two were last touching add-revision.py, is this related to bumpy landing/backout/landing in bug#897559?


per hwine, jhford is going to re-back out these changes (thanks!) and investigate.

Comment 6

[Diego Wilson [:diego]]

I'm pretty check the sanity of sources.xml after the change in bug 897559. Let me recheck.

Comment 7

[Diego Wilson [:diego]]

OK, seems like that before and after patch in bug 897559 I get the same sources.xml *except* it's missing the x-mozbuildid element. That may be the root of the problem here. Feel free to revert my patch if it's causing problems to you trouble. I'll just have to suck it up and try again :-)

P.S. Is there a better way to try out changes to add-revision.py before landing to minimize breakage? I really think this script could use some love but it'd be a shame if people were discouraged from fixing it for fear of process blow back.

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to John Ford [:jhford] -- please use 'needinfo?' instead of a CC from comment #4)
> Dave, I don't think this bug is blocking you any more, is it?

We're using the inari engineering builds on pvtbuilds, so I can't verify until we get a new build. I'm happy to wait for the next daily build, or if someone can trigger a build I'm happy to verify the fix sooner.

Comment 9

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Dave Hunt (:davehunt) from comment #8)
> (In reply to John Ford [:jhford] -- please use 'needinfo?' instead of a CC
> from comment #4)
> > Dave, I don't think this bug is blocking you any more, is it?
> 
> We're using the inari engineering builds on pvtbuilds, so I can't verify
> until we get a new build. I'm happy to wait for the next daily build, or if
> someone can trigger a build I'm happy to verify the fix sooner.

I can confirm this is fixed with the latest builds, thanks!

Comment 10

[Dave Hylands [:dhylands]]

I was just backing out as a request from RyanVM due to suspected tree bustage.(In reply to John O'Duinn [:joduinn] from comment #5)
> We've re-verified and are not aware of any changes here to how these
> source.xml files are generated. However, I note that there were
> land/backout/reland changes to add-revision.py between "last known good"
> manifest and "broken" manifest. dhyland, dwilson: as you two were last
> touching add-revision.py, is this related to bumpy landing/backout/landing
> in bug#897559?

I was just backing out due to a request from RyanVM, who wanted it backed out due to suspected bustage to the tree.

Comment 11

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Diego Wilson [:diego] from comment #7)
> OK, seems like that before and after patch in bug 897559 I get the same
> sources.xml *except* it's missing the x-mozbuildid element. That may be the
> root of the problem here. Feel free to revert my patch if it's causing
> problems to you trouble. I'll just have to suck it up and try again :-)
Thanks Diego, we care about that x-mozbuildid element, so that explains the difference between before+after XML files, and also explains the followon bustage.


> P.S. Is there a better way to try out changes to add-revision.py before
> landing to minimize breakage? I really think this script could use some love
> but it'd be a shame if people were discouraged from fixing it for fear of
> process blow back.
Both finding a better way for devs to help here, and also better alerting on our side when this script goes sideways need to be spun out as separate bugs.

Comment 12

[Chris AtLee [:catlee]]

Just met with jhford about this, and I think I know what's going on. I misunderstood some of the bug comments here and in the related bugs.

The source of the problem is that the contents of the manifest is changing and loses the comment line with "Gonk specific things" in it.

The build script is currently using this comment as a marker for where to insert the new gecko/gaia nodes as well as the informational comments about mercurial revisions [1]. If that comment is missing then we resulting manifest ends up without a gecko or gaia project.

The fix is to modify b2g_build.py so that it doesn't rely on this comment. Perhaps it should also use XML dom methods to insert the nodes rather than just adding lines to the raw XML text.


[1] http://hg.mozilla.org/build/mozharness/file/4ecc13e691a6/scripts/b2g_build.py#l851

Comment 13

[John Ford [:jhford] CET/CEST Berlin Time]

Yes, this happened because of the comment being the marker.  Please make sure that all mutations of the xml manifests are done with a real xml parser and serializer.