1224861 - All Sony AOSP L ports are broken for v2.5 because of missing manifest

Status: RESOLVED FIXED

(Firefox OS Graveyard :: GonkIntegration, defect)

Description

[Adam Farden [:adfad666]]

reeeaaal nice move, removing all the manifests that people will need to build Firefox OS v2.5 for a whole ton of devices.

Comment 1

[Adam Farden [:adfad666]]

Attachment: b2g-manifest PR

Comment 2

[:gerard-majax]

So it looks all the base manifest gets merged with hardcoded revisions each time branching is done. Which explains why base-l.xml got killed. But then I don't understand that commit: https://github.com/mozilla-b2g/b2g-manifest/commit/84b36d382f60c5225aca78df09ba9f8c0c92822e

And the bug 1203186 referenced does not explain anything. Why was this brought back like this? Can we get the same for L manifests or should we hardcode changes?

The problem is that this is breaking the sony-aosp-l.xml manifest that now relies on a unexistent base-l-aosp.xml: https://github.com/mozilla-b2g/b2g-manifest/blob/v2.5/sony-aosp-l.xml

Those devices are core part of the participation initiative.

Comment 3

[:gerard-majax]

All Sony AOSP L ports are broken on v2.5.

Comment 4

[:gerard-majax]

Failure:
> $ BRANCH=v2.5 ./config.sh aries-l
>
> [...]
>
> fatal: manifest 'sony-aosp-l.xml' not available
> fatal: include base-l-aosp.xml doesn't exist or isn't a file
> Repo sync failed

Comment 5

[Rail Aliiev [:rail]]

(In reply to Alexandre LISSY :gerard-majax from comment #2)
> So it looks all the base manifest gets merged with hardcoded revisions each
> time branching is done. Which explains why base-l.xml got killed.

Yeah, this has been our common practice to "freeze" revisions of repos that we either can't branch (not under mozilla's github account) or for those where branching point is not obvious (multiple manifests pointing to different branches/revisions). We also "flatten" the manifests (include the include for reals and delete them). This way we try to prevent unexpected bustages.


> But then I
> don't understand that commit:
> https://github.com/mozilla-b2g/b2g-manifest/commit/
> 84b36d382f60c5225aca78df09ba9f8c0c92822e
>
> And the bug 1203186 referenced does not explain anything. Why was this
> brought back like this?

This is just to please the travis test as stated in the commit message, no other purposes.


> Can we get the same for L manifests or should we
> hardcode changes?

Feel free to change the manifests as you want. The intention of those automated changes was to have a more or less stable branch, independent of upstream changes.

> The problem is that this is breaking the sony-aosp-l.xml manifest that now
> relies on a unexistent base-l-aosp.xml:
> https://github.com/mozilla-b2g/b2g-manifest/blob/v2.5/sony-aosp-l.xml
> 
> Those devices are core part of the participation initiative.

I see what you mean. The main issue here is that we have many manifests in the b2g-manifest repo. Some of the manifests are supported by Mozilla, some by community, some are outdated. The manifests contain pointers to different repos and sometimes these pointers conflict with each other from branching point of view. For example:

* prebuilts/misc (https://github.com/mozilla-b2g/prebuilts) can be branched (we have powerz!)
* it is referenced by multiple devices
* these devices use different revisions of the repo
* none of the revisions can be considered as a common point in git history as a base revision for v2.5

In this case we try to come with some solution:

1) don't touch
2) ignore some devices and let the maintainers to deal with them. This is especially very useful for really old devices, which are still in the repos, but have no real value (like otoro), expect messing up the branching decisions. So we just ignore them (can be configured in https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/merge_day/b2g_branch_repos.py#l37).


It's been ages since releng branched off of m-c (more than a year, I think) and the expectations from an automated branching may be changed. It would be great to prepare in advance next time and go through all changes to see if we still want them as before.

Thank you for the feedback BTW, and I hope it helps.

Comment 6

[:gerard-majax]

Thanks for the explanations, this is what I suspected but it's nice to have it written down :)

Can you explain why that sony-aosp-l.xml was not considered? Just that nobody thought it was important? I don't blame anyone, given it is not on treeherder and bug 1183823 has been lagging for months now for that :(. But I want to make sure it's not because of some issue.

What would be the best way to fix that? Flatten and freeze like others? If so, can you handle it? Thanks!

Comment 7

[Rail Aliiev [:rail]]

(In reply to Alexandre LISSY :gerard-majax from comment #6)
> Thanks for the explanations, this is what I suspected but it's nice to have
> it written down :)
> 
> Can you explain why that sony-aosp-l.xml was not considered? Just that
> nobody thought it was important? I don't blame anyone, given it is not on
> treeherder and bug 1183823 has been lagging for months now for that :(. But
> I want to make sure it's not because of some issue.

The only reason was that the manifest was causing some branching issues and it's not supported by our CI (buildbot and taskcluster). We could have listed the problematic repos in https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/merge_day/b2g_branch_repos.py#l14 and skip them, but we skipped the device all together, because we considered it "not important enough" as many others.

> What would be the best way to fix that? Flatten and freeze like others? If
> so, can you handle it? Thanks!

It would be great to:

1) clean up https://github.com/mozilla-b2g/b2g-manifest/ and remove old unsupported manifests
2) have clear understanding which manifests are important and which are not so. Something like Tier-1, Tear-2, etc manifests. We can create a readme-like file in b2g-manifests and list the devices and how much we care about them. Or maybe some structured (JSON) file that we can use instead ofhttps://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/merge_day/b2g_branch_repos.py#l37
3) add more travis tests (see https://github.com/mozilla-b2g/b2g-manifest/blob/master/run_travis_tests.sh) so we fail if something goes unexpected.

Can you also go through https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/merge_day/b2g_branch_repos.py#l37 and remove manifests that we care about? This way we won't be skipping those manifests next time.

Comment 8

[:gerard-majax]

Can you give more details ?! "was causing some branching issues and it's not supported by our CI (buildbot and taskcluster)"

Branching issues ? What does that means ? I know it's not on TaskCluster even though we asked for it four months ago.

From your list, we care about:
 - sony-aosp-l.xml,
 - nexus-4-kk.xml
 - nexus-5.xml

Comment 9

[:gerard-majax]

So, can we fix the Sony manifest? What are those "brancing issues" that you referred to?

Comment 10

[Adam Farden [:adfad666]]

For the future, it will a good idea to actually NI the people who are using those manifests if you have some issues, 5 seconds of research will tell you those are all devices being actively worked on.

Such actions are going to cause more and more problems now that Mozilla is actively seeking ports for other devices. How can we expect people to contribute if they wake up one day and find something like their entire manifest has been blown away on a whim?

Comment 11

[Rail Aliiev [:rail]]

(In reply to Alexandre LISSY :gerard-majax from comment #9)
> So, can we fix the Sony manifest?

I submitter a PR (https://github.com/mozilla-b2g/b2g-manifest/pull/400) and then realized there is https://github.com/mozilla-b2g/b2g-manifest/pull/399 as well. Pick one! :)

> What are those "brancing issues" that you referred to?

I'll run the script to see what are the complains.

Comment 12

[:gerard-majax]

https://github.com/mozilla-b2g/b2g-manifest/commit/ba2fc430a8e899fc5972bb8c77a997e23062c8f7

Comment 13

[:gerard-majax]

Thanks all!

Comment 14

[Rail Aliiev [:rail]]

I ran the merge script with https://bug1224897.bmoattachments.org/attachment.cgi?id=8687656 applied and got this:

$ mozharness/scripts/merge_day/b2g_branch_repos.py -c mozharness/configs/merge_day/b2g_branch_repos.py --branch-name v2.5 --clean-repos --pull

....

11:59:09    FATAL - Sanity: Not clear where to branch for platform_bionic git://github.com/mozilla-b2g/platform_bionic {u'b2g-4.0.4_r2.1': ['/home/rail/work/mozilla/merge/build/b2g-manifest/emulator.xml'],
11:59:09    FATAL -  u'b2g-4.4.2_r1': ['/home/rail/work/mozilla/merge/build/b2g-manifest/shinano.xml',
11:59:09    FATAL -                    '/home/rail/work/mozilla/merge/build/b2g-manifest/flame-kk.xml',
11:59:09    FATAL -                    '/home/rail/work/mozilla/merge/build/b2g-manifest/emulator-kk.xml']}
11:59:09    FATAL - Sanity: Not clear where to branch for device_lge_hammerhead-kernel git://github.com/mozilla-b2g/device_lge_hammerhead-kernel {u'b2g-4.4.2_r1': ['/home/rail/work/mozilla/merge/build/b2g-manifest/nexus-5.xml'],
11:59:09    FATAL -  u'b2g-5.1.0_r1': ['/home/rail/work/mozilla/merge/build/b2g-manifest/nexus-5-l.xml']}
11:59:09    FATAL - Sanity: Not clear where to branch for device-hammerhead git://github.com/mozilla-b2g/device-hammerhead {u'b2g-4.4.2_r1': ['/home/rail/work/mozilla/merge/build/b2g-manifest/nexus-5.xml'],
11:59:09    FATAL -  u'b2g-5.1.0_r1': ['/home/rail/work/mozilla/merge/build/b2g-manifest/nexus-5-l.xml']}
11:59:09    FATAL - Use --branch-order or self.config['no_branch_repos'] to fix!
11:59:09    FATAL - Running post_fatal callback...
11:59:09    FATAL - Exiting -1


If you translate the message to a human language :), in the first case it tries to indicate the following:

* git://github.com/mozilla-b2g/platform_bionic can be branched (and it's not in "no_branch_repos", but
* it is used in multiple manifests using different revisions and none of them are specified in https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/merge_day/b2g_branch_repos.py#l34 (branch_order) nor the corresponding repos specified in https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/merge_day/b2g_branch_repos.py#l14 (no_branch_repos)
** b2g-4.0.4_r2.1 is used in emulator.xml
** b2g-4.4.2_r1 is used in shinano.xml, flame-kk.xml, emulator-kk.xml

To resolve this issue we have a few ways to go:

1) add the problematic manifests to https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/merge_day/b2g_branch_repos.py#l38 (ignored_manifests). This is what we did actually. The script skips the manifests in that list.

2) add preferred revision (branch in this case) to "branch_order". In this case we use that revision (let's say b2g-4.4.2_r1) as a branch base for v2.5. Probably not ideal in this case.

3) add the problematic repo (not the manifest) to "no_branch_repos" (https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/merge_day/b2g_branch_repos.py#l14) so the script skips that repo and doesn't touch already existing revisions/branches. I think we should have used this strategy here.

As you see, now it's clear, that we should have used the third method. However, we have so many outdated devices and it's not clear which ones are important. If you apply the same logic to otherold devices we could have ended up with a huge list of "no_branch_repos", what could be either wise or not.

Comment 15

[:gerard-majax]

Ok, so it means it's because of nexus-5 failure that base aosp l manifest got removed, hence breaking sony ones?

Comment 16

[Rail Aliiev [:rail]]

Correct.