Closed Bug 835431 Opened 12 years ago Closed 12 years ago

releng b2g manifest munging script are generating manifests with the wrong project names

Categories

(Release Engineering :: General, defect, P2)

Product:
Release Engineering
Component:
General
Platform:
x86
macOS
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jhford, Assigned: catlee)

References

Details

(Whiteboard: [b2g])

Attachments

(1 file)

fix remote parsing
5.20 KB, patch
catlee
: review+
mozilla
: checked-in+
Details | Diff | Splinter Review
This is breaking our partners. Example manifest: http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/latest/source_unagi_2013-01-26-07.xml In the XML markup generated by the manifest munging scripts, the project nodes for gecko.git and gaia.git are incorrectly generated. The name attribute should be "gecko.git" and "gaia.git", respectively. The manifests that are being generated are also using the wrong set of remotes. In the manifests that are uploaded to stage and pvtbuild, the 'mozillaorg' has a url of "http://git.mozilla.org" and the 'b2g' has a url of "git://github.com/mozilla-b2g/". This is not inline with our actual, authoritative manifests and defeats the purpose of having git.mozilla.org's permanent URLs. The remotes that should be used are: 'b2g': 'https://git.mozilla.org/b2g' 'mozillaorg': 'https://git.mozilla.org/releases' Please use the correct base manifest from "https://github.com/mozilla-b2g/b2g-manifest" for the manifests we ship. There haven't been any updates to the otoro or unagi manifests or builds on http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/latest since Jan 26. Is this no longer the place where partners can track our latest changes?
Component: Release Engineering → Release Engineering: Automation (General)
QA Contact: catlee
There are two separate issues here: the contents of the manifests, and missing manifests. The 2nd issue is bug 835378. Let's focus this bug on the first issue.
Component: Release Engineering: Automation (General) → Release Engineering
QA Contact: catlee
Summary: releng b2g manifest munging script are generating invalid manifests with the wrong remotes and haven't updated since Jan 26 → releng b2g manifest munging script are generating manifests with the wrong remotes
Ok, let me see if I can restate this: a) the 'b2g' remote should point to https://git.mozilla.org/b2g (currently git://github.com/mozilla-b2g/) b) the 'mozillaorg' remote should point to https://git.mozilla.org/releases (currently http://git.mozilla.org/) c) gaia's project name should be gaia.git d) gecko's project name should be gecko.git Is that correct? Which of these is causing bustage for partners? Exactly what is breaking?
FYI, we had to change the code around this due to your rename of mozillaorg from http://git.mozilla.org to http://git.mozilla.org/releases ; this was all working fine before that.
Some more background: bug 822970 changed the default branch for b2g manifests to 'nightly'. Since that change we've been using 'nightly' to generate the base sources.xml. Many of the changes requested above haven't been merged to the 'nightly' branch of b2g-manifests.
The issue breaking our partners is that the full repository url is in the project node's name attribute, which duplicates part of the remote node's remote attribute. Repo is currently trying to clone: "http://git.mozilla.org/http://git.mozilla.org/releases/gecko.git" instead of "http://git.mozilla.org/releases/gecko.git"
found in triage.
Component: Release Engineering → Release Engineering: Automation (General)
QA Contact: catlee
Summary: releng b2g manifest munging script are generating manifests with the wrong remotes → releng b2g manifest munging script are generating manifests with the wrong project names
The name of the gecko.git/gaia.git is currently broken because mozilla.org is http:// instead of https:// currently. We'll still need to update all the toolchains to update the remote lines, and deal with the b2g manifests nightly branch question. But this patch should help in the meantime. This patch: * uses urlparse to only get the path of the remote url for mozillaorg. Instead of taking the full url, it appends the path to https://git.mozilla.org. ** Ugly hardcode, but we're already hardcoding all over this file. * gets rid of a duplicate json import * some whitespace changes We're already taking the remote url and replacing it with '' in the gecko.git/gaia.git/l10n repo urls here: http://hg.mozilla.org/build/mozharness/file/7bcf253585ab/scripts/b2g_build.py#l568 The replace should actually do something once b2g_base_url starts with https:// instead of http://.
Attachment #707291 - Flags: review?(catlee)
We're shutting down the nightly branch in favour of the v1-train branch, so no need to regenerate things for nightly I don't think.
I believe our toolchain regeneration docs/scripts point to https://github.com/mozilla-b2g/b2g-manifest nightly, which doesn't have the latest changes (e.g. git.m.o -> git.m.o/releases for mozillaorg). So comment 8 sounds like we have to change which branch of b2g-manifest we point at, and we have to regenerate our toolchains once that's done.
Attachment #707291 - Flags: review?(catlee) → review+
(In reply to John Ford [:jhford] from comment #8) > We're shutting down the nightly branch in favour of the v1-train branch, so > no need to regenerate things for nightly I don't think. does that include the nightly branch of b2g-manifests, or are you referring to the nightly branch of gaia?
(In reply to Chris AtLee [:catlee] from comment #10) > (In reply to John Ford [:jhford] from comment #8) > > We're shutting down the nightly branch in favour of the v1-train branch, so > > no need to regenerate things for nightly I don't think. > > does that include the nightly branch of b2g-manifests, or are you referring > to the nightly branch of gaia? My understanding until checking my email this morning was that we were definitely shutting both down. There are now questions about whether we're going to do this, so I'll post information here as I have it.
(In reply to John Ford [:jhford] from comment #12) > (In reply to Chris AtLee [:catlee] from comment #10) > > (In reply to John Ford [:jhford] from comment #8) > > > We're shutting down the nightly branch in favour of the v1-train branch, so > > > no need to regenerate things for nightly I don't think. > > > > does that include the nightly branch of b2g-manifests, or are you referring > > to the nightly branch of gaia? > > My understanding until checking my email this morning was that we were > definitely shutting both down. There are now questions about whether we're > going to do this, so I'll post information here as I have it. In that case, can you merge the default branch of b2g-manifests to nightly (or at least the relevant changes), so we can fix up the manifests on our side to match?
Assignee: nobody → catlee
Priority: -- → P2
Whiteboard: [b2g]
Looking at bug 834789, it looks like we're back to deleting the nightly branch of both gaia and b2g-manifest.
The most recent manifests have sane project names in xml: http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/2013-01-30-07/source_otoro_2013-01-30-07.xml http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/2013-01-30-07/source_unagi-stable_2013-01-30-07.xml http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/2013-01-30-07/source_unagi_2013-01-30-07.xml This should un-break partners. We also need to re-generate the toolchains against the appropriate branch of b2g-manifests to get the recent <remote /> tag changes. Right now it sounds like we're killing the nightly branch, which reverts bug 822970 and makes bug 820955 an issue again.
Depends on: 820955
(In reply to Aki Sasaki [:aki] from comment #15) > Right now it sounds like we're killing the nightly branch, which reverts bug > 822970 and makes bug 820955 an issue again. Can you please clarify what you mean by bug 820955 being an issue again?
Aiui, bug 822970 worked around bug 820955 by pointing at the nightly branch. The nightly branch is going away. Therefore the workaround for bug 820955 is going away.
(In reply to comment #17) > Aiui, bug 822970 worked around bug 820955 by pointing at the nightly branch. > The nightly branch is going away. > Therefore the workaround for bug 820955 is going away. I think I'm missing something here. What does the nightly branch track? Is that mozilla-central? Is that pointing to my github mirror? (I'm mostly trying to figure out if anything in the b2g builds could be using the github mirror...)
(In reply to :Ehsan Akhgari (Away 2/7-2/15) from comment #18) > I think I'm missing something here. What does the nightly branch track? Is > that mozilla-central? Is that pointing to my github mirror? (I'm mostly > trying to figure out if anything in the b2g builds could be using the github > mirror...) nightly branch is a branch in github. https://github.com/mozilla-b2g/b2g-manifest/tree/nightly https://github.com/mozilla-b2g/gaia/tree/nightly Once the b2g-manifest nightly branch goes away, we will be pointing at the 'master' branch in gecko.git. I think we on the releng side have resolved our questions here and have a path forward; do you have further questions?
(In reply to Aki Sasaki [:aki] from comment #19) > Once the b2g-manifest nightly branch goes away, we will be pointing at the > 'master' branch in gecko.git. do you mean the master branch of gecko.git or all branches in gecko.git?
(In reply to :Ehsan Akhgari (Away 2/7-2/15) from comment #18) > I think I'm missing something here. What does the nightly branch track? Is > that mozilla-central? Is that pointing to my github mirror? (I'm mostly > trying to figure out if anything in the b2g builds could be using the github > mirror...) Ehsan, the nightly branch was a stabilization branch. master was similar to mozilla-inbound and nightly was similar to mozilla-central. Things that landed on master had to pass a set of smoketests before they were uplifted to nightly.
(In reply to John Ford [:jhford] from comment #20) > (In reply to Aki Sasaki [:aki] from comment #19) > > Once the b2g-manifest nightly branch goes away, we will be pointing at the > > 'master' branch in gecko.git. > > do you mean the master branch of gecko.git or all branches in gecko.git? As I stated in comment 19, the master branch of gecko.git.
i read that as saying that a file checked into gecko.git was going to point master of gaia, on all branches of gecko.git, since we don't have an updating master branch of gecko.git. I do have a question for comment 17 and 19, what's the path forward -- are we going to resume updating gecko.git's master branch?
Aiui, that's part of the plan. IIRC: * resume updating gecko.git's master branch * point to master branch of b2g-manifests * regenerate toolchains against b2g-manifests, land on all branches * point to gaia-central (master branch hg instance) instead of gaia-nightly on m-c * potentially kill panda_gaia_central [on all branches?], since panda on m-c will already be building against gaia-central I *think* that's it.
> * resume updating gecko.git's master branch done > * point to master branch of b2g-manifests we'll use whatever branch is appropriate for the gecko branch we're making the toolchain for > * regenerate toolchains against b2g-manifests, land on all branches underway > * point to gaia-central (master branch hg instance) instead of gaia-nightly > on m-c done > * potentially kill panda_gaia_central [on all branches?], since panda on m-c > will already be building against gaia-central will happen on next reconfig anything else?
We don't have the changes landed to bake in commits in b2g-manifest yet, so the toolchain you're going to generate might have slightly different commits than the final ones we use for v1.0.0 and v1-train. I think you should wait until those land to generate the v1-train and v1.0.0 toolchains.
We should also verify that the toolchains get regenerated for the appropriate branches whenever one of the mozilla-owned repositories other than Gecko and Gaia change. An example would be a gonk-misc change, if the toolchains aren't regenerated when gonk-misc changes, the builds won't contain the fix. Since there are dedicated v1.0.0 and v1-train branches for these repositories now, any fixes landed to them should be considered important.
(In reply to John Ford [:jhford] from comment #26) > We don't have the changes landed to bake in commits in b2g-manifest yet, so > the toolchain you're going to generate might have slightly different commits > than the final ones we use for v1.0.0 and v1-train. I think you should wait > until those land to generate the v1-train and v1.0.0 toolchains. good to know - thanks. should we start on refreshing the snapshot used by mozilla-central? (In reply to John Ford [:jhford] from comment #27) > We should also verify that the toolchains get regenerated for the > appropriate branches whenever one of the mozilla-owned repositories other > than Gecko and Gaia change. An example would be a gonk-misc change, if the > toolchains aren't regenerated when gonk-misc changes, the builds won't > contain the fix. Since there are dedicated v1.0.0 and v1-train branches for > these repositories now, any fixes landed to them should be considered > important. our medium term plan is to do away with the snapshots. we'll use the sources.xml that's checked into gecko to figure out which version of the b2g repos to use. Now that bug 835933 is fixed, this is within our reach now. In this system, if you want to pick up a gonk-misc change, you would edit the appropriate sources.xml (e.g. https://hg.mozilla.org/mozilla-central/file/default/b2g/config/unagi/sources.xml) and bump the gonk-misc revision.
(In reply to Chris AtLee [:catlee] from comment #28) > good to know - thanks. should we start on refreshing the snapshot used by > mozilla-central? I don't have any plans to change b2g-manifest's master branch, so I don't see any reason to wait on mozilla-central > our medium term plan is to do away with the snapshots. we'll use the > sources.xml that's checked into gecko to figure out which version of the b2g > repos to use. Now that bug 835933 is fixed, this is within our reach now. Good to see the snapshots going away, but it doesn't sound like you're using Repo. Out of curiousity, why not? > In this system, if you want to pick up a gonk-misc change, you would edit > the appropriate sources.xml (e.g. > https://hg.mozilla.org/mozilla-central/file/default/b2g/config/unagi/sources. > xml) and bump the gonk-misc revision. And you'll have monitoring of branch landings for gonk-misc and friends so you guys know when to change that file?
(In reply to John Ford [:jhford] from comment #29) > (In reply to Chris AtLee [:catlee] from comment #28) > > good to know - thanks. should we start on refreshing the snapshot used by > > mozilla-central? > > I don't have any plans to change b2g-manifest's master branch, so I don't > see any reason to wait on mozilla-central ok, we'll get started on that > > our medium term plan is to do away with the snapshots. we'll use the > > sources.xml that's checked into gecko to figure out which version of the b2g > > repos to use. Now that bug 835933 is fixed, this is within our reach now. > > Good to see the snapshots going away, but it doesn't sound like you're using > Repo. Out of curiousity, why not? The primary reason is that AFAICT it forces you to clone all the repositories for each type of build you want to do. Ideally we'd like something that supports the notion of a local cache of the repos on the build slave (like hg-shared). > > In this system, if you want to pick up a gonk-misc change, you would edit > > the appropriate sources.xml (e.g. > > https://hg.mozilla.org/mozilla-central/file/default/b2g/config/unagi/sources. > > xml) and bump the gonk-misc revision. > > And you'll have monitoring of branch landings for gonk-misc and friends so > you guys know when to change that file? No, that will be up to whoever wants the change to update the manifest on mozilla-central.
(In reply to Chris AtLee [:catlee] from comment #30) > > Good to see the snapshots going away, but it doesn't sound like you're using > > Repo. Out of curiousity, why not? > > The primary reason is that AFAICT it forces you to clone all the > repositories for each type of build you want to do. Ideally we'd like > something that supports the notion of a local cache of the repos on the > build slave (like hg-shared). Repo already does this using .repo/projects > > And you'll have monitoring of branch landings for gonk-misc and friends so > > you guys know when to change that file? > > No, that will be up to whoever wants the change to update the manifest on > mozilla-central. Have you received confirmation that this is an acceptable solution?
I think the manifests are correct at this point.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: