about:buildconfig shows wrong source

RESOLVED FIXED in Thunderbird 64.0

Status

defect
RESOLVED FIXED
Last year
5 months ago

People

(Reporter: t.matsuu, Assigned: rjl)

Tracking

unspecified
Thunderbird 64.0

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments, 3 obsolete attachments)

Daily shows wrong source URL in about:buildconfig.
The URL is generated with base URL for comm-central and revision changeset ID of mozilla-central.

comm-central with comm-central changeset ID is expected (and additional mozilla-central changeset URL would be great).

about:buildconfig in Daily (Build ID: 20180508100117):
Source
Built from https://hg.mozilla.org/comm-central/rev/59005ba3cd3e7b3f9e8804bea881bf4c3a755d7c
Output .txt files at https://archive.mozilla.org/pub/thunderbird/nightly/ also have the same issue.
comm-central URL is generated with base URL for comm-central and revision changeset ID of mozilla-central.

In addition, mozilla source URL is from mozilla-unified, not mozilla-central.
Is this right repository?

https://archive.mozilla.org/pub/thunderbird/nightly/2018/06/2018-06-02-10-02-03-comm-central/thunderbird-62.0a1.en-US.win64.txt
20180602100203
https://hg.mozilla.org/comm-central/rev/8c926373039374cd1a47d92215e9efb4d5557983
https://hg.mozilla.org/mozilla-unified/rev/8c926373039374cd1a47d92215e9efb4d5557983
Jorg K,

If we need some work in comm-central side, I'll set this bug as blocker of bug 456360.
Otherwide I'll set this bug as dup. of bug 456360.

What do you think?
Flags: needinfo?(jorgk)
Sadly I'm not familiar with build issues at all. I'll NI Tom who can pass on the task to the build/release engineer we're in the process of hiring.
Flags: needinfo?(jorgk) → needinfo?(mozilla)
Flags: needinfo?(mozilla) → needinfo?(rob)
It looks like the .txt files come from toolkit/mozapps/installer/informulate.py (I think). There's some MOZ_SOURCE_CHANGESET in there that I think comes out of an environment variable of the same name. There's also MOZ_SOURCE_REPO that's pointing to comm-central, but the CHANGESET value is M-C's changeset. Those originate in comm/configure.in.
I need to play with this some more and figure out what to change, but those are the pieces at least.
The 'SourceRepository' and 'SourceStamp' entries in "application.ini" and "platform.ini" are also the same.

This might be related.

"platform.ini" should point to Mozilla-Central and
"application.ini" should point to Comm-Central.

However, both SourceRepository entries point to Comm-Central and both SourceStamp refer to Mozilla-Central
I think once bug 1491907 is fixed, this will fall into place.
Flags: needinfo?(rob)
Depends on: 1491907
Assignee: nobody → rob
about:buildconfig on Thunderbird shows an incorrect mercurial source URL
that's a comm- repo, but a mozilla- changeset revision.

For Taskcluster builds at least, MOZ_SOURCE_CHANGESET is set as an environment
variable by mozharness. During the build, build/variables.py writes out
source-repo.h with a few defines, including MOZ_SOURCE_URL that gets picked
up by buildconfig.html.
Attachment 9012763 [details] addresses most of the issues raised in this bug.

- about:buildconfig now shows the correct mercurial URL for the build. It's only showing the comm- information, not the mozilla- information. Getting the mozilla- changeset information would require making mozharness/variable.py smart enough to know that there are two repositories in play when doing a comm- build. source-repo.h would need to be updated accordingly. Last, buildconfig.html would need updating to display the additional information. 

- The target.txt file generated by my try build contains the correct information.

- application.ini now has the correct information.

- platform.ini still has the wrong SourceRepository. I suspect this is because with the current state of things, there's really no support for multiple repositories. Solving this piece would likely get us a step closer to having the mozilla- changeset information in about:buildconfig.
FWIW, I'm currently looking into having a single repo for comm so we'd not have to deal with the multi repository junk. But that's not a solution for anything that needs a quick resolution.
Comment on attachment 9012763 [details]
Bug 1460487 - Update mozharness to set MOZ_SOURCE_CHANGESET for comm builds.

Dustin J. Mitchell [:dustin] pronoun: he has approved the revision.
Attachment #9012763 - Flags: review+
Attachment #9012763 - Attachment description: Bug 1460487 - Update mozharness to set MOZ_SOURCE_CHANGESET for comm builds → Bug 1460487 - Update mozharness to set MOZ_SOURCE_CHANGESET for comm builds.
Pushed by mozilla@jorgk.com:
https://hg.mozilla.org/integration/autoland/rev/ba7f9ff7e4c5
Update mozharness to set MOZ_SOURCE_CHANGESET for comm builds. r=dustin
https://hg.mozilla.org/mozilla-central/rev/ba7f9ff7e4c5
Status: NEW → RESOLVED
Closed: 11 months ago
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 64.0
Rob, please manage backport requests here. m-esr60? m-b?
Flags: needinfo?(rob)
Jorg, this will need to be brought into esr60 and beta. 

Rico, I'll take another pass at this bug to see if I can clean the output up a bit more.
Flags: needinfo?(rob)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to Rob Lemley [:rjl] from comment #15)
> Jorg, this will need to be brought into esr60 and beta. 
OK, I'll uplift when ready. Please stick the approval requests onto the patch in due course.
The latest nightly build still have an issue.
Base URL is from comm-central and  revision changeset ID is from mozilla-unified.

Build ID: 20181002100152
Generated URL in about:buildconfig: https://hg.mozilla.org/comm-central/rev/7cda6e1eb528ba81b10548e858f91689b36dfe3b

Treeherder log: https://treeherder.mozilla.org/logviewer.html#?job_id=202834477&repo=comm-central
Component: General → Build Config
No longer depends on: 1491907
Referencing bug 1267270 for some historical context.
As it turns out, this bug is a very similar problem with having incorrect source repo information in Thunderbird builds. The solution that went into place is along the same lines as what I did with the first pass at this patch. Set MOZ_SOURCE_CHANGESET and MOZ_SOURCE_REPO to comm- values, overriding the mozilla- ones.

They also ran into the same problem we are having with platform.ini have the incorrect information from comm- instead of mozilla- after applying their fix. There's a bit of code in comm/mail/installer/Makefile.in that takes care of this in theory, but I suspect that in practice it does not seem to run for some reason.


Re comment 17: I just looked at the latest nightly builds at https://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-central/. The win32 and win64 builds have the incorrect source data, linux, linux64, and macosx64 have the correct information in their target.txt files. I think I know where to fix that.
I've thought about this over the weekend, and this is how I plan to proceed.

- Revert my earlier patch that was applied to m-c in comment 12. I just don't think that relying on environment variables is the most stable way of doing this,
- Expand/enhance build/variables.py
  - The source_repo_header function will be rewritten to not worry about environment variables if being built from a VCS checkout
  - Get the URL and current revision hash for files found at comm/mail ($MOZ_BUILD_APP). It's not an hg root, but by either using slightly different commands or perhaps with hglib we can get the values we need. Store these values as MOZ_SOURCE_*.
  - Get the same info from the repo at $topsrcdir. Store these values as PLATFORM_SOURCE_*. Some magic may need to happen to identify this repository as something other than "moz-unified",
  - Write out source-info.h with the MOZ_SOURCE and PLATFORM_SOURCE values.

MOZ_SOURCE and PLATFORM_SOURCE values will wind up being the same on a Firefox build (where MOZ_BUILD_APP is "browser" and is in the same repository as $topsrcdir). In cases where it's a different repository the MOZ_SOURCE variables will have the application values while PLATFORM_SOURCE is guaranteed to have Firefox values, or platform.ini values.

There will be some cleanup that needs to happen in some buried Makefile's under comm/ that run sed against platform.ini in an attempt to fix the current state of affairs. The code that generates platform.ini will need updating.

The target.txt build artifact is generated by informulate.py which will need some changes.

Sound reasonable?
MOZ_SOURCE_URL and MOZ_SOURCE_CHANGESET get used in the build process to create
application.ini and source-repo.h. It refers to the application's HG repo
("browser/" or "comm/mail")
The previous iteration did not address issues with the repo source and changeset
found in platform.ini,

variables.py will now output additional #defines PLATFORM_SOURCE_REPO,
PLATFORM_SOURCE_CHANGESET, and PLATFORM_SOURCE_URL which refer to the Mozilla
checkout at $topsrcdir.

I kept the same resolution order for determining the values.
  1) Check relevent environment variables. If set, use those values.
  2) Check for presence of sourcestamp.txt and use it's data if valid.
     (MOZ_SOURCE_* values only)
  3) Interrogate Mercurial for the values.

The MOZ_SOURCE_* will continue to refer to the repo for MOZ_BUILD_APP's dir.
In the case of Firefox ("browser/"), the repo is the same as
PLATFORM_SOURCE_REPO. In the case of Thunderbird ("comm/mail/"), the repository
at comm/ is found

Depends on D8151
Attachment #9012763 - Attachment is obsolete: true
Attachment #9015692 - Attachment is obsolete: true
about:buildconfig on Thunderbird 65.0a1
Patch on its way after more testing.
Attachment #9015691 - Attachment is obsolete: true
Sets MOZ_SOURCE_CHANGESET to be the comm-* changeset when building
Thunderbird on Windows. This mirrors a change at line 196 that sets the
same when building for Linux and OSX.
Ran successfully on try-comm-central producing a buildconfig page as shown.
Pushed by thunderbird@calypsoblue.org:
https://hg.mozilla.org/integration/autoland/rev/ae0a5c8948e3
Set MOZ_SOURCE_CHANGESET appropriately for TB. r=dustin
https://hg.mozilla.org/mozilla-central/rev/ae0a5c8948e3
Status: REOPENED → RESOLVED
Closed: 11 months ago8 months ago
Resolution: --- → FIXED
Duplicate of this bug: 1356115
Duplicate of this bug: 1511732
You need to log in before you can comment on or make changes to this bug.