Closed Bug 1042432 Opened 5 years ago Closed 5 years ago

make upload ignores UPLOAD_EXTRA_FILES

Categories

(Firefox Build System :: General, defect)

x86
All
defect
Not set

Tracking

(firefox32 fixed, firefox33+ fixed, firefox34 fixed)

RESOLVED FIXED
mozilla34
Tracking Status
firefox32 --- fixed
firefox33 + fixed
firefox34 --- fixed

People

(Reporter: massimo, Assigned: glandium)

References

Details

Attachments

(3 files)

We are building firefox 32.0b1 build1 and the release-mozilla-beta-updates step fails because there are no *info.txt files in: 

http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/32.0b1-candidates/build1/

the files are generated during the build process:

echo buildID=20140722093241 > linux_info.txt,

and they are in UPLOAD_EXTRA_FILES

UPLOAD_EXTRA_FILES="linux_info.txt update/linux-i686/en-US/firefox-31.0b8-32.0b1.partial.mar ..."

but they are not present in the output of make upload:

http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/32.0b1-candidates/build1/linux-i686/en-US/firefox-32.0b1.tar.bz2
http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/32.0b1-candidates/build1/update/linux-i686/en-US/firefox-32.0b1.complete.mar
http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/32.0b1-candidates/build1/linux-i686/xpi/en-US.xpi
http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/32.0b1-candidates/build1/linux-i686/en-US/firefox-32.0b1.tests.zip
http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/32.0b1-candidates/build1/linux-i686/en-US/firefox-32.0b1.crashreporter-symbols.zip
http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/32.0b1-candidates/build1/linux-i686/en-US/firefox-32.0b1.txt
http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/32.0b1-candidates/build1/linux-i686/en-US/firefox-32.0b1.json
http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/32.0b1-candidates/build1/jsshell-linux-i686.zip
http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/32.0b1-candidates/build1/host/bin/mar
http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/32.0b1-candidates/build1/host/bin/mbsdiff
http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/32.0b1-candidates/build1/linux-i686/en-US/firefox-32.0b1.checksums
http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/32.0b1-candidates/build1/linux-i686/en-US/firefox-32.0b1.checksums.asc
Uploading /builds/slave/rel-m-beta-lx_bld-000000000000/build/obj-firefox/dist/linux-i686/en-US/firefox-32.0b1.tar.bz2

The same applies for partial files too, they are generated but they are not uploaded.

full log here:
http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/32.0b1-candidates/build1/logs/release-mozilla-beta-linux_build-bm84-build1-build7.txt.gz
glandium points to bug 1005495 regressing this.
Blocks: 1005495
Attachment #8460696 - Flags: review?(mh+mozilla) → review+
Comment on attachment 8460696 [details] [diff] [review]
[m-b] Bandaid for beta - remove AC_SUBST from bug 1005495

http://hg.mozilla.org/releases/mozilla-beta/rev/6a87b4799b21
Attachment #8460696 - Attachment description: [m-b] Remove AC_SUBST from bug 1005495 → [m-b] Bandaid for beta - remove AC_SUBST from bug 1005495
[Tracking Requested - why for this release]: we should track a proper fix for this bug, to land on central and aurora this cycle. Beta 32 has a bandaid so is OK.
OS: Mac OS X → All
Nick, are you going to take care of a proper fix?
Flags: needinfo?(nthomas)
No, it would be best if glandium or another build system person did that.
Flags: needinfo?(nthomas)
Interestingly, I tried that at the time the bug was filed and that failed. I don't know what I was doing wrong, but this appears to work just fine: the sccache.log.gz is in the build directory after a try build.
Attachment #8469144 - Flags: review?(mshal)
Assignee: nobody → mh+mozilla
Status: NEW → ASSIGNED
Comment on attachment 8469144 [details] [diff] [review]
Change how UPLOAD_EXTRA_FILES is passed down from mozconfig.cache to the build system

Nick, would you mind testing this in real conditions on a staged release?
Attachment #8469144 - Flags: feedback?(nthomas)
Comment on attachment 8469144 [details] [diff] [review]
Change how UPLOAD_EXTRA_FILES is passed down from mozconfig.cache to the build system

This seems to add sccache.log.gz to UPLOAD_EXTRA_FILES multiple times, at least when going through client.mk. I think in part this is because there's no include guards on .mozconfig.mk, but I can't account for all the extra occurrences. I might just be testing it incorrectly - I'm just printing out $(UPLOAD_EXTRA_FILES) at the top of obj/Makefile and testing it with/without it set in the environment.

If it's not easy to only add it once, we may want to $(sort) UPLOAD_EXTRA_FILES when adding to UPLOAD_FILES to prune the duplicates.

nthomas, do you see sccache.log.gz getting uploaded multiple times with this patch? If it's just something on my end, I'll change to r+.
Attachment #8469144 - Flags: review?(mshal) → feedback+
Comment on attachment 8469670 [details] [diff] [review]
Change how UPLOAD_EXTRA_FILES is passed down from mozconfig.cache to the build system

Chatted with glandium in IRC - so the duplicates were coming from client.mk re-invoking itself and re-including .mozconfig.mk, but since we don't actually run 'make upload' through client.mk, that shouldn't matter. The new patch unexport's UPLOAD_EXTRA_FILES to match the environment that client.mk had before this patch.

nit: A comment in client.mk of why we're unexporting it will be helpful in the future.
Attachment #8469670 - Flags: review?(mshal) → review+
Attachment #8469144 - Flags: feedback?(nthomas)
Comment on attachment 8469670 [details] [diff] [review]
Change how UPLOAD_EXTRA_FILES is passed down from mozconfig.cache to the build system

Will run this in a staging release on the current tip of mozilla-beta.
Attachment #8469670 - Flags: feedback?(nthomas)
Comment on attachment 8469670 [details] [diff] [review]
Change how UPLOAD_EXTRA_FILES is passed down from mozconfig.cache to the build system

Linux64 en-US build uploaded the same set of files in staging as the production.
Attachment #8469670 - Flags: feedback?(nthomas) → feedback+
https://hg.mozilla.org/mozilla-central/rev/fab1f203cab0
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Whiteboard: [fixed-in-fx-team]
Target Milestone: --- → mozilla34
Blocks: 1054727
Mike, can we have an uplift for aurora? Thanks
Flags: needinfo?(mh+mozilla)
Comment on attachment 8469670 [details] [diff] [review]
Change how UPLOAD_EXTRA_FILES is passed down from mozconfig.cache to the build system

Approval Request Comment
[Feature/regressing bug #]: regression bug 1005495
[User impact if declined]: results of the build are not all uploaded for releases/betas
[Describe test coverage new/current, TBPL]: tested by nthomas on staging.
[Risks and why]: No risk. This doesn't touch anything that is really important to the build.
[String/UUID change made/needed]: None
Attachment #8469670 - Flags: approval-mozilla-aurora?
Flags: needinfo?(mh+mozilla)
Attachment #8469670 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
QA Whiteboard: [qa-]
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.