1242035 - No test packages (except jsshell) get uploaded for release builds

Status: RESOLVED WONTFIX

(Firefox Build System :: General, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So firefox-ui-tests moved into mozilla-central now and will ride the trains. As result the github repository will go away at some point in the future. 

That means for any kind of release testing we would need the common.tests.zip file beside release builds (this included also candidates and is both for beta and release). Right now this test file is not present on the FTP server for the en_US build.

I talked with Ben on IRC... so here our conversation:

<whimboo> so its just about http://ftp.mozilla.org/pub/firefox/candidates/44.0b9-candidates/build1/
<bhearsum> your goal is to get common.tests.zip uploaded as part of en-US builds?
<whimboo> and that its missing the test packages
<whimboo> yes
<bhearsum> okay
<bhearsum> so you need two things for that to happen
<whimboo> i can file a release automation bug then
<bhearsum> 1) the package must be generated
<bhearsum> you probably want to do that as part of "make package-tests"
<bhearsum> 2) the package must be in UPLOAD_FILES
<bhearsum> all of this can (and probably should) be done in the build system

Given that this is only a problem for release automation, there must be a specific if condition or such which excludes test packages right now.

If we cannot get this fixed in time we will no longer be able to run any kind of tests on final release builds with mozmill ci because we don't have access to the tests. We would only run tests for the tinderbox builds on the beta/release branches.

Maybe a workaround for now would be to somehow retrieve the tests by using the parent changeset of the release tags, and retrieve the test package via the tinderbox build. But this is not really something I would prefer.

Comment 1

[[e-mail] (:bhearsum)]

This is almost certainly an issue in the build system...most likely there's a difference in how some variables are set depending on whether MOZ_PKG_PRETTYNAMES=1 is set or not. Or maybe one of the MOZ_AUTOMATION_* vars is different? Those should be printed at the top of the build logs.

For reference, here's the nightly and release mozconfigs:
https://dxr.mozilla.org/mozilla-central/source/browser/config/mozconfigs/linux64/nightly
https://dxr.mozilla.org/mozilla-central/source/browser/config/mozconfigs/linux64/release

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So I wonder if this behavior is wanted. Maybe Rail or Nick can shed some light in here? If that would be the case and this bug ends up as wontfix, the workaround as explained above seems to work great for me locally. So it's not a definitive blocker for us.

Comment 3

[Rail Aliiev [:rail]]

Release builds are a bit different and they will be going away "very soon" (TM) :)

They don't upload test packages, because we don't schedule any tests after them.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Rail and I discussed that on IRC and with release promotion no extra builds will be generated anymore. That means the release will be a per checkin generated build which will have test packages available. 

Lets close this bug as wontfix.