Closed Bug 1233314 Opened 4 years ago Closed 4 years ago

beta repacks are failing trying to find checksums file

Categories

(Release Engineering :: Release Requests, defect)

defect
Not set

Tracking

(firefox44 fixed, firefox45 fixed, firefox46 fixed)

RESOLVED FIXED
Tracking Status
firefox44 --- fixed
firefox45 --- fixed
firefox46 --- fixed

People

(Reporter: jlund, Assigned: rail)

References

Details

Attachments

(1 file)

create-release-repacks.py are failing to find the checksums file for iterating over and submitting repack info to balrog blob

looks to be platform wide. interesting that not every chunk has failed.

code:

https://dxr.mozilla.org/build-central/source/tools/scripts/l10n/create-release-repacks.py?case=true&from=create-release-repacks.py#138

log:

Starting new HTTPS connection (1): aus4-admin.mozilla.org
/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/lib/python/vendor/requests-2.7.0/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/lib/python/vendor/requests-2.7.0/requests/packages/urllib3/connection.py:251: SecurityWarning: Certificate has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SecurityWarning
The following tracebacks were detected during repacks:
as:
Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/scripts/l10n/create-release-repacks.py", line 138, in createRepacks
    checksums = parseChecksumsFile(open(checksums_file).read())
IOError: [Errno 2] No such file or directory: '/builds/slave/rel-m-beta-l64_rpk_1-000000000/mozilla-beta/obj-l10n/browser/locales/dist/linux-x86_64/as/firefox-44.0b1.checksums'


Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/scripts/l10n/create-release-repacks.py", line 386, in <module>
    bucket_prefix=branchConfig['bucket_prefix'],
  File "/builds/slave/rel-m-beta-l64_rpk_1-000000000/scripts/scripts/l10n/create-release-repacks.py", line 183, in createRepacks
    "Failed locales: %s" % " ".join([x for x, _ in failed]))
On linux64 it's 'as' in 1/10, 'mk' in 7/10, and 'ta' in 9/10. There's unexpected output like:

command: START
command: make --no-print-directory echo-variable-CHECKSUM_FILE AB_CD=ta
command: cwd: /builds/slave/rel-m-beta-l64_rpk_9-000000000/mozilla-beta/obj-l10n/browser/locales
command: env: {'MOZ_MAKE_COMPLETE_MAR': '1', 'AB_CD': 'ta', ...
command: END (0.25s elapsed)

command: output:
Missing searchplugin: creativecommons.xml
'../../dist/linux-x86_64/ta//firefox-44.0b1.checksums'

Traceback (most recent call last):
  File "/builds/slave/rel-m-beta-l64_rpk_9-000000000/scripts/scripts/l10n/create-release-repacks.py", line 138, in createRepacks
    checksums = parseChecksumsFile(open(checksums_file).read())
IOError: [Errno 2] No such file or directory: '/builds/slave/rel-m-beta-l64_rpk_9-000000000/mozilla-beta/obj-l10n/browser/locales/dist/linux-x86_64/ta/firefox-44.0b1.checksums'

'as' also complains about creativecommons.xml, and mk has

Missing searchplugin: answers.xml
Missing searchplugin: creativecommons.xml

logs:
http://archive.mozilla.org/pub/firefox/candidates/44.0b1-candidates/build1/logs/release-mozilla-beta-linux64_repack_1-bm77-build1-build9.txt.gz
http://archive.mozilla.org/pub/firefox/candidates/44.0b1-candidates/build1/logs/release-mozilla-beta-linux64_repack_7-bm77-build1-build8.txt.gz
http://archive.mozilla.org/pub/firefox/candidates/44.0b1-candidates/build1/logs/release-mozilla-beta-linux64_repack_9-bm91-build1-build12.txt.gz

The other platforms are failing too.

None of the locales have changed revision between 43.0b9 and 44.0b1, but
  hg diff -r FIREFOX_43_0b9_RELEASE:FIREFOX_44_0b1_RELEASE browser/locales/en-US/searchplugins/
indicates that answers.xml and creativecommons.xml were removed, so perhaps they should have.
Flags: needinfo?(l10n)
The build system was supposed to survive that
https://bugzilla.mozilla.org/show_bug.cgi?id=1215413#c2
Flags: needinfo?(l10n)
The current theory is that the code 
  http://hg.mozilla.org/build/tools/file/default/lib/python/build/l10n.py#l235
is expecting to parse:
  '../../dist/linux-x86_64/ta//firefox-44.0b1.checksums'
and gets this instead
  Missing searchplugin: creativecommons.xml
  '../../dist/linux-x86_64/ta//firefox-44.0b1.checksums'

We fail at http://hg.mozilla.org/build/tools/file/default/scripts/l10n/create-release-repacks.py#l138 when checksums_file is bogus.

The only thing that's failing is Balrog submission, so in theory I could look up all the hashes and submit them. In practise it would be easier to fix the l10n repos, move the FIREFOX_44_0b1_{RELEASE,BUILD1} tags, and use the standalone_repack builder to rerun them.
All l10n repos have already been fixed, e.g. mk in bug 1205544
https://bugzilla.mozilla.org/showdependencytree.cgi?id=1068456&hide_resolved=0

The problem is if the locale is inactive, there's no update sign-off on beta or release, and there's not much that we can do.
Sadly I'm not of much use right now, my loaner laptop doesn't like me programming.

https://dxr.mozilla.org/mozilla-central/source/browser/locales/Makefile.in#75 looks like the culprit, and I wonder if changing the := to a = would fix it?

Practically speaking, I'm not too opposed to ship beta 1 without these locales. mk has 92 users on beta, which sounds like a good reason to serve something in b2. But I'm not sure we need to block.
(In reply to Axel Hecht [:Pike] from comment #5)
> Sadly I'm not of much use right now, my loaner laptop doesn't like me
> programming.
> 
> https://dxr.mozilla.org/mozilla-central/source/browser/locales/Makefile.
> in#75 looks like the culprit, and I wonder if changing the := to a = would
> fix it?

It's unlikely to fix it. I'm pretty sure the code in rules.mk will effectively expand SEARCHPLUGINS anyways.

However, changing $(info) with $(warning) might fix it: $(info) is sent to stdout, while $(warning) is sent to stderr. Assuming get_output doesn't slurp stderr...
(In reply to Mike Hommey [:glandium] from comment #6)
> Assuming get_output doesn't slurp stderr...

it shouldn't:
https://dxr.mozilla.org/build-central/source/tools/lib/python/build/l10n.py#235 doesn't pass include_stderr, which is False by default in https://dxr.mozilla.org/build-central/source/tools/lib/python/util/commands.py#148
poor man's test:

$ cat Makefile
all:
        $(info this is info)
        $(warning this is warninig)

$ make 2>/dev/null
this is info
make: 'all' is up to date.

$ make
this is info
Makefile:2: this is warninig
make: 'all' is up to date.
Attachment #8699490 - Flags: review?(gps)
Comment on attachment 8699490 [details] [diff] [review]
info_to_warning.diff

From bug 1073212 I don't see a particular reason why we would prefer $(info) over $(warning), so r+. Would be nice not to have to rely on random make commands someday :/
Attachment #8699490 - Flags: review?(gps) → review+
Comment on attachment 8699490 [details] [diff] [review]
info_to_warning.diff

This should not affect any non l10n builds
Attachment #8699490 - Flags: approval-mozilla-beta?
Attachment #8699490 - Flags: approval-mozilla-aurora?
Comment on attachment 8699490 [details] [diff] [review]
info_to_warning.diff

We need this to get 44.0b1 out the door. A+
Attachment #8699490 - Flags: approval-mozilla-beta?
Attachment #8699490 - Flags: approval-mozilla-beta+
Attachment #8699490 - Flags: approval-mozilla-aurora?
Attachment #8699490 - Flags: approval-mozilla-aurora+
https://hg.mozilla.org/mozilla-central/rev/cc002565421a
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Assignee: nobody → rail
See Also: → 1242771
You need to log in before you can comment on or make changes to this bug.