Closed Bug 1026480 Opened 10 years ago Closed 10 years ago

Missing hsb and dsb single-locale Aurora Fennec builds

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: gueroJeff, Unassigned)

References

Details

After landing bug 1018222, we're should have single-locale Aurora builds for both the hsb and dsb locales in http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-aurora-android-l10n/ but none have kicked off since landing the bug. Each locale has mobile resources in their repositories.
The automation for including a locale in the multilocale build should be tracking maemo-locales, but single-locale builds should be tracking all-locales.
Whoops, good point...looking into this further.
Found this in a job that tried to build dsb:
02:34:02    ERROR - Can't determine the upload url for dsb!
02:34:02    ERROR - You either need to run --upload-repacks before --create-nightly-snippets, or specify the 'snippet_base_url' in self.config!

Same thing for hsb.

I'm out of my league now...Aki?
Flags: needinfo?(aki)
http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_1-unknown-bm71-build1-build1.txt.gz
bld-linux64-spot-126

02:29:37     INFO -  (  echo 'export ANT_HOME=/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/apache-ant';  echo 'export PATH=/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/gcc/bin:/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/apache-ant/bin:/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/'; ) > obj-l10n/.mozconfig.mk
02:29:37    ERROR -  client.mk:316: *** Could not find autoconf 2.13.  Stop.
02:29:37    ERROR - Return code: 2
02:29:37    FATAL - Configure failed!
02:29:37    FATAL - Running post_fatal callback...
02:29:37    FATAL - Exiting -1

http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_2-unknown-bm82-build1-build6.txt.gz
b-linux64-hp-0033

02:20:44    ERROR - caught OS error 2: No such file or directory while running ['make', '-f', 'client.mk', 'configure']
02:20:44    FATAL - Configure failed!
02:20:44    FATAL - Running post_fatal callback...
02:20:44    FATAL - Exiting -1

http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_3-unknown-bm71-build1-build1.txt.gz
bld-linux64-spot-1010

02:25:43     INFO -  (  echo 'export ANT_HOME=/builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/apache-ant';  echo 'export PATH=/builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/gcc/bin:/builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/apache-ant/bin:/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/'; ) > obj-l10n/.mozconfig.mk
02:25:45    ERROR -  client.mk:316: *** Could not find autoconf 2.13.  Stop.
02:25:45    ERROR - Return code: 2
02:25:45    FATAL - Configure failed!
02:25:45    FATAL - Running post_fatal callback...
02:25:45    FATAL - Exiting -1

http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_4-unknown-bm71-build1-build5.txt.gz
bld-linux64-spot-127

02:29:15     INFO -  (  echo 'export ANT_HOME=/builds/slave/m-aurora-and-l10n_4-0000000000/build/mozilla-aurora/apache-ant';  echo 'export PATH=/builds/slave/m-aurora-and-l10n_4-0000000000/build/mozilla-aurora/gcc/bin:/builds/slave/m-aurora-and-l10n_4-0000000000/build/mozilla-aurora/apache-ant/bin:/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/'; ) > obj-l10n/.mozconfig.mk
02:29:15    ERROR -  client.mk:316: *** Could not find autoconf 2.13.  Stop.
02:29:15    ERROR - Return code: 2
02:29:15    FATAL - Configure failed!
02:29:15    FATAL - Running post_fatal callback...
02:29:15    FATAL - Exiting -1

http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_5-unknown-bm73-build1-build3.txt.gz
bld-linux64-spot-443
02:29:54     INFO -  (  echo 'export ANT_HOME=/builds/slave/m-aurora-and-l10n_5-0000000000/build/mozilla-aurora/apache-ant';  echo 'export PATH=/builds/slave/m-aurora-and-l10n_5-0000000000/build/mozilla-aurora/gcc/bin:/builds/slave/m-aurora-and-l10n_5-0000000000/build/mozilla-aurora/apache-ant/bin:/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/'; ) > obj-l10n/.mozconfig.mk
02:29:54    ERROR -  client.mk:316: *** Could not find autoconf 2.13.  Stop.
02:29:54    ERROR - Return code: 2
02:29:54    FATAL - Configure failed!
02:29:54    FATAL - Running post_fatal callback...
02:29:54    FATAL - Exiting -1
Flags: needinfo?(aki)
It sounds like there's at least two different problems here. The logs that Aki points at show complete failure of repacks altogether, starting with this morning's builds.

The error I found was from http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_1-unknown-bm85-build1-build2.txt.gz, which didn't hit the autoconf error.
dsb_failure: "dsb failed in make installers-dsb!  Can't create a snippet for dsb without an upload url."

The installer failed, therefore there was no upload, therefore there was no upload url.

Search for | ERROR -|.

dsb core dumped:
02:30:00     INFO -  /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/android-sdk-linux/build-tools/19.0.2/aapt: /lib/libz.so.1: no version information available (required by /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/android-sdk-linux/build-tools/19.0.2/aapt)
02:30:01     INFO -  make[2]: *** [.aapt.nodeps] Segmentation fault (core dumped)
02:30:01     INFO -  make[2]: *** Deleting file `.aapt.nodeps'
02:30:01     INFO -  make[2]: Leaving directory `/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/obj-l10n/mobile/android/base'
02:30:01    ERROR -  make[1]: *** [repackage-zip] Error 2
02:30:01     INFO -  make[1]: Leaving directory `/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/obj-l10n/mobile/android/locales'
02:30:01     INFO -  make: *** [repackage-zip-dsb] Error 2
02:30:02    ERROR - Return code: 2

af:
02:27:00     INFO -  /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/obj-l10n/_virtualenv/bin/python -m mozbuild.action.preprocessor  -I /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/af/mobile/profile/bookmarks.inc -DAB_CD=af /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/mobile/locales/generic/profile/bookmarks.json.in -o .deps/generated_af/bookmarks.json
02:27:01    ERROR -  Traceback (most recent call last):
02:27:01     INFO -    File "/tools/python27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
02:27:01     INFO -      "__main__", fname, loader, pkg_name)
02:27:01     INFO -    File "/tools/python27/lib/python2.7/runpy.py", line 72, in _run_code
02:27:01     INFO -      exec code in run_globals
02:27:01     INFO -    File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/action/preprocessor.py", line 16, in <module>
02:27:01     INFO -      main(sys.argv[1:])
02:27:01     INFO -    File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/action/preprocessor.py", line 12, in main
02:27:01     INFO -      pp.handleCommandLine(args, True)
02:27:01     INFO -    File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/preprocessor.py", line 472, in handleCommandLine
02:27:01     INFO -      self.processFile(input=input, output=out)
02:27:01     INFO -    File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/preprocessor.py", line 380, in processFile
02:27:01     INFO -      self.do_include(input, False)
02:27:01     INFO -    File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/preprocessor.py", line 774, in do_include
02:27:01     INFO -      self.handleLine(l)
02:27:01     INFO -    File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/preprocessor.py", line 552, in handleLine
02:27:01     INFO -      self.write(aLine)
02:27:01     INFO -    File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/preprocessor.py", line 421, in write
02:27:01     INFO -      filteredLine = self.applyFilters(aLine)
02:27:01     INFO -    File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/preprocessor.py", line 403, in applyFilters
02:27:01     INFO -      aLine = f[1](aLine)
02:27:01     INFO -    File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/preprocessor.py", line 731, in filter_substitution
02:27:01     INFO -      return self.varsubst.sub(repl, aLine)
02:27:01     INFO -    File "/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/python/mozbuild/mozbuild/preprocessor.py", line 729, in repl
02:27:01     INFO -      raise Preprocessor.Error(self, 'UNDEFINED_VAR', varname)
02:27:01     INFO -  mozbuild.preprocessor.Error: ('/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/mobile/locales/generic/profile/bookmarks.json.in', 6, 'UNDEFINED_VAR', 'bookmarks_aboutBrowser')
02:27:01    ERROR -  make[3]: *** [.deps/generated_af/bookmarks.json] Error 255
02:27:01     INFO -  make[3]: *** Deleting file `.deps/generated_af/bookmarks.json'
02:27:01     INFO -  make[3]: Leaving directory `/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/obj-l10n/mobile/locales'
02:27:01    ERROR -  make[2]: *** [libs-af] Error 2
02:27:01     INFO -  make[2]: Leaving directory `/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/obj-l10n/mobile/locales'
02:27:01    ERROR -  make[1]: *** [libs-af] Error 2
02:27:01     INFO -  make[1]: Leaving directory `/builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/obj-l10n/mobile/android/locales'
02:27:01     INFO -  make: *** [repackage-zip-af] Error 2
So, it sounds like we're missing a package for autoconf-2.13, and also missing a packaging that something in the l10n logic depends on? Not sure exactly, and I don't see anything that's changed around the installed packages recently...
Possibly of note: the log I linked to shows that the repack was done on a machine that was re-imaged recently. Aki's logs are a mix of recently reimaged machines, and spot instances (which re-image at least once a day).
This might be the root error in Aki's logs:
02:25:43     INFO -  client.mk:213: obj-l10n/.mozconfig.mk: No such file or directory
I can't find any relevant changes to anything in buildbot-configs, buildbotcustom, mozharness, or even mozilla-aurora...
Summary: Missing hsb and dsb single-locale Aurora Fennec builds → Missing hsb and dsb single-locale Aurora Fennec builds - all single locale repacks possibly broken
Looking further it appears that Aki's log (http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_3-unknown-bm71-build1-build1.txt.gz) never installs mock packages, even though they're in the config. My log log (http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_1-unknown-bm85-build1-build2.txt.gz) does.

Aki's:
02:24:48     INFO - #####
02:24:48     INFO - ##### Running setup step.
02:24:48     INFO - #####
02:24:48     INFO - Running main action method: setup
02:24:48     INFO - Copying /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/mobile/android/config/mozconfigs/android/l10n-nightly to /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/.mozconfig
02:24:48     INFO - Running command: ['cat', '/builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/.mozconfig']
02:24:48     INFO - Copy/paste: cat /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/.mozconfig
02:24:48     INFO -  . "$topsrcdir/mobile/android/config/mozconfigs/common"
02:24:48     INFO -  # L10n

Mine:
02:16:27     INFO - #####
02:16:27     INFO - ##### Running setup step.
02:16:27     INFO - #####
02:16:27     INFO - Running main action method: setup
02:16:27     INFO - Copying /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/mobile/android/config/mozconfigs/android/l10n-nightly to /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/.mozconfig
02:16:27     INFO - Getting output from command: ['mock_mozilla', '-r', 'mozilla-centos6-x86_64', '--print-root-path']
02:16:27     INFO - Copy/paste: mock_mozilla -r mozilla-centos6-x86_64 --print-root-path
02:16:28     INFO - Reading from file tmpfile_stdout
02:16:28     INFO - Output received:
02:16:28     INFO -  /builds/mock_mozilla/mozilla-centos6-x86_64/root/
02:16:28     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stderr',), kwargs: {}, attempt #1
02:16:28     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stdout',), kwargs: {}, attempt #1
02:16:28     INFO - no previous package list found
02:16:28     INFO - Running command: ['mock_mozilla', '-r', 'mozilla-centos6-x86_64', '--init']
02:16:28     INFO - Copy/paste: mock_mozilla -r mozilla-centos6-x86_64 --init


I think the overall failure of repacks is more fallout from bug 1025842. So, that needs to be fixed before we can confirm any sort of fix for hsb and dsb being missing.

However, I think the issues with dsb and hsb are with the locales themselves. For hsb I found this:
02:25:58    ERROR -  /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/obj-l10n/mobile/android/base/res/values/strings.xml:976: error: Error parsing XML: undefined entity
02:25:58    ERROR -  /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout/bookmark_edit.xml:12: error: Error: No resource found that matches the given name (at 'hint' with value '@string/bookmark_edit_name').
02:25:58    ERROR -  /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout/bookmark_edit.xml:19: error: Error: No resource found that matches the given name (at 'hint' with value '@string/bookmark_edit_location').
02:25:58    ERROR -  /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout/bookmark_edit.xml:27: error: Error: No resource found that matches the given name (at 'hint' with value '@string/bookmark_edit_keyword').
02:25:58    ERROR -  /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout/browser_toolbar.xml:56: error: Error: No resource found that matches the given name (at 'contentDescription' with value '@string/menu').
02:25:58    ERROR -  /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout/browser_toolbar.xml:100: error: Error: No resource found that matches the given name (at 'contentDescription' with value '@string/edit_mode_cancel').
02:25:58    ERROR -  /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout-large-v11/browser_toolbar.xml:53: error: Error: No resource found that matches the given name (at 'contentDescription' with value '@string/back').
02:25:58    ERROR -  /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout-large-v11/browser_toolbar.xml:87: error: Error: No resource found that matches the given name (at 'contentDescription' with value '@string/menu').
02:25:58    ERROR -  /builds/slave/m-aurora-and-l10n_3-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout-large-v11/browser_toolbar.xml:112: error: Error: No resource found that matches the given name (at 'contentDescription' with value '@string/edit_mode_cancel').
(and much more - look at http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_3-unknown-bm86-build1-build2.txt.gz for the full list)

And for dsb I found similar:
02:31:22    ERROR -  /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/obj-l10n/mobile/android/base/res/values/strings.xml:976: error: Error parsing XML: undefined entity
02:31:22    ERROR -  /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout/bookmark_edit.xml:12: error: Error: No resource found that matches the given name (at 'hint' with value '@string/bookmark_edit_name').
02:31:22    ERROR -  /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout/bookmark_edit.xml:19: error: Error: No resource found that matches the given name (at 'hint' with value '@string/bookmark_edit_location').
02:31:22    ERROR -  /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout/bookmark_edit.xml:27: error: Error: No resource found that matches the given name (at 'hint' with value '@string/bookmark_edit_keyword').
02:31:22    ERROR -  /builds/slave/m-aurora-and-l10n_1-0000000000/build/mozilla-aurora/mobile/android/base/resources/layout/browser_toolbar.xml:56: error: Error: No resource found that matches the given name (at 'contentDescription' with value '@string/menu').
(full log at http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-l10n/mozilla-aurora-android-l10n_1-unknown-bm84-build1-build5.txt.gz)

Jeff, any idea about the two errors above?
Depends on: 1025842
Flags: needinfo?(jbeatty)
Summary: Missing hsb and dsb single-locale Aurora Fennec builds - all single locale repacks possibly broken → Missing hsb and dsb single-locale Aurora Fennec builds
Rail tells me that bug 1025842 doesn't impact in-house machines like the ones these builds run on, so it's probably unrelated. I'm filing a new bug for the general issue with these repacks.
No longer depends on: 1025842
Oh, that above reads dramatically. What can I do as translator? According to the sign-off pages for hsb and dsb there are no errors - two files are still missing but they are not on Pootle yet.
@Michael
Unfortunately nothing you can do, we just need to figure out the issue.

Dashboard reports only missing productization files for hsb/dsb (Jeff, do we have bugs for those?), no errors. The strings reported as missing are definitely there, and they look OK, not sure if the build system gets mad at the missing files.
I tried a dsb repack locally, and I get an apk (can't test it, I don't have an Android device anymore)

It fails with

cp: /src/central/releases/mozilla-aurora/obj-repack-android/mobile/android/base/fennec_ids.txt: No such file or directory

after successfully verifying the apk. No idea what that means.

IOW, I don't see anything on the localization side being wrong right now.

And then I looked at ftp (should have done that before), and today dsb and hsb did in fact get apks generated, http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-aurora-android-l10n/ has http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-aurora-android-l10n/fennec-32.0a2.hsb.android-arm.apk and http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-aurora-android-l10n/fennec-32.0a2.dsb.android-arm.apk
Flags: needinfo?(jbeatty)
I'm resolving this FIXED, as the infra backout seems to have helped, it'd be good to still keep watching the builds if these are just random outages. At least I don't know if we understand what broke the builds when.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
@Comment 17: The file fennec_ids.txt does not exist on Pootle, just as the files chrome/region.properties and searchplugins/list.txt.

Unfortunately I do not have an Android device, either. I tried to test the Upper Sorbian apk in the emulator Jar of Beans, installation succeeded but when starting Aurora I get the message "Aurora has been stopped". I have a very old computer, 1.024 MB RAM only. Maybe that's the reason for the stop.
I don't understand that this bug has been marked as "resolved" - there are no builds for dsb and hsb now. Still some days ago there were Aurora builds at least. I don't understand, either that now beta builds are created for dsb and hsb. Please don't say it is because of the three character language code. I saw that Maithili has a beta build - it has a three character language code as well - mai.
Sorry, it must be "**no** beta builds" instead of "**now** beta builds".
The aurora builds are busted in general, that's bug 1062880. doesn't have anything to do with what was going on in this bug.
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.