946058 - Run all tests against multilocale builds, ideally using non-English locales

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect, P2)

Description

[Richard Newman [:rnewman]]

Today Mike put out a fire which affected both Android Nightlies and B2G builds (Bug 945750).

The root cause was an error in Bug 943728, which resulted in multilocale builds that contained zero locales.

The symptom was (apparently; Mike, please correct me if I'm wrong) a failure in getSelectedLocale, which naturally resulted in a recent locale-related change being backed out, to no avail.

Meanwhile, tests were merrily passing on local single-locale builds and on Tinderbox.

I propose two things. Shoot down one or the other, or both if you feel so inclined.

1. We should be running tests on the builds we ship: multilocale builds.

2. We should be running tests using the locales that our users use: non-en_US. Pick a random locale each time? For some selected build, use a fixed non-en_US locale? Don't care. Just don't run 100% of jobs on en_US.

Comment 1

[[e-mail] (:bhearsum)]

Thanks for filing this Richard. I agree 100% that we should run tests against multilocale builds. There's a bunch of things that we would need to do to do this, including:
* Generate multilocale builds as part of all Android jobs on RelEng hardware. We currently do this for nightlies, but not for try or on-change jobs.
* Make whatever adjustments to the test runners that are needed. This might be nothing as long as we feed in the right binaries. This would be a separate bug, probably in ateam's court.
* Fix up/disable whatever new test failures we uncover -- I'm certain we'll find at least some new ones.
* Make some changes to our own automation scripts to send the right things to the right places/scripts.

(There's probably a few more things too, this is just off the top of my head.)

With that said, it should be noted that this would not _prevent_ us from publishing a bad build, merely let us catch failures like this (and disable updates) more quickly. We do not run any tests before publishing builds to Nightly users. Because Nightly is a less stable channel and users are more capable of fixing themselves up when they get stuck, we haven't prioritized things such as running basic or more in depth tests before publishing updates.

(In reply to Richard Newman [:rnewman] from comment #0)
> 2. We should be running tests using the locales that our users use:
> non-en_US. Pick a random locale each time? For some selected build, use a
> fixed non-en_US locale? Don't care. Just don't run 100% of jobs on en_US.

I think this is a cool idea but tough to make useful given how we analyze test results currently. It's very difficult for any test that doesn't run consistently to be considered useful, because it becomes very unclear what change actually broke somethnig when we see a failure. When Treeherder (the TBPL succesor) comes along, maybe we have better ways to do things like this.

Comment 2

[Justin Wood (:Callek)]

Does this bugs premise still hold value, or should it be wontfixed at this point?

Comment 3

[Richard Newman [:rnewman]]

So long as we ship multi-locale builds, we would benefit from running tests against them -- even a very basic smoketest that menus, page load, etc. work on a non-en_US locale.

Comment 4

[Mike Kaply [:mkaply]]

The builds broke over the weekend due to this bug.

I had built with try and everything looked good, but official builds were broken because of multilocale.

So yes, this would be really good.

Comment 5

[Aki Sasaki (not active)]

Android nightlies have moved out of Gecko; B2G is EOL; we may be moving to a langpack model.