Closed Bug 1541419 Opened 8 months ago Closed 6 months ago

Write tests to ensure that engines use the correct domains in all regions/locales where they are present

Categories

(Firefox :: Search, task)

task
Not set
Points:
3

Tracking

()

RESOLVED FIXED
Firefox 68
Iteration:
68.4 - Apr 29 - May 12
Tracking Status
firefox68 --- fixed

People

(Reporter: standard8, Assigned: standard8)

References

Details

Attachments

(3 files)

We need to add some unit tests to ensure that we are using the correct domains for search engines in all region/locales.

This should extend the work done in bug 1541415.

Iteration: --- → 68.4 - Apr 29 - May 12
Assignee: nobody → standard8
Status: NEW → ASSIGNED
Depends on: 1549821
Pushed by mbanner@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3d13d2cb56d1
Correct the search form url for Ebay in 'be' locale. r=mikedeboer
https://hg.mozilla.org/integration/autoland/rev/e661822f6730
Adds test to ensure that search engines are using the correct domain in all regions/locales. r=daleharvey

Backed out frequent xpcshell failures e.g. test_duckduckgo.js

https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=windows%2C7%2Cshippable%2Copt%2Cxpcshell%2Ctests%2Ctest-windows7-32-shippable%2Fopt-xpcshell-e10s-2%2Cx%28x2%29&tochange=73254a69497b209b372ed209ce9d004080ea8052&fromchange=e661822f67309406c695209d4be605ff281a2777&group_state=expanded&selectedJob=245750510 - the failure rate is somewhere at <=50% per number of retriggers, and they're either memory errors or thread creation errors after tests related to a search engine.
Example of each:

backout: https://hg.mozilla.org/integration/autoland/rev/44db460adb2b732a8444e0f35c968bd828313e12

push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=windows%2C7%2Cshippable%2Copt%2Cxpcshell%2Ctests%2Ctest-windows7-32-shippable%2Fopt-xpcshell-e10s-2%2Cx%28x2%29&tochange=73254a69497b209b372ed209ce9d004080ea8052&fromchange=e661822f67309406c695209d4be605ff281a2777&group_state=expanded&selectedJob=245746598

e.g failure snippet:
14:14:34 INFO - TEST-PASS | toolkit/components/search/tests/xpcshell/searchconfigs/test_amazon.js | test_searchConfig_amazon - [test_searchConfig_amazon : 108] Should be available for region: "tk" locale: "hy-AM" - true == true
14:14:34 INFO - <<<<<<<
14:14:34 INFO - c:\users\task_1557492576\appdata\local\temp\xpc-plugins-5dvxji could not be cleaned up.
14:14:34 INFO - c:\users\task_1557492576\appdata\local\temp\xpc-plugins-9_ejbx could not be cleaned up.
14:14:34 INFO - Following exceptions were raised:
14:14:34 INFO - Traceback (most recent call last):
14:14:34 INFO - File "Z:\task_1557492576\build\tests\xpcshell\runxpcshelltests.py", line 185, in run
14:14:34 INFO - self.run_test()
14:14:34 INFO - File "Z:\task_1557492576\build\tests\xpcshell\runxpcshelltests.py", line 715, in run_test
14:14:34 INFO - process_output, _ = self.communicate(proc)
14:14:34 INFO - File "Z:\task_1557492576\build\tests\xpcshell\runxpcshelltests.py", line 249, in communicate
14:14:34 INFO - self.process_line(line)
14:14:34 INFO - File "Z:\task_1557492576\build\tests\xpcshell\runxpcshelltests.py", line 602, in process_line
14:14:34 INFO - self.report_message(line_object)
14:14:34 INFO - File "Z:\task_1557492576\build\tests\xpcshell\runxpcshelltests.py", line 572, in report_message
14:14:34 INFO - self.output_lines.append(message)
14:14:34 INFO - MemoryError
14:14:34 INFO - Traceback (most recent call last):
14:14:34 INFO - File "Z:\task_1557492576\build\tests\xpcshell\runxpcshelltests.py", line 1669, in <module>
14:14:34 INFO - main()
14:14:34 INFO - File "Z:\task_1557492576\build\tests\xpcshell\runxpcshelltests.py", line 1664, in main
14:14:34 INFO - if not xpcsh.runTests(options):
14:14:34 INFO - File "Z:\task_1557492576\build\tests\xpcshell\runxpcshelltests.py", line 1431, in runTests
14:14:34 INFO - mobileArgs, **kwargs)
14:14:34 INFO - File "Z:\task_1557492576\build\tests\xpcshell\runxpcshelltests.py", line 1627, in runTestList
14:14:34 INFO - raise exceptions[0]
14:14:34 INFO - MemoryError
WARNING | IO Completion Port failed to signal process shutdown
Parent process 5964 exited with children alive:
PIDS: 3560
Attempting to kill them, but no guarantee of success
14:17:41 ERROR - Return code: 1
14:17:41 ERROR - No tests run or test summary not found
14:17:41 INFO - TinderboxPrint: xpcshell-xpcshell<br/><em class="testfail">T-FAIL</em>

Flags: needinfo?(standard8)

I've been looking into this, and it seems the fundamental issue is that the harness can't cope with the amount of logging we're throwing at it.

I'm therefore working on some patches to only output the logging if we're in test debug mode, or if there's actual failures. These seem to help a lot.

I'm currently pondering between splitting the tests to run in 4 chunks per test, however that isn't really showing any gains at the moment, so I might stay with the structure as it is already.

Flags: needinfo?(standard8)

To follow up on my previous comment, it seems splitting the tests to run in 4 chunks does make it faster on Windows & Mac. It is also better than leaving them as one chunk, with some extra parallel running being handled pretty well on Windows & Mac.

This splits running of locales across 4 chunks, which can run in parallel better.

The chunks can be run individually with '--tag=searchconfig1' etc.

It also stops logging test output in the pass cases (unless we're in _testDebug=true mode). This makes less work on the python harness which was causing a bottleneck and slowing the tests down.

Depends on D30399

Pushed by mbanner@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/afbd9e47febe
Correct the search form url for Ebay in 'be' locale. r=mkaply,mikedeboer
https://hg.mozilla.org/integration/autoland/rev/26e0b9e416db
Adds test to ensure that search engines are using the correct domain in all regions/locales. r=daleharvey
https://hg.mozilla.org/integration/autoland/rev/0e0c4b2166dc
Split up and reduce test output of the xpcshell searchconfigs tests to improve test times. r=daleharvey
Status: ASSIGNED → RESOLVED
Closed: 6 months ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 68
You need to log in before you can comment on or make changes to this bug.