1541419 - Write tests to ensure that engines use the correct domains in all regions/locales where they are present

Status: RESOLVED FIXED

(Firefox :: Search, task)

Description

[Mark Banner (:standard8)]

We need to add some unit tests to ensure that we are using the correct domains for search engines in all region/locales.

This should extend the work done in bug 1541415.

Comment 1

[Mark Banner (:standard8)]

Attachment: Bug 1541419 - Correct the search form url for Ebay in 'be' locale. r?mkaply

Comment 2

[Mark Banner (:standard8)]

Attachment: Bug 1541419 - Adds test to ensure that search engines are using the correct domain in all regions/locales. r?daleharvey

Depends on D30398

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3d13d2cb56d1
Correct the search form url for Ebay in 'be' locale. r=mikedeboer
https://hg.mozilla.org/integration/autoland/rev/e661822f6730
Adds test to ensure that search engines are using the correct domain in all regions/locales. r=daleharvey

Comment 4

[Natalia Csoregi [:nataliaCs]]

Backed out frequent xpcshell failures e.g. test_duckduckgo.js

https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=windows%2C7%2Cshippable%2Copt%2Cxpcshell%2Ctests%2Ctest-windows7-32-shippable%2Fopt-xpcshell-e10s-2%2Cx%28x2%29&tochange=73254a69497b209b372ed209ce9d004080ea8052&fromchange=e661822f67309406c695209d4be605ff281a2777&group_state=expanded&selectedJob=245750510 - the failure rate is somewhere at <=50% per number of retriggers, and they're either memory errors or thread creation errors after tests related to a search engine.
Example of each:

• https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=245731070&repo=autoland&lineNumber=6527
• https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=245750511&repo=autoland&lineNumber=247659

backout: https://hg.mozilla.org/integration/autoland/rev/44db460adb2b732a8444e0f35c968bd828313e12

push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=windows%2C7%2Cshippable%2Copt%2Cxpcshell%2Ctests%2Ctest-windows7-32-shippable%2Fopt-xpcshell-e10s-2%2Cx%28x2%29&tochange=73254a69497b209b372ed209ce9d004080ea8052&fromchange=e661822f67309406c695209d4be605ff281a2777&group_state=expanded&selectedJob=245746598

e.g failure snippet:
14:14:34 INFO - TEST-PASS | toolkit/components/search/tests/xpcshell/searchconfigs/test_amazon.js | test_searchConfig_amazon - [test_searchConfig_amazon : 108] Should be available for region: "tk" locale: "hy-AM" - true == true
14:14:34 INFO - <<<<<<<
14:14:34 INFO - c:\users\task_1557492576\appdata\local\temp\xpc-plugins-5dvxji could not be cleaned up.
14:14:34 INFO - c:\users\task_1557492576\appdata\local\temp\xpc-plugins-9_ejbx could not be cleaned up.
14:14:34 INFO - Following exceptions were raised:
14:14:34 INFO - Traceback (most recent call last):
14:14:34 INFO - File "Z:\task_1557492576\build\tests\xpcshell\runxpcshelltests.py", line 185, in run
14:14:34 INFO - self.run_test()
14:14:34 INFO - File "Z:\task_1557492576\build\tests\xpcshell\runxpcshelltests.py", line 715, in run_test
14:14:34 INFO - process_output, _ = self.communicate(proc)
14:14:34 INFO - File "Z:\task_1557492576\build\tests\xpcshell\runxpcshelltests.py", line 249, in communicate
14:14:34 INFO - self.process_line(line)
14:14:34 INFO - File "Z:\task_1557492576\build\tests\xpcshell\runxpcshelltests.py", line 602, in process_line
14:14:34 INFO - self.report_message(line_object)
14:14:34 INFO - File "Z:\task_1557492576\build\tests\xpcshell\runxpcshelltests.py", line 572, in report_message
14:14:34 INFO - self.output_lines.append(message)
14:14:34 INFO - MemoryError
14:14:34 INFO - Traceback (most recent call last):
14:14:34 INFO - File "Z:\task_1557492576\build\tests\xpcshell\runxpcshelltests.py", line 1669, in <module>
14:14:34 INFO - main()
14:14:34 INFO - File "Z:\task_1557492576\build\tests\xpcshell\runxpcshelltests.py", line 1664, in main
14:14:34 INFO - if not xpcsh.runTests(options):
14:14:34 INFO - File "Z:\task_1557492576\build\tests\xpcshell\runxpcshelltests.py", line 1431, in runTests
14:14:34 INFO - mobileArgs, **kwargs)
14:14:34 INFO - File "Z:\task_1557492576\build\tests\xpcshell\runxpcshelltests.py", line 1627, in runTestList
14:14:34 INFO - raise exceptions[0]
14:14:34 INFO - MemoryError
WARNING | IO Completion Port failed to signal process shutdown
Parent process 5964 exited with children alive:
PIDS: 3560
Attempting to kill them, but no guarantee of success
14:17:41 ERROR - Return code: 1
14:17:41 ERROR - No tests run or test summary not found
14:17:41 INFO - TinderboxPrint: xpcshell-xpcshell<br/><em class="testfail">T-FAIL</em>

Comment 5

[Mark Banner (:standard8)]

I've been looking into this, and it seems the fundamental issue is that the harness can't cope with the amount of logging we're throwing at it.

I'm therefore working on some patches to only output the logging if we're in test debug mode, or if there's actual failures. These seem to help a lot.

I'm currently pondering between splitting the tests to run in 4 chunks per test, however that isn't really showing any gains at the moment, so I might stay with the structure as it is already.

Comment 6

[Mark Banner (:standard8)]

To follow up on my previous comment, it seems splitting the tests to run in 4 chunks does make it faster on Windows & Mac. It is also better than leaving them as one chunk, with some extra parallel running being handled pretty well on Windows & Mac.

Comment 7

[Mark Banner (:standard8)]

Attachment: Bug 1541419 - Split up and reduce test output of the xpcshell searchconfigs tests to improve test times. r?daleharvey

This splits running of locales across 4 chunks, which can run in parallel better.

The chunks can be run individually with '--tag=searchconfig1' etc.

It also stops logging test output in the pass cases (unless we're in _testDebug=true mode). This makes less work on the python harness which was causing a bottleneck and slowing the tests down.

Depends on D30399

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/afbd9e47febe
Correct the search form url for Ebay in 'be' locale. r=mkaply,mikedeboer
https://hg.mozilla.org/integration/autoland/rev/26e0b9e416db
Adds test to ensure that search engines are using the correct domain in all regions/locales. r=daleharvey
https://hg.mozilla.org/integration/autoland/rev/0e0c4b2166dc
Split up and reduce test output of the xpcshell searchconfigs tests to improve test times. r=daleharvey

Comment 9

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/afbd9e47febe
https://hg.mozilla.org/mozilla-central/rev/26e0b9e416db
https://hg.mozilla.org/mozilla-central/rev/0e0c4b2166dc