1315953 - browser_google.js and browser_google_behavior.js fail after 'mach build faster' and on artifact builds

Status: RESOLVED FIXED

(Firefox :: Search, task, P3)

Description

[:Gijs (he/him)]

STR:

1. ./mach build
2. ./mach mochitest browser/components/search/test/browser_google_behavior.js 

(pass)
3. ./mach build faster
4. ./mach mochitest browser/components/search/test/browser_google_behavior.js 

(fail)

including:


55 INFO TEST-UNEXPECTED-FAIL | browser/components/search/test/browser_google_behavior.js | Check search URL for 'foo' - Got https://www.google.com/search?q=foo&ie=utf-8&oe=utf-8&client=firefox-b, expected https://www.google.com/search?q=foo&ie=utf-8&oe=utf-8


Feels like there's some kind of initialization that happens that makes things dirty and then fails the test. Or something.

Comment 1

[Florian Quèze [:florian]]

Mike, do you know why this doesn't work in this case?

Comment 2

[Mark Banner (:standard8)]

This happens with artifact builds for me as well, opt builds are fine.

Comment 3

[Mike Kaply [:mkaply]]

So the specific reason this doesn't work is because in the artifact build case, we use the packaged list.json as the default and it has google in the list, not google-nocodes (which it should).

What I really want is for the search engines to be rebuild in the artifact case.

They are front end and should be treated as such. I'm investigating that.

Comment 4

[:Gijs (he/him)]

This affects artifact builds, locally as well as on try.

Comment 5

[Mike Kaply [:mkaply]]

Apparently there are other tests that fail in artifact builds as well.

Mike Hommey is working on something that might fix this.

What I'd really like is for the search engine stuff to happen as part of front end. I'm not entirely sure how to do that.

Comment 6

[Mark Banner (:standard8)]

Mike (Hommey) is the problem here that we're using the libs: target to install the searchplugins:

https://dxr.mozilla.org/mozilla-central/rev/8103c612b79c2587ea4ca1b0a9f9f82db4b185b8/browser/locales/Makefile.in#76

Is there an easy way around this?

Comment 7

[Chris Manchester (limited bugmail, email directly)]

These are the only remaining tests I know of that are failing against artifact builds, we should disable them there until we're able to figure this out.

Comment 8

[Chris Manchester (limited bugmail, email directly)]

Attachment: Bug 1315953 - Skip browser_google_nocodes.js on artifact builds.

Review commit: https://reviewboard.mozilla.org/r/110760/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/110760/

Comment 9

[Mike Kaply [:mkaply]]

Comment on attachment 8835065 [details]
Bug 1315953 - Skip browser_google_nocodes.js on artifact builds.

https://reviewboard.mozilla.org/r/110760/#review112148

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6c97dfdfe6f9
Skip search related browser chrome tests on artifact builds. r=mkaply

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/6c97dfdfe6f9

Comment 12

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8835065 [details]
Bug 1315953 - Skip browser_google_nocodes.js on artifact builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/110760/diff/1-2/

Comment 13

[Mike Kaply [:mkaply]]

Comment on attachment 8835065 [details]
Bug 1315953 - Skip browser_google_nocodes.js on artifact builds.

I thought this was already r+ed in mozreview...

Comment 14

[Chris Manchester (limited bugmail, email directly)]

(In reply to Mike Kaply [:mkaply] from comment #13)
> Comment on attachment 8835065 [details]
> Bug 1315953 - Skip browser_google_nocodes.js on artifact builds.
> 
> I thought this was already r+ed in mozreview...

It was... this is a different patch for another failing test I found this week.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f68ac7a260c1
Skip browser_google_nocodes.js on artifact builds. r=mkaply

Comment 16

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/f68ac7a260c1

Comment 17

[Julien Cristau [:jcristau]]

Mass wontfix for bugs affecting firefox 52.

Comment 18

[Dan Mosedale (:dmosedale, :dmose)]

One thing I noticed when logged into a one-click loaner to debug this was that if I run

./mach mochitest -f browser --run-until-failure browser/components/search/test/browser_google_codes.js

it never fails.


However, if I do

./mach mochitest -f browser --run-until-failure browser/components/search/test

it quickly hits a failure.


After doing a bunch of test disabling by editing browser.ini, the minimal necessary thing to cause failure is to have browser.ini set up to only run browser_google_codes.js and browser_about_searchReset.js.  Removing browser_about_searchReset.js from browser.ini makes the problem go away.

That said, I have yet to reproduce this on a machine that was not one of the taskcluster loaners.

Comment 19

[Mike Kaply [:mkaply]]

The problems the google tests are all fixed, so this particular bug can probably be closed.

I did find that:

./mach mochitest -f browser --run-until-failure browser/components/search/test/browser_searchEngine_behaviors.js

Always fails on artificate and non artifact builds, so that seems unrelated. I'm guessing that test doesn't properly clean up?

Comment 20

[Mike Kaply [:mkaply]]

I opened

https://bugzilla.mozilla.org/show_bug.cgi?id=1473710

for the browser_searchEngine_behaviors.js failure. 

This bug is fixed (I removed the artifact only stuff when I checked in bug 1437942).

Comment 21

[Mark Banner (:standard8)]

I just found a couple of instances of skips still referencing this bug, and we no longer need them, so lets use this to clean those up.

Comment 22

[Mark Banner (:standard8)]

Attachment: Bug 1315953 - Allow more browser search tests to run in artifact mode now that the underlying issues are fixed. r?daleharvey

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a68c01cb895d
Allow more browser search tests to run in artifact mode now that the underlying issues are fixed. r=daleharvey

Comment 24

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/a68c01cb895d