785074 - Enabling <iframe mozbrowser> in b2g reftests causes additional failures

Status: RESOLVED INCOMPLETE

(Testing :: Reftest, defect, P2)

Description

[Andrew Halberstadt [:ahal]]

Attachment: reftest-sanity log with reftest.browser.iframe.enabled = true

With the landing of bug 778072 we switched from <browser> to <iframe mozbrowser> since that is what will be shipping. This causes some additional test failures.

I attached a reftest-sanity log with 9 failures (where previously there were only 3).
The six new failures are:
== test-async.xul test-async-ref.xul
== test-async.html test-async-ref.html
== zoom-invalidation.html zoom-invalidation-ref.html
== corners-2.html corners-2-ref.html
== corners-3.html corners-3-ref.html
== corners-4.html corners-4-ref.html

There are many other failures in other manifests too, but fixing these will probably fix the rest.

Comment 1

[Andrew Overholt [:overholt]]

Blocking so we can investigate and determine whether or not there is a deeper issue.

Comment 2

[Andrew Overholt [:overholt]]

Jet, can the layout team investigate?

Comment 3

[Andrew Halberstadt [:ahal]]

Instructions for running reftests on b2g at: https://developer.mozilla.org/en-US/docs/Mozilla/Boot_to_Gecko/B2G_Reftests

I'd recommend downloading a nightly emulator build.

Comment 4

[Jet Villegas (inactive)]

To Emmanuelle for investigation...

Comment 5

[Emmanuele Bassi (:ebassi)]

(In reply to Andrew Halberstadt [:ahal] from comment #3)
> Instructions for running reftests on b2g at:
> https://developer.mozilla.org/en-US/docs/Mozilla/Boot_to_Gecko/B2G_Reftests
> 
> I'd recommend downloading a nightly emulator build.

I tried by building b2g from mercurial (+ gaia from git), which worked pretty much okay, but the wiki is a bit scant on details on how to go from that to running the reftests python script; I'll try running from a nightly build, but it'd probably be worth adding details on the wiki page, like where should the B2G_HOME directory point to, and what "--emulator arm" should do.

Comment 6

[Emmanuele Bassi (:ebassi)]

yeah, I *could* test a nightly build, if there was one for x86_64 on Linux. I'll either have to test under Mac or I will need some guidance on how to go from a build of b2g+gaia to running the runreftestb2g.py script.

Comment 7

[Andrew Halberstadt [:ahal]]

The script is run from objdir-gecko/tests/reftest

--b2gpath can point to the B2G repo or an extracted emulator nightly build.
--emulator arm just identifies the type of emulator being used, it shouldn't change.

I'll add these details to the wiki.

Comment 8

[Emmanuele Bassi (:ebassi)]

thanks for the explanation; sadly, I still get this error:

  Exception: File not found: /home/ebassi/mozilla/b2g/out/host/linux-x86/bin/emulator

do I need some specific flag to build mozilla-central apart from the ones listed on the wiki?

Comment 9

[Andrew Halberstadt [:ahal]]

I have noticed the reftest extension wasn't getting installed anymore for a period last week. I haven't had a chance to look into it and I'm not sure if it's still a problem (or reproducible to others). So it might be better to punt on this for a little while.

If you do want to look into it, did you run ./config.sh emulator before ./build.sh? Depending on the device you are building for the ./config.sh script will pull down all of the required libraries.

Comment 10

[Emmanuele Bassi (:ebassi)]

(In reply to Andrew Halberstadt [:ahal] from comment #9)
> If you do want to look into it, did you run ./config.sh emulator before
> ./build.sh? Depending on the device you are building for the ./config.sh
> script will pull down all of the required libraries.

okay, I'm completely missing this step on the wiki. also, and I'm sorry for not saying this earlier, I'm trying to build the desktop client given that I don't have a b2g device to test with.

Comment 11

[Andrew Halberstadt [:ahal]]

(In reply to Emmanuele Bassi (:ebassi) from comment #10)
> okay, I'm completely missing this step on the wiki. also, and I'm sorry for
> not saying this earlier, I'm trying to build the desktop client given that I
> don't have a b2g device to test with.

Yeah, I just linked to the normal build instructions in the prerequisites. You'll want to build the emulator (not the b2g desktop) since this is where I noticed these test failures occurring.

But like I said, the reftests might not be working at all with the latest build. My aim is to investigate/fix by the end of the week.

Comment 12

[Andrew Halberstadt [:ahal]]

:ebassi, fyi the extension being installed issue is fixed (on m-c, I don't think it made it to aurora yet).

So if you end up tackling this let me know if you need any help. If you are having trouble building I can also give you an m-c based emulator.

Comment 13

[Emmanuele Bassi (:ebassi)]

(In reply to Andrew Halberstadt [:ahal] from comment #12)
> :ebassi, fyi the extension being installed issue is fixed (on m-c, I don't
> think it made it to aurora yet).

thanks for looking into it; I'll have a look ASAP.

> So if you end up tackling this let me know if you need any help. If you are
> having trouble building I can also give you an m-c based emulator.

thanks! I have a build set up, but if I have any issue I'll poke you.

Comment 14

[Dietrich Ayala (:dietrich)]

Re-nominating. It was only blocking for investigation, which isn't enough to hold the entire release for.

Comment 15

[Andrew Overholt [:overholt]]

I spoke about this with Andrew via IRC.  We talked about it and decided that since not many reftests are enabled ATM, it's probably not worth blocking on this.  As always, people should feel free to re-nom with a reason as to why something should block and we'll discuss it during triage.

Comment 16

[Andrew Halberstadt [:ahal]]

https://tbpl.mozilla.org/?tree=Cedar&fromchange=993e66df0419&tochange=21163b0bd235&jobname=ics_armv7a_gecko

See the differences in reftest failures with that last commit (ignore debug).

Comment 17

[Andrew Halberstadt [:ahal]]

Attachment: Enables <iframe mozbrowser remote="true"> in the b2g reftest harness

This enables both <iframe mozbrowser> and remote="true". Note that tests still fail even with remote="false", so in the interest of separating concerns you may want to leave it at false to begin with.