Open Bug 1142129 Opened 5 years ago Updated 5 years ago

Running reftests on Fennec locally sometimes fails because the window is zero-sized

Categories

(Testing :: Reftest, defect)

ARM
Android
defect
Not set
normal

Tracking

(Not tracked)

People

(Reporter: kats, Unassigned)

References

Details

Attachments

(1 file)

There appears to be no way to run reftests locally on Fennec. There used to be some instructions on the wiki at [1] and while those appeared to work, they didn't actually produce the correct output, and do not replicate results from tryserver. Specifically I would always get garbage drawn to the screen, and the tests would always fail.

Neither mach reftest nor mach reftest-remote appears to support Fennec either.

[1] https://wiki.mozilla.org/Mobile/Fennec/Android#Reftests
We never got around to adding mach support, but the make target and old docs "should" have worked. I'll check into it.
Assignee: nobody → gbrown
This works fine for me on my Galaxy Nexus running Android 4.0.1:

export TEST_PATH=layout/reftests/reftest-sanity/reftest.list
export MOZ_HOST_BIN=<path to my host utils>
make reftest-remote

I see the tests running on the phone, all tests pass, and the log looks okay.
gbrown@mozpad:~/objdirs/droid$ make reftest-remote
INFO | automation.py | Launching: /home/gbrown/hostutils-march2015/host-utils-37.0a2.en-US.linux-x86_64/xpcshell -g /home/gbrown/hostutils-march2015/host-utils-37.0a2.en-US.linux-x86_64 -v 170 -f /home/gbrown/objdirs/droid/_tests/modules/httpd.js -e "const _PROFILE_PATH = '/tmp/tmpr_44SS';const _SERVER_PORT = '8888'; const _SERVER_ADDR ='192.168.0.82';" -f /home/gbrown/objdirs/droid/_tests/reftest/server.js
INFO | remotereftests.py | Server pid: 21476
Device info:
  uptime:  18:01:18 up 8 min,  0 users,  load average: 0.53, 0.77, 0.50
  memtotal: 646228 kB
  process:
     PID   USER     TIME   COMMAND
         1 root       0:01 /init
         2 root       0:00 [kthreadd]
         3 root       0:00 [ksoftirqd/0]
         4 root       0:03 [kworker/0:0]
         5 root       0:00 [kworker/u:0]
         6 root       0:00 [migration/0]
         7 root       0:00 [migration/1]
         9 root       0:00 [ksoftirqd/1]
        10 root       0:00 [khelper]
        11 root       0:00 [suspend]
        12 root       0:00 [irq/329-pogo_de]
        13 root       0:00 [sync_supers]
        14 root       0:00 [bdi-default]
        15 root       0:00 [kblockd]
        16 root       0:00 [omap2_mcspi]
        17 root       0:00 [khubd]
        18 root       0:00 [twl6030-irq]
        19 root       0:00 [irq/374-TWL6030]
        20 root       0:00 [kworker/0:1]
        21 root       0:00 [cfg80211]
        22 root       0:00 [musb-otg]
        23 root       0:01 [kinteractiveup]
        25 root       0:00 [khungtaskd]
        26 root       0:00 [kswapd0]
        27 root       0:00 [fsnotify_mark]
        28 root       0:00 [crypto]
        41 root       0:00 [hdmi_hotplug]
        42 root       0:00 [hdcp]
        44 root       0:00 [dsscomp_apply]
        45 root       0:00 [dsscomp_apply]
        46 root       0:00 [dsscomp_apply]
        47 root       0:00 [dsscomp_cb]
        48 root       0:00 [irq/335-sii9234]
        50 root       0:00 [pvr_timer]
        51 root       0:00 [bmp180_wq]
        52 root       0:00 [sec_jack_wq]
        54 root       0:00 [irq/184-fsa9480]
        55 root       0:00 [irq/164-fsa9480]
        56 root       0:00 [irq/159-temp_se]
        57 root       0:00 [irq/246-tshut]
        58 root       0:00 [irq/151-twl6040]
        59 root       0:00 [mtdblock0]
        66 root       0:00 [f_mtp]
        67 root       0:00 [file-storage]
        69 root       0:00 [gp2a_wq]
        70 root       0:00 [irq/379-rtc0]
        71 root       0:00 [irq/204-fuel ga]
        72 root       0:00 [khotplug]
        73 root       0:00 [binder]
        74 root       0:00 [irq/131-ABE]
        75 root       0:00 [kworker/u:2]
        76 root       0:00 [twl6040-codec]
        77 root       0:00 [twl6040-hf]
        78 root       0:00 [twl6040-hs]
        79 root       0:00 [twl6040-ep]
        80 root       0:12 [mmcqd/0]
        81 root       0:00 [mmcqd/0boot0]
        82 root       0:00 [mmcqd/0boot1]
        83 root       0:00 [kworker/1:1]
        84 root       0:00 [krfcommd]
        85 root       0:00 [irq/142-omap_em]
        86 root       0:00 [irq/143-omap_em]
        87 root       0:00 [mipi_tx_wq]
        88 root       0:00 [mipi_tx_raw_wq]
        89 root       0:00 [mipi_cmd_wq]
        90 root       0:00 [kworker/u:1]
        91 root       0:00 [dhd_watchdog]
        92 root       0:00 [dhd_dpc]
        93 root       0:00 [dhd_sysioc]
        94 root       0:00 /sbin/ueventd
        98 root       0:00 [jbd2/mmcblk0p10]
        99 root       0:00 [ext4-dio-unwrit]
       100 root       0:03 [jbd2/mmcblk0p12]
       101 root       0:00 [ext4-dio-unwrit]
       102 root       0:00 [irq/206-mms_ts]
       103 root       0:00 [jbd2/mmcblk0p11]
       104 root       0:00 [ext4-dio-unwrit]
       105 root       0:00 [jbd2/mmcblk0p3-]
       106 root       0:00 [ext4-dio-unwrit]
       107 1000       0:00 /system/bin/servicemanager
       108 root       0:00 /system/bin/vold
       113 root       0:00 /system/bin/netd
       114 root       0:00 /system/bin/debuggerd
       115 1001       0:01 /system/bin/rild
       116 1000       0:01 /system/bin/surfaceflinger
       117 root       0:04 zygote /bin/app_process -Xzygote /system/bin --zygote --s
       118 1019       0:00 /system/bin/drmserver
       119 1013       0:00 /system/bin/mediaserver
       120 1002       0:00 /system/bin/dbus-daemon --system --nofork
       121 root       0:00 /system/bin/installd
       122 1017       0:00 /system/bin/keystore /data/misc/keystore
       124 1023       0:00 /system/bin/sdcard /data/media 1023 1023
       125 2000       0:00 /system/bin/sh
       131 root       0:03 /sbin/adbd
       140 root       0:00 [pvr_workqueue]
       155 root       0:00 [omaplfb]
       167 root       0:00 [kworker/u:3]
       174 root       0:00 [flush-179:0]
       182 1000       0:23 system_server
       244 1010       0:00 /system/bin/wpa_supplicant -Dnl80211 -iwlan0 -puse_p2p_gr
       255 1000       0:02 {ndroid.systemui} com.android.systemui
       333 1000       0:00 {ndroid.settings} com.android.settings
       351 10017      0:00 {putmethod.latin} com.android.inputmethod.latin
       369 1001       0:00 {m.android.phone} com.android.phone
       387 1025       0:00 {om.android.nfc3} com.android.nfc3
       401 10018      0:05 {ndroid.launcher} com.android.launcher
       424 10012      0:34 {d.process.media} android.process.media
       457 10037      0:00 {android.smspush} com.android.smspush
       475 10000      0:00 {d.process.acore} android.process.acore
       541 10000      0:00 {ndroid.contacts} com.android.contacts
       570 10006      0:00 {viders.calendar} com.android.providers.calendar
       593 10010      0:00 {droid.deskclock} com.android.deskclock
       611 10013      0:00 {m.android.email} com.android.email
       630 10014      0:00 {ndroid.exchange} com.android.exchange
       649 10021      0:00 com.android.mms
       690 10036      0:00 {oid.voicedialer} com.android.voicedialer
       717 1014       0:00 /system/bin/dhcpcd -ABKL -h android-350e0001b3dc37da wlan
       739 10005      0:00 {ndroid.calendar} com.android.calendar
       798 root       0:00 [kworker/1:2]
       869 root       0:00 [kworker/1:0]
       880 root       0:00 [kworker/u:1]
       914 10015      0:00 {droid.gallery3d} com.android.gallery3d
       929 1000       0:00 {ndroid.keychain} com.android.keychain
       942 10009      0:00 {id.defcontainer} com.android.defcontainer
       958 10023      0:00 {android.musicfx} com.android.musicfx
       972 10029      0:00 {.quicksearchbox} com.android.quicksearchbox
      1009 10026      0:00 com.svox.pico
      1065 root       0:00 /system/bin/sh -c ps; echo $?
      1067 root       0:00 ps
  systime: Wed Mar 11 18:01:18 UTC 2015
  disk:
     Filesystem           1K-blocks      Used Available Use% Mounted on
     /dev/block/platform/omap/omap_hsmmc.0/by-name/userdata
                           13975888    369124  13606764   3% /data
     /dev/block/platform/omap/omap_hsmmc.0/by-name/system
                             659152    165980    493172  25% /system
     /dev/fuse             13975888    369124  13606764   3% /mnt/sdcard
  os: full_maguro-eng 4.0.1 ITL41F eng.mozdev.20120306.224313 test-keys
  id: 01498B300600B008
Test root: /sdcard/tests
REFTEST INFO | runreftest.py | Running tests: start.

INFO | automation.py | Application pid: 10038
REFTEST INFO | Discovered 120 tests, after filtering SKIP tests, we have 111
REFTEST TEST-START | data:text/html,<body>
REFTEST TEST-LOAD | data:text/html,<body> | 0 / 120 (0%)
REFTEST INFO | [CONTENT] Using browser remote=false

REFTEST INFO | drawWindow flags = DRAWWINDOW_DRAW_CARET | DRAWWINDOW_DRAW_VIEW | DRAWWINDOW_USE_WIDGET_LAYERS; window size = 720,1134; test browser size = 800,1000
REFTEST TEST-LOAD | about:blank | 0 / 120 (0%)
REFTEST TEST-PASS | data:text/html,<body> | image comparison (==)
REFTEST INFO | Loading a blank page
REFTEST TEST-END | data:text/html,<body>
REFTEST TEST-START | data:text/plain,
REFTEST TEST-LOAD | data:text/plain, | 1 / 120 (0%)
REFTEST TEST-PASS | data:text/plain, | image comparison (==)
REFTEST INFO | Loading a blank page
REFTEST TEST-END | data:text/plain,
REFTEST TEST-START | data:text/plain,HELLO
REFTEST TEST-LOAD | data:text/plain,HELLO | 2 / 120 (1%)
REFTEST TEST-PASS | data:text/plain,HELLO | image comparison (!=)
REFTEST INFO | Loading a blank page
REFTEST TEST-END | data:text/plain,HELLO
REFTEST TEST-START | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/test-async.xul
REFTEST TEST-LOAD | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/test-async.xul | 3 / 120 (2%)
REFTEST TEST-LOAD | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/test-async-ref.xul | 3 / 120 (2%)
REFTEST TEST-PASS | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/test-async.xul | image comparison (==)
REFTEST INFO | Loading a blank page
REFTEST TEST-END | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/test-async.xul

...

REFTEST TEST-START | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/reftest-no-paint.html
REFTEST TEST-LOAD | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/reftest-no-paint.html | 117 / 120 (97%)
REFTEST TEST-LOAD | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/reftest-no-paint-ref.html | 117 / 120 (97%)
REFTEST TEST-KNOWN-FAIL | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/reftest-no-paint.html | failed reftest-no-paint
REFTEST INFO | Loading a blank page
REFTEST TEST-END | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/reftest-no-paint.html
REFTEST TEST-KNOWN-FAIL | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/async-scroll-1a.html | (SKIP)
REFTEST TEST-KNOWN-FAIL | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/async-scroll-1b.html | (SKIP)
REFTEST FINISHED: Slowest test took 1032ms (http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/invalidation.html)
REFTEST INFO | Result summary:
REFTEST INFO | Successful: 91 (84 pass, 7 load only)
REFTEST INFO | Unexpected: 0 (0 unexpected fail, 0 unexpected pass, 0 unexpected asserts, 0 unexpected fixed asserts, 0 failed load, 0 exception)
REFTEST INFO | Known problems: 30 (7 known fail, 0 known asserts, 3 random, 20 skipped, 0 slow)
REFTEST INFO | Total canvas count = 3
REFTEST TEST-START | Shutdown

INFO | automation.py | Application ran for: 0:01:04.336465
INFO | zombiecheck | Reading PID log: /tmp/tmpByIB0Apidlog
Contents of /data/anr/traces.txt:


MOZ_UPLOAD_DIR not defined; tombstone check skipped
WARNING | leakcheck | refcount logging is off, so leaks can't be detected!

REFTEST INFO | runreftest.py | Running tests: end.

Device info:
  uptime:  18:02:27 up 9 min,  0 users,  load average: 0.54, 0.74, 0.50
  memtotal: 646228 kB
  process:
     PID   USER     TIME   COMMAND
         1 root       0:01 /init
         2 root       0:00 [kthreadd]
         3 root       0:00 [ksoftirqd/0]
         4 root       0:03 [kworker/0:0]
         5 root       0:00 [kworker/u:0]
         6 root       0:00 [migration/0]
         7 root       0:00 [migration/1]
         9 root       0:00 [ksoftirqd/1]
        10 root       0:00 [khelper]
        11 root       0:00 [suspend]
        12 root       0:00 [irq/329-pogo_de]
        13 root       0:00 [sync_supers]
        14 root       0:00 [bdi-default]
        15 root       0:00 [kblockd]
        16 root       0:00 [omap2_mcspi]
        17 root       0:00 [khubd]
        18 root       0:00 [twl6030-irq]
        19 root       0:00 [irq/374-TWL6030]
        20 root       0:00 [kworker/0:1]
        21 root       0:00 [cfg80211]
        22 root       0:00 [musb-otg]
        23 root       0:01 [kinteractiveup]
        25 root       0:00 [khungtaskd]
        26 root       0:00 [kswapd0]
        27 root       0:00 [fsnotify_mark]
        28 root       0:00 [crypto]
        41 root       0:00 [hdmi_hotplug]
        42 root       0:00 [hdcp]
        44 root       0:00 [dsscomp_apply]
        45 root       0:00 [dsscomp_apply]
        46 root       0:00 [dsscomp_apply]
        47 root       0:00 [dsscomp_cb]
        48 root       0:00 [irq/335-sii9234]
        50 root       0:00 [pvr_timer]
        51 root       0:00 [bmp180_wq]
        52 root       0:00 [sec_jack_wq]
        54 root       0:00 [irq/184-fsa9480]
        55 root       0:00 [irq/164-fsa9480]
        56 root       0:00 [irq/159-temp_se]
        57 root       0:00 [irq/246-tshut]
        58 root       0:00 [irq/151-twl6040]
        59 root       0:00 [mtdblock0]
        66 root       0:00 [f_mtp]
        67 root       0:00 [file-storage]
        69 root       0:00 [gp2a_wq]
        70 root       0:00 [irq/379-rtc0]
        71 root       0:00 [irq/204-fuel ga]
        72 root       0:00 [khotplug]
        73 root       0:00 [binder]
        74 root       0:00 [irq/131-ABE]
        75 root       0:00 [kworker/u:2]
        76 root       0:00 [twl6040-codec]
        77 root       0:00 [twl6040-hf]
        78 root       0:00 [twl6040-hs]
        79 root       0:00 [twl6040-ep]
        80 root       0:12 [mmcqd/0]
        81 root       0:00 [mmcqd/0boot0]
        82 root       0:00 [mmcqd/0boot1]
        83 root       0:00 [kworker/1:1]
        84 root       0:00 [krfcommd]
        85 root       0:00 [irq/142-omap_em]
        86 root       0:00 [irq/143-omap_em]
        87 root       0:00 [mipi_tx_wq]
        88 root       0:00 [mipi_tx_raw_wq]
        89 root       0:00 [mipi_cmd_wq]
        90 root       0:00 [kworker/u:1]
        91 root       0:00 [dhd_watchdog]
        92 root       0:00 [dhd_dpc]
        93 root       0:00 [dhd_sysioc]
        94 root       0:00 /sbin/ueventd
        98 root       0:00 [jbd2/mmcblk0p10]
        99 root       0:00 [ext4-dio-unwrit]
       100 root       0:03 [jbd2/mmcblk0p12]
       101 root       0:00 [ext4-dio-unwrit]
       102 root       0:00 [irq/206-mms_ts]
       103 root       0:00 [jbd2/mmcblk0p11]
       104 root       0:00 [ext4-dio-unwrit]
       105 root       0:00 [jbd2/mmcblk0p3-]
       106 root       0:00 [ext4-dio-unwrit]
       107 1000       0:00 /system/bin/servicemanager
       108 root       0:00 /system/bin/vold
       113 root       0:00 /system/bin/netd
       114 root       0:02 /system/bin/debuggerd
       115 1001       0:01 /system/bin/rild
       116 1000       0:03 /system/bin/surfaceflinger
       117 root       0:04 zygote /bin/app_process -Xzygote /system/bin --zygote --s
       118 1019       0:00 /system/bin/drmserver
       119 1013       0:00 /system/bin/mediaserver
       120 1002       0:00 /system/bin/dbus-daemon --system --nofork
       121 root       0:00 /system/bin/installd
       122 1017       0:00 /system/bin/keystore /data/misc/keystore
       124 1023       0:00 /system/bin/sdcard /data/media 1023 1023
       125 2000       0:00 /system/bin/sh
       131 root       0:04 /sbin/adbd
       140 root       0:00 [pvr_workqueue]
       155 root       0:00 [omaplfb]
       167 root       0:00 [kworker/u:3]
       174 root       0:00 [flush-179:0]
       182 1000       0:25 system_server
       244 1010       0:00 /system/bin/wpa_supplicant -Dnl80211 -iwlan0 -puse_p2p_gr
       255 1000       0:02 {ndroid.systemui} com.android.systemui
       333 1000       0:00 {ndroid.settings} com.android.settings
       351 10017      0:00 {putmethod.latin} com.android.inputmethod.latin
       369 1001       0:00 {m.android.phone} com.android.phone
       387 1025       0:00 {om.android.nfc3} com.android.nfc3
       401 10018      0:05 {ndroid.launcher} com.android.launcher
       424 10012      0:34 {d.process.media} android.process.media
       457 10037      0:00 {android.smspush} com.android.smspush
       475 10000      0:00 {d.process.acore} android.process.acore
       541 10000      0:00 {ndroid.contacts} com.android.contacts
       570 10006      0:00 {viders.calendar} com.android.providers.calendar
       593 10010      0:00 {droid.deskclock} com.android.deskclock
       611 10013      0:00 {m.android.email} com.android.email
       630 10014      0:00 {ndroid.exchange} com.android.exchange
       690 10036      0:00 {oid.voicedialer} com.android.voicedialer
       717 1014       0:00 /system/bin/dhcpcd -ABKL -h android-350e0001b3dc37da wlan
       739 10005      0:00 {ndroid.calendar} com.android.calendar
       798 root       0:00 [kworker/1:2]
       869 root       0:00 [kworker/1:0]
       880 root       0:00 [kworker/u:1]
       914 10015      0:00 {droid.gallery3d} com.android.gallery3d
       929 1000       0:00 {ndroid.keychain} com.android.keychain
       942 10009      0:00 {id.defcontainer} com.android.defcontainer
       958 10023      0:00 {android.musicfx} com.android.musicfx
       972 10029      0:00 {.quicksearchbox} com.android.quicksearchbox
      1009 10026      0:00 com.svox.pico
      1202 root       0:00 [flush-0:17]
      1280 root       0:00 /system/bin/sh -c ps; echo $?
      1282 root       0:00 ps
  systime: Wed Mar 11 18:02:27 UTC 2015
  disk:
     Filesystem           1K-blocks      Used Available Use% Mounted on
     /dev/block/platform/omap/omap_hsmmc.0/by-name/userdata
                           13975888    369740  13606148   3% /data
     /dev/block/platform/omap/omap_hsmmc.0/by-name/system
                             659152    165980    493172  25% /system
     /dev/fuse             13975888    369740  13606148   3% /mnt/sdcard
  os: full_maguro-eng 4.0.1 ITL41F eng.mozdev.20120306.224313 test-keys
  id: 01498B300600B008
Test root: /sdcard/tests
/bin/sh: 12: @errors=: not found
Your output says this:

window size = 720,1134;

whereas mine says 0,0 in the same spot. Do you know where that comes from?
Ah, it looks like 0-length timeout at [1] isn't long enough. When I bump it up to 1000 it works.

[1] http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/bootstrap.js?rev=e51a54f1817d#31
Summary: No way to run reftests on Fennec locally → Running reftests on Fennec locally sometimes fails because the window is zero-sized
Interesting.

Something else I noticed: reftests work as I showed above with a new install, but if I run the same reftest job again, I get a 0,0 window size and the tests fail. Uninstall/install firefox and all is well again (until the next run).
Attachment #8576220 - Flags: review?(jmaher)
Attachment #8576220 - Flags: review?(gbrown)
Assignee: gbrown → bugmail.mozilla
Comment on attachment 8576220 [details] [diff] [review]
Robustify startup

Review of attachment 8576220 [details] [diff] [review]:
-----------------------------------------------------------------

Works great for me. Thanks!
Attachment #8576220 - Flags: review?(gbrown) → review+
Comment on attachment 8576220 [details] [diff] [review]
Robustify startup

Review of attachment 8576220 [details] [diff] [review]:
-----------------------------------------------------------------

::: layout/tools/reftest/bootstrap.js
@@ +13,5 @@
>  }
>  
> +function startRefTestsWhenReady(win) {
> +    if (win.innerHeight == 0) {
> +        win.setTimeout(startRefTestsWhenReady, 100, win);

can we put a log message here indicating we are waiting 100ms for win.innerHeight != 0 ?  this way if we get in a loop it can be easier to detect if this is the cause!
Attachment #8576220 - Flags: review?(jmaher) → review+
Thanks! I added a dump statement to that effect and pushed to try to make sure it doesn't bust anything. If that's green I'll land it.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=aaef04ef628f
I don't know wtf is going on here.. that try job has lots of failures (android 11+ only). Another variant that I pushed at https://treeherder.mozilla.org/#/jobs?repo=try&revision=f2c35c858017 shows similar results. What's different about android 11+ vs android 9?
After many many try pushes it looks like there's something fundamentally wrong with the testing framework. See for example https://treeherder.mozilla.org/#/jobs?repo=try&revision=4ad1197b7327 which just delays the test start by 10 seconds and causes everything to fail. It looks like the tests *expect* to run with a window innerHeight of 0 and if they don't then they fail. There must be a race somewhere. A delay of 1 second is still (mostly) acceptable, https://treeherder.mozilla.org/#/jobs?repo=try&revision=2fd647ddf545
Gonna give up on this for now. If that means Fennec can't have new features, so be it.
Assignee: bugmail.mozilla → nobody
Depends on: 1156817
You need to log in before you can comment on or make changes to this bug.