Running reftests on Fennec locally sometimes fails because the window is zero-sized

NEW
Unassigned

Status

defect
5 years ago
4 years ago

People

(Reporter: kats, Unassigned)

Tracking

(Depends on 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

There appears to be no way to run reftests locally on Fennec. There used to be some instructions on the wiki at [1] and while those appeared to work, they didn't actually produce the correct output, and do not replicate results from tryserver. Specifically I would always get garbage drawn to the screen, and the tests would always fail.

Neither mach reftest nor mach reftest-remote appears to support Fennec either.

[1] https://wiki.mozilla.org/Mobile/Fennec/Android#Reftests
We never got around to adding mach support, but the make target and old docs "should" have worked. I'll check into it.
Assignee: nobody → gbrown
This works fine for me on my Galaxy Nexus running Android 4.0.1:

export TEST_PATH=layout/reftests/reftest-sanity/reftest.list
export MOZ_HOST_BIN=<path to my host utils>
make reftest-remote

I see the tests running on the phone, all tests pass, and the log looks okay.
gbrown@mozpad:~/objdirs/droid$ make reftest-remote
INFO | automation.py | Launching: /home/gbrown/hostutils-march2015/host-utils-37.0a2.en-US.linux-x86_64/xpcshell -g /home/gbrown/hostutils-march2015/host-utils-37.0a2.en-US.linux-x86_64 -v 170 -f /home/gbrown/objdirs/droid/_tests/modules/httpd.js -e "const _PROFILE_PATH = '/tmp/tmpr_44SS';const _SERVER_PORT = '8888'; const _SERVER_ADDR ='192.168.0.82';" -f /home/gbrown/objdirs/droid/_tests/reftest/server.js
INFO | remotereftests.py | Server pid: 21476
Device info:
  uptime:  18:01:18 up 8 min,  0 users,  load average: 0.53, 0.77, 0.50
  memtotal: 646228 kB
  process:
     PID   USER     TIME   COMMAND
         1 root       0:01 /init
         2 root       0:00 [kthreadd]
         3 root       0:00 [ksoftirqd/0]
         4 root       0:03 [kworker/0:0]
         5 root       0:00 [kworker/u:0]
         6 root       0:00 [migration/0]
         7 root       0:00 [migration/1]
         9 root       0:00 [ksoftirqd/1]
        10 root       0:00 [khelper]
        11 root       0:00 [suspend]
        12 root       0:00 [irq/329-pogo_de]
        13 root       0:00 [sync_supers]
        14 root       0:00 [bdi-default]
        15 root       0:00 [kblockd]
        16 root       0:00 [omap2_mcspi]
        17 root       0:00 [khubd]
        18 root       0:00 [twl6030-irq]
        19 root       0:00 [irq/374-TWL6030]
        20 root       0:00 [kworker/0:1]
        21 root       0:00 [cfg80211]
        22 root       0:00 [musb-otg]
        23 root       0:01 [kinteractiveup]
        25 root       0:00 [khungtaskd]
        26 root       0:00 [kswapd0]
        27 root       0:00 [fsnotify_mark]
        28 root       0:00 [crypto]
        41 root       0:00 [hdmi_hotplug]
        42 root       0:00 [hdcp]
        44 root       0:00 [dsscomp_apply]
        45 root       0:00 [dsscomp_apply]
        46 root       0:00 [dsscomp_apply]
        47 root       0:00 [dsscomp_cb]
        48 root       0:00 [irq/335-sii9234]
        50 root       0:00 [pvr_timer]
        51 root       0:00 [bmp180_wq]
        52 root       0:00 [sec_jack_wq]
        54 root       0:00 [irq/184-fsa9480]
        55 root       0:00 [irq/164-fsa9480]
        56 root       0:00 [irq/159-temp_se]
        57 root       0:00 [irq/246-tshut]
        58 root       0:00 [irq/151-twl6040]
        59 root       0:00 [mtdblock0]
        66 root       0:00 [f_mtp]
        67 root       0:00 [file-storage]
        69 root       0:00 [gp2a_wq]
        70 root       0:00 [irq/379-rtc0]
        71 root       0:00 [irq/204-fuel ga]
        72 root       0:00 [khotplug]
        73 root       0:00 [binder]
        74 root       0:00 [irq/131-ABE]
        75 root       0:00 [kworker/u:2]
        76 root       0:00 [twl6040-codec]
        77 root       0:00 [twl6040-hf]
        78 root       0:00 [twl6040-hs]
        79 root       0:00 [twl6040-ep]
        80 root       0:12 [mmcqd/0]
        81 root       0:00 [mmcqd/0boot0]
        82 root       0:00 [mmcqd/0boot1]
        83 root       0:00 [kworker/1:1]
        84 root       0:00 [krfcommd]
        85 root       0:00 [irq/142-omap_em]
        86 root       0:00 [irq/143-omap_em]
        87 root       0:00 [mipi_tx_wq]
        88 root       0:00 [mipi_tx_raw_wq]
        89 root       0:00 [mipi_cmd_wq]
        90 root       0:00 [kworker/u:1]
        91 root       0:00 [dhd_watchdog]
        92 root       0:00 [dhd_dpc]
        93 root       0:00 [dhd_sysioc]
        94 root       0:00 /sbin/ueventd
        98 root       0:00 [jbd2/mmcblk0p10]
        99 root       0:00 [ext4-dio-unwrit]
       100 root       0:03 [jbd2/mmcblk0p12]
       101 root       0:00 [ext4-dio-unwrit]
       102 root       0:00 [irq/206-mms_ts]
       103 root       0:00 [jbd2/mmcblk0p11]
       104 root       0:00 [ext4-dio-unwrit]
       105 root       0:00 [jbd2/mmcblk0p3-]
       106 root       0:00 [ext4-dio-unwrit]
       107 1000       0:00 /system/bin/servicemanager
       108 root       0:00 /system/bin/vold
       113 root       0:00 /system/bin/netd
       114 root       0:00 /system/bin/debuggerd
       115 1001       0:01 /system/bin/rild
       116 1000       0:01 /system/bin/surfaceflinger
       117 root       0:04 zygote /bin/app_process -Xzygote /system/bin --zygote --s
       118 1019       0:00 /system/bin/drmserver
       119 1013       0:00 /system/bin/mediaserver
       120 1002       0:00 /system/bin/dbus-daemon --system --nofork
       121 root       0:00 /system/bin/installd
       122 1017       0:00 /system/bin/keystore /data/misc/keystore
       124 1023       0:00 /system/bin/sdcard /data/media 1023 1023
       125 2000       0:00 /system/bin/sh
       131 root       0:03 /sbin/adbd
       140 root       0:00 [pvr_workqueue]
       155 root       0:00 [omaplfb]
       167 root       0:00 [kworker/u:3]
       174 root       0:00 [flush-179:0]
       182 1000       0:23 system_server
       244 1010       0:00 /system/bin/wpa_supplicant -Dnl80211 -iwlan0 -puse_p2p_gr
       255 1000       0:02 {ndroid.systemui} com.android.systemui
       333 1000       0:00 {ndroid.settings} com.android.settings
       351 10017      0:00 {putmethod.latin} com.android.inputmethod.latin
       369 1001       0:00 {m.android.phone} com.android.phone
       387 1025       0:00 {om.android.nfc3} com.android.nfc3
       401 10018      0:05 {ndroid.launcher} com.android.launcher
       424 10012      0:34 {d.process.media} android.process.media
       457 10037      0:00 {android.smspush} com.android.smspush
       475 10000      0:00 {d.process.acore} android.process.acore
       541 10000      0:00 {ndroid.contacts} com.android.contacts
       570 10006      0:00 {viders.calendar} com.android.providers.calendar
       593 10010      0:00 {droid.deskclock} com.android.deskclock
       611 10013      0:00 {m.android.email} com.android.email
       630 10014      0:00 {ndroid.exchange} com.android.exchange
       649 10021      0:00 com.android.mms
       690 10036      0:00 {oid.voicedialer} com.android.voicedialer
       717 1014       0:00 /system/bin/dhcpcd -ABKL -h android-350e0001b3dc37da wlan
       739 10005      0:00 {ndroid.calendar} com.android.calendar
       798 root       0:00 [kworker/1:2]
       869 root       0:00 [kworker/1:0]
       880 root       0:00 [kworker/u:1]
       914 10015      0:00 {droid.gallery3d} com.android.gallery3d
       929 1000       0:00 {ndroid.keychain} com.android.keychain
       942 10009      0:00 {id.defcontainer} com.android.defcontainer
       958 10023      0:00 {android.musicfx} com.android.musicfx
       972 10029      0:00 {.quicksearchbox} com.android.quicksearchbox
      1009 10026      0:00 com.svox.pico
      1065 root       0:00 /system/bin/sh -c ps; echo $?
      1067 root       0:00 ps
  systime: Wed Mar 11 18:01:18 UTC 2015
  disk:
     Filesystem           1K-blocks      Used Available Use% Mounted on
     /dev/block/platform/omap/omap_hsmmc.0/by-name/userdata
                           13975888    369124  13606764   3% /data
     /dev/block/platform/omap/omap_hsmmc.0/by-name/system
                             659152    165980    493172  25% /system
     /dev/fuse             13975888    369124  13606764   3% /mnt/sdcard
  os: full_maguro-eng 4.0.1 ITL41F eng.mozdev.20120306.224313 test-keys
  id: 01498B300600B008
Test root: /sdcard/tests
REFTEST INFO | runreftest.py | Running tests: start.

INFO | automation.py | Application pid: 10038
REFTEST INFO | Discovered 120 tests, after filtering SKIP tests, we have 111
REFTEST TEST-START | data:text/html,<body>
REFTEST TEST-LOAD | data:text/html,<body> | 0 / 120 (0%)
REFTEST INFO | [CONTENT] Using browser remote=false

REFTEST INFO | drawWindow flags = DRAWWINDOW_DRAW_CARET | DRAWWINDOW_DRAW_VIEW | DRAWWINDOW_USE_WIDGET_LAYERS; window size = 720,1134; test browser size = 800,1000
REFTEST TEST-LOAD | about:blank | 0 / 120 (0%)
REFTEST TEST-PASS | data:text/html,<body> | image comparison (==)
REFTEST INFO | Loading a blank page
REFTEST TEST-END | data:text/html,<body>
REFTEST TEST-START | data:text/plain,
REFTEST TEST-LOAD | data:text/plain, | 1 / 120 (0%)
REFTEST TEST-PASS | data:text/plain, | image comparison (==)
REFTEST INFO | Loading a blank page
REFTEST TEST-END | data:text/plain,
REFTEST TEST-START | data:text/plain,HELLO
REFTEST TEST-LOAD | data:text/plain,HELLO | 2 / 120 (1%)
REFTEST TEST-PASS | data:text/plain,HELLO | image comparison (!=)
REFTEST INFO | Loading a blank page
REFTEST TEST-END | data:text/plain,HELLO
REFTEST TEST-START | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/test-async.xul
REFTEST TEST-LOAD | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/test-async.xul | 3 / 120 (2%)
REFTEST TEST-LOAD | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/test-async-ref.xul | 3 / 120 (2%)
REFTEST TEST-PASS | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/test-async.xul | image comparison (==)
REFTEST INFO | Loading a blank page
REFTEST TEST-END | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/test-async.xul

...

REFTEST TEST-START | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/reftest-no-paint.html
REFTEST TEST-LOAD | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/reftest-no-paint.html | 117 / 120 (97%)
REFTEST TEST-LOAD | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/reftest-no-paint-ref.html | 117 / 120 (97%)
REFTEST TEST-KNOWN-FAIL | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/reftest-no-paint.html | failed reftest-no-paint
REFTEST INFO | Loading a blank page
REFTEST TEST-END | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/reftest-no-paint.html
REFTEST TEST-KNOWN-FAIL | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/async-scroll-1a.html | (SKIP)
REFTEST TEST-KNOWN-FAIL | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/async-scroll-1b.html | (SKIP)
REFTEST FINISHED: Slowest test took 1032ms (http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/invalidation.html)
REFTEST INFO | Result summary:
REFTEST INFO | Successful: 91 (84 pass, 7 load only)
REFTEST INFO | Unexpected: 0 (0 unexpected fail, 0 unexpected pass, 0 unexpected asserts, 0 unexpected fixed asserts, 0 failed load, 0 exception)
REFTEST INFO | Known problems: 30 (7 known fail, 0 known asserts, 3 random, 20 skipped, 0 slow)
REFTEST INFO | Total canvas count = 3
REFTEST TEST-START | Shutdown

INFO | automation.py | Application ran for: 0:01:04.336465
INFO | zombiecheck | Reading PID log: /tmp/tmpByIB0Apidlog
Contents of /data/anr/traces.txt:


MOZ_UPLOAD_DIR not defined; tombstone check skipped
WARNING | leakcheck | refcount logging is off, so leaks can't be detected!

REFTEST INFO | runreftest.py | Running tests: end.

Device info:
  uptime:  18:02:27 up 9 min,  0 users,  load average: 0.54, 0.74, 0.50
  memtotal: 646228 kB
  process:
     PID   USER     TIME   COMMAND
         1 root       0:01 /init
         2 root       0:00 [kthreadd]
         3 root       0:00 [ksoftirqd/0]
         4 root       0:03 [kworker/0:0]
         5 root       0:00 [kworker/u:0]
         6 root       0:00 [migration/0]
         7 root       0:00 [migration/1]
         9 root       0:00 [ksoftirqd/1]
        10 root       0:00 [khelper]
        11 root       0:00 [suspend]
        12 root       0:00 [irq/329-pogo_de]
        13 root       0:00 [sync_supers]
        14 root       0:00 [bdi-default]
        15 root       0:00 [kblockd]
        16 root       0:00 [omap2_mcspi]
        17 root       0:00 [khubd]
        18 root       0:00 [twl6030-irq]
        19 root       0:00 [irq/374-TWL6030]
        20 root       0:00 [kworker/0:1]
        21 root       0:00 [cfg80211]
        22 root       0:00 [musb-otg]
        23 root       0:01 [kinteractiveup]
        25 root       0:00 [khungtaskd]
        26 root       0:00 [kswapd0]
        27 root       0:00 [fsnotify_mark]
        28 root       0:00 [crypto]
        41 root       0:00 [hdmi_hotplug]
        42 root       0:00 [hdcp]
        44 root       0:00 [dsscomp_apply]
        45 root       0:00 [dsscomp_apply]
        46 root       0:00 [dsscomp_apply]
        47 root       0:00 [dsscomp_cb]
        48 root       0:00 [irq/335-sii9234]
        50 root       0:00 [pvr_timer]
        51 root       0:00 [bmp180_wq]
        52 root       0:00 [sec_jack_wq]
        54 root       0:00 [irq/184-fsa9480]
        55 root       0:00 [irq/164-fsa9480]
        56 root       0:00 [irq/159-temp_se]
        57 root       0:00 [irq/246-tshut]
        58 root       0:00 [irq/151-twl6040]
        59 root       0:00 [mtdblock0]
        66 root       0:00 [f_mtp]
        67 root       0:00 [file-storage]
        69 root       0:00 [gp2a_wq]
        70 root       0:00 [irq/379-rtc0]
        71 root       0:00 [irq/204-fuel ga]
        72 root       0:00 [khotplug]
        73 root       0:00 [binder]
        74 root       0:00 [irq/131-ABE]
        75 root       0:00 [kworker/u:2]
        76 root       0:00 [twl6040-codec]
        77 root       0:00 [twl6040-hf]
        78 root       0:00 [twl6040-hs]
        79 root       0:00 [twl6040-ep]
        80 root       0:12 [mmcqd/0]
        81 root       0:00 [mmcqd/0boot0]
        82 root       0:00 [mmcqd/0boot1]
        83 root       0:00 [kworker/1:1]
        84 root       0:00 [krfcommd]
        85 root       0:00 [irq/142-omap_em]
        86 root       0:00 [irq/143-omap_em]
        87 root       0:00 [mipi_tx_wq]
        88 root       0:00 [mipi_tx_raw_wq]
        89 root       0:00 [mipi_cmd_wq]
        90 root       0:00 [kworker/u:1]
        91 root       0:00 [dhd_watchdog]
        92 root       0:00 [dhd_dpc]
        93 root       0:00 [dhd_sysioc]
        94 root       0:00 /sbin/ueventd
        98 root       0:00 [jbd2/mmcblk0p10]
        99 root       0:00 [ext4-dio-unwrit]
       100 root       0:03 [jbd2/mmcblk0p12]
       101 root       0:00 [ext4-dio-unwrit]
       102 root       0:00 [irq/206-mms_ts]
       103 root       0:00 [jbd2/mmcblk0p11]
       104 root       0:00 [ext4-dio-unwrit]
       105 root       0:00 [jbd2/mmcblk0p3-]
       106 root       0:00 [ext4-dio-unwrit]
       107 1000       0:00 /system/bin/servicemanager
       108 root       0:00 /system/bin/vold
       113 root       0:00 /system/bin/netd
       114 root       0:02 /system/bin/debuggerd
       115 1001       0:01 /system/bin/rild
       116 1000       0:03 /system/bin/surfaceflinger
       117 root       0:04 zygote /bin/app_process -Xzygote /system/bin --zygote --s
       118 1019       0:00 /system/bin/drmserver
       119 1013       0:00 /system/bin/mediaserver
       120 1002       0:00 /system/bin/dbus-daemon --system --nofork
       121 root       0:00 /system/bin/installd
       122 1017       0:00 /system/bin/keystore /data/misc/keystore
       124 1023       0:00 /system/bin/sdcard /data/media 1023 1023
       125 2000       0:00 /system/bin/sh
       131 root       0:04 /sbin/adbd
       140 root       0:00 [pvr_workqueue]
       155 root       0:00 [omaplfb]
       167 root       0:00 [kworker/u:3]
       174 root       0:00 [flush-179:0]
       182 1000       0:25 system_server
       244 1010       0:00 /system/bin/wpa_supplicant -Dnl80211 -iwlan0 -puse_p2p_gr
       255 1000       0:02 {ndroid.systemui} com.android.systemui
       333 1000       0:00 {ndroid.settings} com.android.settings
       351 10017      0:00 {putmethod.latin} com.android.inputmethod.latin
       369 1001       0:00 {m.android.phone} com.android.phone
       387 1025       0:00 {om.android.nfc3} com.android.nfc3
       401 10018      0:05 {ndroid.launcher} com.android.launcher
       424 10012      0:34 {d.process.media} android.process.media
       457 10037      0:00 {android.smspush} com.android.smspush
       475 10000      0:00 {d.process.acore} android.process.acore
       541 10000      0:00 {ndroid.contacts} com.android.contacts
       570 10006      0:00 {viders.calendar} com.android.providers.calendar
       593 10010      0:00 {droid.deskclock} com.android.deskclock
       611 10013      0:00 {m.android.email} com.android.email
       630 10014      0:00 {ndroid.exchange} com.android.exchange
       690 10036      0:00 {oid.voicedialer} com.android.voicedialer
       717 1014       0:00 /system/bin/dhcpcd -ABKL -h android-350e0001b3dc37da wlan
       739 10005      0:00 {ndroid.calendar} com.android.calendar
       798 root       0:00 [kworker/1:2]
       869 root       0:00 [kworker/1:0]
       880 root       0:00 [kworker/u:1]
       914 10015      0:00 {droid.gallery3d} com.android.gallery3d
       929 1000       0:00 {ndroid.keychain} com.android.keychain
       942 10009      0:00 {id.defcontainer} com.android.defcontainer
       958 10023      0:00 {android.musicfx} com.android.musicfx
       972 10029      0:00 {.quicksearchbox} com.android.quicksearchbox
      1009 10026      0:00 com.svox.pico
      1202 root       0:00 [flush-0:17]
      1280 root       0:00 /system/bin/sh -c ps; echo $?
      1282 root       0:00 ps
  systime: Wed Mar 11 18:02:27 UTC 2015
  disk:
     Filesystem           1K-blocks      Used Available Use% Mounted on
     /dev/block/platform/omap/omap_hsmmc.0/by-name/userdata
                           13975888    369740  13606148   3% /data
     /dev/block/platform/omap/omap_hsmmc.0/by-name/system
                             659152    165980    493172  25% /system
     /dev/fuse             13975888    369740  13606148   3% /mnt/sdcard
  os: full_maguro-eng 4.0.1 ITL41F eng.mozdev.20120306.224313 test-keys
  id: 01498B300600B008
Test root: /sdcard/tests
/bin/sh: 12: @errors=: not found
Your output says this:

window size = 720,1134;

whereas mine says 0,0 in the same spot. Do you know where that comes from?
Ah, it looks like 0-length timeout at [1] isn't long enough. When I bump it up to 1000 it works.

[1] http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/bootstrap.js?rev=e51a54f1817d#31
Summary: No way to run reftests on Fennec locally → Running reftests on Fennec locally sometimes fails because the window is zero-sized
Interesting.

Something else I noticed: reftests work as I showed above with a new install, but if I run the same reftest job again, I get a 0,0 window size and the tests fail. Uninstall/install firefox and all is well again (until the next run).
Attachment #8576220 - Flags: review?(jmaher)
Attachment #8576220 - Flags: review?(gbrown)
Assignee: gbrown → bugmail.mozilla
Comment on attachment 8576220 [details] [diff] [review]
Robustify startup

Review of attachment 8576220 [details] [diff] [review]:
-----------------------------------------------------------------

Works great for me. Thanks!
Attachment #8576220 - Flags: review?(gbrown) → review+
Comment on attachment 8576220 [details] [diff] [review]
Robustify startup

Review of attachment 8576220 [details] [diff] [review]:
-----------------------------------------------------------------

::: layout/tools/reftest/bootstrap.js
@@ +13,5 @@
>  }
>  
> +function startRefTestsWhenReady(win) {
> +    if (win.innerHeight == 0) {
> +        win.setTimeout(startRefTestsWhenReady, 100, win);

can we put a log message here indicating we are waiting 100ms for win.innerHeight != 0 ?  this way if we get in a loop it can be easier to detect if this is the cause!
Attachment #8576220 - Flags: review?(jmaher) → review+
Thanks! I added a dump statement to that effect and pushed to try to make sure it doesn't bust anything. If that's green I'll land it.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=aaef04ef628f
I don't know wtf is going on here.. that try job has lots of failures (android 11+ only). Another variant that I pushed at https://treeherder.mozilla.org/#/jobs?repo=try&revision=f2c35c858017 shows similar results. What's different about android 11+ vs android 9?
After many many try pushes it looks like there's something fundamentally wrong with the testing framework. See for example https://treeherder.mozilla.org/#/jobs?repo=try&revision=4ad1197b7327 which just delays the test start by 10 seconds and causes everything to fail. It looks like the tests *expect* to run with a window innerHeight of 0 and if they don't then they fail. There must be a race somewhere. A delay of 1 second is still (mostly) acceptable, https://treeherder.mozilla.org/#/jobs?repo=try&revision=2fd647ddf545
Gonna give up on this for now. If that means Fennec can't have new features, so be it.
Assignee: bugmail.mozilla → nobody
You need to log in before you can comment on or make changes to this bug.