Closed
Bug 1142129
Opened 9 years ago
Closed 4 years ago
Running reftests on Fennec locally sometimes fails because the window is zero-sized
Categories
(Testing :: Reftest, defect)
Tracking
(Not tracked)
RESOLVED
INCOMPLETE
People
(Reporter: kats, Unassigned)
References
Details
Attachments
(1 file)
|
2.24 KB,
patch
|
jmaher
:
review+
gbrown
:
review+
|
Details | Diff | Splinter Review |
There appears to be no way to run reftests locally on Fennec. There used to be some instructions on the wiki at [1] and while those appeared to work, they didn't actually produce the correct output, and do not replicate results from tryserver. Specifically I would always get garbage drawn to the screen, and the tests would always fail. Neither mach reftest nor mach reftest-remote appears to support Fennec either. [1] https://wiki.mozilla.org/Mobile/Fennec/Android#Reftests
|
||
Comment 1•9 years ago
|
||
We never got around to adding mach support, but the make target and old docs "should" have worked. I'll check into it.
Assignee: nobody → gbrown
|
|
||
Comment 2•9 years ago
|
||
This works fine for me on my Galaxy Nexus running Android 4.0.1: export TEST_PATH=layout/reftests/reftest-sanity/reftest.list export MOZ_HOST_BIN=<path to my host utils> make reftest-remote I see the tests running on the phone, all tests pass, and the log looks okay.
|
|
||
Comment 3•9 years ago
|
||
gbrown@mozpad:~/objdirs/droid$ make reftest-remote
INFO | automation.py | Launching: /home/gbrown/hostutils-march2015/host-utils-37.0a2.en-US.linux-x86_64/xpcshell -g /home/gbrown/hostutils-march2015/host-utils-37.0a2.en-US.linux-x86_64 -v 170 -f /home/gbrown/objdirs/droid/_tests/modules/httpd.js -e "const _PROFILE_PATH = '/tmp/tmpr_44SS';const _SERVER_PORT = '8888'; const _SERVER_ADDR ='192.168.0.82';" -f /home/gbrown/objdirs/droid/_tests/reftest/server.js
INFO | remotereftests.py | Server pid: 21476
Device info:
uptime: 18:01:18 up 8 min, 0 users, load average: 0.53, 0.77, 0.50
memtotal: 646228 kB
process:
PID USER TIME COMMAND
1 root 0:01 /init
2 root 0:00 [kthreadd]
3 root 0:00 [ksoftirqd/0]
4 root 0:03 [kworker/0:0]
5 root 0:00 [kworker/u:0]
6 root 0:00 [migration/0]
7 root 0:00 [migration/1]
9 root 0:00 [ksoftirqd/1]
10 root 0:00 [khelper]
11 root 0:00 [suspend]
12 root 0:00 [irq/329-pogo_de]
13 root 0:00 [sync_supers]
14 root 0:00 [bdi-default]
15 root 0:00 [kblockd]
16 root 0:00 [omap2_mcspi]
17 root 0:00 [khubd]
18 root 0:00 [twl6030-irq]
19 root 0:00 [irq/374-TWL6030]
20 root 0:00 [kworker/0:1]
21 root 0:00 [cfg80211]
22 root 0:00 [musb-otg]
23 root 0:01 [kinteractiveup]
25 root 0:00 [khungtaskd]
26 root 0:00 [kswapd0]
27 root 0:00 [fsnotify_mark]
28 root 0:00 [crypto]
41 root 0:00 [hdmi_hotplug]
42 root 0:00 [hdcp]
44 root 0:00 [dsscomp_apply]
45 root 0:00 [dsscomp_apply]
46 root 0:00 [dsscomp_apply]
47 root 0:00 [dsscomp_cb]
48 root 0:00 [irq/335-sii9234]
50 root 0:00 [pvr_timer]
51 root 0:00 [bmp180_wq]
52 root 0:00 [sec_jack_wq]
54 root 0:00 [irq/184-fsa9480]
55 root 0:00 [irq/164-fsa9480]
56 root 0:00 [irq/159-temp_se]
57 root 0:00 [irq/246-tshut]
58 root 0:00 [irq/151-twl6040]
59 root 0:00 [mtdblock0]
66 root 0:00 [f_mtp]
67 root 0:00 [file-storage]
69 root 0:00 [gp2a_wq]
70 root 0:00 [irq/379-rtc0]
71 root 0:00 [irq/204-fuel ga]
72 root 0:00 [khotplug]
73 root 0:00 [binder]
74 root 0:00 [irq/131-ABE]
75 root 0:00 [kworker/u:2]
76 root 0:00 [twl6040-codec]
77 root 0:00 [twl6040-hf]
78 root 0:00 [twl6040-hs]
79 root 0:00 [twl6040-ep]
80 root 0:12 [mmcqd/0]
81 root 0:00 [mmcqd/0boot0]
82 root 0:00 [mmcqd/0boot1]
83 root 0:00 [kworker/1:1]
84 root 0:00 [krfcommd]
85 root 0:00 [irq/142-omap_em]
86 root 0:00 [irq/143-omap_em]
87 root 0:00 [mipi_tx_wq]
88 root 0:00 [mipi_tx_raw_wq]
89 root 0:00 [mipi_cmd_wq]
90 root 0:00 [kworker/u:1]
91 root 0:00 [dhd_watchdog]
92 root 0:00 [dhd_dpc]
93 root 0:00 [dhd_sysioc]
94 root 0:00 /sbin/ueventd
98 root 0:00 [jbd2/mmcblk0p10]
99 root 0:00 [ext4-dio-unwrit]
100 root 0:03 [jbd2/mmcblk0p12]
101 root 0:00 [ext4-dio-unwrit]
102 root 0:00 [irq/206-mms_ts]
103 root 0:00 [jbd2/mmcblk0p11]
104 root 0:00 [ext4-dio-unwrit]
105 root 0:00 [jbd2/mmcblk0p3-]
106 root 0:00 [ext4-dio-unwrit]
107 1000 0:00 /system/bin/servicemanager
108 root 0:00 /system/bin/vold
113 root 0:00 /system/bin/netd
114 root 0:00 /system/bin/debuggerd
115 1001 0:01 /system/bin/rild
116 1000 0:01 /system/bin/surfaceflinger
117 root 0:04 zygote /bin/app_process -Xzygote /system/bin --zygote --s
118 1019 0:00 /system/bin/drmserver
119 1013 0:00 /system/bin/mediaserver
120 1002 0:00 /system/bin/dbus-daemon --system --nofork
121 root 0:00 /system/bin/installd
122 1017 0:00 /system/bin/keystore /data/misc/keystore
124 1023 0:00 /system/bin/sdcard /data/media 1023 1023
125 2000 0:00 /system/bin/sh
131 root 0:03 /sbin/adbd
140 root 0:00 [pvr_workqueue]
155 root 0:00 [omaplfb]
167 root 0:00 [kworker/u:3]
174 root 0:00 [flush-179:0]
182 1000 0:23 system_server
244 1010 0:00 /system/bin/wpa_supplicant -Dnl80211 -iwlan0 -puse_p2p_gr
255 1000 0:02 {ndroid.systemui} com.android.systemui
333 1000 0:00 {ndroid.settings} com.android.settings
351 10017 0:00 {putmethod.latin} com.android.inputmethod.latin
369 1001 0:00 {m.android.phone} com.android.phone
387 1025 0:00 {om.android.nfc3} com.android.nfc3
401 10018 0:05 {ndroid.launcher} com.android.launcher
424 10012 0:34 {d.process.media} android.process.media
457 10037 0:00 {android.smspush} com.android.smspush
475 10000 0:00 {d.process.acore} android.process.acore
541 10000 0:00 {ndroid.contacts} com.android.contacts
570 10006 0:00 {viders.calendar} com.android.providers.calendar
593 10010 0:00 {droid.deskclock} com.android.deskclock
611 10013 0:00 {m.android.email} com.android.email
630 10014 0:00 {ndroid.exchange} com.android.exchange
649 10021 0:00 com.android.mms
690 10036 0:00 {oid.voicedialer} com.android.voicedialer
717 1014 0:00 /system/bin/dhcpcd -ABKL -h android-350e0001b3dc37da wlan
739 10005 0:00 {ndroid.calendar} com.android.calendar
798 root 0:00 [kworker/1:2]
869 root 0:00 [kworker/1:0]
880 root 0:00 [kworker/u:1]
914 10015 0:00 {droid.gallery3d} com.android.gallery3d
929 1000 0:00 {ndroid.keychain} com.android.keychain
942 10009 0:00 {id.defcontainer} com.android.defcontainer
958 10023 0:00 {android.musicfx} com.android.musicfx
972 10029 0:00 {.quicksearchbox} com.android.quicksearchbox
1009 10026 0:00 com.svox.pico
1065 root 0:00 /system/bin/sh -c ps; echo $?
1067 root 0:00 ps
systime: Wed Mar 11 18:01:18 UTC 2015
disk:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/block/platform/omap/omap_hsmmc.0/by-name/userdata
13975888 369124 13606764 3% /data
/dev/block/platform/omap/omap_hsmmc.0/by-name/system
659152 165980 493172 25% /system
/dev/fuse 13975888 369124 13606764 3% /mnt/sdcard
os: full_maguro-eng 4.0.1 ITL41F eng.mozdev.20120306.224313 test-keys
id: 01498B300600B008
Test root: /sdcard/tests
REFTEST INFO | runreftest.py | Running tests: start.
INFO | automation.py | Application pid: 10038
REFTEST INFO | Discovered 120 tests, after filtering SKIP tests, we have 111
REFTEST TEST-START | data:text/html,<body>
REFTEST TEST-LOAD | data:text/html,<body> | 0 / 120 (0%)
REFTEST INFO | [CONTENT] Using browser remote=false
REFTEST INFO | drawWindow flags = DRAWWINDOW_DRAW_CARET | DRAWWINDOW_DRAW_VIEW | DRAWWINDOW_USE_WIDGET_LAYERS; window size = 720,1134; test browser size = 800,1000
REFTEST TEST-LOAD | about:blank | 0 / 120 (0%)
REFTEST TEST-PASS | data:text/html,<body> | image comparison (==)
REFTEST INFO | Loading a blank page
REFTEST TEST-END | data:text/html,<body>
REFTEST TEST-START | data:text/plain,
REFTEST TEST-LOAD | data:text/plain, | 1 / 120 (0%)
REFTEST TEST-PASS | data:text/plain, | image comparison (==)
REFTEST INFO | Loading a blank page
REFTEST TEST-END | data:text/plain,
REFTEST TEST-START | data:text/plain,HELLO
REFTEST TEST-LOAD | data:text/plain,HELLO | 2 / 120 (1%)
REFTEST TEST-PASS | data:text/plain,HELLO | image comparison (!=)
REFTEST INFO | Loading a blank page
REFTEST TEST-END | data:text/plain,HELLO
REFTEST TEST-START | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/test-async.xul
REFTEST TEST-LOAD | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/test-async.xul | 3 / 120 (2%)
REFTEST TEST-LOAD | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/test-async-ref.xul | 3 / 120 (2%)
REFTEST TEST-PASS | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/test-async.xul | image comparison (==)
REFTEST INFO | Loading a blank page
REFTEST TEST-END | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/test-async.xul
...
REFTEST TEST-START | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/reftest-no-paint.html
REFTEST TEST-LOAD | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/reftest-no-paint.html | 117 / 120 (97%)
REFTEST TEST-LOAD | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/reftest-no-paint-ref.html | 117 / 120 (97%)
REFTEST TEST-KNOWN-FAIL | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/reftest-no-paint.html | failed reftest-no-paint
REFTEST INFO | Loading a blank page
REFTEST TEST-END | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/reftest-no-paint.html
REFTEST TEST-KNOWN-FAIL | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/async-scroll-1a.html | (SKIP)
REFTEST TEST-KNOWN-FAIL | http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/async-scroll-1b.html | (SKIP)
REFTEST FINISHED: Slowest test took 1032ms (http://192.168.0.82:8888/tests/layout/reftests/reftest-sanity/invalidation.html)
REFTEST INFO | Result summary:
REFTEST INFO | Successful: 91 (84 pass, 7 load only)
REFTEST INFO | Unexpected: 0 (0 unexpected fail, 0 unexpected pass, 0 unexpected asserts, 0 unexpected fixed asserts, 0 failed load, 0 exception)
REFTEST INFO | Known problems: 30 (7 known fail, 0 known asserts, 3 random, 20 skipped, 0 slow)
REFTEST INFO | Total canvas count = 3
REFTEST TEST-START | Shutdown
INFO | automation.py | Application ran for: 0:01:04.336465
INFO | zombiecheck | Reading PID log: /tmp/tmpByIB0Apidlog
Contents of /data/anr/traces.txt:
MOZ_UPLOAD_DIR not defined; tombstone check skipped
WARNING | leakcheck | refcount logging is off, so leaks can't be detected!
REFTEST INFO | runreftest.py | Running tests: end.
Device info:
uptime: 18:02:27 up 9 min, 0 users, load average: 0.54, 0.74, 0.50
memtotal: 646228 kB
process:
PID USER TIME COMMAND
1 root 0:01 /init
2 root 0:00 [kthreadd]
3 root 0:00 [ksoftirqd/0]
4 root 0:03 [kworker/0:0]
5 root 0:00 [kworker/u:0]
6 root 0:00 [migration/0]
7 root 0:00 [migration/1]
9 root 0:00 [ksoftirqd/1]
10 root 0:00 [khelper]
11 root 0:00 [suspend]
12 root 0:00 [irq/329-pogo_de]
13 root 0:00 [sync_supers]
14 root 0:00 [bdi-default]
15 root 0:00 [kblockd]
16 root 0:00 [omap2_mcspi]
17 root 0:00 [khubd]
18 root 0:00 [twl6030-irq]
19 root 0:00 [irq/374-TWL6030]
20 root 0:00 [kworker/0:1]
21 root 0:00 [cfg80211]
22 root 0:00 [musb-otg]
23 root 0:01 [kinteractiveup]
25 root 0:00 [khungtaskd]
26 root 0:00 [kswapd0]
27 root 0:00 [fsnotify_mark]
28 root 0:00 [crypto]
41 root 0:00 [hdmi_hotplug]
42 root 0:00 [hdcp]
44 root 0:00 [dsscomp_apply]
45 root 0:00 [dsscomp_apply]
46 root 0:00 [dsscomp_apply]
47 root 0:00 [dsscomp_cb]
48 root 0:00 [irq/335-sii9234]
50 root 0:00 [pvr_timer]
51 root 0:00 [bmp180_wq]
52 root 0:00 [sec_jack_wq]
54 root 0:00 [irq/184-fsa9480]
55 root 0:00 [irq/164-fsa9480]
56 root 0:00 [irq/159-temp_se]
57 root 0:00 [irq/246-tshut]
58 root 0:00 [irq/151-twl6040]
59 root 0:00 [mtdblock0]
66 root 0:00 [f_mtp]
67 root 0:00 [file-storage]
69 root 0:00 [gp2a_wq]
70 root 0:00 [irq/379-rtc0]
71 root 0:00 [irq/204-fuel ga]
72 root 0:00 [khotplug]
73 root 0:00 [binder]
74 root 0:00 [irq/131-ABE]
75 root 0:00 [kworker/u:2]
76 root 0:00 [twl6040-codec]
77 root 0:00 [twl6040-hf]
78 root 0:00 [twl6040-hs]
79 root 0:00 [twl6040-ep]
80 root 0:12 [mmcqd/0]
81 root 0:00 [mmcqd/0boot0]
82 root 0:00 [mmcqd/0boot1]
83 root 0:00 [kworker/1:1]
84 root 0:00 [krfcommd]
85 root 0:00 [irq/142-omap_em]
86 root 0:00 [irq/143-omap_em]
87 root 0:00 [mipi_tx_wq]
88 root 0:00 [mipi_tx_raw_wq]
89 root 0:00 [mipi_cmd_wq]
90 root 0:00 [kworker/u:1]
91 root 0:00 [dhd_watchdog]
92 root 0:00 [dhd_dpc]
93 root 0:00 [dhd_sysioc]
94 root 0:00 /sbin/ueventd
98 root 0:00 [jbd2/mmcblk0p10]
99 root 0:00 [ext4-dio-unwrit]
100 root 0:03 [jbd2/mmcblk0p12]
101 root 0:00 [ext4-dio-unwrit]
102 root 0:00 [irq/206-mms_ts]
103 root 0:00 [jbd2/mmcblk0p11]
104 root 0:00 [ext4-dio-unwrit]
105 root 0:00 [jbd2/mmcblk0p3-]
106 root 0:00 [ext4-dio-unwrit]
107 1000 0:00 /system/bin/servicemanager
108 root 0:00 /system/bin/vold
113 root 0:00 /system/bin/netd
114 root 0:02 /system/bin/debuggerd
115 1001 0:01 /system/bin/rild
116 1000 0:03 /system/bin/surfaceflinger
117 root 0:04 zygote /bin/app_process -Xzygote /system/bin --zygote --s
118 1019 0:00 /system/bin/drmserver
119 1013 0:00 /system/bin/mediaserver
120 1002 0:00 /system/bin/dbus-daemon --system --nofork
121 root 0:00 /system/bin/installd
122 1017 0:00 /system/bin/keystore /data/misc/keystore
124 1023 0:00 /system/bin/sdcard /data/media 1023 1023
125 2000 0:00 /system/bin/sh
131 root 0:04 /sbin/adbd
140 root 0:00 [pvr_workqueue]
155 root 0:00 [omaplfb]
167 root 0:00 [kworker/u:3]
174 root 0:00 [flush-179:0]
182 1000 0:25 system_server
244 1010 0:00 /system/bin/wpa_supplicant -Dnl80211 -iwlan0 -puse_p2p_gr
255 1000 0:02 {ndroid.systemui} com.android.systemui
333 1000 0:00 {ndroid.settings} com.android.settings
351 10017 0:00 {putmethod.latin} com.android.inputmethod.latin
369 1001 0:00 {m.android.phone} com.android.phone
387 1025 0:00 {om.android.nfc3} com.android.nfc3
401 10018 0:05 {ndroid.launcher} com.android.launcher
424 10012 0:34 {d.process.media} android.process.media
457 10037 0:00 {android.smspush} com.android.smspush
475 10000 0:00 {d.process.acore} android.process.acore
541 10000 0:00 {ndroid.contacts} com.android.contacts
570 10006 0:00 {viders.calendar} com.android.providers.calendar
593 10010 0:00 {droid.deskclock} com.android.deskclock
611 10013 0:00 {m.android.email} com.android.email
630 10014 0:00 {ndroid.exchange} com.android.exchange
690 10036 0:00 {oid.voicedialer} com.android.voicedialer
717 1014 0:00 /system/bin/dhcpcd -ABKL -h android-350e0001b3dc37da wlan
739 10005 0:00 {ndroid.calendar} com.android.calendar
798 root 0:00 [kworker/1:2]
869 root 0:00 [kworker/1:0]
880 root 0:00 [kworker/u:1]
914 10015 0:00 {droid.gallery3d} com.android.gallery3d
929 1000 0:00 {ndroid.keychain} com.android.keychain
942 10009 0:00 {id.defcontainer} com.android.defcontainer
958 10023 0:00 {android.musicfx} com.android.musicfx
972 10029 0:00 {.quicksearchbox} com.android.quicksearchbox
1009 10026 0:00 com.svox.pico
1202 root 0:00 [flush-0:17]
1280 root 0:00 /system/bin/sh -c ps; echo $?
1282 root 0:00 ps
systime: Wed Mar 11 18:02:27 UTC 2015
disk:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/block/platform/omap/omap_hsmmc.0/by-name/userdata
13975888 369740 13606148 3% /data
/dev/block/platform/omap/omap_hsmmc.0/by-name/system
659152 165980 493172 25% /system
/dev/fuse 13975888 369740 13606148 3% /mnt/sdcard
os: full_maguro-eng 4.0.1 ITL41F eng.mozdev.20120306.224313 test-keys
id: 01498B300600B008
Test root: /sdcard/tests
/bin/sh: 12: @errors=: not found
|
Reporter | |
Comment 4•9 years ago
|
||
Your output says this: window size = 720,1134; whereas mine says 0,0 in the same spot. Do you know where that comes from?
|
|
Reporter | |
Comment 5•9 years ago
|
||
Ah, it looks like 0-length timeout at [1] isn't long enough. When I bump it up to 1000 it works. [1] http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/bootstrap.js?rev=e51a54f1817d#31
|
|
Reporter | |
Updated•9 years ago
|
Summary: No way to run reftests on Fennec locally → Running reftests on Fennec locally sometimes fails because the window is zero-sized
|
|
||
Comment 6•9 years ago
|
||
Interesting. Something else I noticed: reftests work as I showed above with a new install, but if I run the same reftest job again, I get a 0,0 window size and the tests fail. Uninstall/install firefox and all is well again (until the next run).
|
|
Reporter | |
Comment 7•9 years ago
|
||
Attachment #8576220 -
Flags: review?(jmaher)
Attachment #8576220 -
Flags: review?(gbrown)
|
|
||
Updated•9 years ago
|
Assignee: gbrown → bugmail.mozilla
|
|
||
Comment 8•9 years ago
|
||
Comment on attachment 8576220 [details] [diff] [review] Robustify startup Review of attachment 8576220 [details] [diff] [review]: ----------------------------------------------------------------- Works great for me. Thanks!
Attachment #8576220 -
Flags: review?(gbrown) → review+
|
||
Comment 9•9 years ago
|
||
Comment on attachment 8576220 [details] [diff] [review] Robustify startup Review of attachment 8576220 [details] [diff] [review]: ----------------------------------------------------------------- ::: layout/tools/reftest/bootstrap.js @@ +13,5 @@ > } > > +function startRefTestsWhenReady(win) { > + if (win.innerHeight == 0) { > + win.setTimeout(startRefTestsWhenReady, 100, win); can we put a log message here indicating we are waiting 100ms for win.innerHeight != 0 ? this way if we get in a loop it can be easier to detect if this is the cause!
Attachment #8576220 -
Flags: review?(jmaher) → review+
|
|
Reporter | |
Comment 10•9 years ago
|
||
Thanks! I added a dump statement to that effect and pushed to try to make sure it doesn't bust anything. If that's green I'll land it. https://treeherder.mozilla.org/#/jobs?repo=try&revision=aaef04ef628f
|
|
Reporter | |
Comment 11•9 years ago
|
||
I don't know wtf is going on here.. that try job has lots of failures (android 11+ only). Another variant that I pushed at https://treeherder.mozilla.org/#/jobs?repo=try&revision=f2c35c858017 shows similar results. What's different about android 11+ vs android 9?
|
|
Reporter | |
Comment 12•9 years ago
|
||
After many many try pushes it looks like there's something fundamentally wrong with the testing framework. See for example https://treeherder.mozilla.org/#/jobs?repo=try&revision=4ad1197b7327 which just delays the test start by 10 seconds and causes everything to fail. It looks like the tests *expect* to run with a window innerHeight of 0 and if they don't then they fail. There must be a race somewhere. A delay of 1 second is still (mostly) acceptable, https://treeherder.mozilla.org/#/jobs?repo=try&revision=2fd647ddf545
|
|
Reporter | |
Comment 13•9 years ago
|
||
Gonna give up on this for now. If that means Fennec can't have new features, so be it.
Assignee: bugmail.mozilla → nobody
|
|
Reporter | |
Comment 14•4 years ago
|
||
Fennec is no more
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → INCOMPLETE
You need to log in
before you can comment on or make changes to this bug.
Description
•