908020 - tests 160 and 161 _appInUse take too long (top 8 duration)

Status: RESOLVED FIXED

(Toolkit :: Application Update, defect)

Description

[Mihnea Dobrescu-Balaur (:mihneadb)]

This test takes too long and it affects performance when running xpcshell tests in parallel. It should be made faster or split across multiple files.

You can see the duration data here[1]. It was collected on a linux x64 laptop.


[1] http://www.mihneadb.net/xpcshelltest-resource-usage/

Comment 1

[Robert Strong (they/them - no direct email)]

test_0160_appInUse_complete.js = 12796.579122543335
test_0161_appInUse_xp_unix_complete.js = 12353.29008102417

This is likely due to the helper app and that we have to use timeouts to make sure that the helper app has launched. I've submitted a patch to try that should lessen the time for the test to run though I suspect it will need additional tuning.

https://tbpl.mozilla.org/?tree=Try&rev=15a978335468

Comment 2

[Gregory Szorc [:gps]]

Rob: Thank you for promptly looking at these tests!

Comment 3

[Robert Strong (they/them - no direct email)]

Attachment: 1. improve timeout usage rev1

try looks good so far. The patch that increased the timeout was for an intermittent orange that I followed up with a real fix so reducing it should be fine. I think the rest of the changes should be fine as well.

Comment 4

[Robert Strong (they/them - no direct email)]

Attachment: patch rev1

For test_0160_appInUse_complete.js I printed out times for completion from loading the head file, for most of the functions and then intentionally threw at the end of the test.

I only tested this on Windows and Linux since the other try run looked good and there are minimal changes between this and the last patch. test_0160_appInUse_complete.js made it to the end where it threw so it looks like the additional changes are fine.
https://tbpl.mozilla.org/?tree=Try&rev=651cf56c2933

On Linux the xpcshell timestamp started at 02:42:26 and ended at 02:42:28 and the times I printed out added up to 1240 ms.

Comment 5

[Robert Strong (they/them - no direct email)]

Another try run just in case
https://tbpl.mozilla.org/?tree=Try&rev=e1d411daea33

Comment 6

[Robert Strong (they/them - no direct email)]

Pushed to fx-team
https://hg.mozilla.org/integration/fx-team/rev/30d13245ff58

Adding leave open keyword since there is a decent chance that the tests will still be slow though hopefully it will be closer to other update tests with these changes.

Mihnea, could you provide duration time with this patch applied?

Comment 7

[Mihnea Dobrescu-Balaur (:mihneadb)]

They still took ~12-13 seconds, which is what they took previously. I don't think the tests are hitting that timeout, at least on my laptop. :(

Comment 8

[Robert Strong (they/them - no direct email)]

These tests definitely use that timeout though not that often. I haven't been keeping track of the parallel xpcshell landing... has it landed yet? When you run the test by itself how long does it take?
From <objdir>/toolkit/mozapps/update/
make SOLO_FILE=test_0160_appInUse_complete.js -C test check-one

Comment 9

[Tim Taubert [:ttaubert] (inactive)]

https://hg.mozilla.org/mozilla-central/rev/30d13245ff58

Comment 10

[Robert Strong (they/them - no direct email)]

Mihnea, can I get you to provide the info in comment #8? Thanks

Comment 11

[Mihnea Dobrescu-Balaur (:mihneadb)]

(In reply to Robert Strong [:rstrong] (do not email) from comment #10)
> Mihnea, can I get you to provide the info in comment #8? Thanks

Sorry I missed it!

make SOLO_FILE=test_0160_appInUse_complete.js -C test check-one  0.42s user 0.18s system 4% cpu 13.242 total

So it still takes a while. Parallel xpcshell has landed but it is turned off in automation for now (ironing out some wrinkles with intermittent failures). If you use mach xpcshell-test locally, it will run tests in parallel by default (because those mentioned failures only happen on the test slaves).

Comment 12

[Mihnea Dobrescu-Balaur (:mihneadb)]

For the record, the situation is different on linux [1]. These tests are not a problem there.

[1] http://www.mihneadb.net/xpcshelltest-resource-usage/

Comment 13

[Robert Strong (they/them - no direct email)]

So, the numbers went from
test_0160_appInUse_complete.js = 12796.579122543335 to 1427.4768829345703
test_0161_appInUse_xp_unix_complete.js = 12353.29008102417 to 1441.99800491333

Mihnea, any idea what accounts for the change?

Comment 14

[Mihnea Dobrescu-Balaur (:mihneadb)]

I'd say linux vs os x. Also, I ran the latest batch in parallel. I'll do some more benchmarks later, I'm a bit busy on something else right now.

Comment 15

[Robert Strong (they/them - no direct email)]

Both datasets are from Linux per comment #0 and comment #12

Comment 16

[Mihnea Dobrescu-Balaur (:mihneadb)]

I did some more testing. I can confirm on OS X as well that at least 160_appInUse (since 161 is skipped) takes less time. Must have been your change.

I'd say we can close the bug, I'll reopen if I find the tests take longer than expected.

Thanks!

Comment 17

[Mihnea Dobrescu-Balaur (:mihneadb)]

I also confirmed with the data in TBPL logs.

Comment 18

[Robert Strong (they/them - no direct email)]

Thanks for the confirmation!