1620827 - Remove raptor-webext local storage clear call for causing lots of intermittents

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P1)

Description

[Greg Mierzwinski [:sparky]]

It seems that the storage clear call in the raptor-webext is causing many intermittent failures because of an unknown error:

03-06 18:14:31.788  8014  8080 E GeckoConsole: [JavaScript Error: "IndexedDB UnknownErr: ActorsParent.cpp:613"]
03-06 18:14:31.789  8014  8029 E GeckoConsole: [JavaScript Error: "UnknownError: The operation failed for reasons unrelated to the database itself and not covered by any other error code." {file: "resource://gre/modules/ExtensionStorageIDB.jsm" line: 812}]

I tested removing it and everything works fine without it, I haven't found any issues with that change. Here's a try run showing that the intermittent rate is drastically reduced with this change: https://treeherder.mozilla.org/#/jobs?repo=try&revision=089bd80b6ca40d2c8434c1a9d94a4b539993ee21

For comparison, here's a push that still has the clear call: https://treeherder.mozilla.org/#/jobs?repo=try&revision=bcc0fce02e7fdda70b032ed30903018e318bc76f

This bug is for removing the clear call, and making a couple other minor adjustments.

Comment 1

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1620827 - Remove raptor-webext clear storage call for causing intermittents. r?#perftest

This patch removes a storage.local.clear() call from raptor-webext for causing a high number of intermittent failures. It also adds a default value for --host in the mozharness raptor. Finally, a delayed startup for raptorRunner is added with the hope that this will further decrease intermittents (right now, it might be starting at a very noisy time).

Comment 2

[Greg Mierzwinski [:sparky]]

Try run using this patch: https://treeherder.mozilla.org/#/jobs?repo=try&revision=db6df2226e158bb9ed1d4944efc8baba8f257c2c

Comment 3

[Greg Mierzwinski [:sparky]]

Note that the Fenix failures are expected since the latest Fenix build is failing to update the tabs:

03-08 13:11:18.987  7528  7556 I Gecko   : console.info: "[raptor-runnerjs] pageload test start"
03-08 13:11:22.025  7528  7556 W GeckoEventDispatcher: No listener for GeckoView:WebExtension:UpdateTab

The fenix branch (without these changes) has the same failures: https://treeherder.mozilla.org/#/jobs?repo=fenix&selectedJob=292141083

03-08 01:19:56.102  9233  9263 W GeckoEventDispatcher: No listener for GeckoView:WebExtension:UpdateTab

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0ea89a168b00
Remove raptor-webext clear storage call for causing intermittents. r=perftest-reviewers,Bebe

Comment 5

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/0ea89a168b00

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Thanks for the (temporarily) removal of the clear() call. I assume it was/is necessary when re-using the same profile across different Raptor tests. I assume we still have warm tests? A simple solution might have been to wrap the code in try/catch instead and only log the error but ignore the rest.

Also regarding the delay, why is it necessary? We already have the post startup delay for any kind of noise, and sleeping another 5s now hardly conflicts with this setting. Also at this point I don't see why the delay is necessary at all.

Comment 7

[Greg Mierzwinski [:sparky]]

That's what I'm thinking as well. That said, I went through the 2+ years of history on that line of code (going back to the github repo) and I haven't found any mention of this, or any other reason it might be there.

We recently reduced the post-startup-delay to 1 second so it's more or less non-existent now. I added this in the hopes that it would help a bit with intermittents but I don't think it has had any effect. I'll make a patch to remove the delay.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Greg Mierzwinski [:sparky] from comment #7)

We recently reduced the post-startup-delay to 1 second so it's more or less non-existent now. I added this in the hopes that it would help a bit with intermittents but I don't think it has had any effect. I'll make a patch to remove the delay.

No need for a patch. I have included it in my patch series for bug 1609295.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I would like to see this test-only patch uplifted to beta given that it blocks the uplift of bug 1622213.

Comment 10

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/releases/mozilla-beta/rev/a2131f596c04