1644913 - Intermittent devtools/server/tests/xpcshell/test_extension_storage_actor.js | xpcshell return code: 0

Status: RESOLVED FIXED

(Thunderbird :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: geoff [at] darktrojan.net
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=305768757&repo=comm-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/dMRVePRAS_mmOxnkhlxZ1g/runs/0/artifacts/public/logs/live_backing.log

---

This is a near-permanent failure, on Windows only. I suspect bug 1620243 has caused it to happen.

https://hg.mozilla.org/mozilla-central/pushloghtml?tochange=b2df79a80c0303df9d710800ae37dce56847eef5&fromchange=7f7b983390650cbc7d736e92fd3e1f629a30ac02

Comment 1

[Geoff Lankow (:darktrojan)]

Cc'ing some devtools people. Could you have a look at this, please?

Comment 2

[Geoff Lankow (:darktrojan)]

Apparently it happens on Mac too.

Comment 3

[Intermittent Failures Robot]

21 failures in 1067 pushes (0.02 failures/push) were associated with this bug yesterday.

Repository breakdown:

• comm-central: 21

Platform breakdown:

• windows7-32: 5
• windows7-32-shippable: 5
• windows10-64: 6
• windows10-64-shippable: 4
• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1644913&startday=2020-06-10&endday=2020-06-10&tree=all

Comment 4

[Julian Descottes [:jdescottes]]

This is probably related to https://bugzilla.mozilla.org/show_bug.cgi?id=1583738 which changed the way we connect to Addon targets.

I don't remember exactly what happened with thunderbird when we removed the ability to debug legacy addons in mozilla central, but I remember talking about duplicating the old DevTools addon target actor and keep it only in TB's codebase ?
I can find an old webextension target actor at https://searchfox.org/comm-central/source/mozilla/devtools/server/actors/targets/webextension.js but I can't find the commit log for this file. And all the content of devtools/server/actors on comm-central searchfox is really old.

Geoff: Does the content on searchfox correspond to what is currently used by comm-central? Otherwise, how can I check which files of mozilla central are used?

Assuming we indeed have an old WebExtensionTargetActor in comm-central, it might just be about the last commit of Bug 1583738, where we started "attaching" by default to webextension targets, since the error in the logs is about a failure while calling "detach". On DevTools side, we catch detach errors, but it seems the test harness is still catching it and considering as a test failure.
But I need to understand comm-central's setup around devtools webextension targets better before working on a fix.

Comment 5

[Geoff Lankow (:darktrojan)]

(In reply to Julian Descottes [:jdescottes] from comment #4)

I don't remember exactly what happened with thunderbird when we removed the ability to debug legacy addons in mozilla central, but I remember talking about duplicating the old DevTools addon target actor and keep it only in TB's codebase ?

Hmm, there shouldn't be any of that hanging around any more. All of our add-ons have been WebExtensions (technically at least) since 68.

I can find an old webextension target actor at https://searchfox.org/comm-central/source/mozilla/devtools/server/actors/targets/webextension.js but I can't find the commit log for this file. And all the content of devtools/server/actors on comm-central searchfox is really old.

Geoff: Does the content on searchfox correspond to what is currently used by comm-central? Otherwise, how can I check which files of mozilla central are used?

That's mozilla-central code (everything under https://searchfox.org/comm-central/source/mozilla is, and AFAIK it's usually up-to-date).

Comment 6

[Julian Descottes [:jdescottes]]

(In reply to Geoff Lankow (:darktrojan) from comment #5)

Hmm, there shouldn't be any of that hanging around any more. All of our add-ons have been WebExtensions (technically at least) since 68.

Thanks, that's good to know.

That's mozilla-central code (everything under https://searchfox.org/comm-central/source/mozilla is, and AFAIK it's usually up-to-date).

Right sorry, I was confused between the webextensiondescriptor and the webextensiontarget. The code looks up to date.
With that I'm not exactly sure why this error pops up only on comm-central. I still suspect https://hg.mozilla.org/mozilla-central/rev/623d3cc37bb4 to be the root cause, since the error seems to arise when we detach the webextension target.

Comment 7

[Julian Descottes [:jdescottes]]

It seems like the failure also happens on central https://bugzilla.mozilla.org/show_bug.cgi?id=1618059

Comment 8

[Julian Descottes [:jdescottes]]

I think the addon failures are not directly responsible for the test failure actually. I got the same failures locally.
The error that seems to trigger the test failure is

TEST-UNEXPECTED-FAIL | devtools/server/tests/xpcshell/test_extension_storage_actor.js | test_panel_live_reload - [test_panel_live_reload : 265] A promise chain failed to handle a rejection: can't access property "update", this.storageActor is null - stack: onWindowReady@resource://devtools/server/actors/storage.js:281:9

Which means that in the following method:

    async onWindowReady(window) {
      const host = this.getHostName(window.location);
      if (host && !this.hostVsStores.has(host)) {
        await this.populateStoresForHost(host, window);
        const data = {};
        data[host] = this.getNamesForHost(host);
        this.storageActor.update("added", typeName, data);
      }
    },

the storage actor gets destroyed while we wait for this.populateStoresForHost(host, window);
Given how the method is written, it's definitely exposed to such race conditions.

The first failure on mozilla-central also happened slightly prior to both Bug 1620243 and Bug 1583738. https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=7f7b983390650cbc7d736e92fd3e1f629a30ac02&selectedTaskRun=AVrDWr5aTgqW6VjBMJhvDg-0

It might be linked to something else than these two bugs, which makes populateStoresForHost run slower than usual?

Comment 9

[Julian Descottes [:jdescottes]]

Attachment: Bug 1644913 - Avoid exceptions if the storage actor is destroyed while processing a new window

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b91a23c94b9a
Avoid exceptions if the storage actor is destroyed while processing a new window r=ochameau

Comment 11

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/b91a23c94b9a

Comment 12

[Intermittent Failures Robot]

19 failures in 1000 pushes (0.019 failures/push) were associated with this bug yesterday.

Repository breakdown:

• comm-central: 19

Platform breakdown:

• windows7-32: 6
• windows10-64: 5
• windows10-64-shippable: 4
• windows7-32-shippable: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1644913&startday=2020-06-11&endday=2020-06-11&tree=all

Comment 13

[Geoff Lankow (:darktrojan)]

Thank you Julian.

Comment 14

[Intermittent Failures Robot]

40 failures in 5211 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

This is the #43 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• comm-central: 40

Platform breakdown:

• windows7-32: 11
• windows7-32-shippable: 9
• windows10-64: 11
• windows10-64-shippable: 8
• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1644913&startday=2020-06-08&endday=2020-06-14&tree=all