Closed Bug 1601079 Opened 6 years ago Closed 1 year ago

Intermittent Return code: 3221225477

Categories

(Core :: Security: Process Sandboxing, defect, P2)

Product:
Core
Component:
Security: Process Sandboxing
Platform:
All
Windows
Type:
defect

Tracking

()

Status:
RESOLVED INCOMPLETE

People

(Reporter: intermittent-bug-filer, Unassigned)

References

Details

(Keywords: intermittent-failure)

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=279396682&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/F9c9JMc_RsaSsfgLd5b6kw/runs/0/artifacts/public/logs/live_backing.log

[task 2019-12-03T19:25:44.377Z] 19:25:44 INFO - PID 7892 |
[task 2019-12-03T19:25:44.457Z] 19:25:44 INFO - PID 7892 | Cycle 1(4): loaded http://127.0.0.1:49777/tests/tp5n/indiatimes.com/www.indiatimes.com/index.html (next: http://127.0.0.1:49777/tests/tp5n/mail.ru/mail.ru/index.html)
[task 2019-12-03T19:25:44.516Z] 19:25:44 INFO - PID 7892 | MOZ_EVENT_TRACE sample 1575401144501 47.136744
[task 2019-12-03T19:25:44.889Z] 19:25:44 INFO - PID 7892 |
[task 2019-12-03T19:25:44.889Z] 19:25:44 INFO - PID 7892 | ###!!! [Child][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2019-12-03T19:25:44.889Z] 19:25:44 INFO - PID 7892 |
[task 2019-12-03T19:25:44.979Z] 19:25:44 INFO - PID 7892 | console.warn: LoginHelper: "Couldn't parse origin for" "httpdisabled://jsso.indiatimes.com/sso/IndiatimesEmailLogin?ru=httpdisabled://mb.indiatimes.com/public/login.jsp&nru=httpdisabled://mb.indiatimes.com/public/login_redo.jsp&siteid=ebdccde5269865f726086aac8047fc06" ({})
[task 2019-12-03T19:25:44.979Z] 19:25:44 INFO - PID 7892 | console.warn: LoginHelper: "Couldn't parse origin for" null ({})
[task 2019-12-03T19:25:44.979Z] 19:25:44 INFO - PID 7892 | console.warn: LoginHelper: "Couldn't parse origin for" "httpdisabled://jsso.indiatimes.com/sso/IndiatimesEmailLogin?ru=httpdisabled://mb.indiatimes.com/public/login.jsp&nru=httpdisabled://mb.indiatimes.com/public/login_redo.jsp&siteid=ebdccde5269865f726086aac8047fc06" ({})
[task 2019-12-03T19:25:44.979Z] 19:25:44 INFO - PID 7892 | console.warn: LoginHelper: "Couldn't parse origin for" "httpdisabled://jsso.indiatimes.com/sso/IndiatimesEmailLogin?ru=httpdisabled://mb.indiatimes.com/public/login.jsp&nru=httpdisabled://mb.indiatimes.com/public/login_redo.jsp&siteid=ebdccde5269865f726086aac8047fc06" ({})
[task 2019-12-03T19:25:44.979Z] 19:25:44 INFO - PID 7892 | console.warn: LoginHelper: "Couldn't parse origin for" "httpdisabled://jsso.itimes.com/sso/ITimesLogin?ru=httpdisabled://www.itimes.com/dologin.php&siteid=bb857c2ade1c6bcc4a9cccc12e89e39b&nru=httpdisabled://www.itimes.com" ({})
[task 2019-12-03T19:25:44.979Z] 19:25:44 INFO - PID 7892 | console.warn: LoginHelper: "Couldn't parse origin for" null ({})
[task 2019-12-03T19:25:44.981Z] 19:25:44 INFO - PID 7892 | console.warn: LoginHelper: "Couldn't parse origin for" "httpdisabled://jsso.itimes.com/sso/ITimesLogin?ru=httpdisabled://www.itimes.com/dologin.php&siteid=bb857c2ade1c6bcc4a9cccc12e89e39b&nru=httpdisabled://www.itimes.com" ({})
[task 2019-12-03T19:25:44.981Z] 19:25:44 INFO - PID 7892 | console.warn: LoginHelper: "Couldn't parse origin for" "httpdisabled://jsso.itimes.com/sso/ITimesLogin?ru=httpdisabled://www.itimes.com/dologin.php&siteid=bb857c2ade1c6bcc4a9cccc12e89e39b&nru=httpdisabled://www.itimes.com" ({})
[task 2019-12-03T19:25:44.981Z] 19:25:44 INFO - PID 7892 | console.warn: LoginHelper: "Couldn't parse origin for" "httpdisabled://www.mocolife.com/itlogin.aspx?itp=1" ({})
[task 2019-12-03T19:25:44.981Z] 19:25:44 INFO - PID 7892 | console.warn: LoginHelper: "Couldn't parse origin for" null ({})
[task 2019-12-03T19:25:44.983Z] 19:25:44 INFO - PID 7892 | console.warn: LoginHelper: "Couldn't parse origin for" "httpdisabled://www.mocolife.com/itlogin.aspx?itp=1" ({})
[task 2019-12-03T19:25:44.983Z] 19:25:44 INFO - PID 7892 | console.warn: LoginHelper: "Couldn't parse origin for" "httpdisabled://www.mocolife.com/itlogin.aspx?itp=1" ({})
[task 2019-12-03T19:25:45.005Z] 19:25:45 INFO - PID 7892 |
[task 2019-12-03T19:25:45.005Z] 19:25:45 INFO - PID 7892 | ###!!! [Child][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2019-12-03T19:25:45.005Z] 19:25:45 INFO - PID 7892 |
[task 2019-12-03T19:25:45.005Z] 19:25:45 INFO - PID 7892 |
[task 2019-12-03T19:25:45.005Z] 19:25:45 INFO - PID 7892 | ###!!! [Child][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2019-12-03T19:25:45.005Z] 19:25:45 INFO - PID 7892 |
[task 2019-12-03T19:25:45.089Z] 19:25:45 ERROR - Return code: 3221225477
[task 2019-12-03T19:25:45.089Z] 19:25:45 WARNING - setting return code to 3221225477
[task 2019-12-03T19:25:45.089Z] 19:25:45 ERROR - # TBPL FAILURE #

TEST-START | tp5o_webext

The error code 3221225477, which is dec10is equal to0xC0000005mainly refer to aSTATUS_ACCESS_VIOLATION` error, which is actually coming from Firefox and not Talos. Not sure what this actually means. Maybe Aaron has an idea.

Flags: needinfo?(aklotz)
OS: Unspecified → Windows
Hardware: Unspecified → All

That would mean that the process in question crashed. That's about all I can add.

Flags: needinfo?(aklotz)
Whiteboard: [perftest:triage]

Should a crash should be reported in this case, and if so, why doesn't that happen? (harness issue?)

Gabriele, any idea why we do not record minidumps in those cases?

Flags: needinfo?(gsvelto)

Crash reporting is enabled but it doesn't look like we hit the exception handler, which is why there's no crash report. Could it be that the crash happened in firefox.exe instead of libxul? If that's the case then we might not have caught it.

Flags: needinfo?(gsvelto)

I have no idea in how to answer that question. Anything we could do in CI via try builds to get more clarification here?

After looking at the logs again I don't think it's crashing before we setup the exception handler especially because it seems that we're returning that value. When the crash reporter handler terminates firefox it does not return the current exception code, it terminates the process with the return code 1. So I started poking around at what else might raise that exception and the only thing I could think of is the sandbox. IIRC sandbox exceptions don't always produce crash reports so that's might be happening here. It's only a guess though since I don't remember the details.

Flags: needinfo?(gsvelto)

The failure rate spiked on March 8th, but we still don't have any actionable data. Interesting that we see this only on opt, and it spans different Talos suites.
Moving this bug based on gsvelto's guess, since it doesn't seem to be a harness issue. Perhaps it can be reproduced locally with a debugger attached?

Component: Talos → Security: Process Sandboxing
Priority: P2 → --
Product: Testing → Core
Version: Version 3 → unspecified
Whiteboard: [perftest:triage]

Bob, did we change anything relevant the 8th? Is there something actionable for us here to get more info?

Assignee: nobody → bobowencode
Priority: -- → P2
Flags: needinfo?(bobowencode)

The only thing that I can think of recently was Toshihito's patches.

Flags: needinfo?(bobowencode) → needinfo?(tkikuchi)

Looking at the graph in the entire history, the failure count tends to spike every Sunday. The latest one on Mar-8 was the biggest, and after that, the failure rate is very low until today.

I don't think it's caused by my change on 6-Mar. I ran this test on Try after making the diagnostics asserts I added 100% hit, but no repro. Without the repro or a crash dump, I have no idea to investigate this failure.

Flags: needinfo?(tkikuchi)

It doesn't spike on Sundays. There are less pushes on Sundays which increases the plotted graph value (failure count per push) for the day if there are occurrences for that day.

Assignee: bobowencode → nobody
Severity: normal → S3

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.