1370356 - Autophone - Cdm3 crash during shutdown does not turn test orange

Status: RESOLVED WONTFIX

(Testing Graveyard :: Autophone, enhancement)

Description

[Bob Clary [:bc] (inactive)]

https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=d47466614ea1ea1a4d9c64ff943a0a3670291529&filter-searchStr=autophone

There was an exploitable crash at

https://treeherder.mozilla.org/logviewer.html#?job_id=104139235&repo=mozilla-beta&lineNumber=855

after the test suite completed but the test is still green. Note we did have a tombstone:

https://autophone.s3.amazonaws.com/v1/task/VoTzS7mzSc6gcHpwqM6Mrg/runs/0/artifacts/public/build/f3d4fded-218e-4eb9-840e-48060b05db9b-tombstone_00.1.txt

Would marking the test as failing if a tombstone exists, be sufficient without having to parse the logcat independently or should treeherder's parse have caught it?

FYI, I believe this crash is already filed as bug 1273678.

Comment 1

[Bob Clary [:bc] (inactive)]

I would also like to hijack this bug a little to discuss in a secure setting how we should deal with classifying crashes in Treeherder.

How can we automatically recognize and flag exploitable crashes in Treeherder?

We could do simple greps of the crash report looking for canary addresses.

We could also add the exploitable tool for Windows crashes and perhaps even Linux crashes.

What do we do when we get an exploitable crash in Treeherder? I'm especially concerned about a shipping or soon to be shipped product? How do we handle the triage and sheriffing of such a crash?

Comment 2

[Geoff Brown [:gbrown]]

Android emulator unit tests use checkForTombstones() - https://dxr.mozilla.org/mozilla-central/rev/8a3aa1701537ea6b8334f432cd030d260d492fa3/build/mobile/remoteautomation.py#175.

If I'm reading checkForTombstones() correctly, it doesn't report anything that would turn a job orange. It just adds the tombstone info to the test log.

On the emulator, crashes that do not produce proper minidumps and crash reports often do not produce tombstones. I think (hope!?) that no-crash-report cases still report failure via the "No tests run or test summary not found." mechanism -- reported in bug 1300313 -- or the "No crash directory (%s) found on remote device" mechanism -- bug 1284320.

Also, I believe we have had tombstones created during test runs by system processes -- apparently unrelated, or only tangentially related, to firefox. So it's potentially problematic to fail a test run because a tombstone is found...but perhaps some parsing of the tombstone could overcome that limitation.

Comment 3

[Ed Morley [:emorley]]

(In reply to Bob Clary [:bc:] from comment #0)
> or should treeherder's parse have caught it?

Treeherder doesn't use the log output to determine the pass/fail of a job, so the harness/job submitter would need to report this as failing. As for the log output, ideally the harness would mark the failure using the mozharness ERROR log level line prefix (which Treeherder recognises), or else use one of the existing typical strings that the Treeherder log parser matches (PROCESS-CRASH, "###!!! ABORT:" and so on [1]). However if a case could be made for a new failure term that should be added to the parser, we'd be open to doing so.

[1] https://github.com/mozilla/treeherder/blob/d690f53e82a2a846e981fdb4dc57d23eca2c8201/treeherder/log_parser/parsers.py#L342-L383

> I would also like to hijack this bug a little to discuss in a secure setting
> how we should deal with classifying crashes in Treeherder.

Coincidentally :Tomcat opened bug 1369067 about this recently - I think it's a good topic to discuss, perhaps we could move the discussion there?

Comment 4

[Bob Clary [:bc] (inactive)]

Great, lets go there then.

Comment 5

[Bob Clary [:bc] (inactive)]

Autophone is going away. Resolving these to wontfix.