1289330 - Intermittent dom/crypto/test/test_WebCrypto.html | application crashed [@ mozilla::dom::GetDOMClass]

Status: RESOLVED FIXED

(Core :: DOM: Core & HTML, defect)

Description

[Treeherder Bug Filer]

Filed by: tomcat [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=32592224&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/NHEYsZfJQq6wvCPLHjWCsA/runs/0/artifacts/public%2Flogs%2Flive_backing.log

Comment 1

[Boris Zbarsky [:bzbarsky]]

This is crashing because the mPublicExponent of a RsaHashedKeyAlgorithm ended up with a null object in it.

At first glance this could happen if the allocations and stuff that RsaHashedKeyAlgorithmStorage::ToKeyAlgorithm fail.  Sadly, it ignores those failures...

Comment 2

[Tim Taubert [:ttaubert] (inactive)]

Attachment: 0001-Bug-1289330-Check-for-success-when-calling-ToKeyAlgo.patch

(In reply to Boris Zbarsky [:bz] from comment #1)
> This is crashing because the mPublicExponent of a RsaHashedKeyAlgorithm
> ended up with a null object in it.
> 
> At first glance this could happen if the allocations and stuff that
> RsaHashedKeyAlgorithmStorage::ToKeyAlgorithm fail.  Sadly, it ignores those
> failures...

Yeah, that's definitely an oversight. This should fix it.

Comment 3

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 8775287 [details] [diff] [review]
0001-Bug-1289330-Check-for-success-when-calling-ToKeyAlgo.patch

>+    JSObject* prime = mPrime.ToUint8Array(aCx);
>+    JSObject* generator = mGenerator.ToUint8Array(aCx);

This is going to fail static analysis: "prime" is not rooted and mGenerator.ToUint8Array can gc.

Please just use JS::Rooted<JSObject*> to store the return values of all your ToUint8Array calls, like so:

  JS::Rooted<JSObject*> exponent(aCx, mPublicExponent.ToUint8Array(aCx));

Also, once an allocation fails, you need to return without trying to do more allocations: at this point an exception is pending on aCx and trying to do more work on it is not ok.  That means if !prime you need to return before trying to create generator.

r=me with those two issues fixed.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/2b89d0c341aa
Check for success when calling ToKeyAlgorithm() r=bz

Comment 5

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/2b89d0c341aa