1020804 - Most Win64 Windows 8 Debug tests are failing -- mozjs.dll issue?

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[(Away)]

Almost all tests are orange for Windows 8 Debug at https://tbpl.mozilla.org/?tree=Date&rev=cd4ed657d827.

The few passing cppunittests are the standalone ones that don't have dependencies on Firefox libraries.

When I try to run the server's build locally with gflags set to "Show loader snaps", I get this debugger message:
LdrpProcessStaticImports - ERROR: Probing for the manifest of DLL "C:\firefox-win64debug\mozjs.dll" failed with status 0xc000007b

That's the same error code (in hex) that the xpcshell tests and failing cppunittests are logging.

Substituting my locally-built mozjs.dll fixes the problem. I see no obvious differences with depends.exe.

Given the above, it is strange that the JIT tests are passing.

Comment 1

[(Away)]

> Given the above, it is strange that the JIT tests are passing.
I guess it's because the JIT tests use js.exe and not mozjs.dll.

Comment 2

[(Away)]

Comparisons of dumpbin.exe /headers:

Shipping mozjs.dll:
               0 [       0] RVA [size] of Resource Directory
          3A7400 [    2070] RVA [size] of Certificates Directory

Date branch opt mozjs.dll:
               0 [       0] RVA [size] of Resource Directory
               0 [       0] RVA [size] of Certificates Directory

Date branch debug mozjs.dll:
          B07400 [     4F0] RVA [size] of Resource Directory
               0 [       0] RVA [size] of Certificates Directory

My debug mozjs.dll:
               0 [       0] RVA [size] of Resource Directory
               0 [       0] RVA [size] of Certificates Directory

At RVA B07400 of the Date debug mozjs.dll, I see some similar bytes as at RVA 3A7400 of the shipping mozjs.dll, except with strings like "Mozilla Fake CA".

So it looks like: we are fake-signing the debug DLL, but calling it a Resource rather than a Certificate (?!)

Comment 3

[(Away)]

So apparently we do this fake-signing-resource thing to all of the DLLs. Why is mozjs.dll the only one failing?

The other DLLs have length 0x4E0 for that directory. Mozjs.dll is the only one with length 0x4F0. Why is it special? The only difference I know of is that mozjs lacks version information (bug 520849 / bug 634282). Could that be confusing the signing servers?

Comment 4

[Mike Hommey [:glandium]]

Ben or Chris, can you help David figuring out what's wrong with binary signing on automation?

Comment 5

[(Away)]

The resource-versus-certificate issue may be a red herring. Some good DLLs are showing the same confusion; maybe it's just something that dumpbin isn't interpreting right.

So for now let's just try to figure out why mozjs.dll has a signature of a different length from all the other DLLs.

Comment 6

[(not currently active) Ted Mielczarek]

The only difference I can see in the link lines for mozjs vs. other DLLs is the lack of a resource file in the mozjs link.

Comment 7

[[e-mail] (:bhearsum)]

(In reply to Mike Hommey [:glandium] from comment #4)
> Ben or Chris, can you help David figuring out what's wrong with binary
> signing on automation?

https://bugzilla.mozilla.org/show_bug.cgi?id=711210

Comment 8

[Chris AtLee [:catlee]]

signing for win64 is completely disabled ATM, due to the bug ben linked to.

Comment 9

[(Away)]

(In reply to Chris AtLee [:catlee] from comment #8)
> signing for win64 is completely disabled ATM, due to the bug ben linked to.

I see that signing is disabled for opt builds, but for debug builds we are still fake-signing. Is that intentional? Can we disable it?

Comment 10

[Chris AtLee [:catlee]]

(In reply to David Major [:dmajor] (UTC+12) from comment #9)
> (In reply to Chris AtLee [:catlee] from comment #8)
> > signing for win64 is completely disabled ATM, due to the bug ben linked to.
> 
> I see that signing is disabled for opt builds, but for debug builds we are
> still fake-signing. Is that intentional? Can we disable it?

It is not intentional! I'll disable it shortly.

Comment 11

[Chris AtLee [:catlee]]

Attachment: disable signing for win64-debug

Comment 12

[(Away)]

Is the change supposed to take effect right away? Does it need to be merged around or something? I still see the same signing happening on the Date branch.

Comment 13

[Aki Sasaki (not active)]

In production.

Comment 14

[(Away)]

The debug tests are loading: https://tbpl.mozilla.org/?tree=Date&rev=f33b631a2cbd 

Thanks Chris!