Data is getting lost in nsPSMComponent::HashEnd(PRUint32 id,char** hash, PRUint32* hashlen, PRUint32 maxLen) because HashEnd(...) calls if(CMT_HASH_End(controlConnection, id, (unsigned char*)*hash, (CMUint32*)hashlen, maxLen) != CMTSuccess) in which the second argument "hash" is an unsigned char, whereas, we have "hash" as char** in HashEnd(...). This is causing the hashing algorithm to return incorrect values.i.e all values higher than 128 are treated as negative, hence we are getting the following results. // results from psm-glue (incorrect) ffffffdd 60 ffffff85 ffffffc7 fffffffb ffffffad 6b ffffffd5 67 61 01 ffffffc4 ffffffce 7e 30 ffffffc7 // results from 4.x hashing dd 60 85 c7 fb ad 6b d5 67 61 01 c4 ce 7e 30 c7 This effects verification of netscape.cfg file by preferences.
Reassigning to dougt.
Summary: 2nd argument in nsPSMComponent::HashEnd(..) should be an unsigned char → 2nd argument in nsPSMComponent::HashEnd(..) should be an unsigned char**
Assignee: norris → dougt
Assignee: dougt → mstoltz
Looking at it. Please bear in mind I'm moving these functions out of nsIPSMComponent into a new interface, nsISignatureVerifier, because of bug 36853. The implementation will remain in nsPSMComponent.
Status: NEW → ASSIGNED
Status: ASSIGNED → RESOLVED
Last Resolved: 18 years ago
Resolution: --- → FIXED
Fixed. This function is now in caps/idl/nsISignatureVerifier.idl, and it uses the native type |unsigned char*| as its out parameter. This makes it non-scriptable, but I don't think anyone was using it from JS anyway.
Verified per Mitchell's comments.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.