DER_GetInteger decodes negative numbers improperly

RESOLVED FIXED

Status

NSS
Libraries
RESOLVED FIXED
15 years ago
15 years ago

People

(Reporter: Nelson Bolyard (seldom reads bugmail), Assigned: Nelson Bolyard (seldom reads bugmail))

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

When DER_GetInteger decodes a negative number, and the number's value is 
in the range      0 > X > -2^24, DER_GetInteger returns a positive number,
not the negative number.  

This patch will correct the problem, I believe:

@@ -211,9 +211,14 @@
     unsigned len = it->len;
     unsigned char *cp = it->data;
     unsigned long overflow = ((unsigned long)0xffL) << ((sizeof(ival) - 1)*8);
+    unsigned long ofloinit;
+
+    if (*cp & 0x80)
+       ival = -1L;
+    ofloinit = ival & overflow;

     while (len) {
-       if (ival & overflow) {
+       if ((ival & overflow) != ofloinit) {
            PORT_SetError(SEC_ERROR_BAD_DER);
            if (ival < 0) {
                return LONG_MIN;

Unless someone objects, I will check this patch in soon.
(Assignee)

Comment 1

15 years ago
I just realized that DER_GetInteger also computes the wrong answer for 

a) positive numbers in the range 2^31 <= X < 2^32  
and
b) negative numbers in the range -2^32 < X <= -2^31 

and the patch I gave above doesn't fix these other two problems.  
I'll write another patch.
Status: NEW → ASSIGNED
(Assignee)

Comment 2

15 years ago
Created attachment 102990 [details] [diff] [review]
Fix DER_GetUInteger and DER_GetInteger

But wait, there's more!

DER_GetInteger is also flawed in the same way for the postive values 
reported above.  

I think this patch fixes all these problems.  
I'm about to do some more testing.
(Assignee)

Comment 3

15 years ago
No, DER_GetUInteger was correct.  Fix checked in.   
Status: ASSIGNED → RESOLVED
Last Resolved: 15 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.