276310 - PK11_DigestFinal will return SECSuccess despite DigestOp never being called

Status: RESOLVED INVALID

(NSS :: Libraries, defect)

Description

[Jason Reid]

digest = (unsigned char *)malloc((MD2_LENGTH + 1)*sizeof(unsigned char));
        digestContext = PK11_CreateDigestContext(SEC_OID_MD2);
        if (NULL == digestContext) {
                fprintf(stderr,
"ERROR: PK11_CreateDigestContext(SEC_OID_MD2) failed.\n");
                rv++;
        }
        secStatus = PK11_DigestBegin(digestContext);
        if (SECSuccess != secStatus) {
                fprintf(stderr,"ERROR: PK11_DigestBegin failed\n");
                rv++;
        }
        secStatus = PK11_DigestFinal(digestContext, digest, &counter,
(MD2_LENGTH + 1));
        if (secStatus == SECSuccess) {
                fprintf(stderr,"ERROR: PK11_DigestFinal returned success for a
digest without PK11_DigestOp ever being called.\n");
                rv++;
        } else {
                fprintf(stderr, "ERROR: PK11_DigestFinal failed (expected)\n");
        }
        PK11_DestroyContext(digestContext, PR_TRUE);
        free(digest);

In the above code, PK11_DigestFinal returns SECSuccess and a digest value
is inserted into digest despite PK11_DigestOp never being called. This gives the
false impression that a valid digest has been computed.

ERROR: PK11_DigestFinal returned success for a digest without PK11_DigestOp ever
being called.

Comment 1

[Wan-Teh Chang]

If PK11_DigestOp is never called, that can be
interpreted as hashing a zero-byte input.  Is a
zero-byte input legal?  If so, does PK11_DigestFinal
write the correct digest value into the digest?

If a zero-byte input is illegal, then I agree
PK11_DigestFinal should fail if PK11_DigestOp is
never called.

Comment 2

[Nelson Bolyard (seldom reads bugmail)]

Zero byte input is legal for all hashes we support.

Comment 3

[Nelson Bolyard (seldom reads bugmail)]

Marked invalid per previous comment.