1047494 - tautological comparison in pkixnss.cpp (VerifySignedData), causing "error: comparison of constant 2147483647 with expression of type 'uint16_t' (aka 'unsigned short') is always false"

Status: RESOLVED FIXED

(Core :: Security: PSM, defect)

Description

[Dana Keeler (she/her) [:keeler]]

Result
VerifySignedData(const SignedDataWithSignature& sd,
                 Input subjectPublicKeyInfo, void* pkcs11PinArg)
{
  // See bug 921585.
  if (sd.data.GetLength() >
        static_cast<unsigned int>(std::numeric_limits<int>::max())) {
    return Result::FATAL_ERROR_INVALID_ARGS;
  }

GetLength returns a uint16_t, so this is tautological on systems where int is larger than 2 bytes. If we don't support building on any of those systems, we can probably just remove this check (it's to protect the call to VFY_VerifyDataDirect later in that function).

Comment 1

[Daniel Holbert [:dholbert]]

To clarify, this is a build warning, which is treated as an error in --enable-warnings-as-errors builds.  This broke my build, with clang version 3.4.)

The full error (for the record & to make this searchable for other folks hitting this problem) looks like:
{
security/pkix/lib/pkixnss.cpp:96:27: error: comparison of constant 2147483647 with expression of type 'uint16_t' (aka 'unsigned short') is always false [-Werror,-Wtautological-constant-out-of-range-compare]
   if (sd.data.GetLength() >
       ~~~~~~~~~~~~~~~~~~~ ^
}

Comment 2

[Dana Keeler (she/her) [:keeler]]

Attachment: patch

Brian, do you think it's safe to just remove the check, or do we care about systems where sizeof(int) is 2?

Comment 3

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Comment on attachment 8466510 [details] [diff] [review]
patch

Review of attachment 8466510 [details] [diff] [review]:
-----------------------------------------------------------------

::: security/pkix/lib/pkixnss.cpp
@@ -95,5 @@
> -  // See bug 921585.
> -  if (sd.data.GetLength() >
> -        static_cast<unsigned int>(std::numeric_limits<int>::max())) {
> -    return Result::FATAL_ERROR_INVALID_ARGS;
> -  }

Please add a assertion that sizeof(std::declval(GetLength()) == 2. It is likely we'll have to change Input and Reader at some point to support inputs larger than 64K, as I've read that there are some real-world certificates that large. When the change happens, the removal of this check would be dangerous if we didn't remember to add this check back.

See http://stackoverflow.com/questions/6837444/decltype-for-return-of-a-function.

Alternatively, you can add an Input::MAX_LENGTH constant and assert that it is less than std::numeric_limits<int>::max().

Note that MSVC doesn't support constexpr so at least for MSVC your assertion can't be a static_assert.

Comment 4

[Dana Keeler (she/her) [:keeler]]

Attachment: patch v2

I'm not sure why using decltype was necessary, but maybe I've misunderstood. In any case, the compiler wasn't happy with it, so I went with assert(sizeof(sd.data.GetLength()) < sizeof(int));

Comment 5

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Comment on attachment 8467864 [details] [diff] [review]
patch v2

Review of attachment 8467864 [details] [diff] [review]:
-----------------------------------------------------------------

::: security/pkix/lib/pkixnss.cpp
@@ +151,5 @@
> +  // The static_cast is safe as long as the length of the data in sd.data can
> +  // fit in an int. Right now that length is stored as a uint16_t, so this
> +  // works. In the future this may change, hence the assertion.
> +  // See also bug 921585.
> +  assert(sizeof(sd.data.GetLength()) < sizeof(int));

This is OK. I suggested using declval so that you could use static_assert so that this would be noticed at compile-time instead of only at runtime (only on debug builds). But, again, this is OK.

Comment 6

[Dana Keeler (she/her) [:keeler]]

Attachment: patch v2 with static_assert

Oh, I get it. I changed it to static_assert. This appears to work locally and on my windows vm, but just to be sure: https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=d00fd1d55254

Comment 7

[Dana Keeler (she/her) [:keeler]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/dab7a08e5bb4

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/dab7a08e5bb4