813046 - Assertion "(under_kind&UNEXPECTED_FLAGS)==0" on Windows in ASN.1 encoder with minimal template

Status: RESOLVED INACTIVE

(NSS :: Libraries, defect, P5)

Description

[Kai Engert (:KaiE:)]

Attachment: testcase

The code that I have landed as part of bug 811317 fails, only on Windows.

On Windows, in function sec_asn1e_init_state_based_on_template
we get an assertion failure:
  PORT_Assert ((under_kind & UNEXPECTED_FLAGS) == 0);

This seems similar to bug 583308, however, in this scenario, the template is much simpler.

It's really surprising that the problem can only be seen on Windows.
Could that be caused by differences of memory initialization on Windows?

I have a minimal testcase. All it requires is an empty NSS database (certutil -N) in either /tmp/db/ or c:\\db\\

I'm attaching the testcase as a patch. It will change the test tool "ocspresp" to run a very small amount of code.

Linux runs fine, Windows aborts with the assertion on the first attempt to encode.

Comment 1

[Kai Engert (:KaiE:)]

Just FYI, why I believe I cannot work around this bug, but need it fixed:

In bug 583308 it has been proposed to work around the issue by using a different template - by using a PointerTo template.

Based on the inspiration found in that bug, I played with multiple variations of the template, following the examples that Nelson had attached.

But I failed to write a nested template that gave me the required encoding.

When using any kind of PointerTo sub-template, I got an encoding that looked like:
  [0] {
    NULL
  }

But what I need is:
  [0]
    zero-length-object

The latter style is what other OCSP servers produce (for the good cert status code + empty good info, according to RFC 2560).

Comment 2

[Kai Engert (:KaiE:)]

I'm learning that asn.1 (sub)templates are indeed implemented differently on Windows.

In particular, we have:

#if defined(_WIN32)
#define SEC_ASN1_GET(x)        NSS_Get_##x(NULL, PR_FALSE)
#define SEC_ASN1_SUB(x)        &p_NSS_Get_##x
#define SEC_ASN1_XTRN          SEC_ASN1_DYNAMIC
#define SEC_ASN1_MKSUB(x) \
static const SEC_ASN1TemplateChooserPtr p_NSS_Get_##x = &NSS_Get_##x;
#else
#define SEC_ASN1_GET(x)        x
#define SEC_ASN1_SUB(x)        x
#define SEC_ASN1_XTRN          0
#define SEC_ASN1_MKSUB(x) 
#endif


At the time the encoder processes the sub-template, state->theTemplate points to p_NSS_Get_SEC_SEC_NullTemplate - but the contents of that template appear corrupted. For example, state->theTemplate->kind contains a 10-digit number.

I don't know yet if that indicates a problem must have happened earlier.

Comment 3

[Kai Engert (:KaiE:)]

If templates involving these macros aren't intended to work with our encoder, then we can mark this bug as wontfix.

I hope I have worked around the problem in bug 811317, by using variations of the templates that don't use these macros.

Comment 4

[Robert Relyea]

The macros are necessary because Windows does not export data symbols from shared libraries. If you reference a symbol in one shared library from your application or another shared library, then you need to use these macros. (This is true even if you reference the symbol inside your own template.

You can see examples of this in templates used by smime.dll in NSS proper.

bob