590991 - Implement a tool to externally "test for existing valid .chk file", without loading the tested shared library (chktest)

Status: RESOLVED FIXED

(NSS :: Tools, defect)

Description

[Chris AtLee [:catlee]]

We have code ready to go that will re-generate the .chk files after the freebl3.dll, softokn3.dll, and nssdmb3.dll files are authenticode signed, but we have no way of validating that the new .chk files are valid.

Is there an existing tool that can do this, or can one be written?

Comment 1

[Nelson Bolyard (seldom reads bugmail)]

Any NSS tool is that tool, when the softoken has been configured to operate 
in FIPS mode (the only mode where NSS cares about .chk files).

Comment 2

[Nelson Bolyard (seldom reads bugmail)]

Try certutil -L -d <SOME_DB_DIR>

Comment 3

[Chris AtLee [:catlee]]

What should <SOME_DB_DIR> be in this case?

Comment 4

[Chris AtLee [:catlee]]

To be clear, the steps I want to do are:

1) Get unsigned freebl3.dll, .etc.
2) Sign freebl3.dll with signcode.exe
3) Regenerate freebl3.chk with `shlibsign -v -i freebld3.dll`
4) Verify that it's correct.

Comment 5

[Chris AtLee [:catlee]]

e.g., how do I verify that the new .chk files generated in http://people.mozilla.org/~catlee/Firefox%20Setup%203.6.12.exe are correct?

Comment 6

[Nelson Bolyard (seldom reads bugmail)]

Please ask support questions in the right place.

Comment 7

[Kai Engert (:KaiE:)]

I think Chris' question doesn't justify to close this bug.
There are two possible ways to see Chris' request:

(1)
Chris requests that someone writes a new NSS application will perform a self test and report either success or failure.
In this case, it would be correct for this bug to be assigned to the NSS/tools component.
However, it might be that nobody is going to write such a program any time soon.

As an alternative approach, Nelson has proposed to use the existing tools in a specific way, which could be used to conclude that FIPS is working or not.

This is a good idea and might work, but clearly, we don't have the required steps yet.

This bug requests to develop a solution an attempt to make the Mozilla-client build process more robust by adding a test for FIPS consistency.
I don't think that request is a support question.

If NSS/tools is not an acceptable component for this problem to solve for the Mozilla build process, we can find a more suitable component.

Comment 8

[Kai Engert (:KaiE:)]

(2)
write the series of commands that can be used to perform the test using the existing tools

I'll try to brainstorm based on the idea.

Comment 9

[Chris AtLee [:catlee]]

Something to keep in mind is this is to be used in the context of signing our windows release files.  The proposed process would work something like this, for each of 70-ish locales:

- unpack Firefox X.Y.Z Setup.exe
- sign all .exe's, .dll's with signcode
- regenerate all .chk files with shlibsign
- repack Firefox X.Y.Z Setup.exe

Later we do:
- unpack Firefox X.Y.Z Setup.exe
- validate authenticode signing with chktrust
- validate .chk files with ???

We do NOT have a build environment, nor any build artifacts other than the final packages on the machine where this signing occurs.

Comment 10

[Kai Engert (:KaiE:)]

I enabled building of "modutil" by changing mozilla/security/manager/Makefile.in
I duplicated the line that contains 
        $(NSSMAKE) -C $(NSS_SRCDIR)/security/nss/cmd/certutil $(DEFAULT_GMAKE_FLAGS)
and introduced another line that builds modutil
        $(NSSMAKE) -C $(NSS_SRCDIR)/security/nss/cmd/modutil $(DEFAULT_GMAKE_FLAGS)

In an output tree:

cd obj/dist/bin
mkdir tmpfipstestdir
echo fIps140 > tmpfipstestdir/passfile
LD_LIBRARY_PATH=. ../../nss/certutil -N -d tmpfipstestdir -f tmpfipstestdir/passfile
LD_LIBRARY_PATH=. ../../nss/modutil -dbdir tmpfipstestdir -chkfips true 2>&1 

if the .chk files are correct I get
  "FIPS mode enabled".
on standard output

if the .chk files are corrupt I get
  modutil: function failed: security library: invalid arguments.
on error output

Comment 11

[Kai Engert (:KaiE:)]

If you don't have a build environment when testing, it should be sufficient to create "cerutil" and "modutil" once for your test environment (and maybe update them once in a while with newer NSS releases)

Comment 12

[Nelson Bolyard (seldom reads bugmail)]

There's no need to make anything.  
We've gone around this track many times.
Do we really need to take more laps around it now?

Comment 13

[Nelson Bolyard (seldom reads bugmail)]

read:

bug 521878 comment 29
bug 489961 comment 6   through bug 489961 comment 9

Comment 14

[Chris AtLee [:catlee]]

(In reply to comment #13)
> read:
> 
> bug 521878 comment 29
> bug 489961 comment 6   through bug 489961 comment 9

I know how to re-generate the .chk files, thank you.  What I'm looking for is a way to validate that it's been done properly.

Comment 15

[Kai Engert (:KaiE:)]

sorry, my series of commands missed on step.
updated below.
the preparation phase is:

  (1) cd obj/dist/bin
  (2) mkdir tmpfipstestdir
  (3) echo fIps140 > tmpfipstestdir/passfile
  (4) LD_LIBRARY_PATH=. ../../nss/certutil -N -d tmpfipstestdir \
        -f tmpfipstestdir/passfile

then we enable FIPS mode (new):

  (5) echo "\n" | LD_LIBRARY_PATH=. ../../nss/modutil \
        -dbdir tmpfipstestdir -fips true

finally we check fips mode:

  (6) LD_LIBRARY_PATH=. ../../nss/modutil -dbdir tmpfipstestdir \
        -chkfips true 2>&1 


If everything is good, step (6) will print
  FIPS mode enabled.
on standard output


However, if things are broken, the failure will be visible earlier, step (5) will fail with:

  security library: invalid arguments.
  ERROR: Unable to switch FIPS modes.

and step (6) will say
  FIPS mode disabled.


So, maybe the test is to simply watch for the output from step 6, which will be either "FIPS enabled" (good) or "FIPS disabled" (broken).

Comment 16

[Nelson Bolyard (seldom reads bugmail)]

Chris, the point you're missing is that the programs you need are ALL to 
be viewed as TOOLS, not as build products.  

No new tool is needed.  The tool already exists.  
Nothing new needs to be built as part of Firefox builds. 
Treat the tools as something you already have when you start the builds,
not something you build.

Comment 17

[Nelson Bolyard (seldom reads bugmail)]

Kai, 
There's no need to repeat all those steps with every build. 
One static secmod.db file copied and used in all tests,
and one command that uses it, will suffice.

Comment 18

[Kai Engert (:KaiE:)]

(In reply to comment #17)
> Kai, 
> There's no need to repeat all those steps with every build. 
> One static secmod.db file copied and used in all tests,
> and one command that uses it, will suffice.

The benefit of using the commands is, that it will work in both database modes, the classic mode that uses file secmod.db, and the sql mode that uses file pkcs11.txt

Comment 19

[Kai Engert (:KaiE:)]

(In reply to comment #16)
> No new tool is needed.  The tool already exists.  
> Nothing new needs to be built as part of Firefox builds.

It's not strictly necessary, but it could be done to simplify testing, without a need to carry other binaries around.


> Treat the tools as something you already have when you start the builds,
> not something you build.

NSS doesn't distribute binaries regularly, therefore IMHO it's a reasonable idea to provide instructions on how to build the required tools. Building NSS separately works, but requires additional work. The approach I provided in the first half of comment 10 is a simpler way to produce the modutil tool in the Firefox build area. I don't request that this should be done regularly, but if we decided to do it, it wouldn't hurt.

The certutil is already present in the binary output tree of Firefox builds.

Comment 20

[Nelson Bolyard (seldom reads bugmail)]

Why does firefox build certutil?

Comment 21

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

It's used in the automated testing infrastructure.

Comment 22

[Nelson Bolyard (seldom reads bugmail)]

Kyle, 
Does that infrastructure use Perl or Python?
Do you build those in every run, too?

Comment 23

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

We build certutil in every run, along with pk12util and some other stuff (see http://mxr.mozilla.org/mozilla-central/source/security/manager/Makefile.in#345)

The testing infrastructure is primarily in Python.  An example of what we do is at http://mxr.mozilla.org/mozilla-central/source/build/pgo/genpgocert.py.in#134, where we create a cert that we can use for the HTTPS test server.

Comment 24

[Robert Relyea]

> The benefit of using the commands is, that it will work in both database modes,
> the classic mode that uses file secmod.db, and the sql mode that uses file
> pkcs11.txt

If you just need to check the .chk files, then either case will do it (sql or dbm). So just keeping a pkcs11.txt file around is sufficient.

For general testing, it's good to check both (which is why we do that out of all.sh).

Comment 25

[Kai Engert (:KaiE:)]

Chris, did you have a chance to try the proposed test?

Does it work for you?

Comment 26

[Kai Engert (:KaiE:)]

Chris, using the utils.zip works for me on all of trunk, 1.9.2 and 1.9.1 branches.

On windows, I used the following steps:

- download latest win32 installer exe from ftp.mozilla.org nightly latest-*

- mkdir c:\users\kaie\catlee
- unzip the contents of your utils.zip to that directory

for each branch I repeated the following test:

    - cd c:\users\kaie\catlee
    - mkdir firefox-directory
    - cd firefox-directory
    - 7z.exe x ..\..\Downloads\firefox-version.exe
(1) - for trunk, cd core, for 3.x, cd nonlocalized
    - your current work directory is the one that contains all the DLLs
      from your distribution
    - copy ..\..\*.exe .
      => copies certutil, modutil, shlibsign to current directory
    - (optional: typing "certutil" or "modutil" shows 
       that the commands can run fine using the locally available DLLs,
       that's good, because that's what we're testing, 
       the consistency of the DLLs)
(2) - mkdir db
(3) - echo fIps140 > db\pw
(4) - certutil -N -d db -f db\pw
(5) - echo "\n" | modutil -dbdir db -fips true
(6) - modutil -dbdir db -chkfips true
    - (you may now remove directory db)

Numbers (1) to (6) are the equivalent windows steps from comment 15

(Make sure you use uppercase i for the password in step (3), it's imporant for FIPS that the password has mixed case and numbers.)

Comment 27

[Kai Engert (:KaiE:)]

Ok. We failed to use the existing tools to perform the self test.

I tried the best I could.

Even though I produced a series of steps that worked for me, in all my tests, we are unable to get it to work in Chris' environment.

Whatever he tries, he gets some windows failure, 0xC0000022 etc.

I don't understand that failure. I don't have access to Chris' machine, so I can't debug what's going wrong for him.

There might be some inconsistency between the nss-util-executables and the shared-libraries from the browser-distribution package.


In my understanding, if you want to use the existing tools we need an environment where
- the NSS shared libraries are being loaded to perform an active self test
- use certutil + modutil that correctly link against those libraries


It seems it's easy enough to get a mismatch on Windows and end up with bizarre error messages, if nss-utils and shared-libraries were not produced in the same build environment.


Even if our tools are capable of performing the self test "in theory",
it's no help if it's too difficult to get used.


Yes, we could argue, the Mozilla build engineers should try harder and make sure they always build modutil+certutil and include that in the distribution package, to make sure they can run their test.


But we should all understand that build slaves and single-signing-machine ought to be different environments, and that intermediate build output from build slaves is not easily available on the signing-machine, in particular not, if there are hundreds of branch/locale combinations.


It should be in our own interest, in the intereste of the NSS team, to enable Mozilla engineering to perform an easy self test, to detect whether or not our self-integrity checkfiles are correct or corrupted.


One way would be to add some parameter to the application binary like "--perform-selftest", but I see that as task that would require approval from too many people to get done.


I agree with Chris that NSS should provide a very simple tool to take a pair of 2 files, dll and chk, and make sure that those match. That really ought to be a standard functionality.


I have used the last 2 hours and wrote a patch that enhances shlibsign with a "test mode". It calls into freebl and provides a filename, and will print either 0 or 1, indicating failure or success.


That patch requires to export a new function from freebl.
I would hope that is acceptable.

Patch attached.

Is this approach acceptable?

Comment 28

[Kai Engert (:KaiE:)]

Attachment: Patch v1 - enhance shlibsign with test mode

Comment 29

[Kai Engert (:KaiE:)]

Attachment: Patch v2

v1 missed the cmd directory

Comment 30

[Kai Engert (:KaiE:)]

Attachment: Patch v3

Cleaned up. Ready for review.

Comment 31

[Robert Relyea]

Comment on attachment 491912 [details] [diff] [review]
Patch v3

r-

There are lots of ways to do this without adding new symbols to libfreebl.

I would rather shlibsign use the freebl function list rather than adding new export symbols to libfreebl.

Even better, why not have shlibsign test all three (or n) files directly with FC_Intialize, that's what the application is going to do.

(actually I just looked at slibsign... If you specify -F it will do what you want.... We just need a -nope or -test option so it doesn't actually try to sign something).



    

Is there an actual need to verify each .so separately?

Comment 32

[Kai Engert (:KaiE:)]

Attachment: Patch v5

(In reply to comment #31)
> There are lots of ways to do this without adding new symbols to libfreebl.
> 
> I would rather shlibsign use the freebl function list rather than adding new
> export symbols to libfreebl.


Ok. New patch attached:
- do not export new symbol
- link against the static freebl library

In order to make static linking easy, I created a new test executable "chktest", that is completely independent of anything else.

I used the "bltest" program as an example to get this done.


> Even better, why not have shlibsign test all three (or n) files directly with
> FC_Intialize, that's what the application is going to do.
> 
> (actually I just looked at slibsign... If you specify -F it will do what you
> want.... We just need a -nope or -test option so it doesn't actually try to
> sign something).


I want a fully external test,
that does not require to load the test target in any way, 
in order to avoid any dynamic link dependencies.

Comment 33

[Kai Engert (:KaiE:)]

Attachment: Patch v6

v6 fixes a small c/c++ syntax error

Tested to build and work on Linux and Windows.


Usage of this new "chktest" tool is very simple.
Simply provide a single parameter on the command line, the full path to the library to be tested.
The tool will use NSS internal BLAPI code to open the file, search for the associated .chk file, and run the verification.

The new chktest tool will print a result value on standard output, either SUCCESS or FAILURE.

I've played with varying combinations of "file good", "file missing", "file with invalid content", and I got the expected results.

Comment 34

[Kai Engert (:KaiE:)]

Program output:

$ chktest 
Usage:  chktest <full-path-to-shared-library>
        Will test for valid chk file.
        Will print SUCCESS or FAILURE.

$ chktest libnssutil3.so 
FAILURE

$ chktest libsoftokn3.so 
SUCCESS

$ chktest libfreebl3.so 
SUCCESS

$ chktest libnssdbm3.so 
SUCCESS

$ mv libfreebl3.chk libfreebl3.chk-hidden

$ chktest libfreebl3.so 
FAILURE

$ cp libsoftokn3.chk libfreebl3.chk

$ chktest libfreebl3.so 
FAILURE

$ mv libfreebl3.chk-hidden libfreebl3.chk

$ chktest libfreebl3.so 
SUCCESS

Comment 35

[Robert Relyea]

Comment on attachment 493282 [details] [diff] [review]
Patch v6

r+ rrelyea with one major caveat...

1) make sure you are using the lastest version of loader.c
2) if you are adding this file for the 3.12 branch, don't bump up the freebl version number (it has already been bumped up for 3.12.9).

bob

Comment 36

[Robert Relyea]

Correction. If you want to make 3.12 branch, you'll need to put your function in version 3.012 side.

If you want the 3.13 branch, add it to then end of clone and don't bump the version number.

bob

Comment 37

[Kai Engert (:KaiE:)]

I don't want to disturb your work for NSS 3.12.9 and J-Pake.

I think you declared last Friday as the freeze-date for the 3.12.9 release, because of this week's test-infrastructure limitations.

Given that this new API is not intended as a Mozilla runtime feature,
but solely inteded to be used by an external testing tool,
I think it's sufficient to have it on NSS trunk, only.

I'll address Bob's request:
- do not increase the version number
- place the new function into the existing 3.013 section
- no new section for 3.014
- land into trunk
- do not land into nss 3.12 branch

Comment 38

[Kai Engert (:KaiE:)]

Attachment: Patch v7

This patch addresses Bob's requests.

Comment 39

[Kai Engert (:KaiE:)]

Checked in to trunk, marking fixed.

Checking in cmd/manifest.mn;
/cvsroot/mozilla/security/nss/cmd/manifest.mn,v  <--  manifest.mn
new revision: 1.30; previous revision: 1.29
done
RCS file: /cvsroot/mozilla/security/nss/cmd/chktest/Makefile,v
done
Checking in cmd/chktest/Makefile;
/cvsroot/mozilla/security/nss/cmd/chktest/Makefile,v  <--  Makefile
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/security/nss/cmd/chktest/chktest.c,v
done
Checking in cmd/chktest/chktest.c;
/cvsroot/mozilla/security/nss/cmd/chktest/chktest.c,v  <--  chktest.c
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/security/nss/cmd/chktest/manifest.mn,v
done
Checking in cmd/chktest/manifest.mn;
/cvsroot/mozilla/security/nss/cmd/chktest/manifest.mn,v  <--  manifest.mn
initial revision: 1.1
done
Checking in lib/freebl/blapi.h;
/cvsroot/mozilla/security/nss/lib/freebl/blapi.h,v  <--  blapi.h
new revision: 1.41; previous revision: 1.40
done
Checking in lib/freebl/ldvector.c;
/cvsroot/mozilla/security/nss/lib/freebl/ldvector.c,v  <--  ldvector.c
new revision: 1.28; previous revision: 1.27
done
Checking in lib/freebl/loader.c;
/cvsroot/mozilla/security/nss/lib/freebl/loader.c,v  <--  loader.c
new revision: 1.52; previous revision: 1.51
done
Checking in lib/freebl/loader.h;
/cvsroot/mozilla/security/nss/lib/freebl/loader.h,v  <--  loader.h
new revision: 1.33; previous revision: 1.32
done
Checking in lib/freebl/shvfy.c;
/cvsroot/mozilla/security/nss/lib/freebl/shvfy.c,v  <--  shvfy.c
new revision: 1.15; previous revision: 1.14
done

Comment 40

[Kai Engert (:KaiE:)]

If Chris Atlee needs follow up tweeks to the behaviour of the chktest tool, we will address it in a separate bug.