Closed Bug 1459456 Opened 6 years ago Closed 6 years ago

NSS failed Assertion

Categories

(NSS :: Libraries, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: robert.merget, Assigned: franziskus)

References

Details

Attachments

(2 files)

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0
Build ID: 20180326160923

Steps to reproduce:

During our research on TLS-Attacker (https://github.com/RUB-NDS/TLS-Attacker) we found a failed assertion in the TLS-Client of NSS. The bug can be reproduced by sending a specially crafted DHE ServerKeyExchange message to NSS. Eg. with the modulus:

2658079891452973548809739558400188375860383141163887766083528780073074556914594928027928213253825200598501499895592873484732168401060736756198489161757904867595271137453080835431617551687934886932413852293292489094575200971072558849227764324432109562680511336379717893842056414705467219501691901416710853724148751171739017545025630417889469283012404014086722353945774960966260669831925981642511609669765127863384769837246730747052815494129014000622871731736587488916596331383433801699374766952301360320820653204154941094122169525608157228613127389323364932877301281318305330996726883884936984568174628200752892634201012258234106215766265461571920775904646055558936709610861937745162024495633854503468523120092237136682443819765614345472245050153900067631141378780850952917402538954544778649154013442463035890479748938493887666087320858121603387669642146709602657792141455397952000890560359534453467449626812228428620969816514057446361116619629956288771726982112532542295947314340101959801483943285800517773103713515230052553874147186710182852980378289405184767594204122465137249367878043180379379476696131853837170038551398869023504430694932566264418228292325038543296459387735751544101417275597574312909986445089211480565574958921396967798802925781532279390872979254402459079048548175162958349074861109070106607415598000842938941644418344396456571622905345087658524932583568242116549610703284720297561184646100760263327388016308984764599386797658398589531869501373885876687852270181772088321462453946839504695830726690190141795025010166083603135202523584612648867345355258793299121533173472487735024785719816179564280096112585792865054925060191040115483912310192586596817319225689894652845388284907189969281495479440478084665270968599050248388931345481189610263419852298711377880922193873425884338459265800675805721858179815087608973569559505404530034590159730930737573166872220057988173341225527735898002741155398428179386659710410019545811977069423704623345521136584008443136750917338518137937723828292940832612849917577387995250882745301769004420205268993886786928347430111866478790495025134901941081968513090662531683981350657205415318505389841493698885000304208845080137347013506816147181971478720735029131602077429337824882272003653163145452462681434134968210216456139088059410752360417797217204073393874698918994148895310100929408835839431312457308595899363290038282648253751737825981282921022426494100067210164705244596200885121779074654314099035162686806395342231143146683666872302827003183651350982273693455049646702098294405979998623693038434388102490831933812667922268562703764916483177807273082872316110142575039958097410311830111879322171926684069750987284439340418096556691303581482351600992203975174848882191408978441886426268423515025537975613336642701149538819312598174394482334163190069000614648131570416734136357997938436795634663388466574850627969574582334222762241271350818731116636194014756049954326240850704998363369831240322634542396451882416484129841712491617361540465304011703997742989125890244331228848640516130789346873343075538531678213922307492649458502103269744075014520574567220560047003458945237522175866589614679213440896514951978445803306414871684377104611363368281960000080719479307718114759021047589777003422950927846704325990310929929101634526832459545685031664091036471367354288901715887606823399497790582101489337593174014867995536876725904667676057704424234441079952544694295913142734435715907084772697085532350596231802545


We attached a TLS-Attacker configuration file with which you can reproduce the problem:
git clone https://github.com/RUB-NDS/TLS-Attacker.git
cd TLS-Attacker
mvn clean install
cd apps
java -jar TLS-Server -port 4433 -config nss_modulus_assertion_config.xml
-> Connect with NSS testclnt / Firefox

We _assume_ that this is harmless. But since this happens in a sensitive part of NSS  we flag this as security relevant (better safe then sorry). We did not test this problem in other Versions of NSS.



Actual results:

In the DEBUG build NSS crashes with:
Assertion failure: newLen <= b->space, at ../../lib/ssl/sslencode.c:33

In the production Build a SEC_ERROR_LIBRARY_FAILURE is returned.


Expected results:

NSS should not crash. It should probably proceed with the Handshake, return a HandshakeFailure Alert or an IllegalParameter Alert - but the current behavior is certainly not expected.
Explicitely check that DH keys aren't too long. The buffer catches this but we can be more explicit about it.
Thank you for reporting the issue.
This is actually the desired behaviour here. The assertion only fires in debug builds. In optimised builds the function returns an error. NSS doesn't support DH keys > 8192 bit. The assertion is triggered when trying to append the key to the handshake message but the used buffer is not big enough. We can be more explicit about the buffer size (see patch) but NSS will still throw an assertion in this case.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Priority: -- → P3
Group: crypto-core-security
Assignee: nobody → franziskuskiefer
https://hg.mozilla.org/projects/nss/rev/aa6678175aade961a3290e1edad69bf9b8548998
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → 3.38
Comment on attachment 8974009 [details]
Bug 1459456 - check for long dh keys

Martin Thomson [:mt:] has approved the revision.

https://phabricator.services.mozilla.com/D1212
Attachment #8974009 - Flags: review+
See Also: → 1441121
See Also: → 1441122
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: