1404169 - fsanitize=enum (ubsan) runtime errors in extensions/universalchardet/src/base/nsCodingStateMachine.h

Status: RESOLVED FIXED

(Core :: Internationalization, defect, P3)

Description

[Arthur Edelstein [:arthur]]

I am working on tracking down and fixing ubsan errors reported in Mozilla's automated tests.

When I use -fsanitize=enum, one of the errors I get is:

> [task 2017-09-25T00:47:57.792Z] 00:47:57     INFO - TEST-START | extensions/universalchardet/tests/test_bug426271-utf-8.html
> [task 2017-09-25T00:47:57.902Z] 00:47:57     INFO - GECKO(4261) | /builds/worker/workspace/build/src/extensions/universalchardet/src/base/nsCodingStateMachine.h:49:12: runtime error: load of value 5, which is not a valid value for type 'nsSMState'
> [task 2017-09-25T00:47:57.903Z] 00:47:57     INFO - GECKO(4261) | /builds/worker/workspace/build/src/extensions/universalchardet/src/base/nsCodingStateMachine.h:39:9: runtime error: load of value 5, which is not a valid value for type 'nsSMState'
> [task 2017-09-25T00:47:57.904Z] 00:47:57     INFO - GECKO(4261) | /builds/worker/workspace/build/src/extensions/universalchardet/src/base/nsCodingStateMachine.h:46:30: runtime error: load of value 5, which is not a valid value for type 'nsSMState'
> [task 2017-09-25T00:47:57.906Z] 00:47:57     INFO - GECKO(4261) | /builds/worker/workspace/build/src/extensions/universalchardet/src/base/nsCodingStateMachine.h:46:30: runtime error: load of value 5, which is not a valid value for type 'nsSMState'

the same error was reported here:
https://bugs.freedesktop.org/show_bug.cgi?id=101032
where they replaced the nsSMState by an unsigned int. Here's the patch:
https://cgit.freedesktop.org/uchardet/uchardet/commit/?id=53f7ad0e0b0894f11c948ee75f74595ab1b61405

Should we do the same thing?

Comment 1

[Makoto Kato [:m_kato]]

Yes, nsSMState shouldn't use enum.  It uses non-enum value too.

Comment 2

[Arthur Edelstein [:arthur]]

Attachment: 0001-Bug-1404169-Fix-ubsan-runtime-error-for-nsSMState-en.patch

Thanks. Here's a version of the patch for mozilla-central.

Comment 3

[Arthur Edelstein [:arthur]]

Thank you for the review.

Try server results: https://treeherder.mozilla.org/#/jobs?repo=try&revision=be58c7cf95e9

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/2e390724da6f
Fix ubsan runtime error for nsSMState enum. r=m_kato

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/2e390724da6f