integer overflow in enum in icc34.h

RESOLVED WORKSFORME

Status

()

Firefox
General
--
major
RESOLVED WORKSFORME
10 years ago
8 years ago

People

(Reporter: ul, Unassigned)

Tracking

({platform})

3.0 Branch
Other
AIX
platform
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(URL)

Attachments

(3 obsolete attachments)

(Reporter)

Description

10 years ago
User-Agent:       Mozilla/5.0 (X11; U; AIX 5.1; en-US; rv:1.8.1.16) Gecko/20080711 SeaMonkey/1.1.11
Build Identifier: Mozilla/5.0 (X11; U; AIX 5.1; en-US; rv:1.8.1.16) Gecko/20080711 SeaMonkey/1.1.11

xlc_r -o cmscnvrt.o -c  -DMOZILLA_INTERNAL_API -D_IMPL_NS_COM -DEXPORT_XPT_API -DEXPORT_XPTC_API -D_IMPL_NS_COM_OBSOLETE -D_IMPL_NS_GFX -D_IMPL_NS_WIDGET -DIMPL_XREAPI -DIMPL_NS_NET -DIMPL_THEBES  -DOSTYPE=\"AIX5.1\" -DOSARCH=AIX -I../include -I/home/ulink/Src/mozilla/modules/lcms/src -I. -I../../../dist/include/lcms -I../../../dist/include   -I../../../dist/include/lcms -I../../../dist/include/nspr     -I../../../dist/sdk/include   -qflag=w:w     -DNDEBUG -DTRIMMED -O2 -q32 -qarch=com -qalign=full -qalias=noansi -qlanglvl=stdc99 -qmaxmem=-1   -DMOZILLA_VERSION=\"1.9.0.1\" -DMOZILLA_VERSION_U=1.9.0.1 -DAIX=1 -DHAVE_SYS_INTTYPES_H=1 -DNSCAP_DISABLE_DEBUG_PTR_TYPES=1 -DD_INO=d_ino -DSTDC_HEADERS=1 -DHAVE_ST_BLKSIZE=1 -DHAVE_SIGINFO_T=1 -DHAVE_INT16_T=1 -DHAVE_INT32_T=1 -DHAVE_INT64_T=1 -DHAVE_INT64=1 -DHAVE_UINT=1 -DHAVE_UINT_T=1 -DHAVE_UINT16_T=1 -DHAVE_DIRENT_H=1 -DHAVE_MEMORY_H=1 -DHAVE_UNISTD_H=1 -DHAVE_NL_TYPES_H=1 -DHAVE_MALLOC_H=1 -DHAVE_X11_XKBLIB_H=1 -DHAVE_SYS_STATVFS_H=1 -DHAVE_SYS_STATFS_H=1 -DHAVE_LIBC_R=1 -DHAVE_LIBM=1 -DHAVE_LIBDL=1 -DHAVE_LIBC_R=1 -DFUNCPROTO=15 -DHAVE_XSHM=1 -DHAVE_FT_BITMAP_SIZE_Y_PPEM=1 -DHAVE_FT_LOAD_SFNT_TABLE=1 -D_REENTRANT=1 -DHAVE_RANDOM=1 -DHAVE_STRERROR=1 -DHAVE_LCHOWN=1 -DHAVE_FCHMOD=1 -DHAVE_SNPRINTF=1 -DHAVE_MEMMOVE=1 -DHAVE_RINT=1 -DHAVE_STAT64=1 -DHAVE_LSTAT64=1 -DHAVE_TRUNCATE64=1 -DHAVE_FLOCKFILE=1 -DHAVE_LOCALTIME_R=1 -DHAVE_STRTOK_R=1 -DHAVE_RES_NINIT=1 -DHAVE_LANGINFO_CODESET=1 -DHAVE_I18N_LC_MESSAGES=1 -DMOZ_EMBEDDING_LEVEL_DEFAULT=1 -DMOZ_EMBEDDING_LEVEL_BASIC=1 -DMOZ_EMBEDDING_LEVEL_MINIMAL=1 -DMOZ_PHOENIX=1 -DMOZ_BUILD_APP=browser -DMOZ_XUL_APP=1 -DMOZ_DEFAULT_TOOLKIT=\"cairo-gtk2\" -DMOZ_WIDGET_GTK2=1 -DMOZ_ENABLE_XREMOTE=1 -DMOZ_X11=1 -DMOZ_DISTRIBUTION_ID=\"org.mozilla\" -DMOZ_ENABLE_GCONF=1 -DMOZ_ENABLE_GNOMEUI=1 -DMOZ_ENABLE_DBUS=1 -DOJI=1 -DIBMBIDI=1 -DMOZ_VIEW_SOURCE=1 -DACCESSIBILITY=1 -DMOZ_XPINSTALL=1 -DMOZ_JSLOADER=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DMOZ_NO_XPCOM_OBSOLETE=1 -DMOZ_XTF=1 -DMOZ_CRASHREPORTER_ENABLE_PERCENT=100 -DMOZ_MATHML=1 -DMOZ_ENABLE_CANVAS=1 -DMOZ_SVG=1 -DMOZ_SVG_FOREIGNOBJECT=1 -DMOZ_UPDATE_CHANNEL=default -DMOZ_PLACES=1 -DMOZ_FEEDS=1 -DMOZ_STORAGE=1 -DMOZ_SAFE_BROWSING=1 -DMOZ_URL_CLASSIFIER=1 -DMOZ_LOGGING=1 -DMOZ_USER_DIR=\".mozilla\" -DMOZ_ENABLE_LIBXUL=1 -DHAVE_INTTYPES_H=1 -DMOZ_TREE_CAIRO=1 -DHAVE_UINT64_T=1 -DMOZ_XUL=1 -DMOZ_PROFILELOCKING=1 -DMOZ_RDF=1 -DMOZ_MORKREADER=1 -DMOZ_DLL_SUFFIX=\".so\" -DXP_UNIX=1 -DUNIX_ASYNC_DNS=1 -DJS_THREADSAFE=1 -DMOZ_ACCESSIBILITY_ATK=1 -DATK_MAJOR_VERSION=1 -DATK_MINOR_VERSION=12 -DATK_REV_VERSION=3 -DMOZILLA_LOCALE_VERSION=\"1.9a1\" -DMOZILLA_REGION_VERSION=\"1.9a1\" -DMOZILLA_SKIN_VERSION=\"1.8\"  -D_MOZILLA_CONFIG_H_ -DMOZILLA_CLIENT /home/ulink/Src/mozilla/modules/lcms/src/cmscnvrt.c
"../../../dist/include/lcms/icc34.h", line 321.43: 1506-243 (S) Value of enumeration constant must be in range of signed integer.
"../../../dist/include/lcms/icc34.h", line 348.43: 1506-243 (S) Value of enumeration constant must be in range of signed integer.
"../../../dist/include/lcms/icc34.h", line 378.43: 1506-243 (S) Value of enumeration constant must be in range of signed integer.
"../../../dist/include/lcms/icc34.h", line 412.43: 1506-243 (S) Value of enumeration constant must be in range of signed integer.
"../../../dist/include/lcms/icc34.h", line 424.43: 1506-243 (S) Value of enumeration constant must be in range of signed integer.
"../../../dist/include/lcms/icc34.h", line 434.43: 1506-243 (S) Value of enumeration constant must be in range of signed integer.
"../../../dist/include/lcms/icc34.h", line 446.43: 1506-243 (S) Value of enumeration constant must be in range of signed integer.
"../../../dist/include/lcms/icc34.h", line 454.43: 1506-243 (S) Value of enumeration constant must be in range of signed integer.
"../../../dist/include/lcms/icc34.h", line 463.43: 1506-243 (S) Value of enumeration constant must be in range of signed integer.
"../../../dist/include/lcms/icc34.h", line 476.43: 1506-243 (S) Value of enumeration constant must be in range of signed integer.
"../../../dist/include/lcms/icc34.h", line 484.43: 1506-243 (S) Value of enumeration constant must be in range of signed integer.
"../../../dist/include/lcms/icc34.h", line 498.43: 1506-243 (S) Value of enumeration constant must be in range of signed integer.
gmake[5]: *** [cmscnvrt.o] Error 1

Reproducible: Always

Steps to Reproduce:
1. compile on AIX 5.1 with VACPP 6.0

Actual Results:  
build error

Expected Results:  
successfull compilation w/o severe warning
(Reporter)

Comment 1

10 years ago
the icMaxEnum* values are set to 0xFFFFFFFFL which is one bit too much for signed int what enums are in C.
Setting to 0x7FFFFFFFL makes things compile, but this must be proved by someone knowing what these values are for.
Severity: normal → major
Keywords: helpwanted
Version: unspecified → 3.0 Branch
(Reporter)

Updated

10 years ago
Keywords: platform
(Reporter)

Comment 2

10 years ago
Created attachment 331473 [details] [diff] [review]
this patch make cmscnvrt.c compile
(Reporter)

Updated

10 years ago
Attachment #331473 - Flags: review?(pavlov)

Comment 3

10 years ago
marti any idea here?

Comment 4

10 years ago
icc34.h file comes from ICC and has been a source of endless trouble. I'm dropping it entirely in the next revision of lcms. icMaxEnum* is not used in any place of lcms, so you can sefely delete all those lines.
(Reporter)

Updated

10 years ago
Attachment #331473 - Attachment is obsolete: true
Attachment #331473 - Flags: review?(pavlov)
(Reporter)

Comment 5

10 years ago
Created attachment 334089 [details] [diff] [review]
patch removes the overflowing integers according to Marti's comment

Patch reworked to Marti's comment
(Reporter)

Updated

10 years ago
Attachment #334089 - Attachment is obsolete: true
(Reporter)

Comment 6

10 years ago
Created attachment 334099 [details] [diff] [review]
reworked patch with Marti's suggestions

Patch reworked removing all values set to 0xFFFFFFFFL
*except* "icMaxEnumData" which may be returned into an "int" as fall-through default in an case in  
icColorSpaceSignature LCMSEXPORT _cmsICCcolorSpace(int OurNotation)
in cmsgmt.c
So I set icMaxEnumData to a value with all bits set

I don't know what this code does on a platform without an thirty-three bit signed int type ;-) aarrrgh!

Now after clearing the lcms objdir a gmake builds fine w/o warnings.
Attachment #334099 - Flags: review?(marti.maria)
(Reporter)

Updated

9 years ago
Keywords: helpwanted
(Reporter)

Updated

9 years ago
Attachment #334099 - Flags: review?(pavlov)
(Reporter)

Updated

8 years ago
Status: UNCONFIRMED → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → WORKSFORME

Updated

8 years ago
Attachment #334099 - Attachment is obsolete: true
Attachment #334099 - Flags: review?(pavlov)
Attachment #334099 - Flags: review?(marti.maria)
You need to log in before you can comment on or make changes to this bug.