1348825 - Windows build is broken in media/mtransport

Status: RESOLVED INVALID

(Core :: WebRTC: Signaling, defect, P2)

Description

[:Gijs (he/him)]

0:09.78 The system cannot find the file specified.
 0:09.80 The system cannot find the file specified.
 0:09.80
 0:09.80
 0:09.80 In the directory  /d/builds/opt/media/mtransport/third_party/nICEr/nicer_nicer
 0:09.80 The following command failed to execute properly:
 0:09.82 d:/builds/opt/_virtualenv/Scripts/python.exe -m mozbuild.action.cl d:/MSVS2015/VC/BIN/amd64_x86/cl.exe -Fonr_crypto.obj -c -DNDEBUG -DTRIMMED=1 -DWIN32 -D_WINDOWS -DNOMINMAX -D_CRT_RAND_S -DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS -DWIN32_LEAN_AND_MEAN -D_ATL_NO_OPENGL -D_HAS_EXCEPTIONS=0 -D_SECURE_ATL -DCHROMIUM_BUILD -DTOOLKIT_VIEWS=1 -DUSE_LIBJPEG_TURBO=1 -DENABLE_ONE_CLICK_SIGNIN -DENABLE_REMOTING=1 -DENABLE_WEBRTC=1 -DENABLE_CONFIGURATION_POLICY -DENABLE_INPUT_SPEECH -DENABLE_NOTIFICATIONS -DENABLE_GPU=1 -DENABLE_EGLIMAGE=1 -DUSE_SKIA=1 -D__STD_C -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_DEPRECATE -DENABLE_TASK_MANAGER=1 -DENABLE_WEB_INTENTS=1 -DENABLE_EXTENSIONS=1 -DENABLE_PLUGIN_INSTALLATION=1 -DENABLE_PROTECTOR_SERVICE=1 -DENABLE_SESSION_SERVICE=1 -DENABLE_THEMES=1 -DENABLE_BACKGROUND=1 -DENABLE_AUTOMATION=1 -DENABLE_PRINTING=1 -DENABLE_CAPTIVE_PORTAL_DETECTION=1 -DSANITY_CHECKS -DUSE_TURN -DUSE_ICE -DUSE_RFC_3489_BACKWARDS_COMPATIBLE -DUSE_STUND_0_96 -DUSE_STUN_PEDANTIC -DNR_SOCKET_IS_VOID_PTR -Drestrict= -DR_PLATFORM_INT_TYPES=<stdint.h> -DR_DEFINED_INT2=int16_t -DR_DEFINED_UINT2=uint16_t -DR_DEFINED_INT4=int32_t -DR_DEFINED_UINT4=uint32_t -DR_DEFINED_INT8=int64_t -DR_DEFINED_UINT8=uint64_t -D_CRT_SECURE_NO_WARNINGS -D__UNUSED__= -DHAVE_STRDUP -DNO_REG_RPC -DDONT_HAVE_ETHTOOL_SPEED_HI -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DUNICODE -D_UNICODE -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -Id:/mozilla-central/media/mtransport/third_party/nICEr -Id:/builds/opt/media/mtransport/third_party/nICEr/nicer_nicer -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/event -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/log -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/plugin -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/registry -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/share -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/stats -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/util -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/util/libekr -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/port/generic/include -Id:/mozilla-central/media/mtransport/third_party/nICEr/src/crypto -Id:/mozilla-central/media/mtransport/third_party/nICEr/src/ice -Id:/mozilla-central/media/mtransport/third_party/nICEr/src/net -Id:/mozilla-central/media/mtransport/third_party/nICEr/src/stun -Id:/mozilla-central/media/mtransport/third_party/nICEr/src/util -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/port/win32/include -Id:/builds/opt/ipc/ipdl/_ipdlheaders -Id:/mozilla-central/ipc/chromium/src -Id:/mozilla-central/ipc/glue -Id:/builds/opt/dist/include -MD -FI d:/builds/opt/mozilla-config.h -DMOZILLA_CLIENT -TC -nologo -wd4091 -D_HAS_EXCEPTIONS=0 -W3 -Gy -Zc:inline -utf-8 -arch:SSE2 -Gw -wd4244 -wd4267 -we4553 -Zi -O1 -Oi -Oy- -Fdgenerated.pdb -FS d:/mozilla-central/media/mtransport/third_party/nICEr/src/crypto/nr_crypto.c
 0:09.82 In the directory  /d/builds/opt/media/mtransport/third_party/nICEr/nicer_nicer
 0:09.82 The following command failed to execute properly:
 0:09.83 d:/builds/opt/_virtualenv/Scripts/python.exe -m mozbuild.action.cl d:/MSVS2015/VC/BIN/amd64_x86/cl.exe -Foice_candidate.obj -c -DNDEBUG -DTRIMMED=1 -DWIN32 -D_WINDOWS -DNOMINMAX -D_CRT_RAND_S -DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS -DWIN32_LEAN_AND_MEAN -D_ATL_NO_OPENGL -D_HAS_EXCEPTIONS=0 -D_SECURE_ATL -DCHROMIUM_BUILD -DTOOLKIT_VIEWS=1 -DUSE_LIBJPEG_TURBO=1 -DENABLE_ONE_CLICK_SIGNIN -DENABLE_REMOTING=1 -DENABLE_WEBRTC=1 -DENABLE_CONFIGURATION_POLICY -DENABLE_INPUT_SPEECH -DENABLE_NOTIFICATIONS -DENABLE_GPU=1 -DENABLE_EGLIMAGE=1 -DUSE_SKIA=1 -D__STD_C -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_DEPRECATE -DENABLE_TASK_MANAGER=1 -DENABLE_WEB_INTENTS=1 -DENABLE_EXTENSIONS=1 -DENABLE_PLUGIN_INSTALLATION=1 -DENABLE_PROTECTOR_SERVICE=1 -DENABLE_SESSION_SERVICE=1 -DENABLE_THEMES=1 -DENABLE_BACKGROUND=1 -DENABLE_AUTOMATION=1 -DENABLE_PRINTING=1 -DENABLE_CAPTIVE_PORTAL_DETECTION=1 -DSANITY_CHECKS -DUSE_TURN -DUSE_ICE -DUSE_RFC_3489_BACKWARDS_COMPATIBLE -DUSE_STUND_0_96 -DUSE_STUN_PEDANTIC -DNR_SOCKET_IS_VOID_PTR -Drestrict= -DR_PLATFORM_INT_TYPES=<stdint.h> -DR_DEFINED_INT2=int16_t -DR_DEFINED_UINT2=uint16_t -DR_DEFINED_INT4=int32_t -DR_DEFINED_UINT4=uint32_t -DR_DEFINED_INT8=int64_t -DR_DEFINED_UINT8=uint64_t -D_CRT_SECURE_NO_WARNINGS -D__UNUSED__= -DHAVE_STRDUP -DNO_REG_RPC -DDONT_HAVE_ETHTOOL_SPEED_HI -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DUNICODE -D_UNICODE -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -Id:/mozilla-central/media/mtransport/third_party/nICEr -Id:/builds/opt/media/mtransport/third_party/nICEr/nicer_nicer -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/event -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/log -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/plugin -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/registry -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/share -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/stats -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/util -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/util/libekr -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/port/generic/include -Id:/mozilla-central/media/mtransport/third_party/nICEr/src/crypto -Id:/mozilla-central/media/mtransport/third_party/nICEr/src/ice -Id:/mozilla-central/media/mtransport/third_party/nICEr/src/net -Id:/mozilla-central/media/mtransport/third_party/nICEr/src/stun -Id:/mozilla-central/media/mtransport/third_party/nICEr/src/util -Id:/mozilla-central/media/mtransport/third_party/nrappkit/src/port/win32/include -Id:/builds/opt/ipc/ipdl/_ipdlheaders -Id:/mozilla-central/ipc/chromium/src -Id:/mozilla-central/ipc/glue -Id:/builds/opt/dist/include -MD -FI d:/builds/opt/mozilla-config.h -DMOZILLA_CLIENT -TC -nologo -wd4091 -D_HAS_EXCEPTIONS=0 -W3 -Gy -Zc:inline -utf-8 -arch:SSE2 -Gw -wd4244 -wd4267 -we4553 -Zi -O1 -Oi -Oy- -Fdgenerated.pdb -FS d:/mozilla-central/media/mtransport/third_party/nICEr/src/ice/ice_candidate.c
 0:09.85 d:/mozilla-central/config/rules.mk:886: recipe for target 'nr_crypto.obj' failed
 0:09.86 mozmake.EXE[5]: *** [nr_crypto.obj] Error 1

Nils / Byron, any idea what's going on here? :-\

Comment 1

[:Gijs (he/him)]

(I clobbered already, same problem. And no, I don't see any concrete indications of *which* file the system cannot find.)

Comment 2

[Byron Campen [:bwc]]

This is m-c right? Does d99df97e4115 build?

Comment 3

[:Gijs (he/him)]

(In reply to Byron Campen [:bwc] from comment #2)
> This is m-c right? Does d99df97e4115 build?

It does not. Same error.

Windows (10, insider builds) updated overnight. It might be related to that, I guess? :-\

Is there a way to get more information from this part of the build system as to what's missing/broken?

Comment 4

[:Gijs (he/him)]

(In reply to :Gijs from comment #3)
> (In reply to Byron Campen [:bwc] from comment #2)
> > This is m-c right? Does d99df97e4115 build?
> 
> It does not. Same error.
> 
> Windows (10, insider builds) updated overnight. It might be related to that,
> I guess? :-\
> 
> Is there a way to get more information from this part of the build system as
> to what's missing/broken?

... though additionally, it seems that if I just rebuild some more (on that rev), eventually I end up with a build and a message that says the build "succeeded". A build that does seem to work... (though I don't know how to exercise the code at issue here to see if it at some point notices that parts of it didn't build right...).

Comment 5

[:Gijs (he/him)]

(In reply to :Gijs from comment #4)
> (In reply to :Gijs from comment #3)
> > (In reply to Byron Campen [:bwc] from comment #2)
> > > This is m-c right? Does d99df97e4115 build?
> > 
> > It does not. Same error.
> > 
> > Windows (10, insider builds) updated overnight. It might be related to that,
> > I guess? :-\
> > 
> > Is there a way to get more information from this part of the build system as
> > to what's missing/broken?
> 
> ... though additionally, it seems that if I just rebuild some more (on that
> rev), 

Egh, sorry for the spam, but I meant: if I rebuild some more without a clobber inbetween. In other words, repetitively invoking './mach build' seems to 'eventually' get me a build. :-\

Comment 6

[Byron Campen [:bwc]]

I haven't the foggiest about what might be going on here. Maybe more complete output in pastebin or something would shed some light?

Comment 7

[Randell Jesup [:jesup] (needinfo me)]

:gijs can you give more output (before the error)?  Thx

Comment 8

[:Gijs (he/him)]

Attachment: Output of ./mach build -v

Pastebin doesn't seem to like this, so let's try a bugzilla attachment...

Comment 9

[:Gijs (he/him)]

(In reply to :Gijs from comment #5)
> (In reply to :Gijs from comment #4)
> > (In reply to :Gijs from comment #3)
> > > (In reply to Byron Campen [:bwc] from comment #2)
> > > > This is m-c right? Does d99df97e4115 build?
> > > 
> > > It does not. Same error.
> > > 
> > > Windows (10, insider builds) updated overnight. It might be related to that,
> > > I guess? :-\
> > > 
> > > Is there a way to get more information from this part of the build system as
> > > to what's missing/broken?
> > 
> > ... though additionally, it seems that if I just rebuild some more (on that
> > rev), 
> 
> Egh, sorry for the spam, but I meant: if I rebuild some more without a
> clobber inbetween. In other words, repetitively invoking './mach build'
> seems to 'eventually' get me a build. :-\

So this 'works' on d99df97e4115 but not on current m-c. :-(

Comment 10

[Nils Ohlmeier [:drno]]

The message before the compiler error appears to be coming from here http://searchfox.org/mozilla-central/source/toolkit/components/osfile/tests/xpcshell/test_osfile_win_async_setPermissions.js#91
Although I don't know what that actually means.

I'm not super familiar with Windows builds, but this path looks fishy to me /d/builds/opt/media/mtransport/third_party/nICEr/nicer_nicer
I don't know where 'nicer_nicer' is coming from.

Is this from a build with visual studio project file maybe?

Comment 11

[:Gijs (he/him)]

(In reply to Nils Ohlmeier [:drno] from comment #10)
> The message before the compiler error appears to be coming from here
> http://searchfox.org/mozilla-central/source/toolkit/components/osfile/tests/
> xpcshell/test_osfile_win_async_setPermissions.js#91
> Although I don't know what that actually means.
> 
> I'm not super familiar with Windows builds, but this path looks fishy to me
> /d/builds/opt/media/mtransport/third_party/nICEr/nicer_nicer
> I don't know where 'nicer_nicer' is coming from.
> 
> Is this from a build with visual studio project file maybe?

I don't think so. I generated MSVS project files at a point in the past, but I've also clobbered and rebuilt multiple times, and I'm just building with ./mach build at this point...

Comment 12

[(not currently active) Ted Mielczarek]

(In reply to Nils Ohlmeier [:drno] from comment #10)
> I'm not super familiar with Windows builds, but this path looks fishy to me
> /d/builds/opt/media/mtransport/third_party/nICEr/nicer_nicer
> I don't know where 'nicer_nicer' is coming from.

That's an artifact of how our gyp file integration works, FYI.

Comment 13

[(not currently active) Ted Mielczarek]

Gijs: you might try running Process Monitor and invoking the build to see if you can catch what file access is failing (and exactly what process is causing it).

Comment 14

[:Gijs (he/him)]

Well, in today's episode of "ugh, computers"... it seems that builds succeed when opening a plain cmd window with the mozillabuild bat files (that start a bash shell), but it fails when using the same mozillabuild bat files inside ConEmu (my console emulator/tool of choice). This includes artifact builds, fwiw. I'm still no wiser as to *why* it fails, but at least I can build again...

Thanks everyone for the helpful hints as well as commiseratory "I haven't the foggiest", and sorry for wasting all your time!