Closed Bug 1448040 Opened 2 years ago Closed Last year

clean up hang monitor threads

Categories

(Core :: General, enhancement)

enhancement
Not set

Tracking

()

RESOLVED FIXED
mozilla62
Tracking Status
firefox61 --- wontfix
firefox62 --- fixed

People

(Reporter: froydnj, Assigned: dthayer)

References

(Blocks 2 open bugs)

Details

(Whiteboard: [fxperf:p1])

Attachments

(1 file)

Looking at my thread list for active content processes (and the parent process!), I see:

* ProcessHangMon[itor]
* BHMgr Processor
* BHMgr Monitor (which is actually consuming a substantial amount of CPU time :( )
* Hang Monitor

Surely we don't actually need *three* different hang monitoring threads, do we?  Doug, do you have cycles to look into what these threads are doing, and possibly unify some of the hang monitoring?  Or do you know somebody who does?
Flags: needinfo?(dothayer)
Yeah I'll take a look. I don't have a wealth of familiarity with this stuff but I do have a bit of time.
Assignee: nobody → dothayer
Status: NEW → ASSIGNED
Flags: needinfo?(dothayer)
Whiteboard: [fxperf:p1]
I think the Hang Monitor and BHMgr Monitor could be merged without much difficulty. They run on different intervals and different logic, but I think we could easily join them nonetheless.

The BHMgr Processor thread looks to me like it _could_ be merged with the BHMgr Monitor thread, but it would sacrifice the accuracy of the BHMgr Monitor, since running the BHMgr Processor's work from inside the monitor could cause us to miss short hangs (Nika, am I understanding this correctly, or was there a different reason for processing the hangs in a separate thread from the monitor?)

I don't really think it makes sense to merge the ProcessHangMon thread, since really it serves an entirely different purpose than the others.

Regarding CPU time on BHMgr Monitor, I'm a little bit surprised - do you have any more details?
Flags: needinfo?(nika)
The fact that the BHMgrProcessor thread and the BHMgrMonitor are separate is intentional. The work that we do for processing hangs is fairly substantial (collecting & processing loaded dynamic module maps - can sometimes take *seconds*), and we want to avoid blocking the monitor thread on that.

The hang monitor thread should probably be cleaned up. I'm not sure if anyone's looking at chrome hangs right now, so it might make a lot of sense to just remove it alltogether.

I'm also at least mildly surprised by the CPU time being spent on the BHMgr Monitor. I'm pretty sure we could rewrite it in a less CPU intensive manner, but I don't have the time to really look into that right now.
Flags: needinfo?(nika)
Comment on attachment 8972211 [details]
Bug 1448040 - Remove HangMonitor/ChromeHangs

https://reviewboard.mozilla.org/r/240878/#review249678

Mostly r- because of the extra file which seems out of place. Otherwise looks pretty good :-)

::: dom/base/nsContentUtils.cpp:10763
(Diff revision 2)
>    // us whether or not the user is interacting.
>    nsCOMPtr<nsIObserverService> obs = mozilla::services::GetObserverService();
>    obs->AddObserver(this, kUserInteractionInactive, false);
>    obs->AddObserver(this, kUserInteractionActive, false);
>  
> -  // We can't register ourselves as an annotator yet, as the HangMonitor hasn't
> +  // We can't register ourselves as an annotator yet, as the BackgroundHangMonitor hasn't

Please re-wrap this comment to 80 columns.

::: dom/ipc/ContentPrefs.cpp:1
(Diff revision 2)
> +/* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 2 -*- */

I think this file shouldn't be here...

::: toolkit/components/backgroundhangmonitor/HangAnnotations.h:21
(Diff revision 2)
> +/**
> + * This type represents an individual hang annotation.
> + */
> +class BackgroundHangAnnotation
> +{
> +public:
> +  BackgroundHangAnnotation() {}
> +  BackgroundHangAnnotation(const nsAString& aName, const nsAString& aValue)
> +    : mName(aName), mValue(aValue)
> +  {}
> +
> +  nsString mName;
> +  nsString mValue;
> +};

It would be awesome if we could switch this type over to using the type we use inside BHM: https://searchfox.org/mozilla-central/rev/a85db9e29eb3f022dbaf8b9a6390ecbacf51e7dd/toolkit/components/backgroundhangmonitor/HangTypes.ipdlh#76-81.

::: toolkit/components/backgroundhangmonitor/HangAnnotations.h:43
(Diff revision 2)
> +  void AddAnnotation(const nsAString& aName, const int32_t aData);
> +  void AddAnnotation(const nsAString& aName, const double aData);
> +  void AddAnnotation(const nsAString& aName, const nsAString& aData);
> +  void AddAnnotation(const nsAString& aName, const nsACString& aData);
> +  void AddAnnotation(const nsAString& aName, const bool aData);

At some point we might want to clean this up and just pass around a TArray?

::: toolkit/components/backgroundhangmonitor/HangAnnotations.h:72
(Diff revision 2)
> +  bool Unregister(BackgroundHangAnnotator& aAnnotator);
> +
> +  BackgroundHangAnnotations GatherAnnotations();
> +
> +private:
> +  Mutex                mMutex;

indentation?

::: toolkit/components/backgroundhangmonitor/HangAnnotations.cpp:94
(Diff revision 2)
> +{
> +  BackgroundHangAnnotations annotations;
> +  { // Scope for lock
> +    MutexAutoLock lock(mMutex);
> +    for (std::set<BackgroundHangAnnotator*>::iterator i = mAnnotators.begin(),
> +                                        e = mAnnotators.end();

indentation fixes

::: toolkit/components/backgroundhangmonitor/HangAnnotations.cpp:117
(Diff revision 2)
> +  WriteParam(aMsg, aParam.mValue);
> +}
> +
> +bool
> +ParamTraits<BackgroundHangAnnotation>::Read(const Message* aMsg,
> +                              PickleIterator* aIter,

indentation fixes?

::: toolkit/components/telemetry/Telemetry.cpp:104
(Diff revision 2)
>  using Telemetry::Common::ToJSString;
>  using Telemetry::Common::SetCurrentProduct;
>  using mozilla::dom::Promise;
>  using mozilla::dom::AutoJSAPI;
> -using mozilla::Telemetry::HangReports;
>  using mozilla::Telemetry::CombinedStacks;

Is CombinedStacks still used? I don't think there will be any remaining observers?

::: toolkit/components/telemetry/Telemetry.cpp:1990
(Diff revision 2)
> -                                  Move(aAnnotations));
> -}
>  
>  void CaptureStack(const nsACString& aKey)
>  {
>  #ifdef MOZ_GECKO_PROFILER

Why is this ifdef here? It's nested inside another ifdef which is identical?

::: toolkit/components/telemetry/docs/data/main-ping.rst:351
(Diff revision 2)
>  
>  .. _chromeHangs:
>  
>  chromeHangs
>  -----------
> +As of Firefox 62, chromeHangs has been removed.

Please mention that the bhr ping should be used instead?
Attachment #8972211 - Flags: review?(nika) → review-
Comment on attachment 8972211 [details]
Bug 1448040 - Remove HangMonitor/ChromeHangs

https://reviewboard.mozilla.org/r/240878/#review252556

r=me with the removal of the ParamTraits stuff :-)

::: toolkit/components/backgroundhangmonitor/HangAnnotations.h:29
(Diff revision 3)
> +  void AddAnnotation(const nsString& aName, const int32_t aData);
> +  void AddAnnotation(const nsString& aName, const double aData);
> +  void AddAnnotation(const nsString& aName, const nsString& aData);
> +  void AddAnnotation(const nsString& aName, const nsCString& aData);
> +  void AddAnnotation(const nsString& aName, const bool aData);

As far as I can tell there are exactly 7 places where we call these methods in gecko:
nsContentUtils (UserInteracting)
ContentChild.cpp (PendingInput)
PluginModuleParent (HangUIShown, HangUIContinued, HangUIDontShow, pluginName, and pluginVersion)

Perhaps we can file a follow-up to just remove the AddAnnotation methods, and just expose a `nsTArray<HangAnnotation>` here instead?

That'll also allow us to completely avoid the ParamTraits stuff.

::: toolkit/components/backgroundhangmonitor/HangAnnotations.h:66
(Diff revision 3)
> +template<>
> +class ParamTraits<mozilla::BackgroundHangAnnotations>
> +  : public ParamTraits<nsTArray<mozilla::HangAnnotation>>
> +{
> +public:
> +  typedef mozilla::BackgroundHangAnnotations paramType;
> +};

If you use mozilla::ipc::IPDLParamTraits instead of IPC::ParamTraits you can avoid the need for the HangAnnotations ParamTraits impl.

::: toolkit/components/backgroundhangmonitor/HangAnnotations.h:67
(Diff revision 3)
> +} // namespace mozilla
> +
> +namespace IPC {
> +
> +template<>
> +class ParamTraits<mozilla::BackgroundHangAnnotations>

ParamTraits is a struct not a class :-)

::: toolkit/components/backgroundhangmonitor/HangAnnotations.h:74
(Diff revision 3)
> +template<>
> +class ParamTraits<mozilla::HangAnnotation>
> +{
> +public:
> +  typedef mozilla::HangAnnotation paramType;
> +  static void Write(Message* aMsg, const paramType& aParam);
> +  static bool Read(const Message* aMsg,
> +                   PickleIterator* aIter,
> +                   paramType* aResult);
> +};

mozilla::HangAnnotation already has IPDLParamTraits implemented for it. Let's use the IPDL-generated version instead.
Attachment #8972211 - Flags: review?(nika) → review+
Blocks: 1464464
Comment on attachment 8972211 [details]
Bug 1448040 - Remove HangMonitor/ChromeHangs

https://reviewboard.mozilla.org/r/240878/#review249678

> Is CombinedStacks still used? I don't think there will be any remaining observers?

There's still two usages in the file, mLateWriteStacks and a local empty variable used in conjunction with mLateWriteStacks. I'm not sure what you mean about observers, though, so let me know if I misunderstood what you meant.
Pushed by dothayer@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/da12c077747f
Remove HangMonitor/ChromeHangs r=Nika
Backed out changeset da12c077747f (bug 1448040) for Android build bustage on build/src/obj-firefox/dist/include/mozilla/HangAnnotations.h. CLOSED TREE

Log:
https://treeherder.mozilla.org/logviewer.html#?job_id=180277766&repo=autoland&lineNumber=19492

[task 2018-05-25T17:03:23.896Z] 17:03:23     INFO -  In file included from /builds/worker/workspace/build/src/obj-firefox/widget/android/jni/Unified_cpp_widget_android_jni0.cpp:2:
[task 2018-05-25T17:03:23.896Z] 17:03:23     INFO -  In file included from /builds/worker/workspace/build/src/widget/android/jni/Utils.cpp:11:
[task 2018-05-25T17:03:23.896Z] 17:03:23     INFO -  In file included from /builds/worker/workspace/build/src/widget/android/nsAppShell.h:11:
[task 2018-05-25T17:03:23.896Z] 17:03:23     INFO -  In file included from /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/BackgroundHangMonitor.h:10:
[task 2018-05-25T17:03:23.896Z] 17:03:23     INFO -  In file included from /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/CPUUsageWatcher.h:12:
[task 2018-05-25T17:03:23.896Z] 17:03:23     INFO -  /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/HangAnnotations.h:12:10: fatal error: 'mozilla/HangTypes.h' file not found
[task 2018-05-25T17:03:23.896Z] 17:03:23     INFO -  #include "mozilla/HangTypes.h"
[task 2018-05-25T17:03:23.896Z] 17:03:23     INFO -           ^~~~~~~~~~~~~~~~~~~~~
[task 2018-05-25T17:03:23.896Z] 17:03:23     INFO -  1 error generated.
[task 2018-05-25T17:03:23.896Z] 17:03:23     INFO -  /builds/worker/workspace/build/src/config/rules.mk:1031: recipe for target 'Unified_cpp_widget_android_jni0.o' failed
[task 2018-05-25T17:03:23.896Z] 17:03:23     INFO -  make[4]: *** [Unified_cpp_widget_android_jni0.o] Error 1
[task 2018-05-25T17:03:23.896Z] 17:03:23     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/widget/android/jni'
[task 2018-05-25T17:03:23.896Z] 17:03:23     INFO -  /builds/worker/workspace/build/src/config/recurse.mk:73: recipe for target 'widget/android/jni/target' failed
[task 2018-05-25T17:03:23.896Z] 17:03:23     INFO -  make[3]: *** [widget/android/jni/target] Error 2
[task 2018-05-25T17:03:23.896Z] 17:03:23     INFO -  make[3]: *** Waiting for unfinished jobs....
[task 2018-05-25T17:03:23.896Z] 17:03:23     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/widget'
[task 2018-05-25T17:03:23.896Z] 17:03:23     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/widget'
[task 2018-05-25T17:03:23.896Z] 17:03:23     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/widget'
[task 2018-05-25T17:03:23.896Z] 17:03:23     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/widget'
[task 2018-05-25T17:03:26.202Z] 17:03:26     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/presentation'
[task 2018-05-25T17:03:26.206Z] 17:03:26     INFO -  /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -std=gnu++14 --target=arm-linux-androideabi -o Unified_cpp_dom_presentation0.o -c -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -DNDEBUG=1 -DTRIMMED=1 -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/dom/presentation -I/builds/worker/workspace/build/src/obj-firefox/dom/presentation -I/builds/worker/workspace/build/src/dom/base -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -gcc-toolchain /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 -D__ANDROID_API__=9 -Qunused-arguments -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wloop-analysis -Wc++1z-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -fno-sized-deallocation -fno-short-enums -fno-exceptions -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++/include -I/builds/worker/workspace/build/src/android-ndk/sources/android/support/include -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++abi/include -march=armv7-a -mthumb -mfpu=vfpv3-d16 -mfloat-abi=softfp -mno-unaligned-access -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pipe -g -Oz -funwind-tables -Werror  -MD -MP -MF .deps/Unified_cpp_dom_presentation0.o.pp   /builds/worker/workspace/build/src/obj-firefox/dom/presentation/Unified_cpp_dom_presentation0.cpp
[task 2018-05-25T17:03:26.206Z] 17:03:26     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/presentation'
[task 2018-05-25T17:03:26.414Z] 17:03:26     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/widget/android'
[task 2018-05-25T17:03:26.414Z] 17:03:26     INFO -  /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -std=gnu++14 --target=arm-linux-androideabi -o Unified_cpp_widget_android0.o -c -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -DNDEBUG=1 -DTRIMMED=1 -DOS_POSIX=1 -DOS_LINUX=1 -DMOZ_NATIVE_DEVICES -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/widget/android -I/builds/worker/workspace/build/src/obj-firefox/widget/android -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/docshell/base -I/builds/worker/workspace/build/src/dom/base -I/builds/worker/workspace/build/src/dom/system/android -I/builds/worker/workspace/build/src/gfx/2d -I/builds/worker/workspace/build/src/gfx/vr -I/builds/worker/workspace/build/src/layout/painting -I/builds/worker/workspace/build/src/netwerk/base -I/builds/worker/workspace/build/src/netwerk/cache -I/builds/worker/workspace/build/src/widget -I/builds/worker/workspace/build/src/xpcom/threads -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -gcc-toolchain /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 -D__ANDROID_API__=9 -Qunused-arguments -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wloop-analysis -Wc++1z-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -fno-sized-deallocation -fno-short-enums -fno-exceptions -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++/include -I/builds/worker/workspace/build/src/android-ndk/sources/android/support/include -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++abi/include -march=armv7-a -mthumb -mfpu=vfpv3-d16 -mfloat-abi=softfp -mno-unaligned-access -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pipe -g -Oz -funwind-tables -Werror -Wno-error=shadow  -MD -MP -MF .deps/Unified_cpp_widget_android0.o.pp   /builds/worker/workspace/build/src/obj-firefox/widget/android/Unified_cpp_widget_android0.cpp
[task 2018-05-25T17:03:26.414Z] 17:03:26     INFO -  In file included from /builds/worker/workspace/build/src/obj-firefox/widget/android/Unified_cpp_widget_android0.cpp:20:
[task 2018-05-25T17:03:26.415Z] 17:03:26     INFO -  In file included from /builds/worker/workspace/build/src/widget/android/AndroidBridge.cpp:20:
[task 2018-05-25T17:03:26.416Z] 17:03:26     INFO -  /builds/worker/workspace/build/src/widget/android/nsAppShell.h:39:26: error: no member named 'HangMonitor' in namespace 'mozilla'
[task 2018-05-25T17:03:26.416Z] 17:03:26     INFO -          typedef mozilla::HangMonitor::ActivityType Type;
[task 2018-05-25T17:03:26.416Z] 17:03:26     INFO -                  ~~~~~~~~~^
[task 2018-05-25T17:03:26.417Z] 17:03:26     INFO -  /builds/worker/workspace/build/src/widget/android/nsAppShell.h:69:20: error: 'nsAppShell::Event::Type' (aka 'int') is not a class, namespace, or enumeration
[task 2018-05-25T17:03:26.417Z] 17:03:26     INFO -              return Type::kGeneralActivity;
[task 2018-05-25T17:03:26.417Z] 17:03:26     INFO -                     ^
[task 2018-05-25T17:03:26.417Z] 17:03:26     INFO -  /builds/worker/workspace/build/src/widget/android/nsAppShell.h:252:28: error: 'nsAppShell::Event::Type' (aka 'int') is not a class, namespace, or enumeration
[task 2018-05-25T17:03:26.417Z] 17:03:26     INFO -                      Event::Type::kUIActivity) ? LATENCY_UI : LATENCY_OTHER;
[task 2018-05-25T17:03:26.417Z] 17:03:26     INFO -                             ^
[task 2018-05-25T17:03:26.417Z] 17:03:26     INFO -  In file included from /builds/worker/workspace/build/src/obj-firefox/widget/android/Unified_cpp_widget_android0.cpp:65:
[task 2018-05-25T17:03:26.417Z] 17:03:26     INFO -  In file included from /builds/worker/workspace/build/src/widget/android/GeckoEditableSupport.cpp:7:
[task 2018-05-25T17:03:26.417Z] 17:03:26     INFO -  /builds/worker/workspace/build/src/widget/android/GeckoEditableSupport.h:154:47: error: 'nsAppShell::Event::Type' (aka 'int') is not a class, namespace, or enumeration
[task 2018-05-25T17:03:26.417Z] 17:03:26     INFO -                      return nsAppShell::Event::Type::kUIActivity;
[task 2018-05-25T17:03:26.418Z] 17:03:26     INFO -                                                ^
[task 2018-05-25T17:03:26.418Z] 17:03:26     INFO -  /builds/worker/workspace/build/src/widget/android/GeckoEditableSupport.h:156:43: error: 'nsAppShell::Event::Type' (aka 'int') is not a class, namespace, or enumeration
[task 2018-05-25T17:03:26.418Z] 17:03:26     INFO -                  return nsAppShell::Event::Type::kGeneralActivity;
[task 2018-05-25T17:03:26.418Z] 17:03:26     INFO -                                            ^
[task 2018-05-25T17:03:26.418Z] 17:03:26     INFO -  5 errors generated.
[task 2018-05-25T17:03:26.419Z] 17:03:26     INFO -  /builds/worker/workspace/build/src/config/rules.mk:1031: recipe for target 'Unified_cpp_widget_android0.o' failed
[task 2018-05-25T17:03:26.419Z] 17:03:26     INFO -  make[4]: *** [Unified_cpp_widget_android0.o] Error 1
[task 2018-05-25T17:03:26.420Z] 17:03:26     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/widget/android'
[task 2018-05-25T17:03:26.421Z] 17:03:26     INFO -  make[4]: *** Waiting for unfinished jobs....
[task 2018-05-25T17:03:26.656Z] 17:03:26     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/widget/android'
[task 2018-05-25T17:03:26.659Z] 17:03:26     INFO -  /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -std=gnu++14 --target=arm-linux-androideabi -o Unified_cpp_widget_android1.o -c -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -DNDEBUG=1 -DTRIMMED=1 -DOS_POSIX=1 -DOS_LINUX=1 -DMOZ_NATIVE_DEVICES -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/widget/android -I/builds/worker/workspace/build/src/obj-firefox/widget/android -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/docshell/base -I/builds/worker/workspace/build/src/dom/base -I/builds/worker/workspace/build/src/dom/system/android -I/builds/worker/workspace/build/src/gfx/2d -I/builds/worker/workspace/build/src/gfx/vr -I/builds/worker/workspace/build/src/layout/painting -I/builds/worker/workspace/build/src/netwerk/base -I/builds/worker/workspace/build/src/netwerk/cache -I/builds/worker/workspace/build/src/widget -I/builds/worker/workspace/build/src/xpcom/threads -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -gcc-toolchain /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 -D__ANDROID_API__=9 -Qunused-arguments -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wloop-analysis -Wc++1z-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -fno-sized-deallocation -fno-short-enums -fno-exceptions -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++/include -I/builds/worker/workspace/build/src/android-ndk/sources/android/support/include -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++abi/include -march=armv7-a -mthumb -mfpu=vfpv3-d16 -mfloat-abi=softfp -mno-unaligned-access -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pipe -g -Oz -funwind-tables -Werror -Wno-error=shadow  -MD -MP -MF .deps/Unified_cpp_widget_android1.o.pp   /builds/worker/workspace/build/src/obj-firefox/widget/android/Unified_cpp_widget_android1.cpp
[task 2018-05-25T17:03:26.659Z] 17:03:26     INFO -  In file included from /builds/worker/workspace/build/src/obj-firefox/widget/android/Unified_cpp_widget_android1.cpp:11:
[task 2018-05-25T17:03:26.661Z] 17:03:26     INFO -  In file included from /builds/worker/workspace/build/src/widget/android/nsScreenManagerAndroid.cpp:14:
[task 2018-05-25T17:03:26.662Z] 17:03:26     INFO -  /builds/worker/workspace/build/src/widget/android/nsAppShell.h:39:26: error: no member named 'HangMonitor' in namespace 'mozilla'
[task 2018-05-25T17:03:26.662Z] 17:03:26     INFO -          typedef mozilla::HangMonitor::ActivityType Type;
[task 2018-05-25T17:03:26.663Z] 17:03:26     INFO -                  ~~~~~~~~~^
[task 2018-05-25T17:03:26.664Z] 17:03:26     INFO -  /builds/worker/workspace/build/src/widget/android/nsAppShell.h:69:20: error: 'nsAppShell::Event::Type' (aka 'int') is not a class, namespace, or enumeration
[task 2018-05-25T17:03:26.664Z] 17:03:26     INFO -              return Type::kGeneralActivity;
[task 2018-05-25T17:03:26.665Z] 17:03:26     INFO -                     ^
[task 2018-05-25T17:03:26.668Z] 17:03:26     INFO -  /builds/worker/workspace/build/src/widget/android/nsAppShell.h:252:28: error: 'nsAppShell::Event::Type' (aka 'int') is not a class, namespace, or enumeration
[task 2018-05-25T17:03:26.668Z] 17:03:26     INFO -                      Event::Type::kUIActivity) ? LATENCY_UI : LATENCY_OTHER;
[task 2018-05-25T17:03:26.668Z] 17:03:26     INFO -                             ^
[task 2018-05-25T17:03:26.669Z] 17:03:26     INFO -  In file included from /builds/worker/workspace/build/src/obj-firefox/widget/android/Unified_cpp_widget_android1.cpp:29:
[task 2018-05-25T17:03:26.669Z] 17:03:26     INFO -  In file included from /builds/worker/workspace/build/src/widget/android/nsWindow.cpp:78:
[task 2018-05-25T17:03:26.669Z] 17:03:26     INFO -  /builds/worker/workspace/build/src/widget/android/GeckoEditableSupport.h:154:47: error: 'nsAppShell::Event::Type' (aka 'int') is not a class, namespace, or enumeration
[task 2018-05-25T17:03:26.670Z] 17:03:26     INFO -                      return nsAppShell::Event::Type::kUIActivity;
[task 2018-05-25T17:03:26.670Z] 17:03:26     INFO -                                                ^
[task 2018-05-25T17:03:26.671Z] 17:03:26     INFO -  /builds/worker/workspace/build/src/widget/android/GeckoEditableSupport.h:156:43: error: 'nsAppShell::Event::Type' (aka 'int') is not a class, namespace, or enumeration
[task 2018-05-25T17:03:26.671Z] 17:03:26     INFO -                  return nsAppShell::Event::Type::kGeneralActivity;
[task 2018-05-25T17:03:26.671Z] 17:03:26     INFO -                                            ^
[task 2018-05-25T17:03:26.672Z] 17:03:26     INFO -  In file included from /builds/worker/workspace/build/src/obj-firefox/widget/android/Unified_cpp_widget_android1.cpp:29:
[task 2018-05-25T17:03:26.672Z] 17:03:26     INFO -  /builds/worker/workspace/build/src/widget/android/nsWindow.cpp:358:39: error: 'nsAppShell::Event::Type' (aka 'int') is not a class, namespace, or enumeration
[task 2018-05-25T17:03:26.673Z] 17:03:26     INFO -              return nsAppShell::Event::Type::kUIActivity;
[task 2018-05-25T17:03:26.673Z] 17:03:26     INFO -                                        ^
[task 2018-05-25T17:03:26.673Z] 17:03:26     INFO -  6 errors generated.
[task 2018-05-25T17:03:26.674Z] 17:03:26     INFO -  /builds/worker/workspace/build/src/config/rules.mk:1031: recipe for target 'Unified_cpp_widget_android1.o' failed
[task 2018-05-25T17:03:26.675Z] 17:03:26     INFO -  make[4]: *** [Unified_cpp_widget_android1.o] Error 1
[task 2018-05-25T17:03:26.675Z] 17:03:26     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/widget/android'
[task 2018-05-25T17:03:26.676Z] 17:03:26     INFO -  /builds/worker/workspace/build/src/config/recurse.mk:73: recipe for target 'widget/android/target' failed
[task 2018-05-25T17:03:26.677Z] 17:03:26     INFO -  make[3]: *** [widget/android/target] Error 2
[task 2018-05-25T17:03:27.178Z] 17:03:27     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/widget'
[task 2018-05-25T17:03:27.180Z] 17:03:27     INFO -  /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -std=gnu++14 --target=arm-linux-androideabi -o nsBaseWidget.o -c -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -DNDEBUG=1 -DTRIMMED=1 -DMOZ_CROSS_PROCESS_IME -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/widget -I/builds/worker/workspace/build/src/obj-firefox/widget -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/dom/base -I/builds/worker/workspace/build/src/dom/ipc -I/builds/worker/workspace/build/src/layout/base -I/builds/worker/workspace/build/src/layout/forms -I/builds/worker/workspace/build/src/layout/generic -I/builds/worker/workspace/build/src/layout/painting -I/builds/worker/workspace/build/src/layout/xul -I/builds/worker/workspace/build/src/view -I/builds/worker/workspace/build/src/widget -I/builds/worker/workspace/build/src/widget/headless -I/builds/worker/workspace/build/src/widget/android -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -gcc-toolchain /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 -D__ANDROID_API__=9 -Qunused-arguments -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wloop-analysis -Wc++1z-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -fno-sized-deallocation -fno-short-enums -fno-exceptions -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++/include -I/builds/worker/workspace/build/src/android-ndk/sources/android/support/include -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++abi/include -march=armv7-a -mthumb -mfpu=vfpv3-d16 -mfloat-abi=softfp -mno-unaligned-access -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pipe -g -Oz -funwind-tables -Werror -I/builds/worker/workspace/build/src/obj-firefox/dist/include/cairo  -MD -MP -MF .deps/nsBaseWidget.o.pp   /builds/worker/workspace/build/src/widget/nsBaseWidget.cpp
[task 2018-05-25T17:03:27.180Z] 17:03:27     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/widget'
[task 2018-05-25T17:03:29.929Z] 17:03:29     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/widget'
[task 2018-05-25T17:03:29.931Z] 17:03:29     INFO -  /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -std=gnu++14 --target=arm-linux-androideabi -o Unified_cpp_widget2.o -c -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -DNDEBUG=1 -DTRIMMED=1 -DMOZ_CROSS_PROCESS_IME -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/widget -I/builds/worker/workspace/build/src/obj-firefox/widget -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/dom/base -I/builds/worker/workspace/build/src/dom/ipc -I/builds/worker/workspace/build/src/layout/base -I/builds/worker/workspace/build/src/layout/forms -I/builds/worker/workspace/build/src/layout/generic -I/builds/worker/workspace/build/src/layout/painting -I/builds/worker/workspace/build/src/layout/xul -I/builds/worker/workspace/build/src/view -I/builds/worker/workspace/build/src/widget -I/builds/worker/workspace/build/src/widget/headless -I/builds/worker/workspace/build/src/widget/android -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -gcc-toolchain /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 -D__ANDROID_API__=9 -Qunused-arguments -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wloop-analysis -Wc++1z-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -fno-sized-deallocation -fno-short-enums -fno-exceptions -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++/include -I/builds/worker/workspace/build/src/android-ndk/sources/android/support/include -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++abi/include -march=armv7-a -mthumb -mfpu=vfpv3-d16 -mfloat-abi=softfp -mno-unaligned-access -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pipe -g -Oz -funwind-tables -Werror -I/builds/worker/workspace/build/src/obj-firefox/dist/include/cairo  -MD -MP -MF .deps/Unified_cpp_widget2.o.pp   /builds/worker/workspace/build/src/obj-firefox/widget/Unified_cpp_widget2.cpp
[task 2018-05-25T17:03:29.932Z] 17:03:29     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/widget'
[task 2018-05-25T17:03:31.814Z] 17:03:31     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/widget'
[task 2018-05-25T17:03:31.817Z] 17:03:31     INFO -  /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -std=gnu++14 --target=arm-linux-androideabi -o Unified_cpp_widget1.o -c -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -DNDEBUG=1 -DTRIMMED=1 -DMOZ_CROSS_PROCESS_IME -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/widget -I/builds/worker/workspace/build/src/obj-firefox/widget -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/dom/base -I/builds/worker/workspace/build/src/dom/ipc -I/builds/worker/workspace/build/src/layout/base -I/builds/worker/workspace/build/src/layout/forms -I/builds/worker/workspace/build/src/layout/generic -I/builds/worker/workspace/build/src/layout/painting -I/builds/worker/workspace/build/src/layout/xul -I/builds/worker/workspace/build/src/view -I/builds/worker/workspace/build/src/widget -I/builds/worker/workspace/build/src/widget/headless -I/builds/worker/workspace/build/src/widget/android -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -gcc-toolchain /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 -D__ANDROID_API__=9 -Qunused-arguments -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wloop-analysis -Wc++1z-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -fno-sized-deallocation -fno-short-enums -fno-exceptions -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++/include -I/builds/worker/workspace/build/src/android-ndk/sources/android/support/include -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++abi/include -march=armv7-a -mthumb -mfpu=vfpv3-d16 -mfloat-abi=softfp -mno-unaligned-access -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pipe -g -Oz -funwind-tables -Werror -I/builds/worker/workspace/build/src/obj-firefox/dist/include/cairo  -MD -MP -MF .deps/Unified_cpp_widget1.o.pp   /builds/worker/workspace/build/src/obj-firefox/widget/Unified_cpp_widget1.cpp
[task 2018-05-25T17:03:31.818Z] 17:03:31     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/widget'
[task 2018-05-25T17:03:34.521Z] 17:03:34     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/widget'
[task 2018-05-25T17:03:34.527Z] 17:03:34     INFO -  /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -std=gnu++14 --target=arm-linux-androideabi -o Unified_cpp_widget0.o -c -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -DNDEBUG=1 -DTRIMMED=1 -DMOZ_CROSS_PROCESS_IME -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/widget -I/builds/worker/workspace/build/src/obj-firefox/widget -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/dom/base -I/builds/worker/workspace/build/src/dom/ipc -I/builds/worker/workspace/build/src/layout/base -I/builds/worker/workspace/build/src/layout/forms -I/builds/worker/workspace/build/src/layout/generic -I/builds/worker/workspace/build/src/layout/painting -I/builds/worker/workspace/build/src/layout/xul -I/builds/worker/workspace/build/src/view -I/builds/worker/workspace/build/src/widget -I/builds/worker/workspace/build/src/widget/headless -I/builds/worker/workspace/build/src/widget/android -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-9/arch-arm/usr/include -gcc-toolchain /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 -D__ANDROID_API__=9 -Qunused-arguments -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wloop-analysis -Wc++1z-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -fno-sized-deallocation -fno-short-enums -fno-exceptions -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++/include -I/builds/worker/workspace/build/src/android-ndk/sources/android/support/include -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++abi/include -march=armv7-a -mthumb -mfpu=vfpv3-d16 -mfloat-abi=softfp -mno-unaligned-access -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pipe -g -Oz -funwind-tables -Werror -I/builds/worker/workspace/build/src/obj-firefox/dist/include/cairo  -MD -MP -MF .deps/Unified_cpp_widget0.o.pp   /builds/worker/workspace/build/src/obj-firefox/widget/Unified_cpp_widget0.cpp
[task 2018-05-25T17:03:34.527Z] 17:03:34     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/widget'

Push with failures:
https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=da12c077747fdd6e168795cbc60d0a6dad7ef644

Backout:
https://hg.mozilla.org/integration/autoland/rev/ff2ef4fe0c6c907e7692c1e92bd1601e2295c468
Flags: needinfo?(dothayer)
Pushed by dothayer@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/531593bacc4e
Remove HangMonitor/ChromeHangs r=Nika
Anyone more familiar with Android than I am: how did this try build succeed for Android 4.0 API16 while the autoland build failed?

https://treeherder.mozilla.org/#/jobs?repo=try&revision=061f93cd71a06d6d542dc1003172995c570a00dc&filter-searchStr=Android
Try builds are always clobbers, perhaps that has something to do with it.  I think that means you got bit by headers/source files cargo-culting includes, rather than explicitly including what was needed.
Doug, did you get a chance to retry these with Nathan's suggestion?
Flags: needinfo?(dothayer)
I haven't yet - sorry, I actually meant to ask - Nathan, can you elaborate a bit for me? I don't know enough about the build system to follow this - how does cargo-culting includes cause us to fail to find "mozilla/HangTypes.h"? And are you saying the solution is to touch the clobber file, or no?
Flags: needinfo?(nfroyd)
(In reply to Doug Thayer [:dthayer] (PTO on June 4) from comment #21)
> I haven't yet - sorry, I actually meant to ask - Nathan, can you elaborate a
> bit for me? I don't know enough about the build system to follow this - how
> does cargo-culting includes cause us to fail to find "mozilla/HangTypes.h"?
> And are you saying the solution is to touch the clobber file, or no?

I misspoke here; it looks like something between your try run and actual landing changed what headers get included in widget/android/jni/, perhaps?  mozilla/HangTypes.h comes from ipc/ipdl/_ipdlheaders, which isn't searched for widget/android/jni/.  Maybe just including chromium-config.mozbuild in that directory, like so:

http://dxr.mozilla.org/mozilla-central/source/widget/android/moz.build#64

would fix things?  Have you tried doing another try run since your patch was backed out?
Flags: needinfo?(nfroyd)
Pushed by dothayer@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/167d3bf700d3
Remove HangMonitor/ChromeHangs r=Nika
https://hg.mozilla.org/mozilla-central/rev/167d3bf700d3
Status: ASSIGNED → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla62
Depends on: 1470938
You need to log in before you can comment on or make changes to this bug.