1970422 - ThreadSanitizer: data race [@ PLDHashTable::MakeEntryHandle] vs. [@ RawRemove]

Status: RESOLVED FIXED

(Core :: Graphics: Text, defect)

Description

[Tyson Smith [:tsmith]]

Attachment: tsan-log.txt

Found with m-c 20250604-b75f7675fbad (--enable-thread-sanitizer)

This was found by visiting a live website with a TSan build.

STR:

• Launch browser and visit: http://domofond.ru/

Or via site-scout:

$ pip install fuzzfetch site-scout --upgrade
$ fuzzfetch -d --fuzzing -n firefox
$ site-scout ./firefox/firefox --explore -u <url>

WARNING: ThreadSanitizer: data race (pid=37743)
  Read of size 4 at 0x720800357e34 by main thread (mutexes: write M0):
    #0 PLDHashTable::MakeEntryHandle(void const*, std::nothrow_t const&) /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:640:7 (libxul.so+0x38edd69) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #1 PLDHashTable::MakeEntryHandle(void const*) /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:674:14 (libxul.so+0x38ee27a) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #2 WithEntryHandle<(lambda at /builds/worker/workspace/obj-build/dist/include/nsTHashtable.h:438:9)> /builds/worker/workspace/obj-build/dist/include/PLDHashTable.h:605:35 (libxul.so+0x50e76ac) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #3 WithEntryHandle<(lambda at /builds/worker/workspace/obj-build/dist/include/nsTHashtable.h:309:15)> /builds/worker/workspace/obj-build/dist/include/nsTHashtable.h:436:25 (libxul.so+0x50e76ac)
    #4 PutEntry /builds/worker/workspace/obj-build/dist/include/nsTHashtable.h:308:12 (libxul.so+0x50e76ac)
    #5 gfxFontEntry::ShareFontTableAndGetBlob(unsigned int, nsTArray<unsigned char>*) /builds/worker/checkouts/gecko/gfx/thebes/gfxFontEntry.cpp:538:48 (libxul.so+0x50e76ac)
    #6 gfxFontEntry::GetFontTable(unsigned int) /builds/worker/checkouts/gecko/gfx/thebes/gfxFontEntry.cpp:571:10 (libxul.so+0x50e7a6d) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #7 gfxFontconfigFontEntry::GetFontTable(unsigned int) /builds/worker/checkouts/gecko/gfx/thebes/gfxFcPlatformFontList.cpp:531:24 (libxul.so+0x5082a4d) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #8 gfxFontEntry::TryGetSVGData(gfxFont const*) /builds/worker/checkouts/gecko/gfx/thebes/gfxFontEntry.cpp:344:27 (libxul.so+0x50d246e) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #9 gfxFont::SetupGlyphExtents(mozilla::gfx::DrawTarget*, unsigned int, bool, gfxGlyphExtents*) /builds/worker/checkouts/gecko/gfx/thebes/gfxFont.cpp:4113:19 (libxul.so+0x50e3238) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #10 gfxTextRun::FetchGlyphExtents(mozilla::gfx::DrawTarget*) const /builds/worker/checkouts/gecko/gfx/thebes/gfxTextRun.cpp:1647:19 (libxul.so+0x5146bad) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    ...

  Previous write of size 4 at 0x720800357e34 by thread T29:
    #0 RawRemove /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:583:14 (libxul.so+0x38ee7a2) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #1 operator() /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:539:9 (libxul.so+0x38ee7a2)
    #2 SearchTable<(PLDHashTable::SearchReason)0, (lambda at /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:538:7), (lambda at /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:542:7)> /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:377:16 (libxul.so+0x38ee7a2)
    #3 PLDHashTable::Remove(void const*) /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:536:3 (libxul.so+0x38ee7a2)
    #4 RemoveEntry /builds/worker/workspace/obj-build/dist/include/nsTHashtable.h:348:12 (libxul.so+0x50e759f) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #5 ~FontTableBlobData /builds/worker/checkouts/gecko/gfx/thebes/gfxFontEntry.cpp:422:19 (libxul.so+0x50e759f)
    #6 gfxFontEntry::FontTableHashEntry::DeleteFontTableBlobData(void*) /builds/worker/checkouts/gecko/gfx/thebes/gfxFontEntry.cpp:508:3 (libxul.so+0x50e759f)
    #7 destroy_user_data /builds/worker/checkouts/gecko/gfx/harfbuzz/src/graph/../hb-blob.hh:47:7 (libxul.so+0x4ee57a8) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #8 ~hb_blob_t /builds/worker/checkouts/gecko/gfx/harfbuzz/src/graph/../hb-blob.hh:41:19 (libxul.so+0x4ee57a8)
    #9 hb_object_destroy<hb_blob_t> /builds/worker/checkouts/gecko/gfx/harfbuzz/src/graph/../hb-object.hh:297:11 (libxul.so+0x4ee57a8)
    #10 hb_blob_destroy /builds/worker/checkouts/gecko/gfx/harfbuzz/src/hb-blob.cc:264:8 (libxul.so+0x4ee57a8)
    ...

  Location is heap block of size 32 at 0x720800357e20 allocated by thread T29:
    #0 malloc /builds/worker/fetches/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:666:5 (firefox-bin+0xc1b10) (BuildId: 65c4347c166fff50757e9720fc6bd27433f27f9e)
    #1 moz_xmalloc /builds/worker/checkouts/gecko/memory/mozalloc/mozalloc.cpp:52:15 (firefox-bin+0x148f88) (BuildId: 65c4347c166fff50757e9720fc6bd27433f27f9e)
    #2 operator new /builds/worker/workspace/obj-build/dist/include/mozilla/cxxalloc.h:33:10 (libxul.so+0x50e786e) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #3 mozilla::detail::UniqueSelector<nsTHashtable<gfxFontEntry::FontTableHashEntry>>::SingleObject mozilla::MakeUnique<nsTHashtable<gfxFontEntry::FontTableHashEntry>, int>(int&&) /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:607:23 (libxul.so+0x50e786e)
    #4 gfxFontEntry::ShareFontTableAndGetBlob(unsigned int, nsTArray<unsigned char>*) /builds/worker/checkouts/gecko/gfx/thebes/gfxFontEntry.cpp:535:23 (libxul.so+0x50e781c) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #5 gfxFontEntry::GetFontTable(unsigned int) /builds/worker/checkouts/gecko/gfx/thebes/gfxFontEntry.cpp:571:10 (libxul.so+0x50e7a6d) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #6 gfxFontconfigFontEntry::GetFontTable(unsigned int) /builds/worker/checkouts/gecko/gfx/thebes/gfxFcPlatformFontList.cpp:531:24 (libxul.so+0x5082a4d) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #7 AutoTable /builds/worker/checkouts/gecko/gfx/thebes/gfxFontEntry.h:378:27 (libxul.so+0x50ca7b7) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #8 gfxFontEntry::UnitsPerEm() /builds/worker/checkouts/gecko/gfx/thebes/gfxFontEntry.cpp:270:13 (libxul.so+0x50ca7b7)
    #9 gfxFont::gfxFont(RefPtr<mozilla::gfx::UnscaledFont> const&, gfxFontEntry*, gfxFontStyle const*, gfxFont::AntialiasOption) /builds/worker/checkouts/gecko/gfx/thebes/gfxFont.cpp:1007:25 (libxul.so+0x50ca4fe) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #10 gfxFT2FontBase::gfxFT2FontBase(RefPtr<mozilla::gfx::UnscaledFontFreeType> const&, RefPtr<mozilla::gfx::SharedFTFace>&&, gfxFontEntry*, gfxFontStyle const*, int, bool) /builds/worker/checkouts/gecko/gfx/thebes/gfxFT2FontBase.cpp:36:7 (libxul.so+0x507cb6d) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    ...

  Mutex M0 (0x7254000d8930) created at:
    #0 pthread_rwlock_init /builds/worker/fetches/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1486:3 (firefox-bin+0xc6800) (BuildId: 65c4347c166fff50757e9720fc6bd27433f27f9e)
    #1 mozilla::detail::RWLockImpl::RWLockImpl() /builds/worker/checkouts/gecko/mozglue/misc/RWLock_posix.cpp:19:3 (firefox-bin+0x1b1c27) (BuildId: 65c4347c166fff50757e9720fc6bd27433f27f9e)
    #2 mozilla::RWLock::RWLock(char const*) /builds/worker/checkouts/gecko/xpcom/threads/RWLock.cpp:11:9 (libxul.so+0x397c665) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #3 gfxFontEntry::gfxFontEntry(nsTSubstring<char> const&, bool) /builds/worker/checkouts/gecko/gfx/thebes/gfxFontEntry.cpp:61:7 (libxul.so+0x50e638a) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #4 gfxFT2FontEntryBase /builds/worker/checkouts/gecko/gfx/thebes/gfxFT2FontBase.h:21:59 (libxul.so+0x508a45d) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #5 gfxFontconfigFontEntry /builds/worker/checkouts/gecko/gfx/thebes/gfxFcPlatformFontList.cpp:285:7 (libxul.so+0x508a45d)
    #6 gfxFcPlatformFontList::CreateFontEntry(mozilla::fontlist::Face*, mozilla::fontlist::Family const*) /builds/worker/checkouts/gecko/gfx/thebes/gfxFcPlatformFontList.cpp:2126:18 (libxul.so+0x508a45d)
    #7 operator() /builds/worker/checkouts/gecko/gfx/thebes/gfxPlatformFontList.cpp:1938:40 (libxul.so+0x512d2ac) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #8 OrInsertWith<(lambda at /builds/worker/checkouts/gecko/gfx/thebes/gfxPlatformFontList.cpp:1938:27)> /builds/worker/workspace/obj-build/dist/include/nsBaseHashtable.h:739:23 (libxul.so+0x512d2ac)
    #9 operator()<nsBaseHashtable<nsPtrHashKey<const mozilla::fontlist::Face>, RefPtr<gfxFontEntry>, gfxFontEntry *, nsDefaultConverter<RefPtr<gfxFontEntry>, gfxFontEntry *> >::EntryHandle> /builds/worker/workspace/obj-build/dist/include/nsBaseHashtable.h:436:26 (libxul.so+0x512d2ac)
    #10 operator()<nsTHashtable<nsBaseHashtableET<nsPtrHashKey<const mozilla::fontlist::Face>, RefPtr<gfxFontEntry> > >::EntryHandle> /builds/worker/workspace/obj-build/dist/include/nsBaseHashtable.h:849:18 (libxul.so+0x512d2ac)
    ...

  Thread T29 'DOM Worker' (tid=37875, running) created by main thread at:
    #0 pthread_create /builds/worker/fetches/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1023:3 (firefox-bin+0xc3671) (BuildId: 65c4347c166fff50757e9720fc6bd27433f27f9e)
    #1 _PR_CreateThread /builds/worker/checkouts/gecko/nsprpub/pr/src/pthreads/ptthread.c:429:10 (libnspr4.so+0x423ce) (BuildId: 1d878503d1e11301c40708d5e04a7e762e565f42)
    #2 PR_CreateThread /builds/worker/checkouts/gecko/nsprpub/pr/src/pthreads/ptthread.c:496:10 (libnspr4.so+0x376d4) (BuildId: 1d878503d1e11301c40708d5e04a7e762e565f42)
    #3 nsThread::Init(nsTSubstring<char> const&) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:615:20 (libxul.so+0x399ff67) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #4 mozilla::dom::WorkerThread::Create(mozilla::dom::WorkerThreadFriendKey const&) /builds/worker/checkouts/gecko/dom/workers/WorkerThread.cpp:97:7 (libxul.so+0x8a81f1b) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #5 mozilla::dom::workerinternals::RuntimeService::ScheduleWorker(mozilla::dom::WorkerPrivate&) /builds/worker/checkouts/gecko/dom/workers/RuntimeService.cpp:1408:37 (libxul.so+0x8a2e643) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #6 mozilla::dom::workerinternals::RuntimeService::RegisterWorker(mozilla::dom::WorkerPrivate&) /builds/worker/checkouts/gecko/dom/workers/RuntimeService.cpp:1291:19 (libxul.so+0x8a2d9ce) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #7 mozilla::dom::WorkerPrivate::Constructor(JSContext*, nsTSubstring<char16_t> const&, bool, mozilla::dom::WorkerKind, mozilla::dom::RequestCredentials, mozilla::dom::WorkerType, nsTSubstring<char16_t> const&, nsTSubstring<char> const&, mozilla::dom::WorkerLoadInfo*, mozilla::ErrorResult&, nsTString<char16_t>, std::function<void (bool)>&&, std::function<void ()>&&, mozilla::ipc::Endpoint<mozilla::dom::PRemoteWorkerNonLifeCycleOpControllerChild>&&) /builds/worker/checkouts/gecko/dom/workers/WorkerPrivate.cpp:3190:24 (libxul.so+0x8a5f924) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #8 mozilla::dom::Worker::Constructor(mozilla::dom::GlobalObject const&, mozilla::dom::TrustedScriptURLOrUSVString const&, mozilla::dom::WorkerOptions const&, mozilla::ErrorResult&) /builds/worker/checkouts/gecko/dom/workers/Worker.cpp:80:41 (libxul.so+0x8a3bbde) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #9 mozilla::dom::Worker_Binding::_constructor(JSContext*, unsigned int, JS::Value*) /builds/worker/workspace/obj-build/dom/bindings/./WorkerBinding.cpp:1084:52 (libxul.so+0x65f830c) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    #10 mozilla::dom::InterfaceObjectJSNative(JSContext*, unsigned int, JS::Value*) /builds/worker/checkouts/gecko/dom/bindings/BindingUtils.cpp:761:10 (libxul.so+0x6b5cad5) (BuildId: 12a97ecef8b3e45cbdf36999b4524bf884757ea3)
    ...

SUMMARY: ThreadSanitizer: data race /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:640:7 in PLDHashTable::MakeEntryHandle(void const*, std::nothrow_t const&)

Comment 1

[Teodor Tanasoaia [:teoxoy]]

Lee, would you be able to take a look at this?

Comment 2

[Jonathan Kew [:jfkthame]]

It looks like the issue here is a race between one thread removing an entry from the font table cache at the same time as another thread is updating it. Normally the font entry's lock is supposed to ensure safety while fields are being updated, but the FontTableBlobData attached to the hb_blob, through which it gets removed from the hashtable when harfbuzz is done with it, has a direct reference to the hashtable and doesn't lock the font entry before touching it.

So to fix this, we should give FontTableBlobData a reference to the font entry itself rather than the hashtable, and ensure it holds the lock before it modifies the hashtable.

Comment 3

[Jonathan Kew [:jfkthame]]

Attachment: Bug 1970422 - Ensure we hold the gfxFontEntry lock while deleting an entry from the table cache. r=#gfx-reviewers

Comment 4

[Jonathan Kew [:jfkthame]]

AFAICS the above patch should fix this, but I have not been able to test it locally because my tsan build fails with an unrelated crash before I have a chance to try loading the site from comment 0, or indeed any site. Tyson, if you have a chance to verify this, that'd be great.

Comment 5

[Tyson Smith [:tsmith]]

I am unable to verify this issue because I keep tripping over another long standing issue bug 1825621, sorry.

Comment 6

[Jonathan Kew [:jfkthame]]

Comment on attachment 9494649 [details]
Bug 1970422 - Ensure we hold the gfxFontEntry lock while deleting an entry from the table cache. r=#gfx-reviewers

Security Approval Request

• How easily could an exploit be constructed based on the patch?: Not easily. It's clear the patch is adding some locking, but unclear how to exploit the potential data race we're fixing.
• Do comments in the patch, the check-in comment, or tests included in the patch paint a bulls-eye on the security problem?: No
• Which branches (beta, release, and/or ESR) are affected by this flaw, and do the release status flags reflect this affected/unaffected state correctly?: all
• If not all supported branches, which bug introduced the flaw?: None
• Do you have backports for the affected branches?: Yes
• If not, how different, hard to create, and risky will they be?: (patch transplants cleanly to beta; very minor rebase needed for esr128 and esr115)
• How likely is this patch to cause regressions; how much testing does it need?: Unlikely: just a localized change to allow us to lock the font entry within the table-removal callback
• Is the patch ready to land after security approval is given?: Yes
• Is Android affected?: Yes

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:lsalzman, could you have a look please?

For more information, please visit BugBot documentation.

Comment 8

[Tom Ritter [:tjr] (2 days until extended leave)]

Comment on attachment 9494649 [details]
Bug 1970422 - Ensure we hold the gfxFontEntry lock while deleting an entry from the table cache. r=#gfx-reviewers

Approved to land and request uplift

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/128b05405784
https://hg.mozilla.org/integration/autoland/rev/fe0b9a15726f
Ensure we hold the gfxFontEntry lock while deleting an entry from the table cache. r=gfx-reviewers,lsalzman

Comment 10

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/fe0b9a15726f

Comment 11

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:jfkthame, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
  • See https://wiki.mozilla.org/Release_Management/Requesting_an_Uplift for documentation on how to request an uplift.
• If no, please set status-firefox141 to wontfix.

For more information, please visit BugBot documentation.

Comment 12

[Jonathan Kew [:jfkthame]]

Attachment: Bug 1970422 - Ensure we hold the gfxFontEntry lock while deleting an entry from the table cache. r=#gfx-reviewers

Original Revision: https://phabricator.services.mozilla.com/D253683

Comment 13

[Phabricator Automation]

firefox-beta Uplift Approval Request

• User impact if declined: Data race, potentially resulting in crash or UAF
• Code covered by automated testing: no
• Fix verified in Nightly: no
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: No reliable STR (see also comments 4-5)
• Risk associated with taking this patch: low
• Explanation of risk level: fairly simple patch just adds locking around a hashtable update
• String changes made/needed: none
• Is Android affected?: yes

Comment 14

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/81ad4cd013d0
https://hg.mozilla.org/releases/mozilla-beta/rev/664d3a2729fe

Comment 15

[Ryan VanderMeulen [:RyanVM]]

This will need rebased patches for ESR128/ESR115 but cherry-picks cleanly to ESR140. Please add those nominations when you get a chance.

Comment 16

[Jonathan Kew [:jfkthame]]

Attachment: (secure)

Original Revision: https://phabricator.services.mozilla.com/D253683

Comment 17

[Phabricator Automation]

firefox-esr140 Uplift Approval Request

• User impact if declined: data race may expose users to potential crash/UAF issues
• Code covered by automated testing: no
• Fix verified in Nightly: no
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: no known str
• Risk associated with taking this patch: low
• Explanation of risk level: fairly simple patch just adds locking around a hashtable update
• String changes made/needed: none
• Is Android affected?: yes

Comment 18

[Jonathan Kew [:jfkthame]]

Attachment: (secure)

Comment 19

[Phabricator Automation]

firefox-esr128 Uplift Approval Request

• User impact if declined: data race may expose users to potential crash/UAF issues
• Code covered by automated testing: no
• Fix verified in Nightly: no
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: no known str
• Risk associated with taking this patch: low
• Explanation of risk level: fairly simple patch just adds locking around a hashtable update
• String changes made/needed: none
• Is Android affected?: yes

Comment 20

[Jonathan Kew [:jfkthame]]

Attachment: (secure)

Original Revision: https://phabricator.services.mozilla.com/D256851

Comment 21

[Phabricator Automation]

firefox-esr115 Uplift Approval Request

• User impact if declined: (see previous comment)
• Code covered by automated testing: no
• Fix verified in Nightly: no
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: (see above)
• Risk associated with taking this patch: (see above)
• Explanation of risk level: (see above)
• String changes made/needed: (see above)
• Is Android affected?: yes

Comment 22

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr115/rev/a314e233cff1

Comment 23

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr128/rev/9c18d1619e12

Comment 24

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr140/rev/1ffa06871046

Comment 25

[Pascal Chevrel:pascalc]

Johnathan, your esr115 and esr128 patches are causing failures:
https://treeherder.mozilla.org/logviewer?job_id=517453472&repo=mozilla-esr128&lineNumber=27064

Could you have a look please? thanks!

Comment 26

[Jonathan Kew [:jfkthame]]

Attachment: (secure)

Comment 27

[Jonathan Kew [:jfkthame]]

(In reply to Pascal Chevrel:pascalc from comment #25)

Johnathan, your esr115 and esr128 patches are causing failures:
https://treeherder.mozilla.org/logviewer?job_id=517453472&repo=mozilla-esr128&lineNumber=27064

Could you have a look please? thanks!

Ugh, sorry about that. The above patch should fix this (I checked locally that esr128 builds with this correction; it should apply to esr115 as well).

Comment 28

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr128/rev/0b14d88ab445

Comment 29

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr115/rev/e63f125f5085