Closed Bug 1261264 Opened 3 years ago Closed 3 years ago

Valgrind jobs are going to permafail when Gecko 48 merges to Aurora (Invalid read of size 4 at dosprintf / nsTextFormatter::vsmprintf / nsTextFormatter::smprintf / CreateErrorText)

Categories

(Core :: XML, defect, critical)

defect
Not set
critical

Tracking

()

RESOLVED FIXED
mozilla48
Tracking Status
firefox46 --- unaffected
firefox47 + fixed
firefox48 + fixed

People

(Reporter: RyanVM, Unassigned)

References

Details

Attachments

(1 file)

[Tracking Requested - why for this release]: Valgrind permafail when Gecko 48 merges to Aurora.

https://treeherder.mozilla.org/logviewer.html#?job_id=18718784&repo=try

 5:52.68 TEST-UNEXPECTED-FAIL | valgrind-test | Invalid read of size 4 at dosprintf / nsTextFormatter::vsmprintf / nsTextFormatter::smprintf / CreateErrorText
 5:52.68 ==3395==    by 0xDC3D7AE: CreateErrorText (nsExpatDriver.cpp:834)
 5:52.68 ==3395==    by 0xDC3D7AE: CreateErrorText (nsExpatDriver.cpp:834)
 5:52.68 ==3395==    by 0xDC3D7AE: CreateErrorText (nsExpatDriver.cpp:834)
 5:52.68 ==3395==    by 0xDC3D7AE: CreateErrorText (nsExpatDriver.cpp:834)
 5:52.68 ==3395==    by 0xDC3D7AE: CreateErrorText (nsExpatDriver.cpp:834)
 5:52.68 ==3395==    by 0xDC3D7AE: nsExpatDriver::HandleError() (nsExpatDriver.cpp:940)
 5:52.68 ==3395==    by 0xDC40AD8: nsExpatDriver::ConsumeToken(nsScanner&, bool&) (nsExpatDriver.cpp:1179)
 5:52.68 ==3395==    by 0xDC3F202: nsParser::Tokenize(bool) (nsParser.cpp:1945)
 5:52.68 ==3395==    by 0xDC3FB80: nsParser::ResumeParse(bool, bool, bool) (nsParser.cpp:1464)
 5:52.68 ==3395==    by 0xDC404B8: nsParser::OnDataAvailable(nsIRequest*, nsISupports*, nsIInputStream*, unsigned long, unsigned int) (nsParser.cpp:1842)
 5:52.68 ==3395==    by 0xDF26B88: nsXMLHttpRequest::StreamReaderFunc(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*) (nsXMLHttpRequest.cpp:1751)
 5:52.68 ==3395==    by 0xD713FAF: nsPipeInputStream::ReadSegments(nsresult (*)(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*), void*, unsigned int, unsigned int*) (nsPipe3.cpp:1304)
 5:52.68 ==3395==    by 0xDF2649A: nsXMLHttpRequest::OnDataAvailable(nsIRequest*, nsISupports*, nsIInputStream*, unsigned long, unsigned int) (nsXMLHttpRequest.cpp:1815)
 5:52.68 ==3395==    by 0xD869894: mozilla::net::nsHttpChannel::OnDataAvailable(nsIRequest*, nsISupports*, nsIInputStream*, unsigned long, unsigned int) (nsHttpChannel.cpp:6305)
 5:52.68 ==3395==    by 0xD7801BD: nsInputStreamPump::OnStateTransfer() (nsInputStreamPump.cpp:603)
 5:52.68 ==3395==    by 0xD780324: nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) (nsInputStreamPump.cpp:430)
 5:52.68 ==3395==    by 0xD70E817: nsInputStreamReadyEvent::Run() (nsStreamUtils.cpp:94)
 5:52.68 ==3395==    by 0xD722A62: nsThread::ProcessNextEvent(bool, bool*) (nsThread.cpp:994)
 5:52.68 ==3395==    by 0xD73AE72: NS_ProcessNextEvent(nsIThread*, bool) (nsThreadUtils.cpp:297)
 5:52.68 ==3395==    by 0xD921C85: mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) (MessagePump.cpp:97)
 5:52.68 ==3395==    by 0xD90DBB8: RunHandler (message_loop.cc:227)
 5:52.68 ==3395==    by 0xD90DBB8: MessageLoop::Run() (message_loop.cc:201)
 5:52.68 ==3395==    by 0xE8DD1CE: nsBaseAppShell::Run() (nsBaseAppShell.cpp:156)
 5:52.68 ==3395==    by 0xEE86421: nsAppStartup::Run() (nsAppStartup.cpp:281)
 5:52.68 ==3395==    by 0xEEBA0E5: XREMain::XRE_mainRun() (nsAppRunner.cpp:4342)
 5:52.68 ==3395==    by 0xEEBA38B: XREMain::XRE_main(int, char**, nsXREAppData const*) (nsAppRunner.cpp:4439)
 5:52.68 ==3395==    by 0xEEBA5E8: XRE_main (nsAppRunner.cpp:4545)
 5:52.68 ==3395==    by 0x404F31: do_main(int, char**, char**, nsIFile*) (nsBrowserApp.cpp:220)
 5:52.68 ==3395==    by 0x4046A1: main (nsBrowserApp.cpp:360)
 5:52.68 ==3395==  Address 0x25740162 is 146 bytes inside a block of size 148 alloc'd
 5:52.68 ==3395==    at 0x4C293D5: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
 5:52.68 ==3395==    by 0xD6DC241: nsStringBuffer::Alloc(unsigned long) (nsSubstring.cpp:217)
 5:52.68 ==3395==    by 0xD6DC4FC: nsAString_internal::MutatePrep(unsigned int, char16_t**, unsigned int*) (nsTSubstring.cpp:133)
 5:52.68 ==3395==    by 0xD6DC553: nsAString_internal::ReplacePrepInternal(unsigned int, unsigned int, unsigned int, unsigned int) (nsTSubstring.cpp:195)
 5:52.68 ==3395==    by 0xD6DE7F8: nsAString_internal::ReplacePrep(unsigned int, unsigned int, unsigned int) (nsTSubstring.cpp:186)
 5:52.68 ==3395==    by 0xD6DE8F2: nsAString_internal::Assign(char16_t const*, unsigned int, mozilla::fallible_t const&) (nsTSubstring.cpp:342)
 5:52.68 ==3395==    by 0xD6DEA01: nsAString_internal::Assign(nsAString_internal const&) (nsTSubstring.cpp:390)
 5:52.68 ==3395==    by 0xDC3D4AA: nsParserMsgUtils::GetLocalizedStringByName(char const*, char const*, nsString&) (nsParserMsgUtils.cpp:42)
 5:52.68 ==3395==    by 0xDC3D78C: CreateErrorText (nsExpatDriver.cpp:828)
 5:52.69 ==3395==    by 0xDC3D78C: nsExpatDriver::HandleError() (nsExpatDriver.cpp:940)
 5:52.69 ==3395==    by 0xDC40AD8: nsExpatDriver::ConsumeToken(nsScanner&, bool&) (nsExpatDriver.cpp:1179)
 5:52.69 ==3395==    by 0xDC3F202: nsParser::Tokenize(bool) (nsParser.cpp:1945)
 5:52.69 ==3395==    by 0xDC3FB80: nsParser::ResumeParse(bool, bool, bool) (nsParser.cpp:1464)
 5:52.69 ==3395==    by 0xDC404B8: nsParser::OnDataAvailable(nsIRequest*, nsISupports*, nsIInputStream*, unsigned long, unsigned int) (nsParser.cpp:1842)
 5:52.69 ==3395==    by 0xDF26B88: nsXMLHttpRequest::StreamReaderFunc(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*) (nsXMLHttpRequest.cpp:1751)
 5:52.69 ==3395==    by 0xD713FAF: nsPipeInputStream::ReadSegments(nsresult (*)(nsIInputStream*, void*, char const*, unsigned int, unsigned int, unsigned int*), void*, unsigned int, unsigned int*) (nsPipe3.cpp:1304)
 5:52.69 ==3395==    by 0xDF2649A: nsXMLHttpRequest::OnDataAvailable(nsIRequest*, nsISupports*, nsIInputStream*, unsigned long, unsigned int) (nsXMLHttpRequest.cpp:1815)
 5:52.69 ==3395==    by 0xD869894: mozilla::net::nsHttpChannel::OnDataAvailable(nsIRequest*, nsISupports*, nsIInputStream*, unsigned long, unsigned int) (nsHttpChannel.cpp:6305)
 5:52.69 ==3395==    by 0xD7801BD: nsInputStreamPump::OnStateTransfer() (nsInputStreamPump.cpp:603)
 5:52.69 ==3395==    by 0xD780324: nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) (nsInputStreamPump.cpp:430)
 5:52.69 ==3395==    by 0xD70E817: nsInputStreamReadyEvent::Run() (nsStreamUtils.cpp:94)
 5:52.69 ==3395==    by 0xD722A62: nsThread::ProcessNextEvent(bool, bool*) (nsThread.cpp:994)
 5:52.69 ==3395==    by 0xD73AE72: NS_ProcessNextEvent(nsIThread*, bool) (nsThreadUtils.cpp:297)
 5:52.69 ==3395==    by 0xD921C85: mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) (MessagePump.cpp:97)
 5:52.69 ==3395==    by 0xD90DBB8: RunHandler (message_loop.cc:227)
 5:52.69 ==3395==    by 0xD90DBB8: MessageLoop::Run() (message_loop.cc:201)
 5:52.69 ==3395==    by 0xE8DD1CE: nsBaseAppShell::Run() (nsBaseAppShell.cpp:156)
 5:52.69 ==3395==    by 0xEE86421: nsAppStartup::Run() (nsAppStartup.cpp:281)
 5:52.69 ==3395==    by 0xEEBA0E5: XREMain::XRE_mainRun() (nsAppRunner.cpp:4342)
 5:52.69 ==3395==    by 0xEEBA38B: XREMain::XRE_main(int, char**, nsXREAppData const*) (nsAppRunner.cpp:4439)
 5:52.69 ==3395==    by 0xEEBA5E8: XRE_main (nsAppRunner.cpp:4545)
 5:52.69 ==3395==    by 0x404F31: do_main(int, char**, char**, nsIFile*) (nsBrowserApp.cpp:220)
 5:52.69 ==3395==    by 0x4046A1: main (nsBrowserApp.cpp:360)
 5:52.69 ==3395==
 5:52.69 {
 5:52.69    <insert_a_suppression_name_here>
 5:52.69    Memcheck:Addr4
 5:52.69    fun:_ZL9dosprintfP15SprintfStateStrPKDsP13__va_list_tag
 5:52.69    fun:_ZN15nsTextFormatter9vsmprintfEPKDsP13__va_list_tag
 5:52.69    fun:_ZN15nsTextFormatter8smprintfEPKDsz
 5:52.69    fun:CreateErrorText
 5:52.69    fun:_ZN13nsExpatDriver11HandleErrorEv
 5:52.69    fun:_ZN13nsExpatDriver12ConsumeTokenER9nsScannerRb
 5:52.69    fun:_ZN8nsParser8TokenizeEb
 5:52.69    fun:_ZN8nsParser11ResumeParseEbbb
 5:52.69    fun:_ZN8nsParser15OnDataAvailableEP10nsIRequestP11nsISupportsP14nsIInputStreammj
 5:52.69    fun:_ZN16nsXMLHttpRequest16StreamReaderFuncEP14nsIInputStreamPvPKcjjPj
 5:52.69    fun:_ZN17nsPipeInputStream12ReadSegmentsEPF8nsresultP14nsIInputStreamPvPKcjjPjES3_jS6_
 5:52.69    fun:_ZN16nsXMLHttpRequest15OnDataAvailableEP10nsIRequestP11nsISupportsP14nsIInputStreammj
 5:52.69    fun:_ZN7mozilla3net13nsHttpChannel15OnDataAvailableEP10nsIRequestP11nsISupportsP14nsIInputStreammj
 5:52.69    fun:_ZN17nsInputStreamPump15OnStateTransferEv
 5:52.69    fun:_ZN17nsInputStreamPump18OnInputStreamReadyEP19nsIAsyncInputStream
 5:52.69    fun:_ZN23nsInputStreamReadyEvent3RunEv
 5:52.69    fun:_ZN8nsThread16ProcessNextEventEbPb
 5:52.69    fun:_Z19NS_ProcessNextEventP9nsIThreadb
 5:52.69    fun:_ZN7mozilla3ipc11MessagePump3RunEPN4base11MessagePump8DelegateE
 5:52.69    fun:RunHandler
 5:52.69    fun:_ZN11MessageLoop3RunEv
 5:52.69    fun:_ZN14nsBaseAppShell3RunEv
 5:52.69    fun:_ZN12nsAppStartup3RunEv
 5:52.69    fun:_ZN7XREMain11XRE_mainRunEv
 5:52.69    fun:_ZN7XREMain8XRE_mainEiPPcPK12nsXREAppData
 5:52.69    fun:XRE_main
 5:52.69 }
Investigating.  My first reaction is that we need to add --partial-loads-ok=yes
to the bundle of flags given to Valgrind.  Or upgrade to 3.11.0, which enables that
by default.

As background, gcc sometimes generates vectorised code that over-reads the end of
a heap allocated block, but using a naturally-aligned, power-of-2 sized load; and
then masks the loaded data in such a way that the bytes read from beyond the end
of the block are unused.  Such a load cannot take a page fault, and the masking
makes it safe.  V understands this but until 3.11 this was not allowed without
the abovementioned flag.

I will check though, that there isn't something more serious going on.
Still working on narrowing down the regression range, but I've got it down to https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=31481f9ebdda&tochange=d1d47ba19ce9 so far.
Component: HTML: Parser → XML
Regression range: https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=b8e6b7d7ff08&tochange=2d59367c985a

Looks like this was caused by switching to gcc 4.8.5. Good call on the GCC angle! Interesting that this only affects !NIGHTLY_BUILD. I wonder what's so special about them.
IIRC aurora doesn't have --enable-profiling, which means no frame pointers, which means possibly different optimizations.

Now, to assess what's going on, it would be useful to have a good trace, and valgrind's is weird: it does say the error happens in dosprintf called from nsTextFormatter::vsmprintf called from nsTextFormatter::smprintf called from CreateErrorText, but the trace only starts at CreateErrorText... and nsTextFormatter is certainly not inlined in CreateErrorText, they both are in different .cpp files and we don't do LTO.
Flags: needinfo?(jseward)
One thing that is clear, though, from the fact the overflow happens on a buffer allocated at nsExpatDriver.cpp:828, is that the buffer in question is the format string.
BTW, here's the funny part with how we run valgrind builds: they build their own Firefox. Which means two things:
- you can't download the binary they use, which may or may not match exactly the opt build.
- we're not testing PGO builds.
(In reply to Mike Hommey [:glandium] from comment #6)
> BTW, here's the funny part with how we run valgrind builds: they build their
> own Firefox. Which means two things:
> - you can't download the binary they use, which may or may not match exactly
> the opt build.

And it looks like they *don't* match the opt build on the same try push.
(In reply to Mike Hommey [:glandium] from comment #4)
> 
> Now, to assess what's going on, it would be useful to have a good trace, and
> valgrind's is weird: it does say the error happens in dosprintf called from
> nsTextFormatter::vsmprintf called from nsTextFormatter::smprintf called from
> CreateErrorText, but the trace only starts at CreateErrorText... and
> nsTextFormatter is certainly not inlined in CreateErrorText, they both are
> in different .cpp files and we don't do LTO.

I talked with Mike on IRC about this: it looks like build/valgrind/output_handler.py might be swallowing some of the lines of output from Valgrind.
Flags: needinfo?(jseward)
Depends on: 1262331
With bug 1262331 fixed, the bottom of the stack frame is:

  5:35.66 TEST-UNEXPECTED-FAIL | valgrind-test | Invalid read of size 4 at dosprintf / nsTextFormatter::vsmprintf / nsTextFormatter::smprintf / CreateErrorText
  5:35.66 ==28296== Invalid read of size 4
  5:35.66 ==28296==    at 0xD737BA3: dosprintf(SprintfStateStr*, char16_t const*, __va_list_tag*) (nsTextFormatter.cpp:857)
  5:35.66 ==28296==    by 0xD7395E6: nsTextFormatter::vsmprintf(char16_t const*, __va_list_tag*) (nsTextFormatter.cpp:1282)
  5:35.66 ==28296==    by 0xD73968F: nsTextFormatter::smprintf(char16_t const*, ...) (nsTextFormatter.cpp:1240)
  5:35.66 ==28296==    by 0xDC3D6FE: CreateErrorText (nsExpatDriver.cpp:834)

Which places the read error on this line of code:
  while ((c = *aFmt++) != 0) {

where c is a char16_t and aFmt a const char16_t*.

Thanks to Nick Thomas, I got the binaries from a valgrind build. The machine code corresponding to that line of code is:
  ae4ba3:       8b 45 00                mov    0x0(%rbp),%eax
  ae4ba6:       4c 8d 65 02             lea    0x2(%rbp),%r12
  ae4baa:       66 85 c0                test   %ax,%ax

And the read error is on the first line. It does read 32 bits, but only tests 16 bits out of it. In practice, that means nothing bad happens, but that doesn't mean GCC is not wrong.

Now, the really interesting part is that, this doesn't match what the corresponding linux64 build on Ryan's try push has for the same C++:
 252ef74:       4c 8d 66 02             lea    0x2(%rsi),%r12
 252ef78:       41 0f b7 6c 24 fe       movzwl -0x2(%r12),%ebp
 252ef7e:       66 85 ed                test   %bp,%bp

Here it does read 16-bits with zero extension.

Also, the addresses are radically different, which is surprising. This points at the builds being too different for their own good... and now that I look, that's because the linux64 build on the same try push was a PGO build...

So the PGO build looks good, while the non-PGO is busted. The question would be how busted. Without looking deeper at all the possible paths leading to this machine code, what can go bad is if aFmt points to the 2 end-of-string bytes (0x0000) and the allocation for aFmt ends at the end of a page, it's going to read 2 bytes from the following page, which may not be mapped. So that would seem like a legitimate GCC bug.
So, playing with gcc 4.8 locally, here is what I can see in a .s output for this snippet of C++:

With -fomit-frame-pointer (which we use on aurora)
        .loc 2 857 0 discriminator 1
        movl    0(%rbp), %eax
        leaq    2(%rbp), %r12
        testw   %ax, %ax

With -fno-omit-frame-pointer (which we use on nightly because of --enable-profiling):
        .loc 2 857 0 discriminator 1
        movw    (%r12), %ax
        leaq    2(%r12), %r13
        testw   %ax, %ax
I managed to reproduce the problem with Mike's advice, using gcc-4.9.2,
-Os and -fomit-frame-pointer.  Running ./mach valgrind-test then produces
the error.

It also produced a second example, like this:

Invalid read of size 4
  at 0x8111D4E: setElementTypePrefix.isra.3 (xmlparse.c:5451)
  by 0x8112B70: storeAtts (xmlparse.c:2721)
  by 0x8115867: doContent (xmlparse.c:2412)
  by 0x8116001: contentProcessor (xmlparse.c:2072)

Address 0x192079ce is 2,046 bytes inside a block of size 2,048 alloc'd
  at 0x4A06C4F: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  by 0x8111003: poolGrow (xmlparse.c:6307)
  by 0x81110E0: poolAppend (xmlparse.c:6206)
  by 0x811111A: poolStoreString (xmlparse.c:6253)
  by 0x8111E83: getAttributeId (xmlparse.c:5485)

This is the same as the previous variant, in that it is a 32-bit load at
a 16-bit aligned address, which only uses the loaded 16 bits:

 25bcd4e:       8b 45 00                mov    0x0(%rbp),%eax
 25bcd51:       66 85 c0                test   %ax,%ax
 25bcd54:       0f 84 ce 00 00 00       je     25bce28 <setElementTypePrefix.isra.3+0xf2>
And here's another one.  This is from a PGO build of m-c using gcc-4.9.2,
at -O2.

Invalid read of size 4
   at 0x667AA14: char16_t* ConvertUnknownBreaks<char16_t>(char16_t const*, int&, char const*) (/home/sewardj/MOZ/MC-REL/xpcom/io/nsLinebreakConverter.cpp:249)
   by 0x667E47F: nsLinebreakConverter::ConvertUnicharLineBreaks(char16_t const*, nsLinebreakConverter::ELinebreakType, nsLinebreakConverter::ELinebreakType, int, int*) (/home/sewardj/MOZ/MC-REL/xpcom/io/nsLinebreakConverter.cpp:381)
   by 0x76A5878: nsFSURLEncoded::URLEncode(nsAString_internal const&, nsCString&) (/home/sewardj/MOZ/MC-REL/dom/html/nsFormSubmission.cpp:368)
   by 0x76A5A2D: nsFSURLEncoded::AddNameValuePair(nsAString_internal const&, nsAString_internal const&) (/home/sewardj/MOZ/MC-REL/dom/html/nsFormSubmission.cpp:141)

 Address 0x1cd4cf5a is 730 bytes inside a block of size 732 alloc'd
   at 0x4A06B2D: malloc (/home/sewardj/VgTRUNK/merge/coregrind/m_replacemalloc/vg_replace_malloc.c:299)
   by 0x86DF544: nsStringBuffer::Alloc(unsigned long) (/home/sewardj/MOZ/MC-REL/xpcom/string/nsSubstring.cpp:217)
   by 0x74D82C3: SetLength (/home/sewardj/MOZ/MC-REL/ff-pgo/dist/include/mozilla/dom/BindingUtils.h:1926)
   by 0x74D82C3: AssignJSString<mozilla::dom::binding_detail::FakeString> (/home/sewardj/MOZ/MC-REL/dom/base/nsJSUtils.h:158)

  a9da14:       8b 55 00                mov    0x0(%rbp),%edx
  a9da17:       66 83 fa 0d             cmp    $0xd,%dx
  a9da1b:       75 40                   jne    a9da5d <_Z20ConvertUnknownBreaksIDsEPT_PKS0_RiPKc+0xcf>
Comment on attachment 8738809 [details]
MozReview Request: Bug 1261264 - Apply GCC PR64905 to fix miscompilation with -fomit-frame-pointer. r?froydnj

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/44703/diff/1-2/
Attachment #8738809 - Flags: review?(nfroyd)
Comment on attachment 8738809 [details]
MozReview Request: Bug 1261264 - Apply GCC PR64905 to fix miscompilation with -fomit-frame-pointer. r?froydnj

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/44703/diff/2-3/
Attachment #8738809 - Attachment description: MozReview Request: Bug 1261264 - Apply GCC PR64905 to fix miscompilation with -fomit-frame-pointer → MozReview Request: Bug 1261264 - Apply GCC PR64905 to fix miscompilation with -fomit-frame-pointer. r?froydnj
Attachment #8738809 - Flags: review?(nfroyd) → review+
Comment on attachment 8738809 [details]
MozReview Request: Bug 1261264 - Apply GCC PR64905 to fix miscompilation with -fomit-frame-pointer. r?froydnj

https://reviewboard.mozilla.org/r/44703/#review41535

WFM.

::: mobile/android/config/tooltool-manifests/android/releng.manifest:45
(Diff revision 3)
>  "filename": "jsshell.tar.xz"
>  },
>  {
> -"version": "gcc 4.8.5",
> -"size": 81065660,
> -"visibility": "public",
> +"version": "gcc 4.8.5 + PR64905",
> +"size": 80160264,
> +"digest": "c1a9dc9da289b8528874d16300b9d13a997cec99195bb0bc46ff665216d8535d6d6cb5af6b4b1f2749af6815dab12e703fdb3849014e5c23a70eff351a0baf4e",

We lost the visibility key here, any particular reason?  Conformity with all the other manifests?
(In reply to Nathan Froyd [:froydnj] from comment #17)
> We lost the visibility key here, any particular reason?  Conformity with all
> the other manifests?

visibility is something for tooltool upload. it's quite pointless for tooltool fetch.
https://hg.mozilla.org/mozilla-central/rev/dfff0b460314
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla48
You need to log in before you can comment on or make changes to this bug.