Closed Bug 1292923 Opened 8 years ago Closed 8 years ago

Crashes in igd10umd32.dll

Categories

(Core :: Audio/Video: Playback, defect, P1)

51 Branch
x86
Windows 8
defect

Tracking

()

RESOLVED FIXED
mozilla52
Tracking Status
firefox51 + fixed
firefox52 --- fixed

People

(Reporter: jchen, Assigned: mattwoodrow)

References

Details

(Keywords: crash, regression)

Crash Data

Attachments

(2 files)

This bug was filed from the Socorro interface and is 
report bp-8e8e42fa-f97d-490e-a1c9-3bc9e2160806.
=============================================================

There are several crashes in igd10umd32.dll (Intel graphics driver) that started with the 20160804030441 Nightly.

"igd10umd32.dll | NDXGI::CDevice::SetSharedResourceCreationArgs" is the most frequent (#13 with 8 crashes for 08-04 nightly), but there are 51 crashes combined for igd10umd32.dll, which would rank as the #2 crash for the 08-04 nightly.
I'm guessing this might have to do with our device reworking.
Component: Graphics → Audio/Video: Playback
Any chance you could try reproduce this in the Toronto office Jeff?

It looks similar to bug 1221348 (CreateTexture2D crashes with keyed mutex + upload data), but the driver versions are significantly newer than mentioned in that bug.

The old range was 8.15.10.[18xx-1994], but we're seeing crashes here with drivers as new as 9.17.10.2963.

I also just realized that we lost the DoesAlphaTextureSharingWork check that was done for the ImageBridge device, we probably need to add that back and disable this upload path if it fails.
Flags: needinfo?(jmuizelaar)
(In reply to Matt Woodrow (:mattwoodrow) from comment #2)
> The old range was 8.15.10.[18xx-1994], but we're seeing crashes here with
> drivers as new as 9.17.10.2963.

These are still pretty old, we could easily just disable this optimized upload path for old intel drivers.
Joe, is crashing in CreateTexture2D (when combined with pInitialData and D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX) a known issue?

If it has been fixed in a more recent driver release we can make sure we avoid calling this on older drivers.

Thanks!
Flags: needinfo?(joseph.k.olivas)
(In reply to Matt Woodrow (:mattwoodrow) from comment #4)
> Joe, is crashing in CreateTexture2D (when combined with pInitialData and
> D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX) a known issue?
> 
> If it has been fixed in a more recent driver release we can make sure we
> avoid calling this on older drivers.
> 
> Thanks!

I will file a bug internally to see if I can get an answer. Is there any known way to reproduce, or workload that may hit this path frequently enough where it might happen?
Flags: needinfo?(joseph.k.olivas)
According to the Intel Auto Update scanner I have the latest version of the driver for my laptop
And I have version: 9.17.10.4229

Using a Intel HD Graphics 3000 for my Lenovo E520 laptop

My nightly now crashes all the time making it almost useless :(
(In reply to Joe Olivas from comment #5)
> (In reply to Matt Woodrow (:mattwoodrow) from comment #4)
> > Joe, is crashing in CreateTexture2D (when combined with pInitialData and
> > D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX) a known issue?
> > 
> > If it has been fixed in a more recent driver release we can make sure we
> > avoid calling this on older drivers.
> > 
> > Thanks!
> 
> I will file a bug internally to see if I can get an answer. Is there any
> known way to reproduce, or workload that may hit this path frequently enough
> where it might happen?

We haven't reproduced this yet, only working off crash reports. It does appear that simply creating a texture with those options crashes on affected Intel drivers, which may include fairly recent ones.
Crash Signature: CResource<T>::CopySubresourceRegion<T>] → CResource<T>::CopySubresourceRegion<T>] [@ igd10umd32.dll | ResourceCopyRegionDDIHelper<T>]
Crash Signature: CResource<T>::CopySubresourceRegion<T>] [@ igd10umd32.dll | ResourceCopyRegionDDIHelper<T>] → CResource<T>::CopySubresourceRegion<T>] [@ igd10umd32.dll | ResourceCopyRegionDDIHelper<T>] [@ igd10umd64.dll | RtlAllocateMemoryBlockLookaside | RtlAllocateHeap | RtlAllocateMemoryBlockLookaside | RtlAllocateHeap | igd10umd64.dll | std::list<T>::_Buyn…
We can sometimes fail to acquire keyed mutex in CompositorD3D11::BeginFrame.  When this happens, it could be because of the timeout (WAIT_TIMEOUT) or because things are "not in consistent state" - WAIT_ABANDONED.

This crash: https://crash-stats.mozilla.com/report/index/df5bad8b-ead5-425d-b9eb-d39042160825 has a lot of abandoned errors.  It's suggested this can happen when the locking thread terminates without releasing the mutex.
Assignee: nobody → cpearce
Henrik, can you get a full minidump for the crash you're experiencing?
Here are some steps to do this:

2. Create a new profile for Firefox (this is to avoid having any of your private data in the minidump)
1. Make sure Firefox is closed.
3. Launch cmd.exe
4. type 'set MOZ_CRASHREPORTER_FULLDUMP=1'
5. Launch firefox.exe -p [your-new-profile-name]
6. When the crash happens and the crash dialog is still up look in %APPDATA%\Mozilla\Firefox\Crash Reports\pending and copy out the most recent file from there.
7. Mail the file to the email address associated with my account.

Thanks.
Flags: needinfo?(jmuizelaar) → needinfo?(bugzilla)
I can only find some files in
AppData\Roaming\Mozilla\Firefox\Profiles\fky0bueq.test\minidumps\
a .dmp file and I can zip them (225 Mb)
is that ok or you need someelse else?
Flags: needinfo?(bugzilla)
(In reply to Henrik Gemal from comment #11)
> I can only find some files in
> AppData\Roaming\Mozilla\Firefox\Profiles\fky0bueq.test\minidumps\
> a .dmp file and I can zip them (225 Mb)
> is that ok or you need someelse else?

That looks to be exactly it. Please send it to me. I've emailed at your bugzilla email a dropbox link that you should be able to upload it to.
I get these crashes pretty reliably (e.g. bp-5e8d6c4a-fb14-4fb0-a6f1-7f70b2160904) while playing the video in [1] (open that page, then open the video itself in multiple tabs, and then switch back and forth). MOZ_CRASHREPORTER_FULLDUMP doesn't seem to work though; there's no crash report at all when I set MOZ_CRASHREPORTER_FULLDUMP=1.

[1] https://www.reddit.com/r/funny/comments/50zfxi/two_drunk_guys_helping_each_other_out/
Flags: needinfo?(milan)
Crash Signature: std::list<T>::_Buynode | igd10umd64.dll | stdext::_Hash<T>::insert] → std::list<T>::_Buynode | igd10umd64.dll | stdext::_Hash<T>::insert][@ igd10umd64.dll | CContext::ID3D11DeviceContext1_SetSamplers_<T>]
¡Hola!

I just crashed like this:

Report ID 	Date Submitted
bp-e3becc9e-6c69-4ff7-bc9a-5e1632160921
	21/09/2016	04:32 p.m.

My steps:
- Load https://www.gnome.org/news/2016/09/gnome-3-22-released-the-future-is-now/
- Play the video in the page

Result:
Windows notifies of the Intel driver crashing and recovering successfully and the content process crashes.

¡Gracias!
Alex
I tried to bisect the crash earlier. It's fairly random so it's a bit hard to pinpoint, but I'm pretty sure this is a regression from this push [1], so bug 1284672 or bug 1289640.

[1] https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=b407b9803516
Flags: needinfo?(matt.woodrow)
Keywords: regression
Reassigning to Mattwoodrow, since he has more clues than I... Matt, please let me know if you can't take this...
Assignee: cpearce → matt.woodrow
(In reply to alex_mayorga from comment #14)
> ¡Hola!
> > My steps:
> - Load
> https://www.gnome.org/news/2016/09/gnome-3-22-released-the-future-is-now/
> - Play the video in the page
> 
> Result:
> Windows notifies of the Intel driver crashing and recovering successfully
> and the content process crashes.
I've just got the same driver crash after opening this video in a new tab: https://www.youtube.com/watch?v=VnDtmcCuKUU
Content process crashed after I opened this video second time. As far as I understand, here is the crash data:
https://crash-stats.mozilla.com/report/index/bb20f9b2-535b-4544-9aa9-e92ec2160925

I am using Intel Graphics 2000 and Intel 9.17.10.4229 driver (the latest available).
I'll follow up with Intel and see if we can find a solution to this.
Flags: needinfo?(matt.woodrow)
(In reply to ajfhajf from comment #17)
> I've just got the same driver crash after opening this video in a new tab:
> https://www.youtube.com/watch?v=VnDtmcCuKUU
> Content process crashed after I opened this video second time. As far as I
> understand, here is the crash data:
> https://crash-stats.mozilla.com/report/index/bb20f9b2-535b-4544-9aa9-
> e92ec2160925
> 
> I am using Intel Graphics 2000 and Intel 9.17.10.4229 driver (the latest
> available).

This doesn't look like the same crash unfortunately. Can you reproduce it?
I'm going to look into working around this myself as well.

Anthony, can you help me figure out if these crashes (specifically the ones coming from CDevice::CreateTexture2D) are limited to a subset of Intel devices or drivers?

We can block calling this code, but it would be nice to restrict it to the minimal set we need.
Flags: needinfo?(anthony.s.hughes)
Lets start with disabling this for all Intel cards and we can try make it more restrictive later if we find out more.
Attachment #8794640 - Flags: review?(dvander)
Track 51+ as the volume of crash is high and high priority.
This could help bug 1295075.
Flags: needinfo?(milan)
Comment on attachment 8794640 [details] [diff] [review]
block-intel-createtexture2d-crash

Review of attachment 8794640 [details] [diff] [review]:
-----------------------------------------------------------------

::: gfx/thebes/DeviceManagerDx.cpp
@@ +677,5 @@
> +  MutexAutoLock lock(mDeviceLock);
> +  if (!mDeviceStatus) {
> +    return false;
> +  }
> +  // Disable this on all Intel devices.

nit, can you note this bug # as well?
Attachment #8794640 - Flags: review?(dvander) → review+
Unfortunately I cannot get correlations as fine grained as just crashes "coming from CDevice::CreateTexture2D" as that would involve going through each report by hand (there are literally thousands of reports). Below are the correlations for the last week of data for all of these crashes.  This seems to be heavily but not exclusively tied to Ivybridge and Sandybridge chipsets (~86%) on the 9.17.10.4229 driver (~24%) although there is a very long tail of drivers affected.

Chipset Family
==============
26.46%	Ivybridge-GT2
22.55%	Sandybridge-GT1
18.97%	Sandybridge-GT2
18.09%	Ivybridge-GT1
 8.21%	GMA4500HD
 5.58%	GMA4500
 0.16%	GMA3500

GPU Chipsets
============
25.90%	0x0166	[Intel HD 4000]
15.38%	0x0152	[Intel HD 2500]
13.86%	0x0102	[Intel HD 2000]
12.75%	0x0116	[Intel HD 3000]
 8.69%	0x0106	[Intel HD 2000]
 8.21%	0x2a42	[Intel GMA X4500]
 6.14%	0x0126	[Intel HD 3000]
 5.18%	0x2e32	[Intel G41 Express]
 2.71%	0x0156	[Intel HD Ivybridge]
 0.56%	0x0162	[Intel HD 4000]
 0.24%	0x2e12	[Intel GMA X4500]
 0.16%	0x2e22	[Intel GMA X4500]
 0.16%	0x2a02	[Intel GMA X3100]
 0.08%	0x0122	[Intel HD 3000]

Driver Versions
===============
9.17.10.4229 	24.28%
8.15.10.2712 	 6.20%
9.17.10.2843 	 5.45%
9.17.10.2932 	 5.45%
9.17.10.2828 	 5.35%
9.17.10.2867 	 4.96%
8.15.10.2618 	 3.17%
9.17.10.3347 	 3.17%
8.15.10.2696 	 2.85%
8.15.10.2752 	 2.82%
9.17.10.2849 	 2.75%
9.17.10.3517 	 2.75%
8.15.10.1840 	 2.32%
8.15.10.2653 	 2.32%
8.15.10.2639 	 1.85%
9.17.10.2817 	 1.85%
9.17.10.2963 	 1.46%
8.15.10.1851 	 1.35%
9.17.10.3372 	 1.35%
8.15.10.1883 	 1.32%
8.15.10.2418 	 1.21%
9.17.10.3062 	 1.21%
8.15.10.2669 	 1.18%
9.17.10.4101 	 1.14%
9.17.10.3040 	 1.03%
8.15.10.2626 	 0.71%
9.17.10.3223 	 0.71%
8.15.10.2778 	 0.61%
8.15.10.2538 	 0.57%
9.17.10.2884 	 0.57%
8.15.10.1855 	 0.53%
8.15.10.2725 	 0.53%
8.15.10.1892 	 0.50%
8.15.10.2656 	 0.50%
8.15.10.2761 	 0.50%
8.15.10.2342 	 0.43%
8.15.10.1872 	 0.32%
8.15.10.2353 	 0.32%
9.17.10.3114 	 0.29%
10.0.10240.16384 0.25%
10.17.10.4229 	 0.25%
9.17.10.2857 	 0.25%
9.18.13.2762 	 0.25%
8.15.10.1808 	 0.21%
8.15.10.1994 	 0.21%
9.17.10.2875 	 0.21%
10.18.10.4358 	 0.18%
10.0.10586.0 	 0.14%
8.15.10.2795 	 0.14%
9.17.10.2792 	 0.14%
9.17.10.3190 	 0.14%
10.0.14393.0 	 0.11%
8.640.4.1000 	 0.11%
8.652.0.0 	 0.11%
8.652.1.0 	 0.11%
8.653.0.0 	 0.11%
9.18.13.1269 	 0.11%
8.15.10.2476 	 0.07%
8.15.10.2598 	 0.07%
8.15.10.2622 	 0.07%
8.15.10.2697 	 0.07%
8.15.10.2900 	 0.07%
9.17.10.4459 	 0.07%
10.18.13.6881 	 0.04%
8.15.10.2086 	 0.04%
8.15.10.2279 	 0.04%
8.15.10.2345 	 0.04%
8.15.10.2361 	 0.04%
8.15.10.2372 	 0.04%
8.15.10.2405 	 0.04%
8.15.10.2430 	 0.04%
8.15.10.2455 	 0.04%
8.15.10.2555 	 0.04%
8.15.10.2869 	 0.04%
8.15.10.2993 	 0.04%
8.16.11.8745 	 0.04%
8.672.4.0 	 0.04%
8.741.1.5000 	 0.04%
9.17.10.2897 	 0.04%
9.18.13.1100 	 0.04%
9.18.13.546 	 0.04%

I would note further that these crashes have been around for at least 6 months (as far back as the data goes) but something recently is causing a multi-factor spike.
Flags: needinfo?(anthony.s.hughes)
.4229 is the latest driver, as I understand it, which probably means that we have a problem with all versions.
Actually, I've seen driver versions 9.17.10.4459
(In reply to Matt Woodrow (:mattwoodrow) from comment #19)
> This doesn't look like the same crash unfortunately. Can you reproduce it?
I can sometimes reproduce this bug on profile with addons installed by opening this video:
https://www.youtube.com/watch?v=B2I8PlFmERU
As far as I understand, here is the crash report:
https://crash-stats.mozilla.com/report/index/b3d94ec3-5292-464c-b046-965592160928
Hi Matt, is this one ready to land on Nightly52 and Aurora51? It was mentioned as a top crasher in today's channel meeting.
Flags: needinfo?(matt.woodrow)
Pushed by mwoodrow@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/e1300d4c0a52
Don't upload to textures during creation on Intel cards as it frequently crashes. r=dvander
Comment on attachment 8794640 [details] [diff] [review]
block-intel-createtexture2d-crash

Review of attachment 8794640 [details] [diff] [review]:
-----------------------------------------------------------------

::: gfx/layers/d3d11/TextureD3D11.cpp
@@ +384,5 @@
>    }
>  
> +  if (aSurface && newDesc.MiscFlags == D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX &&
> +      DeviceManagerDx::Get()->CanInitializeKeyedMutexTextures()) {
> +    return nullptr;

I don't know this code, but this seems opposite from the comments on the patch, where we're proposing to do something "different" for Intel.  Here we're doing something "different" for everything except Intel.
¡Hola Matt!

Is https://crash-stats.mozilla.com/report/index/3c7edf13-3b49-4959-9632-49d222161004 also a variant of this bug or a separate one?

¡Gracias!
Alex
Are these all from IMFYCbCrImage::GetTextureClient?
Flags: needinfo?(matt.woodrow)
Attachment #8797857 - Flags: review?(dvander)
(In reply to Milan Sreckovic [:milan] from comment #33)
> Comment on attachment 8794640 [details] [diff] [review]
> block-intel-createtexture2d-crash
> 
> Review of attachment 8794640 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: gfx/layers/d3d11/TextureD3D11.cpp
> @@ +384,5 @@
> >    }
> >  
> > +  if (aSurface && newDesc.MiscFlags == D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX &&
> > +      DeviceManagerDx::Get()->CanInitializeKeyedMutexTextures()) {
> > +    return nullptr;
> 
> I don't know this code, but this seems opposite from the comments on the
> patch, where we're proposing to do something "different" for Intel.  Here
> we're doing something "different" for everything except Intel.

Heh, good catch, the condition is indeed backwards.

The fun bit is that if I had this correct then treeherder wouldn't have caught the other bug that got it backed out.
(In reply to Milan Sreckovic [:milan] from comment #35)
> Are these all from IMFYCbCrImage::GetTextureClient?

The crash that I'm trying to fix comes from IMFYCbCrImage::GetTextureClient and DXGITextureData::Create.

I believe those crashes are the cause of the spike that brought this to prominence.

This bug (as filed) sort of appears to be a catch all for every crash in Intel drivers, so it won't go away entirely. We might need to clone it once the patches stick so we can track what crashes remain.
Adding [@ igd10umd32.dll | CContext::TID3D11DeviceContext_Draw_<T>] to this bug with the following correlations:

Version
=======
> 1 	51.0a2 	257 	48.95 %
> 2 	50.0b3 	122 	23.24 %
> 3 	50.0b2 	75 	14.29 %
> 4 	52.0a1 	27 	5.14 %
> 5 	49.0.1 	23 	4.38 %
> 6 	50.0b4 	16 	3.05 %
> 7 	51.0a1 	2 	0.38 %
> 8 	45.0b6 	1 	0.19 %
> 9 	48.0.2 	1 	0.19 %
> 10 	50.0b1 	1 	0.19 %

Adapter
=======
> 1 	Intel Sandybridge-GT2 [0x0116, 0x0126, 0x0112]  276  52.00 %
> 2 	Intel Sandybridge-GT1 [0x0102, 0x0106]          220  41.91 %
> 3     Intel Ironlake        [0x0046, 0x0042]           25   4.76 %

Driver
======
> 1 	9.17.10.4229 	491 	94.06 %
> 2 	8.15.10.2900 	25 	4.79 %
> 3 	9.17.10.4459 	3 	0.57 %
> 4 	10.17.10.4229 	1 	0.19 %
> 5 	9.17.10.2843 	1 	0.19 %
> 6 	9.17.10.2932 	1 	0.19 %

More Reports: https://crash-stats.mozilla.com/signature/?signature=igd10umd32.dll%20|%20CContext%3A%3ATID3D11DeviceContext_Draw_%3CT%3E
Crash Signature: std::list<T>::_Buynode | igd10umd64.dll | stdext::_Hash<T>::insert][@ igd10umd64.dll | CContext::ID3D11DeviceContext1_SetSamplers_<T>] → std::list<T>::_Buynode | igd10umd64.dll | stdext::_Hash<T>::insert][@ igd10umd64.dll | CContext::ID3D11DeviceContext1_SetSamplers_<T>] [@ igd10umd32.dll | CContext::TID3D11DeviceContext_Draw_<T>]
Attachment #8797857 - Flags: review?(dvander) → review+
Blocks: 1305490
Pushed by mwoodrow@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d35d695f921d
Don't upload to textures during creation on Intel cards as it frequently crashes. r=dvander
Pushed by mwoodrow@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3855dd4690ab
Don't upload to textures during creation on Intel cards as it frequently crashes. r=dvander
Pushed by cbook@mozilla.com:
https://hg.mozilla.org/mozilla-central/rev/38442ad9a422
Don't upload to textures during creation on Intel cards as it frequently crashes. r=dvander
https://hg.mozilla.org/mozilla-central/rev/da986c9f1f72
Backed out changeset d35d695f921d for troubles with windows refrests
had to back this out again from m-c seems this caused together with a what it seems incomplete backout reftest failures like https://treeherder.mozilla.org/logviewer.html#?job_id=5175443&repo=mozilla-central and so got backout from m-c and the integration branches
Flags: needinfo?(matt.woodrow)
Backout by cbook@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c93ea6480724
Backed out changeset 3855dd4690ab for conflicts with m-c
I installed the latest inbound which has this patch as well as Bug 1292923.  I think one of these might have caused Hardware H264 Decoding to be turned off as noted in about:support.  It's like this fixed patch... Bug 1305213

Just wanted to let you know if you don't already.
Sorry, the other patch is Bug 1305326.
Just installed the latest inbound, which still has this and the other patch and I have H264 Decoding.  So I guess it was something else.
Pushed by mwoodrow@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/74b89fbf2130
Don't upload to textures during creation on Intel cards as it frequently crashes. r=dvander
Flags: needinfo?(matt.woodrow)
https://hg.mozilla.org/mozilla-central/rev/74b89fbf2130
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla52
Hi :mattwoodrow,
Since this also affects 51, do you think the patch is worth uplifting to 51?
Flags: needinfo?(matt.woodrow)
Is 50 affected?
Based on the correlations script https://mozilla.github.io/stab-crashes/correlations.html?product=Firefox&channel=beta&signature=igd10umd32.dll, the 32-bit version of this signature does show up on 50.0b4. Given that, if the fix looks good on 52/51, we should uplift to Beta50.
Comment on attachment 8797857 [details] [diff] [review]
block-intel-createtexture2d-crash v2

Approval Request Comment
[Feature/regressing bug #]: Bug 1289640
[User impact if declined]: Crashes, mainly during video.
[Describe test coverage new/current, TreeHerder]: Not tested much since I can't repro, but it avoids the optimized code paths on all affected devices.
[Risks and why]: Very low risk, falls back to normal path that is well tested.
[String/UUID change made/needed]: None
Flags: needinfo?(matt.woodrow)
Attachment #8797857 - Flags: approval-mozilla-aurora?
Comment on attachment 8797857 [details] [diff] [review]
block-intel-createtexture2d-crash v2

Fix a crash. Take it in 51 aurora.
Attachment #8797857 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Hey Matt, is this something that needs to be uplifted to Beta50? Please see comment 52.
Flags: needinfo?(matt.woodrow)
My patch fixed a change that was introduced by bug 1289640, which is only in 51.

Unfortunately this bug is very vague (all crashes in intel gpu drivers, anywhere) so it still shows up in 50 (and earlier).
Flags: needinfo?(matt.woodrow)
has problems to apply to aurora:
grafting 368649:74b89fbf2130 "Bug 1292923 - Don't upload to textures during creation on Intel cards as it frequently crashes. r=dvander"
merging gfx/layers/IMFYCbCrImage.cpp
merging gfx/layers/client/TextureClient.cpp
merging gfx/layers/client/TextureClient.h
merging gfx/layers/d3d11/TextureD3D11.cpp
merging gfx/thebes/DeviceManagerDx.cpp
merging gfx/thebes/DeviceManagerDx.h
warning: conflicts while merging gfx/layers/client/TextureClient.cpp! (edit, then use 'hg resolve --mark')

could you take a look, thanks!
Flags: needinfo?(matt.woodrow)
Flags: needinfo?(matt.woodrow)
Just to be certain: we're not uplifting to beta (50), right?
Flags: needinfo?(matt.woodrow)
(In reply to Andrew Overholt [:overholt] from comment #59)
> Just to be certain: we're not uplifting to beta (50), right?

That is correct, 50 is not affected by the change that caused the spike.
Flags: needinfo?(matt.woodrow)
Version: unspecified → 51 Branch
See Also: → 1346253
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: