1286307 - Crash in dalvik-main space 1 (deleted)@0xc7ffffe

Status: RESOLVED DUPLICATE of bug 1461724

(Core :: JavaScript Engine: JIT, defect, P1)

Description

[Marcia Knous [:marcia]]

This bug was filed from the Socorro interface and is 
report bp-a652d2c4-1678-4eaf-bfda-c62772160712.
=============================================================

Seen while reviewing crash stats. On Firefox 47 this is currently a top crash with over 8K crashes.

Looks as if the crash is also present on Firefox 48b6.

Comment 1

[Brad Lassey [:blassey] (use needinfo?)]

Jim, do you know anything about this?

Comment 2

[Marcia Knous [:marcia]]

Link to crashes on 47: http://bit.ly/29v34OO.

Comment 3

[Kevin Brosnan [Ex-Mozilla]]

The second frame is EnterBaseline seems like a variant of https://crash-stats.mozilla.com/signature/?signature=EnterBaseline link it to the meta bug 858032? We also fixed bug 1247312 in 48 which should cause the EnterBaseline numbers to drop.

Comment 4

[(inactive) Jim Chen [:jchen] [:darchons]]

Yeah seems like a JIT crash. "dalvik-main space 1" is a bogus frame in the stack that Breakpad picked up, and it's right below the top frame with a hex address, which makes sense for JIT code.

Because we recently started ignoring hex-address top frames in Socorro, "dalvik-main space 1" became the new top frame and crash signature.

Comment 5

[Brad Lassey [:blassey] (use needinfo?)]

Naveed, this is the top crash for Fennec right now. Anything you can do to make it actionable?

Comment 6

[Naveed Ihsanullah [:naveed]]

Jan, can you please take a look at this trending crasher?

Comment 7

[Marcia Knous [:marcia]]

Adding a similar stack signature - http://bit.ly/29IdP06 - which accounts for another 3K in crashes.

Comment 8

[Marcia Knous [:marcia]]

This signature has over 20403 crashes in 50 in the last 7 days - seems to have spiked a bit.

Comment 9

[Marco Castelluccio [:marco]]

"dalvik-main space 1 (deleted)@0xbbffffe" has ~20000 reports over the last week.

It seems highly correlated with a specific graphic card:
(99.49% in signature vs 26.82% overall) adapter_driver_version = OpenGL ES 3.1 v1.r7p0-03rel0.b596bd02e7d0169c10574b57180c8b57
(99.49% in signature vs 28.33% overall) adapter_device_id = Mali-T760
(99.49% in signature vs 45.43% overall) adapter_vendor_id = ARM
(56.76% in signature vs 16.46% overall) CPU Info = ARMv1 ARM part(0x4100d070) features: half,thumb,fastmult,vfpv2,edsp,neon,vfpv3,vfpv4,idiva,idivt
(41.55% in signature vs 13.94% overall) CPU Info = ARMv1 ARM part(0x4100d030) features: half,thumb,fastmult,vfpv2,edsp,neon,vfpv3,vfpv4,idiva,idivt

Comment 10

[Liz Henry (:lizzard) (relman/hg->git project)]

Eric, any luck with this JS crash?

As a more drastic measure, we might block SM-G92[XX] phones in the Play Store.

Comment 11

[Johnny Stenback (:jst)]

Naveed, this one looks like it needs more attention, and probably another owner.

Comment 12

[Naveed Ihsanullah [:naveed]]

Hannes please look into this bug. The volume is spiking. Thanks

Comment 13

[Kevin Brosnan [Ex-Mozilla]]

Note if you want to reproduce this it requires specific hardware. Below are the top 5 device model numbers. If you are ordering a phone for this bug please reference the model number as Samsung ships two different SOC boards depending on your location. North America gets Broadcom SOC. The rest of the world has Exynos SOC phones. Exynos is the SOC that is needed to reproduce this bug. The phone needs to be updateable to Android API 23 which is Android 6.0 (marshmallow).

Mfg.        Model          And. API    CPU ABI    #      
samsung  SM-G920F          23 (REL)  armeabi-v7a  8785  37.9%
samsung  SM-G925F          23 (REL)  armeabi-v7a  5097  22.0%
samsung  SAMSUNG-SM-G920A  23 (REL)  armeabi-v7a  1396   6.0%
samsung  SM-G920V          23 (REL)  armeabi-v7a  1296   5.6%
samsung  SM-G920I          23 (REL)  armeabi-v7a  1014   4.4%

Comment 14

[Johnny Stenback (:jst)]

Kevin, do we know of anyone having success reproducing this locally?

Comment 15

[Kevin Brosnan [Ex-Mozilla]]

I'm ordering one.

Comment 16

[Kevin Brosnan [Ex-Mozilla]]

I hit this signature. I played videos linked from http://www.bbc.com/news/video_and_audio/headlines crashed right at the start of playback. bp-7377478a-5343-42ba-873e-0f0492161221 There was about 3 hours of browsing before I hit this.

Comment 17

[Hannes Verschore [:h4writer]]

In all generality JIT crashes are very hard to investigate given only a stacktrace. There is no information to look into and the data in the core dumps often also don't contain much to go on. We have been internally discussing this a lot in how we could improve this information, but we have only found ways to classify it better, not to make it more actionable.

That is also the reason we rely heavily on fuzzers for finding such issues and we are actively coordinating with the fuzzer team to improve it. E.g. searching why fuzzers didn't find particular crashes and improving the instrumentation used for fuzzers. That has worked quite well, but we keep trying to improve the situation. Fuzzers are not the holy grail or the only source of finding/fixing bugs.

Reproducible crashes are actionable for us. That way we can interact and try to fit the missing pieces on what went wrong.

@kbrosnan: You have such a machine right? Could you try to run a debug build and try to hit this? Note: JS might be very slow in debug builds. Let us know if that is making it impossible to reproduce or use. We could try to decrease the amount of instrumentation if needed.

Comment 18

[Carsten Book [:Tomcat]]

Hi Kevin, i have also a samsung mobile (s6 edge latest patches) and was running into this crash here when i was trying to delete a typo in a url bar (like i forgot a space in the search) but couldn't reproduce it so far, so this crash might be really a tricky one with lots of possible steps to crash just reproducing seem to be the problem

Comment 19

[Hannes Verschore [:h4writer]]

(In reply to Hannes Verschore [:h4writer] from comment #17)
> @kbrosnan: You have such a machine right? Could you try to run a debug build
> and try to hit this? Note: JS might be very slow in debug builds. Let us
> know if that is making it impossible to reproduce or use. We could try to
> decrease the amount of instrumentation if needed.

Seems that people are able to hit it in normal browser. Though nobody was able to reproduce in a debug browser. Going to order a device to try that myself and have the hardware next to me while debugging it. Hope that makes this actionable.

Comment 20

[Kevin Brosnan [Ex-Mozilla]]

I have spent time trying to reproduce one of the S6 crashes using a local debug build with JimDB attached. So far I have not had any luck.

Whatever caused this was pushed in part of 50.1.0. In 50.0.2 there are ~350 crashes with this signature. In 50.1.0 there are 22,000. https://crash-stats.mozilla.com/signature/?product=FennecAndroid&signature=dalvik-main%20space%201%20%28deleted%29%400xbbffffe&date=%3E%3D2016-10-10T23%3A11%3A21.000Z&date=%3C2017-01-10T23%3A11%3A21.000Z

The set of changesets is https://hg.mozilla.org/releases/mozilla-release/pushloghtml?fromchange=FENNEC_50_0_2_RELEASE&tochange=FENNEC_50_1_0_RELEASE it looks to be 30 to 50 changes.

Comment 21

[Hannes Verschore [:h4writer]]

Tomcat his girlfriend can reproduce this (4 times in a week). He himself couldn't. Seemed that one difference was that she had myKnox installed. She is now trying with that disabled and see if she still see crashes.  We should know something about that end this week or next week, hopefully.

(In reply to Kevin Brosnan [:kbrosnan] from comment #20)
> The set of changesets is
> https://hg.mozilla.org/releases/mozilla-release/
> pushloghtml?fromchange=FENNEC_50_0_2_RELEASE&tochange=FENNEC_50_1_0_RELEASE
> it looks to be 30 to 50 changes.

@Jan: Can you have a quick look through the changesets if something catches your eye that could potentially cause these jit crashes?

Comment 22

[Jan de Mooij [:jandem]]

(In reply to Hannes Verschore [:h4writer] from comment #21)
> @Jan: Can you have a quick look through the changesets if something catches
> your eye that could potentially cause these jit crashes?

I don't see anything that looks related. There are some JS patches in there, but I don't see how they could cause JIT crashes and only on ARM/Android.

Hannes, do these reports contain some code around the IP? If yes it would be useful to look at a number of them to see if there's a pattern.

Comment 23

[Hannes Verschore [:h4writer]]

Couldn't reproduce this yet, though I got a lead that I'm investigating.

Comment 24

[Hannes Verschore [:h4writer]]

I've been looking into this for some serious time already.

Tried to reproduce this on the phone I ordered. Didn't get it to fail yet.
That is when using the regular browser (release). But I also tried an specific attack and compiled the JS engine.
I know this is mostly happening in baseline and I think switching between the cores might be related.
As a result I constructed a payload to run a benchmark that would exhibit this problem if running long enough.
But couldn't get it to reproduce.
Starting to think this might be a newer version of the chip?
Asked Standard8 for his cpuinfo, since he crashes on it. To see if it is older. Didn't get an answer yet.

I'm quite convinced this is a CPU bug (in ARM big.LITTLE implementation of A53 and A57).
There are a lot of erratas on Aarch64 programs. One that would fit this issue quite nicely:
http://www.mono-project.com/news/2016/09/12/arm64-icache/
Now we are running Aarch32 which as far as I know is not affected. Or I'm misreading the source.
Now switching to Arm64 for firefox is definitely not a solution yet.
Also our logs show that we crash on any address. Not specific the top 64bits.
One of the specifics this report is mentioning.

I've looked at different dumps, but lately more to the SIGILL onces. Since the logs are dominated by them.
I can definitely see that this happens mostly in baseline code. Main code and stubs.
All happening on valid code (ldr/str/mov/...). Not specific memory opcodes.
I haven't been able to find a correlation between logs yet.

I also skimmed the internet for specific Aarch32 a53/a57 errata. But haven't found anything useful yet.
I think people mostly run arm64 on that architecture.

Comment 25

[Hannes Verschore [:h4writer]]

Related to all my investigation I'm able to trigger a crash in the Galaxy S6.
I'm hoping this is this bug, but I cannot see it. Every time I press I want to send the crash information, but I cannot see this crash report in "about:crashes". Is there something I can do about that. It would be easier to fix if I could reproduce and it would be stupid that I'm already able to reproduce but I don't see it... What can I do to see the crash report?

Comment 26

[Kevin Brosnan [Ex-Mozilla]]

Fill out the email address section of the crash reporter. You can then log onto crash-stats.mozilla.org with that email account's Google account. Either click the Your profile link in the lower left or visit https://crash-stats.mozilla.com/profile/

Else fill out the email section and people with rawdump access can provide you the crash ids. Ping me or ask in #crashkill if you need this.

Leaving the Ted NI in case he has other ideas.

Comment 27

[Hannes Verschore [:h4writer]]

(In reply to Kevin Brosnan [:kbrosnan] from comment #26)
> Fill out the email address section of the crash reporter. You can then log
> onto crash-stats.mozilla.org with that email account's Google account.
> Either click the Your profile link in the lower left or visit
> https://crash-stats.mozilla.com/profile/
> 
> Else fill out the email section and people with rawdump access can provide
> you the crash ids. Ping me or ask in #crashkill if you need this.
> 
> Leaving the Ted NI in case he has other ideas.

Thanks for the tip. I did fill out the email address: [e-mail]. Nowhere visible on https://crash-stats.mozilla.com/profile/ either?

Comment 28

[(not currently active) Ted Mielczarek]

Are you saying that you're submitting the crash report but it doesn't show up in about:crashes? If that's the case then something must be going wrong with the submission.

Note there's also bug 1319071, which makes unsubmitted crash reports not visible in about:crashes on linux systems (including Android).

(In reply to Hannes Verschore [:h4writer] from comment #27)
> Thanks for the tip. I did fill out the email address:
> [e-mail]. Nowhere visible on
> https://crash-stats.mozilla.com/profile/ either?

I searched for crash reports with that email and didn't find any, so something is clearly not working.

Comment 29

[Hannes Verschore [:h4writer]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #28)
> Are you saying that you're submitting the crash report but it doesn't show
> up in about:crashes? If that's the case then something must be going wrong
> with the submission.
> 
> Note there's also bug 1319071, which makes unsubmitted crash reports not
> visible in about:crashes on linux systems (including Android).
> 
> (In reply to Hannes Verschore [:h4writer] from comment #27)
> > Thanks for the tip. I did fill out the email address:
> > [e-mail]. Nowhere visible on
> > https://crash-stats.mozilla.com/profile/ either?
> 
> I searched for crash reports with that email and didn't find any, so
> something is clearly not working.

Yes, I submitted the crash report, but I'm not seeing them in about:crashes nor find them online. Which is quite annoying. What can I do to diagnose or fix this. (I hope I can still reproduce, since I flashed an older rom on the device now).

Comment 30

[(not currently active) Ted Mielczarek]

Can you capture logcat while reproducing the crash and submitting? If we're failing to submit there ought to be some information there.

Comment 31

[Hannes Verschore [:h4writer]]

I thought this might be caused by an older kernel and flashed my device. I only found out later that we record the builds in the "app notes" and looking into them there doesn't seem a particular version that occurs more, except for the last builds. Seems like everybody nicely upgrades their phones.

Now it does mean that my crash reports are now getting submitted.

I also found a way to reproduce. Every 24h I have 1-2 crash reports (very irregular). I looked at the 3 I have:
https://crash-stats.mozilla.com/report/index/20d1071c-9dc1-488a-ab3e-196772170126
https://crash-stats.mozilla.com/report/index/ddd1ae46-2856-427c-8473-274de2170126
https://crash-stats.mozilla.com/report/index/a080cc91-8a53-48c7-9222-9c5822170125

The ddd1ae46-2856-427c-8473-274de2170126 is definitely the same issue we have here. The other two look similar and could be the same cause, but also could be something else.

I will run this for the rest of the week. That way I have better idea of how much crashes we have and how they look.
Reading the errata of the A53 and fixes others are doing (in aarch64) I have potentially an idea what workarounds I can try. Next week I'll compile a build with a workaround and search for a workaround that works.

Comment 33

[Hannes Verschore [:h4writer]]

Yesterday I tried "workaround 1". Here I flush the cache twice, making sure the start address of the flush differs.
It has been happily buzzing for almost 24h. No crash yet! Now it could be I only made it less likely to hit this bug.
Though it would already be a 4x reduction, currently. Which would already be an improvement. I'll keep it buzzing for some more time and see. In the meantime I'll try to add some cpu detection to make sure we only do this workaround on the affected cpus.

Comment 34

[Hannes Verschore [:h4writer]]

Attachment: Workaround by doing two cacheflushes

I didn't expected my first workaround to be good immediately, but it has been running for 2 days without crashes. (Not saying that it fully fixes it, but it will at least decreases it 10x. Crash-stat will probably give a better picture. I'll also keep the phone running.)

When I went through the errata there were some know bugs w.r.t. the big.LITTLE architecture and when switching between cores. In some cases the instruction cache wouldn't get updated. There were also issues with the kernel code (in arm64) that was caching the cache line length. But found none that would match our case exactly. 

Some fixes I saw (on arm64) were to add NOPS and to increase the size of flushcaching. Now a few of the errata are arm64 specific. And I assume a few of the errata for arm32 are not found yet.

Doing the cacheflush twice works and I deliberate made sure the starting address is different for both. I think I read somewhere that the start address made a difference for some bugs. In our case it is also fine, since we put the pointer to JitCode first.

I'm very happy we don't have to add extra NOPs, since that would be awful for our stubs.

Comment 35

[Jan de Mooij [:jandem]]

Comment on attachment 8832716 [details] [diff] [review]
Workaround by doing two cacheflushes

Review of attachment 8832716 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for digging into this! It's definitely worth a try to see how it affects crash stats.

::: js/src/jit/ExecutableAllocator.cpp
@@ +327,5 @@
> +    // The exynos7420 cpu (EU galaxy S6/S7 (Note)) has a bug where sometimes
> +    // flushing doesn't invalidate the instruction cache. As a result we force
> +    // it by calling the cacheFlush twice on different start address.
> +    FILE* fp = fopen("/proc/cpuinfo", "r");
> +    if (fp) {

I'm removing initStatic in another (s-s) bug and we don't need this on non-Linux/ARM. What do you think about moving this into Architecture-arm.cpp, maybe in InitARMFlags where we have other code that opens /proc/cpuinfo? We could expose a NeedsDoubleCacheFlush() function and call that in ExecutableAllocator.h.

(We should move the cache flush code for each platform into Architecture-*.cpp/h, the code in ExecutableAllocator.h is becoming quite unwieldy, but that's not necessary for this bug.)

Comment 36

[Hannes Verschore [:h4writer]]

Attachment: Workaround by doing two cacheflushes

My fault. I tried this, but included the wrong file and as a result thought we had cyclic dependencies.

Comment 37

[Jan de Mooij [:jandem]]

Comment on attachment 8832983 [details] [diff] [review]
Workaround by doing two cacheflushes

Review of attachment 8832983 [details] [diff] [review]:
-----------------------------------------------------------------

Sorry for the delay. Will be interesting to see how it affects crash stats :)

::: js/src/jit/ExecutableAllocator.cpp
@@ +30,5 @@
>  #include "mozilla/Atomics.h"
>  
> +#ifdef JS_CODEGEN_ARM
> +#include "jit/arm/Architecture-arm.h"
> +#endif

Nit: these lines can be removed I think.

::: js/src/jit/arm/Architecture-arm.cpp
@@ +247,5 @@
>          }
> +
> +        // The exynos7420 cpu (EU galaxy S6 (Note)) has a bug where sometimes
> +        // flushing doesn't invalidate the instruction cache. As a result we force
> +        // it by calling the cacheFlush twice on different start address.

Nit: addresses

Comment 38

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/44e3e172cb97
Double flush the instruction cache as workaround on the exynos7420 chipset, r=jandem

Comment 39

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/abfb63650780
Include Architecture-arm in ExecutableAllocator, r=bustage ON CLOSED TREE

Comment 40

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/44e3e172cb97
https://hg.mozilla.org/mozilla-central/rev/abfb63650780

Comment 41

[(PTO June 19-July 7) Ryan VanderMeulen]

Please request Aurora/Beta approval on this when you get a chance.

Comment 42

[Hannes Verschore [:h4writer]]

Attachment: Beta patch

Comment 43

[Hannes Verschore [:h4writer]]

Attachment: Aurora patch

Comment 44

[Hannes Verschore [:h4writer]]

Comment on attachment 8834860 [details] [diff] [review]
Beta patch

Approval Request Comment
[Feature/Bug causing the regression]:
The JITS on exynos7420 cpus (galaxy s6)

[User impact if declined]:
Crashes at random.

[Is this code covered by automated tests?]:
Yes

[Has the fix been verified in Nightly?]:
Yes

[Needs manual test from QE? If yes, steps to reproduce]: 
/

[List of other uplifts needed for the feature/fix]:
/

[Is the change risky?]:
No

[Why is the change risky/not risky?]:
It tests if it is this specific cpu and will flush the cache twice. I.e. the features are already quite good tested. It could add a little bit of slowdown on flushing. But that should not be noticeable.

[String changes made/needed]:
/

Comment 45

[Hannes Verschore [:h4writer]]

Comment on attachment 8834863 [details] [diff] [review]
Aurora patch

Approval Request Comment
[Feature/Bug causing the regression]:
The JITS on exynos7420 cpus (galaxy s6)

[User impact if declined]:
Crashes at random.

[Is this code covered by automated tests?]:
Yes

[Has the fix been verified in Nightly?]:
Yes

[Needs manual test from QE? If yes, steps to reproduce]: 
/

[List of other uplifts needed for the feature/fix]:
/

[Is the change risky?]:
No

[Why is the change risky/not risky?]:
It tests if it is this specific cpu and will flush the cache twice. I.e. the features are already quite good tested. It could add a little bit of slowdown on flushing. But that should not be noticeable.

[String changes made/needed]:
/

Comment 46

[Gerry Chang [:gchang]]

Comment on attachment 8834863 [details] [diff] [review]
Aurora patch

Fix a crash. Aurora53+.

Comment 47

[Carsten Book [:Tomcat]]

 https://hg.mozilla.org/releases/mozilla-aurora/rev/522951ff7046fd2bdf3916c973fbfde27be0af31

Comment 48

[Julien Cristau [:jcristau]]

Comment on attachment 8834860 [details] [diff] [review]
Beta patch

force double cache flush on galaxy s6 to fix random crashes, beta52+

Comment 49

[(not currently active) Ted Mielczarek]

FTR, I think this is the same bug as described here: http://www.mono-project.com/news/2016/09/12/arm64-icache/

Comment 50

[Carsten Book [:Tomcat]]

needs rebasing for beta

patching file js/src/jit/ExecutableAllocator.h
Hunk #1 FAILED at 30
Hunk #2 FAILED at 277
2 out of 2 hunks FAILED -- saving rejects to file js/src/jit/ExecutableAllocator.h.rej
patching file js/src/jit/arm/Architecture-arm.cpp
Hunk #2 FAILED at 190
1 out of 2 hunks FAILED -- saving rejects to file js/src/jit/arm/Architecture-arm.cpp.rej
patch failed, unable to continue (try -v)
patch failed, rejects left in working directory
errors during apply, please fix and qrefresh file_1286307.txt

Comment 51

[Hannes Verschore [:h4writer]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #49)
> FTR, I think this is the same bug as described here:
> http://www.mono-project.com/news/2016/09/12/arm64-icache/

see comment 24, where I listed that already. Though that bug is ARM64. We are using ARM32.

Comment 52

[Hannes Verschore [:h4writer]]

Attachment: Rebased beta patch

Comment 53

[Carsten Book [:Tomcat]]

Attachment: .rej file

still have problems to apply also with a new clone attaching js/src/jit/ExecutableAllocator.h.rej

Comment 54

[Hannes Verschore [:h4writer]]

Attachment: Rebased beta patch

Comment 55

[Hannes Verschore [:h4writer]]

Attachment: Rebased beta patch

Comment 56

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-beta/rev/dd8a1f27c391b0669bd6890f7efe3c9efd35e7ac

Comment 57

[Cornel Ionce [:noni] [Hubs QA]]

Setting qe-verify- per comment 45.

Comment 58

[(PTO June 19-July 7) Ryan VanderMeulen]

https://hg.mozilla.org/releases/mozilla-esr52/rev/dd8a1f27c391

Comment 59

[Hannes Verschore [:h4writer]]

Preliminary data shows this decreased crash-rate with 50%. We only have 2 days of data on beta though. I'm keeping my eyes on the graphs.

Comment 60

[Hannes Verschore [:h4writer]]

Re-opening. Looking now it hasn't changed the crash-rate in beta. Looking again.

Comment 61

[Liz Henry (:lizzard) (relman/hg->git project)]

Noting that this is still the #2 top crash on release 53.

Comment 62

[Jason Orendorff [:jorendorff]]

This still seems to be high-volume. Can anything else be done here?

Comment 63

[Jan de Mooij [:jandem]]

Looking at 62.0.3 mobile crashes, any process, I see the first dalvik-main crash at #68 and it seems to be improving:

https://crash-stats.mozilla.com/topcrashers/?product=FennecAndroid&version=62.0.3&_facets_size=200&process_type=any

Given this and the fact our workarounds so far have failed, I don't think this is worth spending a lot of time on right now.

Comment 64

[Nicolas B. Pierron [:nbp]]

(In reply to Jan de Mooij [:jandem] from comment #63)
> Given this and the fact our workarounds so far have failed, I don't think
> this is worth spending a lot of time on right now.

Marking this bug as stalled until we find better avenue for investigating this issue.

Comment 65

[Ted Campbell [:tcampbell]]

Folding this bug into Bug 1461724 where we see the same sort of issue for two more big.LITTLE processors developed by Qualcomm this time. The Exynos7420 crashes still remain. The crashes apply to both arm32 and arm64 builds.

Comment 66

[BugBot [:suhaib / :marco/ :calixte]]

Since the bug is closed, the stalled keyword is now meaningless.
For more information, please visit auto_nag documentation.