1431140 - Android nightly Google Play publishing failures due to "APK is marked as debuggable." error

Status: RESOLVED FIXED

(Firefox Build System :: Android Studio and Gradle Integration, defect, P1)

Description

[Ryan VanderMeulen [:RyanVM]]

We haven't been able to publish Fennec nightlies to Google Play for the last two days due to the below error:
https://public-artifacts.taskcluster.net/Utdwv3cUSWSr4EiIqzm_iw/0/public/logs/live_backing.log

googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/upload/androidpublisher/v2/applications/org.mozilla.fennec_aurora/edits/05751475843230325114/apks?uploadType=media&alt=json returned "APK is marked as debuggable.">

Regression range below shows a lot of Android build changes during that window.
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=b7d66e4e60ef&tochange=e4107773cffb

Nick, can you please look ASAP? This is presumably going to be a blocker for shipping Fennec 59 betas next week (not to mention the whole lack of nightlies right now).

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Assuming this was a regression from bug 1411654, I have to wonder why in the world such a big change was landed during the Nightly 59 soft freeze anyway.

Comment 2

[Johan Lorenzo [:jlorenzo]]

For the record, the same error occurred on the 13th[1]. Backing out bug 1411654 made the Google Play upload working again[3]

[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=pub&fromchange=739484451a6399c7f156a0d960335606aa6c1221&selectedJob=156163173
[2] https://hg.mozilla.org/mozilla-central/rev/64f82460345884dd0f5935645765acccf771d7c4
[3] https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=pub&fromchange=739484451a6399c7f156a0d960335606aa6c1221&selectedJob=156240316

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Unless there's a very good reason for bug 1411654 having to ship in 59, I'd vote for backing it out and not re-landing until after the Gecko 60 bump next week.

Comment 4

[Nick Alexander :nalexander [he/him]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #3)
> Unless there's a very good reason for bug 1411654 having to ship in 59, I'd
> vote for backing it out and not re-landing until after the Gecko 60 bump
> next week.

Let's do that, even though it slows things down.  Clearly my APK comparison process hasn't been rigorous enough for this change.

It's only landing this late because of all the things that happened in the 59 cycle (All-Hands and Christmas), so it can slip to 60.

Comment 5

[Nick Alexander :nalexander [he/him]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #2)
> For the record, the same error occurred on the 13th[1]. Backing out bug
> 1411654 made the Google Play upload working again[3]

Oh yeah, this is definitely Bug 1411654 -- we have fragile handling for Debug vs. Release in production, and I triggered changes there that I didn't think to check for.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

Fixed by the backout of bug 1411654 et al.

https://hg.mozilla.org/mozilla-central/rev/ef04f3ad8475

Comment 7

[Intermittent Failures Robot]

2 failures in 657 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 2

Platform breakdown:
* Android: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1431140&startday=2018-01-15&endday=2018-01-21&tree=all