1464034 - Perma googleapiclient.errors.HttpError: <HttpError 500 when requesting https://www.googleapis.com/androidpublisher/v2/applications/org.mozilla.fennec_aurora/edits/05450589015340056014:commit?alt=json returned "">

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: ebalazs [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=179985824&repo=mozilla-central

https://queue.taskcluster.net/v1/task/XptGCTWCSdy4nMawupC1TA/runs/0/artifacts/public/logs/live_backing.log

Comment 1

[Eliza Balazs [:ebalazs_]]

Traceback (most recent call last):
  File "/builds/scriptworker/bin/pushapkscript", line 11, in <module>
    load_entry_point('pushapkscript==0.7.0', 'console_scripts', 'pushapkscript')()
  File "/builds/scriptworker/lib/python3.6/site-packages/pushapkscript/script.py", line 72, in main
    client.sync_main(async_main, config_path=config_path, default_config=get_default_config())
  File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/client.py", line 161, in sync_main
    loop.run_until_complete(_handle_asyncio_loop(async_main, context))
  File "/tools/python36/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/client.py", line 200, in _handle_asyncio_loop
    await async_main(context)
  File "/builds/scriptworker/lib/python3.6/site-packages/pushapkscript/script.py", line 43, in async_main
    context, all_apks_paths, google_play_strings_path,
  File "/builds/scriptworker/lib/python3.6/site-packages/pushapkscript/googleplay.py", line 19, in publish_to_googleplay
    push_apk.run()
  File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/push_apk.py", line 98, in run
    self.upload_apks(apks_metadata, package_name, l10n_strings)
  File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/push_apk.py", line 70, in upload_apks
    edit_service.commit_transaction()
  File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/common/googleplay.py", line 77, in _transaction_required
    return method(*args, **kwargs)
  File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/common/googleplay.py", line 83, in commit_transaction
    self._service.commit(editId=self._edit_id, packageName=self._package_name).execute()
  File "/builds/scriptworker/lib/python3.6/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/builds/scriptworker/lib/python3.6/site-packages/googleapiclient/http.py", line 840, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 500 when requesting https://www.googleapis.com/androidpublisher/v2/applications/org.mozilla.fennec_aurora/edits/05450589015340056014:commit?alt=json returned "">
exit code: 1

Comment 2

[Johan Lorenzo [:jlorenzo]]

The 500 error comes from Google Play which shows they're having trouble to process our request. The fact they don't provide any error message makes me think they deployed a buggy version of their servers. Let's try again later today to see if this is resolved.

Comment 3

[Johan Lorenzo [:jlorenzo]]

The last rerun didn't pass either. I contacted the Google Play support. They don't have any answer by now. They'll follow up by email.

Comment 4

[Johan Lorenzo [:jlorenzo]]

I haven't got any tangible answer from Google Play, but the last rerun passed https://tools.taskcluster.net/groups/X3b9IZlNSveQXocHyCI2MQ/tasks/XptGCTWCSdy4nMawupC1TA/runs/4

Comment 5

[Intermittent Failures Robot]

4 failures in 632 pushes (0.006 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 4

Platform breakdown:
* Android: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1464034&startday=2018-05-21&endday=2018-05-27&tree=trunk

Comment 6

[Raul Gurzau (:RaulG)]

This seems to occur again here:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=87b46cd1c941782b75fcaa6b311bd9d7bd7f2120

Log link: https://treeherder.mozilla.org/logviewer.html#?job_id=182906535&repo=mozilla-central&lineNumber=209

Log snippet:

Traceback (most recent call last):
  File "/builds/scriptworker/bin/pushapkscript", line 11, in <module>
    load_entry_point('pushapkscript==0.7.0', 'console_scripts', 'pushapkscript')()
  File "/builds/scriptworker/lib/python3.6/site-packages/pushapkscript/script.py", line 72, in main
    client.sync_main(async_main, config_path=config_path, default_config=get_default_config())
  File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/client.py", line 160, in sync_main
    loop.run_until_complete(_handle_asyncio_loop(async_main, context))
  File "/tools/python36/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/client.py", line 199, in _handle_asyncio_loop
    await async_main(context)
  File "/builds/scriptworker/lib/python3.6/site-packages/pushapkscript/script.py", line 43, in async_main
    context, all_apks_paths, google_play_strings_path,
  File "/builds/scriptworker/lib/python3.6/site-packages/pushapkscript/googleplay.py", line 19, in publish_to_googleplay
    push_apk.run()
  File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/push_apk.py", line 98, in run
    self.upload_apks(apks_metadata, package_name, l10n_strings)
  File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/push_apk.py", line 70, in upload_apks
    edit_service.commit_transaction()
  File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/common/googleplay.py", line 77, in _transaction_required
    return method(*args, **kwargs)
  File "/builds/scriptworker/lib/python3.6/site-packages/mozapkpublisher/common/googleplay.py", line 83, in commit_transaction
    self._service.commit(editId=self._edit_id, packageName=self._package_name).execute()
  File "/builds/scriptworker/lib/python3.6/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/builds/scriptworker/lib/python3.6/site-packages/googleapiclient/http.py", line 840, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 500 when requesting https://www.googleapis.com/androidpublisher/v2/applications/org.mozilla.fennec_aurora/edits/11941889134371185798:commit?alt=json returned "">
exit code: 1

Comment 7

[Johan Lorenzo [:jlorenzo]]

Thanks for reopening this bug. The error is the same. Let's try again in a few hours or tomorrow. Last time I contacted the Google Play support, they weren't useful.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

We haven't successfully uploaded a Nightly APK to the Play Store since June 11th. This needs to be escalated ASAP.

Comment 9

[Cristian Brindusan [:cbrindusan]]

The same error occured after today's merges:

https://treeherder.mozilla.org/logviewer.html#?job_id=183485227&repo=mozilla-central&lineNumber=209

Comment 10

[Ryan VanderMeulen [:RyanVM]]

In case Johan isn't in a position to investigate, can you please help find an owner for this, Chris? I assume that this will also block any Beta/Release uploads in the coming week as well.

Comment 11

[Chris AtLee [:catlee]]

I'm not sure anyone else in releng has access to the play store. Perhaps Sylvestre could re-raise this issue with Google?

It sounds like last time this issue resolved itself, and at the same time we didn't get useful error messages back from the API, nor did we get a resolution from Google.

Worst case is we have to upload manually I think.

Comment 12

[Intermittent Failures Robot]

18 failures in 408 pushes (0.044 failures/push) were associated with this bug in the last 7 days. 

This is the #36 most frequent failure this week.   

Repository breakdown:
* mozilla-central: 18

Platform breakdown:
* Android: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1464034&startday=2018-06-11&endday=2018-06-17&tree=trunk

Comment 13

[Sylvestre Ledru [:Sylvestre]]

I uploaded them by hand and saw the following warning:
---
Target API level requirements from August 2018
Warning:

Your app currently targets API level 23 and must target at least API level 26 to ensure it is built on the latest APIs optimized for security and performance.

From August 2018, new apps must target at least Android 8.0 (API level 26).
From November 2018, app updates must target Android 8.0 (API level 26).

Tip:

Change your app's target API level. Learn how.
---

Maybe it is the source of the problem?

Comment 14

[Johan Lorenzo [:jlorenzo]]

I just had a chat with the Google Play support. This time, I made sure the person understood we are using the API[1] and not the web interface (what they call the "Play Console"). The person took note of the error reported by the API and that was the only piece of context we had. Therefore, I hope they won't get back to me asking for another screenshot. 

The technical team will look into the issue and come back to us shortly, via email.

[1] Documented at https://developers.google.com/android-publisher/api-ref/

Comment 15

[Johan Lorenzo [:jlorenzo]]

For a second, I thought this was caused by a fallout of bug 1369668 comment 21, but the fix was deployed after we noticed the current issue.

Comment 16

[Johan Lorenzo [:jlorenzo]]

:jcristau had a good idea: how about running push_apk.py with our dev creds? (in order to make sure bug 1369668 is not impacted). I ran the script against mine. I got the same 500 error with no message.

Comment 17

[Johan Lorenzo [:jlorenzo]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #14)
> Therefore, I hope they won't get back to me asking for another screenshot. 

Damn, they did ask for it:

> I'm sorry to hear that you're having issues with Google Developer API. In order for me to help you, please provide as much of the following information as you can:
>     A screenshot of the problematic behavior you’re reporting
>     What steps did you take to reproduce this issue(important)? Did you meet the error while accessing the API or trying to publish new APK?
>     What is the exact error message that is displayed, if any?
> If there is any other relevant information you'd like to add, please feel free to do so. 


I replied with the real answers + pointers to what we already stated before.

Comment 18

[Intermittent Failures Robot]

9 failures in 521 pushes (0.017 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 9

Platform breakdown:
* Android: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1464034&startday=2018-06-18&endday=2018-06-24&tree=trunk

Comment 19

[Johan Lorenzo [:jlorenzo]]

For the record, yesterday the support "documented your issue and escalated it to our technical team for further investigation.". I'm surprised we just reached the technical team yesterday. After the initial chat, I though we already fit in that category :/

For future reference, it took 4 back-and-forths to get the whole information they apparently need. That is to say:

* can we have a screenshot? (I told them that was poorly relevant in this context)
* what are the STR? (before this question, I remember explaining in plain English what we do to get this error. I also pointed them to our logs. Moreover, the error message also contains a transactionID, which seems reasonable to be looked up in their logs to see what we tried to do)
* does it happen constantly? (that piece of info was given in the original chat)
* what account do you use? (I'm surprised they need this piece, when we provided transactionIDs in the first place. I asked why, I didn't get any answer)
* does it happen on other Google Play Developer API calls? (that question was actually unanswered. I told them Nightly was the only product impacted)

Comment 20

[Intermittent Failures Robot]

4 failures in 718 pushes (0.006 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 4

Platform breakdown:
* Android: 3
* android-em-4-3-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1464034&startday=2018-06-25&endday=2018-07-01&tree=trunk

Comment 21

[Johan Lorenzo [:jlorenzo]]

Julien noted yesterday's job passed[1]. It's actually the case since Friday[2]. I checked with Google Play, they confirmed they deployed a fix. We're done here, then! 

[1] https://tools.taskcluster.net/groups/VBu4DboRToanyvJnRD9AcQ/tasks/LSbrR8SmSZGgdFOfIDHudw/runs/0/logs/public%2Flogs%2Flive_backing.log#L183
[2] https://tools.taskcluster.net/groups/fsOQyFp2QNSij4UDCyAm6Q/tasks/Ta8HHLrvSgiRVx0Gvsb6KQ/runs/0/logs/public%2Flogs%2Flive_backing.log#L183