1505514 - Maven upload issues on fennec 63.0.2

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Justin Wood (:Callek)]

For the ship graph of fennec 63.0.2 (https://tools.taskcluster.net/groups/fzkRkMt9SpOBW3yxqZK-aw) we hit 3 beetmover task failures for maven...

This did not block the Fennec 63.0.2 release (google play pushing succeeded)

Those tasks were:

https://tools.taskcluster.net/groups/fzkRkMt9SpOBW3yxqZK-aw/tasks/Ta2RKEBXSEeaSDA-OK6heA/details
https://tools.taskcluster.net/groups/fzkRkMt9SpOBW3yxqZK-aw/tasks/c678QILXTEeLMLHsEN5jGA/details
and https://tools.taskcluster.net/groups/fzkRkMt9SpOBW3yxqZK-aw/tasks/RejAzZSPRp2gZ_hRvVkJ3g/details

From the log of one of those:

2018-11-07 18:07:59,282 - beetmoverscript.zip - INFO - Archive "/builds/scriptworker/work/cot/LwarfUeeRSud4fZYiCOWfA/public/build/target.maven.zip" contains files with legitimate sizes.
2018-11-07 18:07:59,282 - scriptworker.client - ERROR - Failed to run async_main
Traceback (most recent call last):
  File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/client.py", line 203, in _handle_asyncio_loop
    await async_main(context)
  File "/builds/scriptworker/lib/python3.6/site-packages/beetmoverscript/script.py", line 287, in async_main
    await action_map[context.action](context)
  File "/builds/scriptworker/lib/python3.6/site-packages/beetmoverscript/script.py", line 172, in push_to_maven
    context.config.get('zip_max_file_size_in_mb', DEFAULT_ZIP_MAX_FILE_SIZE_IN_MB)
  File "/builds/scriptworker/lib/python3.6/site-packages/beetmoverscript/script.py", line 184, in _extract_and_check_maven_artifacts_to_beetmove
    artifacts, expected_files, zip_max_file_size_in_mb
  File "/builds/scriptworker/lib/python3.6/site-packages/beetmoverscript/zip.py", line 61, in check_and_extract_zip_archives
    task_id, expected_files_per_archive, zip_max_size_in_mb
  File "/builds/scriptworker/lib/python3.6/site-packages/beetmoverscript/zip.py", line 75, in _check_and_extract_zip_archives_for_given_task
    archive_path, expected_files, zip_max_size_in_mb
  File "/builds/scriptworker/lib/python3.6/site-packages/beetmoverscript/zip.py", line 95, in _check_extract_and_delete_zip_archive
    _ensure_all_expected_files_are_present_in_archive(zip_path, relative_paths_in_archive, expected_files)
  File "/builds/scriptworker/lib/python3.6/site-packages/beetmoverscript/zip.py", line 186, in _ensure_all_expected_files_are_present_in_archive
    file_, zip_path, unique_expected_files
scriptworker.exceptions.TaskVerificationError: File "org/mozilla/geckoview/geckoview-release-armeabi-v7a/63.0.20181105152108/geckoview-release-armeabi-v7a-63.0.20181105152108.aar.md5" present in archive "/builds/scriptworker/work/cot/LwarfUeeRSud4fZYiCOWfA/public/build/target.maven.zip" is not expected. Expected: {'org/mozilla/geckoview/geckoview-armeabi-v7a/63.0.20181105152108/geckoview-armeabi-v7a-63.0.20181105152108.pom.sha1', 'org/mozilla/geckoview/geckoview-armeabi-v7a/63.0.20181105152108/geckoview-armeabi-v7a-63.0.20181105152108.pom.md5', 'org/mozilla/geckoview/geckoview-armeabi-v7a/63.0.20181105152108/geckoview-armeabi-v7a-63.0.20181105152108-sources.jar.sha1', 'org/mozilla/geckoview/geckoview-armeabi-v7a/63.0.20181105152108/geckoview-armeabi-v7a-63.0.20181105152108-javadoc.jar.md5', 'org/mozilla/geckoview/geckoview-armeabi-v7a/63.0.20181105152108/geckoview-armeabi-v7a-63.0.20181105152108-javadoc.jar.sha1', 'org/mozilla/geckoview/geckoview-armeabi-v7a/63.0.20181105152108/geckoview-armeabi-v7a-63.0.20181105152108-sources.jar.md5', 'org/mozilla/geckoview/geckoview-armeabi-v7a/63.0.20181105152108/geckoview-armeabi-v7a-63.0.20181105152108.aar.sha1', 'org/mozilla/geckoview/geckoview-armeabi-v7a/63.0.20181105152108/geckoview-armeabi-v7a-63.0.20181105152108-javadoc.jar', 'org/mozilla/geckoview/geckoview-armeabi-v7a/63.0.20181105152108/geckoview-armeabi-v7a-63.0.20181105152108.aar', 'org/mozilla/geckoview/geckoview-armeabi-v7a/63.0.20181105152108/geckoview-armeabi-v7a-63.0.20181105152108.pom', 'org/mozilla/geckoview/geckoview-armeabi-v7a/63.0.20181105152108/geckoview-armeabi-v7a-63.0.20181105152108-sources.jar', 'org/mozilla/geckoview/geckoview-armeabi-v7a/63.0.20181105152108/geckoview-armeabi-v7a-63.0.20181105152108.aar.md5'}

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

At first glance, it seems like there's and extra "release" within it in the extra in the file naming. 

We have "org/mozilla/geckoview/geckoview-release-armeabi-v7a/63.0.20181105152108/geckoview-release-armeabi-v7a-63.0.20181105152108.aar.md5" when we should be having "org/mozilla/geckoview/geckoview-armeabi-v7a/63.0.20181105152108/geckoview-armeabi-v7a-63.0.20181105152108.aar.md5" (trimming the both occurrence of "release").

Might be related to the changes that were happening in bug 1500897 but it's weird as those patches (mainly https://phabricator.services.mozilla.com/D9551) were never uplifted. So it's either expected to fail or there's something else going on.

Keeping the NI to investigate more.

Comment 2

[Johan Lorenzo [:jlorenzo] - PTO - Back April 13th]

I'm sorry, I forgot to tell you over IRC, Mihai. I'm on the fix. I have something ready on my machine.

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Johan Lorenzo [:jlorenzo] from comment #2)
> I'm sorry, I forgot to tell you over IRC, Mihai. I'm on the fix. I have
> something ready on my machine.

No worries, thanks for working on it. Looking forward for the fix!

Comment 4

[Johan Lorenzo [:jlorenzo] - PTO - Back April 13th]

Attachment: Bug 1505514 - geckoview artifacts don't specify branch when mozilla-release r=snorp

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c38af115dcd9
geckoview artifacts don't specify branch when mozilla-release r=snorp

Comment 6

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/c38af115dcd9

Comment 7

[Intermittent Failures Robot]

4 failures in 3007 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-release: 4

Platform breakdown:
* android-5-0-aarch64: 1
* android-4-2-x86: 1
* android-4-0-armv7-api16: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1505514&startday=2018-11-05&endday=2018-11-11&tree=all

Comment 8

[Johan Lorenzo [:jlorenzo] - PTO - Back April 13th]

https://hg.mozilla.org/releases/mozilla-beta/rev/ec9a256c6703565eafd888030b9502f55092de38

Comment 9

[Johan Lorenzo [:jlorenzo] - PTO - Back April 13th]

Comment 8 was backed out[1] because it caused "gradle-dependencies"[2] to fail:

> [task 2018-11-12T09:40:03.017Z]  1:11.04 7: Task failed with an exception.
> [task 2018-11-12T09:40:03.017Z]  1:11.04 -----------
> [task 2018-11-12T09:40:03.017Z]  1:11.04 * What went wrong:
> [task 2018-11-12T09:40:03.017Z]  1:11.04 Could not resolve all files for configuration ':app:officialWithoutGeckoBinariesNoMinApiPhotonDebugAndroidTestRuntimeClasspath'.
> [task 2018-11-12T09:40:03.017Z]  1:11.04 > Could not find play-services-basement.aar (com.google.android.gms:play-services-basement:15.0.1).
> [task 2018-11-12T09:40:03.017Z]  1:11.04   Searched in the following locations:
> [task 2018-11-12T09:40:03.017Z]  1:11.04       http://localhost:8081/nexus/content/repositories/jcenter/com/google/android/gms/play-services-basement/15.0.1/play-services-basement-15.0.1.aar
> [task 2018-11-12T09:40:03.017Z]  1:11.04 * Try:
> [task 2018-11-12T09:40:03.017Z]  1:11.04 Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

One interesting thing about this job is that it only whenever a change is made on the gradle-side. If the current changes are already cached, then no job is spawned. So I manually retriggered "gradle-dependencies" on top of the backout[3] and the problem remained. Therefore, this patch is not the root cause of the breakage. I believe bug 1476271 is. Petru, what do you think? 

[1] https://hg.mozilla.org/releases/mozilla-beta/rev/4fe80a60a6a04458add149d68b2dd2f7224f65f9
[2] https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&selectedJob=211160959&revision=ec9a256c6703565eafd888030b9502f55092de38
[3] https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=a4175191b29b274d59ed117567ec03df9d6b8cf5&selectedJob=211224139

Comment 10

[Petru-Mugurel Lingurar [:petru]]

My patch from bug 1476271 was for for another dependency.

Regarding Play Services Basement, it seems that that aar is missing atm from jcenter - https://jcenter.bintray.com/com/google/android/gms/play-services-basement/15.0.1/play-services-basement-15.0.1.aar
But it can be found on found on Google's Maven [1][2] and based on bug 1501351 that is where it should look first.

[1] https://dl.google.com/dl/android/maven2/com/google/android/gms/group-index.xml
[2] https://dl.google.com/dl/android/maven2/com/google/android/gms/play-services-basement/15.0.1/play-services-basement-15.0.1.aar

Comment 11

[Johan Lorenzo [:jlorenzo] - PTO - Back April 13th]

Thank you for the super quick and informative answer, Petru! It seems I need bug 1501351 on beta, then!

Comment 12

[Petru-Mugurel Lingurar [:petru]]

Probably. If a try build with it works, then it's an easy fix.

Comment 13

[Johan Lorenzo [:jlorenzo] - PTO - Back April 13th]

Green try job https://treeherder.mozilla.org/#/jobs?repo=try&revision=f10da03e7f0229f572cd4e4b069b130dc6bfa21d&selectedJob=211416418 \o/

Comment 14

[Johan Lorenzo [:jlorenzo] - PTO - Back April 13th]

Relanded on beta at https://hg.mozilla.org/releases/mozilla-beta/rev/17a9ab8d9200549682861bde9c01e27e4e22c308

Comment 15

[Johan Lorenzo [:jlorenzo] - PTO - Back April 13th]

gradle-dependencies is green on beta: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=17a9ab8d9200549682861bde9c01e27e4e22c308&selectedJob=211421531

Comment 16

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/7dc20948e0ff