1589065 - maven.mozila.org: maven-metadata.xml has not been generated by the lambda function

Status: RESOLVED FIXED

(Cloud Services :: Operations: Kinto, defect, P1)

Description

[Johan Lorenzo [:jlorenzo]]

From :sebastian on Slack:

@jlorenzo I was unable to update an app to AC 18.0.0-SNAPSHOT.
Looking at the repo I noticed that for the components I couldn't load there was maven-metadata.xml missing
e.g.: gradle tells me it can't find this: https://snapshots.maven.mozilla.org/?prefix=maven2/org/mozilla/components/feature-media/18.0.0-SNAPSHOT/
but it didn't have issues with this: https://snapshots.maven.mozilla.org/?prefix=maven2/org/mozilla/components/feature-session/18.0.0-SNAPSHOT/
do you know what could cause that?
I get an error for ~9 components in Fenix.
looks like in all nine cases we do have the aars etc. but not maven-metadata.xml

Offhand, I don't see why feature-media would behave differently.

@CloudOps: Would it possible to get some logs to see what happened?

Comment 1

[Johan Lorenzo [:jlorenzo]]

For the context, the maven-metadata.xml are generated by a lambda function. Its code lives there https://github.com/mozilla-releng/maven-lambda

Comment 2

[Sebastian Kaspari (:sebastian; :pocmo)]

Note that now the next snapshot release was triggered and maven-metadata.xml has appeared for all components.

Comment 3

[Sebastian Kaspari (:sebastian; :pocmo)]

We just noticed again once we started shipping snapshots with a new version (19.0.0-SNAPSHOT). For support-base the maven-metadata.xml file is missing: https://snapshots.maven.mozilla.org/?prefix=maven2/org/mozilla/components/support-base/19.0.0-SNAPSHOT/

Comment 4

[Johan Lorenzo [:jlorenzo]]

I just heard from :jonalmeida, they've hit this problem again.

I know :autrilla used to own this service on the CloudOps side. Would you be the person to escalate this issue to, Habib?

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

I've been encountering issues with AWS in the past few days, could have been something on their end too?
Lambda workers worked for me an hour ago when I pushed something to production: https://maven.mozilla.org/?prefix=maven2/org/mozilla/telemetry/glean/ - files were correctly generated by the lamda workers at 2019-10-24T05:58:34.000Z (UTC)

Could be isolated, definitely worth investigating by CloudOps furthermore.

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

Couldn't retrieve anything unusual in the past two days from the https://status.aws.amazon.com/ though.

Comment 7

[Jonathan Almeida [:jonalmeida]]

I'm still seeing this happen with our 20.0.0 release which happened yesterday: https://snapshots.maven.mozilla.org/?prefix=maven2/org/mozilla/components/feature-findinpage/20.0.0-SNAPSHOT/

If so it is an AWS issue, it may still be happening.

Comment 8

[Sebastian Kaspari (:sebastian; :pocmo)]

I'm currently looking at an issue where a Fenix Nightly build did not pick up the latest snapshot and everything looks good in the process. I'm suspecting that this is a variant of this issue here: If maven-metadata.xml exists (from a previous snapshot build) but wasn't updated in the latest build then this may cause us to build with an older snapshot (basically whatever last snapshot managed to update maven-metadata.xml).

Comment 9

[Jeremy Orem [:oremj]]

I see the following exception in the logs:

"0.35.0-TESTING" does not match the pattern: re.compile('\n ^(?P<major_number>\\d+)\n \\.(?P<minor_number>\\d+)\n (\\.(?P<patch_number>\\d+))?\n (?P<is_snapshot>-SNAPSHOT)?$', re.VERBOSE): PatternNotMatchedError
Traceback (most recent call last):
File "/var/task/metadata.py", line 50, in lambda_handler
metadata_function=generate_release_maven_metadata
File "/var/task/metadata.py", line 115, in craft_and_upload_maven_metadata
metadata = metadata_function(bucket_name, pom_files)
File "/var/task/metadata.py", line 133, in generate_release_maven_metadata
latest_version = get_latest_version(all_versions, exclude_snapshots=False)
File "/var/task/metadata.py", line 260, in get_latest_version
maven_versions = [MavenVersion.parse(version) for version in versions]
File "/var/task/metadata.py", line 260, in <listcomp>
maven_versions = [MavenVersion.parse(version) for version in versions]
File "/var/task/mozilla_version/maven.py", line 27, in parse
return super(MavenVersion, cls).parse(version_string, regex_groups=('is_snapshot', ))
File "/var/task/mozilla_version/version.py", line 40, in parse
raise PatternNotMatchedError(version_string, cls._VALID_ENOUGH_VERSION_PATTERN)
mozilla_version.errors.PatternNotMatchedError: "0.35.0-TESTING" does not match the pattern: re.compile('\n ^(?P<major_number>\\d+)\n \\.(?P<minor_number>\\d+)\n (\\.(?P<patch_number>\\d+))?\n (?P<is_snapshot>-SNAPSHOT)?$', re.VERBOSE)

Not sure if this is related. Feel free to ping me on slack for faster back and forth as well.

Comment 10

[Johan Lorenzo [:jlorenzo]]

Thanks for the logs, Jeremy! I see why the lambda function is failing. Although, I don't know where 0.35.0-TESTING comes from. Jon, Sebastian, have you guys already encountered such a version number?

Comment 11

[Sebastian Kaspari (:sebastian; :pocmo)]

I've seen this mostly in the context of publishing locally for testing purposes. Sometimes this may show up in a PR but that should never end up in master. In this particular case I wonder how old the log statement is since we are way past 0.35.0 (maybe we have been using that suffix in the past, it's possible, but not recently).

Comment 12

[Johan Lorenzo [:jlorenzo]]

Oh, good point! :Oremj, do you know when this exception was raised?

Comment 13

[Jeremy Orem [:oremj]]

Attachment: metadata-exceptions.txt

Here are the tracebacks from the snapshots lambda for the last week.

Comment 14

[Jeremy Orem [:oremj]]

Attachment: default-snapshot-tracebacks.txt

Tracebacks for default metadata lambda.

Comment 15

[Jonathan Almeida [:jonalmeida]]

Clearing NI, sebastian already responded.

Comment 16

[Sebastian Kaspari (:sebastian; :pocmo)]

This has been causing serious issues the last days. Fenix Nightly was built from incompatible snapshots (since it didn't see some newer releases due to them missing in maven-metadata.xml) causing crashes. Our only workaround is to continue releasing snapshots until this intermittent (but somehow frequent recently) failure does not happen.

Comment 17

[Johan Lorenzo [:jlorenzo]]

I see 2 things in the logs:

1. We're having trouble to invalidate the cache. That looks like the root cause of this bug.
2. 0.35.0-TESTING is likely an artifact that we did upload on the bucket, but that we don't want to have. Is it possible to get its path?

What do you think :oremj?

Comment 18

[Jeremy Orem [:oremj]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #17)

I see 2 things in the logs:

1. We're having trouble to invalidate the cache. That looks like the root cause of this bug.
   I think the best thing here is to modify the script so that it doesn't total bail out of it can't invalidate the cache. It should just warn and keep going.

2. 0.35.0-TESTING is likely an artifact that we did upload on the bucket, but that we don't want to have. Is it possible to get its path?
   Let's modify the code to print the full path when we hit this error.

Comment 19

[Johan Lorenzo [:jlorenzo]]

Attachment: [maven-lambda] PR

Sounds good. I'm surprised these print() statements don't output anything in the logs, but anyway, I made the exception more explicit about what path is failing. r? :oremj

[1] Like https://github.com/mozilla-releng/maven-lambda/blob/402b33947e8138c9c9d280b56921bba9a0f0c285/maven_lambda/metadata.py#L47

Comment 20

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 9107169 [details] [review]
[maven-lambda] PR

It seems I cannot `r?`, but just `f?`

Comment 21

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 9107169 [details] [review]
[maven-lambda] PR

r+'d by oremj: at [1].

Jeremy, could you deploy these functions[2] on the production environment? I don't know the process, :autrilla used to take it from here. 

[1] https://github.com/mozilla-releng/maven-lambda/pull/20#pullrequestreview-313483939
[2] https://community-tc.services.mozilla.com/api/queue/v1/task/cTeotP8sTbe6HBbZ5pfF-Q/runs/0/artifacts/public/build/function.zip

Comment 22

[Jeremy Orem [:oremj]]

I've updated the code with the zip from comment 21.

Comment 23

[Johan Lorenzo [:jlorenzo]]

Thanks Jeremy! Sebastian, Jon, could you guys let us know whenever snapshots are broken again?

Comment 24

[Hiroyuki Ikezoe (:hiro)]

Is this issue being happening again?

I am seeing "Could not find any matches for org.mozilla.components:concept-toolbar:+ as no versions of org.mozilla.components:concept-toolbar are available."

Comment 25

[Jeremy Orem [:oremj]]

Looks like we are seeing another error now:

2019-11-09 11:04:07.171 'str' object has no attribute 'decode': AttributeError
Traceback (most recent call last):
  File "/var/task/metadata.py", line 52, in lambda_handler
    metadata_function=generate_release_maven_metadata
  File "/var/task/metadata.py", line 120, in craft_and_upload_maven_metadata
    bucket_name, folder, METADATA_BASE_FILE_NAME, metadata, content_type='text/xml'
  File "/var/task/metadata.py", line 283, in upload_s3_file
    invalidate_cloudfront(path=key)
  File "/var/task/metadata.py", line 291, in invalidate_cloudfront
    request_id = slugid.nice().decode('utf-8')  # nice() returns bytes by default
AttributeError: 'str' object has no attribute 'decode'

Comment 26

[Sebastian Kaspari (:sebastian; :pocmo)]

Yeah, we started to build version 22.0.0-SNAPSHOT yesterday and maven-metadata.xml is missing for a lot (or even all?) components.

Comment 27

[Johan Lorenzo [:jlorenzo]]

Ouch! The error reported in comment 25 blocks the generations of all metadata. Therefore maven-metadata.xmls haven't been updated since comment 22.

This is now fixed. Jeremy could you deploy https://community-tc.services.mozilla.com/api/queue/v1/task/YMVsB3sRR_OJdLdCyxd_3Q/runs/0/artifacts/public/build/function.zip ?

Upgrading priority because the system is now busted.

Comment 28

[Jeremy Orem [:oremj]]

I've deployed the new zip. Marking this as fixed, but let's reopen if there are still issues.