1493942 - maven.mozilla.org: Support SNAPSHOT repositories for android-components

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement)

Description

[Johan Lorenzo [:jlorenzo]]

SNAPSHOT repositories behave differently from releases repo[1]. :sebastian told me earlier today they would like to release android-components builds from the master branch to SNAPSHOT. Like bug 1464807 comment 3 states, this is an untested scenario for now.

On top of my head, I think we have some things to do to support it:
1. Generate a new set of credentials to let SNAPSHOT builds be uploaded only under SNAPSHOT directories. We should also lock the existing ones down.
2. Let beetmover support paths like
> /io/packagecloud/client/3.0.0-SNAPSHOT/client-3.0.0-20161003.234325-2.jar
3. Make sure maven-metadata.xml is correctly generated in this scenario. More details about the format in [1].

[1] https://blog.packagecloud.io/eng/2017/03/09/how-does-a-maven-repository-work/#snapshot-repositories

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [android-components] Enable snapshots.

Status update:
* this tracks the development of snapshots in android-components.
* so far, it generates and exposes each of the target.maven.zip per module correctly
* by slipping the "-SNAPSHOT" within the version, maven itself names and generates the files according to the default schema "timestamp-buildnumber". 

Since all modules are parallelized, each of them potentially runs on a different machine at a different timestamp, hence all the HHMMSS substring of the timestamp are all different. 

Possible solutions:
1. We leave this be and we rename the artifacts in beetmover, given a timestamp properly set at decision task and distributed to all tasks
2. We manage to do this elegantly via a maven plugin (see[1][2][3] for more) that takes its input from decision task. 

Leftovers until we take a decision here:
a) potentially separate logic with decision_task_pull_request and decions_task_master as more and more will differ given CoT
b) add beetmover tasks as well in the CI graphs
c) tweak an existing beetmover machine to take care of snapshots world
d) confirm artifacts are getting pushed to snapshots.maven.mozilla.org (staging instance)

Note to self: question for later (production): we'll need to figure out how do we tackle the two set of credentials of the same product within a beetmover instance. We could consider snapshots a different product but that's wrong. 

[1]: https://www.mojohaus.org/buildnumber-maven-plugin/usage.html#
[2]: https://julien.ponge.org/blog/an-experiment-in-maven-to-gradle-migration/
[3]: https://github.com/nemerosa/versioning

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

Note to self: @sebastian mentioned to me - I need to take care of it once the PR is ready.

"FYI: `-pCoverage` is tricky. We want that for running the tests and uploading the coverage reports; but I think we want to assemble the snapshot without that flag enabled (afaik it modifies the binary)
We could do "./gradlew clean -Pcoverage test && upload ... && ./gradlew clean assemble" ... but then we'd build twice"

Comment 4

[Johan Lorenzo [:jlorenzo]]

Attachment: [maven-lambda] Support SNAPSHOT artifacts

Was r+'d by mtabara at https://github.com/mozilla-releng/maven-lambda/pull/6#issuecomment-435462230.
Landed on master at https://github.com/mozilla-releng/maven-lambda/commit/8254c3b5d17797b88f59e1ce6e811a7c9d286773

Adrian, could you deploy 8254c3b5d17797b88f59e1ce6e811a7c9d286773 onto https://maven-snapshots.stage.mozaws.net/ ? We just want stage at the moment. We're not sure it's gonna work on prod.

Comment 5

[Johan Lorenzo [:jlorenzo]]

Attachment: [maven-lambda] Sort by timestamps in lieu of build numbers

r+'d by :mtabara at: https://github.com/mozilla-releng/maven-lambda/pull/7#pullrequestreview-178380189
Landed on master at: https://github.com/mozilla-releng/maven-lambda/commit/1ae7cd6bff9483bb23827fbcf42ad04cb2198684

Adrian, could you put this revision ^ on stage, instead?

Comment 6

[Adrian Utrilla [:autrilla]]

That revision has been deployed to snapshots stage.

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [beetmoverscript] Support for snapshots in beetmoverscript

Comment 8

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [puppet] Add dep snapshots support for beetmoverworkers

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 9031137 [details] [review]
[beetmoverscript] Support for snapshots in beetmoverscript

r+'ed by Jlorenzo in PR.

https://github.com/mozilla-releng/beetmoverscript/commit/1f1401540d559d998983651400056f3c2392560d

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 9031175 [details] [review]
[puppet] Add dep snapshots support for beetmoverworkers

r+'ed by Johan in PR.

https://github.com/mozilla-releng/build-puppet/commit/67141303a3ed94d660a7a29b5f83bd41e1e7ae93

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

Followed-up with https://github.com/mozilla-releng/beetmoverscript/pull/199 to fix a breakage in geckoview in-tree that was broken after snapshots PR landed.

Comment 12

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 9025353 [details] [review]
[android-components] Enable snapshots.

r+'ed with comments/nits by Johan, Mitch and Sebastian \o/

https://github.com/mozilla-mobile/android-components/commit/d0f1dda5530b649172bc1707fa450b3bdeed5829

Comment 13

[Mihai Tabara [:mtabara]⌚️GMT]

http://snapshots.maven.mozilla.org/ is now properly set-up and serving content. Mobile team successfully imported snapshots releases from this repo so I think it's safe to close this bug.

Let's reopen if more follow-ups are needed.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1a9d527301b0
[components] Bug 1493942 - enable snapshots releases.
https://hg.mozilla.org/integration/autoland/rev/710a62a2e00b
[components] Bug 1493942 - Fix rendering issue.
https://hg.mozilla.org/integration/autoland/rev/d64467858c26
[components] [ci skip] Bug 1493942 - fix syntax error.
https://hg.mozilla.org/integration/autoland/rev/43b1dc1a2cce
[components] Bug 1493942 - fix scope issues.
https://hg.mozilla.org/integration/autoland/rev/083de9a40b54
[components] Bug 1493942 - fix version from gradle.