Closed Bug 1433198 Opened 7 years ago Closed 6 years ago

Publish GeckoView on a maven-compatible repository

Categories

(Release Engineering :: General, defect, P2)

defect

Tracking

(firefox58 wontfix, firefox59 wontfix, firefox60 affected)

RESOLVED FIXED
Tracking Status
firefox58 --- wontfix
firefox59 --- wontfix
firefox60 --- affected

People

(Reporter: snorp, Assigned: jlorenzo)

References

(Blocks 1 open bug)

Details

(Whiteboard: [geckoview:klar:p2][releng:q32018])

Attachments

(1 obsolete file)

As part of the GeckoView build, a Maven repository is generated. This is a directory tree, the GeckoView AAR, and some metadata files. I would like to publish this tree under https://ftp.mozilla.org/pub/mobile/nightly/latest-mozilla-central-android-api-16/ with the intent that folks can then point Android Studio there and things will Just Work. If we decided to do stable releases of GeckoView, the same Maven repo would be available at the appropriate ftp URL. The Maven repo has already been added to the build artifacts, but it looks like TaskCluster (or something) flattens the tree, as you see each individual file listed in treeherder. I'm not sure if that's something that would need fixed first or not.
Whiteboard: [geckoview:m1]
Chris, can you help us with this?
Flags: needinfo?(catlee)
It looks like the artifacts do have their paths preserved. e.g. https://tools.taskcluster.net/groups/aeV9uG2cSmmwv50M1_WaYw/tasks/fn_4xGUVRQCZXf6XXywX1w/runs/0/artifacts So the idea is to have a maven/ sub-directory from the taskcluster artifacts published to our nightly S3 bucket as well.
Flags: needinfo?(catlee)
Depends on: 1436577
James, I filed bug 1436577 about publishing GeckoView AAR builds on ftp.mozilla.org. How do you want to share the Beta and Release GeckoView builds with downstream product teams? * A unified Maven repo that points to the latest Nightly, Beta, and Release builds? * Separate Maven repos for Nightly, Beta, and Release channels? * No Maven repo pointing to Beta or Release builds and just let other teams manually find and download the Beta and Release builds themselves on ftp?
Flags: needinfo?(snorp)
(In reply to Chris Peterson [:cpeterson] from comment #3) > James, I filed bug 1436577 about publishing GeckoView AAR builds on > ftp.mozilla.org. How do you want to share the Beta and Release GeckoView > builds with downstream product teams? > > * A unified Maven repo that points to the latest Nightly, Beta, and Release > builds? > * Separate Maven repos for Nightly, Beta, and Release channels? > * No Maven repo pointing to Beta or Release builds and just let other teams > manually find and download the Beta and Release builds themselves on ftp? I think option 2) will be the easiest here, and I believe we get that for free with this bug.
Flags: needinfo?(snorp)
Whiteboard: [geckoview:m1] → [geckoview:klar]
Currently the service ("beetmover") that uploads files from Taskcluster artifacts to S3 needs to be told explicitly which files to upload where as part of the task definition. e.g. look at the upstreamArtifacts section of https://tools.taskcluster.net/groups/WRmkUnJASjeJ3i5stsv_Iw/tasks/JQHh36NDQAinICj6YTQz8w/details We don't currently have a way to upload a subdirectory and all its contents to S3. Does the set of files that make up the maven repository change often, or is it relatively stable?
(In reply to Chris AtLee [:catlee] from comment #6) > Currently the service ("beetmover") that uploads files from Taskcluster > artifacts to S3 needs to be told explicitly which files to upload where as > part of the task definition. e.g. look at the upstreamArtifacts section of > https://tools.taskcluster.net/groups/WRmkUnJASjeJ3i5stsv_Iw/tasks/ > JQHh36NDQAinICj6YTQz8w/details > > We don't currently have a way to upload a subdirectory and all its contents > to S3. > > Does the set of files that make up the maven repository change often, or is > it relatively stable? I don't think the maven repository layout will change frequently, but snorp will know.
Flags: needinfo?(snorp)
(In reply to Chris AtLee [:catlee] from comment #6) > Currently the service ("beetmover") that uploads files from Taskcluster > artifacts to S3 needs to be told explicitly which files to upload where as > part of the task definition. e.g. look at the upstreamArtifacts section of > https://tools.taskcluster.net/groups/WRmkUnJASjeJ3i5stsv_Iw/tasks/ > JQHh36NDQAinICj6YTQz8w/details > > We don't currently have a way to upload a subdirectory and all its contents > to S3. > > Does the set of files that make up the maven repository change often, or is > it relatively stable? The directories and files change when the Gecko version changes. Right now it looks like: maven/org/mozilla/geckoview-default-x86/60.0a1/geckoview-default-x86-60.0a1.aar So obviously that '60' will change to a '61' after the next merge. Additionally, we'll want to have builds in the other branches at some point, so that version number may change more frequently with dot releases, etc. The architecture is also part of the name, so that will differ depending on the build (x86, arm, aarch64, etc).
Flags: needinfo?(snorp)
Is it relatively straightforward to zip up the maven repo before copying to the artifact dir? If the zipfile has a predictable name, we could potentially download it, extract it, and upload it with its current layout. Otherwise we may have to list out the contents of the maven repo before it's created, which can be fragile. I'm also guessing we want to upload new files and remove old files (e.g., on version change) without a hiccup in service? - In theory we could get a listing of the contents of the current latest/maven/ dir, push the new repo, then remove any remote file paths that don't exist locally. However, I think we don't have deletion perms for security reasons, so this may not be doable. In practice this means several months later the maven repo will have many old-versioned files inside. Also, I'm not sure what happens if someone tries to update from the dir during mid-upload. - Is it possible for us to upload the latest maven repo to a dated dir, and point a bouncer link or webapp to it? If so, Taskcluster indexes may also work, with a frontend that makes it look more like an ftp directory structure.
(In reply to Aki Sasaki [:aki] from comment #9) > Is it relatively straightforward to zip up the maven repo before copying to > the artifact dir? If the zipfile has a predictable name, we could > potentially download it, extract it, and upload it with its current layout. > Otherwise we may have to list out the contents of the maven repo before it's > created, which can be fragile. I do not think this would be difficult. > I'm also guessing we want to upload new files and remove old files (e.g., on > version change) without a hiccup in service? I haven't thought too much about this, but removing old files from a package repository is almost always a bad idea. > - In theory we could get a listing of the contents of the current > latest/maven/ dir, push the new repo, then remove any remote file paths that > don't exist locally. However, I think we don't have deletion perms for > security reasons, so this may not be doable. In practice this means several > months later the maven repo will have many old-versioned files inside. Also, > I'm not sure what happens if someone tries to update from the dir during > mid-upload. > > - Is it possible for us to upload the latest maven repo to a dated dir, and > point a bouncer link or webapp to it? If so, Taskcluster indexes may also > work, with a frontend that makes it look more like an ftp directory > structure. The more I think about using ftp.m.o as a Maven repository, the worse it sounds. Maven repositories collect a single XML file with all the metadata about every known version; we can't produce that collected artifact without doing work at upload time. We don't want to do that work as part of the build _or as part of the upload_. That tells me that we shouldn't do this at all: we should do https://bugzilla.mozilla.org/show_bug.cgi?id=1405129 properly, with a dependent Task Cluster task. This is morally equivalent to jchen's recent https://bugzilla.mozilla.org/show_bug.cgi?id=1426244.
We could also publish to a distinct S3 bucket solely for this purpose.
(In reply to Nick Alexander :nalexander from comment #10) > That tells me that we shouldn't do this at all: we should do > https://bugzilla.mozilla.org/show_bug.cgi?id=1405129 properly, with a > dependent Task Cluster task. This is morally equivalent to jchen's recent > https://bugzilla.mozilla.org/show_bug.cgi?id=1426244. The use case for this bug is that we want other apps and developers to be able to easily download and use GeckoView in their builds. For 2018, we only plan to support GeckoView for a few Mozilla apps: Fennec, Focus/Klar, and Crow. Is a Maven repo or Bintray overkill? Can we just copy the geckoview-*.aar files from the artifacts directory to the same archive.mozilla.org directory as the fennec-*.apk files? That was my suggestion in bug 1436577. Focus/Klar and Crow are not built out of mozilla-central. Their Nightly builds can pull the latest GeckoView Nightly from the archive.mozilla.org symlink URLs like https://archive.mozilla.org/pub/mobile/nightly/latest-mozilla-central-android-aarch64/en-US/. Focus/Klar and Crow beta builds might want to vendor their own copies of GeckoView in their trees. I don't know if a human would be in the loop for that. We don't have archive.mozilla.org symlink URLs for `latest-mozilla-beta-android-*` builds, but the GeckoView Beta builds will have fairly predictable URLs like https://archive.mozilla.org/pub/mobile/releases/57.0b3/android-aarch64/en-US/https://archive.mozilla.org/pub/mobile/releases/57.0b3/android-aarch64/en-US/fennec-57.0b3.en-US.android-aarch64.apk that a Focus/Klar or Crow build script could guess. Or we could just add `latest-mozilla-beta-android-*` symlink URLs.
(In reply to Chris Peterson [:cpeterson] from comment #12) > (In reply to Nick Alexander :nalexander from comment #10) > > That tells me that we shouldn't do this at all: we should do > > https://bugzilla.mozilla.org/show_bug.cgi?id=1405129 properly, with a > > dependent Task Cluster task. This is morally equivalent to jchen's recent > > https://bugzilla.mozilla.org/show_bug.cgi?id=1426244. > > The use case for this bug is that we want other apps and developers to be > able to easily download and use GeckoView in their builds. For 2018, we only > plan to support GeckoView for a few Mozilla apps: Fennec, Focus/Klar, and > Crow. Is a Maven repo or Bintray overkill? Perhaps it is, but I doubt it. More below. > Can we just copy the geckoview-*.aar files from the artifacts directory to > the same archive.mozilla.org directory as the fennec-*.apk files? That was > my suggestion in bug 1436577. We can, but it doesn't make it particularly convenient to use them. More below. > Focus/Klar and Crow are not built out of mozilla-central. Their Nightly > builds can pull the latest GeckoView Nightly from the archive.mozilla.org > symlink URLs like > https://archive.mozilla.org/pub/mobile/nightly/latest-mozilla-central- > android-aarch64/en-US/. > > Focus/Klar and Crow beta builds might want to vendor their own copies of > GeckoView in their trees. I don't know if a human would be in the loop for > that. We don't have archive.mozilla.org symlink URLs for > `latest-mozilla-beta-android-*` builds, but the GeckoView Beta builds will > have fairly predictable URLs like > https://archive.mozilla.org/pub/mobile/releases/57.0b3/android-aarch64/en-US/ > https://archive.mozilla.org/pub/mobile/releases/57.0b3/android-aarch64/en-US/ > fennec-57.0b3.en-US.android-aarch64.apk that a Focus/Klar or Crow build > script could guess. Or we could just add `latest-mozilla-beta-android-*` > symlink URLs. We can definitely put the AAR files in some well known location. That's a good use for ftp.m.o. However, that's not what Maven consumers do: Maven downloads metadata (a root XML file and a bunch of POM files) to describe what versions are available before choosing the artifact. When I did this "by hand" without Gradle, I used Ivy, since the metadata was easier to work with. (I forget why.) You can see where I remove all that "by hand" stuff in https://bugzilla.mozilla.org/show_bug.cgi?id=1268944, if it's interesting. I just think that FTP doesn't actually serve the real purpose, which is "make it easy to consume GeckoView". That's true for Mozilla (we'd need to hard code AAR locations) and for external customers (who will never want to give up real version pinning using ftp.m.o symlinks).
We need this but it doesn't technically block Klar 1.0
Priority: -- → P2
I'm taking this bug while I am figuring out our Maven requirements.
Assignee: nobody → cpeterson
We need strong compatibility with the way modern Android apps are being put together. Some hints about that are on: https://developer.android.com/studio/build/dependencies.html and specifically about repositories: https://developer.android.com/studio/build/dependencies.html#remote-repositories It looks like a Maven repo is a good approach. We could also consider to publish to JCenter. That would mean less configuration for developers since that is the default place where AS looks. (I don't think specifying a Mozilla repo location is a big deal - as long as we are compatible with how the modern Android tooling works - which basically is Android Studio)
Depends on: 1443822
Assigning to catlee because he is experimenting with a Maven repo hosted on Bintray. This bug is P2 because it does not block shipping Klar+GeckoView.
Assignee: cpeterson → catlee
Summary: Publish GeckoView Maven repo on ftp.mozilla.org → Publish GeckoView Maven repo on Bintray/JCenter or S3
[geckoview:klar:p2] because this is not a Klar+GeckoView blocker, but we would like to publish a Maven repo on Bintray/JCenter for Focus/Klar's convenience soon.
Summary: Publish GeckoView Maven repo on Bintray/JCenter or S3 → Publish GeckoView Maven repo on Bintray/JCenter
Whiteboard: [geckoview:klar] → [geckoview:klar:p2]
Whiteboard: [geckoview:klar:p2] → [geckoview:klar:p2][releng:q22018]
Assignee: catlee → jlorenzo
QA Contact: catlee
Depends on: 1464807
In a risk analysis done earlier this month, we decided to host a Maven repository on an S3 bucket we own. Enforcing signatures on jar dependencies is not a well-known process and can be easily deactivated. I didn't find any project that does it on Gradle, so I created [1]. As a point of data, Google doesn't sign their binaries either [2]. They do host their own repo, though. I'm renaming this bug to reflect the new target. [1] https://github.com/JohanLorenzo/gradle-jarsigner [2] https://dl.google.com/dl/android/maven2/index.html
Summary: Publish GeckoView Maven repo on Bintray/JCenter → Publish GeckoView on a maven-compatible repository
Depends on: 1470942
STATUS UPDATE I'm making changes in the taskgraph to add a new beetmover task. Beetmover is what uploads our artifacts to an S3 bucket. I'm hitting an issue: the beetmover task needs to know the exact path of artifacts to upload. At the moment path have following pattern (eg [1]): > public/android/maven/org/mozilla/geckoview-${branch}-${platform}/${version}/geckoview-${branch}-${platform}-${version}-javadoc.jar I think it would be simpler if we just had: > public/android/javadoc.jar I know the former pattern is made so that the task artifacts simulate a Maven repository. Thanks to the beetmover task, we won't need this anymore. I looked into how we could change the artifact location and I'm unsure what to do. HOW ARTIFACTS ARE PLACED THIS WAY? The aar, javadoc, source packages already exist at various locations on the disk. They have simple names that don't depend on the branch/platfor/version. It should be fairly straightforward to declare them as artifacts to upload. The gradle task "uploadArtifact" moves everything at the aforementioned path. This line [2] outputs on-disk. It also generates the md5 and sha1 files used by maven clients. I'm concerned about them. :nalexander, is there an easy way to generate them with Gradle? I found [3], but it doesn't support sha1. I don't know if the maven echosystem has a strong assertion on the sha1 file or if we can just upload a sha256 one. [1] https://tools.taskcluster.net/groups/ZeVkONGGR8-6VBGYs152-w/tasks/ewbtOCGHTTui6mI_xMeOpg/runs/0/artifacts [2] https://searchfox.org/mozilla-central/rev/ed2763bea882619ccb48b0aecc54e523d2bdd2ae/mobile/android/geckoview/build.gradle#292 [3] https://github.com/gradle/gradle-checksum
Flags: needinfo?(nalexander)
(In reply to Johan Lorenzo [:jlorenzo] from comment #22) > STATUS UPDATE > > I'm making changes in the taskgraph to add a new beetmover task. Beetmover > is what uploads our artifacts to an S3 bucket. I'm hitting an issue: the > beetmover task needs to know the exact path of artifacts to upload. At the > moment path have following pattern (eg [1]): > > > public/android/maven/org/mozilla/geckoview-${branch}-${platform}/${version}/geckoview-${branch}-${platform}-${version}-javadoc.jar > > I think it would be simpler if we just had: > > > public/android/javadoc.jar > > I know the former pattern is made so that the task artifacts simulate a > Maven repository. Thanks to the beetmover task, we won't need this anymore. > I looked into how we could change the artifact location and I'm unsure what > to do. We can accommodate this at the Gradle level pretty easily. However, _something_ has to name files in a Maven compatible layout. You're saying that it's awkward to have Beetmover accommodate files in a Maven compatible layout. But can Beetmover layout files itself, taking into account the non-trivial version numbering scheme we have in place? (Hopefully the answer is yes and we already rename say target.apk into fennec-$platform-$version.apk or whatever.) > HOW ARTIFACTS ARE PLACED THIS WAY? > > The aar, javadoc, source packages already exist at various locations on the > disk. They have simple names that don't depend on the > branch/platfor/version. It should be fairly straightforward to declare them > as artifacts to upload. Yes. In fact, it looks like the Javadoc (at least) already has a simple name, see https://searchfox.org/mozilla-central/source/mobile/android/mach_commands.py#458-459 However, you really need to have the POM file as well, and that is only produced by an `uploadArchives` task. So we can't remove that task entirely (without teaching Beetmover or some other tool lots about Maven, which I think is foolish.) > The gradle task "uploadArtifact" moves everything at the aforementioned > path. This line [2] outputs on-disk. It also generates the md5 and sha1 > files used by maven clients. I'm concerned about them. :nalexander, is there > an easy way to generate them with Gradle? I found [3], but it doesn't > support sha1. I don't know if the maven echosystem has a strong assertion on > the sha1 file or if we can just upload a sha256 one. I am not aware of a _requirement_ for sha1 in the Maven ecosystem, and would personally prefer to use only sha256 if it "just works" out of the box. Easy to test! > [1] > https://tools.taskcluster.net/groups/ZeVkONGGR8-6VBGYs152-w/tasks/ > ewbtOCGHTTui6mI_xMeOpg/runs/0/artifacts > [2] > https://searchfox.org/mozilla-central/rev/ > ed2763bea882619ccb48b0aecc54e523d2bdd2ae/mobile/android/geckoview/build. > gradle#292 > [3] https://github.com/gradle/gradle-checksum There is an interaction between `uploadArchives` and jchen's |mach android archive-geckoview| task, which uploads the Javadoc to GitHub (https://mozilla.github.io/geckoview/javadoc/mozilla-central/). I don't know if that task requires the special naming. If it doesn't, I think you want to apply the following patch: diff --git a/mobile/android/geckoview/build.gradle b/mobile/android/geckoview/build.gradle --- a/mobile/android/geckoview/build.gradle +++ b/mobile/android/geckoview/build.gradle @@ -267,23 +267,18 @@ android.libraryVariants.all { variant -> configureLibraryVariantWithJNIWrappers(variant, "Generated") } apply plugin: 'maven' uploadArchives { repositories.mavenDeployer { pom.groupId = 'org.mozilla' - pom.artifactId = "geckoview-${mozconfig.substs.MOZ_UPDATE_CHANNEL}-${mozconfig.substs.ANDROID_CPU_ARCH}" - - if (mozconfig.substs.RELEASE_OR_BETA == 'true') { - pom.version = mozconfig.substs.MOZ_APP_VERSION - } else { - pom.version = getAppVersionWithoutMilestone() + "." + getBuildId() - } + pom.artifactId = 'geckoview' + pom.version = 'unversioned' pom.project { licenses { license { name 'The Mozilla Public License, v. 2.0' url 'http://mozilla.org/MPL/2.0/' distribution 'repo' } which in turn produces the following output: ⋊> ~/M/gecko find /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla 09:56:24 /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/maven-metadata.xml /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/unversioned /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/unversioned/geckoview-unversioned-sources.jar.sha1 /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/unversioned/geckoview-unversioned.pom.md5 /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/unversioned/geckoview-unversioned.pom /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/unversioned/geckoview-unversioned-javadoc.jar.md5 /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/unversioned/geckoview-unversioned.aar.sha1 /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/unversioned/geckoview-unversioned-sources.jar.md5 /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/unversioned/geckoview-unversioned-javadoc.jar /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/unversioned/geckoview-unversioned-javadoc.jar.sha1 /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/unversioned/geckoview-unversioned.pom.sha1 /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/unversioned/geckoview-unversioned-sources.jar /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/unversioned/geckoview-unversioned.aar /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/unversioned/geckoview-unversioned.aar.md5 /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/maven-metadata.xml.md5 /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/maven-metadata.xml.sha1 Easy peasy, lemon squeezy. jchen can say more about the uploading to GitHub.
Flags: needinfo?(nalexander)
Flags: needinfo?(nchen)
(In reply to Nick Alexander :nalexander from comment #23) > > There is an interaction between `uploadArchives` and jchen's |mach android > archive-geckoview| task, which uploads the Javadoc to GitHub > (https://mozilla.github.io/geckoview/javadoc/mozilla-central/). I don't > know if that task requires the special naming. If it doesn't, I think you > want to apply the following patch: > The "geckoview-docs" task only depends on one of the "geckoview:javadoc" tasks, so I don't think that affects things.
Flags: needinfo?(nchen)
Bug 1433198 - part 2: Declare maven zip archive as only maven artifacts r=jchen
Comment on attachment 8990731 [details] Bug 1433198 - part 1: Craft zip archive of maven artifacts r=jchen Jim Chen [:jchen] [:darchons] has approved the revision. https://phabricator.services.mozilla.com/D2031
Attachment #8990731 - Flags: review+
Whiteboard: [geckoview:klar:p2][releng:q22018] → [geckoview:klar:p2][releng:q32018]
Comment on attachment 8990731 [details] Bug 1433198 - part 1: Craft zip archive of maven artifacts r=jchen I'm gonna split this patch into 2: 1. Create the zip archive and expose it in the taskcluster artifacts, along the other maven artifacts. 2. Remove the other maven artifacts. I need 1. in bug 1470942, so I'll move it there. I'll use this current bug to track the clean step (2.). I expect the clean up to happen after projects depending on geckoview moved to the maven repo.
Attachment #8990731 - Attachment is obsolete: true
Depends on: 1484932
Depends on: 1486419
Depends on: 1486831
Depends on: 1489132
Depends on: 1489486
Blocks: 1494913
Depends on: 1496262
I forgot to close this bug. Geckoview has been published to maven.mozilla.org since August 30th, 2018 : https://maven.mozilla.org/?prefix=maven2/org/mozilla/geckoview/geckoview-nightly-armeabi-v7a/63.0.20180830100125/
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: