Closed Bug 1443873 Opened 7 years ago Closed 5 years ago

update nightly, beta, rc, release, devediton builds buildhub.json file

Categories

(Release Engineering :: Release Automation, enhancement)

Product:
Release Engineering
Component:
Release Automation
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox-esr60 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox-esr60 --- fixed

People

(Reporter: kmoir, Assigned: lguo)

References

Details

(Keywords: leave-open)

Attachments

(1 file)

adding support for buildhub.json
58 bytes, text/x-github-pull-request
mtabara
: review+
Details | Review
Buildhub is a service that services engineering maintains that collects meta data on various release artifacts that is then consumed by other services. Currently this is populated by extracting artifacts and parsing files and thus is very brittle and fragile. https://mozilla-services.github.io/buildhub/ Services engineering would like to make the process to populate buildhub more robust. Metadata will generated during the build and release process to provide the information that can be uploaded to buildhub. The first part of this work will be done by the build team. In bug 1442306, a buildhub.json file will be populated with data that matches the schema here https://bugzilla.mozilla.org/show_bug.cgi?id=1442306#c0 as part of the build process. The timeframe for this work to be complete is 2-3 weeks. The second part of this work is for the releng team. The buildhub.json file needs to be updated during the release promotion of ci artifacts to release artifacts. The buildhub.json file needs to be updated to reflect things such as the current size of the artifacts, and the locales associated with the build. The are more examples in the schema of data that needs to be updated/added. Peterbe is the contact for schema questions. As well, buildhub.json needs to be sent to buildhub. Notes from the meeting on the releng side of this work Part of release automation publish JSON data to an endpoint Current build process is lots of separate tasks, compared to old model which was one monolithic task Build should produce artifacts that Buildhub needs This way Buildhub is not a secondary consume of the data and have to chase / coordinate changes Catlee suggestion: do what symbols does. TC will has a task that POSTs the JSON into buildhub and if that fails it’ll fail the job Catlee is not worried because it’ll be a leaf on the build TC graph If it fails, releng doesn’t care Product Delivery will deal with failures in the task. +1 … releng doesn’t really care Initially we can make it a non-fatal task. Eventually it’ll break the build when we work out the edge case bugs The build will continue even if the buildhub task fails How does catlee feel about Prod Delivery adding files to archive - don’t care :) The file can exist as a taskcluster artifact … doesn’t have to exist in archive.mozilla.org It’ll be a lot easier to scape / index / find this data than to walk/scrape through archive.mozilla.org Schema / example: JSON Schema https://buildhub.prod.mozaws.net/v1/buckets/build-hub/collections/releases/records/firefox_nightly_2018-02-20-10-34-56_60-0a1_win64_az How do we deal with schema changes? catlee - make the data early in the build and validate it against a schema Product delivery would be a shared consumer / user of the build data schema
I've got patches up for review on bug 1442306, and the builds on my latest try push have buildhub.json artifacts: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d8a81f06ca93dc2bcc029e86af2183c29ba70ecc If you look at one of them (this is the linux opt build): https://taskcluster-artifacts.net/WhdAjkTmS-OcmNpUWWgUWA/0/public/build/buildhub.json you can see that we'll need some other system to properly set the `download.url` key, and also ensure that it updates the `download.{date,size}` keys after signing.
mihai: do you think this is something Lisa could take a look at? It involves artifacts so it's .... somewhat related :)
Flags: needinfo?(mtabara)
Flags: needinfo?(lguo)
Really sorry for delays in answering here, somehow it slipped my radar. I still have to wrap-up some other Shipit v2 and flatpacks work this week but this is something that definitely Lisa could look into since she's touched beetmoverscript already and ramping up in in-tree development. I'll grab more context today and I'll be back with a more realistic ETA for this. Apologies again for delays, mea culpa. P.S. Leaving the NI here until I come up with that ETA.
Lisa, Kim and I met today to discuss this. To sum-up, it sounds like the story is like this: a) Services team needs a more reliable way to feed their Buildhub service rather than extracting and parsing various artifacts. To this matter, they need some centralized files to encompass the metadata needed. b) Build team already added the code to generate this at build time per platform / locale so their job is done c) it's releng's job now to update that builhub.json file with the proper size (and potentially other fields) and publish it accordingly (on-push graphs are different than nightly/release graph as branding + signing affects the builds, hence the final file size and such) Implementation details aside, there's still a few questions for which we need answers: 1. Looking at one of these buildhub.json files[1], should we be concerned about any of the fields except the "download" :: size s if any? 2. Are there other release artifacts that one cares about or is it just the main installer per platform? (e.g. target.tar.bz2, target.dmg and target.installer.exe) 3. Do we want RelEng to publish this to Buildhub via API or something similar or are we happy with placing that (updated) particular buildhub.json file(s) under specific locations in archive.mozilla.org (per each type of release: nightly, release, beta,etc) from where existing parsers / scrapers can get them? 4. What's the story for l10n builds? Do we need a buildhub.json per each locale (per each platform) as well, similar to what we have for en-US? [1]: https://taskcluster-artifacts.net/T7TASdAVQX2ZQW7dIsQ7kQ/0/public/build/buildhub.json Sent an email with all these questions to :peterbe. Once we have all details scoped out, we can talk an implementation plan + ETA. Clearing NI for me and Lisa for now as we're pending an answer before getting to work.
Flags: needinfo?(mtabara)
Flags: needinfo?(lguo)
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #4) > 1. Looking at one of these buildhub.json files[1], should we be > concerned about any of the fields except the "download" :: size s if > any? Scratch this. I did a bit of digging here[1] and here[2] and found the buildhub.json file schema[3] that explains each of these fields. At a second glance at an example task[4], I think we'll need to update the following: target => channel download => size download => date download => url ... since all these fields change for the installers during the release process (due to branding changes and signing). Questions 2-4 remain valid. [1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1442306#c0 [2]: https://mozilla-services.github.io/buildhub/?products[0]=firefox [3]: https://github.com/mozilla-services/buildhub/blob/master/jobs/buildhub/initialization.yml#L71-L198 [4]: https://taskcluster-artifacts.net/T7TASdAVQX2ZQW7dIsQ7kQ/0/public/build/buildhub.json
1. buildhub.json fields that should be changed: download.size download.date download.url target.channel target.locale 2. Just the main installers per platform. 3. Put them on archive.mozilla.org, preferably in the same directory as the main installer 4. Yes one buildhub.json per locale. `target.locale` should also be updated accordingly.
Thanks for providing us with all this information, I think we have everything we need. Lisa is going to tackle this. We're likely going to finish before the All-hands.
Assignee: nobody → lguo
Status: NEW → ASSIGNED
Hello, intern here I want to clarify where to grab these values from to update 'download.size' and 'download.url' I'm assuming 'download.date' can just be taken from the system's current datetime value. As for 'target.channel' and 'target.locale', could you please clarify why these need to change? Thank you.
Flags: needinfo?(kmoir)
Are the values for 'download.size', 'download.url' ,and 'download.date' values in the final graph for the build promotion for the build artifacts in different variable names? I don't know. Here are task group ids for a recent beta build promotion (https://github.com/mozilla-releng/releasewarrior-data/blob/master/archive/firefox/firefox-beta-61.0b9.md) from release warrior, the tool that is used to promote builds, probably Mihai mentioned it). I would look at the promote graph and see what values are in there for the build artifacts or talk to Mihai) promote PJGtuUacT_icCHaIqk1AfQ push UjBSFmg_QCGSggSZumqqzw ship LA10EsfDRC6I76PwJhBK8w It might be worth confirming with someone from the buildhub team like Peter that these reflect the values you are looking for. The target.channel changes when the build is promoted. For instance if a build is promoted to a release build it will have a different channel than the original artifact. Some artifacts have specific locales associated with them, ie. target.locale
Flags: needinfo?(kmoir)
I'm out of my depth here but the least I can tell is that we know what we want for those fields. (pardon me if this is patronizing/obvious) * download.url - The full public URL to the file on archive.mozilla.org (front for S3 bucket) that leads to the binary file * download.size - The size of that file. * download.date - The datetime the file was uploaded to S3.
Just a summary of what I understand the parameters to be. So you want to change the fields in this file https://taskcluster-artifacts.net/T7TASdAVQX2ZQW7dIsQ7kQ/0/public/build/buildhub.json into fields similar to the ones here https://buildhub.prod.mozaws.net/v1/buckets/build-hub/collections/releases/records/firefox_nightly_2018-06-06-10-01-33_62-0a1_win64_en-us where the url is a full path instead of a file name. And I don't believe the target locale and channel values need to be touched.
Flags: needinfo?(peterbe)
Yes! That's right. Basically, download.url needs to be a real full URL that you can run `curl --head` on. And download.date should ideally be the datetime when the file was uploaded but I doubt that date is critical so if it's easier I guess you could just set it to "now" when you figure out the full URL. Note, *technically* Buildhub could figure out the download.size and download.mimetype *from* the URL. E.g. ▶ curl --head https://archive.mozilla.org/pub/firefox/nightly/2018/06/2018-06-06-10-01-33-mozilla-central/firefox-62.0a1.en-US.win64.installer.exe ... HTTP/1.1 200 OK Content-Type: application/octet-stream Content-Length: 41034112 Date: Wed, 06 Jun 2018 15:21:10 GMT ... However, I think the overall architecture is that Buildhub is going to NOT get involved in the content of the JSON but just suck it up, throw it in a database and make it searchable.
Flags: needinfo?(peterbe)
Summary: update nightly, beta, rc, release, devediton builds buildhub.json file and submit to buildhub → update nightly, beta, rc, release, devediton builds buildhub.json file
(In reply to Lisa Guo [:lguo] EST from comment #11) > > And I don't believe the target locale and channel values need to be touched. I *think* the way it's going to work is that the "repacks" (or whatever it's called) generates the "firefox-installer-fr.exe" for the French locale in a separate step. So you end up with: { "target": { "locale": "fr", ... }, "download": { "url": "firefox-installer-fr.exe", "date": "", "size": "", }, ... } And your job is to turn that into: { "target": { "locale": "fr", ... }, "download": { "url": "https://archive.mozilla.org/path/to/where/it/was/uploaded/firefox-installer-fr.exe", "date": "2018-06-13T12:00:00.123546", "size": 123456789, }, ... } I.e. that locale and channel business happens separately and distinctly from this. Right Kim, Ted?
To sum this up: * builds and l10n repacacks jobs are creating a builhub.json file containing a bunch of information about that platform specific installer file For example: 1. For Linux: * sample of builhub.json file https://queue.taskcluster.net/v1/task/exiQeFApTcm0N0jtRlBiWQ/runs/0/artifacts/public/build/buildhub.json * the targetted file is target.tar.bz2 2. For Windows: * sample of buildhub.json file https://queue.taskcluster.net/v1/task/TW04RbxeSXGKTtGY6qJ70w/runs/0/artifacts/public/build/buildhub.json * the targetted file is target.installer.exe 3 For MacosX: * sample of buildhub file https://queue.taskcluster.net/v1/task/EueIeVmkR2m2ZJhp8u_AJg/runs/0/artifacts/public/build /buildhub.json * the targetted file is target.dmg These files are already generated at build time and describe the installer files properies (size, channel, locale, etc). However, some of the fields are yet to be completed (like final location, etc). For this reason we need to update these fields later in the Release Automation graph, when we have all these bits of information at hand. Best place to update them is in Beetmover (bit mover - component that takes artifacts after signing/repackaging step and pretty names it and transfer to S3). So in order to solve this, we need to: a) (in-tree changes) make sure the builhub.json file is present in upstreamArtifacts for beetmover jobs so that we download it in the beetmoverworkers. This means we need to tweak beetmover* transforms to include this particular file as a dependency from the build jobs b) (beetmoverscript changes). Enrich beetmoverscript[1] to handle this new operation. First things that are coming to my mind are: * a buildhub.json schema validation (just in case this changes, we should fail early) * if buildhub.json file is present, load it as dict * read information on the installer file and update the buildhub dict with the new values * submit the buildhub file to its proper location in S3 along the installer file (e.g. buildhub.json should live near its target.dmg counterpart here [2]) The values we need to update are: * download size - we already compute this for checksums + balrog manifest so we have this at hand in beetmover here[3] * download.date - we can just defer to utcnow() or w\e as :peterbe mentioned. It'll be relative to miliseconds to the real one when it gets updated to S3 so I think that's fine * download.url - we can get this from the manifest since we know where to update these files * target.channel - looking at these files across nightly/beta/release they seem to be already set (I blame some of the branding changes maybe that get propagated in the build configs) but we can always define a new mapping in beetmover to map something like mozilla-central => "nightly", "mozilla-beta" => "beta", etc * target.locale - we have this information already in the beetmover task payload Happy to provide more input if needed. [1]: https://github.com/mozilla-releng/beetmoverscript [2]: https://archive.mozilla.org/pub/firefox/candidates/61.0b11-candidates/build1/mac/en-US/ [3]: https://github.com/mozilla-releng/beetmoverscript/blob/master/beetmoverscript/script.py#L288
There's two things that worry me though. a) I see l10n jobs in nightly graphs *not* having the buildhub.json file generated in artifacts. * android l10n jobs are missing this[1] * win64 l10n jobs are missing this[2] * linux l10n jobs are missing this[3] @ted: just to confirm, in order to obtain the same artifact for l10n jobs, we'd have to tweak our releng l10n scripts or is this something the Build team can help with? If former, I might need to rope in :Callek as he's our team's expert on this topic b) I'm not sure what's the naming convention for these files, for nightly in particular. For beta and any other type of releases, things are relatively simpler as we have dedicated firectories for installers. E.g. we could place a file named "buildhub.json" in here[4] along with the "Firefox 61.0b11.dmg" file. Or in here[5] for the en-US counterpart . @peterbe: However, for nightly the story is slightly different as all files live in the same directory[6], hence we can't name them "buildhub.json" as they're going to overlap. Should we opt instead for a prefix (similar to what other artifacts have) like: {product}-{version}.{locale}.{platform}.buildhub.json? Thank you both in advance. [1]: https://tools.taskcluster.net/groups/XANmaL3gRIW937eP-9t93A/tasks/Mcv5XSqMS9OZFSLZkuiOUw/runs/0/artifacts [2]: https://tools.taskcluster.net/groups/HoH-VrfZQRSRmoNxKE6qDQ/tasks/SnAcVCM8SCaxA09z_vkBWQ/runs/0/artifacts [3]: https://tools.taskcluster.net/groups/HoH-VrfZQRSRmoNxKE6qDQ/tasks/GMiMmZAIR0WwRVlAXrvIGA/details [4]: https://archive.mozilla.org/pub/firefox/candidates/61.0b11-candidates/build1/mac/en-GB/ [5]: https://archive.mozilla.org/pub/firefox/candidates/61.0b11-candidates/build1/mac/en-US/ [6]: https://archive.mozilla.org/pub/firefox/nightly/2018/06/2018-06-01-10-01-02-mozilla-central/
Flags: needinfo?(ted)
Flags: needinfo?(peterbe)
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #15) > > @peterbe: > However, for nightly the story is slightly different as all files live in > the same directory[6], hence we can't name them "buildhub.json" as they're > going to overlap. Should we opt instead for a prefix (similar to what other > artifacts have) like: > {product}-{version}.{locale}.{platform}.buildhub.json? > Our intention was to be able to do something like this: for file in sqs_queue.consume(): if file.endswith('/buildhub.json'): do_something_with(file) else: ignore() But we didn't think about this potential problem but we *could* change the code to be this instead: for file in sqs_queue.consume(): if file.endswith('buildhub.json'): do_something_with(file) else: ignore() That means it would match on "root/some/dir/buildhub.json" and "root/bigdir/firefox-62.0a1.en-US.win32.buildhub.json" I think that would work. I'd just need to update some of my docs :) Would be nice to hear what Ted thinks about that.
Flags: needinfo?(peterbe)
Note to self: we're going to need to add "buildhub.json" in the beetmover jobs deps, something similar to what we have for "balrog_props.json" that just got removed. I'm thinking we can use some of that in-tree code from transforms to avoid wasting too much time on that. The bug 1424482 tracked that.
See Also: → 1424482
specific beetmoverscript implementation question for Mihai: for a beetmover task like this one https://queue.taskcluster.net/v1/task/SzOvgYtkRxasaEJW_EZT1Q gotten from this task https://tools.taskcluster.net/groups/VIWyOqTaQZKWo5Fuj0MgaQ/tasks/SzOvgYtkRxasaEJW_EZT1Q/details will the upstreamArtifacts look like this, with the addition of the buildhub.json artifact? "upstreamArtifacts": [ { "locale": "ach", "paths": [ "public/build/ach/buildhub.json" ], "taskId": "UTXVV9wITluSTh4gcqYENw", [taskId of the build or l10n job] "taskType": "build" }. { "locale": "ach", "paths": [ "public/build/ach/target.tar.bz2", "public/build/ach/target.tar.bz2.asc" ], "taskId": "UTXVV9wITluSTh4gcqYENw", "taskType": "signing" }, { "locale": "ach", "paths": [ "public/build/ach/target.complete.mar" ], "taskId": "ORiSdfLHTjOBUcncz6Wglw", "taskType": "repackage" }, { "locale": "ach", "paths": [ "public/build/ach/target-61.0b9.partial.mar", "public/build/ach/target-61.0b7.partial.mar", "public/build/ach/target-61.0b8.partial.mar" ], "taskId": "HvyRE3kOQpObz93OrY1LbA", "taskType": "signing" } ],
Flags: needinfo?(mtabara)
(In reply to Lisa Guo [:lguo] EST from comment #18) > specific beetmoverscript implementation question for Mihai: > > for a beetmover task like this one > https://queue.taskcluster.net/v1/task/SzOvgYtkRxasaEJW_EZT1Q gotten from > this task > https://tools.taskcluster.net/groups/VIWyOqTaQZKWo5Fuj0MgaQ/tasks/ > SzOvgYtkRxasaEJW_EZT1Q/details > > will the upstreamArtifacts look like this, with the addition of the > buildhub.json artifact? > > "upstreamArtifacts": [ > { > "locale": "ach", > "paths": [ > "public/build/ach/buildhub.json" > ], > "taskId": "UTXVV9wITluSTh4gcqYENw", [taskId of the build or l10n > job] > "taskType": "build" > }. > { > "locale": "ach", > "paths": [ > "public/build/ach/target.tar.bz2", > "public/build/ach/target.tar.bz2.asc" > ], > "taskId": "UTXVV9wITluSTh4gcqYENw", > "taskType": "signing" > }, > { > "locale": "ach", > "paths": [ > "public/build/ach/target.complete.mar" > ], > "taskId": "ORiSdfLHTjOBUcncz6Wglw", > "taskType": "repackage" > }, > { > "locale": "ach", > "paths": [ > "public/build/ach/target-61.0b9.partial.mar", > "public/build/ach/target-61.0b7.partial.mar", > "public/build/ach/target-61.0b8.partial.mar" > ], > "taskId": "HvyRE3kOQpObz93OrY1LbA", > "taskType": "signing" > } > ], Precisely! ;-)
Flags: needinfo?(mtabara)
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #15) > There's two things that worry me though. > > a) I see l10n jobs in nightly graphs *not* having the buildhub.json file > generated in artifacts. Correct. This is bug 1459302. (l10n repacks are a PITA to make changes to.)
Flags: needinfo?(ted)
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #20) > (In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #15) > > There's two things that worry me though. > > > > a) I see l10n jobs in nightly graphs *not* having the buildhub.json file > > generated in artifacts. > > Correct. This is bug 1459302. (l10n repacks are a PITA to make changes to.) Thanks a lot for confirming this. We'll proceed with adding the builhub functionality for en-US for now and we'll follow-up with the l10n counterpart whenever that piece is ready, to make sure we're not blocking anyone.
See Also: → 1467845
First staging release with builhub.json testing is here[1]. Sounds like some of the buildhub.json worked smoothly and got transfered under proper S3 directories, e.g. [2]. At first glance, we need to fix two quick things: a) size is wrong. It should be the size of the corresponding installer (macosx beetmover job will have the dmg, windows the exe counterpart, etc b) url misses the prefix. That's available in the context.config We're getting closer with this, nice! [1]: https://tools.taskcluster.net/groups/HX1dc27sRl610fPnTLtq6Q [2]: http://ftp.stage.mozaws.net/pub/firefox/candidates/61.0b15-candidates/build8/linux-i686/en-US/buildhub.json
c) the target.version is "61.0" but judging from the URL it's "61.0b15"
Lisa pushed a bunch of fixes in beetmoverscript side. I landed that in staging environment and triggered a new staging release. Let's see how this[1] one goes. [1]: https://tools.taskcluster.net/groups/CujlHlj8Rg2YhWkfIPu7YQ
This is how the buildhub.json file looks in the latest staging release, after the fixes http://ftp.stage.mozaws.net/pub/firefox/candidates/61.0b15-candidates/build9/win64/en-US/buildhub.json Is it okay that the url doesn't work because of the space? Does all other fields look okay?
Flags: needinfo?(peterbe)
Flags: needinfo?(mtabara)
I'm slightly worried we haven't touched the target.channel and target.locale but they seem to be correctly defined as part of the build phase. The other fields look fine to me. As to the space there, we might need to encode it, we might not, I'll defer to :peterbe.
Flags: needinfo?(mtabara)
Paths and query parameters in `url` should be encoded. It'd be onerous for consumers of the data to split apart the url and encode the relevant parts after.
(In reply to Benson Wong [:mostlygeek] from comment #28) > Paths and query parameters in `url` should be encoded. It'd be onerous for > consumers of the data to split apart the url and encode the relevant parts > after. Sounds about right. @lisa: let's encode those urls and respin another beetmoverscript + staging release and see if that looks better. We're getting closer, this is exciting.
Not sure the spaces in the URL need to be encoded actually. Since we're not using it like a HTML document would. These lines work:: import requests url = 'http://ftp.stage.mozaws.net/pub/firefox/candidates/61.0b15-candidates/build9/win64/en-US/buildhub.json' download_url = requests.get(url).json()['download']['url'] r = requests.head(download_url) assert r.status_code == 200 I.e. it can happily reach the binary download even with spaces.
Right, that's what I wanted to verify - so is what we have now good? What's the consensus here? Thoughts @Benson
We should encode all unsafe characters in URLs. We shouldn't expect clients to be smart. If clients want to make a URL prettier for display but that's not our concern. Our goal is fidelity of data. RFC 1738 states unambiguously: Unsafe: Characters can be unsafe for a number of reasons. The space character is unsafe because significant spaces may disappear and insignificant spaces may be introduced when URLs are transcribed or typeset or subjected to the treatment of word-processing programs. ... All unsafe characters must always be encoded within a URL. For example, the character "#" must be encoded within URLs even in systems that do not normally deal with fragment or anchor identifiers, so that if the URL is copied into another system that does use them, it will not be necessary to change the URL encoding.
For reference, here is the code[1] that generates the url; I just added the url encoding method. Is this what you need, or are you only looking to encode the whitespace? (ie. the space in "Firefox Setup 61.0b15.exe") [1]: https://github.com/mozilla-releng/beetmoverscript/pull/155/commits/2f49f2d7d19780b310439e7673fc2951f8da3b20#diff-21f3f939e3375901322bd3639f4394bcR175
Flags: needinfo?(bwong)
(In reply to Lisa Guo [:lguo] EST from comment #34) > Benson, is this what you have in mind? > http://ftp.stage.mozaws.net/pub/firefox/candidates/61.0b15-candidates/ > build10/win64/en-US/buildhub.json > > From the latest staging release: > https://tools.taskcluster.net/groups/P45R5UvjS_-GX66dsMyw_g No, that's not a valid URL. I think all you need to do run urllib.parse.quote_plus on the filename part of the URL. That'll also work. I.e. if the URL was: https://ftp.stage.mozaws.net/pub/firefox/candidates/61.0b15-candidates/build10/win64/en-US/Firefox+Setup+61.0b15.exe Granted, I still don't know why it would be necessary since it works fine with spaces in it. But Benson might know.
Seeing as the only URL-encoding that should ever be necessary is for any whitespace in the filename part of the URL, I think a simple Python replace method should suffice: ie. "https://ftp.stage.mozaws.net/pub/firefox/candidates/61.0b15-candidates/build10/win64/en-US/Firefox Setup 61.0b15.exe".replace(' ', '+') returns: https://ftp.stage.mozaws.net/pub/firefox/candidates/61.0b15-candidates/build10/win64/en-US/Firefox+Setup+61.0b15.exe
(In reply to Lisa Guo [:lguo] EST from comment #37) > Seeing as the only URL-encoding that should ever be necessary is for any > whitespace in the filename part of the URL, I think a simple Python replace > method should suffice: > ie. > "https://ftp.stage.mozaws.net/pub/firefox/candidates/61.0b15-candidates/ > build10/win64/en-US/Firefox Setup 61.0b15.exe".replace(' ', '+') > > returns: > https://ftp.stage.mozaws.net/pub/firefox/candidates/61.0b15-candidates/ > build10/win64/en-US/Firefox+Setup+61.0b15.exe Sorry if it's overly verbose but you should use urllib.parse.quote but only on the file parts. E.g. `https://domain.tdl/{quote(some_path)}/{quote(other_path)}/{quote(file_name)}`. I.e. don't use quote_plus(). Use quote(). Perhaps it's overkill to run urllib.parse.quote() on every path of the URL because we know it's always ascii safe characters.
Okay, makes sense for good coding practice. Made the changes here: https://github.com/mozilla-releng/beetmoverscript/pull/155/commits/14f14c3cbc4cadba97a86fe27d4bcdb4a6b2529c The test, as you can see in the diffs, seems to be correct. Thanks for the feedback :)
Flags: needinfo?(bwong)
Last staging releases looks promising - https://tools.taskcluster.net/groups/AJG4k7djR-eKSLQsP4G-FQ There are still some few templates nits we need to address before rolling-this out to production but it looks promising. More here https://github.com/mozilla-releng/beetmoverscript/pull/155#pullrequestreview-131601130 Note to self: we need to also trigger a Fennec/Devedition staging release to make sure things work as expected.
See Also: → 1470948
Comment on attachment 8987151 [details] [review] adding support for buildhub.json Lisa's PR in beetmoverscript is ready to be landed via a production beetmoverscript version bump - https://github.com/mozilla-releng/beetmoverscript/pull/155
Attachment #8987151 - Flags: review+
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #41) > Last staging releases looks promising - > https://tools.taskcluster.net/groups/AJG4k7djR-eKSLQsP4G-FQ > There are still some few templates nits we need to address before > rolling-this out to production but it looks promising. More here > https://github.com/mozilla-releng/beetmoverscript/pull/155#pullrequestreview- > 131601130 > > Note to self: we need to also trigger a Fennec/Devedition staging release to > make sure things work as expected. All staging releases look good. Nothing breaking elsewhere, tests pass, artifacts are in their location. I suppose we're ready for production roll-out.
Regarding production landing, I have a few questions. What needs to happen from our side? 1. We need to bump beetmoverscript to 7.5.0 following this[0] to pick up Lisa's freshly merged PR[1] 2. We need land in-tree changes as well. Currently we've played with them on maple tree. This is here[2]. Question(s): a) do we land this to central to ride the trains or do we want this support across all trees? (central/beta/release/esr60). I'm not sure that the code generating the buildhub.json files is already in all these tree, we'd need to double-check b) We can definitely land this to central only, or central + beta or w\e other combination, but let's decide. @peterbe: how do you feel about this? [0]: https://github.com/mozilla-releng/beetmoverscript/#deploy-a-new-version-to-production [1]: https://github.com/mozilla-releng/beetmoverscript/pull/155 [2]: https://hg.mozilla.org/projects/maple/rev/ed0b4efd4f93
Flags: needinfo?(peterbe)
I actually don't have a qualified opinion to answer that. "Missunderstand me correctly", but all I really care about is files ending up in S3. :)
Flags: needinfo?(peterbe)
Pushed by mtabara@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/6716fba67cd8 update beetmover in-tree payload to include buildhub. r=mtabara a=release
* Rolled-out production 7.5.0 beetmoverscript that encompasses changes for enabling buildhub.json. * landed the in-tree patch to inbound / beta / release => https://hg.mozilla.org/integration/mozilla-inbound/rev/6716fba67cd8 => https://hg.mozilla.org/releases/mozilla-beta/rev/801112336847960bbb9a018695cf09ea437dc137 => https://hg.mozilla.org/releases/mozilla-release/rev/1d6943a03afb33584d53e3aaf2f6fcea0e10fa53 We should start seeing the buildhub.json files starting 62.0b5, 62.0 release and (most-likely) tomorrow's nightlies. * for releases (beta/release) the `buildhub.json` files will be present in the same dirs where the installers are * for nightlies, they have different naming (that purposefully are suffixed to `....buildhub.json` to ease the parsing in buildhub2 Let's give this 1-2 days to see how files are looking like and amend if needed.
Buildhub.json files for Firefox, Fennec and Devedition :) (some of them ...) https://archive.mozilla.org/pub/mobile/candidates/62.0b5-candidates/build1/android-aarch64/en-US/buildhub.json https://archive.mozilla.org/pub/mobile/candidates/62.0b5-candidates/build1/android-api-16/en-US/buildhub.json https://archive.mozilla.org/pub/mobile/candidates/62.0b5-candidates/build1/android-api-16/multi/buildhub.json https://archive.mozilla.org/pub/mobile/candidates/62.0b5-candidates/build1/android-x86/multi/buildhub.json https://archive.mozilla.org/pub/mobile/candidates/62.0b5-candidates/build1/android-x86/en-US/buildhub.json https://archive.mozilla.org/pub/mobile/candidates/62.0b5-candidates/build1/android-x86/en-US/buildhub.json https://archive.mozilla.org/pub/firefox/candidates/62.0b5-candidates/build1/linux-x86_64/en-US/buildhub.json https://archive.mozilla.org/pub/firefox/candidates/62.0b5-candidates/build1/mac/en-US/buildhub.json https://archive.mozilla.org/pub/firefox/candidates/62.0b5-candidates/build1/win32/en-US/buildhub.json https://archive.mozilla.org/pub/firefox/candidates/62.0b5-candidates/build1/win64/en-US/buildhub.json https://archive.mozilla.org/pub/devedition/candidates/62.0b5-candidates/build1/linux-x86_64/en-US/buildhub.json https://archive.mozilla.org/pub/devedition/candidates/62.0b5-candidates/build1/mac/en-US/buildhub.json https://archive.mozilla.org/pub/devedition/candidates/62.0b5-candidates/build1/win32/en-US/buildhub.json https://archive.mozilla.org/pub/devedition/candidates/62.0b5-candidates/build1/win64/en-US/buildhub.json (...)
As soon as the in-tree patch lands on central, we should be seeing the nightlies following that within the S3 buckets too. Upcoming 62.0+ releases and also the dot releases are going to have these as well.
Anything else here or can we soon declare victory? \o/
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #53) > Anything else here or can we soon declare victory? \o/ That's great progress! But I don't want to celebrate until we have the locale repacks also working :) By the way, where is that bug?
(In reply to Peter Bengtsson [:peterbe] from comment #54) > (In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #53) > > Anything else here or can we soon declare victory? \o/ > > That's great progress! But I don't want to celebrate until we have the > locale repacks also working :) > By the way, where is that bug? That's bug 1459302.
Depends on: 1459302

Is this done?

Hi Kim, I think we can close this?

Flags: needinfo?(kmoir)

I think so, I don't think this is used anymore.

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Flags: needinfo?(kmoir)
Resolution: --- → INVALID

(In reply to Kim Moir [:kmoir] ET from comment #59)

I think so, I don't think this is used anymore.

Actually we do use these files for buildhub, it's just that there isn't anything more to work on: the feature is complete.

I'll just mark this as fixed.

Resolution: INVALID → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: