Closed
Bug 1325377
Opened 8 years ago
Closed 7 years ago
Public API to Balrog's database
Categories
(Release Engineering Graveyard :: Applications: Balrog (backend), defect, P2)
Release Engineering Graveyard
Applications: Balrog (backend)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: catlee, Assigned: asilva)
References
Details
(Whiteboard: [lang=python][ready])
Attachments
(2 files)
It would make things like funsize scheduling much easier if we had a way to query the list of blobs and inspect individual blobs without requiring credentials or special network access.
Comment 1•8 years ago
|
||
Benson suggested adding an API to the existing public app. This seems like it wouldn't be too difficult, and might be a good opportunity to start using Swagger. Something feels funny to me about allowing anyone in the world to query parts of the database, but I can't find a rational reason not to allow it. We already export a large swath of the database to a public location, and there's nothing secret in it...so it should be fine.
To the queries specifically, it sounds like you want two endpoints:
1) Get a list of releases (possibly with some way to filter them on the server?)
2) Get the full blob for a specific release
And maybe 3) Get a specific platform+locale from a specific release?
Flags: needinfo?(catlee)
Reporter | ||
Comment 2•8 years ago
|
||
1) yes, maybe with some basic sorting / pagination. like the most recent N releases matching some pattern.
2) yup, definitely need this
3) I think funsize in this case would be interested in an entire blob at once, not necessarily platform/locales specifically. The release or nightly decision task logic would be constructing the set of partials to generate spanning all platforms and locales.
Flags: needinfo?(catlee)
Comment 3•8 years ago
|
||
Catlee and I were talking about this a bit more just now, and a couple of things came up:
1) Even if it's not hosted separately, having a different domain for the API would probably be good. It lets us distinguish the API uses from actual updates, move the API to its own thing later if needed, and possibly do some traffic shaping or rate limiting if it becomes necessary.
2) If we build this as a Swagger API, it would be good to build in a way that it can be re-used and extended by a new admin API later (so that the admin API doesn't have to write its own read operations, too).
Updated•8 years ago
|
Priority: -- → P2
Summary: Provide public read-only access to release blobs → Provide public read-only access to release blobs (aka public balrog api)
Whiteboard: [lang=python]
Comment 4•8 years ago
|
||
Updating this summary to expand the scope. Releases are clearly the #1 priority here, but I also think it would be good to have some access to the Rules, particularly so that anyone can see what Releases are live at any given point in time.
We should also expose the history tables for Rules & Releases, to allow people to do archaelogy on their own.
I don't think that we need to provide public access to Permissions, User Roles, or any of the Scheduled Changes tables.
Summary: Provide public read-only access to release blobs (aka public balrog api) → Public API to Balrog's database
Updated•8 years ago
|
Whiteboard: [lang=python] → [lang=python][ready]
Updated•8 years ago
|
Component: Balrog: Frontend → Balrog: Backend
Comment 5•8 years ago
|
||
Allan told me he wants to work on this soon, let's get it assigned :).
Assignee: nobody → allan.tavares
Comment 6•8 years ago
|
||
Commit pushed to master at https://github.com/mozilla/balrog
https://github.com/mozilla/balrog/commit/e04ab19b22fd7fd59913ebf19a2009235efe4672
bug 1325377: restructure web layers and convert public app to swagger (#276). r=bhearsum
Comment 7•8 years ago
|
||
(In reply to [github robot] from comment #6)
> Commit pushed to master at https://github.com/mozilla/balrog
>
> https://github.com/mozilla/balrog/commit/
> e04ab19b22fd7fd59913ebf19a2009235efe4672
> bug 1325377: restructure web layers and convert public app to swagger
> (#276). r=bhearsum
This was part 1 to this bug - Allan did some really great foundational work to make it easier to add the public API.
Allan, can we chat about following up with the actual public API when you have time?
Flags: needinfo?(allan.tavares)
Assignee | ||
Updated•8 years ago
|
Flags: needinfo?(allan.tavares)
Comment 8•8 years ago
|
||
We pushed part 1 of this to production yesterday and it uncovered an issue where update URLs with unicode characters in them do not get served updates. It turns out that we get tons of requests from Ubuntu and Mint builds with update URLs like: https://aus5.mozilla.org/update/3/GMP/53.0/20170421105455/Linux_x86_64-gcc3/null/release-cck-%20%C3%A2%C2%80%C2%9Cmint%C3%A2%C2%80%C2%9D/Linux%204.4.0-53-generic%20(GTK%203.18.9,libpulse%208.0.0)/mint/1.0/update.xml. I think this is pretty serious, because it means none of these builds are receiving GMP or SystemAddon updates, which I'm pretty sure we want them to get.
Even with older code, I seem to get 500s locally, but the new code caused a massive increase in the number of 500s we saw in production. I'm having trouble explaining that, but I think we should strive to gracefully handful unicode in the URL. We probably want to convert it to a string as best we can.
Assignee | ||
Comment 9•8 years ago
|
||
On my tests I got errors on validation level. This new PR the strings are encoded to utf-8.
Do you have another error type in logs?
Flags: needinfo?(bhearsum)
Attachment #8865167 -
Flags: review?(bhearsum)
Comment 10•8 years ago
|
||
(In reply to Allan [:ex-dev] from comment #9)
> Created attachment 8865167 [details] [review]
> PR with fixes the unicode issues in URL.
>
> On my tests I got errors on validation level. This new PR the strings are
> encoded to utf-8.
> Do you have another error type in logs?
That was the only one that came up.
Flags: needinfo?(bhearsum)
Comment 11•8 years ago
|
||
Comment on attachment 8865167 [details] [review]
PR with fixes the unicode issues in URL.
Reviewed in github.
Attachment #8865167 -
Flags: review?(bhearsum)
Comment 12•8 years ago
|
||
Commit pushed to master at https://github.com/mozilla/balrog
https://github.com/mozilla/balrog/commit/3afc89072d986e832bcd67afa58e04361ac49c08
bug 1325377 - reorg of web code and swaggerification of existing public endpoints (#314). r=bhearsum
Comment 13•8 years ago
|
||
Allan, any idea when you'll be able to have a look at the public API for Rules and Releases?
Flags: needinfo?(allan.tavares)
Assignee | ||
Comment 14•8 years ago
|
||
Next saturday, probably. Do you have a deadline to this bug?
Flags: needinfo?(allan.tavares)
Comment 15•8 years ago
|
||
(In reply to Allan [:ex-dev] from comment #14)
> Next saturday, probably. Do you have a deadline to this bug?
No deadline, but there's some other work we're looking at that is much easier once we have the public API, so we'd just like to plan for it. Do you mean Saturday the 20th or 27th? No problem either way.
Assignee | ||
Comment 16•8 years ago
|
||
(In reply to Ben Hearsum (:bhearsum) from comment #15)
> (In reply to Allan [:ex-dev] from comment #14)
> > Next saturday, probably. Do you have a deadline to this bug?
>
> No deadline, but there's some other work we're looking at that is much
> easier once we have the public API, so we'd just like to plan for it. Do you
> mean Saturday the 20th or 27th? No problem either way.
Both, I think. It depends on the demands of my work. haha
Comment 17•8 years ago
|
||
(In reply to Allan [:ex-dev] from comment #16)
> (In reply to Ben Hearsum (:bhearsum) from comment #15)
> > (In reply to Allan [:ex-dev] from comment #14)
> > > Next saturday, probably. Do you have a deadline to this bug?
> >
> > No deadline, but there's some other work we're looking at that is much
> > easier once we have the public API, so we'd just like to plan for it. Do you
> > mean Saturday the 20th or 27th? No problem either way.
>
> Both, I think. It depends on the demands of my work. haha
No worries, thanks for the quick reply!
Assignee | ||
Comment 18•7 years ago
|
||
Attachment #8877143 -
Flags: review?(bhearsum)
Comment 19•7 years ago
|
||
Comment on attachment 8877143 [details] [review]
Public API - PR Part 2
Reviewed in Github
Attachment #8877143 -
Flags: review?(bhearsum)
Attachment #8877143 -
Flags: review-
Attachment #8877143 -
Flags: feedback+
Comment 20•7 years ago
|
||
Commit pushed to master at https://github.com/mozilla/balrog
https://github.com/mozilla/balrog/commit/2954a626c6b07d270bc1d703b01ef23136eb491d
bug 1325377: Public API for Rules & Releases (#320). r=bhearsum,sfraser,aksareen
Comment 21•7 years ago
|
||
This appears to be working well in production, thank you a TON Allan - many people have been looking forward to this. I believe there's a couple of things we wanted to follow up with, but we can do that in separate bugs.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Assignee | ||
Comment 22•7 years ago
|
||
Very Nice,
Is the endpoint to API already available?
Comment 23•7 years ago
|
||
(In reply to Allan [:ex-dev] from comment #22)
> Very Nice,
> Is the endpoint to API already available?
Yup. The swagger docs are at https://aus5.mozilla.org/ui/, and the API is at https://aus5.mozilla.org/api/v1 (eg: https://aus5.mozilla.org/api/v1/rules)
Assignee | ||
Comment 24•7 years ago
|
||
Newbie question: -Is the database protected against DDOS attack?
If, for instance, I get all rules, then all revisions for each rule for a lot times, from different threads. Will it impact the /update endpoints ?
Flags: needinfo?(bhearsum)
Comment 25•7 years ago
|
||
(In reply to Allan [:ex-dev] from comment #24)
> Newbie question: -Is the database protected against DDOS attack?
>
> If, for instance, I get all rules, then all revisions for each rule for a
> lot times, from different threads. Will it impact the /update endpoints ?
There's no special protection we have against doing that, but aus5.mozilla.org is set-up to handle a very high volume of requests, and automatically scales up if load goes up. The primary bottleneck we may have is the database server. I don't think we need to worry about DDoS for the moment (I think it would be tough to generate enough requests to really bog down the server), but I could certainly see a use case for doing some amount of rate limiting. That might be something we'd want to do at the ELB or ngnix level, though.
Flags: needinfo?(bhearsum)
Assignee | ||
Comment 26•7 years ago
|
||
Cool, good to know a little more of releng services structure.
Updated•5 years ago
|
Product: Release Engineering → Release Engineering Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•