1296376 - Publish JSON files to S3 whenever something is changed

Status: RESOLVED WORKSFORME

(Release Engineering :: Applications: Shipit, defect, P3)

Description

[Rail Aliiev [:rail]]

product-details consumes files generated by ship-it (curl'ing them, see bug 1289826). It would be better to change this model from PULL to PUSH, so we get updates instantly without major race conditions (see bug 1296370).

The idea is to generate the files whenever we have something changed in the database and publish the files to s3.

Some ideas:


* publish the files to /v1/<uniqeid>/path/to/file.json
* copy /v1/<uniqeid>/path/to/file.json to * publish the files to /v1/latest/path/to/file.json

* since the operation may take long, we may need to schedule it from shipit instead of running (bump some field in the db or something)

Comment 1

[Sylvestre Ledru [:Sylvestre]]

Rail, we had the issue again today.
Can we change the priority?
This is a pain :(

Comment 2

[Rail Aliiev [:rail]]

This may require some significant changes. :/

As a possible workaround we can increase the frequency of the jobs pulling the data from ship-it to product-details.m.o and product-details-json.


Shyam, can we tweak product-details.m.o to sync from ship-it every 15m?

Paul, can we do the same for product-details-json?

Thanks in advance!

Comment 3

[Paul [:pmac] McLanahan]

I'll change the frequency to 15 min in bedrock and let you know here when it's done.

Comment 4

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/4d0a1ab46f1b6ffd3b8dcbd9c013cf9680e292b3
Bug 1296376: update product-details every 15 min.

https://github.com/mozilla/bedrock/commit/dcdf2fb2b9a5417c6142ccf479277c921ab5c046
Merge pull request #4708 from pmac/change-product-details-update-frequency

Bug 1296376: update product-details every 15 min.

Comment 5

[Paul [:pmac] McLanahan]

Both our Jenkins job that updates our git repo from product-details.m.o and the bedrock servers have been updated to pull in p-d updates every 15 min.

Comment 6

[Rail Aliiev [:rail]]

Thank you!

Comment 7

[Sylvestre Ledru [:Sylvestre]]

Eric, can you help with the cron? Thanks

Comment 8

[Eric Ziegenhorn :ericz]

The product-details cron already runs every 10 minutes.

Comment 9

[Rail Aliiev [:rail]]

(In reply to Eric Ziegenhorn :ericz from comment #8)
> The product-details cron already runs every 10 minutes.

Oh, sweet. Thank you!

Comment 10

[Rail Aliiev [:rail]]

<jlorenzo> rail: how hard would it be if we had a TC task in charge of pushing product-details? 
<aki> sounds like something for vcspush scriptworker, but we probably want something sooner
<jlorenzo> we don't really need to access vcs, do we? 
<rail> we'd need to use BB (VPN), but yeah, shouldn't be hard

Comment 11

[Julien Cristau [:jcristau]]

(In reply to Eric Ziegenhorn :ericz from comment #8)
> The product-details cron already runs every 10 minutes.

ship-it has been listing 52.0.2 as shipped in mobile_versions.json for at least half an hour, and in firefox_versions.json for longer, but product-details is still not updated (shows 52.0.1).  Is there an issue with that cron job?  Are the logs available somewhere?

Comment 12

[Eric Ziegenhorn :ericz]

It shows 52.0.2 for me: https://product-details.mozilla.org/1.0/firefox_versions.json  Maybe your browser is caching it?

Comment 13

[Eric Ziegenhorn :ericz]

When I first hit https://product-details.mozilla.org/1.0/mobile_versions.json I saw 52.0.1, but a shift-reload made it discard the browser cache and showed 52.0.2.

Comment 14

[Julien Cristau [:jcristau]]

I was running:

while ! diff -u <(curl -s https://product-details.mozilla.org/1.0/mobile_versions.json)  <(curl -s https://ship-it.mozilla.org/json/1.0/mobile_versions.json); do sleep 1m; done

Started around 14:20 UTC, and it didn't stop until 15:32 UTC.

Comment 15

[Julien Cristau [:jcristau]]

(so no, not a browser cache issue)

Comment 16

[Eric Ziegenhorn :ericz]

Hmm ok well I don't have logs for the script other than showing it ran, but the script does remove everything in that directory, pulls a new copy of everything using curl then redeploys.  So if it failed to pull something, it just wouldn't show up at all most likely.  Offhand, I can't tell where this is being slow to update.  I'll add some logging to the script for the time being and see if it shows any issues.

Comment 17

[Julien Cristau [:jcristau]]

Nowadays shipit triggers a product-details update whenever a release is shipped, so I believe this is fixed.