1306875 - Document ingesting old builds-4h pushes

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder, enhancement, P4)

Description

[Jonathan French (:jfrench)]

During work on bug 1294525 camd showed me jgraham's groovy way to ingest a push from an old builds-4h. I'm wondering if this would be a useful addition to the docs, below our existing "Ingesting a single push (at a time)" section https://treeherder.readthedocs.io/installation.html#ingesting-a-single-push-at-a-time

If I understand it correctly, the general steps are:

1) Find an old push on prod that you'd like to ingest locally
2) Find the corresponding date/time builds-4h url .gz in http://builddata.pub.build.mozilla.org/buildjson/
3) Download, unzip, open it and find the 'got_revision' SHA you need *
4) Optional: thpurge, re-provision your vagrant to be sure what you're ingesting

5) Switch treeherder/config/settings.py BUILDAPI_BUILDS4H_URL to your custom value
eg. BUILDAPI_BUILDS4H_URL = "http://builddata.pub.build.mozilla.org/buildjson/builds-2016-08-28.js.gz"

6) In vagrant, ingest the push
./manage.py ingest_push [repo] [SHA]
eg. ./manage.py ingest_push comm-central 2540c39cc958

* avoids any browser hang directly opening a large builds-4h .gz

Just wondering what you guys think? I wasn't aware it was possible reading the current docs. For a title, perhaps we could create a separate block called "Ingesting an old push", or flesh out the existing section to describe everything.

Comment 1

[Ed Morley [:emorley]]

I'm a massive fan of good docs, however I think there's sometimes a fine line between documenting lots of useful things and the docs becoming so verbose people dislike looking through them, so miss even the most important items.

For this workflow, if we were going to try and improve it, I'd rather add it to the management command (ie fetch the Hg revision and use the date to automatically fetch the correct builds-4hr archive) than add things to the docs I think? Though with buildbot gradually going away, that may not even be worth it.

Happy to see what the others say about it though?

Something that we definitely do need to document more (and make sure works out of the box, on Vagrant) is Pulse ingestion from Taskcluster however.

Comment 2

[Jonathan French (:jfrench)]

By coincidence I jotted out a docs addition I think while your reply was being posted here :) I'll open a PR for what it's worth, but ya please close it if it's not the desired way forward or is no longer applicable with buildbot disappearing. You can see the appearance at least, in the .rst file.

Comment 3

[GitHub Autolander Bot]

Attachment: [treeherder] tojon:docs-ingest-old-push > mozilla:master