1126356 - Add support for |manage.py ingest_push <repo> tip|

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect, P3)

Description

[Ed Morley [:emorley]]

It would be a nice shortcut for getting the latest revision for a repo - and would mean in the docs I can state a working command for a recent push, rather than "<enter tip revision>".


(venv)vagrant@precise32:~/treeherder-service$ ./manage.py init_master_db
...
(venv)vagrant@precise32:~/treeherder-service$ ./manage.py init_datasources --reset
(venv)vagrant@precise32:~/treeherder-service$ ./manage.py export_project_credentials
(venv)vagrant@precise32:~/treeherder-service$ ./manage.py ingest_push mozilla-central tip

MySQL.MySQL debug message:
        host:localhost db:treeherder host_type:read_host proc:reference.selects.get_all_repository_info
        Executing SQL:SELECT * FROM `repository` WHERE `active_status` = 'active'
        Execution Time:6.7496e-04 sec

[2015-01-27 08:22:23,364] INFO [treeherder.etl.pushlog:107] Getting all pushes for 'mozilla-central' corresponding to changeset 'tip'
[2015-01-27 08:22:24,149] INFO [treeherder.etl.mixins:157] collection loading request: http://local.treeherder.mozilla.org/api/project/mozilla-central/resultset/
datasource.hubs.MySQL.MySQL debug message:
        host:localhost db:mozilla_central_objectstore_1 host_type:master_host proc:objectstore.selects.get_num_unprocessed
        Executing SQL:SELECT COUNT(*) as count FROM `objectstore` WHERE `processed_state` = 'ready' AND `error` = 'N'
        Execution Time:3.3689e-04 sec

[2015-01-27 08:22:28,588] WARNING [treeherder.etl.buildapi:112] skipping job on unsupported branch idle
...
[2015-01-27 08:22:28,598] WARNING [treeherder.etl.buildapi:123] property 'revision' not found in build4h
...
[2015-01-27 08:22:28,646] WARNING [treeherder.etl.buildapi:123] property 'revision' not found in build4h
datasource.hubs.MySQL.MySQL debug message:
        host:localhost db:mozilla_central_objectstore_1 host_type:master_host proc:objectstore.selects.get_num_unprocessed
        Executing SQL:SELECT COUNT(*) as count FROM `objectstore` WHERE `processed_state` = 'ready' AND `error` = 'N'
        Execution Time:2.5010e-04 sec

===

The pushlog is ingested fine, we just don't load the jobs.

I also noticed as making calls like:

[27/Jan/2015 08:22:35] "GET /api/project/mozilla-central/revision-lookup/?revision=38e4719e71af,bd91233cd18e,Unknown,63f8a47cfdf5,232401a6d1cc,6e7b69e6d3eb,1dd013ece082 HTTP/1.1" 200 191

I don't know if this is specific to ingest_push, or whether we're doing it generally, but it seems pointless to query revision-lookup for revision "Unknown".

Comment 1

[Ed Morley [:emorley]]

Attachment: Add support for |manage.py ingest_push <project> tip|

Comment 2

[William Lachance (:wlach)]

Comment on attachment 8555336 [details] [review]
Add support for |manage.py ingest_push <project> tip|

LGTM!

Comment 3

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/45824d8507d3cdc154ff452b1abefa3b6f3d75d7
Bug 1126356 - Add support for |manage.py ingest_push <project> tip|

The most recent push from a repository is given the tag 'tip' in
Mercurial. To save having to look up a specific revision to ingest when
testing, 'tip' can now be passed to ingest_push, and the revision that
'tip' currently maps to, will be used during push and jobs ingestion.
Previously, jobs ingestion would have used the non-existent SHA of 'tip'
when trying to associate jobs with pushes, resulting in no jobs being
imported from builds-{pending,running,4hr}.

Comment 4

[Ed Morley [:emorley]]

Thanks! :-)

We should really sort out the whole short vs long SHA inconsistencies, but for now this works. (See also bug 1079796).