Closed Bug 1035222 Opened 8 years ago Closed 7 years ago

We are not ingesting jobs from the Thunderbird trees

Categories

(Tree Management :: Treeherder, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mdoglio, Assigned: emorley)

References

Details

Attachments

(1 file)

This is possibly due to Bug #1035220
The sheriffs don't generally sheriff thunderbird, so this doesn't need to block switching to treeherder for Firefox/Gecko/B2G.
OS: Mac OS X → All
Priority: -- → P3
Hardware: x86 → All
[It does block turning off tbpl, although I don't think that's anywhere on the cards yet.]
(In reply to Joshua Cranmer [:jcranmer] from comment #2)
> [It does block turning off tbpl, although I don't think that's anywhere on
> the cards yet.]

Agreed, but bug 1030636 is about what we need to fix before Treeherder is to be used as the primary tool by the sheriffs (ie likely still alongside TBPL as the secondary tool, even for some gecko tasks - eg odd features that haven't yet been ported), not EOL of TBPL.
Priority: P3 → P2
Depends on: 1035220
Blocks: tbpl-eol
Blocks: treeherder-dev-transition
No longer blocks: tbpl-eol
Assignee: nobody → emorley
Status: NEW → ASSIGNED
Bug 1035220 is ui-side, whereas this bug is in the service.
Once we change the repo name used in the service, this will mean bug 1035220 needs updating to work with the new name.
Blocks: 1035220
No longer depends on: 1035220
The repository name must match that used by buildbot in builds-4hr,
builds-running, builds-pending, clobberer & self-serve otherwise
data ingestion and many links in the UI will break.
The only service that requires a different repo name is treestatus,
which will require a treeherder-ui change after this lands.
Attachment #8489013 - Flags: review?(cdawson)
Blocks: 1066989
To give a bit more context, TBPL uses:
https://hg.mozilla.org/webtools/tbpl/file/default/js/Config.js#l197

   198     "Thunderbird-Trunk": {
   199       primaryRepo: "comm-central",
...
   201       buildbotBranch: "comm-central",
   202       buildbotBranchExtra: "-thunderbird",
...
   205     "Thunderbird-Try": {
   206       primaryRepo: "try-comm-central",
...
   208       buildbotBranch: "try-comm-central",

From code inspection, "buildbotBranchExtra" is really "suffixForTreestatus" and isn't used anywhere else. All of builds-4hr, builds-running, builds-pending, clobberer & self-serve use what TBPL calls the buildbotBranch.

As such, the attached patch changes the repo name in treeherder for the various Thunderbird repos to match their respective buildbotBranch values in TBPL. Bug 1066989 will make the UI changes required to keep treestatus et al working in the UI - and longer term we should switch to storing the treestatus name in the repositories fixture to avoid having to hardcode in the UI, as well as to support other projects that don't use treestatus.
(In reply to Ed Morley (At work week 6th-13th Sept) [:edmorley] from comment #6)
> From code inspection, "buildbotBranchExtra" is really "suffixForTreestatus"
> and isn't used anywhere else

else... apart from treestatus that is.
Commits pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/4151178794d634e75703cb7f83c75777e02beaf8
Bug 1035222 - Use correct Thunderbird repo name to fix ingestion

The repository name must match that used by buildbot in builds-4hr,
builds-running, builds-pending, clobberer & self-serve otherwise
data ingestion and many links in the UI will break.
The only service that requires a different repo name is treestatus,
which will require a treeherder-ui change after this lands.

https://github.com/mozilla/treeherder-service/commit/2ea74beeabf4f391caa86bc4ac8a63409692d865
Merge pull request #218 from mozilla/fix-thunderbird-ingestion

Bug 1035222 - Use correct Thunderbird repo name to fix ingestion
Attachment #8489013 - Flags: review?(cdawson) → review+
Thank you :-)
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
The attached patch just changed the repo name, but left the pushlog URL the same.
The pushlog URL has always worked, so aiui:
* the DB should contain the pushes already
* we won't have old jobs from builds-4hr, but should pick up new ones from now on

After:
1) Running ./update-to-master as the user treeherder on th-dev-worker
2) Running ./update-to-master as the user treeherder on th-dev
3) Running load_initial_data on th-dev
4) Doing a varnish + supervisord restart as root on th-dev

...the comm-central repo now appears in the repos menu on dev -> great.

...however loading this repo results in a blank page (I've filed bug 1068810 for improving the error message).

Does something else need restarting?
Or does the repo name get hardcoded in the DB somewhere and we need to fix things up?
Flags: needinfo?(mdoglio)
I think we need to either change the thunderbird projects names in the treeherder.datasource table or run init_datasources to create the datasource entries with the new names. I didn't realize it in the first place, sorry about that
Flags: needinfo?(mdoglio)
On treeherder-dev I've run:

(venv)[treeherder@dev-3 treeherder-service]$ python manage.py init_datasources --settings treeherder.settings
(venv)[treeherder@dev-3 treeherder-service]$ python manage.py export_project_credentials --settings treeherder.settings

So we should be all set, right?

I'm guessing init_datasources should also be in update.py for the prod deployment, as well as added to the list of things in bug 1067262?
To let the log parser submit the parsed artifacts to the web service you also need to copy the credentials.json file to the processor machine. That will not be necessary on production as the exported credentials are rsynced to the processor machines with the rest of the code
Also, we could call export_project_credentials at the end of init_datasources to save one step. It's very likely that you will want to update your credentials file if you add a datasource.
Depends on: 1069267
Depends on: 1069283
(In reply to Ed Morley [:edmorley] from comment #12)
> I'm guessing init_datasources should also be in update.py for the prod
> deployment

Filed bug 1069283.

(In reply to Mauro Doglio [:mdoglio] from comment #15)
> Also, we could call export_project_credentials at the end of
> init_datasources to save one step. 

Filed bug 1069293.

(In reply to Mauro Doglio [:mdoglio] from comment #14)
> To let the log parser submit the parsed artifacts to the web service you
> also need to copy the credentials.json file to the processor machine.

Ah I didn't know that. I've added a comment to bug 1067262. We should get all this scripted :-)

I've also just found:
https://treeherder-service.readthedocs.org/en/latest/common_tasks.html
(doh)
Blocks: 1067447
Ok so I copied credentials to my user directory on th-dev, scp'd that to my local machine and then to th-dev-worker (my local version of scp doesn't support the '-3' param), moved it to /home/treeherder/treeherder-service/treeherder/etl/data, chowned it to treeherder:treeherder, then did a |/etc/init.d/supervisord restart| as root. (Yuck, we should script this, bug 1067262)

But still nothing on:
http://treeherder-dev.allizom.org/ui/#/jobs?repo=comm-central

Any ideas?
Status: RESOLVED → REOPENED
Flags: needinfo?(mdoglio)
Resolution: FIXED → ---
I will try to debug this, I don't have any more ideas
Flags: needinfo?(mdoglio)
Hurrah, thank you Mauro!
http://treeherder-dev.allizom.org/ui/#/jobs?repo=comm-central
(also checked comm's try, aurora, beta, esr)
Status: REOPENED → RESOLVED
Closed: 7 years ago7 years ago
Resolution: --- → FIXED
Summary: We are not ingesting any job from Thunderbird-[try|trunk] → We are not ingesting jobs from the Thunderbird trees
15:28 <•mdoglio> I have just cleared the cache, nothing else
15:28 <•mdoglio> apparently if you restart memcached the cache doesn't clear :o
15:28 <•mdoglio> I did it using the python shell
15:29 <•mdoglio> this will mitigate this problem https://github.com/mozilla/treeherder-service/pull/222
(bug 1069267)
Depends on: 1071570
You need to log in before you can comment on or make changes to this bug.