1339179 - Set up pine for activity-stream builds & tests

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Chris AtLee [:catlee]]

We need to get the pine project branch set up to run the regular Firefox desktop builds & tests. No mobile automation needed.

Comment 1

[Alin Selagea [:aselagea]]

Attachment: bug_1339179_bb-configs.patch

Similar jobs to what we have on graphics.

Comment 2

[Alin Selagea [:aselagea]]

Attachment: tests_diff.txt

Comment 3

[Alin Selagea [:aselagea]]

Attachment: builds_diff.txt

Comment 4

[Kim Moir [:kmoir] ET]

Comment on attachment 8839478 [details] [diff] [review]
bug_1339179_bb-configs.patch

It looks good except for linux64 and linux64-debug.  These need to be enabled in taskcluster if this is not already done.  We don't run linux64 and linux64-debug on buildbot anymore.

Comment 5

[Alin Selagea [:aselagea]]

Attachment: disable_mobile_jobs.patch

Only asking for feedback since I followed the steps here: 
http://gecko.readthedocs.io/en/latest/taskcluster/taskcluster/how-tos.html

but I get no diff on my local computer (not sure why).

Comment 6

[Kim Moir [:kmoir] ET]

Comment on attachment 8839478 [details] [diff] [review]
bug_1339179_bb-configs.patch

actually just remembered that if you need talos tests for Linux64 on this branch, you need to enable the buildbot jobs. Is that a requirement?

Comment 7

[Kim Moir [:kmoir] ET]

Comment on attachment 8839492 [details]
disable_mobile_jobs.patch

hmm, so what steps do you use to generate the new graph locally?  Is a new full task graph generated locally? I would think that this would include the new pine tasks, the patch looks good to me

Comment 8

[Alin Selagea [:aselagea]]

(In reply to Kim Moir [:kmoir] from comment #6)
> Comment on attachment 8839478 [details] [diff] [review]
> bug_1339179_bb-configs.patch
> 
> actually just remembered that if you need talos tests for Linux64 on this
> branch, you need to enable the buildbot jobs. Is that a requirement?

I'll shift that to :catlee, maybe he has an idea if talos tests are needed on pine.

Comment 9

[Alin Selagea [:aselagea]]

(In reply to Kim Moir [:kmoir] from comment #7)
> Comment on attachment 8839492 [details]
> disable_mobile_jobs.patch
> 
> hmm, so what steps do you use to generate the new graph locally?  Is a new
> full task graph generated locally? I would think that this would include the
> new pine tasks, the patch looks good to me

From what I can see, TC jobs are already enabled on pine: https://treeherder.mozilla.org/#/jobs?repo=pine
Unfortunately, the desktop builds are all failing, so we don't have any tests to be run.

In order to generate the old graph, I copied the 'parameters.yml' file for the most recent pine run on my local m-i clone and ran './mach taskgraph'. 

As for the new graph, I applied the patch above and ran the same command again (redirecting the output to a different file of course). Yet no task difference between the two can be spotted.

Comment 10

[Kim Moir [:kmoir] ET]

Attachment: bug1339179.patch

I tried it with patch and the following parameters.yml 

base_repository: https://hg.mozilla.org/mozilla-central
build_date: 1487259202
filters:
- target_tasks_method
head_ref: 782b85278179d243acb385ddbb032355620e360f
head_repository: https://hg.mozilla.org/try/
head_rev: 782b85278179d243acb385ddbb032355620e360f
level: '1'
message: 'try: -b o -p linux64-stylo -u refest-stylo-1 -t none'
moz_build_date: '20170216153322'
optimize_target_tasks: true
owner: [e-mail]
project: try
pushdate: 1487259202
pushlog_id: '170732'
target_tasks_method: pine_tasks
triggered_by: push

And it seemed to filter out the android tasks

Comment 11

[Alin Selagea [:aselagea]]

Attachment: bug_1339179_bb-configs.patch

From what I recall from our buildduty meeting, we should move on and don't enable talos on pine. Updated the patch to account for that.

Comment 12

[Alin Selagea [:aselagea]]

(In reply to Kim Moir [:kmoir] from comment #10)
> Created attachment 8840106 [details] [diff] [review]
> bug1339179.patch
> 
> I tried it with patch and the following parameters.yml 
...
> pushlog_id: '170732'
> target_tasks_method: pine_tasks
> triggered_by: push
> 
> And it seemed to filter out the android tasks

Thanks for looking into this Kim!
It seems the reason I wasn't getting any difference was that I used 'target_tasks_method: default' instead of 'target_tasks_method: pine_tasks'.

Comment 13

[Alin Selagea [:aselagea]]

Attachment: task_diff.txt

The android tasks are indeed filtered properly now.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/dd8d6b4f3054
Set up pine for activity-stream builds & tests, r=kmoir

Comment 15

[Alin Selagea [:aselagea]]

Comment on attachment 8840885 [details] [diff] [review]
bug_1339179_bb-configs.patch

https://hg.mozilla.org/build/buildbot-configs/rev/b8e5c4be83aa
https://hg.mozilla.org/build/buildbot-configs/rev/d816f8ed797e

Comment 16

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/dd8d6b4f3054

Comment 17

[Chris AtLee [:catlee]]

(In reply to Alin Selagea [:aselagea][:buildduty] from comment #8)
> (In reply to Kim Moir [:kmoir] from comment #6)
> > Comment on attachment 8839478 [details] [diff] [review]
> > bug_1339179_bb-configs.patch
> > 
> > actually just remembered that if you need talos tests for Linux64 on this
> > branch, you need to enable the buildbot jobs. Is that a requirement?
> 
> I'll shift that to :catlee, maybe he has an idea if talos tests are needed
> on pine.

yes, we need talos tests on pine.

Comment 18

[Alin Selagea [:aselagea]]

Attachment: enable_talos.patch

I checked the list of tasks for pine and noticed that 'talos' was enabled. Since these tests are scheduled in TC, but run in BB, we would need to alter bb-configs as well.

Comment 19

[Kim Moir [:kmoir] ET]

Comment on attachment 8841612 [details] [diff] [review]
enable_talos.patch

Could you provide a builder diff?  Do these also enable the associated builders or are they removed somewhere else since this builds now run on tc? (we just want to enable talos)

Comment 20

[Alin Selagea [:aselagea]]

Attachment: enable_talos.patch

I missed the fact that 'enable_talos' was set to False, so no talos tests showed up. Updated the patch.

Comment 21

[Alin Selagea [:aselagea]]

Attachment: diff_talos.txt

Here's the tests diff following the change above. I also tested the patch on a build master, but no extra builds were triggered.

Comment 22

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/9125de07c8e0

Comment 23

[Kim Moir [:kmoir] ET]

Can this be closed?

Comment 24

[Alin Selagea [:aselagea]]

(In reply to Kim Moir [:kmoir] from comment #23)
> Can this be closed?

Yeah, I think so - automation has been enabled for the pine branch. 
I was just waiting to confirm that talos tests were enabled too. There are some job failures on this branch, but I don't think they make the scope of this bug. 

Thanks!

Comment 25

[Aki Sasaki (not active)]

Attachment: fix_pine.diff

Comment 26

[Aki Sasaki (not active)]

Attachment: fix_pine2.diff

Also disable asan, per dmose.

20:02 <dmose> aki: wonderful; thanks!   if you're so inclined, i think it'd be fine to remove ASAN as well.

Comment 27

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8855163 [details] [diff] [review]
fix_pine2.diff

LGTM!

Comment 28

[Aki Sasaki (not active)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/8a3519624432cad53d657ab0452f3ca11aafd936
bug 1339179 - adjust pine's taskcluster tasks. r=jlorenzo DONTBUILD

Comment 29

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/8a3519624432