1301747 - TC "latest" url paths are pointing to older versions of m-c asan

Status: RESOLVED FIXED

(Taskcluster :: General, defect)

Description

[Kamil Jozwiak [:kjozwiak]]

I'm not really sure if this is an issue, or already known, but the "latest" url paths [1][2] in TC for m-c asan builds are pointing to older versions of the build that are from 20160809064620. Even if it's a known issue, this is pretty confusing and bad UX.

Are m-c asan builds currently broken? Every source [3] is pointing to versions that are about a month old :/

[1] https://tools.taskcluster.net/index/artifacts/#gecko.v2.mozilla-central.latest.firefox/gecko.v2.mozilla-central.latest.firefox.linux64-asan
[2] https://tools.taskcluster.net/index/artifacts/#gecko.v2.mozilla-central.nightly.latest.firefox/gecko.v2.mozilla-central.nightly.latest.firefox.linux64-asan
[3] https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux64-asan/1470837316/

Comment 1

[Michael Shal [:mshal]]

:kmoir, it looks like the name we're using to index these jobs was "linux64-asan" in buildbot, but now it's "linux64-asan-opt" in Taskcluster. I assume this was probably a copy/paste error? If so I can update the task definition and backfill to "linux64-asan".

:jonasfj, is there a way we can delete or hide entries in the index? It would be nice if we could clean up some of the entries in "latest" - right now linux asan has "linux64-asan", "linux64-asan-dbg", "linux64-asan-debug", and "linux64-asan-opt", but only 2 of them are actually the latest, and the other 2 are stale. Or could we somehow setup anything with "latest" in the name to only show tasks less than a month old or something? Then stale things would disappear faster...

Comment 2

[Kim Moir [:kmoir] ET]

Hmm, I just named them that in TC because they were both ASAN opt and debug builds.  If this needs to be corrected, please go ahead.

Comment 3

[Jonas Finnemann Jensen (:jonasfj)]

@mshal, nope...
Things in the index don't even expire as it currently is.. We never wrote the expiration process :) hehe

There is bug 1150179 for adding a deprecation bit to index entries, which seems like the right way to handle this.

Comment 4

[Kamil Jozwiak [:kjozwiak]]

I'm not sure if this helps or adds some context, but it seems like the m-c asan builds have been moved [1] to a new location.

[1] https://blog.mozilla.org/security/2016/09/09/firefox-addresssanitizer-builds-have-been-moved/

Comment 5

[Michael Shal [:mshal]]

decoder, will it cause problems if I change the indexing for ASan builds back to 'linux64-asan' instead of 'linux64-asan-opt'? Or can we just update the link in your blog post mentioned in #c4 after I make the switch? I'd like to switch back to the original and then backfill linux64-asan's pushdate & revision routes to point to the Taskcluster tasks that were created after the switchover.

Comment 6

[Christian Holler (:decoder)]

I'm pretty sure it will cause lots of problems if we change the name again.

We already got a lot of complaints from community security testers when we did the first switch (because we kept stale builds on our FTP and made zero communication about it), changing the URL again now will probably cause a lot of people to consider our builds too unreliable to use them at all and do their own builds instead.

Comment 7

[Michael Shal [:mshal]]

So it sounds like the cat's out of the bag at this point, and the platform is being renamed to linux64-asan-opt. I'll see if I can backfill the old pushdate & revision routes for linux64-asan with linux64-asan-opt entries.

Comment 8

[Michael Shal [:mshal]]

Attachment: taskcluster-backfill.py

I slightly modified the backfill script from bug 1285647 to take an old & new platform. Review is mostly to make sure the script operates correctly - temporary routes are in project.releng.garbage.mshal-testing4.gecko.v2: https://tools.taskcluster.net/index/#project.releng.garbage.mshal-testing4.gecko.v2/project.releng.garbage.mshal-testing4.gecko.v2

Please double-check to make sure that the new routes map to the correct tasks & the formatting matches what we get in the top-level gecko.v2 for linux64-asan-opt.

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 8794347 [details]
taskcluster-backfill.py

1. the while True loop using continuationToken can be a separate function that you can reuse, but not a big deal here, only used twice.

2 print "64-bit FAIL: %s, %s" % (dt, pushdate) can be removed or rephrased ;)

3. I'm a bit worried about substring replacements you apply on line 87. It may lead to unpredictable results. Maybe use re.sub instead, so you can match word boundaries?

Comment 10

[Michael Shal [:mshal]]

(In reply to Rail Aliiev [:rail] from comment #9)
> 1. the while True loop using continuationToken can be a separate function
> that you can reuse, but not a big deal here, only used twice.

Good idea - I made a query_index() generator.

> 2 print "64-bit FAIL: %s, %s" % (dt, pushdate) can be removed or rephrased ;)

Yeah, that's pretty useless now. I think it was just leftover from debugging in linux32 / linux64 routes, but now it doesn't add anything over what the "Create route:" line tells you.

> 3. I'm a bit worried about substring replacements you apply on line 87. It
> may lead to unpredictable results. Maybe use re.sub instead, so you can
> match word boundaries?

Changed to re.sub, so the "index." removal can only remove at the beginning of the string, and the platform transform can only take place at the end of the string.

Comment 11

[Michael Shal [:mshal]]

Attachment: taskcluster-backfill.py

Were you also able to verify the routes created in mshal-testing4?

Comment 12

[Rail Aliiev [:rail]]

Comment on attachment 8795835 [details]
taskcluster-backfill.py

LGTM
r+ if you use raw strings in regexes, eg

newroute = re.sub(r'^index\.', '', route)

Otherwise you need 2 backslashes.

I also spot checked some routes at https://tools.taskcluster.net/index/#project.releng.garbage.mshal-testing4.gecko.v2/project.releng.garbage.mshal-testing4.gecko.v2

Comment 13

[Michael Shal [:mshal]]

The script is checked into braindump as taskcluster-backfill.py: http://hg.mozilla.org/build/braindump/file/tip/taskcluster/route-backfill/taskcluster-backfill.py

I ran it as follows for mozilla-central, mozilla-inbound, fx-team, and autoland.

taskcluster-backfill.py mozilla-central linux64-asan linux64-asan-opt 2015/09/01 2016/09/22

So we now have the linux64-asan-opt routes available for older builds in the .pushdate and .revision namespaces. The old linux64-asan (and linux64-asan-dbg) still show up in the .latest namespace - that should be fixed by bug 1150179.