1098250 - when requesting a talos test via titanic backfill, I always get an error

Status: RESOLVED WONTFIX

(Testing :: General, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

I have done many requests for titanic to backfill jobs.  These requests almost always require a build for each revision.  I seem to get errors, and what I believe is happening is that we launch the builds successfully, then immediately go to launch the tests and it fails instead of waiting for the builds to finish.  I suspect we need to adjust our test launching process.

Here is an edited version of the output:
INFO:titanic:Your return code is: 202
INFO:titanic:https://secure.pub.build.mozilla.org/buildapi/revision/mozilla-inbound/e59e29b14cd6
Building for Job...
Builds are triggered!
Error: For e59e29b14cd6 Windows XP 32-bit mozilla-inbound pgo talos chromez

Comment 1

[Gautam Akiwate [:gakiwate]]

Can you list a few other tests which error out?

As an optimization we basically decided to let the processing "fall through" ie. process multiple stages in one go. The problem is I think the build system takes some time to register the existence of the builds via the API we are using. So my suggest would be to not fall through from "updated" status to "building" status and instead break when we update status to "building".

I think this could be a simple fix.
https://github.com/gakiwate/titanic/blob/master/backfill.py#L81

What do you think?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

I have not seen any of my pushes work in the last 10 or so tries without an error- in fact all of them need at least one build if not many.  If we could:
schedule builds
sleep 2 minutes
check builds / schedule tests

If we run this in a cron job every 10-15 minutes, then we don't need a sleep but just do this:
if anybuilds are needed:
    schedule_builds()
    return
else:
    schedule_tests()
    return

this is a good place to weigh in here that if we need to do builds, waiting 15 extra minutes to schedule some tests is not a big deal;  The reason for backfilling is to look at data from point a->b which cannot be completed until all data is there.

Comment 3

[Gautam Akiwate [:gakiwate]]

Attachment: 0001-BUG-1098250-Proposed-Fix.patch

I was thinking something as simple as this. I am not very keen on having a sleep inside the loop as it might lead to trouble later on and it's just bad practice! :)

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8522739 [details] [diff] [review]
0001-BUG-1098250-Proposed-Fix.patch

Review of attachment 8522739 [details] [diff] [review]:
-----------------------------------------------------------------

this looks great!

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

we are backfilling with mozilla_ci_tools (https://github.com/armenzg/mozilla_ci_tools) which has taken a lot of titanic pieces and knowledge.