910066 - [email/testing] Have GELAM Travis test-runs post results to ArbPL instance or other accessible server location

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::E-Mail, defect)

Description

[Andrew Sutherland [:asuth] (he/him)]

We're getting intermittent failure in our travis GELAM runs.  Our tests actually produce ridiculously detailed logs that should make it easy to identify what's going wrong but there's no way to easily get them out of travis right now without risking going over Travis's 4 meg limit and making the travis log output painful for humans.

Briefly discussed with :lightsofapollo, we could have Travis post them somewhere.  The Mozilla Petri project lets us stand up node.js apps pretty easily and ArbPL was acutally designed to be able to do stuff like this, so it would be reasonably easy to make that happen.  It doesn't really understand the github pull request stuff and instead will want to present a linearized view of things that's not quite right and without affected file summaries, but that's not really a big deal for us.  As long as we can have the travis output write-out hyperlinks that can be clicked on to directly bring up an ArbPL UI displaying the page, I think we're good.

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

Done!

Example excerpt from failed GELAM build
https://travis-ci.org/mozilla-b2g/gaia-email-libs-and-more/builds/16978925

===
Test: test_imap_bad_servers
PASS blacklist broken SPECIAL-USE implementation (imap:fake)
FAIL use working SPECIAL-USE implementation (imap:fake)
   failing step: 65,Has Inbox and Custom folders
   http://clicky.visophyte.org/tools/arbpl-standalone/?log=http://asuth-gelam-arbpl.s3.amazonaws.com/artifacts/183/183.1/test_imap_bad_servers-imap_fake.log
===

And if you go to that URL, you get to see a log!  Hooray!

The bad news is that this does not work on pull requests, just the main branch.  But it should hopefully help us track down our intermittents more.

Note that we only post failures to S3, although that's controlled via a flag, so it's not a huge change to post everything if we wanted to know what the timing looked like.