929222 - Store eideticker data in Amazon S3

Status: RESOLVED WONTFIX

(Testing Graveyard :: Eideticker, defect, P3)

Description

[William Lachance (:wlach)]

Once we really get the firefoxos testing going continuously, we're probably going to be generating a fair number of videos. This will strain the capacity of the EC2 instance we're currently storing dashboard data on. Instead, we should try to store this (and any other files that we generate as part of the capture) in some kind of permanent redundant storage. Amazon S3 sounds like a logical choice.

Comment 1

[William Lachance (:wlach)]

Since we're likely not going to use datazilla, I am increasingly thinking of storing *everything* in S3 and just redirecting eideticker.mozilla.org to that account!

Comment 2

[William Lachance (:wlach)]

I think we can probably do this over the commandline, using http://s3tools.org/s3cmd. This might actually be a quite trivial change.

Comment 3

[William Lachance (:wlach)]

Attachment: First pass

First pass at allowing this. Note that s3cmd requires a configuration file to be set up for the user in $HOME/.s3cfg. The simplest possible looks like:

[default]
access_key = XXX
secret_key = YYY

(replace XXX,YYY with actual values)

Once this is added, we should be able to start syncing dashboard stuff to an arbitrary s3 bucket. I have a test setup going at: 

https://s3-us-west-1.amazonaws.com/test-ateam-mozilla/b2g-test/index.html

(note that you need to specify index.html because indexes are not yet working... should be possible to fix that I assume)

We should be able to configure DNS to direct eideticker.mozilla.org to our bucket and then we should be able to migrate away from ssh and custom ec2 instances (yay!). :)

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 8404273 [details] [diff] [review]
First pass

Review of attachment 8404273 [details] [diff] [review]:
-----------------------------------------------------------------

It's that simple? Nice! I would like to see a combined sync-dashboard.sh as we call this directly in eideticker-ci, though perhaps we could just make run-update-dashboard.sh compatible with the b2g tests? I certainly like the idea of syncing the dashboard after each test rather than at the end, otherwise a single test failure will cause no results from that job to upload. I'll address this in my next patch for bug 979498.

Comment 5

[William Lachance (:wlach)]

I decided to implement bug 1015422 first, should make it easier to not have to bother with a bunch of separate shell scripts.

Comment 6

[BMO Automation]

Eideticker has been discontinued, see bug 1361056