555892 - don't reclone all repositories every n900 test run

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Aki Sasaki (not active)]

Our MV<->MPT pipe is at a premium currently.

We switched from downloading a tarball every run from MV->MV (n810s), to cloning tools,talos&maemkit every run from hg/cvs on the n900s.

I think my preferred solution is to have a clean checkout somewhere on device that we can update & export to a working area (so we only grab diffs per run); a second possible solution is to have a clean checkout on the master that we update&tar up nightly.

Comment 1

[John Ford [:jhford] CET/CEST Berlin Time]

I don't know that these are optimizations we need to be making.

I am not sure how the cvs -z parameter works, but I am going to assume that it does something similar to a per-file gziping.  Given that the tarball only beats per-file gzip -9 by 2.3MB, I don't know if that is worth the time to implement.  I also don't know if cvs does some sort of tar operation before.

For unittests, doing the maemkit checkout has 112KB overhead vs a total of 32MB of browser+tests.  I really don't think that we should be trying to optimize that.

In the patch I am about to attach, I will remove the dependency on the tools repository for our talos runs as we only need one file from the repository.



jhford-wifi:~/mozilla/new-n810-image $ du -sc talos/ talos-no-CVS-dirs/ talos.tar.gz talos.tar.bz2
16956   talos/
16460   talos-no-CVS-dirs/
4628    talos.tar.gz
3896    talos.tar.bz2

jhford-wifi:~/mozilla/new-n810-image/talos $ find -type f -exec gzip -9 {} \;
jhford-wifi:~/mozilla/new-n810-image/talos $ du -hsc
6228    .
6228    total

jhford-wifi:~/mozilla/new-n810-image/maemkit $ du -sc
188     .
188     total
jhford-wifi:~/mozilla/new-n810-image/maemkit $ rm -rf .hg
jhford-wifi:~/mozilla/new-n810-image/maemkit $ du -sc
76      .
76      total
jhford-wifi:~/mozilla/new-n810-image/maemkit $

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: buildbotcustom-fixes

use the dns name and only download the one file we need.

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to comment #1)
> I don't know that these are optimizations we need to be making.

Gah, i forgot that if we have it as a tarball on a machine in MV, we don't need to do any transfer.  Maybe the solution for talos is to have a CVS mirror in MV for *all* talos machines, not just mobile

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: buildbotcustom fixes, this time more -z9

should've included the -z9 flag as a freebie bandwidth usage reduction.

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: buildbotcustom-fixes

sorry about the bugmail spam :S

Comment 6

[Aki Sasaki (not active)]

(In reply to comment #1)
> I don't know that these are optimizations we need to be making.

Have you *seen* all the redness from broken checkouts?
I think it is an optimization we desperately need to make.

Comment 7

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to comment #6)
> (In reply to comment #1)
> > I don't know that these are optimizations we need to be making.
> 
> Have you *seen* all the redness from broken checkouts?
> I think it is an optimization we desperately need to make.

yes, i have.  those are influenced by at least:
1. using the MPT dns server on the n900s, will be fixed in next n900 image by pointing them to the MV mirror of the build dns server
2. using hardcoded ip for talos checkouts which is fixed in this patch
3. tools repository checkouts failing which is removed by this patch.

I'd like to fix these three things before we make a call on this.

Comment 8

[Aki Sasaki (not active)]

Comment on attachment 442185 [details] [diff] [review]
buildbotcustom-fixes

Ok, worth a shot as a first step towards green.

Comment 9

[Aki Sasaki (not active)]

15 test runs died this morning because of this.

Comment 10

[Aki Sasaki (not active)]

36 failures since comment 9 because of this. I have the feeling things have gotten worse in the past couple days.

Comment 11

[Aki Sasaki (not active)]

44 failures since comment 10.

Comment 12

[John Ford [:jhford] CET/CEST Berlin Time]

Lets investigate what is going wrong in bug 576335

Comment 13

[Aki Sasaki (not active)]

66 failures since comment 11.

Comment 14

[Aki Sasaki (not active)]

151 failures over the long weekend.

Comment 15

[Aki Sasaki (not active)]

77 since Wednesday.

Comment 16

[Aki Sasaki (not active)]

59 since Friday, but I think pmm was wedged over the weekend. Kicked it earlier today.

I'm probably going to create tarballs to fix this.

Comment 17

[Aki Sasaki (not active)]

41 since comment 16.
I really want this bug resolved; taking.

Comment 18

[Aki Sasaki (not active)]

Attachment: quick n dirty shell script to create test tarballs

Comment 19

[Aki Sasaki (not active)]

Attachment: include maemkit in the tarball list

33 failures since the last update =\

Comment 20

[Aki Sasaki (not active)]

Attachment: start using tarballs -- configs

27 failures since the last comment.

Comment 21

[Aki Sasaki (not active)]

Attachment: start using tarballs -- custom

Names the pageloader/talos methods correctly, and adds maemkit tarball functionality for unit tests.

After this lands, our main vulnerability (besides the devices' overall fragility) will be downloading builds/tests from ftp.m.o and bc-proxy01. We'll be in the exact same boat as desktop tests there, though.

Downloaded&extracted the tarballs on smm2 just fine.

Comment 22

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 457172 [details] [diff] [review]
start using tarballs -- configs

looks good

Comment 23

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 457179 [details] [diff] [review]
start using tarballs -- custom

looks good

Comment 24

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 457150 [details]
include maemkit in the tarball list

it would be good if the script could do the initial checkouts, but that's definitely a p5 enhancement for the future!

Comment 25

[Aki Sasaki (not active)]

Comment on attachment 457179 [details] [diff] [review]
start using tarballs -- custom

http://hg.mozilla.org/build/buildbotcustom/rev/539e1039c6dd

Comment 26

[Aki Sasaki (not active)]

Comment on attachment 457172 [details] [diff] [review]
start using tarballs -- configs

http://hg.mozilla.org/build/buildbot-configs/rev/cb9660188d97

Comment 27

[Aki Sasaki (not active)]

25 failures since comment 20 =\
There may still be some queued up either in mail or currently running.
However, I expect to see these green up (or at least go orange/red after the cloning steps would have happened) real soon.

Comment 28

[Aki Sasaki (not active)]

There were 2 failures last night (for tarball downloads) due to dns lookups.
Definitely looks better.