966412 - limit pvtbuilds uploads

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Aki Sasaki (not active)]

I think we can turn off all non-eng non-emulator dep uploads to pvtbuilds.
This is the #2 source of traffic from AWS through our VPC, and we *can't* send this over the internet, even via https, so limiting what we upload will help reduce our load.

Comment 1

[Aki Sasaki (not active)]

Tony, Jonathan: Is comment 0 accurate?  Or do we have use cases for specific dep uploads?

Comment 2

[Aki Sasaki (not active)]

Both this bug and bug 914111 will probably need us to spin up more nightlies.

Comment 3

[Tony Chung [:tchung]]

Hi Aki, are you referring to stop uploading the full device production builds on pvtbuilds like:

 [ ]	hamachi.zip	31-Jan-2014 10:27 	86M	 
	leo.zip	        31-Jan-2014 13:57 	93M	 

etc.?   trying to understand what the question here.

Comment 4

[Jonathan Griffin (:jgriffin)]

There won't be any impact on automation for turning off the non-eng uploads (we only use eng builds).

Comment 5

[Aki Sasaki (not active)]

(In reply to Tony Chung [:tchung] from comment #3)
> Hi Aki, are you referring to stop uploading the full device production
> builds on pvtbuilds like:
> 
>  [ ]	hamachi.zip	31-Jan-2014 10:27 	86M	 
> 	leo.zip	        31-Jan-2014 13:57 	93M	 
> 
> etc.?   trying to understand what the question here.

Essentially, anything in http://pvtbuilds.pvt.build.mozilla.org/pvt/mozilla.org/b2gotoro/tinderbox-builds/ that isn't -eng will not be uploaded on push anymore.

They'll be uploaded to http://pvtbuilds.pvt.build.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/ if they have associated nightly builds.

Comment 6

[Tony Chung [:tchung]]

(In reply to Aki Sasaki [:aki] from comment #5)
> (In reply to Tony Chung [:tchung] from comment #3)
> > Hi Aki, are you referring to stop uploading the full device production
> > builds on pvtbuilds like:
> > 
> >  [ ]	hamachi.zip	31-Jan-2014 10:27 	86M	 
> > 	leo.zip	        31-Jan-2014 13:57 	93M	 
> > 
> > etc.?   trying to understand what the question here.
> 
> Essentially, anything in
> http://pvtbuilds.pvt.build.mozilla.org/pvt/mozilla.org/b2gotoro/tinderbox-
> builds/ that isn't -eng will not be uploaded on push anymore.
> 
> They'll be uploaded to
> http://pvtbuilds.pvt.build.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/ if
> they have associated nightly builds.

in that case, kill it.   Thanks.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

This would be extremely helpful for reducing disk usage on ftp.m.o.

Comment 8

[Aki Sasaki (not active)]

(In reply to Nick Thomas [:nthomas] from comment #7)
> This would be extremely helpful for reducing disk usage on ftp.m.o.

The latest patch in bug 914111 turns off public dep uploads anyway, so this specific bug should have no effect on ftp.m.o (but would reduce disk usage on pvtbuilds).

Comment 9

[Aki Sasaki (not active)]

Attachment: limit-private-mozharness

* updates the upload_dep_target_exclusions for in-use config files (excluding -eng and try)
* removes unused config files

Comment 10

[Aki Sasaki (not active)]

Attachment: limit-private-mozharness

er, with wasabi

Comment 11

[Aki Sasaki (not active)]

Attachment: limit-private-buildbot-configs

Enable wasabi nightlies, since we won't have dep uploads.

Comment 12

[Chris AtLee [:catlee]]

Comment on attachment 8369267 [details] [diff] [review]
limit-private-mozharness

Review of attachment 8369267 [details] [diff] [review]:
-----------------------------------------------------------------

lesscode++

Comment 13

[Aki Sasaki (not active)]

Comment on attachment 8369267 [details] [diff] [review]
limit-private-mozharness

https://hg.mozilla.org/build/mozharness/rev/99cf72f2f85f

Comment 14

[Aki Sasaki (not active)]

Comment on attachment 8369268 [details] [diff] [review]
limit-private-buildbot-configs

https://hg.mozilla.org/build/buildbot-configs/rev/76564ec6b160

Comment 15

[Aki Sasaki (not active)]

Landed https://hg.mozilla.org/webtools/tbpl/rev/6f2354446159 for wasabi nightlies on tbpl.

Comment 16

[Ryan VanderMeulen [:RyanVM]]

Follow-up typo fix: https://hg.mozilla.org/webtools/tbpl/rev/2e096737fbc1

Comment 17

[Ryan VanderMeulen [:RyanVM]]

(In reply to Aki Sasaki [:aki] from comment #15)
> Landed https://hg.mozilla.org/webtools/tbpl/rev/6f2354446159 for wasabi
> nightlies on tbpl.

(In reply to Ryan VanderMeulen [:RyanVM UTC-5][Limited connectivity due to mass power outage] from comment #16)
> Follow-up typo fix: https://hg.mozilla.org/webtools/tbpl/rev/2e096737fbc1

In production :)

Comment 18

[Aki Sasaki (not active)]

Looks like I need to s,nexus-4,mako, in mozharness.

Comment 19

[Aki Sasaki (not active)]

Attachment: s,nexus-4,mako

Just noticed that tinderbox-builds/b2g-inbound-mako/ etc were still being populated.

Comment 20

[Aki Sasaki (not active)]

Attachment: enable_hamachi_uploads

Same, but enable hamachi depend uploads per bug 969542 comment 11.

Comment 21

[Aki Sasaki (not active)]

Comment on attachment 8373632 [details] [diff] [review]
enable_hamachi_uploads

https://hg.mozilla.org/build/mozharness/rev/55fe1c06b93d

Comment 22

[Aki Sasaki (not active)]

Transplanted to production.

Comment 23

[Aki Sasaki (not active)]

Ok, I'm seeing hamachi depend builds uploading. and I'm seeing nexus-4 depend builds not uploading.

Comment 24

[Armen [:armenzg]]

Mozharness change live in production.