838299 - building b2g should not require downloading files from external sites

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: GonkIntegration, defect)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Spinning out from bug#838261.

(In reply to Aki Sasaki [:aki] from comment #3)
> https://github.com/mozilla-b2g/android-device-panda/blob/master/download-
> blobs.sh#L24
> 
> We need to fix the b2g build system to not rely on Google for tbpl builds.

(In reply to Aki Sasaki [:aki] from comment #4)
> This is showing up because this link is now a 404 at Google:
> wget -p -nd
> https://dl.google.com/dl/android/aosp/imgtec-panda-imm76i-67545da7.tgz
> --2013-02-05 10:54:46-- 
> https://dl.google.com/dl/android/aosp/imgtec-panda-imm76i-67545da7.tgz
> Resolving dl.google.com... 74.125.226.206, 74.125.226.196, 74.125.226.199,
> ...
> Connecting to dl.google.com|74.125.226.206|:443... connected.
> HTTP request sent, awaiting response... 404 Not Found
> 2013-02-05 10:54:47 ERROR 404: Not Found.

Downloading files from external locations like this causes three serious problems:
1) surprise tree closures when the file is removed (the emergency being resolved in bug#838261).
2) is a vector for security breach.
3) makes it hard for developers to debug intermittent code/test errors when downloadable files get updated without warning.

Please remove all external downloads / curl / wget from b2g build steps. If that means moving additional files into b2gsnapshot, like we are doing in bug#838261, please file new dep bugs and do that.

If there is a need to keep a download step for a specific developer need, move it behind a non-production developer-only flag.

Comment 1

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to John O'Duinn [:joduinn] from comment #0)
> Please remove all external downloads / curl / wget from b2g build steps. If
> that means moving additional files into b2gsnapshot, like we are doing in
> bug#838261, please file new dep bugs and do that.

Putting the files that are downloaded into the snapshot is the real solution.  Your snapshots must already have the Adreno drivers and backup-* folders because those are *not* fetched over the wire.

> If there is a need to keep a download step for a specific developer need,
> move it behind a non-production developer-only flag.

We should be optimizing the default behaviour for the common case, which is a developer who wants things to work out of the box.  The only steps I know of in a default build that require external servers are:

1) config.sh accessing the repository servers -- this is not being done on releng machines
2) blob extraction -- this only occurs if the snapshot is not generated correctly, which was the cause of bug 838261
3) xulrunner-sdk downloading for gaia profile creation -- this is being changed in bug 837340

Respectfully, I disagree that automation options should be defaults and that we should special case every developer.  If we add such a flag, the default should be targeted to the human developers not the automated build machines.  It already takes a long time and thought for a developer to ramp up their environment, adding complexity is counter productive.

I think that the real solution here is bug 617414, since no amount of build system flag creation can enforce this type of policy in the way that shutting down external network access can.