818329 - Requesting b2g images from pvtbuilds.m.o fails

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Armen [:armenzg]]

e.g. http://mobile-imaging-001.p1.releng.scl1.mozilla.com/ui/log.html?device=panda-0094

2012-12-04T16:56:24 statemachine writing pxe_config panda-b2g.1, boot config {"b2gbase": "https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cedar-panda/20121204144926"} to db
2012-12-04T16:56:24 statemachine entering state start_pxe_boot
2012-12-04T16:56:24 statemachine entering state pxe_power_cycling
2012-12-04T16:56:24 bmm setting PXE config to 'panda-b2g.1'
2012-12-04T16:56:24 bmm initiating power cycle
2012-12-04T16:56:27 statemachine entering state pxe_booting
2012-12-04T16:57:45 syslog second stage URL found: http://10.12.128.33/scripts/b2g-second-stage.sh
2012-12-04T16:57:45 syslog wget success: --2025-11-08 17:54:59-- http://10.12.128.33/scripts/b2g-second-stage.sh#012Connecting to 10.12.128.33:80... connected.#012HTTP request sent, awaiting response... 200 OK#012Length: 2605 (2.5K) [application/x-sh]#012Saving to: `/opt/scripts/second-stage.sh'#012#012 0K .. 100% 81.4M=0s#012#0122025-11-08 17:54:59 (81.4 MB/s) - `/opt/scripts/second-stage.sh' saved [2605/2605]
2012-12-04T16:57:45 syslog Executing second-stage.sh
2012-12-04T16:57:45 syslog beginning b2g-second-stage.sh
2012-12-04T16:57:45 syslog setting time with ntpdate
2012-12-04T16:57:55 syslog Submitting lifeguard event at http://10.12.128.33/api/device/panda-0094/event/b2g_downloading/
2012-12-04T16:57:55 bmm clearing PXE config
2012-12-04T16:57:55 statemachine entering state b2g_downloading
2012-12-04T16:57:55 syslog getting B2G_URL from http://10.12.128.33/api/device/panda-0094/bootconfig/
2012-12-04T16:57:56 syslog B2G URL: https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cedar-panda/20121204144926
2012-12-04T16:57:56 syslog fetching https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cedar-panda/20121204144926/system.tar.bz2
2012-12-04T16:57:56 syslog wget failed:

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Brian, is this something you're in a good place to fix up?  If not, I can take a peek, but I know you're working on this right now.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

And to be clear, the need here is to have private builds available, without HTTP authentication, within the build network.  That probably means HTTP, not HTTPS.  It can also mean a different VIP and/or hostname (I'd suggest pvtbuilds.pvt.build.mozilla.org), if required.

Comment 3

[Armen [:armenzg]]

I can't reach any of the following (HTTP rather than HTTPS):
http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cedar-panda
http://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cedar-panda

I can ping one of them though:
Armens-MacBook-Air:mozharness_hg armenzg$ ping pvtbuilds.pvt.build.mozilla.org
ping: cannot resolve pvtbuilds.pvt.build.mozilla.org: Unknown host
Armens-MacBook-Air:mozharness_hg armenzg$ ping pvtbuilds.mozilla.org
PING pvtbuilds.mozilla.org (63.245.215.32): 56 data bytes
64 bytes from 63.245.215.32: icmp_seq=0 ttl=48 time=430.668 ms

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

I didn't mean to indicate any of those would work now -- the suggestion was for :digi :)

Comment 5

[Armen [:armenzg]]

Ah I see.

On another note< I got another similar (not related) problem from dev-stage:
http://mobile-imaging-001.p1.releng.scl1.mozilla.com/ui/log.html?device=panda-0086
2012-12-04T18:06:00 syslog getting B2G_URL from http://10.12.128.33/api/device/panda-0086/bootconfig/
2012-12-04T18:06:01 syslog B2G URL: http://dev-stage01.build.mozilla.org/pub/mozilla.org/mozilla/libraries/b2g_temporary/b2g-2012112900
2012-12-04T18:06:01 syslog fetching http://dev-stage01.build.mozilla.org/pub/mozilla.org/mozilla/libraries/b2g_temporary/b2g-2012112900/system.tar.bz2
2012-12-04T18:09:03 statemachine entering state failed_b2g_downloading
2012-12-04T18:09:03 statemachine device has failed: failed_b2g_downloading

Should I file a separate bug for it? Or should I just wait until this bug gets fixed so I won't need the work around?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Is that where staging B2G builds will be stored?  If so, then yes, please file another bug.  We'll need to open that flow.

Comment 7

[Armen [:armenzg]]

(In reply to Dustin J. Mitchell [:dustin] from comment #6)
> Is that where staging B2G builds will be stored?  If so, then yes, please
> file another bug.  We'll need to open that flow.

That was just temporarily for my testing purposes until this bug would be resolved.

Has the work on this been started? This is quite important.

Comment 8

[Brian Hourigan [:digi]]

:armenzg

Just wanted to update you - I'm implementing a fix now, should be done in a few minutes. I'll keep you posted.

Comment 9

[Brian Hourigan [:digi]]

:armenzg

I've stood up a new service on pvtbuilds.pvt.build.mozilla.org, the existing pvtbuilds.m.o won't work for this as it is on a public IP and our firewall conceals the true source address.

Network flows need to be created to permit access (bug 818687), but once that is established I expect that this will begin working.

(Upgrading back to major - downgraded the priority in error)

Comment 10

[Armen [:armenzg]]

FTR, we can download files from pvtbuilds.pvt.build.mozilla.org:/{pub,pvt}
http://hg.mozilla.org/build/mozharness/file/default/scripts/b2g_build.py#l595
e.g: http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-panda/20121205175527/
e.g: http://pvtbuilds.pvt.build.mozilla.org/pvt/mozilla.org/b2g/

We currently are downloading from "pub".


BTW, I could not upload to pvtbuilds.pvt.build.mozilla.org:/pvt until I created the parent folder it wanted to rsync to.
Should we run a "mkdir -p" command before the rsync?

I also cannot see the folder through http:
http://pvtbuilds.pvt.build.mozilla.org/pvt/mozilla.org/b2g/tinderbox-builds/cedar-panda/

[cltbld@bld-centos6-hp-009 ~]$ rsync -e "ssh -oIdentityFile=/home/cltbld/.ssh/b2gbld_dsa" -azv test [e-mail]:/pvt/mozilla.org/b2g/tinderbox-builds/cedar-panda/test/
reverse mapping checking getaddrinfo for pvtbuilds-zlb.dmz.scl3.mozilla.org [10.22.74.161] failed - POSSIBLE BREAK-IN ATTEMPT!
sending incremental file list
rsync: mkdir "/pvt/mozilla.org/b2g/tinderbox-builds/cedar-panda/test" failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(576) [receiver=3.0.6]
rsync: connection unexpectedly closed (9 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]

[cltbld@bld-centos6-hp-009 ~]$ ssh -oIdentityFile=/home/cltbld/.ssh/b2gbld_dsa [e-mail] mapping checking getaddrinfo for pvtbuilds-zlb.dmz.scl3.mozilla.org [10.22.74.161] failed - POSSIBLE BREAK-IN ATTEMPT!
Last login: Thu Dec  6 12:19:41 2012 from zlb1.dmz.scl3.mozilla.com
[[e-mail]3 ~]$ cd /pvt/mozilla.org/b2g/
[[e-mail]3 b2g]$ mkdir -p /pvt/mozilla.org/b2g/tinderbox-builds/cedar-panda/
[[e-mail]3 ~]$ ls -l /pvt/mozilla.org/b2g/tinderbox-builds/cedar-panda/
total 4
drwxr-x--- 3 b2gbld b2gbld 4096 Dec  6 13:08 test
[[e-mail]3 b2g]$ exit
logout
Connection to pvtbuilds.pvt.build.mozilla.org closed.
[cltbld@bld-centos6-hp-009 ~]$ rsync -e "ssh -oIdentityFile=/home/cltbld/.ssh/b2gbld_dsa" -azv test [e-mail]:/pvt/mozilla.org/b2g/tinderbox-builds/cedar-panda/test/
reverse mapping checking getaddrinfo for pvtbuilds-zlb.dmz.scl3.mozilla.org [10.22.74.161] failed - POSSIBLE BREAK-IN ATTEMPT!
sending incremental file list
created directory /pvt/mozilla.org/b2g/tinderbox-builds/cedar-panda/test
test/
test/README.txt

sent 114 bytes  received 35 bytes  298.00 bytes/sec
total size is 0  speedup is 0.00


Nevertheless, being able to download from pvtbuilds's *pub* folder is doing the job for now.

Comment 11

[Brian Hourigan [:digi]]

Please re-try the rsync/scp/wget, it should be fixed. Ping me on irc if you have any questions

Comment 12

[Armen [:armenzg]]

Sorry digi. I completely spanned after my meeting. I will try this tomorrow.

Comment 13

[Armen [:armenzg]]

* I managed to rsync to it [1]
* I does not have the same permissions [2]
* changing the permissions allows me to see the folder through http:// [3]

What do I need to do to set the permissions to 755 rather than 750?

############

On another note, I believe we should close this bug and open a different one to track switching to upload to pvtbuilds/pvt.

My understanding is that we are investigating how to switch from uploading to:
[e-mail]:/pub
to this:
[e-mail]:/pvt

Is this a correct assesment?

############

[1]
rsync -e "ssh -oIdentityFile=/home/cltbld/.ssh/b2gbld_dsa" -azv test [e-mail]:/pvt/mozilla.org/b2g/tinderbox-builds/cedar-panda/test/
reverse mapping checking getaddrinfo for pvtbuilds-zlb.dmz.scl3.mozilla.org [10.22.74.161] failed - POSSIBLE BREAK-IN ATTEMPT!
sending incremental file list
created directory /pvt/mozilla.org/b2g/tinderbox-builds/cedar-panda/test
test/
test/README.txt

sent 114 bytes  received 35 bytes  298.00 bytes/sec
total size is 0  speedup is 0.00

[2]
[cltbld@bld-centos6-hp-009 ~]$ ssh -oIdentityFile=/home/cltbld/.ssh/b2gbld_dsa [e-mail] "ls -l /pvt/mozilla.org/b2g/tinderbox-builds/cedar-panda"
reverse mapping checking getaddrinfo for pvtbuilds-zlb.dmz.scl3.mozilla.org [10.22.74.161] failed - POSSIBLE BREAK-IN ATTEMPT!
total 28
drwxrwxr-x 3 b2gbld b2gbld 4096 Dec  7 16:00 20121207145552
drwxrwxr-x 3 b2gbld b2gbld 4096 Dec  8 10:20 20121208091253
drwxrwxr-x 3 b2gbld b2gbld 4096 Dec 10 08:05 20121210064555
drwxrwxr-x 3 b2gbld b2gbld 4096 Dec 10 08:39 20121210073754
drwxrwxr-x 3 b2gbld b2gbld 4096 Dec 10 12:21 20121210102505
drwxrwxr-x 3 b2gbld b2gbld 4096 Dec 10 12:21 20121210112102
drwxr-x--- 3 b2gbld b2gbld 4096 Dec 11 08:06 test
[3]
http://pvtbuilds.pvt.build.mozilla.org/pvt/mozilla.org/b2g/tinderbox-builds/cedar-panda/test/

Comment 14

[Brian Hourigan [:digi]]

Your assessment is correct. Lets move to 793152 for the migration from pvtbuilds2 to productdelivery.