1396768 - nightly partials are not detected

Status: RESOLVED WONTFIX

(SeaMonkey :: Release Engineering, enhancement)

Description

[Edmund Wong (:ewong)]

The nightly runs aren't finding the partials/complete mars, because the get previous 
mar step is not finding the right mar file.  In fact, I think we haven't
updated the mar file download for our nightly as it's still using ssh.
I think it needs to use the S3 copy method.

Need to rewrite the get previous mar steps.

Comment 1

[Edmund Wong (:ewong)]

Attachment: [tools] fix get previous mar step with this script [checked-in]

the rationale of this script is to replace the ssh command for the
get_prev_marfilename step since ssh no longer works.

We need to grab the list of mar files from the latest directory
and return the latest one.

Comment 2

[Edmund Wong (:ewong)]

Attachment: [custom] replace ssh command with script [checked-in w/fix]

Comment 3

[Frank-Rainer Grahl (:frg)]

Comment on attachment 8904814 [details] [diff] [review]
[custom] replace ssh command with script [checked-in w/fix]

LGTM

Comment 4

[Frank-Rainer Grahl (:frg)]

Comment on attachment 8904814 [details] [diff] [review]
[custom] replace ssh command with script [checked-in w/fix]

Doesn't this also download the mar files? If yes wouldn't something like in comment 3 here be faster?

https://superuser.com/questions/545316/getting-all-the-filenames-not-content-recursively-from-an-http-directory

How does TB do it?

FRG

Comment 5

[Edmund Wong (:ewong)]

(In reply to Frank-Rainer Grahl (:frg) from comment #4)
> Comment on attachment 8904814 [details] [diff] [review]
> [custom] replace ssh command with script
> 
> Doesn't this also download the mar files? If yes wouldn't something like in
> comment 3 here be faster?

The point is not download the mar file but check if it exists.  Downloading
the mar file goes to the next step.

As for TB, from what I gather from their logs, they don't do this.

Comment 6

[Frank-Rainer Grahl (:frg)]

> The point is not download the mar file but check if it exists.

That is what I mean. Doesn't the wget implementation in the patch actually downloads it?

According to the docs -spider might be needed?

Comment 7

[Edmund Wong (:ewong)]

(In reply to Frank-Rainer Grahl (:frg) from comment #6)
> > The point is not download the mar file but check if it exists.
> 
> That is what I mean. Doesn't the wget implementation in the patch actually
> downloads it?
> 
> According to the docs -spider might be needed?

No, it downloads the list index of https://archive.mozilla.org/pub/seamonkey/nightly/latest-comm-central-trunk.

However, -spider is also an option, especially with the -l1 parameter.

Comment 8

[Frank-Rainer Grahl (:frg)]

Comment on attachment 8904813 [details] [diff] [review]
[tools] fix get previous mar step with this script [checked-in]

r+++

Comment 9

[Edmund Wong (:ewong)]

Pushed to buildbotcustom:
https://hg.mozilla.org/build/buildbotcustom/rev/a75d59e3670c
https://hg.mozilla.org/build/buildbotcustom/rev/1974796a5bae

Pushed to tools:
https://hg.mozilla.org/users/Callek_gmail.com/tools/rev/ede40f6ab87b

Comment 10

[Edmund Wong (:ewong)]

Attachment: [custom] use archiveServer and not stageServer

Comment 11

[Edmund Wong (:ewong)]

with the archiveServer patch manually applied, it seems like the partials are being generated.

to note: our last complete partial generation was from 2010 March 29th.  So yeah...  7+ years ;/

Comment 12

[Frank-Rainer Grahl (:frg)]

Comment on attachment 8907395 [details] [diff] [review]
[custom] use archiveServer and not stageServer

> So yeah...  7+ years

Becoming better like old Whisky :)

Comment 13

[Edmund Wong (:ewong)]

Attachment: [custom] use archiveServer and not stageServer [bitrot fix] [checked-in]

fixed bitrot while also moved the new parameters to the end of the
parameters list.

Comment 14

[Edmund Wong (:ewong)]

Pushed https://hg.mozilla.org/build/buildbotcustom/rev/179730acc03a to fix a faux pas...

Comment 15

[Edmund Wong (:ewong)]

Attachment: [custom] add archiveServer to CCNightlyRepackFactory

Comment 16

[Edmund Wong (:ewong)]

We no longer use buildbot. So closing as WF.