557760 - Tweaks to the wait times emails

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P5)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

1) Can the "Build/Unittest" wait times emails be changed from:
win32, macosx, linux
... to:
win2k3, osx10.5, centos5, maemo4, maemo5gtk, maemo5qt

2) Can the "Talos" wait times emails be changed from:
leopard, win7, tiger, linux, xp, snowleopard
...to:
osx10.4, osx10.5, osx10.6, WinXP, Win7, fedora12, fedora12x64, maemo4, maemo5

3) The "Talos" wait times emails should probably be renamed to "Unittest/Talos", now that we have started running unittests on these minis.

Comment 1

[Rail Aliiev [:rail]]

Attachment: production-master.b.m.o print_waits.sh changes

Comment 2

[Rail Aliiev [:rail]]

Attachment: production-master02.b.m.o print_waits.sh changes

Instead of `find ... -name '?*platform*' -not -name '?*platform64*'` style find(1) parameters, -regex and strict word boundaries (\b) are used.

Comment 3

[Rail Aliiev [:rail]]

Attachment: sm-try-master.b.m.o crontab changes

"Tagging" this patch as "not sure" because we have "wince-hg" builder, not sure if it "win2k3"...

Comment 4

[Rail Aliiev [:rail]]

Attachment: talos-master.b.m.o crontab changes

Comment 5

[Rail Aliiev [:rail]]

Attachment: talos-master.b.m.o print_waits.sh changes

1. Assuming that linux and fedora, linux64 and fedora64 are the same platforms, I've combined them.

2. Vista addedd

(In reply to comment #0)
> 2) Can the "Talos" wait times emails be changed from:
> leopard, win7, tiger, linux, xp, snowleopard
> ...to:
> osx10.4, osx10.5, osx10.6, WinXP, Win7, fedora12, fedora12x64, maemo4, maemo5

I haven't found any talos job for maemo4 and maemo5. Any idea which builders handle them?

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #5)
> Created an attachment (id=438327) [details]
> talos-master.b.m.o print_waits.sh changes
> 
> 1. Assuming that linux and fedora, linux64 and fedora64 are the same platforms,
> I've combined them.
Great.


> 2. Vista addedd
k.

> (In reply to comment #0)
> > 2) Can the "Talos" wait times emails be changed from:
> > leopard, win7, tiger, linux, xp, snowleopard
> > ...to:
> > osx10.4, osx10.5, osx10.6, WinXP, Win7, fedora12, fedora12x64, maemo4, maemo5
> 
> I haven't found any talos job for maemo4 and maemo5. Any idea which builders
> handle them?

Aki would know.



also, I note that recently the "production talos" for snowleopard are always zero. Did something change there in the last few days?

Comment 7

[Aki Sasaki (not active)]

(In reply to comment #6)
> > I haven't found any talos job for maemo4 and maemo5. Any idea which builders
> > handle them?
> 
> Aki would know.

a) assuming this is run on the master, that would be production-mobile-master.
b) why do we want wait times for mobile? Jay/Stuart aren't asking for this.

Comment 8

[Chris AtLee [:catlee]]

Comment on attachment 438322 [details] [diff] [review]
production-master.b.m.o print_waits.sh changes

Do we need centos5x64 and osx10.6 builder support for production-master.b.m.o?

Comment 9

[Chris AtLee [:catlee]]

Comment on attachment 438323 [details] [diff] [review]
production-master02.b.m.o print_waits.sh changes

I'm not sure how I feel about this since the mobile builds use the same pool of slaves as the desktop builds, the wait times for centos5/maemo* will all be about the same.

Wouldn't it make more sense to count the wait times for maemo along with centos?

Comment 10

[Armen [:armenzg]]

(In reply to comment #8)
> (From update of attachment 438322 [details] [diff] [review])
> Do we need centos5x64 and osx10.6 builder support for production-master.b.m.o?

centos5x64 -> no
osx10.6 -> yes (we enabled 10.6 64 bit builds last week).

Comment 11

[Chris AtLee [:catlee]]

Comment on attachment 438325 [details] [diff] [review]
talos-master.b.m.o crontab changes

I thought we disabled jss and are now running new dromaeo tests?  We should probably be getting the list of builders at run-time like we do for builds.

Comment 12

[Chris AtLee [:catlee]]

Comment on attachment 438327 [details] [diff] [review]
talos-master.b.m.o print_waits.sh changes


> python tools/buildfarm/maintenance/print_waits.py \
>-    -blinux:$LINUX_BUILDERS \
>-    -bxp:$XP_BUILDERS \
>-    -bwin7:$WIN7_BUILDERS \
>-    -bleopard:$LEOPARD_BUILDERS \
>-    -btiger:$TIGER_BUILDERS \
>-    -bsnowleopard:$SNOWLEOPARD_BUILDERS \
>+    -bfedora12:$LINUX_BUILDERS \
>+    -bfedora12x64:$LINUX64_BUILDERS \
>+    -bWinXP:$XP_BUILDERS \
>+    -bWin7:$WIN7_BUILDERS \
>+    -bWinVista:$WIN7_BUILDERS \

Should be -bWinVista:$VISTA_BUILDERS I think?

Comment 13

[Rail Aliiev [:rail]]

Attachment: production-master.b.m.o print_waits.sh changes

(In reply to comment #10)
> (In reply to comment #8)
> > (From update of attachment 438322 [details] [diff] [review] [details])
> > Do we need centos5x64 and osx10.6 builder support for production-master.b.m.o?
> 
> centos5x64 -> no
> osx10.6 -> yes (we enabled 10.6 64 bit builds last week).

osx10.6x64 added

Comment 14

[Rail Aliiev [:rail]]

/me thinks about creating a hg repo for these wrappers and crontab entries.
Not sure if this repo should be private or public.

Comment 15

[Rail Aliiev [:rail]]

Attachment: talos-master.b.m.o print_waits.sh changes

(In reply to comment #12)
> Should be -bWinVista:$VISTA_BUILDERS I think?

Exactly. Fixed.

Comment 16

[Rail Aliiev [:rail]]

Attachment: talos-master.b.m.o print_waits.try-talos.sh.diff

(In reply to comment #11)
> (From update of attachment 438325 [details] [diff] [review])
> I thought we disabled jss and are now running new dromaeo tests?  We should
> probably be getting the list of builders at run-time like we do for builds.

something like this?

Comment 17

[Rail Aliiev [:rail]]

Attachment: talos-master.b.m.o crontab changes

Assuming we use print_waits.try-talos.sh

Comment 18

[Aki Sasaki (not active)]

(In reply to comment #14)
> /me thinks about creating a hg repo for these wrappers and crontab entries.
> Not sure if this repo should be private or public.

Yes please.
It would also help w/ stage cleanup etc.

Comment 19

[Rail Aliiev [:rail]]

Comment on attachment 438716 [details] [diff] [review]
talos-master.b.m.o crontab changes

applied

Comment 20

[Rail Aliiev [:rail]]

Comment on attachment 438715 [details] [diff] [review]
talos-master.b.m.o print_waits.try-talos.sh.diff

applied

Comment 21

[Rail Aliiev [:rail]]

Comment on attachment 438712 [details] [diff] [review]
talos-master.b.m.o print_waits.sh changes

applied

Comment 22

[Rail Aliiev [:rail]]

Comment on attachment 438701 [details] [diff] [review]
production-master.b.m.o print_waits.sh changes

applied

Comment 23

[Rail Aliiev [:rail]]

Comment on attachment 438324 [details] [diff] [review]
sm-try-master.b.m.o crontab changes

applied

Comment 24

[Rail Aliiev [:rail]]

Comment on attachment 438323 [details] [diff] [review]
production-master02.b.m.o print_waits.sh changes

applied

Comment 25

[Rail Aliiev [:rail]]

(In reply to comment #7)
> b) why do we want wait times for mobile? Jay/Stuart aren't asking for this.

John, Aki

do we want to see wait times for the mobile Talos jobs?

Comment 26

[Aki Sasaki (not active)]

(In reply to comment #25)
> do we want to see wait times for the mobile Talos jobs?

Up to John. I don't particularly want them.

Comment 27

[Armen [:armenzg]]

Attachment: production-master.b.m.o print_waits.sh changes add Centos x64 to PM

Comment 28

[Armen [:armenzg]]

Attachment: production-master02.b.m.o print_waits.sh changes add OSX10.6 x64 to PM

Comment 29

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #26)
> (In reply to comment #25)
> > do we want to see wait times for the mobile Talos jobs?
> 
> Up to John. I don't particularly want them.

This came up in context of making mobile more integrated with rest of our infrastructure. Even if mobile group are not asking for this yet, it would be good for us to know how bad mobile wait times are - data like this helps when decisions about buying more phones, so good to have this data in advance.

(Note: if this is complicated/hard to do right now, and should wait until mobile infra is better integrated with desktop infra, thats fine - this is nice to have, not urgent.)

Comment 30

[Chris AtLee [:catlee]]

(In reply to comment #29)
> (In reply to comment #26)
> > (In reply to comment #25)
> > > do we want to see wait times for the mobile Talos jobs?
> > 
> > Up to John. I don't particularly want them.
> 
> This came up in context of making mobile more integrated with rest of our
> infrastructure. Even if mobile group are not asking for this yet, it would be
> good for us to know how bad mobile wait times are - data like this helps when
> decisions about buying more phones, so good to have this data in advance.
> 
> (Note: if this is complicated/hard to do right now, and should wait until
> mobile infra is better integrated with desktop infra, thats fine - this is nice
> to have, not urgent.)

Well, this would represent only mobile build wait times, not test wait times.  We'd need to set this script up on the mobile test master to get test wait times.  This is probably the source of the confusion here.  The patches and questions here have been specifically for the build/talos masters.

We could include the mobile builders into the count of jobs that the host platform is doing.

Comment 31

[Armen [:armenzg]]

Attachment: centralize wait times on one file and example of what a specific master would use

I was going to add few more adjustments for Linux 64 and I thought of centralizing the code.

What do you think?

There would a print_wait.master.sh per master.

Not that I have time to work on it but it was easier to write a sample than trying to explain with words on IRC.

I think it will help with the mobile jobs situation.

Comment 32

[Armen [:armenzg]]

Attachment: talos-master.b.m.o print_waits.sh changes to add Fedora x64

To add Fedora x64 to the list of platforms.

Comment 33

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 442798 [details] [diff] [review]
talos-master.b.m.o print_waits.sh changes to add Fedora x64

>--- print_waits.sh      2010-03-08 22:35:56.000000000 -0800

>+LINUX64_BUILDERS=$(find $MASTER_DIR -maxdepth 1 -type d -name '?*fedora64*' -not -name '*release*' -not -name '*fedora*' -not -name '*l10n*' -printf "%p,")

This returns no matches - anything that matches *fedora64* is removed by the not *fedora*. What's the ? in '?*fedora64' there for ?

Comment 34

[Rail Aliiev [:rail]]

Attachment: production-mobile-master.b.m.o print_waits.sh

Comment 35

[Aki Sasaki (not active)]

Comment on attachment 443317 [details]
production-mobile-master.b.m.o print_waits.sh

The n810s will slowly be migrating over to mobile-master2, but this is good for now.

Comment 36

[Rail Aliiev [:rail]]

Comment on attachment 443317 [details]
production-mobile-master.b.m.o print_waits.sh

deployed on production-mobile-master.b.m.o

Comment 37

[Chris AtLee [:catlee]]

Attachment: Ignore builds with "rebuild" in the reason

Let's ignore rebuilds, as their wait times get mis-represented.

Comment 38

[Rail Aliiev [:rail]]

Comment on attachment 444081 [details] [diff] [review]
Ignore builds with "rebuild" in the reason

Looks fine.

Comment 39

[Chris AtLee [:catlee]]

Comment on attachment 442797 [details] [diff] [review]
centralize wait times on one file and example of what a specific master would use

I like this idea!

Do all our builders on the various masters obey the same naming conventions?

Comment 40

[Armen [:armenzg]]

(In reply to comment #39)
> (From update of attachment 442797 [details] [diff] [review])
> I like this idea!
> 
> Do all our builders on the various masters obey the same naming conventions?

WRT naming, the naming I considered were for pm and pm02 (for _BUILDERS) and talos master (for _TESTERS). I am not sure about the try master and the mobile masters.

Comment 41

[Rail Aliiev [:rail]]

Comment on attachment 444081 [details] [diff] [review]
Ignore builds with "rebuild" in the reason

Could somebody push this changeset, so I can deploy it on servers.

Comment 42

[Aki Sasaki (not active)]

Comment on attachment 444081 [details] [diff] [review]
Ignore builds with "rebuild" in the reason

http://hg.mozilla.org/build/tools/rev/afc26be7f219

Comment 43

[Rail Aliiev [:rail]]

(In reply to comment #42)
> (From update of attachment 444081 [details] [diff] [review])
> http://hg.mozilla.org/build/tools/rev/afc26be7f219

deployed on:

talos-master.mozilla.org
sm-try-master.mozilla.org
production-master02.build.mozilla.org
production-master.build.mozilla.org
talos-master02.build.mozilla.org
production-mobile-master.build.mozilla.org

Comment 44

[Rail Aliiev [:rail]]

Comment on attachment 442797 [details] [diff] [review]
centralize wait times on one file and example of what a specific master would use

A nit:

>+LINUX_BUILDERS="      find $MASTER_DIR $FIND_OPTIONS '.*linux.*'           $FIND_DISCARD"

This will eat the next one. Use '.*\blinux\b.*' instead.

>+LINUX64_BUILDERS="    find $MASTER_DIR $FIND_OPTIONS '.*\blinux64\b.*'     $FIND_DISCARD"

Comment 45

[Rail Aliiev [:rail]]

Back to pool. All of the requested features have been implemented, some refactoring (Armen's idea! :) ) still remaining.

Comment 46

[Aki Sasaki (not active)]

Is anyone going to work on this?
Should we close?

Comment 47

[Armen [:armenzg]]

Let's close. I am not interested anymore on doing the refactoring.