1191352 - Run purge_builds from runner on windows

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

We're hitting this a lot on releases, and I assume we're hitting it elsewhere too. Eg: http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/40.0-candidates/build2/logs/release-mozilla-release-win32_beta_update_verify_3-bm82-build1-build2.txt.gz

Comment 1

[Chris AtLee [:catlee]]

runner _should_ be regularly cleaning up old build directories...it's not clear here if we're deleting these dirs because they're old, or because we don't have enough space.

We should add more logging here to indicate why we're deleting these, and also how much space we had at the beginning.

Also, we need to dig into runner logs to see if purging isn't working as expected there.

Comment 2

[Chris AtLee [:catlee]]

Looks like we're not running purge_builds under runner on windows yet. We should!

Morgan, Mark: Any ideas about how hard this is to set up?

Comment 3

[Mark Cornmesser [:markco] OOO 2024/04/15]

From looking at the purge build task in Puppet, it looks like we could implement it without much difficulty, but I defer to Morgan in case there is an issue there that I am unaware of.

Comment 4

[Morgan Phillips [:mrrrgn]]

(In reply to Chris AtLee [:catlee] from comment #2)
> Looks like we're not running purge_builds under runner on windows yet. We
> should!
> 
> Morgan, Mark: Any ideas about how hard this is to set up?

I would like to see puppet + runner up and going before we add it into the mix; but once that's deployed, it should straightforward to whip up a task and put it in puppet. :)

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Morgan Phillips [:mrrrgn] from comment #4)
> (In reply to Chris AtLee [:catlee] from comment #2)
> > Looks like we're not running purge_builds under runner on windows yet. We
> > should!
> > 
> > Morgan, Mark: Any ideas about how hard this is to set up?
> 
> I would like to see puppet + runner up and going before we add it into the
> mix; but once that's deployed, it should straightforward to whip up a task
> and put it in puppet. :)

Any idea the timeline on this? We can wait a bit, but if it's going to take ~1 month we should do something else in the meantime - this is actively bogging down releases, and I'd like to see it fixed one way or the other before 41.0 goes to build.

Comment 6

[Amy Rich [:arr] [:arich]]

Mark, now that we have runner working on GPO and puppet, can you take a look at this? It's burning release builds as well as normal stuff (so GPO more important than puppet, but when done for one, it should be easy to add to the other).

Comment 7

[Mark Cornmesser [:markco] OOO 2024/04/15]

I am looking into this today.

mrrrgn: from looking at the current POSIX task, what would be the Windows equivalent for $SLAVEDIR? 


# Call purge builds to make sure we have at least 10G free
TOOLS=$($RUNNER_CONFIG_CMD -g hg.tools_path)
SLAVEDIR=$($RUNNER_CONFIG_CMD -g buildbot.slave_dir)
cd $SLAVEDIR
python $TOOLS/buildfarm/maintenance/purge_builds.py --not info --not 'rel-*:10d' --not 'tb-rel-*:10d' -s <%= @required_space %> $SLAVEDIR

Comment 8

[Morgan Phillips [:mrrrgn]]

(In reply to Mark Cornmesser [:markco] from comment #7)
> I am looking into this today.
> 
> mrrrgn: from looking at the current POSIX task, what would be the Windows
> equivalent for $SLAVEDIR? 
> 
> 
> # Call purge builds to make sure we have at least 10G free
> TOOLS=$($RUNNER_CONFIG_CMD -g hg.tools_path)
> SLAVEDIR=$($RUNNER_CONFIG_CMD -g buildbot.slave_dir)
> cd $SLAVEDIR
> python $TOOLS/buildfarm/maintenance/purge_builds.py --not info --not
> 'rel-*:10d' --not 'tb-rel-*:10d' -s <%= @required_space %> $SLAVEDIR

Looks like the windows equivalent is /c/builds/moz2_slave

Comment 9

[Mark Cornmesser [:markco] OOO 2024/04/15]

OK. That is what I figured. Just wanted to verify.

Comment 10

[Mark Cornmesser [:markco] OOO 2024/04/15]

Attachment: 1-purge_builds.bat

This may be relatively straight forward. It seems that the purge_builds.py will work on Windows without changes.

I think will just need to copy the purge_builds.py to the local machine and include the attached bat file.

Comment 11

[Morgan Phillips [:mrrrgn]]

Comment on attachment 8662020 [details]
1-purge_builds.bat

+ Given that this works in tests. I would prefer to see /C/builds/moz2_slave living inside of moco-config.pp along with its Linux counterpart before it's pushed.

Thank you ! :)

Comment 12

[Mark Cornmesser [:markco] OOO 2024/04/15]

Will do. Initially I need get the gpo pieces in place. 

I will probably push the gpo portion out to a small test pool tomorrow AM.

Comment 13

[Mark Cornmesser [:markco] OOO 2024/04/15]

This was deployed first to small set of try machines, and is now deployed to all domain machines that are with in the try OU. 

The GPO changes consists of adding file copy for the attached 1-purge_builds.bat and a current copy of purge_builds.py.

Comment 14

[Mark Cornmesser [:markco] OOO 2024/04/15]

Has been implemented in AWS.