If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

TriggerBouncerCheck should survive after reconfig



Release Engineering
Release Automation
7 years ago
2 years ago


(Reporter: rail, Unassigned)


Firefox Tracking Flags

(Not tracked)


(Whiteboard: [releases])


(1 attachment)



7 years ago
The poller itself continues working but can't trigger dependent builders:

        Traceback (most recent call last):
          File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-i686.egg/twisted/internet/defer.py", line 441, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-i686.egg/twisted/internet/defer.py", line 664, in _cbDeferred
          File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-i686.egg/twisted/internet/defer.py", line 318, in callback
          File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-i686.egg/twisted/internet/defer.py", line 424, in _startRunCallbacks
        --- <exception caught here> ---
          File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/Twisted-10.1.0-py2.6-linux-i686.egg/twisted/internet/defer.py", line 441, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/builds/buildbot/rail/buildbotcustom/scheduler.py", line 326, in checkUptake
            Triggerable.trigger(self, self.ss, self.set_props)
          File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/schedulers/triggerable.py", line 66, in trigger
            d = self.parent.db.runInteraction(self._trigger, ss, props)
        exceptions.AttributeError: 'NoneType' object has no attribute 'db'


7 years ago
Priority: -- → P4

Comment 1

7 years ago
Dustin, do you have any idea how to fix this or in which direction should I dig here?
This means that the poller that's running is the *old* poller, from before the reconfig.  When a reconfig occurs, all schedulers that have changed get shut down, and all the new schedulers are started.  When they're shut down, their .parent gets set to None.

So your scheduler should delay the completion of stopService (by returning a Deferred) until the current poll operation is complete, lest it trigger a downstream build *after* the Scheduler has been stopped.

Comment 3

7 years ago
Created attachment 513093 [details] [diff] [review]

Yeah, probably there is no easy way to retrigger this triggerable poller after reconfig/restart with the same values set.

* Not tested in staging, just want to make sure that I understand the issue properly.
* Added stopService method
* Added a standalone builder, w/o a scheduler, for triggering mirror uptake monitoring if someone has to reconfigure the master _after_ push to mirrors and _before_ the pollers finish their work.
Attachment #513093 - Flags: feedback?(dustin)
What about saving state in the DB, and then deciding to trigger on startup or not depending on the saved state?
Comment on attachment 513093 [details] [diff] [review]

This may help, as it looks like previously the LoopingCall would continue forever, even after the scheduler was stopped.

However, this doesn't address the following ordering:
 1. poll() method invoked, get_release_uptake starts
 2. scheduler stopped
 3. uptake is OK and Triggerable.trigger invoked

Since the poll() method is not atomic, you need to be sure it isn't interrupted.  Generally the best way to do this is to use a DeferredLock for the polling operation, and to try to acquire that DeferredLock during stopService and shut down the loop with the lock held.
Attachment #513093 - Flags: feedback?(dustin) → feedback-


6 years ago
Priority: P4 → P5


6 years ago
Component: Release Engineering → Release Engineering: Automation (Release Automation)
QA Contact: release → bhearsum

Comment 6

5 years ago
Back to the pool...
Assignee: rail → nobody


4 years ago
Product: mozilla.org → Release Engineering


3 years ago
Depends on: 1082823
Not going to fix this because we're moving to Taskcluster.
Last Resolved: 2 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.