'buildbot reconfig' causes WebStatus to give tracebacks for awhile

[bhearsum]

After doing a reconfig, even one that doesn't change anything, WebStatus? stops working for a few minutes. It then magically starts working again. There's nothing in the log to indicate how it recovered. Here's the traceback:

File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/web/server.py", line 160, in process

self.render(resrc)

File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/web/server.py", line 167, in render

body = resrc.render(self)

File "/tools/buildbot/lib/python2.5/site-packages/buildbot/status/web/base.py", line 210, in render

data = self.content(request)

File "/tools/buildbot/lib/python2.5/site-packages/buildbot/status/web/base.py", line 245, in content

data += self.fillTemplate(s.header, request)

File "/tools/buildbot/lib/python2.5/site-packages/buildbot/status/web/base.py", line 239, in fillTemplate

valuestitle? = self.getTitle(request)

File "/tools/buildbot/lib/python2.5/site-packages/buildbot/status/web/waterfall.py", line 417, in getTitle

status = self.getStatus(request)

File "/tools/buildbot/lib/python2.5/site-packages/buildbot/status/web/base.py", line 220, in getStatus

return request.site.buildbot_service.getStatus()

File "/tools/buildbot/lib/python2.5/site-packages/buildbot/status/web/baseweb.py", line 458, in getStatus

return self.parent.getStatus()

<type 'exceptions.AttributeError?'>: 'NoneType?' object has no attribute 'getStatus'

[bhearsum]

It turns out that I can't consistently reproduce this. It only seems to happen with one of my Buildbots.

[dustin]

The reconfig operation is pretty dark magic. It involves divorcing 'old' objects from the object graph, but if they are still in use (e.g., by the web), then problems will ensue. In this case, for example, the web service has been divorced from its parent service. I'm not sure there's a good fix for this problem.

[ijon]

Identical to #139.

[bhearsum]

I've now noticed that if I reload a ton of times (by holding down the keyboard shortcut for 'reload') - it comes back immediately.

[warner]

I'm seeing this a lot at work too.

[dbailey]

I get this relatively consistently.

The most recent occurrence was when I updated the master.cfg file to change the FileUpload? step on the 3 builders defined to use a WithProperties? to set the filename.

Only solution in most of the cases I encounter is to complete restart the buildbot master.

[dustin]

I see this too. My theory is that my browser is using HTTP/1.1 with connection caching, and I'm still connected to the old status object. I'm not sure there's a good solution to this.

[dbailey]

cache-Control directive may solve the problem.

 http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html Look for 14.9

I haven't checked to see if the necessary HTTP headers can be set by buildbot, but since it's using twisted for its own web server, I'm assuming it should be possible.

I haven't read the options in detail to see if there is a nice option to inform browsers that they should ignore any cached output prior to a given time/date (i.e update that value after any reconfig).

The alternative is to request the browser to disable caching.

[dustin]

hmm, I don't like the idea of disabling connection caching altogether just to fix this bug. If anything, this is a bug in twisted -- not terminating existing connections when the service is shut down.

Another solution may be to delay removing the old WebStatus? object from the service hierarchy for some longish time like 5 minutes.

[dustin]

let's see if we can fix this in 0.7.10, eh?

[dustin]

commit 48a0947ad8e829963f9564ab27848a66230f381a
Author: Dustin J. Mitchell <dustin@zmanda.com>
Date:   Wed Feb 25 13:22:05 2009 -0500

    (refs #176) use buildmaster_service.master, not ..parent, so that cached web connections can still get reasonable info