contrib/windows/setup.py does not create fully working buildbot_service.exe

[marcusl]

While it creates working buildbot.exe, and the buildbot_serivce.exe works too. It can't start and gives the following traceback:

BuildBot for directory u'c:\\buildbot' terminated with exit code 255.
Traceback (most recent call last):
  File "boot_service.py", line 200, in <module>
  File "buildbot_service.pyc", line 502, in HandleCommandLine
  File "buildbot_service.pyc", line 482, in _RunChild
  File "os.pyc", line 444, in __delitem__
KeyError: 'PYTHONPATH'

It looks like PYTHONPATH is not defined when running in py2exe env? (I don't know enough about py2exe to debug this currently)

[marcusl]

It seems as PYTHONPATH is not there, so I added a check for that (has_key()), then I get this:

BuildBot for directory 'c:\\buildbot' terminated with exit code 255.
Traceback (most recent call last):
  File "boot_service.py", line 200, in <module>
  File "buildbot_service.pyc", line 502, in HandleCommandLine
  File "buildbot_service.pyc", line 486, in _RunChild
  File "buildbot\scripts\runner.pyc", line 889, in run
  File "buildbot\scripts\startup.pyc", line 73, in start
  File "buildbot\scripts\startup.pyc", line 116, in launch
  File "twisted\scripts\twistd.pyc", line 27, in run
  File "twisted\application\app.pyc", line 374, in run
  File "twisted\application\app.pyc", line 354, in parseOptions
  File "twisted\python\usage.pyc", line 177, in parseOptions
  File "twisted\python\usage.pyc", line 333, in <lambda>
  File "twisted\application\app.pyc", line 276, in opt_reactor
  File "twisted\application\reactors.pyc", line 82, in installReactor
twisted.application.reactors.NoSuchReactor: 'win32'

And that is strange, since win32eventreactor is included explicitly in setup.py.

[dustin]

Sounds like this would be good to fix, but .. I don't even understand the question..

[marcusl]

The contrib/windows/setup.py compiles/packages the buildbot-python code into two win32-executables, buildbot.exe and buildbot-service.exe. These would help windows-deployment quite a lot, as it reduces the amount of dependencies quite a lot, as everything is baked into the exe. (Making an stand-alone installer would be very easy, etc.)

The exes are valid but the buildbot_service.exe fails immediately (see above stack traces). They occur because the runtime environment is a bit different when running in "exe-mode" as compared to the "normal" python.exe-runs-foo.py-on-disk mode.

I fixed one error where it was assumed that PYTHONPATH was in the environment. (first trace). Second, the win32-reactor doesn't get installed/included/imported/initialized properly. I didn't/don't know how to fix that.

Perhaps we should track down whoever created the setup.py in the first place?

[dustin]

 https://sourceforge.net/tracker2/index.php?func=detail&aid=1401121&group_id=73177&atid=537003 looks like mark hammond - mhammond@…. Do you want to get in touch?

[marcusl]

Thanks for looking into that! I've sent Mark an e-mail with a link to this ticket.

[mhammond]

The problem is almost simply a failure by py2exe to detect all the twisted modules correctly, and specifically the win32 reactor. The setup.py in the original patch has:

+includes.extend(""" ... + twisted.internet.win32eventreactor + twisted.web.resource""".split())

Which was to work around this lack of detection for the listed modules, so I expect that needs to be adjusted to reflect a reorg in twisted itself. If you can determine exactly what module the reactor lives in, and it is no longer 'twisted.internet.win32eventreactor', then simply adjust accordingly.

[marcusl]

Hm. twisted.internet.win32eventreactor exists in my twisted install, so it doesn't seem to have moved.

I do get the error (just reprocuced on a fresh pull from dustin's master-branch), even when just trying to run a slave using buildbot.exe.

[mhammond]

twisted.internet.win32eventreactor exists in my twisted install

But is that all which is necessary to get that reactor 'importable'? I'm not sure how twisted attempts to find the reactors, or what diagnostics it offers when they aren't found, but I suspect *some* module is missing.

It might be worth having a look at py2exe's library.zip to see if any obvious part of twisted is missing. Attempting to include the entire twisted package (via 'package=...) might also be worth trying...

[mhammond]

Sorry - make that 'package*s*=...'

[dustin]

The better question might be - how does py2exe figure out which modules are required? I imagine that it imports a base set of modules, but doesn't call any functions, and then looks at sys.modules. Twisted does some kind of delayed processing to pick a reactor. Is there a way to trigger that delayed processing to occur before py2exe takes its snapshot? The packages= idea doesn't sound bad, either.

[marcusl]

With packages=twisted? added to py2exe_options, I get this error:

Traceback (most recent call last):
  File "twisted\python\usage.pyc", line 373, in <lambda>
  File "twisted\application\app.pyc", line 566, in opt_reactor
  File "twisted\application\reactors.pyc", line 79, in installReactor
  File "twisted\plugin.pyc", line 200, in getPlugins
--- <exception caught here> ---
  File "twisted\plugin.pyc", line 179, in getCache

exceptions.AttributeError: ZipPath instance has no attribute 'setContent'

[dustin]

wow, I have no idea what twisted/plugin.py is trying to do. It looks like it's being clever. I wonder if you could monkey-patch that with a simpler version that would work with py2exe?

[marcusl]

Perhaps. I'm a bit busy right now and I don't know how to hack twisted, so I can't address this right away. (i.e this week).

But if we're patching twisted anyway, we might as well include the patch here:  http://twistedmatrix.com/trac/ticket/2726 which is pretty essential on Win32. (Otherwise the slaves are very vulnerable to hung processes, i.e. dialogboxes or otherwise.)

[marcusl]

I'm working on it now. Related ticket is #440.

[marcusl]

Update:

I'm fairly sure the last traceback is actually "safe", albeit performance-hit-incurring, see  http://twistedmatrix.com/trac/ticket/2409. (Twisted want's to write dropin.cache, which it can't, cause it's in a zip.)

I haven't figured out how to coax py2exe to include extra files (such as said cache file) in the library-package yet, but I'm working on it. Telling twisted to store the cache file somewhere else is apparently not something the twisted-devs want to do (part of story here:  http://twistedmatrix.com/trac/ticket/3348)

Anyway, the good news is I've got a buildbot.exe working, and it even has the windows gtk+pygtk stuff in, so debugclient works too. :) I just have to test a bit to make sure all dependent dlls are in there.

It might be a bloated release (currently at 35 mb, as I'm probably including more than I need) but there's no test-suite for the built buildbot.exe, so I'll have to go safe until then.

[marcusl]

I'm dropping this too, as I did with #440.

Partial work on:  http://github.com/marcusl/buildbot/commits/bug273 for anyone who wants to resume it.