should have a way of running Python code provided by the master on a slave

[bhearsum]

Right now, ShellCommand? is generally the easiest way to do $CustomTask? on a slave. Using ShellCommand? is often dirty and hard to make cross-platform. Unfortunately, it is difficult to write python code to be run on a slave since it requires upgrading the Buildbot installation there - and a new SlaveCommand?. I think it'd be great if we had something like ShellCommand? -- but that executed Python code. I envision that the master-side BuildStep? would require a function with a defined API and possibly some other information. I'm not exactly sure how this would work on the slave side yet.

[dustin]

There's an additional question of a means of transmitting Python code over the wire. It seems like just sending a string would be adequate.

In general, I'm in support of this idea, but I would want to make sure that the executed Python code was fairly well sandboxed -- not for security purposes, but for purposes of not crashing the buildslave.

This would be especially sweet if the client-side python code could get arbitrary arguments from the server side (rather than a list of strings), and could return arbitrary python objects (rather than zero or more logfiles). This would basically let us implement completely new client-side SlaveCommand objects without modifying any code on the slave side.

[Pike]

I somehow think that putting the python code onto the slave would make things harder to debug. I'm in particular concerned about python version compat here, that bites me so often.

Wouldn't the dynamic python'ism of that defeat the purpose of checking the slave commands up-front on slave startup?

When I was thinking about "hey, this should be easier", I was more thinking of an extra argument to create-slave to add a list of slave command classes to import into buildbot.tac.

[dustin]

It really depends on the control you have over your buildslaves. For the moment, we (zmanda) only run internal buildslaves, so we can make them do whatever we want. But we're going to be recruiting buildslaves from others RSN, and we most likely won't have a login of any sort on those machines, nor even the ability to restart a crashed buildslave instance. So being able to "push" algorithmic complexity to the slave by providing snippets of Python will be helpful. For example, we may want to "explore" the build environment a little bit, and set some build properties to taylor the rest of the build appropriately.

Yes, writing to a minimum Python version will be hard, but that's life :)

[bhearsum]

I'm going to take a first stab at this, once I'm satisfied with it I'll post it for feedback.

[bhearsum]

I've been looking at this off and on and I'm stuck. Seems there's no way to take a complex type like a class or function and pack it up. We could require functions to be in an external file, zip the file to the slave, and run it like that...somehow. But that seems less than ideal. In any case, I won't have time to keep hacking on this for awhile.

[cunger]

imagine the following scenario: we have some python scripts which define and build our projects checked into our repository. upon checkout every developer gets this toolset at their disposal. it would be so cool if I could load this versioned project configuration into the buildbot since also defines the dependencies of the projects and use it to configure the factory on the slave side. this way it would be psossible to re-configure the factories without restarting the master but by updating the configuration inside the repository.

[dustin]

I don't want to rain on the parade, but I think that this is fundamentally _not_ something Buildbot should do. In fact, I don't even think we should have the slave-side Source step implementations. I would rather move in the direction of a simple, clean, remote-execution system. The POSIX API for executing, and monitoring processes is simple and bounded, and the buildmaster should be the one to perform any other processing based on the results of those processes.

As for cunger's proposal -- Buildslaves can't configure a factory anyway, so there's no need for them to dynamically reload their configuration. You could achieve the same effect by (re)loading the configuration on the buildmaster and using it from master_cfg.py

[bhearsum]

I'm curious what "remote-execution system" actually means. Process execution over ssh? I'm sure I'm missing something here.

What would this mean for win32 slaves if we adopt something POSIX compliant?

[dustin]

Sorry, I wasn't clear. It means that we should really only be using buildbot.steps.shell.ShellCommand?.

[Pike]

While I agree with this bug being WONTFIX, I don't agree with the rationale of having to restrict slaves to ShellCommands?.

In my buildbot, I'm running an analysis step which is implemented in python. The output of that step is rather involved and structured, and hard to decypher if written as plain text. If I had to encode all that information into a stdout log on the one side and then parse that again on the other side, I'd be writing a lot of code, compared to just sending the data across via sendUpdate(). Man, I tried to just send it to a custom log, which was a lot of work, for no good output.

It wouldn't buy me anything, too, as I still had to install my code on the slave and make sure that the versions on the slave can talk to the versions on the master.

I really like the idea that I can use my python code on the slave as python code, without having to cumbersomely encode and decode the information that I need out of those builds.

[dustin]

A side-benefit of not running Python on the slave side is that slaves then have *very* few requirements, and there's virtually no need to upgrade. We can probably also write a C version for embedded systems.

As for your case, my recommendation would be to run a Python script (using a ShellCommand?) which takes this complex data structure and pickles it. This is assuming that the easier solution -- shipping the *input* for your analysis to the buildmaster and analyzing it there -- is not possible for some reason.

[Pike]

The step does a syntax and completeness check over the whole source, and possibly even a compilation preparation phase. The output can be as large as  http://l10n.mozilla.org/buildbot/compare/linux-langpack/11907, for example, and is produced by comparing several hundred files from two different repositories.