Ticket #76 (new enhancement)

Opened 1 year ago

Last modified 7 months ago

add mercurial RSS poller change source

Reported by: TedMielczarek Assigned to: warner
Priority: major Milestone: 0.8.0
Component: changesources Version: 0.7.5
Keywords: Cc: Pike, bhearsum@mozilla.com

Description

To use a Mercurial repo as a change source, you currently have to install a hook script in the repo. I've written a change source that polls the RSS feed available in the Hg web interface.

Attachments

buildbot-hgpoller.patch (4.2 kB) - added by TedMielczarek on 08/03/07 06:35:06.
hgpoller.py (unified diff)
hgpoll.diff (0.9 kB) - added by bhearsum on 01/31/08 05:37:18.
fix traceback when change_list is empty
hgPoller-pushlog-v2.diff (9.2 kB) - added by bhearsum on 05/02/08 06:58:26.
hgpoller using pushlog

Change History

08/03/07 06:35:06 changed by TedMielczarek

  • attachment buildbot-hgpoller.patch added.

hgpoller.py (unified diff)

08/03/07 06:41:58 changed by Pike

  • cc set to Pike.

r- from me, urllib.urlopen is evil, as it hangs the buildmaster if the server is slow or down. Use twisted.web.client.getPage instead. I have an example patch in #76, though I need to find out why it has test failures. The other comment I have is that you should pick the revision from the RSS and set it in the changeset. I wonder if there's a halfway cheap way to get the files for that revision from hg, any idea?

08/03/07 06:47:39 changed by Pike

duh, #61 that is

08/03/07 07:02:19 changed by TedMielczarek

Alright, I can replace urllib.urlopen. I'm not sure what you mean about getting the revision from the RSS, as Mercurial only supports changeset IDs (that I know of), see view-source:http://hg.mozilla.org/mozilla-central/?rss-log

08/03/07 08:58:33 changed by Pike

As discussed on #developers, the changeset ID, that is, the numbers like d7e93861f3f3 uniquely identify a snapshot of the repository and thus make a good revision to give to the changeset.

Getting the list of files likely requires modifications to hg, which Ted may or may not do at some point in time or another. Or so. Vague enough? ;-)

08/27/07 12:29:25 changed by Pike

I was wondering, would we end up with a repository on the master anyway? If so, it might be more powerful to just pull (and update?) and to get the changes from the local rep directly.

09/29/07 01:47:27 changed by warner

  • milestone changed from undecided to 0.7.7.

12/21/07 20:35:25 changed by warner

  • milestone changed from 0.7.7 to 0.7.8.

no progress on this yet, bumping to 0.7.8

01/31/08 05:37:18 changed by bhearsum

  • attachment hgpoll.diff added.

fix traceback when change_list is empty

03/19/08 10:09:11 changed by bhearsum

  • cc changed from Pike to Pike, bhearsum@mozilla.com.

04/29/08 08:37:33 changed by bhearsum

I'm working on a different version of this, that works in conjunction with the pushlog extension (http://hg.mozilla.org/users/bsmedberg_mozilla.com/hgpoller/index.cgi/file/76f6d3a9078a/pushlog-feed.py). This works much much better with huge pushes/merges. I'm not sure if this will be as useful to others (since it relies on an external extension). I'll post it here when I'm done and let others decide.

05/02/08 06:58:26 changed by bhearsum

  • attachment hgPoller-pushlog-v2.diff added.

hgpoller using pushlog

05/02/08 06:59:36 changed by bhearsum

Here's the version we're using now. It requires the 'pushlog' hgweb extension that I mentioned in my last comment. I'm not sure how useful this makes it for others, but I wanted to post it nonetheless.

05/02/08 10:42:05 changed by Pike

I frown upon urlopen. Can you use twisted's getPage for the actual loading of the feed so that it's asynch? Like we do in bonsaipoller now, http://buildbot.net/trac/browser/buildbot/changes/bonsaipoller.py#L299.

In _parse_changes, I'd do the check for updated first thing, and continue in the loop. Mozilla early-return style, maybe. I'd be tempted to play with a generator function instead of the for loop, too.

In _process_changes, we got the changes in chronological order, didn't we? I see you doing the max over all changes, instead of just the last?

I read that pushlog has the files in the output, we're not reading those in?