Ticket #76 (new enhancement)

Opened 3 years ago

Last modified 9 months ago

add mercurial RSS poller change source

Reported by: TedMielczarek Owned by: warner
Priority: major Milestone: 0.8.+
Version: 0.7.5 Keywords:
Cc: Pike, durin42@…

Description

To use a Mercurial repo as a change source, you currently have to install a hook script in the repo. I've written a change source that polls the RSS feed available in the Hg web interface.

Attachments

buildbot-hgpoller.patch Download (4.2 KB) - added by TedMielczarek 3 years ago.
hgpoller.py (unified diff)
hgpoll.diff Download (0.9 KB) - added by bhearsum 2 years ago.
fix traceback when change_list is empty
hgPoller-pushlog-v2.diff Download (9.2 KB) - added by bhearsum 23 months ago.
hgpoller using pushlog

Change History

Changed 3 years ago by TedMielczarek

hgpoller.py (unified diff)

Changed 3 years ago by Pike

  • cc Pike added

r- from me, urllib.urlopen is evil, as it hangs the buildmaster if the server is slow or down. Use twisted.web.client.getPage instead. I have an example patch in #76, though I need to find out why it has test failures. The other comment I have is that you should pick the revision from the RSS and set it in the changeset. I wonder if there's a halfway cheap way to get the files for that revision from hg, any idea?

Changed 3 years ago by Pike

duh, #61 that is

Changed 3 years ago by TedMielczarek

Alright, I can replace urllib.urlopen. I'm not sure what you mean about getting the revision from the RSS, as Mercurial only supports changeset IDs (that I know of), see view-source:http://hg.mozilla.org/mozilla-central/?rss-log

Changed 3 years ago by Pike

As discussed on #developers, the changeset ID, that is, the numbers like d7e93861f3f3 uniquely identify a snapshot of the repository and thus make a good revision to give to the changeset.

Getting the list of files likely requires modifications to hg, which Ted may or may not do at some point in time or another. Or so. Vague enough? ;-)

Changed 3 years ago by Pike

I was wondering, would we end up with a repository on the master anyway? If so, it might be more powerful to just pull (and update?) and to get the changes from the local rep directly.

Changed 2 years ago by warner

  • milestone changed from undecided to 0.7.7

Changed 2 years ago by warner

  • milestone changed from 0.7.7 to 0.7.8

no progress on this yet, bumping to 0.7.8

Changed 2 years ago by bhearsum

fix traceback when change_list is empty

Changed 2 years ago by bhearsum

  • cc bhearsum@… added

Changed 23 months ago by bhearsum

I'm working on a different version of this, that works in conjunction with the pushlog extension ( http://hg.mozilla.org/users/bsmedberg_mozilla.com/hgpoller/index.cgi/file/76f6d3a9078a/pushlog-feed.py). This works much much better with huge pushes/merges. I'm not sure if this will be as useful to others (since it relies on an external extension). I'll post it here when I'm done and let others decide.

Changed 23 months ago by bhearsum

hgpoller using pushlog

Changed 23 months ago by bhearsum

Here's the version we're using now. It requires the 'pushlog' hgweb extension that I mentioned in my last comment. I'm not sure how useful this makes it for others, but I wanted to post it nonetheless.

Changed 23 months ago by Pike

I frown upon urlopen. Can you use twisted's getPage for the actual loading of the feed so that it's asynch? Like we do in bonsaipoller now, http://buildbot.net/trac/browser/buildbot/changes/bonsaipoller.py#L299.

In _parse_changes, I'd do the check for updated first thing, and continue in the loop. Mozilla early-return style, maybe. I'd be tempted to play with a generator function instead of the for loop, too.

In _process_changes, we got the changes in chronological order, didn't we? I see you doing the max over all changes, instead of just the last?

I read that pushlog has the files in the output, we're not reading those in?

Changed 14 months ago by Pike

Note, none of the attached versions are still in use on our setups.

Changed 13 months ago by dustin

  • milestone changed from 0.8.0 to 0.7.+

I'm with Pike re: using getPage. Once that is fixed, and there are docs and tests, this is good to go.

Changed 13 months ago by bhearsum

I might have time to fix this up in the next month. No promises, though.

Changed 11 months ago by bhearsum

  • cc bhearsum@… removed

Changed 9 months ago by durin42

  • cc durin42@… added
Note: See TracTickets for help on using tickets.