Work in progress

For the love of god why?

I had some free time so decided I wanted to combine learning more about the Trac project, contributing something useful back to the community, and benefit myself and others in Greenpeace all in one go. Okay okay I love a problem and programming isn't in my blood but should be.

The task: create or update a ticket in Trac when an email is received. Simple? Certainly should be, Trac already has a user contributed script to do the job albeit the version for Trac 0.9.6 (latest released) is very basic compared to the newer version for the as yet unreleased Trac 0.10. However, by the time I'd found that script I'd already written v1 of mine and identified some requirements not met by the existing script, and then I spotted the cake I wanted to have and eat ... the Scheduler was born.

A child is born...

I now have not one but four newly created Trac plugins:

• Email To Ticket - Uses IEmailObserver to process inbound email into a ticket create/update.
• Email Receiver - can receive email from a local MTA, or fetch email from remote POP/IMAP account. Provides IEmailObserver.
• Scheduler - Provides IScheduledTask which any component can implement to have its code 'run' at a time or times of its choosing. Used by Email Receiver to regularly check remote inboxes.
• Ping - Abstraction which connects the passage of time in the outside world to Trac. Provides IPingObserver which Scheduler uses to check if tasks need to be run. Pings can be caused by an external Python script which can either talk to Ping via a HTTP POST to /ping or by loading the component code directly and executing ping().

These connect to each other in sequence building from a tiny piece of functionality into something much larger and more useful. Anybody wanting to build on this work can just take whatever they need - just need to run a script regularly, just use Ping - want more complex scheduling of tasks, use the scheduler.. and so on.

    Python script
        -> Ping
            -> Scheduler
                -> Email Receiver
                    -> Email To Ticket

The plugins provide pages for the TracWebAdmin plugin so that the Scheduled job queue can be seen manipulated and the plugin settings can be configured / remote inbox connections tested.

What's left?

Lots. All the plugins are working but half baked, no test cases (bad Ximon), IMAP code exists but not committed in the Email Receiver yet, no SSL support for POP or IMAP yet, more admin page functionality is needed, testing on more systems (so far tested with TracD/Apache and SQLite/PostgreSQL all on Windows).

I'm also still on the fence about four key issues:

1. How to handle pings and emails when there are multiple projects - especially how to route email to the right project. Request Tracker has this easier because ticket numbers are (IIRC) RT wide, not per queue, but in Trac each queue has its own ticket id sequence meaning ticket 52 could be for any project. Inbound email can be split by account, inbox, address separator (postfix), X-Trac-Project email headers, subject line pattern matching, etc.
2. Where the TracWebAdmin code belongs. TracWebAdmin is slated for Trac 0.11 and currently has some problems with plugin load order. Also my plugins don't _need_ TracWebAdmin so asking customers to install it to use my plugins is harsh - I have coded them so they can operate without TracWebAdmin although easy_install and setup.py took some figuring out. Packaging the web interfaces as separate plugins bumps the number of plugins from four to eight! I suppose I could supply the downloads in two versions - with and without web admin pages.
3. Whether running Trac code outside the webserver is wise (only happens if loading component code outside of the webserver - using HTTP POST to /ping doesn't have this problem but does consume a request thread, conceivably all of them if the scheduler doesn't come back due to a bug in the scheduler or a buggy scheduled task :( )
4. Integration with persistence mechanisms in web servers / use of existing scheduler functionality. AOLserver has ns_schedule_proc, Mod Python and FastCGI keep the Python code memory resident which might be beneficial in some way, etc. Also in this category is the use of threading in the scheduler to both allow tasks to be timed out and to allow quick return to caller.

I guess in a week or so I'll have v1 licked and will post about it and will link to the Subversion repository for you all to enjoy - whether it is finished or not. Oh and I have to stop renaming things - I just realised in the webadmin interface Ping is called Heartbeat now but not in the code...

Screenshots

Scheduler	Heartbeat
Email Receiver	Ticket via Email

The scheduler screen shot shows a job that is 10 hours overdue - this is because I wasn't running the python script that 'pings' the heartbeat mechanism which in turn prods the Scheduler. This script would be invoked by say cron on Linux, or Task Scheduler on Windows, or can be left running since it can sleep between successive pings - but in this latter mode it needs a restart whenever the web server needs a restart for the same reason, to pick up code and configuration changes. The latter mechanism is the most portable however since it requires no knowledge of the serving operating system.