Main | May 2006 »

April 30, 2006

Getting started with Django on Windows

A short (?) story about how I got Django working on my Windows 2000 laptop.

I wanted to get Django working quickly and easily so that I could, if in future necessary, work on a Django project locally rather than edit files remotely on a Linux server. Working locally has certain limitations but is sometimes a god send.

I used the Python 2.4.3 Windows installer and the Django 0.9.1 tar ball, the latter being installed by unpacking it (you can use something like WinRAR for this) then invoking setup.py install. Since I'm just testing if I can get going with the basics of Django on my laptop I'll stick with the built-in webserver for now and worry about Apache later.

Installing Python & Django

Neither python.exe nor django-admin.py are in my path. Searching for django-admin.py turned up a wrapper to it in the Scripts subdirectory of my Python installation. I started a new project like so (in a command shell, start menu -> run -> type cmd and press OK):

c:\ximon\installed_apps\python_2.4.3\python.exe c:\ximon\installed_apps\python_2.4.3\Scripts\django-admin.py startproject tut1
Optional:

What a mouthful. As Valter says on the Installation Guide page of the Django Website you can simplify this:

One trick to make life easier on Windows:

- append ".py" to PATHEXT (Settings - Control Panel - System - Advanced - Environment...)
- append C:\Program Files\Python\Scripts to PATH

This way you can run django-admin.py by just typing "django-admin".

That wasn't enough on my Windows 2000 system, I also had to change the file association (an editor had it) to python.exe (right click on django-admin.py, properties, opens with... change, python). Then the complex command above becomes as simple as:

django-admin startproject tut1

Setting up a database

I chose not to use MySQL since that requires the .NET framework in order to build and install MySQL for Python. Instead I chose SQL Lite since pysqlite has Windows precompiled downloads for Python 2.4 available. I used pysqlite 2.2.2 and SQL Lite 3.3.5.

I edited settings.py in my tut1 project directory (the one created by running django-admin startproject) so that I had:

DATABASE_ENGINE = 'sqlite3'

Starting Django is then just a case of doing:

cd \ximon\data\django_apps\tut1

manage runserver

Tada! At http://127.0.0.1:8000/ you should now have a running Django application, albeit absolutely minimal.

Writing your first Django app, part 1

Now I'm following the tutorial on the Django website. First, edit settings.py to tell Django about my database.All I have to do for sqlite is tell it my DATABASE_NAME which I set to the full path to where I want sqlite to store my database files (c:\ximon\data\sqlite_dbs\tut, but be warned: Django wants forward slashes, not backslashes!) which is the same as the name as my project. You don't have to name them the same thing. Make sure the path to the file exists (in my case I had to create the sqlite_dbs directory since it didn't exist yet). Then tell Django to initialise:

cd \ximon\data\django_apps\tut1

manage init

I continued with the steps of the Django tutorial. The tutorial uses the project name myproject whereas I am using tut1 and so my INSTALLED_APPS entry in settings.py looks like this:

INSTALLED_APPS = ( 'tut1.polls', )

Other than that I completed tutorial one without a hitch. Django running on Windows, tick. Next!

Steps to starting a Django application

django-admin startproject PROJNAME

cd PROJNAME

edit settings.py
- set DATABASE_ENGINE
- set and DATABASE_NAME
- add 'django.contrib.admin' to INSTALLED_APPS

edit urls.py
- uncomment the line after # Uncomment this for admin:

mkdir PATH_TO_DATABASE_FILES (for sqlite)

manage init

manage install admin

manage createsuperuser

manage startapp APPNAME

manage install MODEL

manage runserver

April 26, 2006

OSS Watch : Slides

The presentations delivered at OSS watch are now available online

http://www.oss-watch.ac.uk/events/2006-04-10-12/

For stylish use of powerpoint I suggest Prodromos Tsiavios' presentation on Creative Commons and it's relationship to Open Source software.

April 24, 2006

Internal Competition

Given that Greenpeace is the kind of organisation that sets a lot of store by things like community, democracy and five year plans you'd be forgiven for thinking that we don't have a lot in the way of internal competition. However you'd be wrong. Because Greenpeace is organised as a kind of franchise of 27 separate organisations there's no shortage of opportunities for competing solutions to develop, leading to an awful lot of 'cyberdiversity' within our systems.

Not everywhere though. Where a service proves itself best in class it tends to get adopted widely, sometimes even universally, indeed once Metcalfes law kicks in it's easy for a system to become 'best for us' even if it's not going to be 'best in class'. One system which is on it's way to such success is our content management system, which is known by the name 'Planet 2' and is currently in use by a large majority of our national offices, with more coming on board all the time.

One office considering moving to Planet 2 is the UK office, and they've engaged the services of Important Projects to help them make the decision. Which means our homegrown OpenACS solution is going to be duking it out with Drupal and Plone based tools for the right to deliver www.greeenpeace.org.uk . These kind of evaluations are a great way for local offices to keep us at the centre aware of whether we're delivering things they actually want, and make us work hard to deliver value through to the offices that rely on us.

I also like these contests. Drupal and Plone are newer platforms than Open ACS, and since they'll only have to support the UK office can offer more flexibility than our one stop shop. However when it comes to comparing a custom implementation to off the shelf software I'm reminded of something Fred Brookes wrote

"The real tiger is never a match for the paper one, unless actual use is wanted. Then the virtues of reality have a satisfaction all their own."

April 22, 2006

Bug tracking with Subversion?

So I had a crazy idea tonight - I thought I'd implement a bug tracker using Subversion. Why go to the trouble of trying to track changes to ticket content over time and changes to ticket metadata when Subversion can do that for me, AND it can handle binary metadata.

Top level directories can be queues, files can be tickets, revisions can be posts to the tickets with the commit message being the body text and the content being a basic attachment and Subversion properties can track things like ticket metadata, e.g. priority, the action performed on the ticket (e.g. 'comment', 'resolve', 'close', etc.). I mimicked ticket 676 from one of our trackers (I saved it as HTML here in Subversion).

Then I realised that svn log --xml plus an XSLT stylesheet gives me the basic view of a ticket almost for free.

Slam it into a rails app to issue some subversion commands to manipulate the data in the repository and i'm done, well for a first effort. I'll try the rails bit tomorrow but i already got the repository+xml+xslt bit working! I'm not pulling the revision properties out of Subversion yet, that can be done by the rails app, and injected into the XML that gets transformed perhaps.

The only bit I don't like so far is using the commit message as the body of the post to the ticket, it works great for the svn log --xml but other than that feels wrong.

Why rails? Errm, good question. No reason except it can generate the forms for manipulating the new ticket data and editing the old for me. I could do it in Python and talk to the Subversion Python API, I could even use Django!

April 15, 2006

Virtual framebuffers, FireFox, Selenium and ImageMagick

Some time soon I hope to setup selenium running on our dev server as an additional means of testing our applications that have instances on there. The dev server is headless (no X server) so there's a copy of FireFox 1.0.7 and a virtual framebuffer (Xvfb) which I hope to use instead. I haven't gotten that far yet but I did try a test yesterday using ImageMagick - I wrote a little bash script (shudder) to capture the front page of every instance of an application running on the server and present them as thumbnails in a HTML page. It works a treat! Let's hope I can get Selenium working as easily.

The core commands required if you want to try this on your own server are:

> Xvfb :1 -screen 0 1024x768x24 &
> DISPLAY=:1 firefox http://whatever.url.you/want/to/see &
> DISPLAY=:1 import -window root captured.png
> convert captured.png -resize 256 resized.png
> killall firefox
> killall Xvfb

The import and convert commands are part of ImageMagick. The resize will determine the vertical dimension automatically if omitted. It's a pretty dumb sequence of commands but you get the point.

I got the information I needed from this page on the semicomplete blog by Jordan Sissel.

Design flaw?

I just noticed that it in our Rhubarb on Rails demo it is possible to submit a letter with a blank subject and/or body text. This shouldn't be possible but tthe functional tests are incomplete and the unit tests for the letter model don't exist. This is what happens when coding as part of an exercise when there are other things to concentrate on in the time available and the code isn't going to be deployed straight away, or perhaps at all. Still, habits are hard to shake and not writing tests is not a good habit, something we need to work on.

So, I added subject and body to the requires_presence_of validation but then find that (a) the fields in the form don't get the fieldWithErrors div treatment and (b) I can never submit the form because even with values the subject and body are thought by Rails to be empty. I also added test cases while I was at it. However I haven't committed these changes. Why? The functional tests fail thanks to the inability to submit the form even when the subject and body are non-empty.

I can see what is wrong, but I don't know the right way to solve it. It's to do with the fact that the form is populated from two different sources, a letter template and values previously entered by the user. Surely our code should not be manually remembering and populating the subject and body text fields? There is something quite wrong here.

Any thoughts? Take a look at the model, view and controller files in question.

April 12, 2006

Rhubarb on Rails demo improved!

A couple of minor commits occured tonight. Firstly the CSS upload feature turned out to be 99% complete but not committed so that was finished and committed - it needs more tests and more functionality but the basic idea is there and working. Secondly, a massive single character change - the intro text is no longer HTML escaped so that the intro can be formatted. These two changes combined mean the demo is now much cleaner!

OSS Watch : Open Source in a Company Business Model

Next up is a presnetnation by Kit Blake of Infrae who will be talking about Open Source in a Company Business Model. He will be followed by Jim Farmer, talking about the comercialization of Open Source

Lecture notes follow

Lecture Notes : Infrae
---------------------------

Not that many differences between open source and closed source business models. Infrae is based in Rotterdam! they build content management tools mostly based on Zope.

No income from selling software, only from provision of services. Spending on sales is limited by the model, promoting the software is problematic when people can use it for free. You have to be sure you're going to get some kind of services deal out of the end.

Suggests that OS ensures honest business practices 'We never sell the same feature twice'.

The basic plan was to be just like a proprietary software provider, but sacrifice the 20-30% of revenue you expect from licenses and make it up with more development and consultancy. Initial market was universities - the company already had contacts and the IT departments knew about Open Source, this is less of an issue today, more people understand what is happening.

[ Third? appearance of the technology adoption life cycle graph at this conference ]

Talks about the benefit of having networks around the technology you're working with. (In this case Zope). Goes on to talk about the need to get customers talking and build networks both social and technical (customer issue trackers are open to all customers!). Use Trac as an issue tracker, sprints are good for bringing developers together - based on a XP basis - have now rolled this out to customers as well bringing customers into the development directly.

[Nice to see another appearance of XP at this conference. So far nothing but good things from the people who've been trying it]

Open source customer relations tend to be different because they can watch the repository, ToDO lists and so on. This works with technical customers, but doesn't work so well with commercial clients.

Back to the crossing the chasm stuff (moving from early adopters to early majority). Part of doing this was about providing 'the whole product' - documentation, training, support, debugging and so on. "Services have to be excellent, communication is the first complaint" - RTFM is not acceptable :-)

Probably some significant benefits to being on 'neutral ground' rather than your own repositories etc.

Notes on launching an OS project
----------------------------------------
Place in a neutral environment
Have a leader who answers *every* query
Community is either going to happen or it won't

Communities need constant attention and infrastructure
Customers don't order refactoring - only features
New customers need a lot of coaching
On the plus side ongoing maintenance etc. can become a significant income stream

Lecture Notes : Jim Farmer, GeorgeTown University
---------------------------------------------------------------
Starts by saying that discussions with industry suggested that Higher Education is not seen as a good customer for commercial companies. A survey suggested that what people saw as problematic were

...
Going too fast, notes are about to stop... which is a shame because this is some really good stuff on business valuations. [Malcolm Gladwell's ideas make their first appearance, universities are full of Mavens quite well off for connectors...]

"O'reilly is the branding agent for Open Source"

OSS Watch :

Three talks in this morning's plenary sessions. One delivered without slides or notes, which is always nice to see but it wasn't much we haven't talked about already. The second is from Aingaran Pillai of Camden city council talking about a system they've developed on Open ACS... The final talk is from Bill Oliver of JISC

Lecture Notes follow

Lecture Notes : APLAWS
-------------------------------
APLAWS is a CMS, based on the latest version of red hat's content management system. (which is open ACS reimplemented in Java?). Made extensive use of open standards including WAI AA for accessibility. 30 local authorities and around 10 international organisations using the software.

Things they've found they need are : Documentation, open standards, structured development, interoperate with other open source software, release framework, single official release, code submission criteria, single public repository and a published release schedule.

User communities are involved in the sustainability of software. There is a lot of benefit to getting people into the same room to collaborate and work together. In descending order of size you have...

Broad community > User Group > Technical steering committee (with write access to code repository), User steering committee

The user groups for APLAWS meet every three months.

Supplier communities were built through a process of open tenders. The suppliers who provide support are required to QA upgrades provided by the developers and then support the whole system. This beats asking developers to support their own code indefinately. (Also makes it easier to work with overseas / unproven suppliers). Provide the suppliers with access to the user group as it makes them happy.

OS license - chose the GPL, now under the lesser GNU Public LIcense. The problem with the GPL was that suppliers weren't interested in assigning copyright back to RedHat, (too much work).

Importance of having a single website to publicise a project and provide a definitive source of information.

Lecture Notes : JISC
-------------------------
JISC want us to tell them about what they do. This is probably really useful for the Higher Education lot, but of limited use to Greenpeace. Pretty good that this is the first session we have little interest in from the Greenpeace side, which isn't bad since it's only 20 minutes.

April 11, 2006

Oh for a ready Rhubarb on Rails!

Today alone I've had three requests for an online letter writing action to be setup, and more for next week!

OSS Watch : Creative Commons and Open Source, Prodromos Tsiavos

Will talk about the creative commons licenses and how they're used. Prodromos is a lawyer, so this will have a grounding in law. Will look at the regulatory structures and how they evolve.

Lecture notes follow

Lecture Notes
----------------

CC was founded in 2001, but only got going late 2002. Was a response to changes in copyright law and the results of the Eldred case which sought to limit the ongoing expansion of copyright control.

CC tries to bring a middleground between no copyright and full copyright. The intention is to provide user friendly interfaces to licenses that are human readable. These licenses should become a global standard, less fragmented than the existing copyright laws. This goal has been partially achieved in that the licenses are interoperable across countries so a single CC deed will have different legal texts in different countries.

Basic idea is that you assert only some of your rights under copyright legislation.

[ Long evolution of CC licenses in the UK ] In England and Wales a licence was agreed in 2004 which adequately reflected the US one. Took until July 2004 to get a human readable version. There was then some work to align things with the BBC's creative archive license and then work to address the different legal jurisdictions in the UK... In April 2006 there will be UK license 2.5

Very interesting section on both CC and the FSF as agents of change within society who do not work through 'revolutionary' methods, but by providing alternatives and routes out of various legal boxes. Followed by another interesting section on the stages of computing - but I didn't get the name of the model being used.

Goes on to look at the ways the GPL breaks down the traditional relationship whereby a developer transfers their rights to an organisation, and an organisation licenses those rights to an end user.

Once you have an easy to tool to manipulate your licensing rights you are likely to use it. It changes the way you think of things and it starts to result in more engagement in the political process. CC and the FSF have been instrumental in debates about things like patent law, which previously attracted almost no interest from the public.

"We do not produce licenses, we produce communities debating about licenses"

[Podromos is winning the award for most stylish use of ppt today...]

OSS Watch : Walking into the Sunset - Elliot Smith

Elliot Smith is going to explain to us how to choose open source software. This sounds like it could be useful...

Lecture Notes follow

Lecture Notes
-----------------

Installing Drupal took about 15 minutes. Result was a 'gleaming machine', which he then filled with stuff. Upgrades occurred, which he didn't use. Result was that within a year the software was 'a pile of rust and junk'. This was a problem of the user not keeping things up to date...

So, how do you create a productive relationship with a piece of software?

Won't be talking about skills or policies. Will be talking about resources - what will I need and how can I tell? You will need to upgrade, extend and fix the software. But that goes for anything. Suggests that it's easier to estimate this stuff if you're using Open Source software because you can see it happening in the open.

Open Source does mitigate the risk that software might break and your vendor won't fix it. You always have the option to do it yourself.

If you're evaluating Open Source you have to engage with the community / content. So read SourceForge, Freshmeat etc. and track projects around the area. You have to get engaged to understand this stuff.

How do you tell the prospects of Open Source - in basic terms, is it being used and is it still being developed? The latter is more important, because Open Source tends to die if it's not being actively developed. Remember that this has to be balanced against what you're planning on doing with the software - dead projects have their uses.

Scalability - not the most important thing in the world. Worry about it once it becomes a problem [hmm. Not too sure about that one]

Numbers to look for : Bug reports - how many and how quickly are they fixed; Code check in - how often does this happen and what proportion are bug fixes; How many people are submitting patches; How many downloads has it had; How much traffic does it get.

Other things - look up the developers, what else do they work on, do they blog? Do they say the project is stable and do they ever expect it to be? Does it ship with anything out of the box?

Formal models for software evaluation : Open Source Maturity Model, Business Readiness Rating, Woods and Guliani (book), JISC QA Focus... [All seem to work. Might want to look into these in more details.]

Examining code quality : Is there an API, is the code modular, is there a set of coding standards in use, consistency of naming files / variables, is it documented, comments in the code, presence of a test suite (and coverage / quality of this).

Before patching anything see if anyone has done it already. Then log a bug report / feature request. Then start work on patching this, and once you've done it encourage the community to use it.

OSS Watch : Micropayments for Microtasks - Felix Klee, Linuxburg

This is about how you can finance the development of small features in OS projects.

Lecture Notes follow

Lecture Notes
-----------------

Micropayments are a few dozen euro / pounds etc.
Microtasks are things like installing software, developing small extensions <1 hour? Note that there are plenty of microtasks available to the world that don't involve open source software.

Existing solutions include things like donations, ad-hoc payments, codeweaver pledges, google answers and so on. There are problems with all of these approaches.

[See Affero for some interesting donor things]

Solution for micropayments needs to be simple, quick, trusted, independent.

Suggested heavyweight solution

This is probably not feasible.

Suggests a central hub where requests are posted with details of the payment on offer. One of the important possibilities is that other individuals with the same request can add their payment to the total on offer. Developers need to be able to address tasks and claim the payment for working on them - there needs to be a level of trust etc. in the system.

Suggested lightweight solution

This may be feasible

Similar to the above, but no payment channels and no hubs. Messages would be posted in public, signed with PGP to establish an identity. URL tags would be used to specify the tasks and payments. Additional requestors can sign up by adding additional payment tags.

This would work better if it was supported / understood by search engines
Something less clunky than PGP would help as well
Guaranteed payments - escrow?

>> Sounds like a web 2.0 thingumy to me. Or an obvious addition to e-Bay... [Martin]

OSS Watch : Highs and Lows of Open Source

In the spirit of yesterday's Dana Blakenhorn comment James Dalziell is not going to be 'pretending' during his presentation about the highs and lows of outsourcing a project. In this case the project is called LAMS (Learning Activity Management System) and it started life as commercial software.

Lecture Notes follow

How many times has open source produced a genuine innovation - that is to say a piece of software which has no commercial equivalent. Suggests that there aren't very many at all.

Extracting the project from the original commercial company took 9 months of negotiations. However at the end of this promised funding vanished leaving a large problem. Interesting that the project was released to beta partners before the code was made available - lots of work to make code available, not just 'putting it out there'. Began with a code snapshot, only released access to the CVS repository later (today in fact)

http://wiki.lamsfoundation.org/display/lams/Building+LAMS

* LAMS 2.0 is not yet ready for teachers - just developers. (1.0 is presumably usable for teachers)
* Got a lot of criticism for using the GPL with FUD coming from some big vendors behind closed doors - suggests that this was because there was a product not an R&D effort.

* Governments were keen on the idea, but some failed to deliver on promises of funding
* Other governments did deliver on funding for specific requirements that they wanted (translation)
* Problem with presenting the product at some levels - first you get treated as a dodgy salesman, then you confuse your target further with talk of 'free' software

To get external support you will need to have a good story to tell about sustainability of the project. If you can't answer those questions things will be difficult.

Has had some extraordinary support for things like translations. Once the process was made open there was no shortage of people willing to get started and work on this.

>> Overall this sounds a lot like the normal struggles of starting a company, getting the first big sales and so on. I'm not sure this is a story this audience hears very often though. What might be interesting is the the OS community tend not to see themselves as having to do traditional entrepreneurship / business and it's possibly a little odd for them when that happens. [Martin]

Interesting point that periodic product evaluations weren't as useful as hoped because they were backward focused. Ongoing user feedback was more useful (Release early, Release often?) .

OSS Watch, Tuesday Morning Plenary

I was a couple of minutes late so I can't tell you who the chap giving the very interesting talk on open source and economic research is. (checks website, it's Paul David, of Stanford and the OII) It seems like Stanford have been attempting to build an economic model of open source software creation. Questions are

Micro - where do resources come from, how are they assigned to tasks?
Meso - how are inputs allocated between projects, how does the system match the software to the needs of users?

What are the social costs of OS software? Is there an opportunity cost to participating in OS projects? Is this misallocation of scarce resources? What would it cost closed source vendors to do the same thing? Suggests that the replacement cost of the Debian distribution is about $14Bn

Lecture notes follow

Lecture Notes
-----------------
FLOSS - Free / LIbre Open Source Software

How do you sustain the committment of FLOSS projects?

Many sceptics suggest that OS is something of a flash in the pan and it's just a matter of time till reality bites, with financial motives and 'hard headed business decisions' driving out ideological committment and enthusiasm.

Survey results of OS developers suggest that most (57%) make no money from it. This suggests that they're getting compensation in a non-monetary means. Suggests that 'ideological' reasons are a major driver behind involvement in the community, people also do it to learn and to socialise with other developers.

However motivations do change and shift around, people often start out with quite weak motives but then tend to develop stronger committment, either to skill improvement or ideology.

When it comes to choosing a given project developers are very pragmatic, they tend to aim for things that are going to be useful to them, or educational for them. This pragmatism increases over time.

How do you sustain the creation of new FLOSS projects?

To what extent are things like SourceForge acting as innovation clusters? Or are people just using it as a place to release existing projects? It seems that there are is a very small number of serial innovators who do regularly start projects. Founders have a lot of entrepreneurial characteristics - they network, they socialise, they communicate well, but not necessarily much more skilled.

How do you sustain maintainability and growth in large projects?

There was an early criticism that OS projects were likely to grow to the stage where they would become impossible to maintain (especially on large projects) since the possible number of connections between files rises exponentially as you add more files. However experience suggests that you can avoid this kind of thing if you try. Surveing large FLOSS projects suggests that file growth is linear and doesn't decelerate over time. In the case of the linux kernel it actually accelerates.

This seems to be possible because code architectures tend to be tree like which makes it more governable. In some cases this has meant a project has had to be redeveloped to meet this requirement. An analysis of the linux code base suggests that this is how it is structured.

Conclusion

We can address some, but not all of the sceptics challenges directly. Questions that remain are about things like generating a supply of developers, ensuring that modular architecture develops within projects, competing with focused and agile commercial developers and potentially generating large law suits.

First use of Rhubarb on Rails

Morning out there. Our shiny newly developed Rhubarb on Rails online letter writing action tool nearly saw its first deployment last night but I declared it not ready for use. Why?
* It's running on a development server
* It's running in development mode
* It's running against MySQL - typically we use and know more about PostgreSQL
* The participation page is not yet configurable enough and (as the demo shows) this means that even a basic action doesn't look correct in FireFox because the body text field is too wide.
* No decision has been made about which mail server should be sending the mails.
* The reporting functionality is extremely limited.
* And last but not least, our existing tool which RoR is to replace is currently capable of more and if problems arise is a better understood beast.

However, I do fully intend to use it as soon as possible, but wish to deploy it properly. Watch this space for the first deployment.

April 10, 2006

The linux Kernel : Evolving into Sustainability

Presentation from Stuart Yeades who said this was going to be a discussion, but it's turning into a good presentation. Here are the bullets on what made this letter good.

Don't fork
Don't pretend (via audience, see Dana Blakenhorn on this)
Running code : Projects succeed in almost direct proportion to the amount of effort it takes to get them to work the first time
Open Standards : Use them
Manage Expectations
Open to input / people

Other things you need : Need (demand), Luck, Hard Work, Skill

Where do open source developers come from?

> Ideally from the user community. Great because they have fantastic domain knowledge
> Raised from the academic community. Lecturers will allow students to submit open source projects as coursework if you give them feedback on the marking
> Convert existing developers
> Poach, share or borrow from other projects

How do you find good ones?

> What do you want people to be good at? Developers have very varies skills and it takes all sorts. You should encourage people to do what they are good at - and hoping that a balance emerges...

Useful things - TODO lists
A list of simple things that people can do is a great way to direct communities
It's also a great way to explain to people where a project is going

Your OSS Strategy Sucks

A light-hearted look at using Open Source in an insitutional context. From Andrew Savory of Luminas Ltd.

Lecture Notes follow
--------------------------

I'm going to start by betting that the question 'what is a strategy' will not be adequately answered :-) . [to be fair there's a more of less adequate quote that's just been put up about it]

Stuff to think about : Costs, Lockin, Empowerment, Solution ownership, Data ownership, Maintenance...

Open source as a spur to competition that makes the software market competitive through the whole cycle. ..

Traps and pitfalls
---------------------
Fake open source - the license is not enough. Look for a community, (no community - not open source???)
Is there an enterprise version (and fee)
Code dumps - opensourcing dead projects
Legal issues - are the licenses appropriate

The need for strategy
--------------------------
Work through a checklist on...

Environment
- is the organisation ready for it? Management committment, cultural change, skills to do DIY IT.
Solution background
- health of the project, size of the community, direction
Project Stability - age, reference customers, books / documentation - books are a sign that a significant number of people care, download statistics, commercial extentions?,
Community How is it run, governed etc. How big is the community, how diverse is the community, what do the community think they're doing?
Legal How well are IP issues managed on this project, are they keeping track of the licenses for third party products, does the license allow the intended uses?
Technical Issues Is it all compatible, how easy is it to deploy, configure, manage etc. Can you guarantee business continuity.
Risk Management Regarding the quality of the code and the community in particular.
Support Is support available, and who from? Can the support do the tricky stuff (integration, extension etc) will your support source be able to feed back into the community?

Developing a plan for overcoming this lot...

1. Commit (from management and those who will be working with it)
2. Expectations (no guarantees that the community will care about your specific problems)
3. Get ready (training plan etc. Make people willing to use the new tools)
4. Evaluate (which of the available solutions are you going for)
5. Plan (Deployment, support, decommissioning)
6. Plan B

Open Source and Open Standards

Brian Kelly has put his presentation (including performance) under a creative commons license. You can read about it all here.

Makes me realise we need to put a CC license on this blog.

Lecture notes below the fold

Open standards are generally seen to be a good thing - usual reasons. However adopting open standards does not guarantee all the benefits that are usually listed. You need to work at getting the benefits.

* Potential problem with standards that are too complex, over engineered etc.
* Users might not watn to change
* Migration might be expensive

As an example both Skype and Powerpoint are defacto standards but are closed formats. So...

What makes a standard open => Owned by a neutral body, published openly (freely?), open development, platform and application neutral.

RSS is a complex example. Several interpretations of the acronymn, never mind the actual specs. RSS 1.0 and 2.0 aren't really related. Dave Winer wrote 2.0 as an alternative to 1.0 . Has now degenerated into a governance war at some levels. RSS 3.0 was released as a joke so people will have to go to 4.0... this is a mess for trying to do any long term work with these standards.

"This is an open source conference so we don't do users" :-)

Strong parallels with OSS, the things that make standards work are much the same. Who's backing it, how long are they in for, how good is the documentation etc. This is at least as much about the context the stuff is being produced in as the actual properties of the stuff itself.

Open standards brings the question of 'what happens if you don't comply' what does it mean to say you 'must comply with the standard'. It also matters where you're applying a standard, to the extent where it interferes with people's ability to comply or understand the standards.

Proposes a three layer model for modelling standards. (see online slides). This looks at things from the point of view of the standard and the projects using them.

How do you handle QA? External check-ups dropped as not culturally in line with the HE community. Instead recommend self-assessment as the way forward. This needed to be simple and lightweight enough to use easily.

Remember that open standards exist to support users, not for their own sake. Proposes some core principles that lead to a more pragmatic adoptions of standards / software.

Watch out for fundamentalists - remember that user advocates can be fundamentalists, as can perfectionsists and these aren't necessarily any better than say vendor or open source fundamentalists. Good list of 'fundamentalists'

How do we extend all this to Open Source?
------------------------------------------------------
Interesting bit on 'accessibility' standards. Basic point is that you don't want to talk about accessibility to document x or y, but to the end result - In the context of eductional software this would be access to the learning outcomes.

Suggests a metaphor of a 'tangram' guidelines that can be combined in a wide variety of ways to produce results that are appropriate to their context.

Questions
------------
Are standards prescriptive or descriptive? Is it just a 'folksonomy of the world'?
A bit. It's very difficult for a small group to mandate stuff that is out of step with the real world. However sometimes decisions need to be made and there there needs to be an awareness of what's going on, what's likely to work and so on.

OSS Watch gets Underway

Marco and I are at the Open Source and Sustainability conference at Said Business School Oxford to learn more about what Open Source can do for an organisation like Greenpeace. It's been organised by OSS Watch.

The big issue - Greenpeace has been involved in plenty of open source projects and uses plenty of open source software, but the dream of community driven software solving our problems has never quite happened. Maybe the people here can help us learn why.

What follows will be my 'lecture notes' I'll try and stitch it together with a overview and some analysis later.

Lecture notes....
---------------------
Early thoughts on Sustainability. John Norman is asking some early questions

What needs to be sustainable?
How long does it need to be sustained?

and providing some answers

Open source projects require motivation and incentives for developers. Customer feedback loops and a sustained demand are important.

Grant funding may not be that useful because it removes the incentive to make a product / project sustainable. The result is projects that make it to the end of their grant funding and fall apart because they're not sustainable. Suggests that handing out grants to projects once they're being used and proving successful might be more useful.

James Dalziel (Macquarie University, Sydney) on why sustainability matters for open source software
----------------------------------------------------------------------------------------------------------------------------

* Sustainability applies to closed source software as well as open source. Open source tends to discuss it more because of it's alternative financial model.
* TCOO is made up of many parts and acquisition costs are rarely a major part of that. The major cost is usually adapting the organisation to the software.
* We don't know what unsures the sustainability of software? - We don't know. Companies fail, communities fall apart. There are no guarantees. However levels of risk and type of risk varies between open source and closed source.

Closed source can be unattractive because your vendor holds the whip hand. They might withdraw support for the project, change direction or abandon R&D for the software. They might go bankrupt.

Open source offers several models for making a project sustainable, but there are multi-model projects possible.

* Direct government support and large levels of adoption. This is like making a strategic commitment, by betting so heavily on a project you create an environment in which the stakeholders cannot allow a project to fail. (See Shibboleth in the UK)
* Charitable giving and donations (see Sakai)
* Revenue splits with partner organisations and income from specific feature development (see Moodle)
* Dual licensing - release under the GPL but charge for those who don't want to be under the GPL (see MySQL) who are not attracting VC funding
* Firefox have a deal with Google for revenue splits based on the pages served using google as the default search page...

Case study on LAMS

* Being open source and free from the start (speech, beer) encouraged people to adopt early
* Being open source encourages other projects to get involved
* Makes contributing to standards bodies easier

Made sustainable through an Open Source Foundation that is able to act as a route into charitable donations etc. while a related commercial company offers consulting etc. to those willing to pay for it. Profits go back into the software. Have gone with the GPL so that there is the possibility of doing dual licensing in the future which would allow them to charge for the software.

>> Open Source can be a profitable business, but you have to be good at running a business, but the profits may not be of interest to most VCs. Also, if you are successful you can expect competition to arrive.

Rebecca Grifiths (Ithaka) on the OOSS project
---------------------------------------------------------
Considering whether there is a need for an organisation to support Open Source Software for Higher Eduction in the USA.

Ithaka were established to accelerate the productive uses of IT for the benefit of higher eduction institutions around the world. Act as an incubator for Open Source projects in the classic sense (facilities, admin support, community etc). The OOSS project is about establishing whether there is a need for a co-ordinating body at the national level within the USA - backed by a number of big universities.

* Define OS as free (beer and speech) and community development
* Higher education in the US spend a lot on IT and aren't happy with the results (is any *sector* happy with the return on their spending on IT? (ML)). Usual problems of flexibility, control and cost.

One of the interesting risks to Open Source as a strategy is that there aren't OS projects for many requirements. Things like payroll software etc. These just aren't things people want to do in their spare time - you're not going to see 'spontaneous generation' of projects in these areas. So an institution might do this kind of thing deliberately.

Another is buy vs build - build is much more risky so people decide not to. An institution could bear some of that risk.

Successful projects at one institution sometimes just fail to get migrated to other institutions. An institution might be able to find and publicise these.

There is a significant risk in OSS projects not reaching sustainability. You don't want to be the organisation left holding the baby... If you see the solution to this as the emergence of community driven software then there are some things that you need. You need a large enough community of volunteers with appropriate incentives and motivation, you need a vision about what the software should do and how it should do it, you need the major customers to make long term commitments to guarantee the ongoing use of the software.

License terms are problematic for HE insitutions, apparently much more so than for commercial operations adopting open source software. Perhaps an insititution could drive the creation of a standard set of licenses.

Requirement to stand up to FUD from commercial organisations. In mega litigious US of A there is plenty of room for commercial organisations to threaten legal action against anyone accidently finding someone elses code in their OS deployment.

What next for Rhubarb on Rails?

Today and tomorrow I'll be planning the next few weeks/months of work in the GPI ICT application development team and as part of that will work out if and how we go forward with the work we've done last week. Watch this space! In the mean time thoughts on the code, the application, the documentation and the information provided here are welcome.

April 8, 2006

Demonstration

I've put together an example of the code in action.

We haven't attempted to make Rhubarb compatible in all browsers yet, the example certainly works better in IE 6 than FireFox 1.5 , and for this development we were targeting only IE 5.5+ and FireFox 1+ only. We tend to use BrowserCam for browser compatibility testing. If you haven't used it I can say that for me at least I find it highly useful and happily recommend it. We haven't used it for this project yet though.

The example highlights several limitations of the basic Rhubarb implementation, some of which are simply in later stories not reached yet. For example, CSS styling of the participate form is not yet possible meaning the form is too wide in FireFox on my screen.

Enjoy.

April 7, 2006

Cleaning up

Alex and I spent today going through the failing test cases from Thursday and fixing either the tests or the code. We started with 37 errors (since the code was committed at the end of the day in whatever state it was in - I *really* don't like working against the trunk) and managed to get that down to zero with a couple of bumps where we wanted to scream at people for changing their code but leaving the tests unchanged and thus breaking them - obviously they were never run prior to commit!!! Finally we had it down to zero, then updated (we'd been updating very regularly) only to find two more errors.... aaaargggh! :)

We got it sorted in the end :p Mysteriously though the continuous builder plugin is still mailing me that the tests are failing yet if I run all the tests with rake they pass.

Anyway we bailed to the bar at around 6pm after a final review of how the days had gone - there was almost unanimous agreement that the chandelier had to go and that fresh air was in far too short supply, and plenty of constructive approval and criticism of what worked and what didn't regarding the retreat planning and implementation, rails, ruby and agile xp processes.

We'll review more next week. I'm tired but in a good way, looking forward to a chilled weekend and maybe just maybe I'll resist the temptation to write more rails for my own enjoyment over the weekend but I doubt it. I want to get to the bottom of the Apache problem I'm seeing too, and upgrade our Subversion server from 1.2.3 to 1.3.0 now that 1.3.1 is out, maybe even 1.3.1 but I need to look at what changed, 1.3.0 has been out a little while so it's clear how that works compared to 1.3.1.

Close of play

It's Friday, we're wrapping up and going to the bar.

Today was a good day, we finished release 1 except for one issue with internationalisation.

We've still got a lot of non-functional stuff to look at though, but all in all a pretty successful three days.

Quotes

There have been some choice quotes in the last few days. Indeed in the last few minutes there's been discussions of the need to 'fix Koyan's depravity' which may or may not have been something he'd committed into the code. Here though are some highlights...

Koyan (Greek) : Why am I speaking English with your accent?
Tino (Scottish): You're doing a good job.

Ximon : This is the Ruby on Whales web page
Nis : Whales?
Martin : It's the greenpeace version

Martin : In Agile we fly by yesterdays weather. Which is no way to fly an aeroplane but might work for software development.

Alex : There's something about writing tests that makes me want to curl up and go to sleep.

Alex : Call me an idiot but...
Nis, Koyan : "Idiot!"

Ximon : 'It's rakey rakey time'

Day Three in process

Day three has started and began with a review of the non-functional requirements. Since we haven't paid much (any) attention to this lot since we started (bad project manager!, bad!) we've realised that we can't claim that any of them are actually done.

On the list for this afternoon then...

To make this less likely to happen in future I've added the non-functional stories to the board. So they can move between in progress and done depending on whether anyone anything has done has broken them. Here's a picture of the new look board, which now includes a bug list and a set of new stories for when we finish release 1.

progress board You can't see this well, but the stories in the middle are in progress. The one's on the right are done and the ones on the bottom are the non-functional stories.

Continuous Integration

Once I got the staging instance operational "continuous integration" kicked in. In this case this is the Continuous Builder plugin for Rails which is run every time a commit is made to the rhubarb_on_rails Subversion repository and svn up's the code then runs the full test suite. On test failure it sends an email notification to its' configured targets.

What it doesn't do and I quite like is something I added to the continuous integration script ThoughtWorks made for the Melt project and I added to - that script keeps a web page updated with the status (success or failure) of each build. I'll try and add this to the Continuous Builder plugin and submit my changes back to the rails community.

Debugging Rails

Yesterday I was going nuts trying to spot a problem in one of my test cases. The test case had proven it's worth by identifying a problem that I couldn't see through the web (a double redirect to the same page) but then I couldn't work out why it was happening.

Eventually I got fed up and started looking into how to debug Ruby on Rails. I used two techniques to help me out:

* Firstly the good old write a message to the log file. This can be done with logger.info "message".
* Secondly I used the command "breakpoint" which in test mode causes the test suite to drop into a console where local_variables can be used to see exactly that and any other ruby commands can be issued. Very handy.

Linked

FingerTips have kindly picked us up on their blog following their participation in the retreat. Check it out here.

April 6, 2006

Admin lock down

If you try using the staging instance and manage to guess any of the URLs that work (sorry, we haven't advertised them yet!) you'll find that you're challenged for a username and password. That's about all you can see until we take it further tomorrow.

What you can't see is that it is possible to create actions, list the actions, edit them, and to send mails but not yet take part in them. We'll make this more public and interactive tomorrow. It's dead ugly at present, CSS niceness isn't a story we've come to yet.

Time for dinner, cya tomorrow.

MySQL gem woes & staging instance operational

Spent a bit of time trying to fix MySQL 4.1.15 on RHEL 4 today since it was complaining on rake that

Mysql::Error: Lost connection to MySQL server during query: SELECT version FROM schema_info

I found help in this post on the Rails blog which led me to this solution on the Rails wiki. In particular I had to do:
gem install mysql
cd /usr/lib/ruby/gems/1.8/gems/mysql-2.7
sudo ruby extconf.rb --with-mysql-lib=/usr/lib/mysql
sudo gem install mysql
.
Once that was fixed the automatic update on commit of the staging instance could be turned on. So now you can see the latest code at the staging instance. The tests are incomplete but we've halted for the night, see you tomorrow.

Oh and I just heard we'll have our superstar developer back from illness tomorrow, woohoo!

Work is progressing

Things are running remarkably smoothly today. Stories have flown across the progress board, those in progress are mostly in progress because tests are being written.

I've been pairing with Ximon, which since I have almost 0 experience as a developer has been a good test of the veteran/newbie pairing combination. Elsewhere things seem to be going well and I'd say people are getting to grips with the system.

Five Stories Down ...

We're getting somewhere now, most of our problems have been with TortoiseSVN or Subversion. I'm definitely going back to working on Linux after this, three or four weeks of using Windows again have been driving me nuts. On to authentication for me and Martin now.

Training Day

Yesterday we suffered from an absence of ruby / rails knowledge. Today Thijs and Manfred sat us down for the ninety minute crash course in Rails. Ninety minutes, several ah-ha moments and one round of questions and a round of applause later we were of one mind.

Training Day

Rip it up and start again.

I have a good feeling about today.

Day Two

Today started with a presentation by Thijs and Manfred of FingerTips on Rails looking at some of the issues we encountered in Day One. Presentation is ongoing while I fix up the staging instance and continuous integration (after fixing a problem with MySQL on the staging instance, groan :D).

Presentation is very interesting so far, it's almost a shame that we have to stop at some point and do some coding!

Right at the minute we're learning how to link tables together via has_many and belongs_to and how to build objects corresponding to these tables that automagically generate the id numbers that link the objects/tables together.

April 5, 2006

Thoughts on Day One

Day one of the retreat was definately a learning experience. At the end of the day we ran through a lessons learned exercise, some of the things we thought we might change included...

Trying to do too many new things at once. It's hard to know if we're going slowly because we're new to Ruby / Rails, because we're not getting XP right, or what. Perhaps this should have been a re evolution.

A real customer would really help Laying our hands on someone to spend three days 'being the customer' wasn't really possible. Without that things like the planning game become a bit more like rituals and a bit less like negotiations

More preparation time We've lost a lot of time today due to avoidable issues, like network problems. Had we managed to test this stuff in advance we'd have been able to make better use of our time available for coding.

On the positive side it's good that we're managing to try this. It's good that we are making some progress, however slow. It's good that the local Ruby community is helping out. It's good that we managed to play a planning game, do some work in pairs, get out of the office, have an end of day retrospective, get the working in public thing up and running and spend some time with each other.

These are not small things. And I for one am confident that tomorrow will go rather better than today.

Public subversion access up

It's not perfect (I believe our Apache server configuration for /view/ URLs has a problem with DAV requests) but you can use it to view the repository from a web browser. See our repository online. It also contains the first files we've uploaded regarding documentation, but expect much more.

We shall overcome

So far we've dealt with a network environment that won't let us access the net or accept DAV methods (so GET and POST work but not PROPGET for example), one Kubuntu machine that couldn't install MySQL 'cause it had no net connection, one existing Rails instance that wouldn't work because it needed the MySQL gem but that was only available online and so on ... but we're making progress!

We've moved to two pairs and me fixing up network/environment/server issues and providing ad-hoc help (heh help from somebody that knows nothing about Rails, smart idea huh :D). At Thijs suggestion we're working directly on the trunk which means we're having to merge conflicts as we go (e.g. migration db file numbering overlaps).

We've now got a limited net connection with a squid proxy in the way but we can now work against our proper subversion server and we can blog blog blog blog blog ... too much time in the darkness of the disconnected world. So, you can follow our progress in our subversion server - i'm configuring it now and will post once that's ready.

Working Space

The Dutch office have done us proud, lots of space, lots of room for whiteboards, lots of light and crucially some outrageous pink chandeliers.

Lighting from Mars


Pairs at work


project definition board

Kick Off

We're off and running. Here are some of the highlights...

Planning game
------------
It wasn't far into the planning game when Thijs - one of the guys helping out from Fingertips - pointed out that at our current rate of progress we were going to be meeting all day. The reason - too much thinking about conseqences - if we need this we'll want that, if this is going to exist then it implies other functionality, what about that? The thing to do is get tunnel vision and stick to estimating what's in front of you.

For the record we're being assisted by Thijs and Manfred of FingerTips and Andy Lo-A-Foe , who I believe is his own man. So far they've been incredibly helpful in getting a group of people who for the most part had barely looked at Ruby or Rails to get things up and running. They're also agilists and XP practitioners, able to look at the noisy room full of talking people and say 'yup, you're doing it right'


First estimates
---------------
The first release contains eleven stories. Many of them are very small indeed, probably too small to be stories by themselves in a real project, but since we're trying to learn a whole new style of working small chunks is a good thing - right?

Anyway, the points system we've gone for is deliberately simple. 10 points is an amount of work a pair think they can achieve in a day. We've got three pairs, so thirty points. The estimates for todays work - the most expensive story is five points. The majority are one point, which leads me to believe that people are being wildly optimistic.

Connectivity
------------
Like all good veteran sysadmins / developers / team managers Ximon brought more of everything than we could possibly need. Sadly however there's only one network and while we've now got our own connections to the server and the wider web running we're having some issues with our SVN server.

April 4, 2006

What the #$%& is Rhubarb?

Rhubarb is the name of a PHP based letter writing action tool we created not long ago in a hurry when we needed something quickly. Now we finally get the chance to take a step back and do the project properly.

So, tomorrow's the big day - six Greenpeace developers, a project manager, a customer and some people from the Rails community present to lend a hand and three days of mixing extreme programming, ruby on rails, and a team of developers that have hardly or not worked at all with Rails (or even Ruby for that matter!) in a room with nothing but a story list, some wireframes and our kind volunteer Rails experts for the few hours we have them, oh and of course the Pick Axe and Agile Rails books. If we're lucky another couple of Greenpeacers will be able to join the fray too though likely more in terms of planning, problem solving and triage than development (six is plenty!).

Like any good project it's already presenting us with unexpected challenges. Earlier today I thought we were going to be two people down from day one, and even now I don't know if one of my team will be off sick tomorrow again. For his sake I hope he can join, this should be great!

I made my first contribution to the rails community tonight in the form of a very simple validation helper plugin. It needs test cases and rdoc but I struggled in the short time I had tonight to work out how to test a validation helper. I think I might need to setup a fake model to test it with. It was created just because I was irked at having to put a big email regular expression directly into my model code and wanted to just succinctly express the need to validate the form of a user entered email address.

In a minute I'll try and finish the continuous integration setup. It's taken me longer than usual because I'm trying to use the continuous integration plugin and of course I know (but am learning fast!) very little about ruby/rails. A few other plugins we might use for this project are the ssl require plugin and the captcha plugin. I think our internationalisation requirements are too simple to need the globalize plugin.

What would be great is if the ruby/rails community can review the work we do over the next few days and afterwards and to that end we'll be blogging here and providing the code we make available in our Subversion server. I don't know until I get to the venue tomorrow how the net connection will work out, I expect it will be fine in which case we'll use our net based dev server and subversion server and you can follow our staging instance and code activity. Otherwise we'll be using a mobile server I put together for this event (a laptop) for Subversion and staging instances and updating the net based ones as and when we can.

What's really not helping me finish this up is the fact that my wireless connection keeps dropping out - grrr.

Time for some sleep. See you tomorrow as we kick off Rhubarb Reloaded, or Rhubarb on Rails, or Rhubarb Revamped ;-)

The Application Development Team Retreat

The project which has led to us getting this blog set up and embarking on what we hope will be a long and successful public engagement is our Application Development Team Retreat.

The Application Development Team do what the name suggests. We develop applications. However we tend to do this in a fairly meta manner, managing third party suppliers, buying in technology, deploying open source software and so on. What development we do is typically limited to integrating different systems, bug fixes, maintenance and enhancement. The annual offsite retreat is a chance to get together as a team and build some software from the ground up.

According to the document I've just been writing the goals of the retreat in order of priority are...

Get out of the office : less stress, less distractions
Work as a team : Seven people, one project. Not seven people twenty projects (give or take)
Learn stuff : From each other, from experimentation, from others (both in person and online)
Try new things : This year it's working in public, agile development (XP), ruby on rails, and anything else we come across
Build some software : We have a project that we're planning to work on, we'd like to think we can deliver something by the end of our three days offsite, but if that doesn't work out we won't mind too much, provided we get the other things on this list.

Welcome to Work In Progress

Work in Progress is the weblog where those of us lucky enough to work in Greenpeace International ICT will be sharing our days with the world at large. We call it 'Working in Public'. The idea is to make as much of our activity available to the outside world as we can, in the hope that this might

Make us more transparent to our donors and supporters
Elicit useful feedback and suggestions on our work
Improve the quality of our work
Encourage others to do the same

We've already tried working in public on another project, and we felt the benefits were sufficiently large that we're going to try it again. This time on a non-project specific weblog. Although as you'll see, the origins of this blog are very project-centric indeed...