« November 2005 | Main | January 2006 »
December 23, 2005
Iteration 6 signed off
Yesterday was a busy day, including signing off and putting iteration 6 online, and making plans for the holidays period here.
Iteration 6 brings one of the key features we envisioned: being able to link anything to everything. For any event, group, or resource you're looking at, there's a button to add it to "Your stuff". And if you have stuff there, to link that to what you're looking at. As with the tagging, this opens up a wealth of new opportunities. With all this tagging and linking now working, we've built the crucial ingredients of the platform, and in just seven weeks.
Additionally, once a group is "certified", the owners now can also show the group's endorsement for events and resources, by "certifying" it.
Preparing for the planning game, we went over all remaining stories, dropping 17 stories for this phase (hey, do we now have 'feature shrink'?). Mainly, we decided to leave out messinging within the system for now. And, as with the tagging, we're discovering the true potential of linking now that we're playing around with that: why not let you link your profile to someone else, thereby granting permission for that person to see your contact details?
Christmas is a time for reflection too. We just had a gathering at the office here, looking at footage of the Greenpeace vessels currently "Defending Our Oceans", and the horrific whaling they're trying to stop, while listening to Shane talk about the events onboard over there. Inspiring at the same time: we need new ways to let people join us to stop these crazy things from happening.
Most of us will be back in January, so wishing you all an insightful Christmas and New Year!
Posted by rolf at 1:25 PM | Comments (0) | TrackBack
December 20, 2005
Custard Ontology
The custard project is the culmination of an awful lot of thinking. Like most social software projects it owes a lot to the Cluetrain Manifesto, and my contribution was probably influenced by stuff like Gonzo Marketing, Permission Marketing and the rest of the usual suspects. Somewhere at the back of every NGO web project is the ghost of the Dean Campaign, promising endless dollars if only we'll do whatever it was they did. But there is one piece of external thinking that has probably done more to shape this project than any other, and it's this article - Ontology is Over-rated - by Clay Shirky.
The Clay Shirky article is interesting because it goes to what has become the heart of the project. Ideas about tagging and classification. Shirky argues that rigid categorisations are difficult to impose and often lose their value in the face of a changing world, besides which they're fundamentally rooted in the idea of arranging physical books on physical shelves.
On the internet, he says, there is no shelf. To prove it he's got a nice series of diagrams... (go and look, then come back)
The one we're most interested in is that last one. Which I'll reproduce here to make life easier.
[Just links (There is no filesystem)]
Here the rigid hierarchical structure of the earlier system has been replaced with a structure composed entirely of links between different pieces of information. The links form an unlimited number of possible shelves and filing systems, and by allowing anyone who wants to make connections between disparate bits of data the system grows and structures itself.
This is why services like del.ico.us and Flikr work, and hopefully it's why Custard will work. Within custard there are three ways to link pieces of information.
1. Tagging allows for one to many relationships. A tag can be shared by any number of pieces of content. 'Power' might be associated with pictures of a power plant, and the President of the USA. When tags become used in combination more clarity emerges 'Coal Power' shows the power plant but not the president, what 'Oil Power' shows is a different matter, and without making a value judgement about the content which we expect our users to share, who are we to tell whoever made the connection that they are wrong?
2. Linking allows for one to one relationships. It's a way to express connections between pieces of information that go beyond a shared subject. For instance you might link an event, to a map (showing where it is), a discussion group (who organised it) and a press report (describing what happened). These items will each have a place of their own among the multiple shelves of the tagging system, but once a visitor has found a way to one, accessing the others should be simple.
3. Geo Location where a thing is, or where a thing is about is a good way to connect things. Pollution near powerplants is one example, activists near other activists is another. Custard will make it easy to make these connections between different pieces of content, providing another dimension for our increasingly flexible shelving system.
At the end of the current iteration (Thursday, due to some folks being ill / on holiday / whatever) we'll be able to show the world a custard where linking, tagging and geo-location are all running together. We expect that this will lead to some changes about the way we see the system and interact with it. With luck it will bring to mind some new possibilities, and hopefully show the rest of the world what it is we've been working on for all this time.
Posted by Martin Lloyd at 1:18 PM | Comments (2) | TrackBack
December 19, 2005
Adventures in Agile : Part 2
To continue some of Martin's observations in part 1, here are some of my findings, being the "Customer" in this project, and dealing with my first exposure to full-fledged agilism.
As I was taking over James's position as "Customer" after the start of the project, it turned out not to be easy to get up to speed. There is a list of 140 or so stories describing many features, usually in some detail, but not all detail. Some wireframes to give a suggestion where these should go. And with the daily meetings, there seems to be urgency around signing off things. But what exactly, and should something go with the current story, or is there a different story for that?
After about three weeks, it's all getting clearer, to the point where I just wrote out all the features I felt where missing, adding another 30 or so stories (some turned out to be duplicates). Also, we've worked out more of a routine using an issue tracker to streamline our workflow. Even looking at some of the "agile stories managers" out there, more people must have tried to keep an overview of all these details...
My own rhythm in the iterations is settling a bit too. No longer running around each day, looking at story lists and half-finished features, and feeling like not having time for All The Other Important Work -- like making sure the campaigners will want to use it, and so. At first encounter, "agile" seems to support the techie notion that developing the tool is the most important thing in the world. Instead, I let Duncan, Paul, Peter and others at ThoughtWorks just do some stories in the first days of the iteration, and then test them more in cohesion.
I do recognise aspects from other approaches. For instance, the ability to adapt, and to add or change current 'stories', and give them new priorities, is also a feature of DSDM, an approach where I've also been the "Customer". DSDM works with time boxes, and prioritises with the "MoSCoW rule", which tends to make assigned priorities more consistent across people than using "1"-"4". I've been using it for other types of projects now as well.
And since methodology wars probably still exist, as they did during my time at university, I quickly found a discussion of the differences. The bottom line seems to be really the bottom line: you pay a license fee for DSDM.
In developing web applications for clients myself, I've often had "Customers" who had far more Important Other Work to deal with -- having far less time for all this continuous testing and looking at details. Some of my customers would have run away at a site where you can join a group, but not leave it ("That's next week's story"). I think almost all would have gone crazy having to deal with so many details. Let alone sit in so many discussions about technical details.
Agile is indeed pretty process-heavy, with its good sides too: discussion topics have their place, either in the daily scrums, the weekly plannings, or the less frequent retrospectives. But it takes quite some commitment from all stakeholders to make it work!
Posted by rolf at 4:09 PM | Comments (1) | TrackBack
December 16, 2005
Polar Bears on Thin Ice
Just saw this in today's Guardian. The photo at the top of the article is fantastic.
Meanwhile the result of the Montreal conference showed that there is widespread international will to do something about climate change. Now we need to make sure that what we do is enough.
Posted by Martin Lloyd at 11:35 AM | Comments (0) | TrackBack
December 15, 2005
Iteration 5 signed off
Iteration 5 is now live on http://melt.staging.greenpeace.org . New features this week include
* Tabbed search results to make searching across the content types easier
* Administrators can delete and edit content
* People who upload content can go back and edit it or add more keywords
* We have a mechanism for leaving and closing discussion groups
* Homepage now shows the most popular, most active or soonest resources, groups and events
But mostly we've got a new look and feel for the hopepage. You might need to hit reload a time or two to clear your cache. It's still functional rather than lovely, but it's heading toward where we want to be.
This week we'll be adding the functionality for linking bits of content to other bits of content. More about why that's important in a future post.
Posted by Martin Lloyd at 11:35 AM | Comments (2) | TrackBack
December 14, 2005
BrowserCam
Despite the name sounding a little suspicious, BrowserCam looks very interesting. One of their two main product lines is a "remote screenshot grabber", which allows you to easily compare how your sites look on a multitude of different browsers, platforms and resolutions.
Warning for the pure of heart: The service is not free, neither beer nor speech. You can create a trial account which will work for a maximum of 24 hours and 200 captures.
I am not sure how much we are going to use it for this project - but definitely a handy tool for all kinds of layout testing.
Here is a screenshot of the "View" functionality of the application:
It compares the look of oceans.greenpeace.org in 6 different browsers, including IE 5 for Mac.
The web interface is very slick, with lots of small niceties. But it is the core idea that is brilliant - the rest is just a good implementation.
Posted by Nis Jorgensen at 4:40 PM | Comments (2) | TrackBack
December 13, 2005
Adventures in Agile : Part 1
For those of us on the Greenpeace side of the development team, this is the first time we've worked on an 'Agile' project. For those on the Thoughtworks side, well every project is an Agile project. So this is the first in a series of posts about what I'm learning from this, and what I see as the strengths and weaknesses of this way of working.
My first reflection on Agile is that it's often mis-sold. It's billed as a 'Process Light' approach in which 'All that Bureaucracy' is thrown out in favour of 'Pragmatic Solutions'. While the Agile Manifesto is carefully worded and comes with caveats the business customer or non-agilist is likely to see the manifesto as saying
'We don't have a process, we're not going to write documents, we won't sign a contract and we don't believe in planning'
Processes, documents, contracts and planning are not meaningless bureaucracy. They're responses to particular problems - namely quality, maintainance, supplier risk and project risk. Taken by itself the manifesto reads like a rejection of tried and tested methods in favour of cowboy independence. That advocates use terms like 'movement' and 'manifesto' adds to the sense that this isn't something responsible businesses would want to get involved with.
Which is unfortunate, because Agile is a well thought out methodology that does address the issues of quality, maintainance and project risk in detail, and often extremely well. It is also anything but process light.
On many web development projects the process is a bit like this. 1. design 2. development 3. testing 4. deployment. Within each stage there are goals and milestones, the project manager can see a list of tasks, calculate the critical path, identify risks and estimate things like delivery date with a high degree of accuracy.
At that stage though process often ceases. The individual programmer, tester or designer is given a task like 'design this bit and make sure it works' with the Project Manager ticking it off when it's done. Some testing is often done by the developer and on big projects finished components are handed over to a QA guy for serious testing. When things fail, they go back into the loop.
Agile changes a lot of this, mostly by redistributing the burden of where the process falls. As 'Iteration Manager' I find that Agile reduces some of my workload, principally by pushing a lot of tasks onto the developers. Developers used to just being told to 'write this and make it work' though could well find Agile much heavier going.
Lets imagine Peter the Programmer has been asked to write some code for a project. In what I'll call 'traditional' development he'll be given a specification, write the code and when he's happy with it hand it over for testing and sign off, if it fails, he'll get it back.
In Agile development it's different. Peter isn't given a specification, he's given a requirement (called a Story). First he estimates how long this will take him. Then he has to write some automated tests which will only pass when the requirement is met. Then he has to work out how he's going to meet the requirement, write the code, test it himself and then pass it over for acceptance testing, which is less about 'does it work' than 'is this an appropriate solution'.
Advocates of Agile argue that what Peter is doing now is best practice. He shouldn't have to work to detailed specifications, because they take a long time to write and don't adapt well to changes. He should write tests and do his own estimating because that's best practice for programming, and he should be made to understand the requirement because then he can formulate the most appropriate response.
My point isn't that this isn't a form of best practice - it is. My point is that it's process, lots of it. Masses of baked in deep process. An Agile programmer spends a lot of his time doing things that the non-agile programmer doesn't have to do - things like writing tests and understanding requirements. He also has to go to daily meetings and weekly planning sessions; yet more evidence of process.
For the project manager, Agile is easier. No detailed technical specification - that's written by the programmer as they go. Estimating is done through 'points' and while things like critical path analysis and estimating timelines get harder, the responses to problems (usually triage) get easier. You only really need one document - a big list of stories, and it's written in a language your customer can understand.
For the programmer (and bear in mind that I'm not one) Agile seems to demand a lot of discipline and an adherence to a methodology that while it does allow a lot of freedom (in terms of how you solve each problem) does force you to work in a tightly defined manner.
So, my first big thought on Agile? There's a lot more process there than some people would have you believe.
Posted by Martin Lloyd at 1:02 PM | Comments (6) | TrackBack
December 9, 2005
The 34th Element
We've just got selenium up and running as part of our test suite for the development. Unfortunately it's not in public so I can't point you to it directly, but I can recommend it as a very useful looking piece of kit.
I've just demoed it to some of the team here at Greenpeace and we're thinking it would be of value on pretty much all our web projects.
There's a demo of Selenium here What there isn't is an explanation of why it's named after the 34th element in the periodic table. Paul?
Posted by Martin Lloyd at 1:00 PM | Comments (2) | TrackBack
December 8, 2005
Iteration 4 Signed Off
Iteration 4 was signed off yesterday and added as the public face of Custard development today. Week 4 saw us add features that let content owners edit their material and get forums up and running within the groups. The search results will also now suggest additional appropriate terms to refine your search by when you're looking for content.
Take a look at it online
This week will mostly be about front page customisation and deleting material from the site.
Posted by Martin Lloyd at 4:55 PM | Comments (0) | TrackBack
December 7, 2005
How to get started with the code
Formatted nicely from the HOWTO:
Running the server
Prerequisites:
Python 2.4
eGenix mx Base
Psycopg 1.x (or get the compiled windows binaries)
PostgreSQL
Use subversion to check out the source to a folder called "melt" (username: anonymous, blank password):
svn co https://svn.greenpeace.org/repositories/custard/production/trunk/melt melt
Create a database in PostgreSQL - use the included pgAdminIII tool or run:
createdb melt
Edit the melt/settings.py file to point to your database. Make sure the following settings are correct:
DATABASE_NAME
DATABASE_USER
DATABASE_PASSWORD
DATABASE_HOST
DATABASE_PORT
Now, you can use our build.py script to do almost everything you need! Run:
python build.py
This will show you all of the build targets that are available.
First, initialise the freshly created database:
python build.py recreateDB
Now, load the necessary data into the database, such as the list of countries:
python build.py loadReferenceData
Then, start up Django's development server:
python build.py runserver
You should now be able to see custard in action at http://localhost:9090 in your web browser. Enjoy!
Doing development
If you want to do development, there are a few more steps. First, you are going to need xmllint, which we use to validate all of our XHTML. You can find xmllint in the libxml2 package:
libxml2
Make sure you have the server running, and run:
python build.py runfulltests
This will run our unit, functional, and nonfunctional tests.
Running in production
Django's development server is great for development, with its autoreloading and quick responsiveness. But if you want to host the site for more than just you, you are going to want to use Apache + mod_python:
Prerequisites:
Apache 2.x
Mod Python 3.1.x
First, you need to configure Apache to serve the site through Django. See our melt.conf as an example:
melt.conf
Notice how the melt.conf does not use Django to serve the static files (CSS, images, etc.) because it would be too slow.
Finally, change the MEDIA_ROOT setting in the melt/settings.py file to be an absolute path, instead of a relative path. Apache's home directory is not necessarily your melt folder, so Django needs an absolute path to specify where it saves your files.
That should be it! Let us know if you have any questions!
Posted by at 5:18 PM | Comments (2) | TrackBack
When Technology isn't the answer
Sometimes you just have to do things the old fashioned way.
http://www.greenpeace.org/international/news/global-warming-global-action-111
and sometimes it helps to do both at once
http://www.asiacleanenergy.org/camp/
Posted by Martin Lloyd at 10:54 AM | Comments (0) | TrackBack
December 6, 2005
Javascript on?
One thing that's come up in several of our recent planning games (well the last two) is the question of whether the system should require JavaScript or not. There are two sides to this discussion
One is that a the vast majority of users have javascript, that it lets us do cool stuff and would speed up development.
The other is that those users without Javascript are often the ones who might find this system most useful. Those using old technology in the developing world, or users accessing the site through cybercafes etc. Does anyone know if a Chinese cybercafe is likely to have Javascript enabled for instance?
We'll need to let the Javascript thing slip anyway for the Google Maps interface, but we've been trying to work toward a site that works without it. The concern I have is that we might be incurring extra work for little benefit.
Thoughts?
Posted by Martin Lloyd at 6:45 PM | Comments (4) | TrackBack
December 5, 2005
Comments enabled
Hmm. I've just unticked the box for moderating comments, so unless we're drowned in a tidal wave of spam comments should be able to appear on this site instantly rather than waiting for us to approve them.
On that note - use this as an open thread...
Posted by Martin Lloyd at 5:42 PM | Comments (2) | TrackBack
Del.icio.us December 3rd
Last Saturday was an international day of action to demand governments to take action against climate change (they are currently discussing the Kyoto protocol in Montreal). over 100 cities in 30 countries participated.
I've started collecting links on del.icio.us/tag/2005-12-03, and invite you add your own! Even better, add the tag "press", and we should be able to simply create an overview of press coverage together (although there seems to be a delay before links show up there when you filter for more than one tag? Anyone familiar with this?). [I changed the date format from 03-12-2005 to 2005-12-03 following Dustin's remark]
And while we're at the subject: help us out and tag things you find relevant for us (eg. climate campaign groups, must-see social software sites) with "custard-melt"!
Posted by rolf at 1:45 PM | Comments (3) | TrackBack
Custard Melt (soon to be) used in 30+ Countries
Protesters From 30 Countries Unite to Fight Global Warming ... and what tool did they use to organize their online and offline activities ? ? ?
I wish we could have simultaneously shouted:
CUSTARD MELT ! ! !
But that will have to wait another month or so.
At present we are networking with the techie/developer crowd, but some of the groups that we should get on board this project (while we are still in the development phase) are listed here:
http://www.google.com/Top/Society/Issues/Environment/Climate_Change/
http://www.google.com/Top/Society/Issues/Environment/Climate_Change/Activism/
Posted by John Carella at 12:50 PM | Comments (0) | TrackBack
December 2, 2005
"Why roll your own?"
Columbus was the last person to discover America. So, are we the last ones to develop a new social software platform? With other options around, and many organisations putting their bets on CivicSpace (based on Drupal) or Plone (based on Zope), are we starting from scratch again?
Melt is intended to be different from most campaigning sites: it wants to be a larger, non-partisan platform for a far-reaching goal: curbing climate change. It will take time to let people find the platform, use it to find others, and jointly organise their own activities.
The software behind it therefore should scale well, both in amount of users as well as features and maintenance of the code base. And it should easily run for years to come.
A lot of thinking and discussion went into the decision. A lot of research too: can we adapt a current platform to our needs, are presentation and code properly separated? Take one of our main requirements, "geo-location for everything": no current system has it built-in, what would be the effort to add it?
Then, if we want to hire contractors, what are the options, what are their levels of maturity in software development? Is there a methodology to assure quality and maintainability? Will it scale to our ambition? And what kind of larger developer community would we be looking at?
The most convincing story was the rigorous approach of working according to XP methods, on Django as rapid development platform, starting from scratch to build around persons, groups, tags, and geo-location. A challenging expedition, with unknown discoveries ahead. Being inspired and learning from others, but with the desire to follow a new path to get there, and beyond!
Posted by rolf at 11:31 AM | Comments (11) | TrackBack
December 1, 2005
Get involved!
Maybe you're a Python developer interested in getting involved in our project. Or maybe you're a web designer and you think you can make Melt look nicer. Perhaps you're a HCI guru and want to tell us how to make Melt more usable. Are you just curious and want to look under the hood? Could it be that you've heard about Django and want to see it in action?
Right now we haven't got a process for taking Python/HTML/CSS/etc contributions from you but we really want to so comment on how you think we could work together.
What you can do now is take a look under the hood. You can also get a copy of Melt for yourself (use a subversion client to check it out from the repository) and try running it and playing with it on your system (check out the INSTALL file for help on this).
Posted by xeightee at 5:27 PM | Comments (9) | TrackBack
Gadgets and build cycles
There are several tools to help you take a look at the underlying source code of Melt but first a quick word about our release cycles is in order so that what you can see with these tools makes some sense.
We've got a roughly weekly cycle and each week the new version is made available on the staging site. Warning! Content will be erased when the staging instance is upgraded! This new version is called an iteration and these are tagged in Subversion. These are the tags you can see in the tags directory of the source structure.
Then of course there is development happening all the time and this is what you see if you look in the trunk directory of the source.
So, back to tools. Subversion (what is Subversion?) contains the source code for Melt, and also allows you to browse it. See the link on the right to dive right in!
Posted by xeightee at 4:53 PM | Comments (0) | TrackBack
Iteration 3 signed off
Iteration 3 was tagged and released into the wild yesterday. Right now you can see it at http://melt.staging.greenpeace.org , the code is here
(Note we're only hosting the 'latest' iteration at any time, so in a week or so the link will be to Iteration 4...)
Developments in the last week included
Being able to promote other users to 'own' a group you have set up
Being able to mark organisations within the system as 'approved'
Allowing approved organisations to 'approve' of other pieces of content
We also added a lot of location based functionality to the searches, although for now we only have data in the system for Japan and the Netherlands.
The most interesting part of last week was adding the approval process. Since Melt is intended to be used by many NGOs we felt that they would need a degree of reassurance before participating. Given how open Melt is anyone could turn up and post a 'Greenpeace press release' or an 'Friends of the Earth Scientific Report'. By accredditing organisations and allowing them to mark particular bits of content, or indeed users, as their own we provide a way of avoiding this.
(I learned yesterday that this goes some way to making us an attention network rather than a social network)
Future upgrades might include replacing the current text indication of approval with an icon or something similar.
The accreditation process itself will be an offline affair, the details of which still have to be worked out. But a phone number, some headed fax paper and something similar are likely to be involved.
Posted by Martin Lloyd at 10:26 AM | Comments (0) | TrackBack