Custard Melt

* The software behind it therefore should scale well, both in amount of users as well as features and maintenance of the code base. And it should easily run for years to come.

Drupal supports over 100 active (committed code in the last month) developers and runs over 50,000 websites some serving millions of visitors a day (theonion.com).

* Take one of our main requirements, "geo-location for everything": no current system has it built-in, what would be the effort to add it?

CivicSpace / Drupal support this - using the location module geo-locative information can be assigned to any content object (node) and user.

http://drupal.org/project/location

* can we adapt a current platform to our needs, are presentation and code properly separated?

Have a developer dive into Drupal code. For this project the answer is unequivocably yes.

* Then, if we want to hire contractors, what are the options

There are currently over 20 companies employing over 50 engineers that service CivicSpace / Drupal. One or two new firms pop up each month on average.

* Is there a methodology to assure quality and maintainability?

Yes. Read up on the Drupal development process, Brows the code.

http://drupal.org/node/316

* And what kind of larger developer community would we be looking at?

300+ engineers have CVS commit access to Drupal. Over 100 committed in the last month. The Drupal community is exploding. (In the past year the number of users on Drupal.org has increased 280% and the number of nodes has increased 270%)

See: http://tinyurl.com/96lrm

* The most convincing story was the rigorous approach of working according to XP methods, on Django as rapid development platform, starting from scratch to build around persons, groups, tags, and geo-location. A challenging expedition, with unknown discoveries ahead. Being inspired and learning from others, but with the desire to follow a new path to get there, and beyond!

90-95 percent of Open Source projects fail.

http://press.oreilly.com/pub/pr/1447

Why take on the enormous risk of trying to start your own open source project when an already proven successful project such as Drupal could easily be used as a platform to build on - saving you large amounts of money on developing your application and garunteeing you will have a pool of developers hacking on your code base you can collaborate with?

I put my reply up here: http://delocalizedham.com/drupal/node/34

I've gone through the feature listed in the MELT master story list[1] and flagged each as:

(109) blue: currently implemented in Drupal
(19) orange: can be implemented by extending functionality in an existing Drupal module
(3) red: would require a new module be created

Download: http://www.uploadtown.com/d.php?file=yea&filepath=2213

[1] https://svn.greenpeace.org/repositories/custard/production/trunk/melt/doc/

-Zack

http://demo.civicspacelabs.com/home/trunk/

I unfortunately shot myself in the foot and worked on a sandbox I did not have server access to which means I was unable to install new modules or mess with the database. Because of this there were a couple problems.....

* If you didn't notice Zip code radius searching isn't working. This is because the install seems to have been loaded without the Zip code -> long / lat database. When I get server access and can dump this data in the right table it should work.
* The groups are kind of fake. I was unable to install the real groups system (Organic Groups - http://drupal.org/project/og). The current groups are using the Drupal flexinode system and thus have no concept of membership. With server access I could install the real group system and it would work fine.

Other than that everything else should work.

Current functionality implemented:
* Site menu's and content all configurable via web interface
* User registration system (hashing passwords, password retrieval, access control)
* Profile collection / displaying for registered user
* Profile information mapped back to full blown CRM back end (CiviCRM)
* Event system
** Hosting events
** Modifying events
** Support for mutliple event types
** Event comments
** Event trackback
** Searching for events by country, state, metropolitan area, and radius serching
** iCal export for events
** Registration for events (CRM back end)
*** View reg list
*** Send message to reg list
** Volunteer management at events (CRM back end)
*** Volunteer rating system
*** Queuing system
** Event invite system (like eVite)
* Group system
** Create groups
** Modify groups
** Assigning geo-locative information to groups
** Group comments
** Edit Groups
** LIst groups
** Group trackback
If I was able to install Organic groups it would also support...
** Group moderation
** Group membership
** Group inviting
** Group blogs
** Group events
** Group forum
** Group images
** Group eCommerce Stores
** Group messaging

Total implementation time was approximately 45 minutes start to finish, from 0 modules enabled and no content to what you see now. Currently utilizing 19 of 63 modules installed and 300+ in CVS

hi Zack,

Obviously, you are disappointed we didn't go with CivicSpace. You are pointing out that CivicSpace indeed has many of the features we are looking for. That is exactly why we spent quite some time researching it.

In fact, based on email contacts with you, one of the leading companies implementing CivicSpace has presented a very attractive proposal. Their proposed budget to build on CivicSpace was at the same height as our current choice, building from scratch.

ThoughtWorks as a partner also brings a strong dedication to "the" open source community, we're not starting that from zero (http://opensource.thoughtworks.com/). And we also had a competitive offer based on Typo3 (http://www.typo3.com), I'm sure you know that has a strong community too?

In the end, it's not only a matter of checking off features. There also needs to be a match in vision and direction, community culture, and working methods. There was a very competitive offer based on CivicSpace on the table, but we had the luxury to choose among a few similarly attractive offers.

With kind regards,
Rolf.

Rolf,

First off, I think the project looks great and I wish you all the sucess in the world.

I think the "open source" part of this discussion has less to do with an open source technology community (Drupal vs. Django), but the open source community of NGOs that continue to reinvent the wheel over and over again, rather than build and extend an ecosystem of software that supports the sector.

The logic of choosing "among a few similarly attractive offers" tends not to lead to collaboration. I posted a thought on this on my blog.

Now that you all have picked your direction, I hope to see open APIs, open data standards, and robust syndication models.

Climate change is a big problem. The environmental movement is at the forefront of network centric advocacy. I look forward to seeing how your technology allows any concerned organization (regardless of their technology platform) "work together to protect the climate and to generate clean and secure energy."

David, I've read your post at http://socialsource.blogspot.com/2005/12/when-technology-decisions-arent.html

I think you're raising important points. I agree that "open source" has become the natural choice, or more even, imperative. In the decision process, I would personally like to add the most pressing question that emerges: (4) What is the process and culture in the community?

I have no doubts about the quality of many people working at Drupal and many other platforms. But do the individual talents and qualities come together in a process that synergizes thos talents and qualities? What are the cultural drivers in the community?

For instance, is the community driven towards building businesses, or towards adding kewl features? Do I talk to DarthFather at IRC, or can I call Mr. Klein at Customer Support? Is it a "non-profit tech community" or an "open source business community"? What is the roadmap, do people have a common goal?

In the stacking of technology and communities, building on PHP-Drupal-CivicSpace meant building on layers of different communities. Building on Python-Django-(Custard) mainly meant going for a specific development methodology covering different layers.

For me personally, learning that adding "a few features" in the former seemed equally expensive as "building everything" in the latter has surely made me think. Does it mean we'll need hundreds of volunteers or big piles of money everytime we need "a few more features"? How much leverage is that? Maybe building a legacy-free new platform is a better investment in "the public good" than throwing more resources into an existing tool.

(In fact, you're capturing it right in the title of your item, "When Technology Decisions aren't Technology Decisions". It might be interesting to learn that the choice for Python and Django was made relatively late, and Ruby on Rails was another serious option.)

It is indeed a choice of "ecosystems" and culture. I see a lot of efforts, both from you, as well as from great people at CivicSpace, to build up a distinct ecosystem based on other products (with their own ecosystem and culture), and within the market of non-profits. There is great promise in that, but in my (current) opinion not more or less promise than in building on an existing methodoloy-driven ecosystem.

It's an interesting discussion, and I think we should find more places to have that discussion. "Diversity is stability", so perhaps also on the "meta-ecosystem level" :-)

Disappointed but hopeful. That's how I describe my reaction to this news. I'm disappointed that Greenpeace's efforts won't do anything to directly support and extend the growing Drupal community, but I'm hopeful that by using open APIs, the system they build will be truly useful and complimentary to Drupal.

Good luck with this project!

Heya Rolf. Let me first preface this conversation with a couple statements...

* I've got a ton of respect for you Rolf and for Greenpeace. Committing to open-source solutions goes well against the grain of the established technology services market. Just about everyone I know working towards open-source technology in this sector makes about 40% than they would in the private sector and about 30% less than they would working for a proprietary vendor. Greenpeace in my mind has been the earliest large organization with a sizable budget investing substantially in open-source solutions. They've been all on their own for years investing large amount of resources into open-source platforms such as OpenACS.

* Drupal will never be everything to every body. There will never be one single platform that will rule all web apps, nor should there be. Competition is (obviously) a good thing for the technology industry as a whole and for open source projects. There will be winners (Plone, Drupal) and there will be losers (Nuke, OpenACS) and there will be competition and knowledge share, spurning the respective communities to develop better products.

* I really don't be want to be an ass whining about a lost contract. CivicSpace would have gained nothing financially from this project (it would have been fulfilled by a contractor) and I very routinely see large projects that in my opinion should have been done with CS/Drupal go a different direction. My issue with MELT is I believe we have fundamental differences of perception about what open-source is and how it works within this community that deserve to be discussed before tens (hundreds?) of thousands of dollars and many months of work are invested into MELT.

Ok. The CS vendor submitted a quote with the same price-tag as ThoughtWorks. This is somewhat surprising to me since there will be multitudes more man-hours invested in creating the solution from scratch than extending CS/Drupal to meet your spec. Of the 142 features in your spec 119 already exist within Drupal and 19 can be reached by extending existing functionality in a straightforward manner while every feature will have to be implemented virtually from scratch by ThoughtWorks (except for the free CRUD and other tidbits you get from Django). Did you get a proposal from multiple CS vendors? Since the billable hourly rate for CS services differs by a factor of 5 within the vendor community I would bet you could get a much lower price quote than the one you received. My guess is the vendor who submitted the quote has higher hourly billable rate than ThoughtWorks and they highballed you on the estimate while ThoughWorks lowballed.

But that aside let's talk about Open-Source. I firmly believe Open-Source is not just code and a license. Open source projects that are successful are much more than that - they are communities - and thats why open-source works. All of the benefits of starting open-source projects such as MELT come from the communities that (hopefully) form around them: free development resources (other hackers), user-contributors, grassroots marketing, money etc. Until those communities of contributors form it's just code in a public CVS.

Without a community, MELT and GreenPeace are destined to lose out on just about all the advantages of going open-source. I do not believe there are good prospects for a substantial community of contributors forming around MELT...

* The raw numbers are against it. Again, 90-95% of all open source projects fail: http://press.oreilly.com/pub/pr/1447
* Who is going to drive this project? It took 4 years of personal commitment from Dries to turn Drupal and the Drupal community into what it is. You have ThoughtWorks on a contract to do initial development, but then what? Does is then fall into GreenPeace's lap?
* MELT is in competition with a fairly established market of open-source communities. Why would someone adopt this platform over others that are intended to do roughly the same thing but with much more established bases of users and developers? It would be one thing if MELT were in an entirely new space of applications, but it clearly isn't.

In any case, why take this risk?

In the Objectives section of the design brief lists "To deliver the system on an open source platform to ensure it can eventually be handed over to the community" and "provides a platform for future development" as objectives of the project, but says nothing about the what / why / and how of actually establishing an open-source development community. This means to me that this is either an after-thought of the project or not one of the primary objectives. Either way this does not bode well for the future of the platform, but it does makes sense to me.

Why would GreenPeace want to create their own open-source project anyway?

Greenpeace can meet it's objectives of assuring that others can extend their platform and they can __garuntee__ there will be a community of contributors working with them on their platform by building this project within an existing open-source community such as Drupal.

Now, to address your questions...

"I would personally like to add the most pressing question that emerges: (4) What is the process and culture in the community?

I have no doubts about the quality of many people working at Drupal and many other platforms. But do the individual talents and qualities come together in a process that synergizes thos talents and qualities? What are the cultural drivers in the community?

For instance, is the community driven towards building businesses, or towards adding kewl features? Do I talk to DarthFather at IRC, or can I call Mr. Klein at Customer Support? Is it a "non-profit tech community" or an open source business community? What is the roadmap, do people have a common goal?"

These are good questions to be asking. The Drupal community is a very wide net. People are running Drupal sites for everything from personal home pages, porn sites, theonion.com, and shortly the EASA (European Union FAA) and everything in between. The developers are come from all around the world: South America, Belgium, Germany, America, Czech Republic, USA, and many more. But everyone has something in common - the Drupal platform. In my opinion the core Drupal hackers are exactly the kind of people you want working on the core of your platform. They obsess over code format and structure, tinker and test for anything that can speed up performance, fly halfway around the world to discuss how to implement "Node Relations" or the ultimate "Forms API" and every other extremely complicated and important technical problem I promise you would not want to be stuck with solving.

The CivicSpace community has a smaller scope, we focus on community organizing - in my opinion exactly the same problem set MELT is intended to tackle. We consist of a network of collaborating organizations and individuals that work day in and day out in the same space as MELT.

So to answer your individual questions:

"But do the individual talents and qualities come together in a process that synergizes thos talents and qualities?"

Yes. The Drupal core hackers solve the hard technical problems and the CivicSpace community figures out how to employ and extend the functionality in the problem space of Community Organizing. The result is more resources (money) to support the platform developers and lots of excellent free code to employ in community organizing. For CivicSpace it is nearly a perfect match.

"What are the cultural drivers in the community?"

The Drupal core engineers are driven by the challenge of solving hard technical challenges and the thrill building "the ultimate platform". The CivicSpace community is driven by figuring out how to best employ technology in community organizing.

"For instance, is the community driven towards building businesses, or towards adding kewl features?"

Its driven towards solving problems and being a useful platform to build on top of.

"Do I talk to DarthFather at IRC, or can I call Mr. Klein at Customer Support?"

If you ask on the forums you get answers from anyone within the community. If you hire a vendor you get customer support.

"Is it a "non-profit tech community" or an open source business community?"

Drupal supports both. Both reinforce and add to one another.

"What is the roadmap, do people have a common goal?"

The common goal of Drupal is solving problems and being a useful platform to build on top of. The common goal of CivicSpace is to figure out how to best employ technology in community organizing. These goals are not exclusive to one another as is proven by the success (adoption) of the CivicSpace platform.

"In the stacking of technology and communities, building on PHP-Drupal-CivicSpace meant building on layers of different communities. Building on Python-Django-(Custard) mainly meant going for a specific development methodology covering different layers."

This is incorrect. All of CivicSpace's modules are completely compatible with Drupal which means that at a technical level they are the same platform. In fact building on top of Django will add an additional layer of technology to your application than if you build on CivicSpace / Drupal. Here is a comparison...

(1) Database:
Drupal - mySql or Postgres
Custard - Postgres

(2) Languages:
Drupal - PHP
Custard - Python

(3) Application frame work
Drupal - None (included in Drupal)
Custard - Django

(4) Application
Drupal - Drupal (+CivicSpace modules)
Custard - Custard

Additionally Custard requires additional libraries: psycopg and egenix.com and is coded in Python which is a web app environment not nearly as widely supported as PHP.

"For me personally, learning that adding "a few features" in the former seemed equally expensive as "building everything" in the latter has surely made me think. Does it mean we'll need hundreds of volunteers or big piles of money everytime we need "a few more features"? How much leverage is that? Maybe building a legacy-free new platform is a better investment in "the public good" than throwing more resources into an existing tool."

Drupal does not have a legacy problem. The going rate for module development for Drupal is about about equal to normal php development ( about $10-$80 an hour). It is extremely straightforward to leverage the pervasive developer API's within Drupal and to create your own modules or extend existing ones. Even better there is an active community of developers already completely up to speed developing and extending modules that you can tap. Conversely anyone having to extend Custard in the future will have to climb it's learning curve on your dime unless you are successful in establishing a community of contributors.

A question Rolf:

"It is indeed a choice of "ecosystems" and culture. I see a lot of efforts, both from you, as well as from great people at CivicSpace, to build up a distinct ecosystem based on other products (with their own ecosystem and culture), and within the market of non-profits. There is great promise in that, but in my (current) opinion not more or less promise than in building on an existing methodoloy-driven ecosystem."

Can you explain in more detail what you mean by that? Particularly expounding on what "existing methodoloy-driven ecosystem" means?

Hey Rolf,
Your project looks really cool and I love your focus on building apps specifically targeted to climate change.

I understand that you are not really starting from scratch since Django is a powerful rapid development platform. That said, I wish you were able to leverage some of the tools that already exist, such as ones from the Drupal community, especially since you have an international focus and an international community.

I can admit that I am biased towards the internationalization of Drupal. We work mostly with international development and advocacy organizations, teasing out client needs, finding strategies to address those needs, deploying solutions, and finally measuring results to see if we got it right :) But in the end we make it so that the organization can maintain its site. Teaching and learning with clients are by far the most important aspect of our work. When we do it right, it ensures a really sustainable web outreach strategy in which the organization is running its own site and coming back to us or another vender for just upgraded functionalities to better target and improve their outreach.

Since Greenpeace often operates on decentralized country-by-country basis and campaign by camping level, it is sad that work is not being invested in a stable, international application like Drupal. Drupal has such a large international user groups and support system, both in the user to user sense and on the for profit level with shops all around the world.

For example if your office in Argentina sees value in sms campaigns, they could invest in expanding or testing civiCRM’s sms functionality, which is really picking up speed, thanks to Ben from Mobile Voter and Lobo. Your field office could be benefiting from this in just a couple of months, and improve it so your Chilean office can influence the new political officials after the election yesterday. Congrats on the Zero Waste policy in Buenos Aries - "HASTA LA VICTORIA, SIEMPRE!."

I think this is a good example of how we look at Drupal - not so much as a solution but as a developers platform that has a killer foundation (like those 119 apps the Drum and Zack mentioned) to build out upon and customize for your cause. This gives you time to target tools you need and sleep comfortably at night that there are always a couple of developers up making sure your version of Drupal will stay secure and grow. We chose OSS for all the obvious reason. We chose Drupal/CivicSpace because of the developer base and commitment. Three years later our clients and staff are Drupal heads.

Good luck with this – I look forward to reading your posts and seeing this project grow!