Digifesto

Category: geospatial software

“SDI Best Practices with GeoNode” Slides

Slides from my FOSS4G tutorial on GeoNode available here:

The talk was well received, and many people I talked to were interested in installing GeoNode. Overall, the conference was a great one.

Preparing for FOSS4G 2010

Two years ago when I had the incredible opportunity to go to FOSS4G 2008 with OpenGeo, I was very new to the scene. I had been working on OpenLayers-based web applications for just a year at that point. Attending that conference gave me a much broader look at the industry I was entering, and was a crucial chance to become better acquainted with the people behind the IRC handles I was learning to recognize.

I’m back this year, with eleven other members of OpenGeo. Our contingent’s preparation in the past few months has been led by Paul Ramsey, who has a deep professional and personal connection to the conference. We are more coordinated than ever before; if you’re at the conference, you’ll see us in matching t-shirts and with advertised “expert hours” at our booth. Our intention is to make a strong showing and become our own self-fulfilling prophecy of a successful and growing open source geospatial company.

A key part of that vision is OpenGeo’s main product, the OpenGeo Suite. As the internal argument goes, software consulting as programming-for-hire doesn’t scale up as a business model. To prove that open source geospatial can really triumph in the industry, open source consulting shops need to evolve into a product-based company that sells support and training, and funds software development indirectly–but more efficiently, as with more flexibility the company can make wiser long-run decisions. So we are here to sell Suite contracts, we were reminded in our team meeting earlier tonight.

But what’s keeping me up at night right now is the knowledge that this year, unlike two years ago, I have something to say: I’m giving a tutorial on Thursday morning with Galen Evans announcing GeoNode to the attending members of the FOSS4G community.

I am quite nervous. Though we have been tweeting and blogging and releasing news bites about GeoNode with increasing frequency in the past year or so of development, we have for the most part been too busy building it to publicize what we are working on. Often when I try to explain the project, I’m still met with “Ok, but what is it?”

Now I’m hoping we can just show people. We’ve been ironing out the bugs from our 1.0-beta release over for a couple weeks now, and despite some worrying regressions I’m confident that what we’ve got to show is something genuinely new, compelling, and full of potential.

But it’s not for me to decide that. I’ve had my nose stuck in this project for a year, and am still fairly green in the geospatial domain. Meanwhile, the community at FOSS4G is full of jaded industry veterans, well aware of the alternatives and the pitfalls of new software projects. And their judgment matters: GeoNode depends on many other FOSS geospatial projects’ communities, and for us the ideal is for more open source geospatial developers to see potential in GeoNode and consider contributing.

So a lot is riding on this conference. Even if GeoNode is a commercial success, it won’t stand for what it ought to unless it is also a community success. And for that to happen, it needs to earn the respect of colleagues at FOSS4G.

I suppose I really should get back to working on my slides…

Job opening at OpenGeo for GeoNode

It looks like OpenGeo is going to be getting a lot of business in the coming year for GeoNode–so much that we can’t handle it with the size of our current team.

So we’re hiring a new Software Engineer.

What we’re looking for is somebody who has both the engineering chops to handle our complex stack (which includes Django, GeoServer, GeoNetwork, and GeoExt) and the inclination and aptitude to provide on-site consulting, training, and support to clients. Because of our partnership with the World Bank, many of these deployments will be around the world. So this job will likely offer the opportunity to go interesting places and meet interesting people, on top of coding for a very well poised open source, open data project.

OpenGeo, I have to say, is a kickass place to work. I’ve been working for it and our parent organization OpenPlans for almost three years now, and I am challenged and inspired daily by the intelligence of the people around me and the dynamism of the organization. It’s also committed to openness with a sincerity that is extraordinarily compelling to the right kind of person.

This blog post, for example, divulges crucial information about our business strategy in an unprofessional medium. “What is going on?” you might ask. “Why is he doing this? Is it a corporate ploy?” No. Our business strategy isn’t a secret. On the contrary, we want as many people to know about it as possible so that more people will get involved in our open source projects and communities. And though our work and life itself demands humility, on rare occasions (just as when we are trying to attract applications from Software Engineers) we get to brag about how great we are.

Something else I like about us? We are scrappy. OpenPlans has subsisted primarily on donations from Mark Gorton for most of its existence. We are in a turbulent time, because the LimeWire lawsuits mean that we can no longer depend on Mark. But seriously in the nick of time, we warming up our business development machines.

The leads are coming in, and we are growing (did I mention that we are hiring?). It feels like the helicopter is lifting off just as the ground beneath us is melting into hot lava. And yet, we aren’t afraid. Yesterday I saw Vanessa Hamer give a budget report for OpenPlans. The gist of it was: even assuming that lots of things go wrong with our funding pipeline, we are well prepared to thrive in the future. I have never had more confidence in this organization than I do now.

This is a great thing. It is great because OpenPlans and OpenGeo are working on great things. OpenGeo in particular is in the early stages of transforming the geospatial software industry from one dominated by proprietary desktop software to free web software. It is providing tools that will help institutions transform their closed and cobwebby data silos to nodes in a vibrant, open geospatial web. For great justice.

“Are you for real?” Yes, I’m for real. OpenGeo is that good. I know we’re that good because we have attracted successful geospatial software industry entrepreneurs (1) (2) (3) as our upper level management in the past year. These folks aren’t just all geniuses. They are geniuses with domain expertise who are passionately dedicated to transforming an industry with an impact on millions of people.

We also have some great connections going on. I believe I’ve mentioned that we’ve been working with the World Bank’s CAPRA initiative on this GeoNode project (the one we are hiring for). The people behind CAPRA are rockstars within a superorganization with a mission to save Central America from natural disasters.

Oh, and we’re going to be working with the Global Earthquake Model soon. Those guys are a crack team of brilliant scientists with a mandate to unite all earthquake modeling under one banner using open source software. If it works, it will have an impact on everyone who is in an area that suffers earthquake risk, because earthquake modeling is used by everyone from humanitarian NGO’s who do disaster relief to insurance companies who wind up taking on a lot of the corporate/economic risk of natural disasters. It’s a mandate that would be ludicrously ambitious if it weren’t happening right now with web technology that has just recently been perfected and available.

It is so fucking cool I can hardly contain myself. I have to suppress these thoughts normally so I don’t get so distracted by how cool it is that I don’t get any work done. It is so cool that it’s hard to talk about publicly because it all just seems so unbelievable.

That is what we are hiring for right now.

Data Uploading on GeoNode

We’ve made great strides in GeoNode development in the past few weeks, due in part to the addition of Luke Tucker, on loan from Civic Works, to the team.

The main improvements lately have been ones slated for the 0.1.2 milestone, which focuses on basic data management through the web application interface.

The most important improvement is the data upload functionality. We provide a web interface for SHP and TIFF upload and then push the file to GeoServer using its REST configuration API and gsconfig.py. The layer is then exposed via OCG services like any normal GeoServer layer, and so is available for use by OpenLayers and GeoExt applications, for example.

Hat tip to GeoCommons for inspiring the UI for uploading the multiple files needed for Shapefile data.

You can get a taste of some of the other improvements by checking out the Data and Map information pages. In GeoNode, both data sets and maps (compilations of data, with a layer order and styles specified) are first-class content. We want to let users transition seamlessly between working with data and working with maps in order to smooth out the web GIS experience.

The “information pages” (I think we are looking for a better name for these, internally) serve as dashboards to a particular data set or map, and expose the relationships internal to the content. By viewing which maps a data set is a part of, the user can find out about who trusts the data, who finds it authoritative. This is just the first of many ways we plan to let social use of data provide relevant metadata.

What’s this GeoNode thing?

I’m going to start blogging here about the GeoNode.

If there is anything I’ve learned at The Open Planning Project and OpenGeo, it’s that it is critical for the success of open source projects that they be “truly open”–not just with an open source license “slapped on,” but with a transparency built into its development and management process.

So I am chagrined that the GeoNode, the project I’m currently most involved with, has been developed in relative obscurity. At OpenGeo, we’ve been treating it like an internal project, and though its task tracker is open, documentation is sparse.

The reason for this is simple: we have been too busy building it to properly groom it. But hopefully that will change soon.

So what is the GeoNode? Here’s the backstory, as articulated by Eddie Pickle:

The GeoNode project is an open source initiative backed by the World Bank, OpenGeo, and others with pilot programs in El Salvador and Guatemala. It aims to give a modern spin to the idea of the spatial data infrastructure with an eye to the use and reuse of data, encouraging collaboration, and promoting open data exchange as a default. Initially, the development of the GeoNode has focused on serving and visualizing the data created by the Central America Probabilistic Risk Assessment (CAPRA) to assess and mitigate the risk due to adverse Natural Disasters.

At its core, the GeoNode is based on open source components GeoServer, GeoNetwork, Django, and GeoExt that provide a platform for sophisticated web browser spatial visualization and analysis. Atop this stack, the project has built a map composer and viewer, tools for analysis, and reporting tools. The GeoNode also supports facilities for styling data as well as collaborative features like ratings, comments, and tagging for data, maps, and styles. The GeoNode is built on four key tenets: Collaboration, Distribution, Cartography and Data collection.

This is the GeoNode vision–a vision which is current under (rapid!) development. But at the time of this writing, the use of the present tense in the paragraphs above is sometimes ambitious. You can check out the demo that we update weekly from GeoNode trunk here.

In its current state, it requires some explanation. What you’re seeing is a Django site that is configured to work against a GeoServer instance that contains the spatial data hosted by the GeoNode. The map on the front page is a GeoExt map embedded as a widget. This Map-As-Embeddable-Widget is a feature of the GeoNode’s Map Composer, which is closely related to GeoExplorer, OpenGeo’s main GeoExt-based web GIS client.

The site’s theming is, as requested by our partner and client, made to look like the World Bank’s CAPRA website. Rolando Penate has been working ideas for generic GeoNode themes that we hope to implement soon. His design wireframes show off some of the potential of these themes as well as the features that we are scheduled to implement in the coming year: user registration, commenting and rating, a groups.

The purpose of all these “Web 2.0” features (cringe!) is to provide a layer of social metadata over the geospatial data and ISO metadata that we will be supporting with an integrated GeoNetwork instance. If the internet has proved anything, it’s that the best catalogs are those created dynamically from real usage. So our plan is to integrate cutting-edge web GIS tools with the social features of the site so that as users participate in the GeoNode environment, we collect data about their activity and use it to refine search results. We joke about whether the GeoNode aims to be the MediaWiki/Twitter of maps, or the Facebook/YouTube of maps, or the Yelp/Wordpress of maps…but really these features are all so commonplace on the web now that it’s not a matter of imitating the market leaders. It’s a matter of taking solid modern principles of data management and bringing them intelligently to the Spatial Data Infrastructure domain.

We are several months away from these features, however. For now we are focusing on building a solid backbone of tools that will make the GeoNode attractive to users.

If you’ve been following GeoExt development lately, you may have noticed Andreas Hocevar’s contributions of JavaScript client code that works against the Mapfish Printing module. That’s all been core development work funded by our GeoNode partnership.

Another neat budding project worth noting is gsconfig.py, developed primarily by David Winslow. It’s a Python library for interacting with GeoServer’s catalog via its REST API. We are going to be using it to provide through-the-web data upload and metadata management through a Django-powered user interface.

These two open source contributions highlight what’s for me one of the coolest parts of our GeoNode development: that we are dedicated to bringing as much as possible of the work for the GeoNode back into widely recognized open source geospatial projects or splitting parts out into general purpose libraries. This is possible because of the NGO and government community partnership around the GeoNode that is being spearheaded by Stu Gill, of World Bank CAPRA. Stu is a visionary who sees very clearly the decisive role openness will play in the future of SDI, GIS, etc. And he understands that openness will play an important role all the way down to the open source core, and has been reaching out to like minds.

OpenGeo’s GeoNode team has been hard at work on the prototype GeoNode software and the open source software improvements that it depends on, but in the coming months we hope to shift our priorities more towards documentation and community. We’ll be announcing those advances through GeoNode.org, the official site of the GeoNode project. But we’ve unintentionally left the lid on this for too long. In the short term, I’ll be blogging informally about GeoNode progress week-by-week here, because it’s high time we got the GeoNode into the open.

Computers are for computing

One of my favorite talks from FOSS4G this year was Josh Livni‘s talk on Walk Score, a web service that calculates the “walkability” of an area based on publicly available data.  Walkability is calculated efficiently right against the database according to an algorithm that takes into account how easy it is to get around–and get to points of interest–by walking.  Then it displays the results using Google Maps.

It took me a while to realize what I liked about Walk Score so much.  It isn’t a fully open source stack, and though “walkability” is important to me, I don’t really have a use for this service beyond checking out the walk score of my home town.  And yet it appeals to me and has been a generally popular site.

Then I realized: this project appeals to me because it computes something interesting.

A frustrating aspect of the world of open source web GIS is that most projects appear to be hung up on the problems of making data available over the internet–in various formats, in certain combinations, with certain metadata, but otherwise essentially untouched.  Where modifying data is supported (say through WFS-T), it has to be done painstakingly by hand.

I don’t want to minimize the challenges of building the foundations that have taken so much effort so far.  But I think that what gets missed in the process is the fact that the most compelling applications compute something useful.  What people really want and need is software that thinks for them.  Or, maybe, discovers something for them.

What Walk Score does, which few applications I’ve seen this week do, is calculate something interesting for people.  Livni had the creativity to turn a human interest into a quantitative, algorithmically calculable metric, and found a way to report that metric back to people in a way they could understand.  It provides people with something two steps ahead of them, just beyond the horizon of what they can imagine.  That’s true progress.  I hope to see more of it in FOSS/Web/GIS applications in the coming year.