Osm2pgsql project 2020-07

From OpenStreetMap Foundation
Revision as of 15:48, 30 July 2020 by Stereo (talk | contribs) (Text from Jochen's email)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Jochen writes by email:

OSMF Support for osm2pgsql Development

Osm2pgsql is an important building block in the toolchain from OSM data to the maps on openstreetmap.org and many other maps. Development and maintenance has so far mostly been dependent on work of a few developers in their free time with some support from companies here or there. As the long list of open issues on Github shows, this has not been enough to keep moving osm2pgsql forward.

I have written up a draft road map that shows where osm2pgsql is now and where future development might go (attached). Approval from the maintainers on this is pending, so it might still change a bit.

Based on this road map I propose that OSMF funds my work on maintenance and development of osm2pgsql along the following lines:

I'll work for one person-month at 50 EUR/h (times 160 hours = 8000 EUR) spread over two to three months on osm2pgsql. Billing for the whole sum will be at the end of that time.

This is, of course, not enough time to tackle everything on the road map. But it should get us a good step forward. I propose work will focus on the following tasks (pending maintainer approval):

  • Ongoing maintenance as needed (looking at issues coming in, fixing bugs, cleaning up code, etc.)
  • Set up of osm2pgsql.org (*) as a new site for documentation and consolidate existing documentation there as well as write new documentation. (About 1 week.)
  • Rethink output of the program to make it more concise and useful and consolidate the code for that. (< 1 week.)
  • Tackle refactoring and cleanup of the "middle" code. This is an important part of the internal infrastructure of osm2pgsql that's needed for other improvements. (About 2 weeks.)
  • Other work from the road map as time permits in consultation with the maintainers.

I have written about my work on osm2pgsql in the lasts months on my blog at https://blog.jochentopf.com/ . While working for the OSMF I would continue to do so. I am also available to the community for questions, ideas, and discussions in the usual channels like the dev mailing list. I would also suggest that we organize some kind of online meeting with interested parties to discuss the mentioned road map and get feedback from the wider community.

Road Map for osm2pgsql

Current as of 2020-07-26.

This document is a kind of road map for osm2pgsql development. It’s not to be understood as a definite “this is what we’ll do” document, but as a rough overview of the shared understanding of the maintainers about where we are and in what areas we see need for work. It is incomplete.

Where we are: A Stable Platform

First and foremost osm2pgsql must be a stable and reliable platform for its many users who use it every day to maintain current maps. Backwards compatibility is important and must be kept wherever possible. This makes developing new features more difficult, but it gives the users a way forward, using new features as they are needed without the need for a break-the-world update.

This also means that often ongoing maintenance, bugfixing, attending to issues, making sure everything keeps working with old and new library and operating system versions etc. is more important than new features.

The development mostly is done on Linux and most users use osm2pgsql on Linux. Osm2pgsql must work well on all current and some older versions of major Linux distributions.

Because we currently have no Windows or OSX developers the support for these operating systems is “best effort”, we try to keep it working but can not promise anything.

Only 64 bit systems are supported.

Osm2pgsql currently needs a C++11 compiler, switching to newer C++ versions will be considered based on availability in the Linux distributions we want to support.

Osm2pgsql must always support all officially supported versions of PostgreSQL (currently 9.5 and above) and maybe more. Some optional features might only be supported in newer PostgreSQL versions.

Osm2pgsql wants to be as resource-friendly as possible. It must always be possible to use it with small data extracts on a hobbyists laptop. Processing the full planet file and running minutely updates must be possible on a reasonably modern machine (64GB RAM, SSD).

Where we want to go

The following sections describe major topics or areas of work. They are roughly ordered from the more important, simpler, more near-term to the less important, more complex, more “out there” ideas. This does not imply an order in which problems will be tackled (or even whether they will be tackled at all).

Note that many of the following topics overlap. Some topics have related issues which are listed below, but not all open concerns appear in Github issues.

Ongoing Maintenance

There is always the ongoing maintenance.

Issues: