Working Group Minutes/EWG 2012-02-13
Jump to navigation Jump to search
|IRC nick||Real name|
- zere asked for feedback on a pseudo-ruby license-change algorithm .
- There was discussion of vector tiles:
- What is a vector tile? There are a range of different possibilities at different points on a scale of generality (OSM XML) and ready-to-useness (SVG).
- It was a general consensus that a middle-way, with some geometry pre-processing and filtering, was probably the most useful. Something like GeoJSON, which can be output with KothicJS' engine or Tilestache.
- Vector tiles discussion led into a discussion of how to get the best of both Imposm and Osm2pgsql together. A simple proposal of using views and partial indexes might produce benefits, but there were doubts over the performance.
18:01 < zere> minutes of the last meeting, for your enjoyment: http://www.osmfoundation.org/wiki/Working_Group_Minutes/EWG_2012-02-06 18:02 < zere> if there's anything wrong, please let me know. 18:04 < zere> last meeting i said i'd put together a code-y algorithm for the license change cleaning, and here it is: https://gist.github.com/1817152 18:05 < zere> it's sort of pseudo-ruby, might be a little verbose, but pretty close to working code 18:05 < zere> it's based on what frederik has told me of the OSMI license change layer process 18:09 < migurski> reads pretty clearly. I'd love something similar for osm2pgsql's polygon internals. =) 18:10 < zere> there's a few areas of uncertainty - particularly around what constitutes a "harmless" edit of a relation. and i'd welcome any feedback. i'm hoping to clean it up a bit and send it on to LWG. if they're happy that it's close to what they want then we can start building real code around it. 18:10 < zere> migurski: thanks :-) 18:14 < zere> last week RichardF asked to have vector tiles put on the agenda, but he's not about today. anyone want to lead off with something on that topic? 18:17 < migurski> Vector tiles == .osm XML? 18:17 < migurski> or SVG? 18:18 < apmon> For what purpose would the vector tiles be? 18:18 < zere> well, exactly... there's a range of "usefulness" - at one end, there's OSM XML which needs a lot of pre-processing and styling before it's presentable on-screen. on the other hand, there's SVG which needs little pre-processing and already contains styling information. 18:20 < apmon> KothicJS tiles are probably half way between them 18:20 < migurski> I've been talking to the DevSeed people about adding Mapnik support for a tiled datasource 18:21 < migurski> so tiled data is useful for that - it would be possible for people to be up and running with OSM + Mapnik without ever installing Postgres 18:21 < zere> the best reason i can see for having vector tiles is that they're potentially more useful as client-side engines become more powerful. if it's possible to style them then it could be that it's a very good alternative to setting up a completely new tileserver for each new style? 18:21 < migurski> exactly 18:21 < zere> which does rather rule out SVG as a candidate... 18:22 < zere> i guess we're talking about something closer to geoJSON? 18:22 < migurski> yeah something like that 18:22 < migurski> we've experimented heavily with this stuff in the past 18:23 < migurski> and found that geoJSON clipped to six digits of floating point precision + gzip was competitive in terms of file size with custom formats, and easy to deal with 18:23 < apmon> As far as I can tell kothicJS uses a form of geoJSON 18:24 < migurski> it was also very necessary to simplify geometries for tiles below z13 or so 18:24 < migurski> just became unmanageable 18:25 < zere> it's pretty easy to pull geoJSON out of postGIS in tiles, and simplify them en-route. could that be a solution, and allow re-use of the osm2pgsql expiry code? 18:25 < migurski> not familiar with the latter 18:25 < migurski> oh 18:25 < migurski> right 18:25 < apmon> Yes, it should be possible 18:25 < migurski> so, keep an osm2pgsql-to-postgis DB 18:26 < migurski> and run tiles out of that 18:26 < zere> just so we can cache the output, not hit the DB every time someone requests tile 0/0/0 ;-) 18:26 < migurski> yeah 18:26 < apmon> You can also use tirex with mod_tile for delivery 18:26 < zere> is what we're discussing similar to the tirex json backend you've been writing, apmon? 18:26 < apmon> mod_tile can already deliver js tiles (or other formats) 18:26 < migurski> these things can have long cache expiry times - days or more 18:26 < apmon> I think so, yes 18:27 < apmon> It should be very simple. I just haven't gotten around to it. 18:27 < apmon> Partly because I am not too familiar with python 18:27 < apmon> or perl, which is what tirex is written in 18:27 < migurski> *shudder* 18:28 < apmon> The communication between the tirex part and the "renderer" is actually very simple text over a socket. 18:29 < apmon> So you could write it in pretty much any language 18:29 < migurski> I can do a first pass of something like this in Python, using Tilestache 18:29 < apmon> It was just that I wanted to reuse the python code from the kothic server code 18:30 < migurski> ah 18:32 < zere> i wouldn't worry too much about implementation language at this point. just a proof-of-concept and then we can try integrating it with current server code - renderd / tirex / whatever. 18:32 < apmon> http://code.google.com/p/kothic/source/browse/src/ json_getter.py is the code for the backend 18:32 < apmon> It does caching and rendering, but no queuing (which is why I wanted to integrate it into tirex) 18:33 < migurski> I see that kothic dynamically simplifies geometries 18:33 < migurski> it's probably worth pre-simplifying geoms in PostGIS ahead of time 18:37 < zere> that kinda touches on another thing i know i've discussed before - getting something with the diff support of osm2pgsql and the flexibility and multiple table support of imposm. 18:39 < migurski> apmon: the built-in Tilestache Vector provider might serve the need we're talking FWIW http://tilestache.org/doc/TileStache.Vector.html 18:41 < migurski> zere: imposm does all it's own importing, but there's no reason it can't be implemented as a set of postgres queries on top of osm2pgsql based on my experiments with it 18:41 < migurski> so, osm2pgsql for the importy bits 18:41 < migurski> then some very based postgres queries to generate the imposm tables 18:42 < migurski> I've not yet seen anything in those tables that's not derivable from osm2pgsql tables 18:42 < zere> i'm not sure i'd want to rebuild the whole set of tables every minute, or whatever the diff import period is... 18:44 < migurski> they could be views 18:44 < zere> but it might be possible to hook into osm2pgsql's knowledge of which features have changed to limit the extent of the rebuilds. 18:48 < migurski> going to duck out - need to actually go to the office today =\ 18:49 < zere> migurski: cool, thanks for coming :) 18:49 < migurski> bye! =) 18:49 -!- migurski [~firstname.lastname@example.org] has quit [Quit: migurski] 18:50 < zere> i just tried the views method, btw, but unfortunately it seems postgresql doesn't support creating indexes on views, which might make run-time query performance unusable. 18:51 < TomH> it won't create indexes on views no 18:52 < TomH> but it will use underlying indexes 18:52 < TomH> views are implemented by query rewriting in postgres you see 18:53 < zere> ah, so a query like "select way,name from planet_osm_lines where highway='residential' and way && bbox(...)" is the same as "select * from planet_osm_residentials where way && bbox" and planet_osm_residentials is a view "... where highway='residential'"? 18:54 < TomH> exactly 18:54 < TomH> postgres has a query rewriting engine that you can write rules for, and views are implemented on top of that 18:55 < zere> ah, and would require the same set of partial indexes to support any of these at low zoom with any speed. 18:56 < zere> for example, pulling capital cities out of a z4-7 tile, just from planet_osm_point, is pretty horrendous without a partial index. 18:56 < TomH> http://www.postgresql.org/docs/9.1/static/rules.html 18:56 < TomH> http://www.postgresql.org/docs/9.1/static/rules-views.html 18:58 < zere> yikes - rules look a bit like triggers on steroids. plenty of firepower to blow one's leg off ;-) 19:00 < TomH> BTW I plan to push tmcw's edits shortly unless anybody has any last minute objections 19:01 < zere> the whitespace/border changes? 19:01 < TomH> what you see at http://tomh.apis.dev.openstreetmap.org/ basically 19:02 < zere> yeah, i've got no objections 19:04 < zere> thanks to everyone for coming! see you next week :-)