Working Group Minutes/EWG 2015-01-12

Attendees

IRC nick Real name
gravitystorm Andy Allan
insertcoffee Peter Johnson
pnorman Paul Norman
RichardF Richard Fairhurst
TomH Tom Hughes
zere Matt Amos

Summary

  • Breaking up the rails_port
    • It would be possible to break up the database using a FDW wrapper or something equivalent, but that would still leave the website itself monolithic.
    • Dangers of breaking things up:
      • Stuff stops working
      • More "pieces" - makes it harder to deploy and/or develop
    • insertcoffee and people from MapZen will have a look and come back to EWG with a recommendation for discussion.
    • CGImap is moving in a similar direction
      • zere is building a Ruby gem around it [1] - it's very early stages at the moment.
      • The plan is to use the gem, so as to still be able to just 'bundle install' everything, but gradually switch all parts of the rails_port which directly access API details to instead use the HTTP API.
  • Hack weekends
    • There's one in Karlsruhe soon [2].
    • RichardF suggests that a hack weekend with a theme (specific goal and/or piece of software) would attract more people and keep them engaged.
    • It would need more 'organisation' and structure, and more people involved in setting it up and running it.
    • It would need more than one room, one for talking/demoing/helping setup and other quieter one for heads-down hacking.
    • ACTION: RichardF, gravitystorm, pnorman, TomH & zere: think of a theme for a hack weekend to discuss next week.

IRC Log

17:31:50 <zere> minutes of the last meeting: http://wiki.osmfoundation.org/wiki/Working_Group_Minutes/EWG_2015-01-05
17:32:01 <zere> not much in there, but if anything needs changing, let me know
17:32:16 <zere> on the agenda for day: breaking up the rails_port :-)
17:32:26 <zere> #topic breaking up the rails_port
17:33:24 <zere> the idea here is that the rails_port is pretty monolithic, but there are areas of it (the GPX traces, the API, perhaps others) which we can start breaking out as separate services
17:33:40 <zere> each of which is then smaller, easier to maintain, lower risk to deploy, etc...
17:34:01 <zere> i was reading http://blog.carbonfive.com/2014/05/29/an-incremental-migration-from-rails-monolithic-to-microservices/ earlier, but haven't fully digensted it yet
17:35:01 <insertcoffee> will read in more detail but the bullets for "A Three Phased Attack" are a good strategy of attack
17:35:53 <insertcoffee> while it's tempting to change an API while also doing a migration it's probably best to do one at a time
17:36:13 <zere> yup, i agree... makes it easier to iterate
17:36:32 <zere> i was thinking about what we'd need to break off GPX API/database to its own service
17:36:51 <zere> and i was wondering if it would require breaking off the user stuff also / first.
17:37:19 <insertcoffee> yes, from what we discussed I think the traces feature is a good starting point as it has the least interdependencies
17:38:28 <insertcoffee> re: user relations, are you thinking we may need to open a user API to ensure the data is valid?
17:39:11 <gravitystorm> it's not clear to me yet how much is work required at the rails_port level and how much is a deployment thing. For example, doing FDI wrapper stuff at the postgresql level rather than through rails
17:39:19 <insertcoffee> or could this be tied to the session data?
17:39:38 <gravitystorm> and I mean "not clear to me" as in "I haven't tried to see what works"
17:39:50 <insertcoffee> ie. you are auth'd as insertcoffee, so you can act on those records and create as insertcoffee
17:40:01 <zere> i was thinking that the point of intersection between the GPX API and "the rest" is the users table. thinking in terms of services, this means having a service (or services) to manage user state - both long-lived and at the session level
17:40:36 <insertcoffee> ACK, but the question is whether is NEEDS to be done before the GPX API
17:40:59 <zere> yes, that's the question :-)
17:41:16 <insertcoffee> We can have an engineer explore this issue and report back here for further dicussion
17:41:19 <gravitystorm> our approach to users wrt rails_port( + gpx) + help + wiki + osmfoundation + forum needs a fair bit of careful thinking, and for once I'd recommend a bit of forward planning over jfdi
17:42:22 <zere> i agree, that's why i brought it up. i'd love to see user accounts extended to other services, but that seems unlikely in the short term (e.g: people often have different account names for the wiki and main API)
17:43:17 <zere> insertcoffee: sounds like a great idea. i've no real experience designing micro-services stuff - i'm kinda trying to think about this intuitively, but there's only so far that can work
17:43:20 <insertcoffee> I personally feel the User API will end up being far more complex than we think and starting small would be better, then tackling the 'harder' APIs once we have something done
17:44:25 <gravitystorm> we already have various levels of integration with help and the forum (iirc) albeit not with the wiki
17:45:00 <gravitystorm> so it's worth making sure we don't break existing stuff
17:45:09 <insertcoffee> So as a first step. mapzen will take a stab at a recommendation of how the traces feature could be split out and we will discuss that as a group at a future meeting
17:45:30 <insertcoffee> Can we document this stuff somewhere, so whoever works on it is well informed?
17:46:30 <zere> backing up to a previous point: there's a difference between splitting at the database level (e.g: with something like FDW) and splitting at the code level - the latter decouples to a greater extent and (hopefully, at least) makes both components smaller and easier to develop & maintain
17:47:05 <zere> insertcoffee: yeah, these meetings are all archived in full at http://www.osmfoundation.org/wiki/Engineering_Working_Group#Minutes
17:47:11 <insertcoffee> I think we try to handle both those points at the same time
17:47:25 <insertcoffee> is there an easier way than reading through all the past meeting notes?
17:48:55 <zere> i'm not sure i understand - an easier way to do what?
17:49:35 <insertcoffee> re: "so it's worth making sure we don't break existing stuff"
17:49:51 <insertcoffee> I mean, so the engineer understands what not to break ;)
17:51:26 <zere> oh, that's about the user API... not sure that's really documented anywhere at all. basically, some services (the forums for sure, but i'm not sure about help) abuse the user/details API in an attempt to use OSM like an OpenID provider.
17:51:38 <insertcoffee> oh ok
17:52:01 <zere> as long as the current API for rails_port is maintained, it shouldn't break anything
17:52:14 <zere> for some values of "shouldn't", anyway ;-)
17:52:15 <insertcoffee> FYI, I have to run in 5 minutes, really sorry but we have a 2015 planning meeting that I must attend at 18:00 GMT
17:52:31 <zere> no worries
17:53:06 <insertcoffee> so, I will figure out who will work on this; someone senior; and get them to look over it to understand it better
17:53:19 <insertcoffee> then we will discuss specific points in a future meeting
17:53:21 <insertcoffee> sound good?
17:53:54 <zere> yeah, that sounds great :-)
17:54:17 <zere> in my searches on the internet earlier, i was hoping to find an article written by someone who'd already done something like this
17:54:28 <zere> i.e: broken up a large, already-existing, complex application into services.
17:54:51 <insertcoffee> ok cool, I better run away then, please email me if you want to discuss further :)
17:54:53 <zere> but most of what i can find is either trivial (as in the article i linked to earlier) or greenfield.
17:55:02 <zere> insertcoffee: cool, thanks!
17:55:09 <insertcoffee> I have a fair bit of experience there, mostly in nodejs
17:55:17 <insertcoffee> kk, thanks, cya!
17:56:54 <zere> gravitystorm, pnorman: what do you think about what we've discussed so far?
17:58:46 <gravitystorm> zere: it's a reasonable place to start, lets see how things go. I have a desire to ensure that the separation remains seamless for hackers and private deployers (i.e. doesn't make the rails_port a nightmare to set up) but that, of course, is secondary to OSMF operations
17:59:23 * pnorman doesn't know rails
17:59:40 <TomH> well I assume we're only talking about a couple of components right? not fragmenting into a billion pieces?
17:59:49 <TomH> rails for web, cgimap for api?
17:59:57 <TomH> maybe something separate for gpx api
18:00:10 <zere> gravitystorm: i completely agree, and i think how the application gets carved up will make a big difference to how "locally deployable" it is.
18:01:36 <zere> yeah, i think we're talking about carving off one component at a time. it will probably make sense when to stop, as we'll see the benefits start to dry up.
18:02:31 <TomH> well I don't even see it that black and white
18:02:44 <zere> it's still not clear to me whether we'd do this at an "internal API" level, and leave all the GPX-related UI stuff in rails_port, or whether it would be a completely separate application.
18:02:44 <TomH> it's more like "finish off moving api to cgimap"
18:02:53 <TomH> then "gradually rewrite web code to use api"
18:03:18 <TomH> I don't think you can sanely split the web into multiple pieces
18:03:34 <TomH> unless you decide to literally make it completely separate with it's own design etc
18:04:19 <TomH> but anyway, do we actually somebody volunteering to do all this?
18:04:27 <TomH> or is this just pie in the sky?
18:04:57 <gravitystorm> TomH: yep, insertcoffee/mapzen are tasking an engineer to investigate
18:05:16 <zere> i've been working (slowly) on the cgimap stuff, and it sounds like insertcoffee and mapzen are willing to put some effort into it as well. (or, what gravitystorm said)
18:05:30 <TomH> cool
18:05:52 <TomH> I mean that definitely one of the the first things (and relatively easy for the read stuff at least) to think about
18:06:35 <zere> sadly, i think for compatibility, i'm going to have to make some changes to cgimap internals. basically, so that i can extract the (marked up) status code, headers and body separately. currently it writes it all to a stream, which i don't fancy parsing.
18:07:43 <zere> gravitystorm: you might find this interesting - the cgimap gem now has a (pretty close to full) test suite: https://github.com/zerebubuth/cgimap-ruby
18:07:56 <gravitystorm> zere: I was just looking at that :-)
18:07:59 <TomH> zere: what do you need to extract them for?
18:09:43 <zere> TomH: unless the rack request supports socket hijacking, the status code, headers and body have to be added separately to the response.
18:10:15 <zere> of course, we'd prefer socket hijacking in practice, but i'm fairly sure whatever internal mocking rails tests use don't support it
18:10:33 <TomH> oh for cgimap-ruby you mean
18:10:42 <zere> yup
18:10:43 <TomH> sorry was just think of fastcgi version
18:11:05 <TomH> forgotten now why we even want the ruby one?
18:11:18 <zere> or, indeed, if we wanted to use a different front-end other than fastcgi. i was looking at adding a libmicrohttpd one
18:12:10 <gravitystorm> TomH: we could use cgimap-ruby to gut out all the xml-building code from within the rails port
18:12:17 <zere> the ruby gem can be included in rails_port, and then it "just works" without needing anyone to run a fleet of separate (and hard to deploy) servers.
18:13:06 <gravitystorm> my plan is to use that to mean we can do authenticated things where the auth stuff is done in rails but the xml-stuff handed over to the gem. zere has other ideas though which are similar but different.
18:13:18 <zere> right, and then you can have a smaller rails_port which either runs cgimap in-process (e.g: for local dev or small deployments), or as separate fastcgi/http servers for better control.
18:14:11 <zere> eventually ("gradually rewrite web code to use api") the rails_port itself will end up not using the gem, which will be removed.
18:14:20 <TomH> yeah I'm conflicted - my real preference would be to rip the api code out of the rails stuff completely
18:14:21 <zere> so it's kind of like a transitional scaffold.
18:14:29 <TomH> but I understand that makes it hard for people to wor on it
18:14:37 <TomH> hell I'd have to look at my own dev setup
18:14:47 <gravitystorm> zere: from what I can see of cgimap-ruby it looks like it's about ready to try crowbaring it into the rails_port (purely as a demo/hack I know). Is that right?
18:15:51 <zere> gravitystorm: i think you might hit this issue where the rack implementation (rails mocks, webrick, etc...) don't support socket hijacking. but i haven't tried - it's certainly close to that point.
18:16:08 <gravitystorm> sounds like a hack weekend kind of thing
18:16:43 <zere> i've been working on it today, but i've wasted some time exploring some dead ends ;-)
18:16:47 <zere> hopefully next monday.
18:16:54 <zere> speaking of hack weekends...
18:17:09 <zere> i know there's one that woodpeck is running in karlsruhe soon
18:17:45 <gravitystorm> yep, can't make that unfortunately. I had an idea to run one in London simultaneously but I'm busy that weekend too.
18:18:21 <zere> more info for the minutes: http://wiki.openstreetmap.org/wiki/Karlsruhe_Hack_Weekend_February_2015
18:18:25 * pnorman is going to be in Karlsruhe
18:21:12 <zere> we could just do what we've done in the past, but i'm not great at running these things on a regular schedule. i was wondering if there might be an opportunity to get a group of people together, to spread the burden of organisation?
18:21:37 <zere> or, more succinctly: gravitystorm, do you fancy helping out?
18:21:56 <RichardF> I wonder if _possibly_ the way to get hack weekends that attract a wider crowd (and hence fulfil EWG's remit) is to theme them
18:21:56 <gravitystorm> zere: yes
18:22:31 <RichardF> with the notion that people who aren't likely to come to a general OSM hack weekend might come to a "OSM C++ hack weekend" or an "OSM mobile app hack weekend" or something
18:22:34 <zere> RichardF: welcome, lurker! :-) i'm intrigued - could you expand on what you mean by "theme them", please?
18:22:40 <zere> #topic hack weekends
18:22:40 <RichardF> i.e. you get C++ developers to the former, mobile developers to the latter
18:23:21 <RichardF> or even a more definite theme than that, e.g. "we are going to write an <n>"
18:24:11 <zere> i've always shied away from that, thinking that we might put off people who don't feel like they could contribute, or add a layer of organisational burden, or something... clearly it would need to be more "organised" than i'm generally able to manage ;-)
18:25:27 <zere> putting it another way, it's a lot less "casual"...
18:25:59 <zere> how would you see it working? we pick a topic, or a goal, and spend some time briefing people on the first day?
18:26:21 <RichardF> it might not be something we need to consider until we actually have a project that needs it. but (say) if we decided OWL needed resuscitating, yet we didn't have anyone with the skillset to do so, organising a hack weekend for that could be a good way to get capable people
18:26:37 <zere> i know the workshops/lectures at previous hack weekends were very enthusiastically received.
18:26:49 <RichardF> ideally you want to provide a briefing in advance (i.e. coming along to the hackday? you might want to read this / check this out first)
18:27:10 <RichardF> given that IME large% of any given hackday is spent installing the thing I want to hack on :|
18:27:56 <zere> is it better to have one project/focus, or allow several to provide coverage/variety?
18:28:31 <RichardF> with my marketing hat on I'd say "one, for clarity". though I suspect you'd probably get the hackday regulars along anyway
18:29:49 <zere> e.g: if we ran one on, for the sake of argument, original XAPI... that might be a tough call. "first, install this random binary kernel module, then download MUMPS..."
18:30:01 <zere> i know many people feel the same way about C++...
18:30:11 <zere> or, for that matter, actionscript :-)
18:30:31 <RichardF> true. you'd probably want to say "don't worry if you can't get it set up, we'll help at the hackday". but if you can get a few people to hit the ground running, so much the better
18:30:49 <RichardF> I guess "familiarising more people with codebase X" is probably a good outcome for any hackday anyway.
18:31:08 <zere> so you at least need a core of >1 person to do 1:1 or small-group tutoring.
18:31:19 <zere> it becomes semi-educational
18:31:59 <RichardF> ideally 2+ people who are clued up on the project in question, yes.
18:32:06 <zere> and, i guess, the thing needs to be broken down into "bite size" chunks previously, so that people can work on multiple aspects concurrently
18:32:27 <zere> damn, that requires a project with a truck number >1 in the first place ;-)
18:32:41 <RichardF> either that or a new project where everyone is starting from 0 ;)
18:33:39 <RichardF> it's just an idea - can you think of any projects that could benefit from such an approach?
18:34:21 <RichardF> osm-carto might be one but I'm not convinced there's a sufficient number of github-savvy cartographers/designers to run one (in the UK, that is - perhaps more likely in the US)
18:34:22 <zere> in terms of the educational aspect - and raising the truck number: all of them
18:35:23 <RichardF> come to think of it, it's maybe a good argument for de-monolithing the Rails port - smaller codebases to grasp => more approachable to new developers => run hackdays on single aspects
18:36:24 <zere> as a very hands-on workshop flavoured hack weekend, i can see that working. prepare materials up-front, spend the saturday morning on setup / orientation, then the afternoon in "education" mode, then the sunday on "now implement this".
18:36:44 <zere> gravitystorm: what do you think?
18:37:00 <TomH> but do we have any project where there are enough people interested in learning it to make this work?
18:37:14 <zere> i don't know. but it might be interesting to find out
18:38:56 <gravitystorm> so my feelings are: * we should go with osm-themes (e.g. owl) rather than technology themes (e.g. c++). Most of the osm projects need a fair amount of cross-discipline work anyway, like designing svg icons + carto + db profiling for osm-carto
18:39:48 <gravitystorm> I do find myself running into a wall with no-theme hackdays since it's often very distracting, but I also know that people like them since they can't otherwise commit to 2x10 hours coding at home of a weekend
18:40:37 <gravitystorm> so I think we should try having a theme, but with a FAQ at the bottom saying you're welcome to attend and work on other osm-things
18:41:12 <gravitystorm> and I think it would be good if, like at sotm-eu, there are multiple rooms e.g. one for talking/demoing/helping setup and other quieter one for heads-down hacking
18:42:52 <RichardF> makes sense.
18:43:59 * TomH doesn't remember that, but it makes sense I guess
18:44:36 <zere> i had an especially quiet room for heads-down hacking ;-)
18:44:52 * RichardF had a quiet mountainside :)
18:45:32 <zere> this all sounds very reasonable so far. so do we start with a theme, or start planning and leave the theme until later...?
18:46:27 <RichardF> are there any themes that immediately spring to mind?
18:47:20 <gravitystorm> well, I want to hack on cgimap-ruby :-)
18:47:53 <gravitystorm> so a general 'rails_port' theme could work. The downside is that it's a big beast.
18:47:53 <zere> OWL has been mentioned already. i don't know what the current state of that is, but it could probably do with some help.
18:48:38 <gravitystorm> could owl be one of those things where a bunch of effort gets it working again? i.e. a hack weekend with a goal?
18:49:01 * gravitystorm reminisces about the i18n hack weekend
18:49:09 <zere> i think "rails_port" as a whole is too big. but some aspect of it might work... but i can't think of such an aspect right now... is "groups" still a thing?
18:49:43 <zere> yeah, i don't know enough about the current state of OWL to know how far it might be from "working again".
18:50:00 <RichardF> I'd love it if groups was... it's been in abeyance for a while because I hit a coding wall, but it'd be great to resuscitate it, and there've been sporadic bits of interest
18:50:32 <gravitystorm> does EWG have an up-to-date list of osm projects that it's interested in? Or is https://github.com/gravitystorm/osm100/blob/master/projects.yml still a reasonable list?
18:50:54 <gravitystorm> (it came from the developer survey a while back)
18:51:46 <zere> RichardF, gravitystorm, pnorman, TomH, zere: we all get homework (a.k.a: an action) - think of a theme for a hack weekend to discuss next week :-)
18:52:10 <RichardF> ok :)
18:52:36 <zere> i think that list is about right, except perhaps jxapi should be taken off and/or replaced with pyxapi. i don't think jxapi is actively maintained any more
18:52:55 <zere> perhaps pyxapi isn't either... overpass has rather taken over that space.
18:53:48 <zere> and i'm sure there's lots of interest in vector tiles, too. not to be too parochial about it, but it would be awesome to get avecado into a "apt-get install working-osm-tileserver" state.
18:53:56 <RichardF> hell you're right
18:54:04 <RichardF> vector-tiles-on-osm.org is a hack weekend in itself
18:54:56 <pnorman> zere: I have instructions for installing avecado
18:55:16 <zere> pnorman: sure, but avecado != working-osm-tileserver ;-)
18:55:48 <zere> RichardF: that's excellent! a more modest goal than working-osm-tileserver, but would still be very useful
18:56:35 <zere> hopefully we can continue next week, when we've all done our homework.
18:56:42 <RichardF> I do have an idea for a future project and hack weekend that has "osm.org can serve vector tiles" as a dependency, but that's a story for another time ;)
18:56:58 <zere> but, until then, is there anything else anyone wanted to discuss?
18:57:02 <zere> #topic AoB
18:57:29 <pnorman> zere: well yes, but that gets into having a render stack that's sane to deploy
18:57:44 <zere> yup, that's why it's so hard :-/
18:58:02 <pnorman> currently we don't - mod_tile/renderd is scary, as are others
18:58:47 <RichardF> it may be obsoleted by vector tiles, but right now I have a project where I'd love UTFGrid in mod_tile/renderd
18:58:49 <zere> mod_tile/renderd is scary, mapnik-master is scary, osm2pgsql (+ postgres, on some platforms) is scary, downloading planet is scary :-)
18:59:15 <RichardF> at present the alternatives are (a) learn Tilestache (b) pregenerate all the tiles. I'm leaning towards (b)
18:59:40 <pnorman> osm2pgsql data loading isn't too bad on ubuntu - I've pointed completely new people at http://switch2osm.org/loading-osm-data/ and had them follow it successfuly
19:02:06 <TomH> RichardF: I think vector tiles is a bit more than a weekend!
19:02:45 <zere> sure, due to the huge efforts of a small number of people, it's much less scary than it used to be. but i'd say it's still pretty scary for a lot of people.
19:02:50 <pnorman> zere: for one of my personal projects I'll be running avecado_server to generate openstreetmap-carto vector tiles
19:02:54 <RichardF> TomH: oh, absolutely. but perhaps we could get the ball rolling within a weekend, such that people had something to go away and hack on afterwards
19:03:51 <zere> to be honest, i don't think we are more than a weekend away from *an* implementation for osm.org. we might be more than weekend away from a *good* implementation, though ;-)
19:05:31 <zere> i set up avecado_server behind an apache reverse proxy & cache, and it was pretty good. i'm not sure it would scale, but that's a problem it would be easy enough to attack at a hack weekend.
19:05:36 <TomH> RichardF: the technology stack might be a weekend - actually migrating the style is the big job
19:06:13 <pnorman> TomH: a dumb migration of the style is an afternoon. I did it
19:06:32 <TomH> really? I imagined it would be a much bigger job than that
19:06:36 <zere> carto -> webgl compiler as a stream of that hack weekend. sounds good to me :-)
19:06:44 <pnorman> Unless you're talking about migrating to mapbox-gl, in which case, oh god no
19:06:50 <TomH> well in a month or so when the tile servers collapse maybe we can use that as an excuse
19:08:23 <pnorman> TomH: actually all that's required is adding min/max zoom to each layer and then listing the layers in the new style yml MML file. I did the former in an afternoon, the latter is trivial.
19:08:32 <zere> tomh: speaking of which, when was the database on orm last vacuum full'ed or the indexes rebuilt?
19:08:56 <TomH> zere: there is insufficent space to rebuild the indexes
19:09:02 <TomH> well we might be able to do the mall ones
19:09:08 <TomH> have never been able to do the biggest
19:09:18 <TomH> autovacuum should be on
19:09:36 <pnorman> the problem is that you gain few advantages from doing that simple a conversion. and openstreetmap-carto is a huge complex beast
19:10:02 <pnorman> is OWG looking at a 3rd server or upgrading the two existing ones?
19:10:13 <TomH> but last June by the looks of it for the indexes
19:10:28 <TomH> got maybe 5% or so back then
19:10:37 <TomH> but as I say I couldn't do some of the largest ones
19:10:53 <RichardF> <devils_advocate> you could take it as an opportunity to deploy a clean-room style as the default for osm.org, and move openstreetmap-carto to a secondary, Osmarender-like style... </devils_advocate>
19:10:56 <pnorman> it's the planet_osm_ways nodes GIN index, right? that one also bloats the fastest
19:11:03 <zere> ah, so, not even worth stopping updates, creating a tablespace on / (1.5TB free), rebuilding the indexes and moving them back?
19:11:14 <pnorman> RichardF: thank you for volunteering to create the new clean-room st yle
19:11:20 <TomH> pnorman: in the first insance, fix yevaud to be less shit (https://github.com/openstreetmap/operations/issues/5)
19:11:20 * RichardF forks osm-bright
19:11:37 <TomH> and make more space (https://github.com/openstreetmap/operations/issues/6)
19:11:45 <zere> RichardF: with vector tiles you can have client-side styles - as many and as eyewatering as you like ;-)
19:12:02 <RichardF> true :)
19:12:17 <TomH> zere: well yes I could spend hours fucking around and have us running on yevaud for a weekend, or we could just sped a few quid
19:12:44 <TomH> if yevaud was less shit I might be more inclined to go with the first option
19:13:55 <zere> i think we're out of scope for EWG by now anyway. thanks for coming, and hope to see you next week (especially gravitystorm, RichardF, pnorman, TomH & myself, who have homework! :-) )