OpenStreetMap Foundation, Operations Meeting* - Draft minutes
These minutes do not go through a formal acceptance process.
Thursday 7 April 2022 at 19:00 London time - Countdown
Location: Video room at https://osmvideo.cloud68.co
* Please note that this is not strictly an Operations Working Group (OWG) meeting.
Minutes by Dorothea Kazazi.
New action items
- Grant to suggest the discussions about community.osm.org / Discourse to move from Matrix to IRC. Topic: community.osm.org / Discourse
- Grant to look in the DIscourse translation plugin. Topic: community.osm.org / Discourse
- Grant to talk to Tom about fixing Wal-G. [Reportage: AWS]
- Paul to park the Github ticket "Adding API key support for tile.osm.org" https://github.com/openstreetmap/operations/issues/342 [Action item updates]
- Grant talked to them again.
- Billing notifications: Updated to automatic 6-month rolling. Set at 15% and 75% total spent.
- Billing tags: Turned on and will get cost per tag.
- Paul wants to do aggregates of logs (e.g. stats on browser agents).
On S3 storage
- Storage backups (bigger storage percentage)
- 53.0 TB Fastly logs, which can be stored on Glacier https://aws.amazon.com/s3/storage-classes/glacier/)
- 50.0 TB Write-ahead log for Postgres (WAL-G, data up to 2017)
- Get compressed before uploaded.
- 40.0 TB Backup of planet (costing little)
- 04.1 TB Fastly processed logs.
Other points mentioned
- We have ~ 204 TB on S3.
- Fastly processed logs used the most.
On tiers and cost
- Paul reluctant to have tier changed automatically, as it has resulted in unexpected costs in the past.
- Intelligent tiering according to Grant is pretty safe. Can cost more if you have insane number of objects.
- Grant prefers to keep them for some time for trend analysis.
- Is costing us (storage costs more than Athena queries).
- Aggregate logs to see trends. Easier to run Athena queries on aggregated data over a year.
- Deletion of IP addresses and some partially personally identifiable data (Privacy/GDPR).
On cost of Athena queries
- Two types of queries:
- Automatic queries for tile-logs - we can't fo anything about their cost.
- Grant is not concerned about the cost.
Action item: Grant to talk to Tom about fixing WAL-G.
community.osm.org / Discourse
Related to https://github.com/openstreetmap/operations/issues/377 - Merge forums, OSQA, MLs to discourse? #ML=mailing lists, OSQA is the software running on help.openstreetmap.org.
community.openstreetmap.org is live.
Discussions about community.osm.org / Discourse
Moved to Matrix.
Action item: Grant to suggest the discussions about community.osm.org / Discourse to move from Matrix to IRC.
- Will do migration from old forum.
- Christian Quest is doing the testing and dealing with some of the issues.
- Some tasks briefly paused as we get ready for migration.
- Google: good from/to English.
- Not DeepL, because:
- doesn't work with Discourse
- languages: has only 20 languages (mainly western European). It is good for the languages it supports. Does have Chinese/Japanese, doesn't have Indian.
- Cost 5000 USD for plugin to be developed by Discourse developers.
Other points mentioned
- Discourse stores cache of translation.
- Christian Quest has tested the plugins.
- Approximate cost of translation plugin: 20 USD/month.
Action item: Grant to look in the Discourse translation plugin.
- To be discussed at next meeting about community.osm.org / Discourse (probably next week).
2022-03-24 Grant to suggest the discussion on Discourse to move from the OSM US Slack channel, probably to Discourse. [Topic: Discourse]
- 2022-03-24 OPS to document user / personal account deletion on Discourse. (Admin console has option to anonymise information related to the deleted users (such as email and IP addresses)) [Topic: Discourse]
- 2022-03-24 Grant/Tom to try remove broken footer link on OSQA. Currently links to link-farm/spam site. [Topic: OSQA]
2022-02-24 Grant to update AWS billing notifications can activate automatised alerts on expected billing. [Topic: AWS]
- 2022-02-24 Paul to start conversations for 10G HE.net connection at Amsterdam. [Topic: Amsterdam]
- 2022-02-24 Grant to get quotes on the AMS switches. [Topic: Amsterdam]
- 2022-01-13 Paul to look at page thanking people for hosting cache-nodes. [Reportage]
- 2022-01-13 Paul to contact Element.io and try to find out about the potential growth. [Topic: Request to use Standard Tile Layer in Element.io]
2021-10-20 Grant to chat to AWS again [Topic: Planet on S3] # 2022-04-09 Merged with "Grant to speak to AWS person about going ahead with open data program with official OSM S3 bucket"
- 2020-09-09 Grant [Topic: AWS] Speak to AWS person about going ahead with open data program with official OSM S3 bucket. # 2021-05-19 & 2021-06-02 pending
- 2020-09-09 Grant [Topic: AWS] Talk to OpenAerial Map/HOT. # 2021-05-19 & 2021-06-02 pending # 2021-06-16 Paul talked to HOT about OpenAerial Map. We can give them geotiff and can produce tiles from it
- 2020-12-02 Grant to develop some thoughts on what is next for us using AWS. [Topic: AWS] # 2021-05-19 & 2021-06-02 & 2021-06-16 postponed for a few weeks.
- 2020-07-29 Grant to enable background sync to AWS S3. [Topic: Ironbelly] #2020-08-12&26 & 2021-06-02 Manually run, automated scripting to be added. # 2021-05-19 Grant to run the script again. # 2022-04-09 Still manually run.
- 2021-09-08 Grant and Guillaume to capture the OSMF emails of suspended users and transfer them to an archive account before deleting them and start paying [Topic: OSMF email provider] # 2021-09-22 Pending. Guillaume may have cleared some. # 2022-03-10 They charge us per archive of individuals (not for groups). Guillaume has emailed some users. Paul can help.
- 2021-09-08 Paul to do the PR to add Fastly to the "supported-by" list. [Topic: Fastly] # 2021-09-22 Added in one place. Probably still needs to be added to hardware.openstreetmap.org.
- 2021-05-05 Grant to email Toby from WMF and suggest chating to MapTiler. [Topic: Wikimedia] # 2021-06-02, 2021-06-16, 2021-06-30,2021-07-14, 2021-08-11, 2021-08-25, 2022-04-07 Status: Pending. Low priority.
- 2021-02-24 Tom to report back on TimescaleDB again at next meeting. [Topic: Reportage] [was: 2021-01-13 Tom to evaluate TimescaleDB] [Topic: Longer term metricretention] #2021-04-21 SSD Disk Failing in US # 2021-05-19 decision to leave on the agenda. # 2021-06-02, 2021-06-16, 2021-06-30 nothing new #2021-07-14 deployed yesterday # 2021-08-11 Failing over again after a week. Hypothesis: issue with Postgres and number of tables.Some of the autovacuum jobs stuck in loops reading the statistics. # 2021-08-25 No update.
- 2021-01-13 OWG to send message to the servers we want to keep. [Reportage. Existing CDN servers] # 2021-03-24 Three servers stopped talking to us (shenron, naga and one more) # 2021-05-19, 2021-06-02 & 2021-06-16 pending # 2021-07-14 in progress, half-done. 2021-08-11, 2021-08-25 Pending. # 2022-04-07 Once Grant goes to AMS will wipe some servers and derack them.
- 2021-01-13 Grant to wipe thorns and the 3 other machines [AMS] [Topic: Longer term metric retention] # 2021-05-19 pending # 2021-06-02 ramoth data drives wiped - decision: Grant to do final wipe of Ramoth and leave it until next site visit. Discussion about 16G DDR3. #2021-07-14 Paul to update ticket with the decision.
- 2020-11-04 OWG to work out tile log archival and deletion policy at later stage. [Topic: Commercial CDN] # 2021-03-24 & 2021-05-19 deferred to future point
- 2020-08-12 Michal to try to rekindle excitement about people helping with imagery (on dev channel/imagery channel or Slack). # 2020-08-26 No progress. # 2022-04-09
- 2020-07-29 Grant to check with Wiki Admins on hCaptcha (reCaptcha replacement). [Topic: Wiki reCaptcha issue] https://github.com/openstreetmap/operations/issues/454 #2020-08-12 hCaptcha people reached out and happy to help. Blocker on Mediawiki 1.35 being released in August. # 2021-05-19 blocker removed. # 2021-06-02 and 2022-04-09 pending
- 2020-07-01 Paul to create a ticket about solutions to reduce incoming comms. [Topic:Revision of acceptable use policy to reduce incoming comms] # 2021-05-19 decision to leave the action item open. # 2021-06-02 discussion about priority for account deletion. # 2022-04-09 Grant can show Paul how to do that with autoresponder which Tom built. Might be better to work on an online form (action item below).
- 2020-07-01 Grant to work out some of the questions for an online form as a solution to reduce incoming comms. [Topic: Revision of acceptable use policy to reduce incoming comms] 2020-08-12 need to think about the reply # 2021-05-19 decision to leave the action item open. # 2022-04-09 Grant is thinking about examples. Suggestion to add what is considered large for tile usage.
2020-06-04 Paul to update the Github ticket "Adding API key support for tile.osm.org" https://github.com/openstreetmap/operations/issues/342
- 2020-04-10 Grant to work out a table of different data bits, work out how they are backed up and what can be potentially improved. [Topic: High Availability / Redundancy of OpenStreetMap.org (and primary services)] # 2021-05-19 decision to leave the action item open. # 2021-06-02 pending
Meeting adjourned 46' after start.
Thursday 21 April 2022, 19:00 London time, unless rescheduled.
Operations meetings are currently being held every two Thursdays, at 19:00 London time.
Online calendar showing the OPS meetings.