Operations/Minutes/2022-04-07

From OpenStreetMap Foundation

OpenStreetMap Foundation, Operations Meeting* - Draft minutes
These minutes do not go through a formal acceptance process.

Thursday 7 April 2022 at 19:00 London time - Countdown
Location: Video room at https://osmvideo.cloud68.co

* Please note that this is not strictly an Operations Working Group (OWG) meeting.

Participants

Minutes by Dorothea Kazazi.

New action items

Reportage

AWS

  • Grant talked to them again.
  • Billing notifications: Updated to automatic 6-month rolling. Set at 15% and 75% total spent.
  • Billing tags: Turned on and will get cost per tag.
  • Paul wants to do aggregates of logs (e.g. stats on browser agents).

On S3 storage

  • Storage backups (bigger storage percentage)
  • 53.0 TB Fastly logs, which can be stored on Glacier https://aws.amazon.com/s3/storage-classes/glacier/)
  • 50.0 TB Write-ahead log for Postgres (WAL-G, data up to 2017)
    • Get compressed before uploaded.
  • 40.0 TB Backup of planet (costing little)
  • 04.1 TB Fastly processed logs.

Other points mentioned

  • We have ~ 204 TB on S3.
  • Fastly processed logs used the most.

On tiers and cost

  • Paul reluctant to have tier changed automatically, as it has resulted in unexpected costs in the past.
  • Intelligent tiering according to Grant is pretty safe. Can cost more if you have insane number of objects.

On logs

  • Grant prefers to keep them for some time for trend analysis.
  • Is costing us (storage costs more than Athena queries).

Suggestions

  • Aggregate logs to see trends. Easier to run Athena queries on aggregated data over a year.
  • Deletion of IP addresses and some partially personally identifiable data (Privacy/GDPR).

On cost of Athena queries

  • Two types of queries:
    • Automatic queries for tile-logs - we can't fo anything about their cost.
    • Ad-hoc.
  • Grant is not concerned about the cost.

Action item: Grant to talk to Tom about fixing WAL-G.

community.osm.org / Discourse

community.openstreetmap.org is live.

Discussions about community.osm.org / Discourse

Moved to Matrix.

Action item: Grant to suggest the discussions about community.osm.org / Discourse to move from Matrix to IRC.

Migration

  • Will do migration from old forum.
  • Christian Quest is doing the testing and dealing with some of the issues.
  • Some tasks briefly paused as we get ready for migration.

Translation plugin

Options

  • Microsoft
  • Google: good from/to English.
  • Not DeepL, because:
    • doesn't work with Discourse
    • languages: has only 20 languages (mainly western European). It is good for the languages it supports. Does have Chinese/Japanese, doesn't have Indian.
    • Cost 5000 USD for plugin to be developed by Discourse developers.

Other points mentioned

  • Discourse stores cache of translation.
  • Christian Quest has tested the plugins.
  • Approximate cost of translation plugin: 20 USD/month.

Action item: Grant to look in the Discourse translation plugin.

Help.osm.org functionality

Action items

  • 2022-03-24 Grant to suggest the discussion on Discourse to move from the OSM US Slack channel, probably to Discourse. [Topic: Discourse]
  • 2022-03-24 OPS to document user / personal account deletion on Discourse. (Admin console has option to anonymise information related to the deleted users (such as email and IP addresses)) [Topic: Discourse]
  • 2022-03-24 Grant/Tom to try remove broken footer link on OSQA. Currently links to link-farm/spam site. [Topic: OSQA]
  • 2022-02-24 Grant to update AWS billing notifications can activate automatised alerts on expected billing. [Topic: AWS]
  • 2022-02-24 Paul to start conversations for 10G HE.net connection at Amsterdam. [Topic: Amsterdam]
  • 2022-02-24 Grant to get quotes on the AMS switches. [Topic: Amsterdam]
  • 2022-01-13 Paul to look at page thanking people for hosting cache-nodes. [Reportage]
  • 2022-01-13 Paul to contact Element.io and try to find out about the potential growth. [Topic: Request to use Standard Tile Layer in Element.io]
  • 2021-10-20 Grant to chat to AWS again [Topic: Planet on S3] # 2022-04-09 Merged with "Grant to speak to AWS person about going ahead with open data program with official OSM S3 bucket"
  • 2020-09-09 Grant [Topic: AWS] Speak to AWS person about going ahead with open data program with official OSM S3 bucket. # 2021-05-19 & 2021-06-02 pending
  • 2020-09-09 Grant [Topic: AWS] Talk to OpenAerial Map/HOT. # 2021-05-19 & 2021-06-02 pending # 2021-06-16 Paul talked to HOT about OpenAerial Map. We can give them geotiff and can produce tiles from it
  • 2020-12-02 Grant to develop some thoughts on what is next for us using AWS. [Topic: AWS] # 2021-05-19 & 2021-06-02 & 2021-06-16 postponed for a few weeks.
  • 2020-07-29 Grant to enable background sync to AWS S3. [Topic: Ironbelly] #2020-08-12&26 & 2021-06-02 Manually run, automated scripting to be added. # 2021-05-19 Grant to run the script again. # 2022-04-09 Still manually run.
  • 2021-09-08 Grant and Guillaume to capture the OSMF emails of suspended users and transfer them to an archive account before deleting them and start paying [Topic: OSMF email provider] # 2021-09-22 Pending. Guillaume may have cleared some. # 2022-03-10 They charge us per archive of individuals (not for groups). Guillaume has emailed some users. Paul can help.
  • 2021-09-08 Paul to do the PR to add Fastly to the "supported-by" list. [Topic: Fastly] # 2021-09-22 Added in one place. Probably still needs to be added to hardware.openstreetmap.org.
  • 2021-05-05 Grant to email Toby from WMF and suggest chating to MapTiler. [Topic: Wikimedia] # 2021-06-02, 2021-06-16, 2021-06-30,2021-07-14, 2021-08-11, 2021-08-25, 2022-04-07 Status: Pending. Low priority.
  • 2021-02-24 Tom to report back on TimescaleDB again at next meeting. [Topic: Reportage] [was: 2021-01-13 Tom to evaluate TimescaleDB] [Topic: Longer term metricretention] #2021-04-21 SSD Disk Failing in US # 2021-05-19 decision to leave on the agenda. # 2021-06-02, 2021-06-16, 2021-06-30 nothing new #2021-07-14 deployed yesterday # 2021-08-11 Failing over again after a week. Hypothesis: issue with Postgres and number of tables.Some of the autovacuum jobs stuck in loops reading the statistics. # 2021-08-25 No update.
  • 2021-01-13 OWG to send message to the servers we want to keep. [Reportage. Existing CDN servers] # 2021-03-24 Three servers stopped talking to us (shenron, naga and one more) # 2021-05-19, 2021-06-02 & 2021-06-16 pending # 2021-07-14 in progress, half-done. 2021-08-11, 2021-08-25 Pending. # 2022-04-07 Once Grant goes to AMS will wipe some servers and derack them.
  • 2021-01-13 Grant to wipe thorns and the 3 other machines [AMS] [Topic: Longer term metric retention] # 2021-05-19 pending # 2021-06-02 ramoth data drives wiped - decision: Grant to do final wipe of Ramoth and leave it until next site visit. Discussion about 16G DDR3. #2021-07-14 Paul to update ticket with the decision.
  • 2020-11-04 OWG to work out tile log archival and deletion policy at later stage. [Topic: Commercial CDN] # 2021-03-24 & 2021-05-19 deferred to future point
  • 2020-08-12 Michal to try to rekindle excitement about people helping with imagery (on dev channel/imagery channel or Slack). # 2020-08-26 No progress. # 2022-04-09
  • 2020-07-29 Grant to check with Wiki Admins on hCaptcha (reCaptcha replacement). [Topic: Wiki reCaptcha issue] https://github.com/openstreetmap/operations/issues/454 #2020-08-12 hCaptcha people reached out and happy to help. Blocker on Mediawiki 1.35 being released in August. # 2021-05-19 blocker removed. # 2021-06-02 and 2022-04-09 pending
  • 2020-07-01 Paul to create a ticket about solutions to reduce incoming comms. [Topic:Revision of acceptable use policy to reduce incoming comms] # 2021-05-19 decision to leave the action item open. # 2021-06-02 discussion about priority for account deletion. # 2022-04-09 Grant can show Paul how to do that with autoresponder which Tom built. Might be better to work on an online form (action item below).
  • 2020-07-01 Grant to work out some of the questions for an online form as a solution to reduce incoming comms. [Topic: Revision of acceptable use policy to reduce incoming comms] 2020-08-12 need to think about the reply # 2021-05-19 decision to leave the action item open. # 2022-04-09 Grant is thinking about examples. Suggestion to add what is considered large for tile usage.
  • 2020-06-04 Paul to update the Github ticket "Adding API key support for tile.osm.org" https://github.com/openstreetmap/operations/issues/342
  • 2020-04-10 Grant to work out a table of different data bits, work out how they are backed up and what can be potentially improved. [Topic: High Availability / Redundancy of OpenStreetMap.org (and primary services)] # 2021-05-19 decision to leave the action item open. # 2021-06-02 pending

Meeting adjourned 46' after start.

Next meeting

Thursday 21 April 2022, 19:00 London time, unless rescheduled.

Operations meetings are currently being held every two Thursdays, at 19:00 London time.
Online calendar showing the OPS meetings.

Retrieved from "https://osmfoundation.org/w/index.php?title=Operations/Minutes/2022-04-07&oldid=9465"