Operations/Minutes/2024-03-07

From OpenStreetMap Foundation

OpenStreetMap Foundation, Operations Meeting - Draft minutes

These minutes do not go through a formal acceptance process.
This is not strictly an Operations Working Group (OWG) meeting.

Thursday 7 March, 19:00 London time
Location: Video room at https://osmvideo.cloud68.co

Participants

Minutes by Dorothea Kazazi

New action items from this meeting

  • Grant Slater to open a ticket about running our own IMAP server, to describe the problem and sample cases. [Topic: Running our own IMAP server]
  • Grant Slater and Guillaume Rischard to discuss further about the redundancy of gateway / IPv6 "private subnet". Benefit / "Cost" [Topic: Redundancy of Gateway / IPv6 "private subnet"]

Equinix AM6 Price Increase

Equinix is requiring a 24% price increase on space, 24% on power, and 5% on cross-connects. The price increase last .

  • Space cost increase: The increase in space cost is likely justifiable, as it has remained unchanged since we moved in.
  • Cross-connects cost increase: appears reasonable.
  • Power cost increase: We were informed of a price increase in October, effective from January, with assurances of no further increases.
    • A ticket regarding this matter has been opened with the salesperson. Resolution may require escalation.
    • The statement of "No price increase" was communicated via a notification email from a no-reply address in November 2023.

Suggestions

  • Pursue clarification on the validity of the "No price increase" statement.
  • Ask for a discount, citing our acknowledgment of Equinix on the thanks page.
  • Verify if the "No price increase" commitment extends to future increases in Dublin.
    • This query will be directed to a different salesperson.
    • Consensus seemed to be to defer this inquiry for now. If a price increase email is received, we'll follow up then.
  • Seek an explanation for the rise in electricity costs, mentioning our budgetary commitments and donor scrutiny.

On formally asking for a discount as we have Equinix on our thanks page

  • We inquired about potential discounts during our initial signup in Amsterdam. There is no clarity about any discounts at the time, although we did secure a favorable price for one rack.
  • Requesting a discount after several years, may seem weird.
  • Should Equinix decline to agree, remove their mention from the thanks page.
  • Grant offered to email Equinix.
  • Will save ~ EUR 8,000 with greater savings anticipated next year, given its mid-year commencement.

Attendance at SotM Europe 2024

  • Paul will attend.
  • Guillaume, Tom and Grant are unsure.

Suggestion: If all OPS go, they could book an extra night for a meeting.

Insurance update

  • They need some locations.
  • They probably have not provided an actual dollar value.

Running our own IMAP server

We forward a lot of spam that we get from the DMCA form or OTRS and sometimes, it doesn't get delivered.

  • The DMCA form submits emails to DWG.
  • Emails to OTRS generate email notifications: the email goes to the forwarder email address before reaching other recipients.

Those emails are frequently spam messages, which we automatically send to others. The spam that appears to be generated from our site. The destination ISPs think it's spam and bounce it.

Suggestions

  • Operating our own IMAP server (with no SMTP functionality, exclusively for internal use without outgoing email capabilities) could resolve the issue. It would act as an incoming SMTP, preventing forwarded emails from being bounced by other servers.
    • An IMAP server without an SMTP service to deliver email into it is irrelevant, except in rare scenarios.
    • There are better alternative solutions than running an IMAP server.
  • Open a ticket.
  • Stop people using their private email addresses and instead have them get an OSMF email address.
    • This will have some admin burden.
  • The DMCA form should post straight into the OTRS API.
    • This approach would move the problem to OTRS, as OTRS will send notifications by email that will get bounced.

Other points mentioned during discussion

  • Forwarding emails through mailbox.org to other addresses is highly unreliable. Mailbox.org does not do the things that Google does.
  • An SPF issue may arise.
    • It's probably fine because it is coming from an authorized OSMF outgoing server.
  • We prefer not to manage email infrastructure unless it becomes an absolute necessity.

Action item

Grant to open a ticket, to describe the problem and sample cases.

Redundancy of Gateway / IPv6 "private subnet"

Concern: running tasks such as replication on old machines, like Ironbelly (Site gateway) and Ridley (Site gateway, Foundation related sites).

Suggestion

Grant and Guillaume to discuss the issue, documenting objectives, constraints, and the workload for implementing changes.

Other points mentioned during discussion

  • We should avoid relying on a single host, as it is a point of failure.
    • Over the past twelve years or more, failures have occurred infrequently and were typically resolved within five minutes. Considering the potential workload for changes, maintaining the current setup may be justified.

Action item

Grant and Guillaume to discuss further. Benefit / "Cost"

Vector tiles

  • Paul put a demo up of his vector tiles work.
  • Currently runs single-threaded on a Hertzner machine (6 core, consumer drive, several year old machine) and performs quite wall.
  • It's less demanding than an image rendering server on disk space and on I/O.

Future

  • Parallel processing still needed for the initial pre-rendering of millions of tiles.
  • Paul wants to dockerise it properly, to simplify deployment of development environments.
    • Grant offered to help.
    • Implementation is expected no sooner than one month from now.

Suggestions

  • Move it to a shared machine, because it doesn't warrant a full machine.
  • Remove the libraries it relies on to ensure complete separation.

Any other business

Nominatim

  • Nominatim is currently on a Debian machine.
  • Sarah Hoffmann asked to keep the old machine for some time, for experimentation purposes.

help.osm.org

help.osm.org is now read-only. The login button has been removed, and the OAuth page returns a 403 error.

Options

  • Convert it into a container.
  • Create a static archive.

Some archive.org people are scraping it, currently running 3 similtaneous gets.

On suggestion to make it read-only, removing user permissions on the tables

  • This process is relatively complex with PostgreSQL.
  • Django might encounter errors, necessitating that we retain the session tables.

Grant truncated the session table to force all users to log out. We can change the session database to use Redis and Memcached, but the benefits may not outweigh the effort.

Other points mentioned during discussion

  • We want to avoid running unmaintained code.

OPS to look for the minimal solution with minimal ongoing administrative overhead.

UCL visit by Grant

Upcoming visit by Grant, discussion of challenges regarding bringing in and removing of hardware. Large number of machines to be decommissioned which may need removing.

Tasks

  • Replace memory.
  • Derack all the machines. . Currently, we lack storage or disposal options for them.

Options:

  • E-waste.
  • Post to inquire if anyone is interested in taking them.

E-waste issues

  • Drop-off is nearly impossible without certification documents and evidence that we're not disposing of hazardous materials like nuclear waste. It involves significant physical effort and paperwork.
  • Collection is usually from commercial businesses.

Suggestions

  • Email UCL to ask them about the possibility of using their storage center for drop-off.
  • Email the data centre directly.
    • They will probably request the opening of a ticket, but we lack access to their ticketing system.
  • Email the small team that assisted with our move from the old site.

Other points mentioned during discussion

  • UCL E-waste bin is in locked separate section.
  • There are public waste bins.
    • Equinix used these.
  • UCL are getting the data center via JYSK, who subcontract the operation to a commercial company.
  • It's is unclear if our initial contacts are there anymore.

Plan

Do maintenance (RAM swaps), take inventory of hardware that needs to be moved, ensure access works. Then put a notice offering the hardware and do a follow-up visit with better timing.

Ridley

Many services still need to be moved. Containers are primary choice.

  • There's a ticket for that.
  • Some OSMF stuff has already moved.

Action items reviewed at the beginning of the meeting

  • 2024-02-22 Paul to email Equinix about the price increase [Topic: Equinix AM6 Price Increase]
  • 2024-02-22 Paul to open a ticket for Fastly root access and ask Grant there whether Ian has access. [Topic: Fastly root access]
  • 2024-02-08 Paul to ping Sarah on the ticket regarding the Overpass move. [New general purpose machines for DB4]
  • 2024-02-08 OWG to review the Editor policy during one of the next calls and possibly vote on it. [Editor Policy adding to OpenStreetMap.org]
  • 2024-01-25 Grant to publish the schedule for OAuth 1 deprecation on mailing lists/Discourse/Mastodon/Twitter [Topic: OAuth 1.0a]
  • 2024-01-25 Tom to investigate what happens to existing OAuth 1 tokens when OAuth 1 gets turned off, and whether error messages can be sent back in the html of the response to basic/OAuth 1 users (yes, PR in progress). [Topic: OAuth 1.0a]
  • 2023-11-30 Grant to revisit the "policy for purchasing" document, which currently is focused on specs, and add information such as the process for obtaining approval for purchases. [Reportage] Added info: Who Approves / Steps etc -> Grant to create GitHub ticket
  • 2023-11-30 OPS to review the issue of spam reports to ISPs in 6 months (May 2024) -> Grant to create GitHub ticket
  • 2023-05-18 Paul to start an open document listing goals for longer-term planning. [Topic: Longer-term planning]

Action items that have been stricken-through are either completed, or have been moved to GitHub tickets.

Retrieved from "https://osmfoundation.org/w/index.php?title=Operations/Minutes/2024-03-07&oldid=12257"