Operations/Minutes/2021-04-21

OpenStreetMap Foundation, Operations Meeting* - Agenda & draft minutes
Wednesday 21 April 2021 18:00 London time
Location: Video room at https://osmvideo.cloud68.co

* Please note that this was not strictly an OWG meeting.

Participants

Present:

Minutes by Dorothea Kazazi, including notes by participants.

Apologies:

Administrative

Previous minutes

2021-04-07

Action items from this meeting

Paul to get list from Equinix. [Topic: Dublin]
Grant to buy the most current model (x1) from more reputable seller. Will use different location/tray. [Topic: SSD Disk Failing in US]
Paul to work out where we need the new HP DL360 servers. [Topic: New HP DL360 servers]
Grant to tweet asking for recommendation of HP/Supermicro resellers in Ireland.
Paul to check how raid1 with hot spare works out with the budget. [Topic: New Rendering server]
Paul to 1) follow up with them and ask about their connection. Ask if the machine will be stable there for 2 year period. Make clear that the machine will belong to us and depending on the circumstances might have to be shipped back. 2) Create a ticket. [Topic: Swedish computer club]

Reportage

Fastly

Communications:
They have a draft that they will run by us and will turn to a blogpost. We are now in contact with Hannah Aubry regarding communications

Listing as top 3 donors:
Michal suggested waiting, as he wants to have a phonecall with them first.

Timescale DB

Upgraded.
Crash in US. - Disk failure. See topic: "SSD Disk Failing in US"

CDN servers

Suggestions:

Make them imagery cache servers.
Accommodate unpredictable community demands.
- In the past when such requests came up, they weren't suited to those machines.
- All machines are different, in different locations around the world. Most new services don't need to be located around the world when starting out. -> administrative cost/ haven't found a good way of optimising routing.
- Suggestion: get a pizza box.

List added on spreadsheet.

Wikimedia

Related to past action item: 2020-09-23 Toby Negrin (Wikimedia) to ask Wikimedia whether they would be interested in OSMF running a tile service available to Wikipedia and if they would be willing to share hardware resources or expertise. [Topic: Wikimedia challenges with Tile CDN delivery]

Decision: to be added as a topic on next meeting's agenda.

Discourse

Background

Related to https://github.com/openstreetmap/operations/issues/377 - Merge forums, OSQA, MLs to discourse? #ML=mailing lists, OSQA is the software running on help.openstreetmap.org.

Past discussions:

2021 Apr, Mar 24, Mar 10
2020 Aug

(carried over from previous meeting)

Maintenance: use of wrapper scripts that manage Docker under the hood. Doesn't play nicely with our Chef setup (e.g. regarding certificate management).

Decisions:

Wait until Grant has more time.
We will return thorn-01 to online state, add external IP. Grant will work on getting chef to manage a docker based install.

New data center: Dublin

Signed with Equinix. There remains some paperwork, but seeing if it's really necessary. Phone call with account manager coming up, pending scheduling.

Planned start date was beginning of June, are we still okay with this?
> July preferable.

Do we want a cage for storage of shipped hardware before Grant visits? It would let us start doing purchasing now.

Shipping
Equinix does not act as importer. Does this rule out shipping from UK? br> > Yes. Suggestion: use Republic of Ireland.

UK: Found companies with the parts we're looking for.
EU: limited selection of SSDs/more expensive.

Can we have shipping company bill us for duties?

ISP
https://github.com/openstreetmap/operations/issues/520
Will get list from Equinix. We need to decide what we want
- Do we need dual ports for redundancy? Yes
- 10 Gbps ports > Yes
- Current Amsterdam traffic peaks at 924 Mbps outbound (mainly planet) and 370 Mbps average (3/4 of it planet) 2Gbps commit, preferred hard cutoff at 2Gbps, not burstable.

https://www.peeringdb.com/fac/2161
https://www.peeringdb.com/fac/1065

Action item: Paul to get list from Equinix.

New HP DL360 servers

Used HPE DL360 Gen9 with v4 processor is still probably sweet spot. ~2000 EUR more to go up to Gen10, and better Gen9 availability.

Proposed specs:

2xE5-2680v4 (14 Core @ 2.4 GHz)
128 GB DDR4-2400 (4x32)
HP P440ar
2x HPE 480GB SATA

Possible upgrades:

2xE5-2690v4 (14 Core @ 2.6 GHz), 120 EUR
192 GB DDR4, 278 EUR
2x HPE 1.6TB SATA, 540 EUR, might be availability issues

Go for all upgrades, and 3x 1.6TB

Action items

Paul to work out where we need the new servers.
Grant to tweet asking for recommendation of HP/Supermicro resellers in Ireland.

New rendering server

https://github.com/openstreetmap/operations/issues/521

We've budgeted for a new rendering server for somewhere outside of Dublin. This could be Amsterdam, but doesn't have to be. Need to decide on location.

See Github ticket for locations.

Amsterdam

easiest but then colocated with Odin.
physical space is mostly used.

Exonetric

Not concerned about use of bandwidth.

Zagreb

Behind NAT.

Is 3.84TB enough to last 5 years? If not, is 4TB?

Go with 3x4TB for storage, 2x SATA DOM for boot

Other points mentioned:

Maxed out on rendering capacity in Europe for most of the day.
Perhaps Yandex can supply machine with more SSD.
UCL not really an option.
Europe: Could swap cache server with rendering machine.
Could also put in Dublin, for the moment.

Suggestion: raid1 with hot spare.

Action item: Paul to check how raid1 with hot spare works out with the budget.

SSD disk failing in US

stormfly-03 has a failed disk. We've had 2 of this model SSD fail (very rare). Do we replace with same or more expensive disk? $400 vs $700 (SAS required).

Bought 6 initially. Patched. 2 failed. No SSD failure before.
Samsung with HP firmware.

Suggestion: buy 3.

Action item: Grant to buy the most current model (x1) from more reputable seller. Will use different location/tray.

Any other business

Swedish computer club

Options:

Take-up their offer.
We supply the machine.
- Check bandwidth available.
- Machine stable for 2 years.

Action item: Paul to 1) follow up with them and ask about their connection. Ask if the machine will be stable there for 2 year period. Make clear that the machine will belong to us and depending on the circumstances might have to be shipped back. 2) Create a ticket.

Eurostar vouchers