View source for OSMF updates/2020-08/Operations Overview

<font color="blue"> '''Not for wide dissemination yet - in progress, will be modified.'''</font> 

[[OSMF_updates/2020-08]]

Contains news from [https://operations.osmfoundation.org/reports/ Operations Working Group reports] and [[Working_Group_Minutes#Operations_meetings|Operations meetings]].

== 2020 budget request ==
The OWG’s 2020 budget request was approved by the OSMF board. Planned new work includes:

* refreshing API DB servers;
* refreshing HP DL360 G6 machines at	UCL and OSUOSL;
* upgrading Nominatim capacity; and
* using a consultant to make chef cookbooks more testable and	make it easier fr others to contribute.

== Policies ==
* The OWG has approved a new [https://operations.osmfoundation.org/policies/banner/ banner policy] for event banners on OpenStreetMap.org.
* The [https://operations.osmfoundation.org/policies/tiles/ tile usage policy] has been [https://github.com/openstreetmap/owg-website/pull/60/files amended] to reflect current practice and the increased load on the tile infrastructure.

== Services ==
=== irc.openstreetmap.org ===
The CGI:IRC page has been upgraded to redirect to the official OFTC webchat IRC. You can continue to use irc.openstreetmap.org, use https://webchat.oftc.net/, or use your choice of [https://wiki.openstreetmap.org/wiki/IRC IRC] client.

=== nominatim.openstreetmap.org moved to nginx for web serving ===
nginx allows us to handle additional traffic and simplifies the quality of service management to fairly serve users following our [https://operations.osmfoundation.org/policies/nominatim/ usage policy]. 

=== planet.openstreetmap.org ===
Downloads from planet.openstreetmap.org will now use the a mirror when downloading PBF files. Bzip2 files already used the mirror.

We will soon be adding BitTorrent as another way to download the weekly Planet.osm exports. Please test the EXPERIMENTAL torrent files here: https://osm.cquest.org/torrents/ 

=== svn.openstreetmap.org ===
Is now closed to new accounts and will be made read-only before the end of August 2020. Many relevant http://trac.openstreetmap.org tickets have already moved to https://github.com/openstreetmap/openstreetmap-website/issues We're available to help developers move their project.

=== trac.openstreetmap.org turned off ===
The time has come for us to retire our trac bug tracker, as it is becoming a burden for us to support and most projects have moved onto better alternatives. 
* [https://lists.openstreetmap.org/pipermail/dev/2020-July/030958.html Details]
* https://github.com/openstreetmap/operations/issues/416

=== tile.openstreetmap.org ===
A number of default user-agents and heavy use apps have had their access to tile.openstreetmap.org restricted. If you are developing software using the OSMF tile servers, be sure to follow the tile usage policy.

Do you run a website or app which uses http://tile.openstreetmap.org? Make sure to switch to the HTTPS tile url. The http:// url has been deprecated. 

Tile servers are upgraded to OpenStreetMap Carto 5.2.0.

== Tile caches ==

=== New caches ===
Many new tile caches were setup
{| class="wikitable"
!Cache
!Location
!Host
|-
|stormfly-01
|Corvallis,				Oregon
|OSUOSL,				hardware provided by OSMF
|-
|gorwen
|Bogotá,				Colombia
|edgeuno
|-
|neak
|France
|faimaison
|-
|meraxes
|Paris,				Fance
|Scaleway
|-
|naga
|Paris,				France
|MilkyWan
|-
|fuchur
|São				Paulo, Brasil
|edgeuno
|-
|idris
|Buenos				Aires, Argentina
|edgeuno
|-
|glaedr
|Bogotá,				Colombia
|edgeuno
|-
|albi
|Paris, France
|Scaleway
|-
|Gackelchen
|Bissen, Luxembourg
|Gandi
|-
|Kokosnuss
|Germany
|Strato,  thanks to Max Michels.
|-
|Falkor
|Prague, Czech Republic 
|Vodafone
|-
|Shruikan
|Dronten, Netherlands.
|greenminihost
|}

Also https://blog.openstreetmap.org/2020/04/ but has 2019 data and incomplete.

=== Would you like to host a tile cache? ===
If you operate an internet exchange, host company, or otherwise have a site with good internet connectivity and high regional bandwidth, you can look at the tile CDN node requirements. We welcome hosting of tile caches elsewhere, and are particularly looking for tile caches in Africa and Asia. If you are interested, please [Mailto:operations@osmfoundation.org contact us].

== www.openstreetmap.org ==
=== Call for featured layers on www.openstreetmap.org ===
Submissions for featured layers are welcome and will be evaluated when they come in. 
* Criteria for possible inclusion: https://operations.osmfoundation.org/policies/new-tile-layers/
* Call: https://blog.openstreetmap.org/2020/03/24/call-for-more-featured-tile-layers-on-osm-org/

=== New featured layer for public transport ===
We have approved [http://xn--pnvkarte-m4a.de/ ÖPNVKarte] as a new featured layer on OpenStreetMap.org. [https://blog.openstreetmap.org/2020/07/10/opnvkarte-a-new-featured-layer-on-www-openstreetmap-org/ blog post].

=== API ===
[https://wiki.openstreetmap.org/wiki/OSM_JSON OSM JSON] format is now available in production. Supported endpoints are listed on the [https://wiki.openstreetmap.org/wiki/API_v0.6#JSON_Format OSM API 0.6] wiki page.

Four additional API 0.6 endpoints are now being directed to CGImap.

The new database servers, [https://hardware.openstreetmap.org/servers/snap-01.openstreetmap.org/ snap-01] and [https://hardware.openstreetmap.org/servers/snap-02.openstreetmap.org/ snap-02] have been racked and connected.

== Other ==

=== Network congestion control ===
BBR network congestion control was tried, then [https://github.com/openstreetmap/operations/issues/359 rolled back after problems].

=== Upgrade of cgimap ===
cgimap was upgraded from 0.7.5 to version 0.8.1. JSON output is enabled by default now.

See [https://github.com/zerebubuth/openstreetmap-cgimap/releases release notes] for a list of all new features/bug fixes.

=== Hardware ===
New API database servers have arrived in Amsterdam and the UK. The UK server is awaiting a trip to UCL which needs to be scheduled with COVID-19 restrictions.

Two new servers to replace stormfly-01 and stormfly-02 have arrived in OSUOSL but are awaiting a trip to the data center which needs to be scheduled with COVID-19 restrictions.

4 1U servers were removed from UCL for disposal.

Several failed or failing disks were replaced in machines.

Two servers to replace stormfly-01 and stormfly-02 have been installed at OSUOSL. One is planned to be a tile cache.

=== Git ===
The [https://github.com/openstreetmap/operations/issues/381 network issues between UCL and Github] were diagnosed as a faulty UCL firewall and fixed.

=== DNS ===
Our domain registrar, Gandi, is now supporting OpenStreetMap with free registration.

We have switched to using https://stackexchange.github.io/dnscontrol/ to managing all OpenStreetMap domains DNS. https://github.com/openstreetmap/dns/blob/master/dnsconfig.js We also have #DNSSEC enabled on open-maps.org, other domains to follow. 

=== osm.org outbound mails === 
All OpenStreetMap.org outbound mails are now automatically DKIM signed to confirm sending server authenticity. Additionally SPF IP listed.

=== Mailing lists ===
The OpenStreetMap mailing lists now have DMARC "Munge From" enabled by default. https://wiki.list.org/DEV/DMARC

=== General ===
Some services have been upgraded to Ubuntu 20.04.

=== Membership ===
Radoje Stojisic and Hrvoje Bogner joined the OWG.

If you are interested in joining, please read our [https://operations.osmfoundation.org/policies/owg-membership/ membership policies] and [Mailto:operations@osmfoundation.org contact us].

== Discussions during operations meetings ==
In April online Operations meetings with guests were started. Minutes or notes are added at the [[Working_Group_Minutes#Operations_Working_Group| Working Group minutes]] page.
Discussion topics are presented below.

=== Also mentioned above ===
==== Turn off Trac ====
''https://github.com/openstreetmap/operations/issues/416''

See above. 

''Notes from meeting: [[Operations/Minutes/2020-06-04#Deprecate_Turn_off_Trac| Jun]]''

==== SVN ====
See above. 

''Notes from meeting: [[Operations/Minutes/2020-05-22#SVN| May]]''

==== Featured layers call ====
See above. 

''Notes from meeting: [[Operations/Minutes/2020-05-04#Featured_layers_call| May]]''

==== [Closed ticket] Fixing the failing CI tests  ====

The [https://github.com/openstreetmap/operations/issues/381 network issues between UCL and Github] were diagnosed as a faulty UCL firewall and fixed.

''Minutes/notes from meetings: [[Operations/Minutes/2020-04-10#Fixing_the_failing_CI_tests| Apr]], [[Operations/Minutes/2020-05-04#On_fixing_the_failing_CI_tests| May 04]]''

=== [Ticket] Migrate help.openstreetmap.org from OSQA === 
''Github ticket: [https://github.com/openstreetmap/operations/issues/149 Migrate help.openstreetmap.org from OSQA #149]''

help.openstreetmap.org is currently using [https://en.wikipedia.org/wiki/OSQA OSQA] (Open Source Question and Answer), which:
* has been inactive upstream for long time.
* is on Python/Python2, which goes away.
* is on a machine that (in 2019 Jan was) running 14.04, which reached End of Life in April 2019.

Options for migration suggested so far include, but are not limited to:
* ASKBOT (OSQA fork) 
* [https://www.question2answer.org Question2answer]
* [https://stackexchange.com/ Stack exchange]/[https://stackoverflow.com/ overflow]
* Discourse

<font color="blue"> Current status on archiving</font>

''Minutes/notes from meetings: [[Operations/Minutes/2020-04-10#Migrate_help.openstreetmap.org_from_OSQA| Apr]], [[Operations/Minutes/2020-05-22#Migrate_help.openstreetmap.org_from_OSQA| May]], [[Operations/Minutes/2020-06-04#Michal.27s_report_into_archiving_help.openstreetmap.org_.28sent_via_email_prior_to_the_meeting.29| Jun]] (archiving help.osm.org), [[Operations/Minutes/2020-07-01#Archiving_help.openstreetmap.org| Jul]] (archiving help.osm.org)''

=== [Ticket] OSM Forum (FluxBB) update ===
''Related to ticket: [https://github.com/openstreetmap/operations/issues/439 Github: OSM Forum (FluxBB) update #439]''

''Notes from meeting: [[Operations/Minutes/2020-07-15#FluxBB_is_old_and_slow| Jul]] (FluxBB is old and slow)''

=== [Ticket] Adding API key support for tile.osm.org? === 
''Github ticket [https://github.com/openstreetmap/operations/issues/342 : Adding API key support for tile.osm.org? #342]''

''Notes from meeting: [[Operations/Minutes/2020-06-04#Adding_API_key_support_for_tile.osm.org.3F| Jun]]''

=== [Ticket] Monitoring and alerting system === 
''Related to [https://github.com/openstreetmap/operations/issues/360 Github: Switch to Prometheus for monitoring and alerting #360]''

We are hitting the limits of Munin. Prometheus was suggested as an alternative but is a lot of work to get the same monitoring and alerting coverage we have right now.

''Minutes/notes from meetings: [[Operations/Minutes/2020-05-04#Monitoring_and_alerting_system| May]]''

=== [Ticket] Wiki reCAPTCHA issue === 

[https://github.com/openstreetmap/operations/issues/449 Github issue: Wiki asks for confirmation code before saving changes without sending one #449] 

[https://www.hcaptcha.com/ hCaptcha] has been [https://github.com/openstreetmap/chef/issues/298 suggested] as a reCAPTCHA replacement and support for it is coming to mediawiki. Previous blocker: Mediawiki 1.35 being released in August.

Please note that we don't have the resources to develop a custom replacement for reCaptcha. If someone else develops a custom replacement with an OpenStreetMap focus, we would be interested.

''Notes from meeting: [[Operations/Minutes/2020-07-29#Wiki_reCAPTCHA_issue| Jul]]''

=== US servers packet loss ===

''Notes from meeting: [[Operations/Minutes/2020-07-01#US_servers_packet_loss| Jul]]''

=== Machines running old software === 

=== planet-dump caused outage ===

''Notes from meeting: [[Operations/Minutes/2020-07-15#planet-dump_caused_outage| Jul]]''

=== High Availability / Redundancy of OpenStreetMap.org (and primary services) === 
We have two main data centers for osm.org: one in Netherlands and a warm/coldish copy in the UK.

''Minutes/notes from meetings: [[Operations/Minutes/2020-04-10#High_Availability_.2F_Redundancy_of_OpenStreetMap.org_.28and_primary_services.29|  Apr]]''

=== New Data Centre Space? === 
Even if we move stuff to virtual providers and hosting companies, we think we still want to maintain a physical presence somewhere as a second site. Current situation in Slough works, but not for next 5 years.

''Minutes/notes from meetings: [[Operations/Minutes/2020-04-10#New_Data_Centre_Space.3F| Apr]], [[Operations/Minutes/2020-07-01#Long_term_plans_for_data_center_space| Jul]] (Long term plans)''

=== AWS Config Using Code (Terraform?) ===

''Notes from meeting: [[Operations/Minutes/2020-07-15#AWS_Config_Using_Code_.28Terraform.3F.29| Jul]]''

=== Tile CDN ===
==== Tech ====
''Squid / nginx / GeoDNS? Is the software right? Issues and improvement.''<br>
''What experiments should we be doing to improve the network.''

''Notes from meeting: [[Operations/Minutes/2020-05-22#Tile_CDN| May]]''

==== Tile CDN usage policy (OSM, Friends, Others?) ====
''https://operations.osmfoundation.org/policies/tiles/'' <br>
''Can we continue to be be open to the world, what policies (not yet actions) can we put in place now to encourage people to move to other providers?''
''eg: Do we continue to allow Commercial Sites to use tile.openstreetmap.org ?''

''Minutes from meeting: [[Operations/Minutes/2020-04-10#Tile_CDN_usage_policy_.28OSM.2C_Friends.2C_Others.3F.29| Apr]]''

==== Commercial CDN for Bulk Tile Users ====
We are considering of using a commercial CDN for bulk tile users. Fastly has expressed interest in supporting OSM via an [https://www.fastly.com/open-source Open source grant] and they are interested in "co-branding" (blogpost). Please see the notes linked below.

''Minutes/notes from meetings: [[Operations/Minutes/2020-04-10#Commercial_CDN_for_Bulk_Tile_Users| Apr]], [[Operations/Minutes/2020-05-04#Commercial_CDN_for_Bulk_Tile_Users| May]], [[Operations/Minutes/2020-07-01#Commercial_CDN_for_Bulk_Tile_Users| Jul 01]], [[Operations/Minutes/2020-07-15#Commercial_CDN_for_Bulk_Tile_Users| Jul 15]], [[Operations/Minutes/2020-07-29#Commercial_CDN Jul| Jul 29]]''

=== Hardware ===

* Equinix servers
* UK replacements
* Status on the discs to Oregon, to be added to remote management network

==== Ironbelly ====
OSMF Servers Info: [https://hardware.openstreetmap.org/servers/ironbelly.openstreetmap.org/ ironbelly.openstreetmap.org]

''Notes from meeting: [[Operations/Minutes/2020-07-29#Ironbelly| Jul]]''

==== Imagery Server Future? ====

OSMF Servers Info: 
* [https://hardware.openstreetmap.org/servers/errol.openstreetmap.org/ errol.openstreetmap.org]  - Development and “tool” server ([https://munin.openstreetmap.org/openstreetmap.org/errol.openstreetmap.org/index.html munin])
* [https://hardware.openstreetmap.org/servers/kessie.openstreetmap.org/ kessie.openstreetmap.org] - Aerial imagery server ([https://munin.openstreetmap.org/openstreetmap.org/kessie.openstreetmap.org/index.html munin])
* [https://hardware.openstreetmap.org/servers/draco.openstreetmap.org/ draco.openstreetmap.org] - ([https://munin.openstreetmap.org/openstreetmap.org/draco.openstreetmap.org/index.html munin])

We'll upgrade Errol.

''Notes from meetings: [[Operations/Minutes/2020-07-15#Replacement_of_Errol.2FKessie| Jul 15]] (Replacement of Errol/Kessie), [[Operations/Minutes/2020-07-29#Imagery_Server_Future.3F| Jul 29]]''

=== Revision of acceptable use policy to reduce incoming comms ===

''Notes from meeting: [[Operations/Minutes/2020-07-01#Revision_of_acceptable_use_policy_to_reduce_incoming_comms| Jul]]''