⚓ T416707 Sunsetting mirrors.wikimedia.org
Page Menu
Phabricator
Create Task
Maniphest
T416707
Sunsetting mirrors.wikimedia.org
Open, Medium
Public
Actions
Edit Task
Edit Related Tasks...
Create Subtask
Edit Parent Tasks
Edit Subtasks
Merge Duplicates In
Close As Duplicate
Edit Related Objects...
Edit Commits
Edit Mocks
Mute Notifications
Protect as security issue
Assigned To
None
Authored By
MoritzMuehlenhoff
Feb 6 2026, 3:28 PM
2026-02-06 15:28:59 (UTC+0)
Tags
SRE
(Backlog)
Infrastructure-Foundations
(Backlog)
Release-Engineering-Team (Radar)
(Limbo)
User-notice
(Already announced/Archive)
Referenced Files
None
Subscribers
A_smart_kitten
Aklapper
Andrew
bd808
bking
dduvall
Fabfur
View All 15 Subscribers
Description
We've been operating a Debian mirror since approx. ten years (it also hosts a mirror of Ubuntu, Tails and Debian Openstack Backports repository). This has historically been a great way to contribute back to our favourite OS. But Debian defaults to the deb.debian.org CDN since many years and that shrank the use of the traditional mirrors significantly.
Despite being part of the ftp.us.debian.org mirror rotation our mirror is hardly used these days, even despite idiotic AI scrapers hitting the server randomly and excessively, our bandwidth usage practically never exceeds 10% of the available bandwidth (and usually even less than that).
The current server is ending it's hardware lifespan, so in the current state it currently does not really make sense to refresh it, so we'll likely decommision it entirely.
Details
Related Changes in Gerrit:
Subject
Repo
Branch
Lines +/-
Switch Cloud VPS to deb.debian.org
operations/puppet
production
+1
-1
Switch our servers to use deb.debian.org
operations/puppet
production
+1
-4
Switch three test systems to deb.debian.org
operations/puppet
production
+3
-0
Switch our servers to use deb.debian.org
operations/puppet
production
+1
-1
Customize query in gerrit
Related Objects
Search...
Task Graph
Mentions
Status
Subtype
Assigned
Task
Open
None
T416707
Sunsetting mirrors.wikimedia.org
Open
None
T423596
Adjust WMCS Gitlab CI/CD repo to stop using mirrors.wikimedia.org
Open
Andrew
T423598
Migrate our use of osbpo away from mirrors.wikimedia.org
Resolved
Jdforrester-WMF
T423622
New base images without mirrors.wikimedia.org
Open
None
T423971
Update default GitLab runner image to a base image without mirrors.wikimedia.org
Open
None
T423972
Rebuild dev-images using a base image without mirrors.wikimedia.org in the apt sources
Mentioned Here
T243056: Set up static-codereview.wikimedia.org to host static HTML dump of CodeReview
Event Timeline
MoritzMuehlenhoff
created this task.
Feb 6 2026, 3:28 PM
2026-02-06 15:28:59 (UTC+0)
Restricted Application
added a subscriber:
Aklapper
View Herald Transcript
Feb 6 2026, 3:29 PM
2026-02-06 15:29:00 (UTC+0)
MoritzMuehlenhoff
triaged this task as
Medium
priority.
Feb 6 2026, 3:29 PM
2026-02-06 15:29:09 (UTC+0)
ayounsi
awarded a token.
Feb 6 2026, 6:17 PM
2026-02-06 18:17:00 (UTC+0)
MoritzMuehlenhoff
mentioned this in
Unknown Object (Task)
Feb 18 2026, 10:10 PM
2026-02-18 22:10:28 (UTC+0)
Fabfur
subscribed.
Mar 16 2026, 4:34 PM
2026-03-16 16:34:15 (UTC+0)
gerritbot
added a comment.
Mar 20 2026, 12:25 PM
2026-03-20 12:25:47 (UTC+0)
Comment Actions
Change #1256371 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Switch our servers to use deb.debian.org
gerritbot
added a project:
Patch-For-Review
Mar 20 2026, 12:25 PM
2026-03-20 12:25:48 (UTC+0)
bking
subscribed.
Mar 23 2026, 9:29 PM
2026-03-23 21:29:34 (UTC+0)
Andrew
subscribed.
Thu, Mar 26, 10:33 PM
2026-03-26 22:33:10 (UTC+0)
Comment Actions
Hi folks! Any idea when this is likely to happen? I will need to coordinate for openstack nodes which use a bespoke openstack repo hosted on the mirror.
LSobanski
subscribed.
Mon, Apr 6, 1:10 PM
2026-04-06 13:10:31 (UTC+0)
Comment Actions
The alert
MirrorHighLag
has started firing 1 month ago. Would it make sense to disable it at this point?
Labels
alertname
MirrorHighLag
cluster
misc
instance
mirror1001:9100
job
node
path
/srv/mirrors/ubuntu
prometheus
ops
severity
warning
site
eqiad
source
prometheus
team
infrastructure-foundations
Annotations
Name
Content
dashboard
description
Mirrors - /srv/mirrors/ubuntu synchronization lag is behind 38d 16h 31m 9s
runbook
summary
Mirrors - /srv/mirrors/ubuntu synchronization lag
Links
Alert dashboard
Alert source
Triage metadata. Do not delete.
fingerprint
60b74e4f03d5302a
gerritbot
added a comment.
Tue, Apr 7, 9:13 AM
2026-04-07 09:13:04 (UTC+0)
Comment Actions
Change #1256371
merged
by Muehlenhoff:
[operations/puppet@production] Switch our servers to use deb.debian.org
Maintenance_bot
removed a project:
Patch-For-Review
Tue, Apr 7, 9:31 AM
2026-04-07 09:31:39 (UTC+0)
gerritbot
added a comment.
Tue, Apr 7, 9:33 AM
2026-04-07 09:33:58 (UTC+0)
Comment Actions
Change #1268515 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Switch three test systems to deb.debian.org
gerritbot
added a project:
Patch-For-Review
Tue, Apr 7, 9:33 AM
2026-04-07 09:33:59 (UTC+0)
gerritbot
added a comment.
Tue, Apr 7, 9:39 AM
2026-04-07 09:39:39 (UTC+0)
Comment Actions
Change #1268515
merged
by Muehlenhoff:
[operations/puppet@production] Switch three test systems to deb.debian.org
gerritbot
added a comment.
Tue, Apr 7, 10:04 AM
2026-04-07 10:04:50 (UTC+0)
Comment Actions
Change #1268522 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Switch our servers to use deb.debian.org
gerritbot
added a comment.
Thu, Apr 9, 1:30 PM
2026-04-09 13:30:19 (UTC+0)
Comment Actions
Change #1268522
merged
by Muehlenhoff:
[operations/puppet@production] Switch our servers to use deb.debian.org
Maintenance_bot
removed a project:
Patch-For-Review
Thu, Apr 9, 1:30 PM
2026-04-09 13:30:49 (UTC+0)
Jdforrester-WMF
created subtask
T423596: Adjust WMCS Gitlab CI/CD repo to stop using mirrors.wikimedia.org
Thu, Apr 16, 1:14 PM
2026-04-16 13:14:24 (UTC+0)
A_smart_kitten
subscribed.
Thu, Apr 16, 1:31 PM
2026-04-16 13:31:52 (UTC+0)
MoritzMuehlenhoff
created subtask
T423598: Migrate our use of osbpo away from mirrors.wikimedia.org
Thu, Apr 16, 1:37 PM
2026-04-16 13:37:53 (UTC+0)
kevinbazira
subscribed.
Thu, Apr 16, 1:46 PM
2026-04-16 13:46:32 (UTC+0)
thcipriani
added a project:
Release-Engineering-Team (Radar)
Thu, Apr 16, 2:31 PM
2026-04-16 14:31:40 (UTC+0)
thcipriani
added subscribers:
dduvall
thcipriani
Comment Actions
Checking my understanding of "sunsetting" here:
We're no longer hosting a mirror? vs.
the
mirrors.wikimedia.org
url will cease to function?
Currently all the base docker images have
mirrors
in their apt source:
(◍•ᴗ•◍)❤ docker run --rm -it docker-registry.wikimedia.org/trixie:20260412 cat /etc/apt/sources.list | grep mirrors
deb http://mirrors.wikimedia.org/debian/ trixie main
deb http://mirrors.wikimedia.org/debian/ trixie-updates main
deb http://mirrors.wikimedia.org/debian/ trixie-backports main contrib
(◍•ᴗ•◍)❤ docker run --rm -it docker-registry.wikimedia.org/bookworm:20260412 cat /etc/apt/sources.list | grep mirrors
deb http://mirrors.wikimedia.org/debian/ bookworm main
deb http://mirrors.wikimedia.org/debian/ bookworm-updates main
deb http://mirrors.wikimedia.org/debian/ bookworm-backports main contrib
(◍•ᴗ•◍)❤ docker run --rm -it docker-registry.wikimedia.org/bullseye:20260412 cat /etc/apt/sources.list | grep mirrors
deb http://mirrors.wikimedia.org/debian/ bullseye main
deb http://mirrors.wikimedia.org/debian/ bullseye-updates main
Trying to think through the consequences of this for CI images and pipeline building (CC
@dduvall
SomeRandomDeveloper
subscribed.
Thu, Apr 16, 2:53 PM
2026-04-16 14:53:48 (UTC+0)
MoritzMuehlenhoff
added a comment.
Thu, Apr 16, 3:07 PM
2026-04-16 15:07:03 (UTC+0)
Comment Actions
In
T416707#11829949
@thcipriani
wrote:
Checking my understanding of "sunsetting" here:
We're no longer hosting a mirror? vs.
the
mirrors.wikimedia.org
url will cease to function?
These are both true. We will no longer operate a mirror (which is running under mirrors.wikimedia.org), so these image should be built using deb.debian.org as the mirror going forward.
thcipriani
created subtask
T423622: New base images without mirrors.wikimedia.org
Thu, Apr 16, 4:10 PM
2026-04-16 16:10:02 (UTC+0)
NightWolf1223
subscribed.
Thu, Apr 16, 4:39 PM
2026-04-16 16:39:29 (UTC+0)
bd808
subscribed.
Thu, Apr 16, 8:18 PM
2026-04-16 20:18:55 (UTC+0)
Comment Actions
In
T416707#11830231
@MoritzMuehlenhoff
wrote:
These are both true. We will no longer operate a mirror (which is running under mirrors.wikimedia.org), so these image should be built using deb.debian.org as the mirror going forward.
And going backward? Is there a way we can help
"cool URIs don't change"
by redirecting mirrors.wikimedia.org elsewhere?
A_smart_kitten
added a comment.
Thu, Apr 16, 9:24 PM
2026-04-16 21:24:43 (UTC+0)
Comment Actions
In
T416707#11831547
@bd808
wrote:
And going backward? Is there a way we can help
"cool URIs don't change"
by redirecting mirrors.wikimedia.org elsewhere?
I'm obviously not WMF staff; but from e.g.
T243056#6095651
it feels like a
.wikimedia.org
URL maybe shouldn't be redirected to an alternative mirror-site that isn't controlled by the WMF. But maybe e.g.
collaboration-services
could host a microsite at the
mirrors.wikimedia.org
domain, that just serves to point to alternative mirrors/sites for the downloads that it's currently providing?
bd808
added subscribers:
Krinkle
Jdforrester-WMF
Thu, Apr 16, 9:47 PM
2026-04-16 21:47:28 (UTC+0)
Comment Actions
In
T416707#11831821
@A_smart_kitten
wrote:
But maybe e.g.
collaboration-services
could host a microsite at the
mirrors.wikimedia.org
domain, that just serves to point to alternative mirrors/sites for the downloads that it's currently providing?
That would fix nothing of value for me. The problem is needing to update the apt sources for anything that is currently pointed to
mirrors.wikimedia.org
. A redirect would keep that working, at least for a bit. A static page telling me that
mirrors.wikimedia.org
has been taken offline as an apt repo would still leave everything broken.
I assume that
@Krinkle
and
@Jdforrester-WMF
were concerned about things like HTTP cookie leaks in that exchange, but maybe there is something even more insidious that I'm not considering. Anyway, I'd like to hear from a WMF SRE making system changing decisions on the general topic.
LSobanski
added a comment.
Fri, Apr 17, 5:58 AM
2026-04-17 05:58:08 (UTC+0)
Comment Actions
@bd808
In addition to the good point raised by
@A_smart_kitten
above the general intent here is to reduce complexity. Leaving a dependency that is no longer tied to a service and has potential to break silently in the future would go against that. The preference is for end users to switch to the canonical (no pun intended) way of accessing repositories.
gerritbot
added a comment.
Fri, Apr 17, 6:52 AM
2026-04-17 06:52:02 (UTC+0)
Comment Actions
Change #1273441 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Switch Cloud VPS to deb.debian.org
gerritbot
added a project:
Patch-For-Review
Fri, Apr 17, 6:52 AM
2026-04-17 06:52:03 (UTC+0)
Nemoralis
added a project:
User-notice
Fri, Apr 17, 7:04 AM
2026-04-17 07:04:43 (UTC+0)
Nemoralis
moved this task from
To Triage
to
Announce in next Tech/News
on the
User-notice
board.
Fri, Apr 17, 7:15 AM
2026-04-17 07:15:09 (UTC+0)
STei-WMF
moved this task from
Announce in next Tech/News
to
In current Tech/News draft
on the
User-notice
board.
Fri, Apr 17, 11:44 AM
2026-04-17 11:44:10 (UTC+0)
gerritbot
added a comment.
Mon, Apr 20, 9:46 AM
2026-04-20 09:46:58 (UTC+0)
Comment Actions
Change #1273441
merged
by Muehlenhoff:
[operations/puppet@production] Switch Cloud VPS to deb.debian.org
Maintenance_bot
removed a project:
Patch-For-Review
Mon, Apr 20, 10:33 AM
2026-04-20 10:33:37 (UTC+0)
Krinkle
unsubscribed.
Mon, Apr 20, 10:57 AM
2026-04-20 10:57:59 (UTC+0)
Jdforrester-WMF
changed the status of subtask
T423622: New base images without mirrors.wikimedia.org
from
Open
to
In Progress
Mon, Apr 20, 2:51 PM
2026-04-20 14:51:41 (UTC+0)
Jdforrester-WMF
closed subtask
T423622: New base images without mirrors.wikimedia.org
as
Resolved
Mon, Apr 20, 6:42 PM
2026-04-20 18:42:36 (UTC+0)
thcipriani
created subtask
T423971: Update default GitLab runner image to a base image without mirrors.wikimedia.org
Tue, Apr 21, 12:00 AM
2026-04-21 00:00:29 (UTC+0)
thcipriani
created subtask
T423972: Rebuild dev-images using a base image without mirrors.wikimedia.org in the apt sources
Tue, Apr 21, 12:02 AM
2026-04-21 00:02:58 (UTC+0)
Novem_Linguae
subscribed.
Tue, Apr 21, 3:00 PM
2026-04-21 15:00:49 (UTC+0)
STei-WMF
moved this task from
In current Tech/News draft
to
Already announced/Archive
on the
User-notice
board.
Thu, Apr 23, 7:53 PM
2026-04-23 19:53:37 (UTC+0)
Log In to Comment
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct.
Wikimedia Foundation
Code of Conduct
Disclaimer
CC-BY-SA
GPL
Credits