2025 RecentChanges Language Agnostic Revert Risk Filtering - MediaWiki
Jump to content
From mediawiki.org
Translate this page
Languages:
Bahasa Indonesia
Bahasa Melayu
Zazaki
italiano
polski
português
português do Brasil
čeština
русский
العربية
हिन्दी
မြန်မာဘာသာ
中文
Wikimedia Foundation projects
RecentChanges Language Agnostic Revert Risk Filtering
Add Language Agnostic Revert Risk Filter to wikis where it will have high impact
Start:
2025-04-01
End:
2025-06-30
Team members:
Moderator Tools
Machine Learning
Background
edit
Recent changes
is a feed of all edits on a wiki. Patrollers and moderators rely on Recent changes to identify edits requiring action.
The
ORES
extension fetches and stores Machine learning model scores for recent edits, which are used to filters changes on
Special:RecentChanges
, highlighting the ones with a high risk of being reverted. These filters help patrollers to find edits that are more likely to be damageable for the wiki.
The ORES infrastructure
is being deprecated
by the Machine Learning team. Plus, this extension and the associated filters are only enabled on select wikis.
The Moderator Tools team would like to add a similar filter to all Wikipedias, by using the
Language-Agnostic Revert Risk
The Moderator Tools team has determined a list of wikis would benefit from this filter (and do not have high technical hurdles), regardless of whether or not the ORES extension is currently enabled for that wiki.
What is changing
edit
A new filter is added to Recent changes filters:
Likely to be reverted
Detects revisions that might be reverted, but does not assess the user's intention.
This filter is shown under a new section: "Risk of being reverted".
It can be used like
any other filter
This filter identifies edits that are
likely to be reverted
by scoring an edit against a revert risk model threshold.
It is a tool to
assist
Recent changes reviewers.
However, some of the changes filtered won't need to be removed, while some other changes that should be reverted might not be present in the filtered results.
In the future, this new filter could replace
existing quality filters
, powered by ORES, which are deployed at a few wikis.
Likely to be reverted threshold analysis findings
edit
Overview
The filter works by scoring an edit against a revert risk model and if the score is above the specified threshold, then the edit is categorized as
"likely to be reverted"
As part of the threshold analysis, we wanted to better understand the distribution of revert risk scores and how they corresponded to real-world activities (i.e. was the edit actually reverted) so that we can provide a process to arrive at buckets for the scores, e.g.“Likely to be reverted”.
Methodology
We reviewed a sample of article namespace edits completed on Indonesian Wikipedia. The edit sample included if the edit was reverted within the reviewed timeframe and the revert risk score based on the scores provided by the language-agnostic model.
Some data analysis was performed to target a threshold which resulted in an approximately 15% false positive rate.
In other words, approximately 15% of edits or approximately 1 out of 7 edits shown as likely to be reverted are falsely classified as such.
We targeted a 15% false positive rate as it seemed like an appropriate number of false positives for a user-facing feature.
The threshold analysis is repeatable (
using this notebook
).
In order to target different levels of false positive rates (lower or higher than 15%) you can re-run the notebook with different parameters and arrive at a more optimized threshold.
The threshold can then be updated via configuration like
in this patch
Note, this process is only available for language agnostic wikis currently.
Thresholds
As a result of the analysis we were able to arrive at the following threshold for Indonesian Wikipedia.
Revert Risk Score
Meaning
Likely Outcome/Tradeoffs
Supplemental data
Over 0.85
Very likely for edit to be reverted.
Moderate false positive rate with high false negative rate.
Correctly identifies a little over half (59.4%) of edits that were actually reverted.
This threshold is good at catching the majority of edits that should be reverted (high recall), but it also flags some edits (~15%) as should be reverted when they weren't (low precision).
Recall:
0.594
Precision:
0.239
False Positive Rate:
14.5%
False Negative Rate:
40.6%
Wiki
Optimal threshold
cywiki
0.11076631
simplewiki
0.90655595
bewiki
0.5724957
kkwiki
0.6048708
nnwiki
0.41624594
mkwiki
0.5058307
lawiki
0.6340628
afwiki
0.7369383
tewiki
0.36725548
mrwiki
0.86735326
swwiki
0.74820924
mlwiki
0.93626505
iswiki
0.88926035
pawiki
0.545889
hawiki
0.4823182
tlwiki
0.60741657
bnwiki
0.64658594
trwiki
0.6082413
azwiki
0.5366108
FAQ
edit
Sometimes, the filters show edits that are not vandalism. Is it normal?
Yes, some edits are false positives, about 15% of the diff filtered. False positives are part of the detection. We can reduce this percentage, but then the number of edits being detected as vandalism would drastically decrease.
How to improve the false positive rate?
To improve the false positive rate, you can increase the threshold to a higher number as desired.
Our language has a high threshold, meaning that not many vandalisms are detected, right?
We are stricter on classifying something as vandalism, but at the same time we are confident that our decision is reliable in not making false predictions.
Targeted wikis
edit
The changes will be applied to wikis identified in
T392144
Indonesian Wikipedia (pilot wiki)
Welsh Wikipedia
Simple English Wikipedia
Belarusian Wikipedia
Kazakh Wikipedia
Norwegian Nynorsk Wikipedia
Macedonian Wikipedia
Latin Wikipedia
Afrikaans Wikipedia
Telugu Wikipedia
Marathi Wikipedia
Swahili Wikipedia
Malayalam Wikipedia
Icelandic Wikipedia
Punjabi Wikipedia
Hausa Wikipedia
Tagalog Wikipedia
Bengali Wikipedia
Turkish Wikipedia
Azerbaijani Wikipedia
Key Milestones
Identify target wikis
T392144
Deploy only to idwiki initially to validate process
Deploy to remaining wikis on list
Including: Deploy to two wikis that have more complexities
phab:T395668
Help & Learning More
edit
The interface can be translated at
translatewiki
We welcome your questions and comments
at the talk page
Retrieved from "
Hidden category:
WMF Projects 2025q2
2025 RecentChanges Language Agnostic Revert Risk Filtering
Add topic