⚓ T141490 Deploy improved FancyCaptcha

⚓ T141490 Deploy improved FancyCaptcha
Page Menu
Phabricator
Create Task
Maniphest
T141490
Deploy improved FancyCaptcha
Closed, Resolved
Public
Actions
Edit Task
Edit Related Tasks...
Create Subtask
Edit Parent Tasks
Edit Subtasks
Merge Duplicates In
Close As Duplicate
Edit Related Objects...
Edit Commits
Edit Mocks
Mute Notifications
Protect as security issue
Assigned To
None
Authored By
tstarling
Jul 27 2016, 10:13 PM
2016-07-27 22:13:18 (UTC+0)
Tags
Wikimedia-Site-requests
(External)
Security
ConfirmEdit (CAPTCHA extension)
(Backlog)
MW-1.42-notes (1.42.0-wmf.14; 2024-01-16)
User-notice-archive
(Backlog)
Referenced Files
F41753329: Figure_1_new_week.png
Feb 2 2024, 4:20 PM
2024-02-02 16:20:55 (UTC+0)
F41723772: grafik.png
Jan 27 2024, 11:15 PM
2024-01-27 23:15:58 (UTC+0)
F41711785: grafik.png
Jan 27 2024, 11:15 PM
2024-01-27 23:15:58 (UTC+0)
F41720804: Screenshot from 2024-01-26 10-46-31.png
Jan 27 2024, 12:13 AM
2024-01-27 00:13:28 (UTC+0)
F41720802: Screenshot from 2024-01-26 14-01-50.png
Jan 27 2024, 12:13 AM
2024-01-27 00:13:28 (UTC+0)
F41720803: Screenshot from 2024-01-26 10-43-55.png
Jan 27 2024, 12:13 AM
2024-01-27 00:13:28 (UTC+0)
F41720801: Screenshot from 2024-01-26 10-44-01.png
Jan 27 2024, 12:13 AM
2024-01-27 00:13:28 (UTC+0)
F41691877: grafik.png
Jan 15 2024, 8:01 PM
2024-01-15 20:01:10 (UTC+0)
View All 26 Files
Subscribers
ABorbaWMF
Aklapper
ArielGlenn
Bawolff
Daimona
Dbrant
dpatrick
View All 41 Subscribers
Description
In 2014, I investigated FancyCaptcha's resistance to OCR. I found that it had essentially no resistance, that it could be trivially broken by open source software without image preprocessing or OCR engine configuration.
In
these
two
changes, I implemented changes which were confirmed to defeat such naïve OCR attacks. Specifically, I tweaked the tunable parameters to improve distortion of the baseline, and added low-spatial-frequency noise and a gradient to defeat thresholding.
These changes were never deployed to WMF. I propose now doing so.
Here is some representative output:
Old
New
The procedure to regenerate the captcha image set is documented at
Details
Related Changes in Gerrit:
Subject
Repo
Branch
Lines +/-
Add negative kerning and lines to captcha
mediawiki/extensions/ConfirmEdit
master
+24
-34
mediawiki: Use the new captcha
operations/puppet
production
+2
-2
captcha.py: Increase number and position of random lines in the text
mediawiki/extensions/ConfirmEdit
master
+6
-6
Customize query in gerrit
Related Objects
Search...
Task Graph
Mentions
Status
Subtype
Assigned
Task
Restricted Task
Resolved
None
T141490
Deploy improved FancyCaptcha
Resolved
BUG REPORT
Ladsgroup
T355505
Beta cluster captcha prevents new account creation when username includes whitespace
Mentioned In
T152219: Statistics on Captcha success/failure rate
T354099: FancyCaptcha isn't compatible with pillow 10.x
T355962: [betacluster] Special:Captcha/image reports 404 (Not Found) on Create account page
T42360: Obscure the letters of the graphical CAPTCHA in a different way
T241921: Fix Wikimedia captchas
T230304: Ongoing spambot attack 2019-08-{10,11,.*}
T204615: Generate new Captcha word list for prod
T174877: Spambots as IP addresses and as accounts again prolific within WMF wikis
T158909: Automatically detect spambot registration using machine learning (like invisible reCAPTCHA)
T150029: Create cronjob for regular captcha regeneration
Mentioned Here
T354099: FancyCaptcha isn't compatible with pillow 10.x
T355962: [betacluster] Special:Captcha/image reports 404 (Not Found) on Create account page
T263223: captcha.py issues due to change in `/` behaviour in python 3
T255208: Catalog and evaluate methods of analysis for Wikimedia captcha performance
T212667: Create mitigations for account creation spam attack [public task]
T187826: Create mediawiki::maintenance server (aka terbium) in deployment-prep
T186244: Deploy AICaptcha data collection
T158909: Automatically detect spambot registration using machine learning (like invisible reCAPTCHA)
T157734: Add threading to captcha.py
T152219: Statistics on Captcha success/failure rate
T157735: Update logging in ConfirmEdit
T150029: Create cronjob for regular captcha regeneration
Event Timeline
There are a very large number of changes, so older changes are hidden.
Show Older Changes
MarcoAurelio
mentioned this in
T230304: Ongoing spambot attack 2019-08-{10,11,.*}
Aug 11 2019, 11:33 PM
2019-08-11 23:33:48 (UTC+0)
Reedy
changed the task status from
Open
to
Stalled
Nov 1 2019, 11:35 AM
2019-11-01 11:35:05 (UTC+0)
Reedy
triaged this task as
Medium
priority.
Tgr
mentioned this in
T241921: Fix Wikimedia captchas
Jan 5 2020, 8:12 AM
2020-01-05 08:12:00 (UTC+0)
Tgr
mentioned this in
T42360: Obscure the letters of the graphical CAPTCHA in a different way
Jan 8 2020, 6:06 AM
2020-01-08 06:06:19 (UTC+0)
Tgr
added a parent task:
Restricted Task
Jan 8 2020, 8:38 AM
2020-01-08 08:38:15 (UTC+0)
chasemp
added a project:
Security
Feb 10 2020, 10:59 PM
2020-02-10 22:59:07 (UTC+0)
chasemp
removed a project:
acl*security
Feb 20 2020, 8:07 PM
2020-02-20 20:07:54 (UTC+0)
Reedy
added a project:
ConfirmEdit (CAPTCHA extension)
Jun 8 2020, 6:47 PM
2020-06-08 18:47:03 (UTC+0)
Aklapper
added a comment.
Aug 22 2020, 7:57 AM
2020-08-22 07:57:39 (UTC+0)
Comment Actions
@Reedy
: Hmm, what exactly is this task
stalled
on?
Reedy
added a comment.
Aug 22 2020, 1:16 PM
2020-08-22 13:16:42 (UTC+0)
Comment Actions
In
T141490#6403690
@Aklapper
wrote:
@Reedy
: Hmm, what exactly is this task
stalled
on?
Lack of any consensus. Lack of decent enough metrics to even deploy the change and see what happens. It's been open 4 years at this point, and no clear path of moving forward for deploying any of the improvements proposed.
It'll probably get declined at some point
Tgr
added a comment.
Aug 22 2020, 2:19 PM
2020-08-22 14:19:14 (UTC+0)
Comment Actions
Mainly lack of metrics; lack of consensus is a consequence. So the path forward would be
T255208: Catalog and evaluate methods of analysis for Wikimedia captcha performance
or something similar - and eventually, some reporting mechanism that gives some reasonably reliable numbers on how well the captcha performs against humans vs. against bots. It's a few weeks of work, IMO, it just has been hotpotato'd around between various departments.
TimSC
subscribed.
Edited
Sep 18 2020, 4:25 AM
2020-09-18 04:25:30 (UTC+0)
Comment Actions
Moved this bug report to its own item
Aklapper
changed the task status from
Stalled
to
Open
Nov 3 2020, 9:56 AM
2020-11-03 09:56:19 (UTC+0)
Aklapper
lowered the priority of this task from
Medium
to
Lowest
Comment Actions
In
T141490#6403867
@Reedy
wrote:
Lack of any consensus. Lack of decent enough metrics to even deploy the change and see what happens. It's been open 4 years at this point, and no clear path of moving forward for deploying any of the improvements proposed.
That doesn't make it
stalled by definition
but low priority :)
Ladsgroup
subscribed.
Nov 27 2023, 11:08 PM
2023-11-27 23:08:18 (UTC+0)
Comment Actions
I might be stating the obvious but a change like this shouldn't take 10 years to be deployed (with the last comment being more than three years ago). Is there anything I can do to get this off the ground? Should I just make the patches and get it deployed and check some (*waves at the air*) metrics? Should I just get
T255208
done first? Anyone willing to help?
sbassett
added a comment.
Nov 28 2023, 7:17 PM
2023-11-28 19:17:59 (UTC+0)
Comment Actions
In
T141490#9361722
@Ladsgroup
wrote:
I might be stating the obvious but a change like this shouldn't take 10 years to be deployed (with the last comment being more than three years ago). Is there anything I can do to get this off the ground? Should I just make the patches and get it deployed and check some (*waves at the air*) metrics? Should I just get
T255208
done first? Anyone willing to help?
Given the amount of time that has passed and the lack of conclusive data on how this might improve FancyCaptcha's efficacy while not further hindering accessibility, I'm not sure a simple code-cleanup and deploy would be the best option here. We definitely wouldn't want to introduce a worse captcha experience for project users at this point.
Tgr
added a comment.
Nov 29 2023, 10:16 PM
2023-11-29 22:16:02 (UTC+0)
Comment Actions
AIUI the fundamental blocker here is not being able to differentate between increased captcha failure rate for bots vs. increased captcha failure rate for humans. So we'd need captcha success rate stats (which we sort of have, but not very good ones) plus some sort of bot detection.
Also IMO the patch proposed here is a non-starter because it makes the captcha a lot harder to read, while the security gains would be limited at best.
@Bawolff
had some ideas for captchas which at least at first glance don't look harder for a human (see comments starting at T125132#4432800).
Bawolff
added a comment.
Nov 30 2023, 7:38 AM
2023-11-30 07:38:42 (UTC+0)
Comment Actions
I feel like we're over emphasizing perfect stats here. Its not like the original captcha had extensive testing. Surely we can test how readable any given new captcha proposal is, by asking for a bunch of volunteers from the community to solve some captchas and see how they do.
Ladsgroup
added a comment.
Dec 1 2023, 6:41 PM
2023-12-01 18:41:39 (UTC+0)
Comment Actions
I agree with Brian here, Captcha is by design hard to measure. If you have a way to detect bot request failure, then why are relying on captcha in the first place? just use that method to block the bots.
Tgr
added a comment.
Dec 1 2023, 10:38 PM
2023-12-01 22:38:29 (UTC+0)
Comment Actions
Sure, user testing works too. I'm not sure it's less effort. You can look at effect of captcha on known-human users (e.g. IPs from some insitutional range), split things by user agent etc.
Nemo_bis
added a comment.
Dec 3 2023, 9:04 PM
2023-12-03 21:04:36 (UTC+0)
Comment Actions
You can look at effect of captcha on known-human users (e.g. IPs from some insitutional range)
Sounds difficult... what network can possibly be guaranteed not to have any bot on it?
I agree it _might_ be possible to pick some statistical proxy which would give us an idea of the impact. For example, what portion of the users pass the captcha and then get blocked, and perhaps more importantly how many capcha attempts were needed by users who did pass the captcha and did not get blocked after making an edit.
kostajh
subscribed.
Dec 14 2023, 7:28 AM
2023-12-14 07:28:20 (UTC+0)
Ladsgroup
added a comment.
Jan 15 2024, 1:34 PM
2024-01-15 13:34:47 (UTC+0)
Comment Actions
Examples of output of the patch I'm about to make
gerritbot
added a comment.
Jan 15 2024, 1:35 PM
2024-01-15 13:35:11 (UTC+0)
Comment Actions
Change 990694 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):
[mediawiki/extensions/ConfirmEdit@master] Add negative kerning and lines to captcha
gerritbot
added a project:
Patch-For-Review
Jan 15 2024, 1:35 PM
2024-01-15 13:35:12 (UTC+0)
gerritbot
added a comment.
Jan 15 2024, 2:10 PM
2024-01-15 14:10:03 (UTC+0)
Comment Actions
Change 990697 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):
[operations/puppet@production] mediawiki: Use the new captcha
ArielGlenn
subscribed.
Jan 15 2024, 2:18 PM
2024-01-15 14:18:38 (UTC+0)
gerritbot
added a comment.
Jan 15 2024, 2:35 PM
2024-01-15 14:35:15 (UTC+0)
Comment Actions
Change 990694
merged
by jenkins-bot:
[mediawiki/extensions/ConfirmEdit@master] Add negative kerning and lines to captcha
Daimona
subscribed.
Jan 15 2024, 2:59 PM
2024-01-15 14:59:43 (UTC+0)
ReleaseTaggerBot
added a project:
MW-1.42-notes (1.42.0-wmf.14; 2024-01-16)
Jan 15 2024, 3:00 PM
2024-01-15 15:00:40 (UTC+0)
Ladsgroup
added a comment.
Jan 15 2024, 4:33 PM
2024-01-15 16:33:59 (UTC+0)
Comment Actions
FWIW, this is deployed in beta cluster as of now.
gerritbot
added a comment.
Jan 15 2024, 4:56 PM
2024-01-15 16:56:50 (UTC+0)
Comment Actions
Change 990726 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):
[mediawiki/extensions/ConfirmEdit@master] captcha.py: Increase number of random lines in the text
Ladsgroup
added a comment.
Jan 15 2024, 5:01 PM
2024-01-15 17:01:11 (UTC+0)
Comment Actions
^ This increases the number of lines, feel free to accept or abandon it. Some examples:
I haven't thrown this into an OCR to check its performance though.
Bawolff
added a comment.
Jan 15 2024, 7:17 PM
2024-01-15 19:17:13 (UTC+0)
Comment Actions
The examples do seem like the lines are a bit low, maybe there should be some lines at the bottom and some near the top of the word
Ladsgroup
added a comment.
Jan 15 2024, 8:01 PM
2024-01-15 20:01:10 (UTC+0)
Comment Actions
oh indeed, made some changes:
Is it better?
Bawolff
added a comment.
Jan 15 2024, 8:39 PM
2024-01-15 20:39:30 (UTC+0)
Comment Actions
I think so, it looks more random to me anyhow (this is just gut feeling, i haven't done any testing)
Ladsgroup
added a comment.
Jan 15 2024, 9:22 PM
2024-01-15 21:22:56 (UTC+0)
Comment Actions
Ran
Tesseract Open Source OCR Engine v4.1.1 with Leptonica
on 1000 generations and only on 71 it produced any output which mostly were garbage:
( CutsSauna-
Ceeuighv ews —
ashySears
U Serubpadgy —
we
newts julips
bongdoar_
( pathghebe
wi
\sewersaez—
——
Ss
~ spagemary—
<= Gegn
“ Fenépunk—
-eedasonja-
—waveduery
(CupssnowY
~Sakhaurged
c_ ahmeddears—
Cstudsrusts——
Only one case it produced the exact value, and if you strip out the garbage, it still doesn't go higher than five cases in total.
gerritbot
added a comment.
Jan 15 2024, 9:51 PM
2024-01-15 21:51:35 (UTC+0)
Comment Actions
Change 990726
merged
by jenkins-bot:
[mediawiki/extensions/ConfirmEdit@master] captcha.py: Increase number and position of random lines in the text
Tamzin
subscribed.
Jan 15 2024, 10:08 PM
2024-01-15 22:08:21 (UTC+0)
Vermont
subscribed.
Jan 15 2024, 10:08 PM
2024-01-15 22:08:36 (UTC+0)
Ladsgroup
added a comment.
Edited
Jan 15 2024, 10:13 PM
2024-01-15 22:13:36 (UTC+0)
Comment Actions
In
T141490#9460756
@Ladsgroup
wrote:
Only one case it produced the exact value, and if you strip out the garbage, it still doesn't go higher than five cases in total.
For comparison, the out of 20 mentioned above (
F4313219
), Tesseract correctly recognizes all twenty of them
Edit: That wasn't a fair comparison, With the old version can, the rate is 33% (
), with the newer one probably higher.
Bawolff
added a comment.
Edited
Jan 15 2024, 10:53 PM
2024-01-15 22:53:55 (UTC+0)
Comment Actions
The options you give tesseracrt can make a big difference, in my old test i used
-psm 13 -c tessedit_char_whitelist=abcdefghijklmnopqrstuvwxyz
We might want to consider making them not real words, to prevent spell checking attacks.
Ladsgroup
added a comment.
Jan 16 2024, 11:54 AM
2024-01-16 11:54:40 (UTC+0)
Comment Actions
Fair, with that options, you get 7.8% accuracy.
Bawolff
added a comment.
Jan 18 2024, 10:22 AM
2024-01-18 10:22:19 (UTC+0)
Comment Actions
In
T141490#9460683
@Ladsgroup
wrote:
oh indeed, made some changes:
Is it better?
I did some testing, and posted the results at T125132#9468294
Bugreporter
added a subtask:
T355505: Beta cluster captcha prevents new account creation when username includes whitespace
Jan 22 2024, 10:15 AM
2024-01-22 10:15:48 (UTC+0)
awight
closed subtask
T355505: Beta cluster captcha prevents new account creation when username includes whitespace
as
Resolved
Jan 23 2024, 11:16 AM
2024-01-23 11:16:46 (UTC+0)
Stashbot
added a comment.
Jan 25 2024, 5:30 PM
2024-01-25 17:30:29 (UTC+0)
Comment Actions
Mentioned in SAL (#wikimedia-operations)
[2024-01-25T17:30:28Z] deploying new captchas (
T141490
Ladsgroup
added a project:
User-notice
Jan 25 2024, 5:44 PM
2024-01-25 17:44:40 (UTC+0)
Comment Actions
This is deployed.
gerritbot
added a comment.
Jan 25 2024, 5:45 PM
2024-01-25 17:45:01 (UTC+0)
Comment Actions
Change 990697
merged
by Ladsgroup:
[operations/puppet@production] mediawiki: Use the new captcha
sbassett
added a comment.
Jan 25 2024, 5:53 PM
2024-01-25 17:53:58 (UTC+0)
Comment Actions
In
T141490#9489154
@Ladsgroup
wrote:
This is deployed.
Resolvable for now then?
DTorsani-WMF
subscribed.
Jan 25 2024, 5:54 PM
2024-01-25 17:54:12 (UTC+0)
Comment Actions
From an accessibility perspective, the proposed solution will decrease usability for people with blurred vision and dyslexia, potentially other users with visual impairments, but it does not drastically decrease the overall accessibility. That being said, I believe this is a fine solution for now.
Dzahn
subscribed.
Edited
Jan 25 2024, 5:55 PM
2024-01-25 17:55:34 (UTC+0)
Comment Actions
@Bawolff
How much does it matter if the font is FreeMonoBoldOblique or DejaVuSans? I see
where it was switched because the former font isn't installed on mwmaint hosts.
Should we look more into how to get that installed or is DejaVuSans fine?
fonts-freefont might be the package.
Ladsgroup
closed this task as
Resolved
Jan 25 2024, 5:56 PM
2024-01-25 17:56:41 (UTC+0)
Comment Actions
I'm resolving this.
Ladsgroup
added a comment.
Jan 25 2024, 5:57 PM
2024-01-25 17:57:23 (UTC+0)
Comment Actions
(Some more work is needed, changing the font if suitable, comms, measurements, etc. but doesn't need to happen in this ticket)
DTorsani-WMF
added a comment.
Jan 25 2024, 6:01 PM
2024-01-25 18:01:13 (UTC+0)
Comment Actions
re: font choice. Visually, there is slightly more noise with the use of FreeMonoBoldOblique because of the serifs. This might have an effect on the ability to break it if that noise is helpful to make the CAPTCHA more difficult to break. From a readability perspective, DejaVuSans seems fine as that lack of additional serif noise may increase readability when all the characters are more close together with the noise of the abstract lines. But again, does that effect the ability to break it? Have these two fonts been tested against one another?
Ladsgroup
updated the task description.
(Show Details)
Jan 25 2024, 6:24 PM
2024-01-25 18:24:10 (UTC+0)
Bawolff
added a comment.
Edited
Jan 25 2024, 6:24 PM
2024-01-25 18:24:20 (UTC+0)
Comment Actions
In
T141490#9489169
@Dzahn
wrote:
@Bawolff
How much does it matter if the font is FreeMonoBoldOblique or DejaVuSans? I see
where it was switched because the former font isn't installed on mwmaint hosts.
Should we look more into how to get that installed or is DejaVuSans fine?
fonts-freefont might be the package.
FreeSans
is more secure against more sophisticated attackers at the cost of somewhat lower readability (due to lower interletter spacing).
DejaVu sans
and
FreeMonoBoldOblique
are roughly the same level of security imo.
Maintenance_bot
removed a project:
Patch-For-Review
Jan 25 2024, 6:30 PM
2024-01-25 18:30:53 (UTC+0)
Seddon
added subscribers:
JTannerWMF
Dbrant
Tsevener
ABorbaWMF
Jan 25 2024, 7:21 PM
2024-01-25 19:21:29 (UTC+0)
Tgr
added a comment.
Jan 25 2024, 7:41 PM
2024-01-25 19:41:58 (UTC+0)
Comment Actions
In
T141490#9489166
@DTorsani-WMF
wrote:
From an accessibility perspective, the proposed solution will decrease usability for people with blurred vision and dyslexia, potentially other users with visual impairments
I think the example in the task description oversells the readability of the current captcha (there is a random factor in how "hairy" it is and the more hairy ones can be quite difficult to read, even for average readers). Brian made a great dashboard where you can check out a more realistic set of examples:
old captcha
new captcha
The old captcha is hard to read in a certain way (characters can get quite distorted), the new captcha is hard to read in a different way (the characters aren't distorted but get squished together). I'm not sure what's the net effect is.
Bawolff
added a comment.
Edited
Jan 25 2024, 8:31 PM
2024-01-25 20:31:09 (UTC+0)
Comment Actions
Fwiw: my intuition around different text captcha methods:
There are basically 3 steps in attacking a captcha - remove extranous elements, separate the word into individual letters, identify each letter. Thus those are the three knobs we can control to adjust difficulty.
non letter elements (lines etc) - about equally hard for a computer and human. If they are obviously different then letters than both humans and computers can discard them. If the look the same as letters then hard for both. The best thing about this method is that off the shelf OCR software can't handle it well, so the attacker is forced to program their own thing, even if the algorithms involved aren't super complex. Thus we can get some benefit for very little drop in readability.
overlapping letters - this is generally hard for computers and medium difficulty for humans (relatively speaking). This aims to prevent computers from seperating words into letters, so the entire captcha has to be taken as a whole. The algorithms involved in segmenting tend to be more complex, forcing the attacker to spend much more effort programming counter measures.
distorting letter shapes. Generally hard for humans and easier for computers. This usually doesn't prevent a computer from segmenting a word into letters, and once the computer is dealing with it a letter at a time, things become much easier as the computer has only 26 choices and only needs to decide which of the 26 it is closest to, which computers can usually do with ease if it is all readable by humans. Additionally this part overlaps with OCR, so there are excellent off the shelf solutions.
Dzahn
added a comment.
Jan 25 2024, 8:36 PM
2024-01-25 20:36:17 (UTC+0)
Comment Actions
Ok, thanks Bawolff, won't worry about the package then.
Quiddity
moved this task from
To Triage
to
In current Tech/News draft
on the
User-notice
board.
Jan 26 2024, 2:27 AM
2024-01-26 02:27:39 (UTC+0)
Dzahn
mentioned this in
T355962: [betacluster] Special:Captcha/image reports 404 (Not Found) on Create account page
Jan 26 2024, 8:21 PM
2024-01-26 20:21:31 (UTC+0)
Comment Actions
Is this (new?) captcha-related issue on beta cluster related?
T355962
Ladsgroup
added a comment.
Jan 26 2024, 8:41 PM
2024-01-26 20:41:22 (UTC+0)
Comment Actions
In
T141490#9491493
@Dzahn
wrote:
Is this (new?) captcha-related issue on beta cluster related?
T355962
It's unrelated, the new captcha was deployed to beta cluster for weeks now. I know what's the root cause of that bug.
TheDJ
subscribed.
Edited
Jan 26 2024, 10:17 PM
2024-01-26 22:17:50 (UTC+0)
Comment Actions
FYI, this rebroke captcha.py for pillow 10, by reintroducing getsize, see also
T354099
Tgr
added a comment.
Jan 27 2024, 12:13 AM
2024-01-27 00:13:28 (UTC+0)
Comment Actions
Some stats (web only; event / minute, 3-hour rolling average, last 7 days compared with previous 7-day periods; for the last one % of failed captcha attempts from all captcha attempts, 6-hour rolling average):
successful registrations
link
successful captcha attempts
link
failed captcha attemps
link
captcha failure rate
link
Zero effect on captcha success, but captcha failures are almost cut in half, and the changes correlate well with the time of the deploy (16:30 on the 25th). What does that mean?
I can think of two explanations:
The average user finds the new captcha easier. We are down from a ~25% human failure rate to ~15%.
Some class of not very smart spambots that could OCR the captcha into a string in the past but with near-100% failure rate now has its OCR broken so hard that it doesn't even try to submit anything.
Either way, this seems like a positive change (although in the second case, not very consequential).
Ladsgroup
added a comment.
Jan 27 2024, 11:15 PM
2024-01-27 23:15:58 (UTC+0)
Comment Actions
I did some measurements similar to what Tim did in
T152219#3405800
. I suggest reading that comment before continuing to read this.
Note that before the deploy, the measurement was for a week, but after the deploy, I have data for a day (and will repost once a full week passes).
Wikis failure ratio graph:
before deploy (for a week)
after deploy (for a day)
The plateau goes down from ~20% to 15-16% which to repeat Tim's prediction from seven years ago (
T152219#3405800
):
If we switched to a different CAPTCHA solution, we would want to see the height of the plateau remain the same, or be reduced.
(Why? The plateau basically represents large-enough wikis with mostly human activities)
Which is exactly what happened here, you could roughly estimate that we had 20% decrease in human failure (from roughly 20% to 16%). Any that is also aligned with what Tgr said in
T141490#9491802
Noting that it doesn't mean it's better for a11y, it's just easier to read for humans overall, it might make it easier for certain people while making it harder for other but the net change seems to be an improvement.
We know it's very likely human failure ratio going down because the slope for wikis with mostly spambot activities didn't move in a negative direction. It is too early to give an exact value for the derivative of the slope after 80% of failure rate but I'm not seeing that slope stopping at below 100% or having a lower derivative. I wait for a week for the exact estimate of impact on the spambots.
Here is list of large wikis failure ratio and the change:
wiki
failure ratio after deploy (1 day)
failure ratio before deploy (1 week)
difference
ukwiki
6.9%
23.8%
-16.9
metawiki
24.3%
40.9%
-16.6
fawiki
19.4%
34.1%
-14.7
kowiki
6.2%
19.9%
-13.7
thwiki
14.5%
27.5%
-12.9
arwiki
24.5%
37.0%
-12.5
bnwiki
22.0%
33.8%
-11.8
enwiktionary
25.3%
36.9%
-11.6
svwiki
14.3%
25.0%
-10.7
hiwiki
30.2%
40.7%
-10.5
ruwiki
13.7%
22.9%
-9.2
mediawikiwiki
19.1%
27.6%
-8.5
viwiki
16.5%
24.3%
-7.8
eswiki
21.0%
28.7%
-7.7
cswiki
12.0%
18.9%
-6.9
jawiki
14.9%
21.1%
-6.3
enwiki
16.0%
22.2%
-6.2
frwiki
16.4%
22.5%
-6.0
itwiki
12.5%
18.0%
-5.5
commonswiki
19.3%
24.4%
-5.1
zhwiki
16.6%
21.4%
-4.8
plwiki
12.7%
17.4%
-4.8
nlwiki
21.5%
26.1%
-4.7
uzwiki
23.9%
27.9%
-3.9
trwiki
18.4%
22.2%
-3.8
hewiki
27.4%
31.1%
-3.6
dewiki
15.4%
17.9%
-2.5
ptwiki
23.3%
23.8%
-0.4
elwiki
18.2%
17.5%
0.7
fiwiki
13.8%
13.0%
0.8
wikidatawiki
28.7%
25.5%
3.2
simplewiki
21.0%
17.8%
3.2
idwiki
28.8%
22.9%
5.9
I want to note that the wikis that had the highest redaction in failure rate were mostly wikis in languages that
are not
written with Latin scripts (Ukrainian, Persian, Korean, Thai, Arabic, Bangladeshi, Hindi, Russian). That would explain the massive redaction in human failure rate.
Johannnes89
subscribed.
Jan 29 2024, 7:52 PM
2024-01-29 19:52:46 (UTC+0)
Stashbot
added a comment.
Jan 30 2024, 3:06 PM
2024-01-30 15:06:41 (UTC+0)
Comment Actions
Mentioned in SAL (#wikimedia-operations)
[2024-01-30T15:06:41Z] Manual run of mediawiki_job_generatecaptcha.service following timer failure -
T141490
UOzurumba
moved this task from
In current Tech/News draft
to
Already announced/Archive
on the
User-notice
board.
Feb 1 2024, 6:22 PM
2024-02-01 18:22:06 (UTC+0)
Ladsgroup
added a comment.
Edited
Feb 2 2024, 4:20 PM
2024-02-02 16:20:55 (UTC+0)
Comment Actions
Results for a week are in:
Regarding impact on human failure rate
The plateau is at 16% which is lower than 22% on the old one, a 27% redaction on failure rate for humans.
Regarding impact on non-latin wikis:
On wikis that had more than 100 captcha attempt in the week of 26 Jan - 2 Feb, here is the result:
wiki
lang code
new captcha failure rate
old captcha failure rate
difference (%)
urwiki
ur
19.8%
38.1%
-18.3
hiwiki
hi
23.8%
40.7%
-16.9
fawiki
fa
17.4%
34.1%
-16.8
rowiki
ro
11.7%
27.7%
-16.0
bnwiki
bn
19.0%
33.8%
-14.7
arwiki
ar
22.3%
37.0%
-14.6
mywiki
my
17.6%
31.5%
-13.9
etwiki
et
5.5%
19.2%
-13.7
kawiki
ka
17.0%
30.3%
-13.3
mlwiki
ml
15.6%
28.8%
-13.2
kkwiki
kk
14.0%
26.8%
-12.9
mnwiki
mn
7.0%
19.6%
-12.6
ukwiki
uk
11.4%
23.8%
-12.5
metawiki
meta
28.5%
40.9%
-12.4
thwiki
th
15.3%
27.5%
-12.2
foundationwiki
foundation
56.4%
68.0%
-11.6
hewiki
he
19.8%
31.1%
-11.3
eswiki
es
18.8%
28.7%
-9.9
svwiki
sv
15.3%
25.0%
-9.7
cswiki
cs
9.7%
18.9%
-9.1
hawiki
ha
9.5%
18.6%
-9.1
viwiki
vi
15.6%
24.3%
-8.7
mediawikiwiki
media
19.0%
27.6%
-8.6
kowiki
ko
11.3%
19.9%
-8.6
trwiki
tr
13.7%
22.2%
-8.5
uzwiki
uz
19.4%
27.9%
-8.5
ruwiki
ru
15.0%
22.9%
-7.9
nowiki
no
10.7%
18.0%
-7.3
jawiki
ja
14.0%
21.1%
-7.2
enwikiversity
en
28.8%
35.9%
-7.1
ptwiki
pt
16.8%
23.8%
-6.9
srwiki
sr
11.6%
18.5%
-6.9
bgwiki
bg
13.8%
20.3%
-6.5
enwiktionary
en
30.5%
36.9%
-6.5
hywiki
hy
18.0%
24.4%
-6.4
enwiki
en
15.8%
22.2%
-6.4
sourceswiki
sources
41.5%
47.9%
-6.4
azwiki
az
16.3%
22.5%
-6.2
commonswiki
commons
18.2%
24.4%
-6.2
zhwiki
zh
15.3%
21.4%
-6.1
enwikibooks
en
28.2%
33.9%
-5.7
hrwiki
hr
14.3%
19.7%
-5.4
jawiktionary
ja
35.8%
41.2%
-5.4
frwiki
fr
18.1%
22.5%
-4.4
elwiki
el
13.1%
17.5%
-4.4
sqwiki
sq
17.1%
21.3%
-4.2
enwikivoyage
en
28.4%
32.3%
-4.0
cawiki
ca
11.4%
15.2%
-3.8
huwiki
hu
16.1%
19.6%
-3.5
mswiki
ms
17.6%
21.1%
-3.5
plwiki
pl
14.4%
17.4%
-3.0
simplewiki
simple
14.8%
17.8%
-3.0
itwiki
it
15.2%
18.0%
-2.8
dewiki
de
15.9%
17.9%
-2.0
dawiki
da
11.7%
13.6%
-2.0
skwiki
sk
13.2%
15.0%
-1.9
frwiktionary
fr
38.2%
39.3%
-1.1
fiwiki
fi
13.7%
13.0%
0.7
wikidatawiki
26.3%
25.5%
0.8
idwiktionary
id
36.9%
33.9%
3.0
eswiktionary
es
52.7%
49.5%
3.2
enwikiquote
en
39.4%
35.3%
4.1
frwikisource
fr
58.2%
53.6%
4.7
slwiki
sl
18.2%
13.4%
4.7
enwikinews
en
50.0%
44.5%
5.5
zh_yuewiki
zh_yue
11.1%
2.8%
8.4
nlwiki
nl
34.9%
26.1%
8.8
enwikisource
en
45.1%
35.6%
9.5
idwiki
id
32.9%
22.9%
10.0
jawikibooks
ja
68.0%
56.4%
11.6
ptwiktionary
pt
50.6%
38.9%
11.7
Difference for non-latin wikis is 9.3% redaction on average (it's not weighted). And for latin wikis it's 2.8% redaction in captcha failure rate (can be anything from non-native speakers or general improvement)
Given what we have, the two-tailed p-value for null hypothesis of latin or non-latin scripts having no impact on captcha failure rate is 0.000139 and as such is rejected. (Warning: My math/statistics is rusty)
Notes: I ignored multilingual wikis, considered sr and sh latin wikis and some other small stuff.
Regarding the impact on bots:
The slop on 80% and above has been cut to one third (from 0.03278 percent/captcha attempt to 0.00947 percent/captcha attempt) which is not what we expected, It can be many reasons, including that improvements on human captcha solving is masking the impact on the bots (as it's quite large) or the fact that there has been less captchas in total (bots not attempting?), etc. etc. I will need to investigate this more but any idea would be more than welcome.
Nikolay_Komarov
subscribed.
Feb 4 2024, 10:04 AM
2024-02-04 10:04:34 (UTC+0)
Comment Actions
I have some experience on the grey market, and I would assure you that these changes didn't make the captcha any stronger against machine OCR. There a lot of weaknesses remaining. Some of them are following.
The random noise lines should not be grey, otherwise they can be easily discarded by a simplest contrast improvement using any graphic library.
The operator of OCR can detect the font you use, then OCR crops every letter and matches it against existing library of letter images for this font type. The font type should be different for each letter to prevent that. The letter size also should vary.
In general, don't invent the bicycle, use open-source captcha solution with higher strength that are still easy to be solved by humans.
And when you do so, a part of the bot makers will simply forward these captchas to some real humans somewhere in India or China, they recognize 1000 captchas for 10 cents.
The real difference could only be made with Google reCaptcha or a non-evil alternative that uses IP-Address matching and browser fingerprinting to match the user against a blacklist.
Ladsgroup
added a comment.
Feb 4 2024, 3:10 PM
2024-02-04 15:10:06 (UTC+0)
Comment Actions
I understand and agree abut this is just about raising the bar a little to avoid captcha being broken with off-the-shelf OCR with default settings. No matter what we do, it'll be some way around it. It's just improving one layer of defense out of many. Further improvements on captchas are being talked about at the moment. We will keep the community updated on those.
Krinkle
attached a referenced file:
F41711785: grafik.png
(Show Details)
Feb 5 2024, 6:39 PM
2024-02-05 18:39:52 (UTC+0)
TheDJ
mentioned this in
T354099: FancyCaptcha isn't compatible with pillow 10.x
Feb 12 2024, 8:55 PM
2024-02-12 20:55:40 (UTC+0)
Maintenance_bot
edited projects, added
User-notice-archive
; removed
User-notice
Feb 22 2024, 9:31 PM
2024-02-22 21:31:05 (UTC+0)
Ladsgroup
mentioned this in
T152219: Statistics on Captcha success/failure rate
Aug 30 2024, 2:00 PM
2024-08-30 14:00:13 (UTC+0)
Log In to Comment
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct.
Wikimedia Foundation
Code of Conduct
Disclaimer
CC-BY-SA
GPL
Credits