4NCL Online

[DavidWalker]

I don't think the number of tests would be halved because each test isn't independent.

Matthew,
You are correct about the lack of independence of each test. I recalled Dr Regan mentioning a similar issue in one of his blog posts, but I couldn't find the article, and anyway I don't think he gave any quantitative means to correct for this. I used the simplistic assumption of independence to give maximum protection to the honest player. To get a lower bound on the z-score, consider testing each distinct block of 5 games exactly once, giving a z-score of 4.1.

I would still be inclined to choose a value of 4.5, in line with Dr Regan's minimum recommendation in one of his articles:

I am proposing a 5.00-sigma standard in chess for statistical evidence to rise above the caveats of selection that go with Littlewood’s Law. This might be shaded to 4.75, giving exactly Littlewood’s quantification of "miracle," something we’d expect to see in chess only once in two decades. Or perhaps to 4.50, which my post "Thirteen Sigma" noted is the end-product criterion for "Six Sigma" in industry.

[Matthew Turner]

David,
That sounds like all good analysis, but by picking a z score of 4.5 what does that actually mean in practice? Let me try to give an example to put you in the 4NCL’s position. Imagine I send you all the games and all the z scores for everybody who has ever played in a 4NCL Congress. I highlight all the players that have ever been banned for cheating. Two questions then

1. What z score would you need to conclude that 4NCL was justified in banning the player
2. What z score would you require to conclude a player should have been banned who wasn’t

[JustinHorton]

Could we have a brief explanation of z-score for readers who are unaware of its meaning

[Matthew Turner]

The z score is a measure of how likely a player of a given rating is to play computer moves . A z score of 4 equates to someone performing 800 points above their rating. This should happen naturally 1 in every 31,000 occasions
A z score of 2 would be 400 points and one in every 44 occasions.
I hope this helps.

[Roger de Coverly]

This should happen naturally 1 in every 31,000 occasions

How many moves per game are tested? If it's 30 moves, that's only around one in a thousand games. I don't believe it's mathematically valid to treat moves in the same game as independent of one another. Indeed that's how phone cheats operate by only needing to consult their engine a handful of times a game.

[Matthew Turner]

Roger,
Of course you don’t test every move and if you match a computer move you get a high z score and get banned. A typical test might be over 100 useable moves (excluding theory, where one side has an overwhelming advantage and repetitions).

[Roger de Coverly]

Imagine I send you all the games and all the z scores for everybody who has ever played in a 4NCL Congress. I highlight all the players that have ever been banned for cheating.

As far as I am aware, there's only ever been one player banned for cheating in a 4NCL Congress. It never got to the stage of revealing either the nature of the cheating or how it was detected, but the player's actual results could have been flagged in that they scored near 100% in a rating restricted tournament. It was by a player whose prior record had not suggested was good enough to get those results. The ban and loss of prize money wasn't challenged, which is perhaps an implicit admission of guilt. There wasn't any action taken by the ECF, but presumably enough people knew who it was, as the player has self banned by withdrawing from graded chess.

[Kevin Thurlow]

"The z score is a measure of how likely a player of a given rating is to play computer moves . A z score of 4 equates to someone performing 800 points above their rating. This should happen naturally 1 in every 31,000 occasions
A z score of 2 would be 400 points and one in every 44 occasions.
I hope this helps."

A really good description. I know it's part of the job, but explaining something complicated and technical in plain English is rare!

[JustinHorton]

Could we clarify "occasions"? It may be unclear what constitutes an occasion.

[Matthew Turner]

Justin,
In general I think 5 or 6 games is about the right number to run a valid test. So if i play in a normal weekend congress, my performance would be one occasion. So if Blackpool congress has 400 players they should have one z score of 4 every 75 years or so.

[Roger Lancaster]

"The z score is a measure of how likely a player of a given rating is to play computer moves . A z score of 4 equates to someone performing 800 points above their rating. This should happen naturally 1 in every 31,000 occasions
A z score of 2 would be 400 points and one in every 44 occasions.
I hope this helps."

A really good description. I know it's part of the job, but explaining something complicated and technical in plain English is rare!

This has become a circular argument which I don't intend to re-enter except to point out, as has been mentioned previously, that the answer highlights that Lichess cannot legitimately arrive at a z-score of 4, broadly equivalent to performance 800 points above one's rating, when it does not know [except in a small minority of cases] what that rating is in the first place.

[Paul Cooksey]

Roger L, also those cases when the performance is 3400? Not all engine use is subtle.

Just to the other Roger's point on the number of available data points. I think Roger overestimates how close amateurs are to GMs and GMs to engines in their quality of play outside the opening. Playing some games against stockfish at knight odds shows you how your advantage diminishes on virtually every move.

[Roger Lancaster]

Roger L, also those cases when the performance is 3400? Not all engine use is subtle.

Just to the other Roger's point on the number of available data points. I think Roger overestimates how close amateurs are to GMs and GMs to engines in their quality of play outside the opening. Playing some games against stockfish at knight odds shows you how your advantage diminishes on virtually every move.

Paul - If performance is as high as 3400, that's another matter. I think we all accept that it's relatively easy to weed out obvious cheats. An intelligent cheat would, I would guess, aim for a much lower figure and so throw in some indifferent moves so that his performance wasn't overly conspicuous.

[Roger de Coverly]

So if Blackpool congress has 400 players they should have one z score of 4 every 75 years or so.

What's being measured though? If assumed that it's possible to estimate a grade or rating from the quality of moves actually played, what does it tell you that a player with a rating of 1100 played to a standard of 1900? External assistance is one hypothesis, but you could also run with Elo's theory that it tells you that the 1100 rating is likely to be wrong and the practical applications of Elo with k factors tell you how much to change it by.

[Matthew Turner]

Roger,
It is strong evidence of cheating isn’t it. The organisers have to decide how to use this information. At Blackpool I would want to investigate further, I might even withhold prize money (temporarily). If the player had a z score of 5 then I would be tempted to ban the player and take my chances in court if it came to it.