4NCL Online

[JustinHorton]

. An intelligent cheat would, I would guess, aim for a much lower figure

I am guessing that this is also much easier to do if

(a) you are online rather than OTB
(b) you are an experienced club and tournament player rather than an online rabbit.

[Roger de Coverly]

It is strong evidence of cheating isn’t it.

Are you claiming that a 1100 player playing to a 1900 standard is evidence of cheating? The arbiters might want to keep a watchful eye, but in the absence of physical evidence it's just the case of improving from "well below average" to "a bit above average" or even that the rating is absolutely and completely incorrect.

[Matthew Turner]

Roger,
I am not claiming it, it is blindingly obvious. If I have 2 1100 players, one of whom is playing at 1900 they are more likely to be cheating than the one who isn’t.

[Roger de Coverly]

If I have 2 1100 players, one of whom is playing at 1900 they are more likely to be cheating than the one who isn’t.

Are you claiming that there's a significant probability that the one playing at 1900 is cheating?

Underpinning the Elo theory of ratings is the notion that rating is an estimate of strength and if there's evidence the estimate is wrong, you change it.

[Matthew Turner]

Roger,
So there are lots of caveats to the answer, but that is not me ducking the question, it is just there are lots of caveats!

If
1. I knew the initial rating was basically accurate
2. It was over enough games/moves
3. I had a sufficiently good way of measuring their performance

Then Yes.

[DavidWalker]

...
1. What z score would you need to conclude that 4NCL was justified in banning the player
2. What z score would you require to conclude a player should have been banned who wasn’t

Matthew,
In response to yesterday's post.

1. 4NCL congresses are OTB events, so any z-score would just be a hint to the arbiter to keep an eye out for suspicious behaviour. Anything north of 1,000-1 odds (z ~ 3.1) would seem reasonable for an arbiter to investigate further and if evidence of cheating is discovered then a ban (subject to appeal) would seem appropriate.

2. I wouldn't suggest banning players from OTB chess based on z-score alone, however if the scores were from an online competition...
I would still attempt to set a minimum z-score of 4.5, but this doesn't have to be from a single event, the idea being that someone who has cheated once will continue to do so. For example, I believe that two consecutive z-scores of 3.5 equates to a z-score around 5. It should also be possible to combine scores from two non-consecutive events, but in this case an allowance would need to be made for the number of possible pairs of scores for a given player.

Having the full set of z-scores, it should be possible to estimate how many players are cheating, even if the score for any individual player is not high enough to accuse them. With this information, the effectiveness of the two-event testing procedure described above could be evaluated. If it still leaves too many players cheating, then consider lowering the 4.5 value.

Any ban should still be subject to a transparent appeal procedure.

[MartinCarpenter]

The above seems very sensible indeed.

To try and explain the confusion I caused a couple of days back:
In many ways its a data problem. It we had a big database of serious ~1 hour, online, games we'd use that, generate grades, configure the statistics and the numbers would work directly. We obviously don't.

We're having to estimate peoples '4NCL online' grades, and that's a bit risky - If someone's underlying strength is 200pts (~1 sigma) above that derived from their current FIDE grade then they trigger a 4 sigma test 40 times more often. A small underrated population will produce a lot of false positives.

Now, if Ken is actually assuming everyone has 4+ hours/game in 4NCL online then nearly everyone is underperforming by default and the probabilities arising from the tests are really very conservative.

If Ken has a FIDE grade <-> move quality mapping for 1 hour (online?) games, then some people will play much nearer to their 4 hour FTF strength than expected and you'll have a good number of 'extra' false positives arising. You'd need data from multiple seasons to quantify this.

It does occur to me that, purely from a statistical point of view, the lower divisions are especially difficult. The FIDE ratings obviously stop being remotely reliable, and aren't ECF grades are much more volatile than FIDE? Especially with the intended move to monthly grading lists. That'll subtly mess the numbers up.

Anyway, mostly just reasons to tread carefully. Its all solid within an order of magnitude and probably rather better than that.

[Matthew Turner]

David,
That is a great answer and I think I would agre3 with all of that. It is pretty much how the OTB 4NCL and similar events operate. However, I am not sure that you’ve completely answered the first question though. The problem I am posing you is based on incomplete information, so let’s say I find out a player has a z score of 3.1 and I search his bag and find a phone and I ban him. When you see the z scores you won’t know about the phone, so would you be happy to trust me and say that based on a score of 3.1 I was justified? Or would you need a higher z score.
What I am trying to do is put you in the same situation that the 4NCL finds itself.
Matt

[DavidWalker]

Personally I would not be in favour of banning from OTB based on z-score alone. However, if z-score alone were used to initiate a ban, I suggest it should be at least as high as the 4.5 I mentioned for online events. In this case there is an additional issue to consider related to the assumed relative scarcity of computer cheating in OTB events.

The issue is described in the article by Dr Regan which Keith Arkell mentioned earlier in this thread. To paraphrase: if 1 in 10,000 OTB players cheat and we expect a false positive rate of 1 in 30,000 (z~4), then for every 3 cheats, we could expect to sanction 1 honest player. This means that if all sanctioned players appeal, any appeal panel could expect to observe a 25% rate of honest players.

[Roger de Coverly]

This means that if all sanctioned players appeal, any appeal panel could expect to observe a 25% rate of honest players.

Would any OTB chess organisation want to run the risk of sanctioning without any physical evidence or even the hint of such evidence? Particularly for lower rated players, when someone plays to a standard several hundred points above their published rating the probability that the rating is incorrect or obsolete would be reasonably high.

But a fraud, someone who couldn't play chess to any respectable standard, ought to be physically detectable by their need for assistance at every move.

Perhaps a reverse applies. What are the probabilities of performing several hundred points or perhaps more than a thousand below a presumed rating? That's where the rating was established with undetected engine usage and the player for whatever reason was playing without their usual assistance.

[Matthew Turner]

David,
You still haven't addressed the question that I posed about what it would take for you to support a ban, rather than initiate one.

To respond to your comments. An OTB event could use a z score of 4 without collaborating evidence, you would just need a robust appeals process and recognize that there will be a significant proportion of successful appeals. I would suggest in your example 50% or more, since some cheaters would know they were hopeless cases and not bother appealing and a functioning appeals process should let off some cheaters, because whilst still guilty they had managed to significantly undermine the evidence.
So Roger is right why would an OTB event set the z score at 4, you may as well set it at 5 or 5.5. If a player gets 4.5 you can always do your civil duty and send the evidence to the next event and then it is their problem.
For online events setting the z score at 4 (or the equaivalent) isn't really an option because the sheer weight of numbers involved would quickly overwhelm the system. Out of necessity they will have to set the bar higher.

[NickFaulks]

Personally I would not be in favour of banning from OTB based on z-score alone.

So why is that different in online chess? There is a strand of thought within FIDE that online bans might be seen as more of a technicality, with no resulting stain on the player's character, but I just don't see how that will work.

[MartinCarpenter]

To respond to your comments. An OTB event could use a z score of 4 without collaborating evidence, you would just need a robust appeals process and recognize that there will be a significant proportion of successful appeals.

More than a 'significant proportion' I fear. All anyone appealing needs to do is point out that there's a 1/4 chance of a false conviction from your only evidence and you'd surely have to acquit them?

Unless you had additional data of course.

[MartinCarpenter]

Personally I would not be in favour of banning from OTB based on z-score alone.

So why is that different in online chess? There is a strand of thought within FIDE that online bans might be seen as more of a technicality, with no resulting stain on the player's character, but I just don't see how that will work.

That last would be convenient but yes

Differences in online chess:
1) Far more cheating in the overall population,
2) It is near impossible to gather useful secondary evidence unless they're inept.

That all makes using purely statistics much more relatively attractive.

[Matthew Turner]

To respond to your comments. An OTB event could use a z score of 4 without collaborating evidence, you would just need a robust appeals process and recognize that there will be a significant proportion of successful appeals.

More than a 'significant proportion' I fear. All anyone appealing needs to do is point out that there's a 1/4 chance of a false conviction from your only evidence and you'd surely have to acquit them?

Unless you had additional data of course.

Martin,
I do feel that my posting addressed this pretty fully so I am not sure why you have chosen just to quote this particular section