4NCL Online

[Ian Thompson]

As a general question which partly relates to this - how easy would it actually be to play like a 2300, suppose a 1500 player determined to cheat? I can see that it could be quite easy to play at 3300, but 2300 seems much harder. I would imagine that the average of a 1500 player and a 3100 player is much closer to 1500; it only takes one really bad move to lose the game.

A related question would be what sort of results would you get with "intelligent cheating"? Say someone selected all moves themselves to start with and then checked them with a computer. They play the move they chose unless the computer evaluates it at more than X pawns worse than the best move. When this happens the player chooses the computer move.

I'd have thought that you could simulate the play of various standards of player by adjusting X. Maybe start off with X = 1, play some games, and then reduce it gradually and see how the results improve.

[DavidWalker]

Does a z-score of 4 really equate to an 800 point rating boost? I know that the ELO standard deviation is supposed to be 200, but this z-score is in a different domain (basically measuring actual moves matched against expected matches I believe). There is a draft paper by Dr Regan in which he gives rating estimates for players based on moves played in their games. Randomly checking a few examples from this paper shows a SD of the rating estimates lower than 200.

[MartinCarpenter]

Surely, You wouldn't do this though. Lets say you were willing to accept x as evidence of cheating, you'd set x+1 as the level for a ban, then the appellant would have to show that you'd made a substantial mistake to get a ban lifted.

I'd think/hope so yes. Of course, if you decide you 'want' Z to be around 4 to have the false positives under control then compensate significantly further you really are asking for a lot of evidence.

Might take more than one season of data.

[Matthew Turner]

The point of the margin is specifically to take into consideration these kind of fluctuations, so in my opinion, no!

OK so let try another example. Bob Bunny is a junior rated 1500. He has a z score of 4 in the JNCL over his first 6 games (i.e selecting computer moves at a rated associated with a 2300 players). His rating chart shows steady progress going up at 50 points a year of average. He says he has been playing a lot online and can point to a number of tournaments where he has performed at 1800-1900. His coach Mathos Lender has also written in to say he has been studying the openings that were played in his JNCL games and knows the ideas very well. Acquit, or let the ban stand?

As a general question which partly relates to this - how easy would it actually be to play like a 2300, suppose a 1500 player determined to cheat? I can see that it could be quite easy to play at 3300, but 2300 seems much harder. I would imagine that the average of a 1500 player and a 3100 player is much closer to 1500; it only takes one really bad move to lose the game.

Can one of the neural net engines (Leela etc) provide a 2300 that would look like a human 2300?

(where this is leading - if its very hard to cheat in a certain fashion, that should count in the player's favour)

Joe,
It is important to note that we are talking about matching a computer's moves at the same rate as a 2300 as distinct from playing at 2300. So Magnus Carlsen is better than me, not only because he plays more computer moves, but also because he is better than me at knowing when it is important to play a computer move.

So, it is perfectly conceivable that a 1500 could 'perform' at 2300 by selecting the computer move half the time rather than all the time. In reality though, I am not sure that this would persist for very long, they would either get bored and stop or be drawn into playing more computer moves.

[Matthew Turner]

Does a z-score of 4 really equate to an 800 point rating boost? I know that the ELO standard deviation is supposed to be 200, but this z-score is in a different domain (basically measuring actual moves matched against expected matches I believe). There is a draft paper by Dr Regan in which he gives rating estimates for players based on moves played in their games. Randomly checking a few examples from this paper shows a SD of the rating estimates lower than 200.

So, the 200 points per one of z is a rule of thumb, but generally it relates pretty accurately to the Regan tests that we are talking about. The difference with the numbers in the paper you quote are down to sample size.
So for example we have
Steinitz in the World Championship match 1886 Est. performance 2352 (2 sigma range) 2150–2553 No of moves 593 (that is from 20 games)

If we take an example from fewer games we have
Larsen Candidates 1971 Est. performance 2187 2 sigma range 1702 - 2602 No of moves 181 (that is from 6 games)

[David Sedgwick]

Let say we use 180 for your Regan test that equates to 2050. So to get a z score of 4 you have been selecting computer moves at a rate associated with a 2850 (basically Magnus Carlsen). Your defence is that you are actually a 200 graded player i.e 2200.
Is that sufficient grounds for a successful appeal (I am not offering an opinion either way)

What about a player rated 2200 who selects computer moves at a rate associated with a 2850? That is a Z score of 3.25.

I presume that that is not high enough for any action to be taken

If your 2050 player can demonstrate that his true strength in the relevant competition is 2200, then he should be treated as though his rating were 2200.

So the appeal should be allowed.

[Matthew Turner]

David,
How about I show that my grade was rounded down to 180 and was actually 180.4 and so 2053 should be used as my rating and consequently my z score should be 3.99 - is that still enough?
I am trying to demonstrate there are quite a few grey areas and people can legitimately have differing views. So there are (or should be) tough decisions for a appeals panel to make.

[David Sedgwick]

So there are (or should be) tough decisions for a appeals panel to make.

I completely agree with that.

How about I show that my grade was rounded down to 180 and was actually 180.4 and so 2053 should be used as my rating and consequently my z score should be 3.99 - is that still enough?

Probably not. De mimimis not curat lex. (The law is not concerned with trifles.)

[MartinCarpenter]

So there are (or should be) tough decisions for a appeals panel to make.

I completely agree with that.

How about I show that my grade was rounded down to 180 and was actually 180.4 and so 2053 should be used as my rating and consequently my z score should be 3.99 - is that still enough?

Probably not. De mimimis not curat lex. (The law is not concerned with trifles.)

Sensible Of course the normal distribution is really very sensitive at this point in its curve.

Z scores:
4 = 1/33,333
3.9 = 1/20,000
3.8 = 1/14,286
3.7 = 1/9090
3.6 = 1/6250
3.5 = 1/4347
3.4 = 1/2941
3.3 = 1/2083
3.2 = 1/1449
3.1 = 1/1030
3 = 1/741

A difference of even 0.2 Z (~40 fide/5 ECF) - beneath notice in grading terms for most purposes - moves the numbers a long way.

Of course Ken's statistics do include the natural variations in FIDE <=> underlying playing strength, so you'd have to be contending something beyond that.

But you can think of plausible things and.... Hard. Even in principle lower down - there's a lot of very low FIDE's in 3/3N, so I presume use ECF. Which, post all the recent & pending changes, really aren't as stable as you'd like for this sort of work. I imagine Z would end up quite a bit wider.

[JustinHorton]

Just out of interest - I don't know that much about what Ken Regan does, but do we know how accurate his system is, and if so, how do we know?

[MartinCarpenter]

Just out of interest - I don't know that much about what Ken Regan does, but do we know how accurate his system is, and if so, how do we know?

Plenty of reasons to trust. He's a sensible, knowledgable person and he's calibrated and tested his numbers against the vast extant database of FIDE rated games. The methodology is open, published, rational and I'm not aware of any holes being flagged up.

If you were feeling paranoid you might want a few people to fully replicate the work he put in - ideally using different methodologies - as a cross check. Otherwise as good as you'd get.

The nature of the work itself is that its always uncertain and tries to quantify how much. It should do that, subject to those sources of uncertainty being taken from a global database of games.

If, eg, your whole country has unusually inaccurate FIDE grades then the answers you get will be more confident than they strictly should be.

[JustinHorton]

Just out of interest - I don't know that much about what Ken Regan does, but do we know how accurate his system is, and if so, how do we know?

Plenty of reasons to trust. He's a sensible, knowledgable person

Oh, it's not this that I'm asking about, I take that to be the case. I was just unsure about the question of whether his results could be or had been replicated.

[Roger de Coverly]

I don't know that much about what Ken Regan does, but do we know how accurate his system is, and if so, how do we know?

There's an underlying assumption that it's possible to assess a player's strength by the quality of their moves. That contrasts with Elo and other rating systems that attempt to assess a player's strength from their results.

[Matthew Turner]

Just out of interest - I don't know that much about what Ken Regan does, but do we know how accurate his system is, and if so, how do we know?

Plenty of reasons to trust. He's a sensible, knowledgable person

Oh, it's not this that I'm asking about, I take that to be the case. I was just unsure about the question of whether his results could be or had been replicated.

I am not sure exactly how you want this answered, but Regan has published various reports on how his system works. Could this be replicated, well in a sense isn’t this exactly what Lichess and Chess.com are doing? (In terms of the underlying philosophy at least). You can see that here
https://github.com/clarkerubber/irwin

[MartinCarpenter]

Sort of, except you'd then need to use whatever LiChess/Chess.com did to a strict replication study on a controlled data set. This is a very strict standard indeed though.