English Chess Forum

Roger:

To me I think arguments over whether the distribution of 'B' or 'C' are precisely normal are not relevant. The question is what is the chance of objective online performance lying in the tail of the distribution (z>3 here).

Unless one can think of a reason why objective online performance should be better than objective OTB performance (and all arguments I can think of go the other way, e.g. shorter time controls) then it doesn't matter about the nature of the distribution, given that 'A' is a data-driven distribution, and so the probabilities in B to fall in the extreme regions should all be strictly less than A.

Thanks Roger.

Joseph - I believe many psychologists say better performance under pressure is a myth. Maybe at home in familiar surroundings with creature comforts and without the minor stresses of a venue, I am under less pressure and my chess is better. Maybe, because I do most of my study on a screen, I play objectively better.

I am not sure how statistically relevant the advantages of playing from home are. But I can at least make an argument they exist. Which I think gets us back to how we interpret the data.

Roger Lancaster wrote: ↑

Sun Aug 16, 2020 2:43 am

John McKenna wrote: ↑

Sat Aug 15, 2020 10:59 pm

As I said, I find it hard to believe the people responsible for the underlying statistical treatments of the data available to them - both in the 4NCL and the wider online & otb chess world - have somehow got it basically wrong.

John, I don't want to dwell on this forever but my comments yesterday related to one specific graph which was the work of a single human or small group of humans rather than endorsed by "the wider online & otb chess world".

Roger, the author of the 4NCL Fair Play: Anti-Cheating Online article included this in it -

We use Professor Ken Regan’s model, which has existed in various forms since the bitter Topalov-Kramnik World Championship Match and has been used in FIDE and national cases since 2011...

That is what I say is the general acceptance of the methodology & s/w and is what you are ignoring when you say "was the work of a single human or small group of humans rather than endorsed by the wider online & otb chess world".

The article also contains -

If a website flags or bans a player for using computer engine assistance in 4NCL Online, I inform the captain of the player’s z score, and the MMP/ASD characteristics I described earlier. There have only been two confessions out of about 30 cases at the time of writing.

The usual response I receive is that they have independently reviewed them and said that there isn’t a shred of evidence to support the allegation. The resulting implication is that the website is wrong, and Professor Regan’s model (which seems to have otherwise worked well for 10 years) is wrong, and then a generally dissatisfied-with-everything article is published somewhere...

I agree with what Joseph Conlon posted, above, about your attempt to reinterpert the graph based on statistical assumptions of your own. It would be best if you could discuss that with the author of the article at some time and inform the forum of the outcome. (However, I believe the author is heavily engaged in the FIDE Online Olympiad at present and therefore unlikely to be available for some time.)

While you, and Paul Cooksey, are right to point out factors that will make the statistics of online performances differ from those of over-the-board ones I do not think they in any significant way account for the exceptional online engine-matching performances that the graph highlights and which rarely, if ever, exist over-the-board because the use of engine assistance is so much more difficult to get away with in the real - as opposed to the virtual - world.

On a slightly separate note, the Junior 4NCL gets underway on Thursday, with player lists to be submitted by (I presume) Wednesday.

Will the fixtures be published tomorrow so?

Thanks to Joseph and Paul for their comments earlier. As to the respective qualities of play [measuring online, sans cheating, against over-the-board] my gut feel tends to be with Joseph although I have no evidence to support this - except that there's a study by three Dutch academics [one an IM] of comparable academic seniority to Ken Regan suggesting that strong chess players play, on average, 200 rating points higher over the board. See "Cognitive Performance in the Home Office - Evidence from Professional Chess [2020]", Maastricht university.

I'll just quote from an abstract: "We use the Artificial Intelligence embodied in a powerful chess engine to assess the quality of chess moves and associated errors. Using within-player comparisons, we find a statistically and economically significant decrease in performance when competing online compared to competing offline". Perhaps someone here more familiar with Ken Regan's approach can tell me whether he agrees with these, presumably equally respected, professors?

Of course, I would point out that this study was conducted with strong chess players only. No inference is invited regarding weaker players.

The other point I should like to put gently to Joseph is this. The 4NCL graph sets out to show actual performance relative to expected performance. If, as Joseph and many others appear to believe, average online performance tends to be worse than over-the-board performance, why are both shown on the graph with the same mean performance?

Thanks also to John. My problem is the widespread belief that Ken Regan's methodology was handed down to Moses on Mount Sinai and must therefore - something that, in fairness, Ken is at pains to deny - be correct in every last detail. That's the same mindset as the Post Office directors who decided to prosecute entirely innocent postmasters, many of whom were convicted and imprisoned, on the basis of 'infallible' expert computer evidence which in hindsight proved only too fallible. There's no lack of other examples where experts, or 'experts', have been proved wrong.

That isn't intended to pass judgment on Ken Regan's methodology, which personally I regard as "best of type", but to draw attention to the fact that - in academic and scientific circles - acceptance of a finding or theory isn't widespread until it has been reviewed by the author's peers. As far as I am aware, that hasn't happened here. In fact, I'm not even sure that Ken's findings are consistent with those of the three academics mentioned earlier. As to FIDE's apparent endorsement, before commenting on this I rather think I'd like to know exactly who within FIDE has the technical/statistical expertise to pass judgment one way or the other.

John makes the entirely legitimate point that I might usefully have discussed the matter with Alex. In fact, we fleetingly discussed the subject several weeks ago, when I indicated to Alex I wasn't entirely happy, but subsequently life has intervened - you won't need me to tell you that Alex is busy while, for my part, when not running a small internet business, I'm regularly in the present situation of needing to get together six junior teams before Thursday. So the discussion never got going.

Roger Lancaster wrote
" As to the respective qualities of play [measuring online, sans cheating, against over-the-board] my gut feel tends to be with Joseph"

So the evidence that you presented as showing the 4NCL exaggerated cheating, your gut feeling now suggests underestimates cheating.

In general, I think this is correct, however we need to address the point that Paul makes. There will certainly be some players who perform better at home, because they feel more comfortable etc. etc. However, I think, in general, it is unlikely that these players will massively outperform at home, because there will be compensating factors. For example, older players might benefit from not traveling, but then it will be less likely that they will be rapidly improving.
I suggest it will only be relatively extreme examples where the benefits of playing at home will have a significant impact on the Z score. For example, if a player had been involved in a car accident then the unease of getting to the 4NCL could have a significant impact on their standard of play. I am not sure that any automated fair play system can take account of all these disparate factors, which is why it is so important that an appeals system is in place to take account of individual circumstances.

Roger Lancaster wrote: ↑

Sun Aug 16, 2020 1:25 pm

The other point I should like to put gently to Joseph is this. The 4NCL graph sets out to show actual performance relative to expected performance. If, as Joseph and many others appear to believe, average online performance tends to be worse than over-the-board performance, why are both shown on the graph with the same mean performance?

The graphs are that way because they simply used over the board FIDE ratings as the expected underlying playing strength for both sets of results. Not a lot else they could have done to be honest.

If the 4NCL online runs for a few years they'll know what the right adjustment factor would be to use.

The 'true mean' of the online 4NCL results does seem to be a good 0.5-1 SD over to the left side, indicating that the population does play a bit worse. A few outliers might obviously still hold steady/improve online.

Matthew Turner wrote: ↑

Sun Aug 16, 2020 3:06 pm

which is why it is so important that an appeals system is in place to take account of individual circumstances.

With the use of chess.com and lichess servers, that's not really happening. You have bans and flags imposed by the server using undisclosed methods and those imposed by the organisers such as the 4NCL or the ECF which utilise the Regan black box.

Matthew Turner wrote: ↑

Sun Aug 16, 2020 3:06 pm

Roger Lancaster wrote
" As to the respective qualities of play [measuring online, sans cheating, against over-the-board] my gut feel tends to be with Joseph"

So the evidence that you presented as showing the 4NCL exaggerated cheating, your gut feeling now suggests underestimates cheating.

Matt, you've asked lots of questions so suppose I ask you a question for a change. Bearing in mind that Alex's graph appears to give identical means for online and OTB performance, do you believe that average online and OTB performance is indeed equal?

Roger,
I believe the online performance will be lower, so the 4NCL underestimates cheating. I believe that is right and proper and gives more protection to players from wrongful accusations.

Matthew Turner wrote: ↑

Sun Aug 16, 2020 4:00 pm

Roger,
I believe the online performance will be lower, so the 4NCL underestimates cheating. I believe that is right and proper and gives more protection to players from wrongful accusations.

Excellent, Matt, we're making progress. Now, as regards the OTB graph, I hope you'll agree that the data reflects the differences between actual OTB performances and expected [ratings-based] OTB performances - with wider differences resulting in higher z values. For the online graph, the data reflects the differences between actual online performances and exactly what, in your opinion?

The graph reflects the difference between actual online performance and expected OTB performance. If you changed the graph to reflect the difference between actual online performance and expected online performance there would be more high Z scores and more players banned for cheating. Is that the change you are wanting to see enacted?

LawrenceCooper wrote: ↑

Thu Aug 13, 2020 9:19 am

LawrenceCooper wrote: ↑

Wed Aug 12, 2020 10:29 am

215 teams registered for season 2: http://www.4ncl.co.uk/data/online_teams_2.htm

Now up to 218 so entries still being accepted? http://www.4ncl.co.uk/data/online_teams_2.htm

A further increase to 221 teams.

Matthew Turner wrote: ↑

Sun Aug 16, 2020 10:26 pm

The graph reflects the difference between actual online performance and expected OTB performance.

Except, Matt, that you answered my earlier question, "Do you believe that average online and OTB performance is indeed equal?" by saying "I believe the online performance will be lower". Let's assume that you believe it to be, on average, X rating points lower [although the same point would apply if you believed it was Y points higher] then the first X points [or Y points] difference between actual online performance and expected OTB performance has nothing whatever to do with cheating. It's a direct consequence of overall online performance being lower/higher.

Since we already seem to have established that no-one knows, with any accuracy, what X or Y might be [and, since you say "I believe", I take it that you're not even sure which applies, never mind the actual figure] then we can't eliminate it to establish the residual effect of cheating. A statistically useful graph would reflect the difference between actual online performance and expected online performance rather than expected OTB performance. In practice, I understand why 'expected online performance' wasn't used but its omission undermines the statistics.