Don't Understand Grading Calculation

General discussions about ratings.
Brian Towers
Posts: 1266
Joined: Tue Nov 18, 2014 7:23 pm

Don't Understand Grading Calculation

Post by Brian Towers » Mon Oct 07, 2019 12:54 pm

Last season we had a foreign student join our club with a FIDE rating that equated to about 115. Playing quick games against him we decided that he was probably a bit stronger than that and placed him in our teams based on our estimates. At the end of the season his results were 8 wins against opponents with the following grades - 132, 129, 121, 118, 125, 128, 144, 107. That looks to me to be an average grade of 125.5 which I would have thought would have resulted in a grade of 176. Instead his grade is 187. What aspect of the calculation have I failed to understand?

BTW, I did warn him that he would have an inflated grade with all wins but he went away over the summer rather than play congresses where he would have faced stronger opposition. Although we moved him up during the course of the season we were constrained by league grading difference limits and by the end of the season were perhaps exceeding them but not by enough for any other teams to lodge an official complaint.
Ah, but I was so much older then. I'm younger than that now.

User avatar
Christopher Kreuzer
Posts: 8782
Joined: Fri Aug 06, 2010 2:34 am
Location: London

Re: Don't Understand Grading Calculation

Post by Christopher Kreuzer » Mon Oct 07, 2019 1:00 pm

One of your player's games was against a junior (the one graded 118), that is what the asterisk (*) indicates. If you click through to that junior's grading page, their actual grade (calculated using the junior grading thingy method) is 124. I think this will account for some of the difference. OK, that is not enough to account for the difference. Maybe your player (as a new player) also has their grade calculated in a particular way (like for juniors).

I think what might be happening is that at least four of the players are "50 points below" the grade that the system is trying to assign to your player, so the performance there is recalculated. Take the initial estimate of 176 and redo the calculation with those four grades redone as "126" grades instead. And see what the result is.

132, 129, 121 [126], 118 [124 then 126], 125 [126], 128, 144, 107 [126].

New average: 180.

Then do this again:

132, 129 [130], 121 [126 then 130], 118 [124 then 126 then 130], 125 [126 then 130], 128 [130], 144, 107 [126 then 130].

New average: 182.

Then do this again:

132 [132], 129 [130 then 132], 121 [126 then 130 then 132], 118 [124 then 126 then 130 then 132], 125 [126 then 130 then 132], 128 [130 then 132], 144, 107 [126 then 130 then 132].

New average: 183.5 (round this to 184)

Basically, only the highest grade of 144 (bolded below) will not get bumped up this way.

The final iteration is:

132 [132 then 134], 129 [130 then 132 then 134], 121 [126 then 130 then 132 then 134], 118 [124 then 126 then 130 then 132 then 134], 125 [126 then 130 then 132 then 134], 128 [130 then 132 then 134], 144, 107 [126 then 130 then 132 then 134].

New average is 185.

Do it one more time (it wasn't quite the final iteration):

132 [132 then 134 then 135], 129 [130 then 132 then 134 then 135], 121 [126 then 130 then 132 then 134 then 135], 118 [124 then 126 then 130 then 132 then 134 then 135], 125 [126 then 130 then 132 then 134 then 135], 128 [130 then 132 then 134 then 135], 144, 107 [126 then 130 then 132 then 134 then 135].

New average is 186.

Maybe do it one more time?

New average is [(136*7 plus 144)/8]+50= 187

There we are! :D

Roger de Coverly
Posts: 21291
Joined: Tue Apr 15, 2008 2:51 pm

Re: Don't Understand Grading Calculation

Post by Roger de Coverly » Mon Oct 07, 2019 1:16 pm

Christopher Kreuzer wrote:
Mon Oct 07, 2019 1:00 pm

I think what might be happening is that at least four of the players are "50 points below" the grade that the system is trying to assign to your player, so the performance there is recalculated. Take the initial estimate of 176 and redo the calculation with those four grades redone as "126" grades instead. And see what the result is.
You always get something for winning, so the minimum grade is yours minus 40.

How strong is someone who chews up 120 players on a regular basis? The grading system struggles, but so do all rating systems?

User avatar
Christopher Kreuzer
Posts: 8782
Joined: Fri Aug 06, 2010 2:34 am
Location: London

Re: Don't Understand Grading Calculation

Post by Christopher Kreuzer » Mon Oct 07, 2019 1:16 pm

From the help page:

"If an opponent (or the player himself) is ungraded, a "starting grade" is estimated, using all available information. See Estimating a starting grade for an ungraded player below."
Estimating a starting Grade for an ungraded player (or a junior)
A Rapid grade, where available, will be used in default of a Standard grade; and vice versa. If the player has no grade at all, a starting grade is calculated as follows, using all their games in the latest three years (for adults) or one year (for juniors), inclusive of the current year.

Stage 1 is to calculate a 'grade' for each ungraded player on all their games against graded opponents in the relevant period. The 40-point rule is not used. If all their opponents are graded, it stops there and Stage 2 is omitted. The result will be used as their starting grade.

Stage 2 brings in games between the ungraded players. Once again the 40-point rule is not used. The players are 'graded' on all their games, using as starting grades the figures obtained from Stage 1. The resulting 'grades' will not be very accurate. So they are fed in again as new starting grades, and Stage 2 is repeated. This continues, with increasing accuracy each time, until the figures (more or less) stop changing. The starting grades can then be considered accurate, and they are used in the grading proper.

The effect of Stage 2 is that the points scored for a game against an ungraded opponent are repeatedly adjusted till the starting grades are right. But it is important to note that the adjustment applies only to games from the most recent (so not previously calculated) grading period. Games from earlier periods are included every time round, but always with the original points value. A previous-period game, once calculated, is never recalculated except for corrections.

User avatar
Christopher Kreuzer
Posts: 8782
Joined: Fri Aug 06, 2010 2:34 am
Location: London

Re: Don't Understand Grading Calculation

Post by Christopher Kreuzer » Mon Oct 07, 2019 1:20 pm

Roger de Coverly wrote:
Mon Oct 07, 2019 1:16 pm
Christopher Kreuzer wrote:
Mon Oct 07, 2019 1:00 pm

I think what might be happening is that at least four of the players are "50 points below" the grade that the system is trying to assign to your player, so the performance there is recalculated. Take the initial estimate of 176 and redo the calculation with those four grades redone as "126" grades instead. And see what the result is.
You always get something for winning, so the minimum grade is yours minus 40.

How strong is someone who chews up 120 players on a regular basis? The grading system struggles, but so do all rating systems?
Oops. I did my iterations doing the grades minus 50 not minus 40. Any iteration would have eventually converged on 187 I think. Is that what has happened here? The grading calculations guidance seems to indicate that this should NOT have happened here, as only one of his opponents was 'ungraded' or a 'junior'. Strange.

[I think my iteration would have eventually got to 194, i.e. 144+50.]
Last edited by Christopher Kreuzer on Mon Oct 07, 2019 1:25 pm, edited 1 time in total.

Roger de Coverly
Posts: 21291
Joined: Tue Apr 15, 2008 2:51 pm

Re: Don't Understand Grading Calculation

Post by Roger de Coverly » Mon Oct 07, 2019 1:23 pm

Christopher Kreuzer (Grading Help) wrote:
Mon Oct 07, 2019 1:16 pm

Estimating a starting Grade for an ungraded player (or a junior)

Stage 1 is to calculate a 'grade' for each ungraded player on all their games against graded opponents in the relevant period. The 40-point rule is not used. If all their opponents are graded, it stops there and Stage 2 is omitted. The result will be used as their starting grade.
Now there's an anomaly. If the player in question hadn't played a junior, the iteration would have stopped at the lower value. That may help to explain why new players and juniors doing extremely well or extremely badly seem to get grades that a player with an established grade and the same performance wouldn't.

Roger de Coverly
Posts: 21291
Joined: Tue Apr 15, 2008 2:51 pm

Re: Don't Understand Grading Calculation

Post by Roger de Coverly » Mon Oct 07, 2019 2:26 pm

Christopher Kreuzer wrote:
Mon Oct 07, 2019 1:20 pm

Oops. I did my iterations doing the grades minus 50 not minus 40. Any iteration would have eventually converged on 187 I think. Is that what has happened here? The grading calculations guidance seems to indicate that this should NOT have happened here, as only one of his opponents was 'ungraded' or a 'junior'.
I have this idea that at least part of the iteration uses minus 50 rather than minus 40. It may have to, otherwise you get a convergence to highest player plus 50.

From the grading Help.
Stage 2 brings in games between the ungraded players. Once again the 40-point rule is not used.

Kevin Thurlow
Posts: 5803
Joined: Wed Apr 30, 2008 12:28 pm

Re: Don't Understand Grading Calculation

Post by Kevin Thurlow » Mon Oct 07, 2019 5:49 pm

I think Chris was on the right track. I also think that all grading/rating systems struggle with 100 (or 0!) % scores. It's a "new" player and only 8 games as well. Many players on this forum would probably get 8/8 against that opposition but as we tend to have established grades the system can handle it without iterations of the results.

Brian Valentine
Posts: 574
Joined: Fri Apr 03, 2009 1:30 pm

Re: Don't Understand Grading Calculation

Post by Brian Valentine » Mon Oct 07, 2019 7:26 pm

The calculation here is fairly straightforward, but difficult to justify.

The iteration method will get to the average of opponent's grades plus 50; which as already pointed out gives 176. In the final iteration the 40 point limit is introduced so 7 of those results become 186 with the other being 194. The new average is 187 and published as the grade.

Angus French
Posts: 2149
Joined: Thu May 15, 2008 1:37 am
Contact:

Re: Don't Understand Grading Calculation

Post by Angus French » Mon Oct 07, 2019 7:47 pm

It's perhaps worth pointing out that between five and eight results with at least one of those in the most recent 12-month grading period gets a player an 'F' category grade. 'F' category grades are not used for future grading calculations (instead a new estimate is calculated) and are treated by some with caution (since the sample size used in the calculation is small and some of it may be dated).

User avatar
Chris Goodall
Posts: 1057
Joined: Sun Oct 10, 2010 6:40 pm
Contact:

Re: Don't Understand Grading Calculation

Post by Chris Goodall » Sun Oct 20, 2019 8:35 pm

Kevin Thurlow wrote:
Mon Oct 07, 2019 5:49 pm
I also think that all grading/rating systems struggle with 100 (or 0!) % scores.
They don't have to. It's a German Tank Problem. If you capture a single tank, and its serial number is 60, estimate how many tanks have been manufactured? Since you only have one data point, the best assumption you can make is that half of all the tanks have a lower serial number, and half have a higher. So 120 tanks. You assume that the tank you observed is the median tank, the same assumption as in the Doomsday Argument.

So if you have a single tank, which is that your new player lost to a player graded 200, what do you do? If you think you can't wring much information out of a sample size of one, consider a sample size of zero. If I meet a chap who tells me he's a competitive chess player, and I know none of his past results, my best estimate is that he's a perfectly average player graded 128. (Let's assume I'm new to the area and don't recognise all the top players already.)

If I then see him lose to a player graded 200, what happens to my estimate? It seems obvious to me that it should go down. I've learnt that he's not one of the small number of players that a 200 can't beat. So now when I assume he's an average player, he is the average of a set that's been chopped off at the top end, therefore its midpoint is slightly lower.

If I were the ECF, my estimate of his grade would jump up to 150! Not only that, but if he then played nine more games against 200 players and lost them all, the ECF would become more confident that my initial grade was an underestimate, because if he wasn't graded within 50 points of 200, why is he playing 200s all the time? We're literally deriving information about the player's strength from the arbitrary decisions of competition organisers.

My Normalised Open Source Online Clarke System is going to estimate new players' grades as if they were German tanks. (In fact, new players with grading codes starting 31 have a median starting grade of 103, unless their "club" is either a country or "4NCL [Club]", in which case it's 189. So that's a further refinement right there.)
Donate to Sabrina's fundraiser at https://gofund.me/aeae42c7 to support victims of sexual abuse in the chess world.

Northumberland webmaster, Jesmond CC something-or-other. Views mine. Definitely below the Goodall Line.

MartinCarpenter
Posts: 3041
Joined: Tue May 24, 2011 10:58 am

Re: Don't Understand Grading Calculation

Post by MartinCarpenter » Sun Oct 20, 2019 9:30 pm

Hummm, I'm not sure you know.

I think team captains tend to have a decent idea how strong players are, and that determines the strength of the field they play to a fair extent. Also people entering the 'right' part of Open tournaments.

Seems like information to me. Weak, but information. Not got anything else to use, and not making any claims about the actual usability of the grades resulting, so why not?

User avatar
Chris Goodall
Posts: 1057
Joined: Sun Oct 10, 2010 6:40 pm
Contact:

Re: Don't Understand Grading Calculation

Post by Chris Goodall » Sun Oct 20, 2019 11:09 pm

MartinCarpenter wrote:
Sun Oct 20, 2019 9:30 pm
Hummm, I'm not sure you know.

I think team captains tend to have a decent idea how strong players are, and that determines the strength of the field they play to a fair extent. Also people entering the 'right' part of Open tournaments.

Seems like information to me. Weak, but information. Not got anything else to use, and not making any claims about the actual usability of the grades resulting, so why not?
We do have something else to use. Guess 127 (not 128; miscalculated), you'll be righter than if you guess any other number. You can appreciate that guessing either 3.1415 or 900 is going to be very wrong, so somewhere in between those extremes must be an optimum value, right? Or if you know the player is new to the system and doesn't have a FIDE code from another country, revise your guess to 103. That's instantly more information than you can get out of one defeat to a 200, because almost everyone loses to 200s.

Look at it this way: if you find the set of people who'd probably score 0/1 against a 200, and assume that your player is bang in the middle of it, their grade drops from 127 to 123. 123 puts them at the 50th percentile within the set. If you instead assume that your player is graded 150, then they are at the 73rd percentile within the set. Of all the people who would probably lose to this 200, this particular one is better than 73% of the others, based on nothing other than the fact that they lost the game :?
Donate to Sabrina's fundraiser at https://gofund.me/aeae42c7 to support victims of sexual abuse in the chess world.

Northumberland webmaster, Jesmond CC something-or-other. Views mine. Definitely below the Goodall Line.

Joseph Conlon
Posts: 339
Joined: Thu Jun 06, 2019 4:18 pm

Re: Don't Understand Grading Calculation

Post by Joseph Conlon » Mon Oct 21, 2019 9:14 am

I don't agree; I think the flaw in the logic is that it does not regard as significant the fact that this unknown X has played a game against a player graded 200 (which is *already* information about their strength, even prior to the result).

i.e. the average grade among players who have played a graded game against someone graded 200 is significantly higher than the overall average of the chess population.

To take an even more extreme example: suppose one had to estimate the strength of player Y, given only that Y is not Norwegian and has played a classical game against Carlsen and lost. Although almost everyone would lose to Carlsen, the reasonable estimated strength of player Y should still be >2300.

User avatar
Chris Goodall
Posts: 1057
Joined: Sun Oct 10, 2010 6:40 pm
Contact:

Re: Don't Understand Grading Calculation

Post by Chris Goodall » Mon Oct 21, 2019 9:23 am

Joseph Conlon wrote:
Mon Oct 21, 2019 9:14 am
I don't agree; I think the flaw in the logic is that it does not regard as significant the fact that this unknown X has played a game against a player graded 200 (which is *already* information about their strength, even prior to the result).

i.e. the average grade among players who have played a graded game against someone graded 200 is significantly higher than the overall average of the chess population.
Sure, but the median grade of people who play against 200s is about 188, and the median grade of the players beaten by 200s is about 176, so why 150 and not 188 or 176?

Let's apply your logic from the other direction. A player walks into your club and claims to be from a country you were sure was still part of Russia. You put him on board 5 in Division 3, and he easily beats a couple of children graded 38 and 49. Do you therefore let him enter the Under 100 section of your congress, because the fact that he played against those children makes him a weaker player than if he hadn't?
Donate to Sabrina's fundraiser at https://gofund.me/aeae42c7 to support victims of sexual abuse in the chess world.

Northumberland webmaster, Jesmond CC something-or-other. Views mine. Definitely below the Goodall Line.

Post Reply