Forum Archive : Ratings

Effect of droppers on ratings

From:   Gary Wong
Address:   gary@cs.auckland.ac.nz
Date:   4 February 1998
Forum:   rec.games.backgammon
Google:   ieypvl4l0y2.fsf@cs20.cs.auckland.ac.nz

Chuck Bower writes:
> How important is a person's rating?  This applies to the 'dropper' AND
> the 'droppee'.

  Let me behave like a politician and start my answer by rewording your
question :-)  I think it can be broken down into:

 * How important is it that ratings are _accurate_?

 * If a rating is _inaccurate_, how much of a problem is it?

 * What value does a rating have to a person?  (would I be happier if my
   rating were 100 points more?  would my meaningless life be fulfilled if
   I reached 2000? etc.)

  The last question is probably the easiest to answer -- it depends on the
person, of course.  To a deliberate dropper, it's obviously very important
(a desire to be rated highly is among the ingredients in their recipe,
besides a lack of morals and having nothing more worthwhile to do).  To a
bot, it's not important in the slightest.  To the vast majority of

it's presumably somewhere in between.

  Answering the first question now: assuming ratings are important (and
there are many good reasons they might be, leaving aside vanity value for
the moment) then it seems clear that accurate ratings are superior.  It has
long since been observed that fluctuations are large -- expect swings up to
100 points, say.

  The last thing I want to address is the second question -- how much of a
problem is an inaccurate rating?  Given that a 100 point discrepancy can
occur from random variation alone, you might be tempted to argue that even
if somebody cheated 50 points off you, the difference is insignificant and
lost in the noise.  At the other extreme, you could claim that losing a
single point has permanently lowered your rating and you will never again
be rated as highly as you would be if you had won the point (which is also
true, modulo rounding error).  Overall I think a fairer answer is to
consider the "half-life" of a result -- assume you are cheated out of a
game and "lose" a few points (not scoring n points that you deserve is
essentially equivalent to having n points unfairly deducted from your
rating).  Your rating is now n points lower than it should be; but to
compensate, your future wins will be worth fractionally more (since you are
now underrated). How many games will you have to play before you are only
rated n/2 below your "true" rating?

  For simplicity, let's assume your experience is 400+ and you only play 1
point matches.  If an equally rated player drops on you on a game that you
would certainly win, then your rating is 2 points lower.  However, you'll
score slightly more in future (0.0023 points on average).  This quantity
will gradually reduce as you approach your "true" rating again, but as
a linear approximation let's say it will take somewhat over 200 games to
gain a point from it -- ie. the "half life" of a FIBS match result is of
the order of 200 experience.

>      While you're mulling over my "how important is ratings" question,
> here is another one for you.  What is the frequency of 'dropped' matches?
> I'm looking for a fraction or percentage--like "On average I'm dropped
> on once every N matches".  What is the value of 'N'?

  N will naturally vary depending on who you are, who you play, etc. etc.
But as an order-of-magnitude estimation, Abbott has been on FIBS for about
a week, plays one point matches only, has 1094 experience and 39 saved
games. It tries hard not to start new matches with players it has saved
games with, and to the best of my knowledge it has succeeded so far.  If we
assume half of the saved games are deliberate drops that will never be
resumed, then N is about 1094/19.5 -- let's call it 50.  For humans I
expect N to be somewhat higher, since I hope droppers are less likely to
cheat against a human; humans are able to apply a little more pressure in
convincing a dropper to resume a game; and humans are more likely to be
suspicious of starting a game with a dropper in the first place (would you
join somebody with an 1800 rating and 50 experience?).  So I'll claim N for
a human is about 100.

  So, what effect do droppers have on your rating overall?  If we take the
same 1-point player as above, and assume she lost 2 points 50 matches ago,
and another 2 150 matches, 250, 350 etc., how much lower will her rating
be?  The 2 points she lost 50 matches old will now be worth -1.68; 150 old
will be worth -1.19, etc. -- the sum to infinity of a geometric series with
a = -1.68, r = 1/sqrt(2).  This works out to be about -5.74.  There have
been several assumptions and estimations in this quantity so let's say our
guess is that the true value is between 2 and 10 and probably about 5.

  _Therefore, I'd estimate that everybody who is not a dropper is rated
typically 5 points lower than they ought to be._ Personally I don't think
this is large enough to be worried about.  Certainly not worth all the
heated accusations, or being put off FIBS over!  Do I dare suggest the
simplest fix for Patti and Marvin to make would be simply to start every
new player on FIBS with a rating of 1505.74 to compensate? ;-)

  And to shed yet more opinions, I think the most constructive suggestions
so far have been the "show saved <name>" extension (though you'd also need
an "oldboard <name1> <name2>" to distinguish between droppers and
droppees -- bots accumulate dozens of saved games a week, but they're not
droppers).  That and the Serenity Prayer :-)

Gary (GaryW on FIBS -- yep, the one whose rating is in a sorry state, and
  he doesn't care).

Gary Wong, Computer Science Department, University of Auckland, New Zealand
gary@cs.auckland.ac.nz        http://www.cs.auckland.ac.nz/~gary/

Tom Keith  writes:

Douglas Zare and Adam Stocks comment on this posting in their article
"Ratings: A Mathematical Study"
Did you find the information in this article useful?          

Do you have any comments you'd like to add?     



Constructing a ratings system  (Matti Rinta-Nikkola, Dec 1998) 
Converting to points-per-game  (David Montgomery, Aug 1998)  [Recommended reading]
Cube error rates  (Joe Russell+, July 2009)  [Long message]
Different length matches  (Jim Williams+, Oct 1998) 
Different length matches  (Tom Keith, May 1998)  [Recommended reading]
ELO system  (seeker, Nov 1995) 
Effect of droppers on ratings  (Gary Wong+, Feb 1998) 
Emperical analysis  (Gary Wong, Oct 1998) 
Error rates  (David Levy, July 2009) 
Experience required for accurate rating  (Jon Brown+, Nov 2002) 
FIBS rating distribution  (Gary Wong, Nov 2000) 
FIBS rating formula  (Patti Beadles, Dec 2003) 
FIBS vs. GamesGrid ratings  (Raccoon+, Mar 2006)  [GammOnLine forum]
Fastest way to improve your rating  (Backgammon Man+, May 2004) 
Field size and ratings spread  (Daniel Murphy+, June 2000)  [Long message]
Improving the rating system  (Matti Rinta-Nikkola, Nov 2000)  [Long message]
KG rating list  (Daniel Murphy, Feb 2006)  [GammOnLine forum]
KG rating list  (Tapio Palmroth, Oct 2002) 
MSN Zone ratings flaw  (Hank Youngerman, May 2004) 
No limit to ratings  (David desJardins+, Dec 1998) 
On different sites  (Bob Newell+, Apr 2004) 
Opponent's strength  (William Hill+, Apr 1998) 
Possible adjustments  (Christopher Yep+, Oct 1998) 
Rating versus error rate  (Douglas Zare, July 2006)  [GammOnLine forum]
Ratings and rankings  (Chuck Bower, Dec 1997)  [Long message]
Ratings and rankings  (Jim Wallace, Nov 1997) 
Ratings on Gamesgrid  (Gregg Cattanach, Dec 2001) 
Ratings variation  (Kevin Bastian+, Feb 1999) 
Ratings variation  (FLMaster39+, Aug 1997) 
Ratings variation  (Ed Rybak+, Sept 1994) 
Strange behavior with large rating difference  (Ron Karr, May 1996) 
Table of ratings changes  (Patti Beadles, Aug 1994) 
Table of win rates  (William C. Bitting, Aug 1995) 
Unbounded rating theorem  (David desJardins+, Dec 1998) 
What are rating points?  (Lou Poppler, Apr 1995) 
Why high ratings for one-point matches?  (David Montgomery, Sept 1995) 

[GammOnLine forum]  From GammOnLine       [Long message]  Long message       [Recommended reading]  Recommended reading       [Recent addition]  Recent addition

  Book Suggestions
Computer Dice
Cube Handling
Cube Handling in Races
Extreme Gammon
Fun and frustration
GNU Backgammon
Luck versus Skill
Magazines & E-zines
Match Archives
Match Equities
Match Play
Match Play at 2-away/2-away
Opening Rolls
Pip Counting
Play Sites
Probability and Statistics
Source Code
Strategy--Bearing Off
Strategy--Checker play


Return to:  Backgammon Galore : Forum Archive Main Page