Variable Reward Schedules: The Math Behind Unpredictable Rewards

B.F. Skinner established that rewards delivered on an unpredictable schedule produce higher response rates and much greater resistance to extinction than rewards delivered on a fixed one. Wolfram Schultz later showed why at the neural level: dopamine responds to prediction error, spiking during the anticipation of an uncertain reward rather than the receipt of a guaranteed one.

The design consequence is the interesting part. A variable schedule and a fixed schedule with the same average cost you exactly the same amount, and one of them works considerably better.

The short answer

A variable ratio schedule paying out with probability p per action has a mean of 1/p actions per reward, identical to a fixed schedule at the same rate, but a variance of (1-p)/p squared. At p = 0.20 that is a mean of 5 actions and a standard deviation of 4.47. The unpredictability is free. What is not free is the tail: 3.52% of users will go 15 actions without a reward, and capping that tail with a guaranteed payout costs 3.6% more in rewards.

Fixed versus variable, same budget

Under a fixed ratio of 5, every fifth action pays. The user learns the pattern within two cycles and the schedule develops a characteristic post-reinforcement pause: having just been paid, there is no reason to act again immediately, because the next reward is known to be four actions away.

Under a variable ratio averaging 5, each action pays with probability 0.20 independently. The number of actions until the next reward follows a geometric distribution.

The costs are identical. The final row of the middle column is the whole mechanism: under a variable schedule the probability of a reward on the next action is 0.20 regardless of how many actions have just passed without one. The geometric distribution is memoryless. There is no point at which a user can reason that a reward is not worth waiting for, and equally no point at which they can reason one is due.

That memorylessness is why the schedule resists extinction. A user on a fixed schedule who stops being paid knows immediately that something changed. A user on a variable schedule cannot distinguish a broken system from an unlucky run.

The frustration tail

Memorylessness cuts both ways. The same property that prevents users from predicting a reward guarantees that some of them wait a long time.

A third of users experience a full mean-length drought at least once. Three and a half percent go 15 actions with nothing, which is three times the advertised average. In a large user base that is a meaningful population having a materially worse experience than the design intends, and they are not a rounding error, they are the people most likely to conclude the system is rigged.

The pity timer, and what it costs

The standard fix is a guaranteed reward after N actions, called a pity timer in game design. It truncates the tail. It also raises your reward cost, because some users who would have waited longer now get paid earlier.

def pity_cost(p, cap):
    """Expected actions per reward with a guaranteed payout at action `cap`."""
    e = sum(k * p * (1 - p) ** (k - 1) for k in range(1, cap))
    e += cap * (1 - p) ** (cap - 1)
    return e


p = 0.20
base = 1 / p
print(f"{'cap':>5} {'E[actions/reward]':>18} {'cost vs pure VR':>17} {'hit the cap':>13}")
for cap in (8, 10, 12, 15, 20, 25):
    e = pity_cost(p, cap)
    print(f"{cap:>5} {e:>18.3f} {base / e - 1:>16.1%} {(1 - p) ** (cap - 1):>13.2%}")

Output:

  cap  E[actions/reward]    cost vs pure VR   hit the cap
    8              4.161             20.2%        20.97%
   10              4.463             12.0%        13.42%
   12              4.656              7.4%         8.59%
   15              4.824              3.6%         4.40%
   20              4.942              1.2%         1.44%
   25              4.981              0.4%         0.47%

A cap at 15 actions costs 3.6% more in rewards and removes the experience of a 15-action drought entirely. That is cheap insurance against your worst-served users, and it is a better use of budget than raising the base rate, which costs far more and does nothing for the tail specifically.

A cap at 8 costs 20.2% more and starts to defeat the purpose, since a fifth of users now hit a predictable ceiling and the schedule drifts back toward fixed.

Where this actually applies

The honest scope matters here. Variable reward schedules are the mechanism behind slot machines, loot boxes and infinite feeds, and their power is precisely what makes them a regulatory and ethical concern. There is a real line between a schedule that makes a genuinely useful product more engaging and one that manufactures compulsion around a product with no underlying value.

The defensible applications share a property: the reward is a bonus on top of something the user already wanted to do. Surprise upgrades on a purchase they were making anyway. Occasional bonus content in a product they subscribe to. Randomised referral bonuses where the base offer already stands on its own.

The indefensible ones invert that: the uncertainty is the product, and removing it would leave nothing.

If you cannot describe why a user would still act with the variable element removed, you have not designed an incentive, you have designed a slot machine. That question is worth answering before the arithmetic, not after.

What to take from this

The unpredictability is free, so if a variable schedule is appropriate for your product there is no cost argument against it. Budget the same, design the variance deliberately.

The tail is not free, and it is where your worst experiences live. Compute it, cap it, and price the cap. At p = 0.20 a cap at 15 costs 3.6%.

State the mean honestly. Users who are told rewards come "about every 5 actions" and then wait 15 are not experiencing bad luck from their point of view, they are experiencing a broken promise. Publishing the actual distribution costs nothing and removes the most common complaint.