Scoring model#

Everything the trainer decides — which cases sit at the top of a list, which three end up in tonight's drill, how far each spoke of the spider graph reaches, whether the projected solve time meets the goal — comes from one module: app/services/scoring.py. There is exactly one definition of "weak", so the dashboard and the drill generator can never disagree.

Read this file before changing a constant. Every number below was chosen for a reason, and several of them are load-bearing for properties the tests enforce.

Times are milliseconds throughout, written plainly (4000 ms, 1.4 s).

The two sources of truth#

A case has at most two kinds of evidence:

A claim is cheap and available immediately; a measurement is expensive and arrives slowly. The model starts with the claim, and hands over to the clock as measurements accumulate.

Three states matter, and they are genuinely different:

• unknown — no claim, and not a single attempt. score is None. Sorts first: unknown work is the most valuable work, because you cannot know how bad it is. Note what this is not: twenty failed attempts are evidence, and such a case scores low rather than reading as unknown.
• claimed — a self-assessment, with too few attempts to trust. Capped at 80.
• measured — enough successful attempts. The clock dominates.

The formula#

1. The target time for one case#

targets.splits[goal] in config.yml gives per-phase seconds for a whole solve. A single case is a fraction of that: F2L is four pairs, OLL and PLL happen once each.

case_target_ms = splits[goal][phase] * 1000 / CASE_UNITS_PER_SOLVE[phase]
CASE_UNITS_PER_SOLVE = {cross: 1, f2l: 4, oll: 1, pll: 1}

For the shipped sub_20 goal: OLL 4000 ms, PLL 4000 ms, one F2L pair 9500 / 4 = 2375 ms.

The goal key falls back gracefully: the profile's goal_key → targets.default_goal → the first goal in the file → the built-in FALLBACK_SPLITS. A hand-edited config.yml can name a goal that does not exist without taking the app down, and individual junk values (a string, a negative number) are repaired per phase.

2. The measured score#

Let r = avg_ms / case_target_ms.

r <= 0.5      ->  100                        mastery plateau
0.5 < r <= 1  ->  100 - 40 * (r - 0.5)       linear, hits 80 exactly at target
r > 1         ->  80 / r                     hyperbolic decay, floor 0

The piecewise function is continuous at both joins (at r = 0.5 and r = 1) and strictly non-increasing in r. Some values:

Why hyperbolic rather than linear-to-zero: a linear tail hits zero at some arbitrary multiple of the target and then stops rewarding improvement, which is exactly wrong for the student who most needs encouragement. 80 / r means halving your time always doubles your score, at every speed.

3. The average that goes into it#

avg_ms is a recency-weighted, winsorised mean of the case's successful attempts (scoring.trimmed_average_ms):

w_i    = max(RECENCY_FLOOR, 0.5 ** (age_days_i / 30))
v_i    = min(ms_i, 2.0 * best_recent_ms)
avg_ms = sum(w_i * v_i) / sum(w_i)      over the 50 most recent attempts

• Recency window. Influence halves every 30 days and stops decaying after 90 days (3 half-lives), where the weight has reached its floor of 0.125. So a fresh attempt counts 8x an ancient one, but ancient evidence never vanishes entirely — a case you nailed a year ago and never touched again is not suddenly unknown. A row dated more than a day in the future is a broken clock rather than the freshest evidence there is, and is demoted to the same floor.
• The trim. An attempt slower than 2x your best recent time is clipped to 2x rather than dropped. A dropped phone, a mis-scramble or a phone call mid-rep cannot dominate the average.

Clipping rather than rank-dropping is deliberate. Dropping the slowest k samples and then weighting the survivors by recency is not monotone: a slow-but-recent attempt getting faster can pull a heavy weight into the counting set and raise the average. Winsorising has no such hole — every term of the sum is non-increasing in every input, so the whole model satisfies

making any attempt faster can never lower a score

which tests/test_scoring.py checks both on the primitives and end to end.

The 50-attempt window is not applied in Python after the fact: it is a ROW_NUMBER() window function in the query, so the rows that cannot matter are never fetched. See Cost.

4. The blend#

self_points = min(self_score / 5 * 100, 80)                or None
measured    = measured_score(avg_ms, case_target_ms)       or None
prior       = self_points if assessed else NEUTRAL_PRIOR_SCORE
blended     = w * measured + (1 - w) * prior               (prior alone if unmeasured)
reliability = 1 - 0.8 * dnf_attempts / total_attempts
score       = blended * reliability

with

w(n) = 0.7 * n / min_attempts                              for n < min_attempts
w(n) = 0.7 + 0.3 * (n - min) / (full_trust(min) - min)     for min <= n <= full
w(n) = 1.0                                                 for n >= full_trust(min)

full_trust(min) = max(12, 4 * min)

min_attempts is targets.min_attempts_for_measured (3 by default), and n counts only successful attempts.

• The 80 cap on a claim is the whole point of the assessment being cheap: 80 is precisely "at target speed", so a student who rates everything a 5 is ranked exactly at, and never above, someone who has proved target speed. A consequence worth knowing: self-ratings 4 and 5 both score 80. They are separated by measurement, not by conviction. (They do differ in the projected solve time — see below.)
• The prior is never zero. With no self-assessment, thin measured evidence is shrunk towards NEUTRAL_PRIOR_SCORE, which is 25 — and that number is derived rather than invented: it is measured_score of a case solved at UNKNOWN_TIME_FACTOR (3.2x) the target, i.e. exactly what the time model already assumes about a case nobody has looked at. Shrinking towards zero instead scored a demonstrated 1.4 s T-perm at 23.3, below the 40 of a case the student merely claims to be slow at — so the drill picker handed reps to the fast case and skipped the slow one. The prior is the state of ignorance, and ignorance is 25, not 0.
• 0.7 at the threshold makes measured evidence dominate the moment it is trusted, as the build contract requires, without the score jumping discontinuously on the third attempt. w is continuous and monotone in the attempt count for every configured minimum — that is what full_trust(min) = max(12, 4 * min) is for. Hard-coding "full trust at 12" gave a coach who set min_attempts_for_measured: 20 a 0.335 cliff at n=20.
• Only successful attempts raise w. A DNF must never buy trust.
• Reliability scales the finished blend, not just the measured part. This matters more than it looks: with the multiplier applied only to measured, a case that is self-rated 5 and then DNFed twenty times in a row kept its full 80 and displayed as "solid", because with no successful attempts there was no measured term to penalise. Scaling the blend puts it at 80 x 0.2 = 16, which reads as "learning" and sorts near the bottom, where a case you have never once completed belongs.
• DNFs are counted, and they hurt. At DNF_PENALTY_WEIGHT = 0.8, a 50% DNF rate costs 40% of the score and an all-DNF case keeps a fifth of it. Adding a DNF always lowers the score and never raises it.
• No self-assessment, few attempts. The score sits between the measurement and 25, so two quick attempts give a provisional score: high enough to leave the unknown bucket, low enough to stay on the drill list until the reps back it up.

5. Status bands#

Worked example#

A sub-20 student, OLL 21. Target for one OLL case: 4000 ms.

Monday. They rate it a 3 in the assessment wizard, nothing timed.

self_points = 3 / 5 * 100 = 60
w           = 0            (no attempts)
score       = 60           -> "known"

Tuesday. Three timed reps: 5200, 4800, 6400 ms. All today, so every recency weight is 1.0. The best is 4800, so the clip ceiling is 9600 and nothing is clipped.

avg_ms   = (5200 + 4800 + 6400) / 3 = 5466.7
r        = 5466.7 / 4000 = 1.367
measured = 80 / 1.367 = 58.5
w        = 0.7                        (3 successful attempts, min = 3)
score    = 0.7 * 58.5 + 0.3 * 60 = 59.0    -> "known"

The claim of 60 was roughly honest, so the score barely moves — which is the point of blending rather than switching.

Three weeks later. Fifteen successful reps now average 3640 ms, and one attempt along the way was a DNF.

r           = 0.91
measured    = 100 - 40 * (0.91 - 0.5) = 83.6
w           = 1.0                    (15 successful attempts >= 12)
blended     = 83.6                   (the claim has dropped out entirely)
reliability = 1 - 0.8 * (1 / 16) = 0.95
score       = 83.6 * 0.95 = 79.4     -> "known", just short of "solid"

One DNF in sixteen cost 4.2 points and kept the case out of the solid band. That is the intended sting: a case you drop one time in sixteen is not yet something you can rely on mid-solve.

Drill selection#

weakest_cases(profile, limit, phases, include_starred, require_algorithm, respect_focus) sorts on

(0 if needs_discovery else 1, score - star_bonus, case.sort_order, case.id)

• Discovery first. A case with no evidence at all, and a case rated 0 ("Never seen it") that has never been attempted, share rank 0. Those two are the same information state; the only difference is that the student confirmed it out loud. Measuring either tells you more than shaving 200 ms off a case you already know.
• Then ascending score.
• Then the library's own sort_order, then the id. Ties therefore always break the same way: the same data yields the same drill list, every time. Drill sessions are reproducible, which matters when a coach and a student compare notes.
• star_bonus is 15 points for a starred case, applied only in this sort key. Fifteen is about one status band — enough to pull a starred case past its immediate neighbours, not enough to bury a genuinely worse case. The stored score is never touched: a star changes what you see, not what is true.
• Focus wins. If the student has set an explicit focus (Focus rows may name a case, a group or a whole phase — "a student can also select focus cases or skills to practice"), the candidate pool is restricted to it. A focus that matches nothing drillable is treated as stale and ignored, rather than handing back an empty drill session.
• Cases with no algorithm in the library are skipped by default (require_algorithm=True) — there is nothing to drill.
• limit defaults to drills.weakest_case_count from config.yml, which is what makes "the worst 3 (configurable)" actually configurable. The same setting controls how many ids PhaseScore.weakest carries.

Projected solve time#

estimated_solve_time(profile) answers the "Expected solve time" question from _plan/idea.md: given what we know today, how long is a solve going to take, and which phase is costing the most?

Per case#

no measurement           ->  case_target_ms * SELF_TIME_FACTORS[self_score]
measurement, no claim    ->  avg_ms
measurement and a claim  ->  w * avg_ms + (1 - w) * target * factor(self_score)

This is deliberately not the same shrinkage the score uses, and the difference is worth understanding. The score answers "should I practise this?", where thin evidence should stay near the prior. The projection answers "how fast are you?", and a recorded time is direct evidence about speed. Blending a single measured 1.4 s T-perm towards the "never seen it" prior projected a 10.1 s PLL for a solver who had visibly just done it in 1.4 s. With no claim to weigh it against, the measurement stands on its own; with a claim, the two are blended on the usual weight, because then there really are two competing opinions.

The self-rating table is a separate calibration from the score, which is also why 4 and 5 tie on score but not on projected time: for ranking what to practise, the difference between "solid" and "instant" is unproven noise; for projecting a solve it is a second and a half.

Per phase#

phase_seconds = weighted_mean(case estimates) * CASE_UNITS_PER_SOLVE[phase]

The mean is weighted by Case.probability from the library ("1/108", "2/54", "4%" all parse) and falls back to uniform weights when the library has no probabilities. This matters for OLL: an algorithm you meet once in 108 solves should not drag the projection as hard as one you meet every twelve. For F2L the weighted mean is the expected pair time, multiplied by four.

The cross has no case library, so its evidence is the cross_ms split on real solves — and those splits win whenever they exist, whether or not the library happens to carry cross cases. Crucially the cross goes through the same combine_score as every case, against the cross target. It used to have a scale of its own, on which a single 1.5 s cross split scored 96 while a single 1.5 s OLL rep scored 23; two panels of the same dashboard cannot disagree about the same student by that much.

With nothing recorded, the cross scores None — not 0.0. That distinction is the difference between "we do not know" and "you are bad at this", and it is what stopped the phase panel reading 0 while the projection next to it reported goal pace.

Total#

The four phase estimates are summed and compared against the sum of the goal splits. The result also carries each phase's share of the total and a confidence (the fraction of cases actually backed by measurements), so the UI can say how much to believe the number.

For a profile with no data at all the projection is exactly the goal — the honest answer to "we know nothing about you" is "we assume you are on target", not "we assume you are terrible".

Phase and spider aggregation#

phase_summary(profile) averages the case scores of a phase, counting an unknown case as 0. A phase where half the cases have never been looked at is not an 80% phase, and a student who has assessed six OLLs should not see a green ring. A phase with no cases at all scores None instead, which is a different statement and renders differently.

The spider chart (stats.spider_data) plots the same numbers with the target ring at 80. Every axis carries has_data, so an axis at 0 because nothing was recorded can be drawn as an empty state rather than as a score of zero.

Cost#

case_scores() is five queries regardless of how many cases exist — cases, assessments, attempt aggregates, capped attempt detail, stars — and, more importantly, a bounded number of rows:

• counts, the DNF tally and the all-time personal best are aggregated by the database, one row per case;
• the detail rows that feed the weighted average are capped at MAX_ATTEMPTS_CONSIDERED per case by a ROW_NUMBER() window function.

A student with 120 reps on each of 119 cases therefore moves ~5950 rows rather than 14 280, and that ceiling does not grow with further practice. A constant query count alone would not have caught this; tests/test_scoring.py counts the rows each statement returns and asserts the number does not move when the practice history doubles.

phase_summary, estimated_solve_time, spider_data, weakest_cases and personal_bests all accept an already-computed scores (or summary) mapping. stats.dashboard_summary computes the table once and threads it through every panel; without that, one dashboard render made four full passes over the attempt table.

Caching#

case_score_cache is a denormalised (profile, case) row so list pages can sort in SQL. refresh_case_score(profile_id, case_id) maintains one row, refresh_all(profile_id) rebuilds the lot and prunes rows for deleted cases. Both take commit=False for callers that own a larger transaction — a drill that records an attempt and refreshes the score should decide for itself when the transaction ends.

Because score is NOT NULL, an unknown case is stored as 0.0; the unknown state is recovered from attempts == 0 and not is_assessed. That predicate is exact, and a test asserts the equivalence across unknown, all-DNF and "never seen it" cases: combine_score returns None only when there is neither an assessment nor a single attempt, so an all-DNF case carries a real low score and is correctly not recovered as unknown.

Constants#

All of them live at the top of app/services/scoring.py.

Statistics (app/services/stats.py)#

Aggregations for the dashboard, the timer and the assessment overview. All of them return plain dicts and lists of primitives — no ORM objects, no datetime — so a route can jsonify the result directly.

spider_data, progress_series, recent_attempts, personal_bests, session_summary, solve_stats, dashboard_summary.

WCA averaging#

wca_average(times, size) drops the best and worst trim_count(size) solves and means the rest. None in the input is a DNF.

trim_count(n) = max(1, ceil(n * 0.05))

One rule that reproduces both conventions instead of special-casing them: WCA regulation 9f8 drops a single best and worst for ao5 and ao12 (5% of 5 and of 12 both round up to 1), and cstimer — the reference _plan/idea.md names — trims 5% of each end for the long averages, so ao100 drops the fastest five and the slowest five. That is not cosmetic: on 100 solves containing five flukes and five disasters, a 1-best-1-worst trim inherits four of each and reports 13 265 ms where the correct answer is 10 000 ms.

• A DNF is by definition the worst result, so DNFs fill the dropped-worst slots first. [1000, 2000, 3000, 4000, DNF] averages the middle three to 3000. An ao5 tolerates one DNF, an ao100 tolerates five; one more leaves a DNF inside the counting set and the whole average is a DNF.
• Fewer than size solves is not enough, which is a third state, not a DNF. Hence AverageResult(value, is_dnf, count) rather than a bare float.
• +2 is applied before averaging; a dnf penalty makes the solve a DNF.

best_rolling_average(times, size) scans consecutive windows in chronological order for the personal-best ao5 / ao12 / ao100.

session_summary passes DNFs through to the average in their original slots, as None. Filtering them out first shortened the sample, which turned a legitimate five-rep ao5 containing one DNF into "not enough solves", and in a six-rep session silently pulled an older rep in to take the DNF's place.

progress_series#

Two dense arrays per metric, both exactly as long as dates, so a chart needs no gap handling:

• daily — the mean of that calendar day, None on empty days (draw as dots)
• rolling — the mean over the trailing 7 days, carried forward across gaps so the trend line never breaks (draw as the line)

The rolling mean pools the raw solves in the window rather than averaging the daily means, so a day with ten solves counts ten times as much as a day with one.

Days are UTC by default. Pass tz_offset_minutes (JavaScript's -new Date().getTimezoneOffset()) to bucket by the student's local midnight instead — otherwise an evening session in UTC+13 lands on "tomorrow" and a late-night one in UTC-8 on "yesterday".

Timestamps#

Everything here assumes every writer uses app.models.utcnow(), which is aware UTC. SQLite's DATETIME silently drops the offset on the way in, so a row written with a non-UTC aware datetime comes back shifted; _as_utc reads naive timestamps as UTC because that is the only interpretation that can be right. If you add a write path, use utcnow().