Case study
A full tournament of predictions, scored against Bet365's closing line. It beat them on first goalscorer, and it repeatedly got a better price than they closed at.
How I built it
9,819 lines of Python and 2,588 lines of tests across 24 modules, on eight libraries. No GPU, no hosted inference, no database. The interesting parts are the pipeline shape and the gate, so that is what the rest of this page is about.
penaltyblogpandasnumpystatsmodelsscipyrequestsstreamlitpyarrowTheStatsAPI ──> ingest.py ──> data/*.parquet ──> models/ ──> predictions/*.json
ESPN (free) ──> espn.py 880 lines 10 files 4 models stamped MODEL_VERSION
23,735 rows
│
v
Bet365 odds ──> paper.py ──> 3 ledgers ──> settle + CLV
odds.parquet 895 lines 173 bets │
│ Kalshi + Polymarket feed the
│ live desk in the dashboard only —
│ read-only quotes, never graded.
v
notebooks/backtest.py ──> data/market_gates.json
walk-forward + bootstrap the betting engine reads this file
GitHub Actions ran the chain and committed data/ back to the repo after every run.
Streamlit read from GitHub every 5 minutes.| Model | Technique | Lines |
|---|---|---|
| Match outcome & goals | Dixon-Coles bivariate Poisson, time-decayed weights, neutral-venue aware | 355 |
| Anytime scorer | Poisson thinning — team goal expectation split by npxG/90 × projected minutes, penalty duty explicit | 142 |
| Corners | Negative binomial GLM on pre-match possession priors | 101 |
| Cards | Poisson on shrunk team booking rates | 45 |
Layer 1 refits Dixon-Coles daily and stores a base prediction. Layer 2 runs ~75 minutes before kickoff, measures the confirmed XI's attacking strength against the pre-lineup projection, scales each team's goal expectation by that ratio and rebuilds the whole probability grid. A prediction made 48h out and one made after team news are different products, so they are stored and graded separately.
Every run commits its parquets, predictions, stakes and settlements straight back into data/. There is no database and no mutable state — the git history is the record, so any figure traces to the commit that produced it and the state of the world at that moment. 444 commits.
The paid trial lapsed mid-tournament. espn.py replaced it in a day against a free no-auth feed: update-only writes that never overwrite a finished score, regulation and after-extra-time scores split by period from the scoring events, shootout kicks excluded. Parsers were written against captured fixtures, so the adapter is tested without a network.
MODEL_VERSION stamps every prediction. make test-regression re-runs the walk-forward and fails the build if the backtest degrades past its committed baseline, so a “harmless” refactor won't cost accuracy.
Method
Paired bootstrap, 20,000 resamples of the per-game Brier differences. Reported confidence is the share of resamples where the model comes out ahead. The same function gates the live betting, with a floor of ten games.
Five experiments tested and dropped: goal-rate market blending, knockout-stage pace adjustment, match-importance weighting, a knockout draw gate, the corners betting gate. Each logged with what it scored and why.
make backtest regenerates the 99-match walk-forward, the gate verdicts and the strategy replay. make calibrate rebuilds the scorecard and reliability bins. make postgame rebuilds the per-game report cards.
Market gates
Each market gets a walk-forward replay: refit on everything before that match day, predict, score against Bet365's de-vigged close. A paired bootstrap then resamples the per-game Brier differences 20,000 times and returns P(model better). Below 95%, the market never places a bet.
Four of the six failed. Two of those four were my own ideas.
Translation — the Brier score. Square the gap between the probability you gave and what happened, then average it over every match. Call a side at 70% and they win, you take (1 − 0.70)² = 0.09. Call them at 70% and they lose, you take 0.49. Zero is perfect, and guessing 33/33/33 on every match scores 0.667. The model came in at 0.4772, the book at 0.4833. Both are far better than guessing, and they sit 0.0061 apart.
| Market | n | Model Brier | Bet365 Brier | Confidence | Verdict |
|---|---|---|---|---|---|
| First goalscorer | 96 | 0.4631 | 0.4965 | 100% | Cleared — betting |
| Asian handicap | 16 | 0.0945 | 0.2517 | 100% | Cleared — betting |
| First-half result | 96 | 0.5752 | 0.5907 | 86% | No measurable difference |
| Corners | 72 | 0.2517 | 0.2492 | 45% | No measurable difference |
| Total goals | 99 | 0.2461 | 0.2357 | 24% | Bookmaker more accurate |
| Both teams to score | 99 | 0.2614 | 0.2479 | 7% | Bookmaker more accurate |
Match results
1X2 is the number every football model gets judged on. Over 99 matches: Brier 0.4772 against the closing line's 0.4833. The gap is +0.0061 on a 95% interval of −0.032 to +0.045 — it straddles zero, and the model was closer on 47 of 99 games.
Calling that a win would fall apart the moment anyone re-ran it.
| Measure | Sample | Result | 95% interval | Verdict |
|---|---|---|---|---|
| Match outcome (1X2) | 99 matches | Brier gap +0.0061 | −0.032 … +0.045 | Crosses zero — not established |
| Closing line value | 131 flat-stake bets | +1.24% | +0.68% … +1.84% | Clear of zero — holds up |
Each row is a 95% bootstrap on its own units — Brier points above, percent of closing price below — zero aligned.
What holds up
Profit over 44 bets is mostly variance. Closing line value converges faster: did the price you took beat the price the market settled on? Repeatedly ahead of the close means you are pricing something before the market does.
| Book | Bets | Mean CLV | 95% interval | Resamples ahead |
|---|---|---|---|---|
| Strict — Kelly-staked, 5%+ edge | 42 | +1.90% | +0.71% … +3.17% | 19,992/20,000 |
| Shadow — every edge, flat $10 | 131 | +1.24% | +0.68% … +1.84% | 20,000/20,000 |
| Book | Bets | Staked | P&L | ROI | 95% interval |
|---|---|---|---|---|---|
| Strict | 44 | $1,658.51 | +$414.59 | +25.0% | −20.2% … +78.0% |
| Shadow | 135 | $1,350.00 | +$5.30 | +0.4% | −22.2% … +25.5% |
| Parlays | 28 | $190.00 | +$83.29 | +43.8% | — |
The strict book turned $1,000 into $1,414.59, on an interval running −20% to +78%. Forty-four bets cannot separate a model from a good fortnight. The closing-line number is the one to argue from.
Where it was wrong
Across 297 outcome probabilities the model spread its mass too evenly — favourites shaded down, longshots up. At a stated 70%, the side won 86% of the time. The bookmaker carries the same bias, milder.
| Predicted | Actual |
|---|---|
| ~12% | 4.8% |
| ~29% | 26.9% |
| ~49% | 61.7% |
| ~70% | 85.7% |
| ~87% | 77.8% |
One temperature parameter on the output grid sharpens every row. Left undone: by the time the pattern was clear there was no held-out data to fit it on. Fitting it on the same games would have improved the scorecard and invalidated it.
Group stage +0.0127 over 72 matches; knockouts −0.0114 over 27, with lineup adjustments frozen from the quarter-finals on.
The gap in the evidence
The odds provider's trial ended 2026-07-08, quarter-finals still to play. Results kept arriving from a free feed, so every ledger settled and the bracket resolved through the final. No closing prices after that date, so from the quarter-finals on there is nothing to score the model against.
The book is Bet365. The ingest layer asks for Pinnacle and falls back to whatever the provider returns first; Pinnacle was never in the response, and the substitution was silent. Bet365 is a recreational book with wider margins, so every “beats the market” figure here is against an easier line than the sharp benchmark.
One thing that works in the numbers' favour: entry price and closing price come from the same book, so closing line value measures Bet365's own line moving toward the model rather than a gap between two venues. Kalshi and Polymarket quotes appear only on the dashboard's live desk — nothing priced from them was ever graded.
Start a project
Have an AI workflow, reporting gap, or system you can't seem to get off the ground? Scope it with me — I'll give you a real read on what to build first and what it should cost.
I review every submission and reply with a real read on fit.