Case study 04
Credit-data quality and benchmarking
Know what the data means before you show it to anyone.
Audited completeness, comparability, and meaning in commercial credit data, and turned the findings into recommendations and more interpretable score displays.
- Role
- Data Science Co-op
- Period
- 2026
- Status
- Analysis delivered · displays shipped
01 — My contribution
I designed the sampling, ran the field-level audit and cross-source comparisons, and built the percentile-based display logic and deterministic explanation templates.
02 — Problem and context
A credit product is only as good as its understanding of the data underneath it. Before relying on a dataset, the team needed to know which fields are actually populated, what the scores mean, and how they compare with independent sources.
Raw scores also arrive on arbitrary scales. A customer cannot read them without a reference point.
From a large file to a readable score
Seeded sample
One streaming pass
Fill rates
Nulls, placeholders, zeros
Distributions
Shape, heaping, tails
Cross-source
Rank-based agreement
Display
Midrank percentiles + templates
03 — Approach and key decisions
Decision 01
Representative samples from large files
Reservoir sampling streams a very large file once and draws a uniform random sample with a recorded seed. That matters because large files are often sorted, so the first rows are not representative of the rest.
Decision 02
Missing is not zero
Fill rates distinguish true nulls, placeholder strings, and zeros that mean “no data”. Treating a placeholder as a real value biases every statistic downstream.
Decision 03
Look at the shape before summarizing
Distribution fitting, normality tests, and density plots, plus checks for heaped values — many records sharing one value — which change how percentiles should be computed.
Decision 04
Compare sources by rank, not raw scale
Providers use different scales, so comparisons used rank correlation, weighted agreement on ordered classes, and equipercentile equating, with bootstrap intervals for uncertainty.
Decision 05
Percentiles customers can read
Score displays use midrank percentiles against a reference population, which places tied values fairly, with tiers set from the population distribution rather than from documentation.
Decision 06
Deterministic explanations
Customer-facing explanations are fixed templates selected by which condition applies, and wording strength was matched to how often independent information agrees with the statement. No generated text is involved.
Illustrative example — synthetic data
Ten synthetic scores. Four businesses share the value 455.
- 410
- 455
- 455
- 455
- 455
- 520
- 560
- 610
- 640
- 700
| Method | Percentile of 455 | Effect |
|---|---|---|
| Share strictly below | 10% | Ties look worse than they are |
| Share at or below | 50% | Ties look better than they are |
| Midrank: below + ½ × equal | 30% | Places ties fairly in the middle |
04 — Evaluation and evidence
This was measurement work, so evaluation meant stating uncertainty and checking that the pipeline read the data correctly.
- Uncertainty stated
- Bootstrap intervals on key statistics, and display lookup tables checked against exact computation.
- Reproduction checks
- Where a provider publishes its own class labels, rebuilding them from the underlying scores confirmed the data was being read correctly.
- Methods chosen by test
- Normalization methods were compared with paired statistical tests rather than chosen by appearance.
05 — What shipped
Findings informed data-selection recommendations and the design of more interpretable score displays.
Percentile bars, population-based tiers, explicit missing-data handling (“a missing score isn't a low score”), and deterministic explanations shipped in the product.
- Status: Analysis delivered
Data-quality audit and benchmarking
Field-level audit, distribution analysis, and cross-source agreement study.
- Status: Shipped
Percentile displays and explanations
Midrank percentile bars, tiers, missing-data handling, and template-based explanations.
06 — Limitations and lessons
Limitations
- Agreement between providers is not ground truth. Low agreement shows uncertainty, not which source is right.
- A data-quality audit does not establish predictive performance.
- Percentile displays are relative to a reference sample and need refreshing as the population changes.
Lessons
- Documentation and data disagree more often than you would expect, so check both.
- “How much do independent sources agree?” is often more answerable than “is this source accurate?”
07 — Tools
- Python
- pandas
- NumPy
- SciPy
- scikit-learn
- Matplotlib
- DuckDB
- Jupyter