Regulated AI & Compliance

AutoML Doctor: Statistical Diagnostics for Tabular ML

Local-first statistical diagnostics for trustworthy tabular machine learning

A second opinion for a tabular model: it checks the data, the splits and the predictions for the reasons a result might look better than it really is.

The problem

Leakage, group overlap between train and test, small test sets and hidden imbalance can all make a model look trustworthy when it is not, and these checks are rarely run in a consistent way.

The approach

Eight diagnostics compute integrity, imbalance, correlation, outlier, leakage, overfitting, metric-uncertainty and error checks from the data alone. Predictions are joined by row ID, never by position, and invalid inputs are rejected instead of silently fixed. Findings are graded by configurable heuristics and written to a validated JSON report.

The outcome

An open-source, local-first CLI with no network calls or telemetry that can gate CI on serious findings.

In detail

Built AutoML Doctor (mldoctor), an open-source command-line tool that inspects training data, held-out data and model predictions and reports the statistical problems that quietly inflate ML results: missing data, class imbalance, redundant features, outliers, leakage indicators such as group overlap across splits, generalisation gaps, imprecise metrics and systematic errors. It runs entirely on the local machine with no network calls, no telemetry and no LLMs, grades findings with configurable screening heuristics rather than a single health score, and writes a versioned, strictly validated JSON report plus a terminal view. A fail-on threshold lets it block merges in CI.

Built with

Trustworthy ML
Data Leakage
Model Validation
Python
CLI

More in Regulated AI & Compliance

Next project: MLang: A Compiler for ML Inference

By Kanishka Ranaweera, PhD, Machine Learning Engineer in Melbourne, Victoria, Australia. hello@kanishkabuilt.me