P R E D I C T O R

Documentation: PREDICTOR Web App Usage Instructions

This web app evaluates binary classification models with ROC and precision-recall curves, class separation histograms, a confusion matrix, threshold metrics, and a calibration plot.

How to Use This App

1. Uploading Data

   Upload Data: Choose a CSV file with a header row. The app looks for a label column (for example actual, label, y_true, target) and a score column (for example prediction, score, prob, y_pred). If it finds neither, it uses the first two columns. Labels must be 0 or 1. Blank or invalid rows are skipped and counted in a status message.

   Generate curves from data: The plots update as soon as a file is chosen. Use this button in the toolbar to regenerate them from the same file.

2. Toggling Full Screen

   Full screen: Click the ⛶ Full screen button to expand the page for easier viewing and interaction with the plots. You can also press the F key on your keyboard (when not typing in a field). Press it again or Esc to exit.

3. Generating Random Data

   Sample Size: Number of simulated observations, from 10 to 1,000,000. Default 1,000.

   Positive Prevalence: Share of observations in the positive class. Lower values show how the precision-recall curve reacts to class imbalance while ROC AUC stays roughly the same.

   Scenario: How scores are generated for each class. Random picks one of the five below on each click.

   • Equal-variance binormal: Both classes are normal with the same spread. Gives the classic symmetric ROC curve, from near chance to near perfect.
   • Unequal-variance binormal: Positives have a wider or narrower spread than negatives. The ROC curve becomes lopsided and can dip below the diagonal at one end.
   • Skewed (beta) scores: Each class follows a beta distribution, giving bounded, skewed scores like many real models produce.
   • Hard-to-detect positive subgroup: Some positives look exactly like negatives. The ROC curve flattens out before reaching the top, and recall cannot climb without many false positives.
   • Negatives with a high-scoring cluster: Some negatives score like positives, so precision drops even at strict thresholds.

   Scores from the latent scenarios pass through a logistic function with a random slope and offset, so calibration also varies from run to run.

   Seed: Leave blank for a new dataset every click. The seed used is shown under the buttons, and entering it again with the same settings reproduces that exact dataset.

   Distortions: When checked, each run may also round scores to a few levels (creating ties and staircase curves) and flip a share of labels to mimic outcome misclassification. The description line shows which were applied.

   Target ROC AUC and Target AP: Optional. Target ROC AUC can be 0.5 to 0.999 and Target AP 0 to 0.999; leave a box blank for no target. Fill in either or both, and the generated data will hit those values on the plotted sample. Negatives are drawn from N(0, 1) and positives from a normal distribution whose mean and spread are solved for:

   • The mean of the positives mainly sets the AUC: moving positives up raises both AUC and AP.
   • At a fixed AUC, the spread of the positives sets the AP. A wider spread puts more positives at the very top of the ranking, which raises AP. A narrower spread leaves a tail of high-scoring negatives above most positives, which lowers AP.
   • With only one target set, the spread is chosen at random, so each click gives a differently shaped curve with the same value.

   AP depends on prevalence, so not every combination is reachable. If a target pair cannot be met, the app uses the closest reachable AP and shows the reachable range. While a target is set, the Scenario and Distortions options are turned off, since rounding and label noise would move the metrics off target.

   Generate random data & curves: Draws a new dataset with these settings and plots it.

4. Choosing a Threshold

   Threshold slider: Observations with a score at or above the threshold are predicted positive. Moving the slider updates the marker on both curves, the dashed line on the histogram, the confusion matrix, and the metrics table. For scores between 0 and 1 the slider runs from 0 to 1 and starts at 0.5; for uploaded scores outside that range it spans the lowest to highest score and starts at the midpoint.

5. Downloading

   Download data as CSV: Saves the current actuals and predictions. Use the camera icon on any plot to save it as a PNG.

6. Understanding the Display

   Toolbar: The Generate random data & curves, Download data as CSV, Generate curves from data, Show/hide documentation, and Full screen buttons stay pinned to the top of the window as you scroll. Both generate buttons take you straight to the curves when they finish, from anywhere on the page; if an input is invalid, the page scrolls to the message instead. On phones the toolbar fits in two rows with shorter labels (Random curves, Download CSV, Curves from data, Documentation, Full screen).

   Layout: The four main plots are drawn as equal squares in a 2 x 2 grid. On phones and narrow windows they stack one per row, and the layout refits when the window is resized or the phone is rotated.

   ROC Curve: True positive rate against false positive rate across thresholds. The ROC AUC is shown in blue above the plot.

   Precision-Recall Curve: Precision against recall across thresholds. Above the plot, AP (PR) in green is the average precision, and AP (No Effect) in purple is the prevalence, which is the AP of a model with no skill and matches the dotted baseline.

   Class Separation: Score distributions for each class, each normalized to its own class size.

   Confusion Matrix: True positives, false negatives, false positives, and true negatives at the selected threshold.

   Calibration: Mean predicted probability against the observed positive rate in 10 bins. Points on the diagonal indicate good calibration, and marker size reflects bin size. Shown only when all scores are between 0 and 1.

   Metrics: Sensitivity, specificity, PPV, NPV, F1, and accuracy at the threshold, plus AUC and AP. The Brier score (mean squared difference between score and outcome) is included when all scores are between 0 and 1.

7. How the Metrics Are Computed

   Tied scores are treated as a single threshold, and both curves start at the origin. AUC uses the trapezoidal rule and AP uses the step definition, matching scikit-learn's roc_auc_score and average_precision_score.