Case study · Research
Reading cognitive overload from eye movements
When interfaces overwhelm people, performance drops and learning stops, and surveys only catch it after the fact. I asked whether eye movements alone could flag the moment of overload, and grounded the answer in Cognitive Load Theory.
The corpus, alive
Watch load happen
Watch one participant from the COLET corpus: thirty seconds of the task they rated easiest and thirty seconds of the one they rated hardest, replayed in sync. Nothing here is simulated.
The easiest task
TLX 14.2 · calm
The hardest task
TLX 83.3 · overload
Synchronized replay, 2x speed. The dot is the gaze; its size is the pupil.
>>> scipy.io.loadmat("data_v3.mat") NotImplementedError: Please use HDF reader for matlab v7.3 files, e.g. h5py >>> from mat73_reader import load >>> load("data_v3.mat")["Data"]["task"][0]["gaze"][0].shape (8205, 21) # participant 1, task 1, unlocked
The wall, then the key
The corpus was locked, so I wrote the decoder
COLET ships as MATLAB v7.3 table objects, an undocumented format no Python tool could read. The proof of concept trained on a synthetic stand-in and said so plainly; every number it produced carried that asterisk.
mat73-reader opened the real recordings: 47 participants, four tasks, 188 labeled task instances. It shipped on PyPI. The maintainer of the mat7.3 library built that library’s affine2d support on my write-up of the format, then asked for the table layer as a pull request. The corpus stopped being an asterisk.
What the real data changed
What the synthetic stand-in hid
Same pipeline, same ten features, same forest. The proof of concept reported 0.897 on the stand-in; the identical pipeline on the real corpus, with participants held out, scores 0.665. Switch worlds and watch the feature story change with the number.
0.897 · synthetic, one split, positive-class F1
The 0.665 is the first experiment’s grouped five-fold run. Repeated over five seeds, the same pipeline scores 0.623, the figure the paper’s table reports.
Where the 0.897 came from
The decomposition experiment replicated the 0.897 exactly, then took it apart. About 0.21 of the gap is the data: the stand-in fabricated structure that real recordings do not have, including the saccade-dominance story. About 0.08 is single-split luck. The rest is the metric, since the proof of concept reported positive-class F1 where the rebuild reports macro F1. Grouping the split moved almost nothing on either dataset. The stand-in had also erased participant-level correlation, so a grouped check run on it would have given false comfort.
From 0.665 to 0.783
The rebuild kept the honest split and changed the rest. Within-person z-scores replaced raw feature levels, because pupil size and gaze velocity are personal; that step alone moved the headline to 0.690 and mean pupil diameter from last to first in importance. A twelve-feature parsimony set replaced the ten. Then a sweep of four model families on two feature sets, five seeds by five participant-grouped folds, picked a logistic regression at 0.783 ± 0.046. The forest the proof of concept used scored 0.775 on the same features. Twelve coefficients beat every tree model.
The audit turned inward
A pipeline that audits stand-ins earns the same audit pointed at itself.
In August I cross-checked mat73-reader against a second decoder, the one I was contributing to the mat7.3 library, on all 799 tables in the corpus. I ran it because numbers that depended on the reader were already out. The two disagreed on forty. Every one was an empty table. Version 0.1.0 read a zero-row table as two rows of [0, 1], so for 40 of the 188 instances the blink features were fabricated: two blinks per span, half a second each, where the truth was none.
I fixed the reader, rebuilt the features, and reran the feature, label, and protocol experiments under the frozen protocol. Every cell without blink features reproduced exactly. The headline moved from 0.762 to 0.783, blink rate became the largest coefficient, and two coefficients flipped sign. The contaminated results stay in the repo beside the corrected ones, with a correction block that says what changed and why. The abstract that had already gone out carries the old numbers; everything after it carries these.
The theory underneath
Three kinds of load, two of them measurable here
Cognitive Load Theory, from John Sweller’s work on why people learn poorly when problem-solving demands exceed working memory, splits mental burden three ways.
Intrinsic load
The difficulty of the task itself. Within-person pupil change tracks it, and mental demand is the subscale where the model scores highest.
Extraneous load
The burden added by poor presentation. Theory says scanning carries it; on real recordings the saccade signature is a fraction of what the stand-in claimed.
Germane load
The productive effort of building understanding. Not measured here. Protecting room for it is the point of measuring the other two.
What survived
The findings that held
The headline moves from 0.640 to 0.841 on labeling choice alone, so the pre-registered tertile split stays primary. The model is strongest on mental demand and near chance on physical demand, the negative control a cognitive model should fail.
Pick the labeling. Watch the headline.
Positive class 36% of instances. Balanced classes flatter the metric; the number is a property of the labeling as much as the model.
Six subscale targets, one negative control
The dashed line is chance.
Both panels are the label-sensitivity run: a Random Forest on the twelve-feature set, five participant-grouped folds, rerun after the reader correction. The 0.783 headline is the later protocol lock, a logistic regression under five seeds by five grouped folds; on the same tertile labels that run reports mental demand at 0.786 and the physical control at 0.501. The control holds across model families.
Twelve coefficients
The model, running right here
The sweep winner is a logistic regression, so the whole model is twelve coefficients and an intercept, and it runs in this page. Load a class profile or move the sliders yourself.
Not high load
probability of high load. Sliders are standard deviations from your own calm baseline.
The coefficients are the full fit of the frozen winner on the corrected features. Inputs are within-person z-scores, standard deviations from a reader’s own calm baseline, which is the calibration a deployment would need. The class profiles are the corpus means for each class.
By the numbers
01234567890123456789.012345678901234567890123456789012345678901234567890123456789
macro F1
logistic regression, five seeds by five participant-grouped folds, on the real corpus
01234567890123456789.012345678901234567890123456789012345678901234567890123456789
mental demand
the same model on the strongest subscale target, where a cognitive model should lead
01234567890123456789.012345678901234567890123456789012345678901234567890123456789
physical demand
chance on the subscale a load model should fail: the negative control
012345678901234567890123456789012345678901234567890123456789
labeled instances
47 participants by 4 tasks, decoded from the real COLET corpus
One program
This is one of four artifacts in one research program; the decoder that opened this corpus is another. How each one forced the next is on the work page.
The stand-in told a tidy story. The real data tells a subtler one, and the gap between them is what the paper is about.
After the paper, the same twelve coefficients run on-device, reading load in real time from a camera feed.