Case study
mat73-reader: opening MATLAB’s undocumented table format
A research dataset was locked inside a file format no Python tool could read. I reverse-engineered the format and shipped the fix for everyone.
The format
Where the table hides
A v7.3 file is HDF5, so h5py opens it happily. Then the trail goes cold: the table variable is a group holding opaque integer references into a hidden group called #refs#, where MCOS, MATLAB’s internal object system, serialized the real data. Which reference holds the column names, which holds the types, and which holds the payload is the undocumented part.
mat73-reader implements that mapping. It follows the references, reassembles the columns, and hands back a pandas DataFrame. For everything MCOS does not touch, the right tools already exist, and the README says so: scipy for pre-7.3 files, mat73 for plain v7.3 arrays. This library exists for the case those tools leave open.
>>> f = h5py.File("data_v3.mat") >>> list(f.keys()) ['#refs#', '#subsystem#', 'Data'] >>> f["Data"]["task"] <HDF5 dataset "task": shape (47, 1), type "|O"> # refs >>> f["#subsystem#"]["MCOS"] <HDF5 dataset "MCOS": shape (1, 5597), type "|O">
01 · The wall
Every existing tool fails on MATLAB tables
MATLAB v7.3 stores tables through MCOS, an undocumented internal object system layered over HDF5. scipy raises. The mat73 library returns None. The data is right there in the file, and none of the standard Python tools could reach it.
>>> scipy.io.loadmat("data_v3.mat") NotImplementedError: Please use HDF reader for matlab v7.3 files, e.g. h5py >>> mat73.loadmat("data_v3.mat") ERROR: MATLAB type not supported: table, (uint32) # ... 799 times, one per table, each returned as None
>>> from mat73_reader import load >>> data = load("data_v3.mat") >>> data["Data"]["task"][0]["gaze"][0].head(3) gaze_timestamp world_index confidence norm_pos_x ... 0 5410.551714 0.0 0.999499 0.446264 ... 1 5410.555834 0.0 0.999653 0.446534 ... 2 5410.559773 0.0 0.999648 0.446660 ... [8205 rows x 21 columns] >>> # participant 1, task 1: a DataFrame, straight from MCOS
02 · The key
Reverse-engineered, then made ordinary
The library decodes MCOS table structures into pandas DataFrames with one call, and handles the numeric arrays, structs, cells, and character data around them. A CLI covers inspection and conversion to CSV or JSON for people who never want to leave the shell.
03 · Built to outlive its origin
The tool knows nothing about eye tracking
The dataset that motivated it is a cognitive science corpus, and the library contains not one line about it. Researchers in neuroscience, signal processing, and physics hit the same wall; the fix is domain-free by design. Apache 2.0, on PyPI, with an honest comparison table pointing at the right tool for every other .mat flavor.
It also had a bug. In August a cell-for-cell cross-check against a second decoder found version 0.1.0 reading zero-row tables as two rows of [0, 1]; 40 of the corpus’s 799 tables are empty. Version 0.1.1 fixed it with a regression test, and the research built on the reader reran and republished its numbers.
The research it unlocked →$ mat73-reader inspect data_v3.mat Variable Type Shape/Children Data group subject_info, task $ mat73-reader convert data_v3.mat --format csv
By the numbers
012345678901234567890123456789012345678901234567890123456789
tables decoded
from the dataset that motivated the tool
01234567890123456789.01234567890123456789
GB validated
a full research corpus, end to end
0123456789012345678901234567890123456789
tests
across standard operations and edge cases
01234567890123456789
languages cracked MCOS
independent decoders exist in C#, Julia, and Python; this one is tables-first with a CLI
Why it matters
The blocker became the contribution: a working key to a closed format, the mat7.3 library’s maintainer building on the format notes, and the research that needed it back in motion.