Real business files are rarely clean. PDFs arrive with mixed fonts, merged cells, skewed scans, and headers that repeat on every page. Spreadsheets contain merged ranges, hidden columns, inconsistent date formats, and notes typed into the wrong cells. AI Speed Index therefore tests whether GLM 5.3 Fast can turn these messy sources into clean, structured data that a human can actually use.
I treat data-extraction tests the same way I treated data-quality checks in product work: start with the ugly inputs teams actually receive, define what “clean enough” means in practical terms, and measure both the model’s output and the human cleanup that still remains. A model that produces a fast but messy table often costs more total time than a slower process that delivers usable structure on the first pass.
What “Ugly” Means in This Test
The test deliberately uses files that reflect common real-world problems rather than perfectly formatted examples:
PDFs with multi-column layouts, repeated headers, footers, and occasional scanned pages
Tables that have been converted to PDF and lost clear cell boundaries
Spreadsheets with merged cells, inconsistent column naming, mixed data types in the same column, and free-text notes scattered among numeric values
Files that contain both the target data and substantial irrelevant material (cover pages, signature blocks, disclaimers)
These characteristics force the model to do more than simple copy-and-paste. They test whether it can identify the relevant regions, normalize formats, and output structure that matches a requested schema.
The files themselves are either synthetic or fully anonymized so that no confidential business data is sent to the API.
Success Criteria Defined in Advance
A run is considered successful when the output meets four practical conditions:
The requested fields or columns are present and correctly labeled
Values are normalized into consistent formats (dates, numbers, categories) where the prompt asked for normalization
Extraneous material (headers, footers, disclaimers, notes) is excluded or clearly separated
A human can load the result into a spreadsheet or database with only light correction
Partial success is also recorded. An output that captures most fields but mangles dates or drops two columns is logged as partial rather than discarded. The goal is to surface the exact cleanup work that remains, not to force every run into a binary pass/fail.
Speed and token counts are measured for reference, but they are secondary to structural usefulness. A fast extraction that still requires extensive manual repair is not treated as a win.
How the Extraction Workflow Is Run
Each file is processed through a consistent prompt sequence. The prompt specifies the desired output schema (column names, expected data types, and any normalization rules) and explicitly instructs the model to ignore non-data regions. The same prompt is reused across the file set so that differences in results reflect file difficulty and model behavior rather than prompt drift.
When a PDF is multi-page, the test records whether the model was given one page at a time or a larger window, and how that choice affected completeness. Spreadsheet tests note whether the full sheet or a selected range was supplied.
Timing follows the standard AI Speed Index method: first-token latency and total completion time are captured. Output is examined for structural fidelity against the requested schema. Any hallucinations (invented values that do not appear in the source), dropped rows, or format inconsistencies are logged as defects.
The test is run under the same ordinary Austin home-office conditions used for all other AI Speed Index measurements. No specialized document-parsing infrastructure is added; the point is to see what the model itself can achieve when the input is simply the file content plus a clear schema request.
What the Test Deliberately Limits
A focused extraction test still has boundaries. It does not claim to evaluate:
Handwritten forms or low-quality mobile photographs
Extremely large files that exceed practical context limits
Complex multi-table relational extraction across many documents
Integration with production OCR or document-intelligence pipelines
Legal or compliance review of the extracted data
Those scenarios belong in separate, more specialized tests. The present workflow stays inside the common case of “someone emailed us a messy PDF or spreadsheet and we need the numbers in a clean table today.”
How Results Will Be Reported
When the full set of extraction results is published it will follow the standard AI Speed Index closing structure:
Use it when…
Skip it when…
What we measured
What we still do not know
The measured section will summarize the file types tested, the observed rate of clean versus partial versus failed extractions, the typical cleanup effort still required, and the token and time costs. The open-questions section will note harder file types, larger documents, or newer model versions still unexamined.
Because extraction quality depends heavily on the specific messiness of the input, the article will also describe the characteristics of the test files so readers can judge how closely they match their own incoming documents.
Why Messy Files Are the Right Test
Many extraction demos use clean, born-digital tables. Those demos show what the model can do under ideal conditions. They do not show what happens when the file has been through two exports, a print-to-PDF step, and a few merged cells added by a previous user. The gap between ideal and ugly is exactly where most teams lose time.
A transparent test on deliberately messy inputs surfaces that gap early. It shows whether the model reduces the manual cleaning load or simply shifts the work into post-editing. Those are the numbers that decide whether the model belongs in the daily data-intake process.
Fast is a feature. Reliable is the product. Clean structure from ugly sources is one of the places reliability becomes measurable. AI Speed Index runs the files, publishes the method and the defect patterns, and lets readers decide how the results apply to the documents they actually receive.
Letters
No letters yet — be the first to write.