Table and Form Extraction: Document AI Annotation for Complex Layouts

Why Table Extraction Annotation Is Harder Than It Looks

A document page is not just text — it is a spatial layout where position carries meaning. Tables encode information in a two-dimensional grid where the relationship between a cell's content and its row and column headers is what makes the data extractable. Annotating a table incorrectly — even if every cell is accurately transcribed — produces training data where the grid topology is wrong, and a model trained on that data will extract values from the wrong columns or rows in production.

A 2023 benchmark paper published at the International Conference on Document Analysis and Recognition (ICDAR) evaluated five commercial table extraction APIs against a held-out set of Australian financial report tables. The best-performing system achieved 91.3% cell-level accuracy on simple tables (fixed structure, visible grid lines, no spanning cells) but dropped to 54.7% on complex tables with multi-level headers and spanning cells. The accuracy gap was not a model capability problem — it was a training data problem. The commercial systems had been trained primarily on simple table structures, because complex table annotation is expensive and the spanning-cell annotation protocols are rarely documented clearly enough to produce consistent inter-annotator agreement.

This is the annotation investment gap that production document-AI teams most often discover after deployment rather than before. Document annotation services that specialise in IDP annotation handle complex table structures routinely — but the annotation format, spanning-cell protocol, and header taxonomy must be agreed before any annotation work begins.

Two Annotation Tasks: Detection and Structure Recognition

Table extraction is a two-stage problem, and the annotation for each stage is distinct:

• Table detection annotation: labels the bounding box of each table on a document page. The annotation challenge is defining what counts as a table — a two-column price list, a two-row comparison grid, and a 40-row financial schedule all qualify, but a bulleted list formatted with tabs does not. Annotation guidelines must include visual examples of edge cases and explicit rules for borderless tables (where columns and rows are implied by whitespace).
• Table structure recognition (TSR) annotation: labels the internal structure of a detected table — individual cell bounding boxes, header row identification, column header identification, spanning cell extent, and the row and column index assigned to each cell. TSR annotation requires annotators who understand the difference between a header row (column labels) and a body row (data), and can correctly assign row and column spans to merged cells.

Most production pipelines train detection and TSR as separate models — or use a multi-task architecture — and need separate annotated datasets for each. A common project mistake is annotating only at the detection level and assuming the production system will infer cell structure from the detected region. Current state-of-the-art TSR models (TableTransformer, TATR, TableMaster) require cell-level annotation — the model cannot infer grid topology from the outer table boundary alone.

Spanning Cells: The Most Common Annotation Error

Spanning cells — cells that occupy multiple rows or columns because their content applies to the entire span — are the annotation challenge where most teams produce systematically wrong training data. The consequences are precise: a model trained on incorrectly annotated spanning cells learns the wrong grid topology and will assign extracted values to the wrong column or row headers in production.

Three annotation decisions must be explicitly specified for each spanning cell:

• Bounding extent: the bounding box must cover the full merged area, not just one grid position within it.
• Span attributes: the row-span count (how many rows the cell covers) and column-span count must be recorded as metadata on the annotation.
• Grid position assignment: which canonical grid positions (row indices × column indices) does the spanning cell occupy? Some architectures infer this from the bounding box geometry; others require explicit listing. The annotation schema must specify which approach the target model architecture uses.

A 2022 analysis of TSR annotation datasets found that spanning cell annotation errors were present in 23.4% of complex tables across three publicly available benchmark datasets, and that these errors accounted for 61% of all cell-assignment failures in models trained on those datasets. This is not a rare edge case — any financial report, scientific paper, or government form with multi-level headers will have spanning cells, and that is the document type most production IDP systems are built to process.

Case Study: Financial Report Table Extraction at a Fund Manager

An Australian funds management firm was extracting financial data from ASX annual reports to populate a portfolio analytics database. Their initial pipeline used a commercial document AI API trained on generic financial documents. Cell-level extraction accuracy on simple tables (income statements, balance sheets with visible grid lines) was 87.4%. On complex tables — fund performance attribution tables with three-row column headers spanning multiple sub-columns, and dividend tables with spanning footnote rows — accuracy fell to 49.1%, requiring manual correction of approximately 51% of table extractions.

The annotation scope we designed for fine-tuning their pipeline:

• 4,200 tables from 380 ASX annual reports across 12 industry sectors (2015–2025)
• Detection annotation for all tables including borderless performance attribution tables
• TSR annotation at cell level using PubTables-1M format (compatible with TableTransformer fine-tuning)
• Explicit spanning-cell protocol with row-span, column-span, and grid-position attributes
• Three-tier header taxonomy: level-1 header (section label), level-2 header (sub-section), column header (data label)
• Two-annotator workflow with table-structure expert adjudicating all spanning-cell disagreements (approximately 6.8% of tables required adjudication)

Results after fine-tuning on the annotated dataset:

• Cell-level accuracy on simple tables: 87.4% → 95.3%
• Cell-level accuracy on complex tables: 49.1% → 88.7%
• Manual correction rate: 51% → 14% of table extractions
• Analyst time on data normalisation: reduced from 3.2 hours/report to 0.7 hours/report
• Database population throughput: 4.1 reports/analyst-day → 16.8 reports/analyst-day

The annotation investment paid back within the first month of production use. This ROI pattern is consistent with broader intelligent document processing case studies — the accuracy gap between generic models and domain-fine-tuned models is largest on complex layouts, and complex layouts are precisely where the commercial value is highest.

Form Field Annotation: Structured vs Unstructured Forms

Form field annotation is distinct from table annotation but often required in the same document-AI project. Forms present two annotation challenges that differ from tables:

• Key-value pair annotation: identifying the field label ("Date of Birth") and its corresponding value region, even when they are spatially separated. The label may be above, to the left, or within the field boundary. Key-value pair annotation must capture the spatial relationship, not just the locations of the two elements independently.
• Checkbox and radio button annotation: binary state annotation (checked / unchecked) plus the label associated with each option. Checkbox annotation is deceptively difficult because "checked" is defined differently across form types — a scanned paper form may show a handwritten tick, an X, or a filled circle; a digital form may show a solid square, a checkmark SVG, or a partially rendered PDF checkbox glyph.

Unstructured forms — where field positions vary across instances of the same form type (common in older paper forms and some government templates) — require layout-normalisation annotation in addition to field-level labelling. The model must learn to find "Date of Birth" regardless of whether it appears in the top-left or mid-right of the page. This is a significantly harder annotation task than fixed-layout form annotation and requires larger annotated datasets to achieve equivalent accuracy.

Multi-Page Tables: Annotation Across Page Breaks

Tables that continue across page breaks are among the most annotation-intensive document structures. The annotation challenge is threefold: identifying that a table continues from the previous page (continuation detection), correctly associating the rows on page N+1 with the column headers from page N, and handling repeated header rows that appear on continuation pages (which should be annotated as headers, not as data rows).

Production annotation pipelines for multi-page tables require two pieces of annotation infrastructure that single-page table annotation does not:

• A document-level table ID that is consistent across all page segments of the same logical table, so the model can associate page-level detections into a single structured output
• A continuation marker annotation on the first row of each continuation page, distinguishing whether that row is a repeated header row (exclude from data extraction), a data row with a continuation label ("continued from page 3"), or a true first data row of the continued segment

Many production IDP systems avoid multi-page table handling by splitting the problem — extracting each page independently and post-processing the results to reconstruct the full table. This works when tables are short and column structures are simple. For complex financial schedules that run 8–15 pages in annual reports, post-processing reconstruction without correct continuation annotation produces unreliable column alignment across page boundaries.

Quality Metrics: TreeEdit Distance, GriTS, and Field Accuracy

Three metrics are used to evaluate table extraction annotation quality, each measuring different aspects of structural correctness:

• GriTS (Grid Table Similarity): the primary TSR evaluation metric from the PubTables-1M paper. Measures topology correctness (are cells in the right grid positions?), content correctness (are cell transcriptions accurate?), and location correctness (are cell bounding boxes accurate?). GriTS scores range 0–1; production fine-tuning targets 0.90+ on the document type being annotated.
• TreeEdit Distance Score (TEDS): measures the edit distance between the annotated table tree structure and the model output structure, normalised by tree size. Used primarily in research settings and ICDAR competition evaluation.
• Field-level extraction accuracy: the end-to-end metric for production systems — the fraction of specific field extractions (e.g. "Total Revenue for FY2024") that are exactly correct in the extracted output. This is the metric that maps directly to business value and is the one that document-AI teams should optimise annotation investment around.

GriTS should be measured separately by table complexity tier — simple, medium (some spanning cells), and complex (multi-level headers, extensive spanning). Aggregate GriTS across all table types is misleading when complex tables drive the majority of production failures. The annotation investment decision should be driven by per-tier GriTS on the table types that represent the highest business value to extract correctly.

Related annotation types that complement table extraction in full document-AI pipelines include OCR annotation for text region transcription and handwriting OCR annotation for documents with mixed printed and handwritten content.

Annotation Format: Matching Your Target Architecture

The annotation format must match the training pipeline of the model being fine-tuned. Mismatches between annotation schema and model input format are a common cause of failed fine-tuning projects — the data is technically annotated correctly but the format cannot be ingested without schema conversion that often introduces errors. Three formats cover most production use cases:

• PubTables-1M format: JSON with table bounding box, cell bounding boxes, row/column indices, and spanning attributes. The native format for TableTransformer and Microsoft's TATR model family. The most widely compatible format for open-source TSR fine-tuning.
• ICDAR XML: XML-based table annotation format used in ICDAR 2013/2019 competition datasets. Appropriate for research pipelines and systems trained on ICDAR benchmark data.
• Vendor fine-tuning schemas: AWS Textract, Azure Document Intelligence, and Google Document AI each have custom annotation formats for fine-tuning. These formats are documented in vendor SDKs but differ significantly from research formats. Annotation done in PubTables-1M format must be converted before vendor fine-tuning — the conversion is non-trivial for spanning cells.

Our document annotation service produces annotation in the target format for your pipeline — whether that is PubTables-1M for open-source fine-tuning or a vendor-specific format for AWS, Azure, or Google Document AI — so no post-annotation schema conversion is required.

Frequently Asked Questions