Section 3.5 Generative model architecture
Multimodal models and diffusion
A multimodal model uses more than one kind of signal. A PDF reader may receive extracted text, block coordinates and visual features of the page. A transformer can process vectors derived from small image patches. Evaluate its output against the structure of the source document.
Read a table from an image
The value “20” is unusable without its row and column. When extracting a scanned timetable, check the link between each cell, header and page, then inspect rows with merged cells. Compare with plain-text extraction and OCR that preserves layout. A multimodal model can complement those methods on irregular pages. A correctly formatted output can still point to the wrong cell.
Generate an image by denoising
In one family of diffusion models, training adds noise to observations at different levels. A network learns an operation that helps reverse the process. During generation, the algorithm starts with noise and applies successive denoising steps, optionally guided by text. Some architectures work in a compressed latent space. The mechanism differs from emitting text tokens left to right. Ho, Jain and Abbeel describe a foundational formulation. In both cases, test plausible outputs against task constraints, including the labels and values on a technical chart.