Section 3.5 Generative model architecture

Multimodal models and diffusion

A multimodal model uses more than one kind of signal. A PDF reader may receive extracted text, block coordinates and visual features of the page. A transformer can process vectors derived from small image patches. Evaluate its output against the structure of the source document.

Read a table from an image

The value “20” is unusable without its row and column. When extracting a scanned timetable, check the link between each cell, header and page, then inspect rows with merged cells. Compare with plain-text extraction and OCR that preserves layout. A multimodal model can complement those methods on irregular pages. A correctly formatted output can still point to the wrong cell.

A page is divided into regions and converted into visual vectors while the instruction is tokenized. Both representations feed an output.
Figure 3.5. Diagram of one architectural family. Position information helps associate text with a region of the page.

Generate an image by denoising

In one family of diffusion models, training adds noise to observations at different levels. A network learns an operation that helps reverse the process. During generation, the algorithm starts with noise and applies successive denoising steps, optionally guided by text. Some architectures work in a compressed latent space. The mechanism differs from emitting text tokens left to right. Ho, Jain and Abbeel describe a foundational formulation. In both cases, test plausible outputs against task constraints, including the labels and values on a technical chart.