Tabular data and images: why encoder choice is critical

Researchers for the first time systematically evaluated tabular models as encoders in multimodal learning tasks, where a neural network must simultaneously understand tables and images. The traditional approach — using a simple MLP for tables — turned out to be suboptimal.

Why is MLP Insufficient for Tabular Data?

Multimodal learning requires a separate encoder for each data type. If a model works with images and tables simultaneously, it needs specialized networks to process each. For images, powerful CNNs or Vision Transformers have long been used; for text — language models. But for tabular data, most researchers simply took a simple MLP.

Tabular data (database tables, CSV files, spreadsheets) is a known problematic area for deep learning. It's called "the last unsolved lock in machine learning." There are better models for working with tables — for example, specialized architectures based on attention and in-context learning, but they are rarely used as encoders in multimodal contexts.

• For the first time, state-of-the-art tabular models evaluated as encoders in multimodal tasks
• Simple MLP is the de-facto standard in many studies, but far from optimal
• In-Context Learning models rank among the best for tabular data

What Problem Do Better Models Create?

The main challenge is that the best tabular models, especially those based on In-Context Learning, require knowing the target variable (label) to correctly process an example. This creates a practical paradox: how to encode training examples (which have labels during training) and test examples (which have no labels) in the same way?

In-Context Learning models work by looking at a few labeled examples to understand the task. When you apply such a model as an encoder for image-table pairs, a mismatch arises. During training, the model sees labels; during testing, it doesn't. The authors of the study developed methods to solve this problem, adapting several families of In-Context Learning models so they work consistently both during training and testing.

What This Means for Engineers

The research emphasizes that encoder choice is often an underestimated critical factor in multimodal learning. Using more powerful tabular models instead of a simple MLP can significantly improve the quality of the entire system.

This is important for real-world applications where tabular data is often combined with other modalities: customer data in a table + product photo in e-commerce, device parameters table + status visualization in industry, patient table + medical scans in healthcare.

What This Means

Results show that tabular data becomes more competitive in multimodal systems with the right encoder choice. ML engineers can achieve better results if they stop economizing on the quality of the tabular part of the architecture. This is especially relevant in applications where tables and tabular information are critical: fintech, medicine, logistics, finance. The research opens the path toward more thoughtful multimodal model design.