Understanding TLV-CoRe: Collaborative Representation Learning for Tactile-Language-Vision Alignment

Introduction

Tactile perception is an underrated part of robotic intelligence. Vision can tell a robot what an object looks like, and language can describe what humans expect from it, but touch gives information that is difficult to infer from pixels alone: roughness, hardness, texture, deformation, slipperiness, and fine-grained material cues.

The paper Collaborative Representation Learning for Alignment of Tactile, Language, and Vision Modalities studies this problem from a multimodal representation-learning perspective. It proposes TLV-CoRe, a CLIP-based tactile-language-vision collaborative representation learning method.

Paper links:

• Paper HTML: ar5iv:2511.11512
• arXiv abstract: arXiv:2511.11512
• arXiv DOI: 10.48550/arXiv.2511.11512
• Proceedings version: AAAI 2026

CLIP-style alignment is a natural starting point because it gives a simple geometric target: paired samples should land close together in an embedding space. But tactile data is not just another image domain. A tactile frame is produced by a physical contact process, a sensor housing, a deformable surface, illumination, calibration, and sometimes force or motion history. The method therefore has to align semantic meaning while also filtering out sensor-dependent artifacts.

The central problem is not simply “how do we add a tactile encoder to CLIP?” The harder question is:

How can tactile, language, and vision learn a shared representation when tactile sensors themselves are heterogeneous and not fully standardized?

TLV-CoRe answers this with three ideas:

1. Sensor-Aware Modulator (SAM): adapt tactile features from different sensors into a more unified representation space.
2. Tactile-irrelevant decoupled learning: reduce sensor-specific noise and force the tactile encoder to focus on task-relevant information.
3. Unified Bridging Adapter (UBA): let tactile, language, and vision communicate through a shared adapter space before final contrastive alignment.

Problem Setting: What Is Being Aligned?

Before looking at the architecture, it helps to make the target problem more concrete. In a tactile-language-vision dataset, one training item can be understood as an aligned triplet:

$$ \left(x_i^T, x_i^V, x_i^L\right), $$

where $x_i^T$ is a tactile observation, $x_i^V$ is a visual observation, and $x_i^L$ is a language description or label related to the same object, material, or property. The goal is to learn encoders that map the three inputs into embeddings:

$$ z_i^T=f_T(x_i^T),\quad z_i^V=f_V(x_i^V),\quad z_i^L=f_L(x_i^L). $$

For a good representation, matched embeddings such as $(z_i^T,z_i^V)$ or $(z_i^T,z_i^L)$ should be close, while mismatched examples should be farther away. That sounds similar to vision-language contrastive learning, but tactile data makes the target more subtle.

The tactile image is not only a picture of an object surface. It is also a record of a contact event. Contact force, pose, sensor elastomer, internal camera, lighting position, color response, and calibration can all change the image. Therefore, the representation must keep object-level information such as texture and hardness, while avoiding shortcuts that only identify the sensor or the recording style.

It is also important not to overread the paper. TLV-CoRe is a representation-learning method, not a complete robot manipulation policy. It improves the feature space that later systems can use for recognition, retrieval, classification, or downstream robotic tasks.

This means the target output is not a direct action such as “grasp here” or “push with this force.” Instead, the target is a reusable embedding that makes tactile observations comparable with visual evidence and language descriptions. A better embedding can then support many downstream behaviors, but the policy or controller still has to decide how to act.

Why Tactile-Language-Vision Alignment Is Hard

Vision-language alignment has a relatively mature recipe. CLIP-style models learn from paired images and text by pulling matching pairs together and pushing mismatched pairs apart. Extending this idea to touch is tempting, but tactile data introduces several extra difficulties.

First, tactile sensors are not standardized. GelSight-like and DIGIT-like sensors may capture the same physical object with different illumination, camera geometry, color response, elastomer properties, and contact artifacts. This means a tactile image can contain both useful object information and sensor-specific style.

Second, tactile images can be misleading. Two different touched objects may produce visually similar tactile patterns if the sensor style dominates the image. Conversely, the same object may look different under different sensors or contact conditions.

Third, tactile information is often property-oriented rather than category-oriented. A visual image may quickly reveal “cup”, “cloth”, or “bottle”, while touch may reveal “soft”, “rough”, “slippery”, or “deformable”. These are exactly the properties robots need during contact-rich interaction, but they are also harder to label and compare across datasets.

Fourth, existing tactile-language-vision methods often align modalities at the final embedding level but do not explicitly encourage intermediate communication among tactile, language, and vision branches. The paper argues that this limits deep tri-modal fusion.

So the target is not only cross-modal alignment. The target is sensor-agnostic, collaborative, and stable representation learning.

Related Work Context

The paper positions TLV-CoRe among several tactile representation learning directions.

CLIP-based tactile-language-vision learning includes methods such as TLV-Link, AnyTouch, UniTouch, and related approaches. Their strength is that they can reuse powerful vision-language priors. Their weakness is that tactile data has sensor-specific structure that ordinary CLIP-style alignment does not automatically remove.

This line of work is attractive because it avoids training a multimodal model from scratch. However, reusing CLIP also creates a mismatch: CLIP’s visual encoder was trained on natural images, not contact images produced by an elastomer and internal camera. TLV-CoRe keeps the benefit of pretrained alignment while adding tactile-specific modules where the mismatch is most damaging.

Sensor-transfer tactile learning focuses on the fact that a representation trained on one sensor may fail on another sensor. This is not a small implementation detail. If a method only performs well under one tactile camera, one lighting setup, or one contact protocol, it is difficult to reuse in a broader robotic system.

This is closely related to domain generalization. The model should preserve material and shape cues that survive across sensors, while suppressing cues that merely reveal the sensor family. In tactile learning, that separation is difficult because the sensor is part of the measurement process rather than a harmless style filter.

Intermediate multimodal adapters are useful when the base model is large and expensive to retrain. Adapter-style methods can add a small number of trainable parameters while preserving the pretrained encoder. TLV-CoRe follows this practical direction, but adds a shared bridge so that the three modalities are not isolated until the final loss.

The shared bridge is especially important for tri-modal learning. If tactile, language, and vision branches only meet at the final contrastive loss, each branch can learn mostly private transformations and only align at the end. A shared adapter gives the model an intermediate place to shape features toward a common representation before the final embedding.

Custom tactile representation models can be built specifically for tactile data, but they make fair comparison harder because the base model, training schedule, batch size, and evaluation protocol often differ.

General multimodal alignment methods such as ImageBind-style work show that many modalities can be projected into a shared embedding space. However, tactile sensing is still less explored than image, text, audio, video, or 3D point clouds.

TLV-CoRe is best understood as a CLIP-based method that tries to keep the comparison setting controlled while adding tactile-specific and tri-modal-specific modules.

High-Level Architecture

The model uses three modality branches:

• Tactile path: tactile input -> tactile encoder -> SAM -> UBA -> tactile representation.
• Vision path: image input -> visual encoder -> UBA -> visual representation.
• Language path: text input -> language encoder -> UBA -> language representation.
• Alignment head: tactile, vision, and language representations are trained with pairwise contrastive alignment in a shared embedding space.

The tactile branch has extra work to do because sensor identity can become a shortcut. SAM handles sensor-aware feature adaptation, while decoupled learning discourages the encoder from preserving tactile-irrelevant sensor information.

The UBA module is inserted into the modality encoders to make the three branches collaborate through a shared adapter space. This is the part that changes the system from only “aligning final embeddings” to “sharing intermediate representation structure.”

Figure: a simplified mechanism view of TLV-CoRe. The tactile branch has two special components, SAM and sensor-shortcut decoupling, while all three modalities meet through the UBA shared bridge before contrastive alignment and RSS evaluation.

The diagram should be read from left to right. Tactile observations first pass through sensor-aware adaptation and a decoupling signal that discourages sensor shortcuts. Vision and language use their own encoders. All three streams then enter the shared UBA bridge, where modality-specific projections connect to a common adapter space before the contrastive losses pull matched examples together.

Sensor-Aware Modulator

Let a tactile input be $x^T$, and let the tactile encoder produce a feature:

$$ h^T = E_T(x^T). $$

The Sensor-Aware Modulator estimates a sensor-related routing vector:

$$ r(h^T)=\operatorname{softmax}(W_r h^T). $$

For a sample from sensor $s$, the paper uses the corresponding routing component to modulate the tactile feature. A simplified way to read the update is:

$$ h^T_{\text{mod}} = h^T + [r(h^T)]_s h^T. $$

The intuition is that the tactile encoder should not treat all sensors as if they produced identical image distributions. SAM gives the model a way to adapt to sensor-specific statistics while still pushing the final representation toward a unified space.

This is a practical compromise:

• ignoring sensor identity is too naive, because sensors really do produce different images;
• overfitting to sensor identity is also dangerous, because the model may learn sensor style instead of object properties;
• SAM gives the network an explicit mechanism to handle this variation.

Tactile-Irrelevant Decoupled Learning

SAM handles sensor-aware adaptation, but the model also needs to prevent the tactile encoder from encoding too much sensor-specific noise. TLV-CoRe uses a decoupled learning objective with sensor centroids.

For sensor label $s$ and tactile representation $h^T$, define a sensor classifier over centroids $\lbrace c_s\rbrace$:

$$ p(s \mid h^T) = \frac{ \exp(\langle h^T,c_s\rangle/\tau) }{ \sum_{s'} \exp(\langle h^T,c_{s'}\rangle/\tau) }. $$

The sensor classification loss is:

$$ \mathcal{L}_{DL} = - \mathbb{E}_{(x^T,s)} \left[ \log p(s \mid h^T) \right]. $$

The important trick is the training direction. With gradient reversal, the sensor classifier learns to recognize sensor identity, while the tactile encoder is pushed to remove information that makes sensor identity too easy to predict. In plain language:

The auxiliary classifier asks, “which sensor produced this tactile feature?”, while the encoder learns to make that question harder unless the information is useful for the task.

This is why the paper calls it tactile-irrelevant decoupling. The goal is not to erase all sensor information, but to suppress the part that hurts cross-sensor generalization.

Unified Bridging Adapter

The Unified Bridging Adapter is the most important tri-modal collaboration module. For each modality $m \in \lbrace T,V,L\rbrace$, the encoder feature $h^m$ is projected down, passed through a shared adapter space, projected back, and added residually:

$$ z^m_{\text{shared}} = W_{\text{sh}} \left( W^m_{\text{down}} h^m \right), $$

$$ \Delta h^m = W^m_{\text{up}} z^m_{\text{shared}}, \quad h^m_{\text{aligned}} = h^m + \Delta h^m. $$

This structure has two useful properties.

First, the down and up projections are modality-specific. Tactile, vision, and language do not have to enter the adapter in exactly the same way.

Second, the middle mapping $W_{\text{sh}}$ is shared. That shared bottleneck gives the model a common route through which the three modalities can exchange alignment-friendly structure.

This is different from only aligning the final embeddings. UBA encourages intermediate representations to become more compatible before the final contrastive objective is applied.

Cross-Modal Contrastive Objective

After the representations are aligned, TLV-CoRe applies pairwise symmetric contrastive learning across tactile, vision, and language.

For a tactile-vision pair, a typical symmetric InfoNCE loss has the form:

$$ \mathcal{L}_{T,V} = - \frac{1}{N} \sum_{i=1}^{N} \left[ \log \frac{\exp(\operatorname{sim}(t_i,v_i)/\tau)} {\sum_{j=1}^{N}\exp(\operatorname{sim}(t_i,v_j)/\tau)} + \log \frac{\exp(\operatorname{sim}(v_i,t_i)/\tau)} {\sum_{j=1}^{N}\exp(\operatorname{sim}(v_i,t_j)/\tau)} \right]. $$

The full supervised contrastive alignment loss combines the three modality pairs:

$$ \mathcal{L}_{SCL} = \mathcal{L}_{T,V} + \mathcal{L}_{T,L} + \mathcal{L}_{V,L}. $$

The total training objective is:

$$ \mathcal{L}_{total} = \mathcal{L}_{SCL} + \lambda_{DL} \mathcal{L}_{DL}. $$

So the system has two coupled goals:

1. bring matched tactile, vision, and language examples together;
2. reduce sensor-specific tactile shortcuts that damage generalization.

RSS Evaluation Framework

One strong part of the paper is that it does not evaluate only one accuracy table. It proposes an RSS framework:

1. Robustness: does the tactile representation generalize across sensors and datasets?
2. Synergy: does alignment help the other modalities instead of weakening them?
3. Stability: does performance remain stable under different training batch sizes and contrastive-learning conditions?

This is important because tactile multimodal papers can otherwise be difficult to compare. If one method uses a stronger base model, different batch size, or different downstream protocol, the method-level conclusion becomes unclear.

RSS tries to make the comparison more controlled. The paper evaluates intra-sensor, cross-sensor, and multi-sensor generalization, then adds modal cross-evaluation tasks and batch-size studies.

The three axes also correspond to three deployment concerns. Robustness asks whether the representation survives a change in tactile hardware or dataset. Synergy asks whether adding touch actually helps multimodal understanding rather than becoming a weak auxiliary branch. Stability asks whether the method is usable under realistic contrastive-learning conditions, where batch size and negative samples can change the training dynamics.

Experimental Takeaways

The experiments use real-world tactile datasets including TAG, SSVTP, TVL, Octopi, and TacQuad, and downstream evaluation datasets such as ObjectFolder 1.0, ObjectFolder 2.0, and Feeling of Success.

The main message is that TLV-CoRe improves tactile representation quality while using a small number of trainable parameters. In the reported main comparison, TLV-CoRe uses about 0.30% trainable parameters, compared with larger percentages for several competing CLIP-based baselines.

A few results are especially informative:

• On TAG-trained evaluation, TLV-CoRe reports strong material, roughness, and hardness performance, and also improves downstream generalization to ObjectFolder and Feeling of Success tasks.
• In modal cross-evaluation, TLV-CoRe gives strong improvements on image-classification benchmarks such as CIFAR-10 and CIFAR-100 when probing those vision-task datasets with the tactile encoder.
• In multi-sensor settings such as GelSight plus DIGIT, TLV-CoRe improves over baselines, suggesting that the sensor-aware and sensor-decoupling design is doing useful work.
• In batch-size experiments, TLV-CoRe is reported as smoother and more stable than methods that degrade or plateau under larger batch sizes.

The ablation results also fit the architecture:

• removing SAM hurts robustness;
• removing tactile-irrelevant decoupled learning causes a larger drop because the encoder keeps more sensor-specific noise;
• removing UBA is most damaging to tri-modal collaboration because the shared intermediate adapter is the main cross-modal communication path.

Theoretical Intuition

The paper includes convergence, robustness, synergy, and stability analysis. The exact proofs are not the main focus of this post, but the intuition is useful.

Convergence. A shared adapter can improve the conditioning of the joint optimization problem. If the shared representation space is better aligned, gradient updates across modalities become less conflicting.

Robustness. Sensor-invariance decoupling reduces the amount of sensor-specific information in tactile features. This can reduce gradient variance and improve cross-sensor generalization.

Synergy. UBA gives each modality a route to absorb useful information from the others. In ideal cases, tactile, vision, and language features become complementary rather than merely coexisting in the same embedding space.

Stability. Contrastive learning is sensitive to batch size because the number and quality of negative samples change. By reducing sensor-specific shortcuts and using shared adapters, TLV-CoRe aims to keep performance less sensitive to this training choice.

Limitations and Open Questions

The paper is not claiming that tactile-language-vision alignment is solved. Several questions remain open.

First, tactile datasets are still much smaller and less standardized than image-text datasets. A method that works well on current datasets still needs broader validation on real robotic platforms and real-time manipulation tasks.

Second, the RSS framework is useful, but no evaluation framework is complete. Robustness, synergy, and stability are important axes, but deployment also cares about latency, data collection cost, contact policy, sensor wear, and closed-loop control.

Third, language supervision can be incomplete. Text can describe object properties, but it may not capture the full physics of contact, force, friction, and deformation.

Finally, UBA introduces a useful shared communication path, but the best depth, placement, and capacity of the adapter may depend on the base encoder and the tactile sensor family.

Conclusion

TLV-CoRe is best understood as a carefully designed bridge between tactile sensing and CLIP-style multimodal learning. Its key contribution is not just adding another modality to a contrastive framework. It recognizes that tactile data has sensor-specific nuisance structure, then combines sensor-aware modulation, adversarial decoupling, and shared tri-modal adapters to improve alignment.

The paper’s main design lesson is clear:

For tactile-language-vision learning, the model must align modalities while also preventing sensor identity from becoming the easiest shortcut.

That makes TLV-CoRe a useful reference for future tactile representation learning work, especially if the goal is not only high in-domain accuracy but also cross-sensor robustness, cross-modal synergy, and stable training behavior.