Understanding TLV-CoRe: Collaborative Representation Learning for Tactile-Language-Vision Alignment

Introduction

Tactile perception is an underrated part of robotic intelligence. Vision can tell a robot what an object looks like, and language can describe what humans expect from it, but touch gives information that is difficult to infer from pixels alone: roughness, hardness, texture, deformation, slipperiness, and fine-grained material cues.

The paper Collaborative Representation Learning for Alignment of Tactile, Language, and Vision Modalities studies this problem from a multimodal representation-learning perspective. It proposes TLV-CoRe, a CLIP-based tactile-language-vision collaborative representation learning method.

Paper links:

• Paper HTML: ar5iv:2511.11512
• arXiv abstract: arXiv:2511.11512
• DOI: 10.48550/arXiv.2511.11512

According to arXiv, the paper was first submitted on November 14, 2025 and the current listed version is v5, revised on February 1, 2026. The authors are Yiyun Zhou, Mingjing Xu, Jingwei Shi, Quanjiang Li, and Jingyuan Chen.

The central problem is not simply “how do we add a tactile encoder to CLIP?” The harder question is:

How can tactile, language, and vision learn a shared representation when tactile sensors themselves are heterogeneous and not fully standardized?

TLV-CoRe answers this with three ideas:

1. Sensor-Aware Modulator (SAM): adapt tactile features from different sensors into a more unified representation space.
2. Tactile-irrelevant decoupled learning: reduce sensor-specific noise and force the tactile encoder to focus on task-relevant information.
3. Unified Bridging Adapter (UBA): let tactile, language, and vision communicate through a shared adapter space before final contrastive alignment.

Why Tactile-Language-Vision Alignment Is Hard

Vision-language alignment has a relatively mature recipe. CLIP-style models learn from paired images and text by pulling matching pairs together and pushing mismatched pairs apart. Extending this idea to touch is tempting, but tactile data introduces several extra difficulties.

First, tactile sensors are not standardized. GelSight-like and DIGIT-like sensors may capture the same physical object with different illumination, camera geometry, color response, elastomer properties, and contact artifacts. This means a tactile image can contain both useful object information and sensor-specific style.

Second, tactile images can be misleading. Two different touched objects may produce visually similar tactile patterns if the sensor style dominates the image. Conversely, the same object may look different under different sensors or contact conditions.

Third, existing tactile-language-vision methods often align modalities at the final embedding level but do not explicitly encourage intermediate communication among tactile, language, and vision branches. The paper argues that this limits deep tri-modal fusion.

So the target is not only cross-modal alignment. The target is sensor-agnostic, collaborative, and stable representation learning.

Related Work Context

The paper positions TLV-CoRe among several tactile representation learning directions.

CLIP-based tactile-language-vision learning includes methods such as TLV-Link, AnyTouch, UniTouch, and related approaches. Their strength is that they can reuse powerful vision-language priors. Their weakness is that tactile data has sensor-specific structure that ordinary CLIP-style alignment does not automatically remove.

Custom tactile representation models can be built specifically for tactile data, but they make fair comparison harder because the base model, training schedule, batch size, and evaluation protocol often differ.

General multimodal alignment methods such as ImageBind-style work show that many modalities can be projected into a shared embedding space. However, tactile sensing is still less explored than image, text, audio, video, or 3D point clouds.

TLV-CoRe is best understood as a CLIP-based method that tries to keep the comparison setting controlled while adding tactile-specific and tri-modal-specific modules.

High-Level Architecture

The model uses three modality branches:

• Tactile path: tactile input -> tactile encoder -> SAM -> UBA -> tactile representation.
• Vision path: image input -> visual encoder -> UBA -> visual representation.
• Language path: text input -> language encoder -> UBA -> language representation.
• Alignment head: tactile, vision, and language representations are trained with pairwise contrastive alignment in a shared embedding space.

The tactile branch has extra work to do because sensor identity can become a shortcut. SAM handles sensor-aware feature adaptation, while decoupled learning discourages the encoder from preserving tactile-irrelevant sensor information.

The UBA module is inserted into the modality encoders to make the three branches collaborate through a shared adapter space. This is the part that changes the system from only “aligning final embeddings” to “sharing intermediate representation structure.”

Sensor-Aware Modulator

Let a tactile input be $x^T$, and let the tactile encoder produce a feature:

$$ h^T = E_T(x^T). $$

The Sensor-Aware Modulator estimates a sensor-related routing vector:

$$ r(h^T)=\operatorname{softmax}(W_r h^T). $$

For a sample from sensor $s$, the paper uses the corresponding routing component to modulate the tactile feature. A simplified way to read the update is:

$$ h^T_{\text{mod}} = h^T + [r(h^T)]_s h^T. $$

The intuition is that the tactile encoder should not treat all sensors as if they produced identical image distributions. SAM gives the model a way to adapt to sensor-specific statistics while still pushing the final representation toward a unified space.

This is a practical compromise:

• ignoring sensor identity is too naive, because sensors really do produce different images;
• overfitting to sensor identity is also dangerous, because the model may learn sensor style instead of object properties;
• SAM gives the network an explicit mechanism to handle this variation.

Tactile-Irrelevant Decoupled Learning

SAM handles sensor-aware adaptation, but the model also needs to prevent the tactile encoder from encoding too much sensor-specific noise. TLV-CoRe uses a decoupled learning objective with sensor centroids.

For sensor label $s$ and tactile representation $h^T$, define a sensor classifier over centroids $\lbrace c_s\rbrace$:

$$ p(s \mid h^T) = \frac{ \exp(\langle h^T,c_s\rangle/\tau) }{ \sum_{s'} \exp(\langle h^T,c_{s'}\rangle/\tau) }. $$

The sensor classification loss is:

$$ \mathcal{L}_{DL} = - \mathbb{E}_{(x^T,s)} \left[ \log p(s \mid h^T) \right]. $$

The important trick is the training direction. With gradient reversal, the sensor classifier learns to recognize sensor identity, while the tactile encoder is pushed to remove information that makes sensor identity too easy to predict. In plain language:

The auxiliary classifier asks, “which sensor produced this tactile feature?”, while the encoder learns to make that question harder unless the information is useful for the task.

This is why the paper calls it tactile-irrelevant decoupling. The goal is not to erase all sensor information, but to suppress the part that hurts cross-sensor generalization.

Unified Bridging Adapter

The Unified Bridging Adapter is the most important tri-modal collaboration module. For each modality $m \in \lbrace T,V,L\rbrace$, the encoder feature $h^m$ is projected down, passed through a shared adapter space, projected back, and added residually:

$$ z^m_{\text{shared}} = W_{\text{sh}} \left( W^m_{\text{down}} h^m \right), $$

$$ \Delta h^m = W^m_{\text{up}} z^m_{\text{shared}}, \quad h^m_{\text{aligned}} = h^m + \Delta h^m. $$

This structure has two useful properties.

First, the down and up projections are modality-specific. Tactile, vision, and language do not have to enter the adapter in exactly the same way.

Second, the middle mapping $W_{\text{sh}}$ is shared. That shared bottleneck gives the model a common route through which the three modalities can exchange alignment-friendly structure.

This is different from only aligning the final embeddings. UBA encourages intermediate representations to become more compatible before the final contrastive objective is applied.

Cross-Modal Contrastive Objective

After the representations are aligned, TLV-CoRe applies pairwise symmetric contrastive learning across tactile, vision, and language.

For a tactile-vision pair, a typical symmetric InfoNCE loss has the form:

$$ \mathcal{L}_{T,V} = - \frac{1}{N} \sum_{i=1}^{N} \left[ \log \frac{\exp(\operatorname{sim}(t_i,v_i)/\tau)} {\sum_{j=1}^{N}\exp(\operatorname{sim}(t_i,v_j)/\tau)} + \log \frac{\exp(\operatorname{sim}(v_i,t_i)/\tau)} {\sum_{j=1}^{N}\exp(\operatorname{sim}(v_i,t_j)/\tau)} \right]. $$

The full supervised contrastive alignment loss combines the three modality pairs:

$$ \mathcal{L}_{SCL} = \mathcal{L}_{T,V} + \mathcal{L}_{T,L} + \mathcal{L}_{V,L}. $$

The total training objective is:

$$ \mathcal{L}_{total} = \mathcal{L}_{SCL} + \lambda_{DL} \mathcal{L}_{DL}. $$

So the system has two coupled goals:

1. bring matched tactile, vision, and language examples together;
2. reduce sensor-specific tactile shortcuts that damage generalization.

RSS Evaluation Framework

One strong part of the paper is that it does not evaluate only one accuracy table. It proposes an RSS framework:

1. Robustness: does the tactile representation generalize across sensors and datasets?
2. Synergy: does alignment help the other modalities instead of weakening them?
3. Stability: does performance remain stable under different training batch sizes and contrastive-learning conditions?

This is important because tactile multimodal papers can otherwise be difficult to compare. If one method uses a stronger base model, different batch size, or different downstream protocol, the method-level conclusion becomes unclear.

RSS tries to make the comparison more controlled. The paper evaluates intra-sensor, cross-sensor, and multi-sensor generalization, then adds modal cross-evaluation tasks and batch-size studies.

Experimental Takeaways

The experiments use real-world tactile datasets including TAG, SSVTP, TVL, Octopi, and TacQuad, and downstream evaluation datasets such as ObjectFolder 1.0, ObjectFolder 2.0, and Feeling of Success.

The main message is that TLV-CoRe improves tactile representation quality while using a small number of trainable parameters. In the reported main comparison, TLV-CoRe uses about 0.30% trainable parameters, compared with larger percentages for several competing CLIP-based baselines.

A few results are especially informative:

• On TAG-trained evaluation, TLV-CoRe reports strong material, roughness, and hardness performance, and also improves downstream generalization to ObjectFolder and Feeling of Success tasks.
• In modal cross-evaluation, TLV-CoRe gives strong improvements on image classification benchmarks such as CIFAR-10 and CIFAR-100 when the aligned representation is transferred through the visual side.
• In multi-sensor settings such as GelSight plus DIGIT, TLV-CoRe improves over baselines, suggesting that the sensor-aware and sensor-decoupling design is doing useful work.
• In batch-size experiments, TLV-CoRe is reported as smoother and more stable than methods that degrade or plateau under larger batch sizes.

The ablation results also fit the architecture:

• removing SAM hurts robustness;
• removing tactile-irrelevant decoupled learning causes a larger drop because the encoder keeps more sensor-specific noise;
• removing UBA is most damaging to tri-modal collaboration because the shared intermediate adapter is the main cross-modal communication path.

Theoretical Intuition

The paper includes convergence, robustness, synergy, and stability analysis. The exact proofs are not the main focus of this post, but the intuition is useful.

Convergence. A shared adapter can improve the conditioning of the joint optimization problem. If the shared representation space is better aligned, gradient updates across modalities become less conflicting.

Robustness. Sensor-invariance decoupling reduces the amount of sensor-specific information in tactile features. This can reduce gradient variance and improve cross-sensor generalization.

Synergy. UBA gives each modality a route to absorb useful information from the others. In ideal cases, tactile, vision, and language features become complementary rather than merely coexisting in the same embedding space.

Stability. Contrastive learning is sensitive to batch size because the number and quality of negative samples change. By reducing sensor-specific shortcuts and using shared adapters, TLV-CoRe aims to keep performance less sensitive to this training choice.

Limitations and Open Questions

The paper is not claiming that tactile-language-vision alignment is solved. Several questions remain open.

First, tactile datasets are still much smaller and less standardized than image-text datasets. A method that works well on current datasets still needs broader validation on real robotic platforms and real-time manipulation tasks.

Second, the RSS framework is useful, but no evaluation framework is complete. Robustness, synergy, and stability are important axes, but deployment also cares about latency, data collection cost, contact policy, sensor wear, and closed-loop control.

Third, language supervision can be incomplete. Text can describe object properties, but it may not capture the full physics of contact, force, friction, and deformation.

Finally, UBA introduces a useful shared communication path, but the best depth, placement, and capacity of the adapter may depend on the base encoder and the tactile sensor family.

Conclusion

TLV-CoRe is best understood as a carefully designed bridge between tactile sensing and CLIP-style multimodal learning. Its key contribution is not just adding another modality to a contrastive framework. It recognizes that tactile data has sensor-specific nuisance structure, then combines sensor-aware modulation, adversarial decoupling, and shared tri-modal adapters to improve alignment.

The paper’s main design lesson is clear:

For tactile-language-vision learning, the model must align modalities while also preventing sensor identity from becoming the easiest shortcut.

That makes TLV-CoRe a useful reference for future tactile representation learning work, especially if the goal is not only high in-domain accuracy but also cross-sensor robustness, cross-modal synergy, and stable training behavior.