Abstract: We propose SG-XDEAT (Sparsity-Guided Cross-Dimensional and Cross-Encoding Attention with Target-Aware Conditioning), a novel framework designed for supervised learning on tabular data. At its core, SG-XDEAT employs a dual-stream encoder that decomposes each input feature into two parallel representations: a raw value stream and a target-conditioned (label-aware) stream. These dual representations are then propagated through a hierarchical stack of attention-based modules. SG-XDEAT integrates three key components: (i) cross-dimensional self-attention, which captures intra-view dependencies among features within each stream; (ii) cross-encoding self-attention, which enables bidirectional interaction between raw and target-aware representations; and (iii) an Adaptive Sparse Self-Attention (ASSA) mechanism, which dynamically suppresses low-utility tokens by driving their attention weights toward zero—thereby mitigating the impact of noise. Empirical results on multiple public benchmarks show consistent gains over strong baselines, confirming that jointly modeling raw and target-aware views—while adaptively filtering noise—yields a more robust deep tabular learner.

SG-XDEAT Tabular Learning/
├── Supplementary_Material/            # Supplementary resources
│ ├── Experiments/                     # Experimental results
│ │ ├── ASSA/                          # Sparsity Related experiments
│ │ ├── Architectural Comparison/      # Comparison across model architectures
│ │ ├── Input Strategies/              # Different input handling strategies ( Raw & Targeted )
│ │ ├── Performance Comparison/        # Performance benchmarking
│ │
│ ├── Appendix.pdf                     # Appendix
│ ├── ReproducibilityChecklist.pdf     # Checklist
│
├── Figure/                            # Figures & Tables
├── SG_XDEAT-v1.pdf                    # Conference Paper
├── Summary.pdf                        # 3 pages Summary For SG_XDEAT (pdf)
├── Summary.docx                       # 3 pages Summary For SG_XDEAT (docx)
├── README.md                          # Documentation

We use five datasets: AD, CA, HI, GE, and GE.

For each dataset, we evaluate:

• 6 benchmark models
• Our proposed method: SG-XDEAT

The corresponding code is located in Supplementary_Material/Experiments/Performance Comparison.

Note: Our evaluation currently covers mainstream deep learning baselines (e.g., FT-Transformer). We are actively integrating state-of-the-art retrieval-based models (TabR) and foundation models (TabPFN) into our unified framework.

Statistical significance tests (Wilcoxon signed-rank test) and standard deviations over 5 seeds are detailed in the Supplementary_Material/Appendix.pdf.

In this experiment, we evaluate four different model variants:

1. Direct ( CDFC )

   • ** Self-Attention ** on all representations.

2. Cross-Dim only ( CD )

   • Uses only the Cross-Dim component in SG-XDEAT.
   • Consider dependencies among features.

3. Cross-Enc only ( CE )

   • Uses only the Cross-Enc component in SG-XDEAT.
   • Consider dependencies among encodings.

4. Cross-Dim + Cross-Enc ( CD+CE )

   • Uses the ** Cross-Dim & Cross-Enc ** component in SG-XDEAT.
   • Consider both dependencies among features and encodings.

The corresponding code is located in Supplementary_Material/Experiments/Architectural Comparison

In this experiment, we evaluate two types of input strategies:

1. Raw Stream : Direct raw feature values.

2. Targeted Stream : Feature values conditioned on labels (using PLE-T & DecisionTreeEncoder).

   • PLE-T (Piecewise Linear Encoding with Target): Adapts numerical features based on target distribution.
     • On Embeddings for Numerical Features in Tabular Deep Learning
   • Decision Tree Encoder: Leverages tree-based partitions to capture non-linear label dependencies.
     • Winning the KDD Cup Orange Challenge with Ensemble Selection

Note: For Targeted Stream, We ensure the target information is only derived from the training set, maintaining the integrity of the validation/testing phases." We also investigate their combinations, such as DFC and CD+CE.

The corresponding code is located in Supplementary_Material/Experiments/Input Strategies

We construct a synthetic regression benchmark where only a subset of features is informative. Each sample x is a vector of dimension d = 100, with a proportion ρ in {0.5, 0.6, ..., 1.0} of features being informative.

In our model, we enhance the attention mechanism by adding a squared-ReLU activation path, instead of relying solely on softmax.

The corresponding code can be found in Supplementary_Material/Experiments/ASSA.