The default model in this repository is SAVER-SIS (main model), with an optional SAVER-RL (RES) extension.

SAVER follows three core principles:

• Use vision only when the current entity (MNER) or marked entity pair (MRE) is likely to be visually groundable.
• When vision is activated, acquire only a small and complementary multi-image evidence set.
• Use a unified scoring head to combine text and optional visual evidence.

The full pipeline has four stages:

1. Encoding: text encoder produces token/span representations; vision encoder first produces global image vectors.
2. CGG (Conformal Groundability Gate): unit-level routing decision for whether visual evidence is needed.
3. Evidence Constructor (SIS or RES) + Set Transformer: after activation, select up to K images and aggregate region evidence.
4. Energy-Inspired Joint Scoring: unified scoring for MNER (span/type) or MRE (relation).

• Input: text + image set.
• Output: entity spans and entity types.
• SAVER performs CGG and evidence selection at candidate-span granularity.

• Input: text with a marked head/tail entity pair + image set.
• Output: relation label of the marked pair.
• SAVER performs pair-level routing and evidence selection (pair gate is derived from entity gates by default).

• Computes groundability score g(u) using global image vectors and text-image similarity statistics.
• Uses threshold-based hard routing at inference: γ(u)=1[g(u)≥τ].
• Chooses τ on a calibration split via a Clopper-Pearson upper-bound constraint under target risk α.

• When gate is active, selects at most K images from N candidates.
• Objective balances relevance and coverage/diversity.
• Uses greedy approximation (1-1/e) for efficient selection.

• Formulates evidence acquisition as sequential decision making with a STOP action.
• Uses cost-aware rewards with CGG-based action masking.
• Reported as an extension in ablations; SIS remains the default model in main tables.

• Uses standard cross-entropy training (energy notation is used for unified formulation).
• Combines task score, text-vision consistency term, and gate sparsity term.

• MRE: MNRE, MRE-MI
• MNER: Twitter-2015, Twitter-2017, MNER-MI, MNER-MI-Plus

Core dataset scales used in the main text:

• MRE: micro-F1 / Precision / Recall
• MNER: strict entity-level F1 (boundary + type)
• Selectivity: Risk-Activation-Coverage curves, AURC, ActCov@0.10
• Efficiency: FLOPs/sample, end-to-end P90 latency

pip install -r requirements.txt

Place data under data/NER_data, or update data paths in run.py.

Use links in the project documentation or prepare data with the expected directory structure.

Note: SAVER supports both single-image and multi-image inputs. In multi-image settings, SIS/RES builds compact evidence subsets only when gate activation occurs.

bash run_twitter15.sh
bash run_twitter17.sh

bash run_re_task.sh

python -u run.py \
  --dataset_name="MRE" \
  --bert_name="bert-base-uncased" \
  --seed=1234 \
  --only_test \
  --max_seq=80 \
  --use_prompt \
  --prompt_len=4 \
  --sample_ratio=1.0 \
  --load_path='your_re_ckpt_path'

• This README now presents SAVER as the primary method and main result set.
• If updated SAVER architecture/risk-coverage figures are available, replace placeholders in this README accordingly.

• Twitter15/Twitter17 data processing follows UMT.
• MNRE data sourcing follows MEGA.
• Early implementation inspirations include HVPNeT.