fix: render # and $ tight against the following token in to_string() and stringify! - #23255

Addresses the second report in #18571.

Root cause

crates/proc-macro-srv/src/token_stream.rs:484 decides whether to emit a space after a punct purely from its jointness:

TokenTree::Punct(Punct { ch, joint, span: _ }) => {
    *emit_whitespace = !*joint;
    write!(f, "{}", *ch as char)?;
}

crates/tt/src/lib.rs:738 does the same thing for tt::pretty, which backs the stringify! builtin at crates/hir-expand/src/builtin/fn_macro.rs:195.

# and $ are prefix sigils, not binary operators. The token after them is normally an ident or a delimiter, not another punct, so jointness is Alone and both renderers insert a space. rustc never does. Jointness cannot express this, because it describes whether two puncts fuse into one compound operator, and #ty is not a compound operator.

Why it is a correctness bug and not a formatting preference

The reporter in #18571 (comment) has a proc macro that calls input.to_string() and does textual replacement of #ty and #inner. Under rustc the string contains #ty, the replacement fires, and real code comes out. Under rust-analyzer the string contains # ty, the replacement never fires, and the macro emits its own template verbatim. rust-analyzer then type checks impl From<#inner<#ty>> and reports cannot define inherent impl on foreign type plus unexpected token in input on code that cargo check accepts. The diagnostics in that report are downstream of this one space.

stringify! has the same problem for a simpler reason. Its result is an observable string literal, so any divergence from rustc is directly wrong.

Evidence

rustc is the only oracle that matters here and it is trivially available, so I built one. A real proc macro compiled by the real compiler:

#[proc_macro]
pub fn show(input: TokenStream) -> TokenStream {
    format!("{:?}", input.to_string()).parse().unwrap()
}

I ran a 43 case corpus through that under rustc 1.95.0 and through TokenStream::from_str(..).to_string() on e96ea7a5, then diffed. Only 5 of 43 cases matched. Most of the divergence is harmless whitespace that re-lexes identically. The sigil cases are not:

rustc: #inner<#ty>                        RA: # inner <# ty >
rustc: #outer::#ty_pc(value)              RA: # outer ::# ty_pc (value)
rustc: impl From<#inner<#ty>> for #outer  RA: impl From <# inner <# ty >> for # outer
rustc: ##a                                RA: ## a
rustc: #a #b #c                           RA: # a # b # c
rustc: #0                                 RA: # 0
rustc: $x                                 RA: $ x
rustc: $crate::foo                        RA: $ crate :: foo
rustc: $($y),*                            RA: $($ y),*

Same for stringify!, checked directly against rustc 1.95.0:

stringify!(#inner<#ty>)     rustc: "#inner<#ty>"     RA: "# inner <# ty >"
stringify!($x)              rustc: "$x"              RA: "$ x"

The fix

Suppress the trailing space after # and $ in both renderers, independently of jointness. Two lines plus comments.

Test proof

Three new or corrected assertions fail on the parent commit and pass on this one. Reverting only the two fix hunks and keeping the tests:

---- token_stream::tests::rustc_parity_prefix_sigils stdout ----
assertion `left == right` failed: rendering of `#a #b #c` diverges from rustc
  left: "# a # b # c"
 right: "#a #b #c"

---- token_stream::tests::doc_comment_from_str stdout ----
assertion `left == right` failed
  left: "# [doc = \" foo\"]"
 right: "#[doc = \" foo\"]"

test result: FAILED. 1 passed; 3 failed; 0 ignored; 0 measured; 19 filtered out

---- macro_expansion_tests::builtin_fn_macro::test_stringify_expand_prefix_sigils ----
fn main() {
    "# inner <# ty >";
    "$ x";
}
test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 503 filtered out

Restoring both hunks:

test token_stream::tests::doc_comment_from_str ... ok
test token_stream::tests::ts_to_string ... ok
test token_stream::tests::rustc_parity_prefix_sigils ... ok
test token_stream::tests::rustc_parity_known_divergences ... ok
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 19 filtered out

test macro_expansion_tests::builtin_fn_macro::test_stringify_expand ... ok
test macro_expansion_tests::builtin_fn_macro::test_stringify_expand_prefix_sigils ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 503 filtered out

Gates. Full proc-macro-srv suite under the CI invocation, cargo test --features in-rust-tree -p proc-macro-srv:

test result: ok. 23 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

cargo test -p tt -p mbe -p hir-def -p syntax-bridge:

test result: ok. 504 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

cargo fmt --check clean. cargo clippy -p tt -p hir-def -p proc-macro-srv --all-targets produces 5 warnings in proc-macro-srv, all pre-existing, none on changed lines.

The existing test encoded the bug

doc_comment_from_str asserted # [doc = " foo"]. rustc renders /// foo as #[doc = r" foo"]. I corrected the #[ half. The literal kind still differs, we produce a normal string where rustc produces a raw one. That is a separate divergence and I left it alone.

What this does not fix, honestly

The original report by @feois is a different bug. feois/rust-enumeration has no [dependencies] and no proc-macro = true, so it is macro_rules! only and never reaches TokenStream::to_string(). Nothing here touches it. #18571 is two unrelated bugs sharing a title, and the first one still has no root cause. It probably deserves its own issue.

I have not written an end to end test that loads a real dylib whose macro does string replacement. The proof above is at the renderer, plus the rustc oracle.

The remaining 30-odd whitespace divergences from rustc are untouched. rustc_parity_known_divergences records eight representative ones so they are visible in the tree and any future change to them shows up as a diff instead of silently. That test asserts both the current output and that it still differs from rustc, so it fails loudly in either direction.

Question for a maintainer

Do you want full space_between parity with rustc_ast_pretty, or is this sigil fix the right scope? Full parity is a much larger change and would churn expectations across the macro expansion tests. My read is that the sigil case is the only one that is semantically load bearing, since everything else re-lexes to the same tokens, but you know the downstream consumers better than I do. Happy to do the larger port in a follow-up if you want it.