Tracking Issue for remainder methods for str split iterators

View all comments

This is a tracking issue for the methods implemented in #75265 and #82570, which allow viewing the remainder of the underlying string in str-split iterators. E.g.:

let mut split = "Mary had a little lamb".split(' '); 
assert_eq!(split.remainder(), Some("Mary had a little lamb"));

_ = split.next();
assert_eq!(split.remainder(), Some("had a little lamb"));

split.by_ref().for_each(drop);
assert_eq!(split.remainder(), None);

The feature gates for the issue are #![feature(str_split_remainder)] (for most split iterators), #![feature(str_split_inclusive_remainder)] (for SplitInclusive) and #![feature(str_split_whitespace_remainder)] (for SplitWhitespace and SplitAsciiWhitespace).

Public API

// mod core::str
impl<'a, P: Pattern<'a>> Split<'a, P> {
    fn remainder(&self) -> Option<&'a str>;
}

// And the same for
// - `RSplit<'a, P>`
// - `SplitTerminator<'a, P>`
// - `RSplitTerminator<'a, P>`
// - `SplitN<'a, P>`
// - `RSplitN<'a, P>`
// - `SplitWhitespace<'a>`
// - `SplitAsciiWhitespace<'a>`
// - `SplitInclusive<'a, P>`

Steps

Implement the feature
Adjust documentation (see instructions on rustc-dev-guide)
Stabilization PR (see instructions on rustc-dev-guide)

Unresolved Questions

Should empty strings point to the end of the haystack, or they can be arbitrary constants? (see Add str::{Split,RSplit,SplitN,RSplitN,SplitTerminator,RSplitTerminator,SplitInclusive}::as_str methods #75265 (comment)) None is now returned instead of an empty string, making this irrelevant

Implementation history

Add str::{Split,RSplit,SplitN,RSplitN,SplitTerminator,RSplitTerminator,SplitInclusive}::as_str methods #75265 — implemented as_str methods on core::str::{Split,RSplit,SplitN,RSplitN,SplitTerminator,RSplitTerminator,SplitInclusive}
Add as_str method for split whitespace str iterators #82570 — implemented as_str methods on core::str::{SplitWhitespace, SplitAsciiWhitespace}
Split*::as_str refactor #95644 — renamed as_str=>remainder and changed return type to Option<&str>

Looking at its name it isn't immediately clear what it returns. Wouldn't remainder or something along that line be a more self-explanatory name?

The name as_str was chosen to be in line with Chars::as_str

Can this be added to SplitWhitespace and SplitAsciiWhitespace?

@TyPR124

@TyPR124 as noted in the implementation PR discussion it is possible, but not trivial.

Both Split*Whitespace are using iterators internally which doesn't allow to access it's inner parts (^1, ^2). It's possible to 'solve' this problem either by making the fields pub(crate::str::iter) (tbh it's a hack) or providing methods on iterator adaptors that allow accessing inner iterators (downside: there are lots of adaptors to change).

I didn't want to complicate the original PR with this, so didn't include anything like this. I can now make a new PR that'll add the same methods to Split*Whitespace, thanks for reminding about it!

Update: #82570 was merged, so now SplitWhitespace and SplitAsciiWhitespace has as_str method too!

I just had a case we're i needed to split a str: "tag:value" into a tag: &str and a value: Option<&str>.
I know i can just use Some(iter.as_str()).filter(str::is_empty) to get the Option, but maybe there could be another method named as_opt_str or remainder. That method would return None if calling next would also return None.
What do you think?

Note that split iterators could yield an empty str as their final item, so as_opt_str could return Some("") while your code with always filter that case.

That's true, to be correct the above code should look like this:

let s = iter.as_str();
iter.next().map(|_| s)

Would be nice to have as_opt_str.

I just had a case we're i needed to split a str: "tag:value" into a tag: &str and a value: Option<&str>.

That sounds like a use case for the recently stabilized split_once.

What's the status wrt stabilisation?

I noticed that SplitWhitespace::remainder() has some counter-intuitive behaviour if tokens are separated by multiple whitespace characters. The remainder() method includes all but the first whitespace character in the returned string:

let s = "foo \tbar";
let mut it = s.split_whitespace();
it.next().unwrap();
assert_eq!(it.remainder(), Some("\tbar"));

I would expect it to either include all of the whitespace or none of it (though I do understand why it behaves the way it does). This behaviour, if intended, should at least be documented.

I needed a functionality of "split the string once on X or Y, whichever is first", and I need to know whether X or Y was first.

I hoped I could use str.split(|c: char| c == X || c == Y) and then inspect the iter.remainder() to see whether X or Y was there, but I was disappointed to learn that the remainder is after the next split, rather than immediately after the previously returned substring.

Rollup merge of rust-lang#107464 - WaffleLapkin:all_that_remains_of_l…

Unrolled build for rust-lang#107464

Could this work with peekable?

https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=73dea5523564c607bef9c4b3a93a2169

I think as_str would be a fine name - I personally just expected it to exist since I am familiar with as_str and as_slice on other iterators.

I feel like the Option may be overcomplicating it. I expected it to just return an empty string as in &string[string.len()..] == "". I think the reasoning for Option doesn't apply for split_inclusive() at least. Not having an Option would allow for simpler code when an empty string is not a special case.

I feel like the Option may be overcomplicating it. I expected it to just return an empty string as in &string[string.len()..] == "". I think the reasoning for Option doesn't apply for split_inclusive() at least. Not having an Option would allow for simpler code when an empty string is not a special case.

I think returning None is more convenient. Assume you got the remainder as iter.as_str().

None to empty string: iter.as_str().unwrap_or_default()
empty string to None: Some(iter.as_str()).filter(|s| !s.is_empty())
(identity conversion: iter.as_str())

I don't remember if there is simpler or shorter way of doing Some(val).filter(pred) without binding to a variable, but from this comparison I think returning Option<&str> from the method is more generic and straightforward to use in any situation.

Here, is another use case of remainder function:
I am trying to implement a Scanner

pub struct Scanner<R> {
    reader: R,
    buffer: Vec<u8>,
    iter: std::str::SplitAsciiWhitespace<'static>,
}

impl<R: std::io::BufRead> Scanner<R> {
    pub fn new(reader: R) -> Self {
        Self {
            reader,
            buffer: vec![],
            iter: "".split_ascii_whitespace(),
        }
    }

    pub fn next<T: std::str::FromStr>(&mut self) -> T {
        loop {
            if let Some(token) = self.iter.next() {
                return unsafe { token.parse().unwrap_unchecked() };
            }
            self.buffer.clear();
            self.reader.read_until(b'\n', &mut self.buffer).unwrap();
            self.iter = unsafe {
                let slice = std::str::from_utf8_unchecked(&self.buffer);
                std::mem::transmute(slice.split_ascii_whitespace())
            }
        }
    }

    pub fn next_line<T: std::str::FromStr>(&mut self) -> T {
        if self.iter.nth(0).is_none() {
            self.buffer.clear();
            self.reader.read_until(b'\n', &mut self.buffer).unwrap();
        }
        self.iter = "".split_ascii_whitespace();
        let result = std::mem::take(&mut self.buffer);
        unsafe {
            let s = std::str::from_utf8_unchecked(&result);
            s.trim().parse().unwrap_unchecked()
        }
    }
}

here remainder function will boil down the code for the next_line function in few lines

I have a use case for this where I want to split a file into a header and a footer, where the end of the header is determined by seeing a line that matches some predicate. I don't particularly care whether the remainder is empty or not, so in this case the Option would be mapped to a an empty string.

I've landed on this issue multiple times after looking for a function like split_once_inclusive.

Navigation Menu

Repository navigation

Tracking Issue for `remainder` methods for `str` split iterators #77998

Public API

Steps

Unresolved Questions

Implementation history

8 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Tracking Issue for remainder methods for str split iterators #77998

Description

Public API

Steps

Unresolved Questions

Implementation history

Activity

SkiFire13 commented on Oct 19, 2020

WaffleLapkin commented on Oct 20, 2020

TyPR124 commented on Feb 20, 2021

WaffleLapkin commented on Feb 20, 2021

WaffleLapkin commented on Mar 20, 2021

soerenmeier commented on Apr 14, 2021

SkiFire13 commented on Apr 14, 2021

soerenmeier commented on Apr 14, 2021

m-ou-se commented on May 4, 2021

8 remaining items

Piturnah commented on Jun 29, 2023

smarnach commented on Aug 5, 2023

kornelski commented on Oct 21, 2023

lkurcak commented on Jul 2, 2024

camsteffen commented on Mar 11, 2025

lo48576 commented on Mar 11, 2025

sumitpsm commented on Nov 6, 2025

cramertj commented on Jun 2, 2026

sffc commented on Sep 24, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Tracking Issue for `remainder` methods for `str` split iterators #77998