Proposal

Problem statement

I need to read a large technically text file that potentially has new lines, but it's more like a markup of text and reading it line by line is wasteful, breaks apart markup structure, and line presence is not guaranteed. Ideally I'd read it character by character as an iterator - making a streaming shift-reduce parser.

Motivating examples or use cases

Ideally, we can have

let file = File::open("big_file").expect("Unable to open file.");
let mut reader = BufReader::new(file);
let mut parser = Parser::new();
for c in reader.chars() {
    parser.shift_reduce(c)
}

Solution sketch

// std/src/io/mod.rs
// ...
// I looked at str::Chars and I suspect we can't just return it here, because we need to 
/// load the next file chunk once we consume all std::Chars of the contained buffer. 
pub struct Chars<R> {
    inner: R
}
// ...
pub trait BufRead: Read {
        pub fn chars(self) -> Chars<Self> {
               Chars(self)
        }
}
// ...
impl Iterator for Chars<R> where R:BufRead {
    type Item = Result<char>;

    fn next(&mut self) -> Option<Self::Item> {
        // if the buffer is empty, fill it up
        // read buffer as UTF8. If invalid code point return Err() and go to  the next byte
        // otherwise return Ok(char_
       // if reached the end of buffer, refill it. 
        // if EOF return None forever.
    }
}

Alternatives

I've looked around and I didn't find any crates that implement anything like this. The functionality seems to be a bit too simple to turn it into a standalone crate.

Right now BufReader provides buffer() -> &[u8], which I can convert to Utf8Chunks, but none of the mutable buffer functionality is exposed, so I cant fill_buffer as needed by the Utf8Chunk data. Module buffered is private too so I can't make my own BufCharReader. To parse arbitrary UTF8 text, we need at least a 4-byte buffer, so a BufReader seemed a perfect fit for this.

Links and related work

In C++ Visual Studio There's a way to stream utf8 chars, but for GCC this requires a 3rd party library.
Anything remotely-related I found in Rust Internals is about UTF8 prefix inspection, which can somewhat allow this ACP to use a 4-byte buffer, but since BufReader's default buffer size is 8Kb, I think it's fine to use reader.buffer().utf8_chunks() to implement this.