You can also be smart about memory with lexing for great winnage. Have a struct for your tokens that has an enum for your token type and either pointer or indices or a string_view (or a &str but lol lotsa luck with the borrow checker). You can then have a vector of your token structs for fast allocation and iteration and you have a slice for the token back into the original input, no substring copying.
Yes, the C FE I write (in C) does exactly this and othen utputs in one pass a flattened intermediate code. I did not see at as AST because it does semantic analysis during parsing, but it sitll has all the information.
