Large language models have the same interface as a Markov chain - they're just predicting likelihoods of next tokens.

You generate text by sampling from that likelihood distribution.