I think you need to consider conditional statistics. "What are high probability options for the next word, given that the text i'm working on starts with the words please rhyme, and that the text 10 words ago was 'sun' and the text 20 words ago was 'fun'?" How it knows which parts of the text are relevant to condition on is the attention mechanism which is like "what is the probability this word is important to how to finish this sentence?". Both of these can be extracted from large enough example data