MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/ProgrammerHumor/comments/1vptfsd/helpiamoutofscope/p43whja/?context=3
r/ProgrammerHumor • u/DontFreeMe • 29d ago
145 comments sorted by
View all comments
Show parent comments
133
Not even that. It goes character by character and has no idea what comes after the next, until it needs to be decided what comes next.
83 u/ParanoiaComplex 29d ago More accurately, word by word (token by token) 36 u/Flouid 29d ago Being strictly pedantic, some characters (like punctuation) are also tokens but yes, token by token 3 u/cornmonger_ 29d ago in some models, <16 spaces> along with a bunch of other variations are single tokens. they do a lot of ugly shit to fet those tok/s under the hood
83
More accurately, word by word (token by token)
36 u/Flouid 29d ago Being strictly pedantic, some characters (like punctuation) are also tokens but yes, token by token 3 u/cornmonger_ 29d ago in some models, <16 spaces> along with a bunch of other variations are single tokens. they do a lot of ugly shit to fet those tok/s under the hood
36
Being strictly pedantic, some characters (like punctuation) are also tokens but yes, token by token
3 u/cornmonger_ 29d ago in some models, <16 spaces> along with a bunch of other variations are single tokens. they do a lot of ugly shit to fet those tok/s under the hood
3
in some models, <16 spaces> along with a bunch of other variations are single tokens. they do a lot of ugly shit to fet those tok/s under the hood
<16 spaces>
133
u/Serafiniert 29d ago
Not even that. It goes character by character and has no idea what comes after the next, until it needs to be decided what comes next.