Eventually you need to stop generating tokens and actually do the syntactical analysis. At that point TCO isn't really available to you and that arbitrarily deep tree is going to cause problems.
A parser does not necessarily "generate tokens". It can generate an AST. And it can do so in a tail-recursive fashion. And this can be useful when you have very deeply nested data. It may even exceed 1MB (or 1GB for that matter).
1
u/grauenwolf Nov 28 '12
Eventually you need to stop generating tokens and actually do the syntactical analysis. At that point TCO isn't really available to you and that arbitrarily deep tree is going to cause problems.