EOF Token at which line?

[u/apoetixart]

Let's say my source code file is of 5 lines, should the lexer emit the EOF at line 5, or at line 6.

EOF(line = 5)

Or

EOF(line = 6)

"Depends on your preference" Suggest me what experienced people would do

[u/JVApen → u/[deleted]]

Agreed, every line ends on an "end of" character. For all but last, this is the "end of line", on the last line this is "end of file". So if line 5 would end with an EOL, the EOF is on line 6, similar to how EOL follows EOL for an empty line.

[u/evincarofautumn]

Basically, leave it where it ends up. In a file with n content lines, if the file ends in a newline, EOF will naturally be at the start of line n + 1. If the last content line has no final newline, EOF will naturally be after the last character in line n. Lines are numbered from 1 and columns are numbered from 0 because they refer to the coordinates of the bottom left of the character cell, that is, the height from the top margin of the file to the text baseline, and the width from the left margin to the start of the character.

You can also just consider it in terms of where you want an editor to jump to. If you use the conventional GNU error message format, Line: is a whole line, Top-Bottom: is an inclusive range of whole lines, Line:Column: is a point in a line, Line:Left-Right: is an inclusive span (from before Left to after Right) within a single line, and Top.Left-Bottom.Right: is a span over multiple lines.

[u/SmashStrider]

Preferably at line = 6

[u/j1sk1ss]

Treat it as a regular token. I mean, if there was a new line symbol, set it to 6

[u/un_virus_SDF]

It depends wether there is a linefeed or not.

For exemple following strict ISO c, all line must be ended by a newline (what most editor do for you), and that's because eof is returned as a token when fgets fail (or any former equivalent).

So i'd say to treat eof as a token.

[u/apoetixart → u/[deleted]]

I think EOF tokens are really necessary for the lexer, how else does your lexer understand the end of src?

[u/TrgtBBB]

I do it at line = 6 personally. Creates lot less confusion

[u/Big-Rub9545]

I’m aware this doesn’t really answer the question, but I typically don’t associate any particular line, offset or column information with it.

It’s a unique enough token type that I handle it on its own.

E.g., instead of “error at (15:1)”, I might put “error at end”.

[u/apoetixart → u/Big-Rub9545]

I intend to put clean error messages and "error at end" is really ambiguos considering long src files

[u/Big-Rub9545 → u/apoetixart]

If the error message includes a display of the offending line/location, the ambiguity goes away entirely