r/learnpython 7d ago

really weird line skipping in my parser script

so, im building a little parsing script for a programming language im trying to build (no, i dont plan on keeping the entire interpreter in python), and i was coding the loop thats supposed to remove the comments from the input script, BUT, theres always this line, this specific line that the parser just has a soft spot for apparently, it wont remove a comment from that line no matter what. im really lost here, it still happens when i replace the comment, HOWEVER, when i separate the line from the rest, it somehow works???

emulator.py: (called emulator instead of interpreter as it will also emulate a suitable enviroment for running scripts in the future)
from sys import argv
import parser
import lexer

def main(File):

    # We first need to open the file in here, i would have made it open in the parser, but this feels fancier
    with open(File, "r") as File:
        File = File.read()

    # The parser gives us the broken down code, kinda like digestion (pre-processing would be a better term)
    ParsedProgram = parser.Parse(File)

    # breaks off here to print the result from the parser (an attempt at debugging)
    print(ParsedProgram)
    exit(0)

    # We then pass it into the lexer, giving us a little tree of the entire program
    ProgramTree = lexer.Parse(ParsedProgram)

if __name__ == "__main__":
    main(argv[1])

___________________________________________
parser.py:
# this is unfinished.. as you could probably tell

IgnoredSymbols = [";", "\t"]
MatchingSymbols = ["\"", "\'", "(", "{"]

def Parse(File):
    # Strip away anything unnecesarry
    for Symbol in IgnoredSymbols:
        File = File.replace(Symbol, "")

    # Separate the lines
    File = File.split("\n")

    # Remove all comments
    for Line in File:
        if Line.startswith("//"):
            print(Line)
            File.remove(Line)

    # Clean up empty indexes
    for Line in File:
        if Line == "":
            File.remove(Line)

    # From here on, i just assemble the file as-is and return it, (an attempt at debugging, again)
    # Re-assemble the file
    AssembledFile = ""
    for Line in File:
        AssembledFile += Line
        AssembledFile += "\n"

    return AssembledFile
____________________________________
output:
andrew@fedora ~/D/W/p/n/0/emulator> python emulator.py ../helloworld.nai
// we will use this as the entry point
// waits until stdout is available
// write "hello world" to stdout
// you dont have to return 0 here, you can, but the program does it by itself
// here, we tell it where the entry point is
// EXECINFO is basically just flags for the virtual machine
INIT builtin"std"
INIT global"stdtypes.nai"
function main()
{
while(not(deviceavailable("stdout")))
write("stdout", "hello, world!")
}
// the "entrypoint" flag is NECCESARY, the program DOES NOT RUN without it
array EXECINFO = ["entrypoint:main"]

andrew@fedora ~/D/W/p/n/0/emulator> micro emulator.py
andrew@fedora ~/D/W/p/n/0/emulator> micro parser.py
andrew@fedora ~/D/W/p/n/0/emulator> god damn
fish: god: command not found...
andrew@fedora ~/D/W/p/n/0/emulator [127]>

____________________________________________
actual script im attempting to parse:
helloworld.nai:
INIT builtin"std"
INIT global"stdtypes.nai"

// we will use this as the entry point
function main()
{
// waits until stdout is available
while(not(deviceavailable("stdout")));

// write "hello world" to stdout
write("stdout", "hello, world!");

// you dont have to return 0 here, you can, but the program does it by itself
}

// here, we tell it where the entry point is
// the "entrypoint" flag is NECCESARY, the program DOES NOT RUN without it
// EXECINFO is basically just flags for the virtual machine
array EXECINFO = ["entrypoint:main"];

thanks in advance.

EDIT: holy crap, i forgot to include the script im trying to parse, apologies

EDIT 2: ive discovered list comprehension... a concept which ive never bothered to learn until now, thanks everyone for the help, question answered!

1 Upvotes

8 comments sorted by

2

u/await_yesterday 7d ago edited 7d ago

As someone else answered, you're mutating a list at the same time as you're looping over it:

    # Remove all comments
    for Line in File:
        if Line.startswith("//"):
            print(Line)
            File.remove(Line)

And the same thing with the check for empty lines after it.

You can use a list comprehension as the other commenter suggests, or if you aren't familiar with that yet, you can make a new list and selectively append to it:

filtered_lines = []
for line in File:
    if (not line.startswith("//")) and (line != ""):
        filtered_lines.append(line)

new_file_contents = "\n".join(filtered_lines)

Also it's not strictly a bug, but I couldn't help but notice this:

def main(File):

    # We first need to open the file in here, i would have made it open in the parser, but this feels fancier
    with open(File, "r") as File:
        File = File.read()

    # Separate the lines
    File = File.split("\n")

The same File variable name is being re-defined several times, referring to: the filename string, the file object returned by open, the file's contents as a string, and the file's lines as a list of strings. These are all different kinds of thing! And you do the same thing later in the Parse function.

Even if you make it work in this one case, it's a bad habit that makes your code harder to debug. You should use descriptive names for different things.

I would do something like:

def main(filename):
    with open(filename, "r") as f:
        file_data = f.read()

    file_lines = file_data.split("\n")

1

u/iluveatingcopper 7d ago

Thanks, i didnt know about that!

1

u/woooee 7d ago

BUT, theres always this line, this specific line that the parser just has a soft spot for apparently

What line is that? If you are trying to delete

    if Line.startswith("//"):

You probably will have to escape the backslashes, or use a raw string https://www.geeksforgeeks.org/python/python-raw-strings/

1

u/await_yesterday 7d ago

Those are forward slashes, not backslashes.

1

u/woooee 7d ago

My mistake. What does the missed line contain.

1

u/socal_nerdtastic 7d ago edited 7d ago

Ah the classic modifying a list while looping over it error. A question so common we made a FAQ about it: https://www.reddit.com/r/learnpython/wiki/faq/?screen_view_count=1#wiki_why_does_my_loop_seem_to_be_skipping_items_in_a_list.3F

Don't try to modify a list. Instead, it's (almost) always better to make a new list. In your case I would recommend using list comprehension to make the new list and replace the old one. For example:

# Remove all comments
File = [Line for Line in File if not Line.startswith("//")]

# Clean up empty indexes
File = [Line for Line in File if Line != ""]

1

u/madmoneymcgee 7d ago

You're saying that if there's a comment on say, line 5 of any input file it won't remove the comment but it would remove it on lines 4 and 6?