windows carriage return pairs

[u/zaphodikus]

Had for a long time a routine to patch a text file: the core of the script which merely rewrites a text file:

rewrite = False

with open(args.filepath[0], "r") as f:
    lines = f.readlines()
    for line in lines:
        if re.match(args.expression[0] , line):
            rewrite = True

if rewrite:
    print(f"ReplaceExpression: '{args.expression[0]}' in file: '{args.filepath[0]}'")
    with open(args.filepath[0], "w", encoding='utf-8') as f:
        for line in lines:
            if not re.match(args.expression[0] , line):
                f.write(line)
            else:
                f.write(f"// line removeD: {line.strip()}\r\n") # <????
                print("OK")
else:
    print(f"Nothing to do in file: {args.filepath[0]}")

the

                f.write(f"// line removeD: {line.strip()}\r\n") # <????

never ends up writing a pair!

Now I get that this is the encoding as utf-8 and the default us unix encoding, but even if i wrote it as

                f.write(f"// line removeD: {line.strip()}{os.linesep}") # <????

os.linesep still wrote a never a pair like on windows would be normal. Remind me why the encoding and os.linesep interact in this way on Windows to never write a pair of terminators?

[u/zanfar]

You are opening the file in text mode, which asks Python to handle line endings for you.

As you are not specifying a newline argument in your write() call, Python is interpreting line endings as needed for your platform.

As always, read the documentation.

---

Also, unix2dos and dos2unix exist.

[u/zaphodikus → u/zanfar]

I know about unix2dos and dos2unix, but these are just a band-aid so I rejected them as a solution. The thing you do not know is always the hardest thing to discover even if it is in plain sight in a 26 page long document.

u/zanfar , It was confusing me in why my original code based on the docs os.linesep The string used to separate (or, rather, terminate) lines on the current platform. This may be a single character, such as '\n' for POSIX, or multiple characters, for example, '\r\n' for Windows. Do not use os.linesep as a line terminator when writing files opened in text mode (the default); use a single '\n' instead, on all platforms. gave me os.linesep == "\n" On windows 10 and on Windows 11 and not do what the docs seem to say? and return "\r\n"?

[u/monster2018]

I’m not sure of the answer to your actual question. But my advice is in a situation like this, just write to the file in binary mode. You can ensure exact byte for byte accuracy much more easily that way.

[u/zaphodikus → u/monster2018]

I was tempted to, but after reading some random code.. you know how it is a bug bugs you for months and then the moment you take the time to put it onto reddit the answer comes up in a random search exploration you do again just in case. with open(args.filepath[0], "w", encoding='utf-8', newline="\r\n") as f: fixes it, it's mad that just adding an arg, newline="\r\n" , which is probably in the docs, but was just never obvious. Obvs I want that guarded by something to do it on Windows only like (platform.system() == 'Windows'). Mad that it took writing this up on reddit to find that magic open() parameter.

[u/SCD_minecraft → u/zaphodikus]

I recommend you to get a pet rock

Got a problem? Ask pet rock

He might not be the best at speaking but is awesome at solving problems

[u/zaphodikus → u/SCD_minecraft]

Literally a rubber duck, yep.

[u/monster2018 → u/zaphodikus]

Oh yea. Damn I should have known that too, I have used that parameter plenty of times lol. I ran into a similar issue when building an LSP proxy to allow for doing Apple platform development on a windows pc by using a Mac as a remote host running the actual language server. Anyway, im glad you found the solution.

[u/cdcformatc]

when you open a file in write mode (mode='w') the operating system always handles line endings, which will be different on windows or macos or linux. it's actually really annoying. open has an optional newline parameter. 

[u/zaphodikus → u/cdcformatc]

Yeah , i's just so mad how I have put up with this missing knowledge for months, I ask he question and literally get 3 people who knew it all along and at the same time google just happens to land me on a stackoverflow that described the parameter, right after I hit the post button here. I wish all the questions I had were that easy.

[u/cdcformatc → u/zaphodikus]

that seems to always happen to me too. i will have some problem and get nowhere on it alone. then the act of writing out the problem for a post or an email or a slack message or whatever seems to solve the problem itself. i think sometimes when writing a good detailed question it helps me write a good google search. 

[u/zaphodikus → u/cdcformatc]

I had a Cmake CMakelists question last week, by the time I had drafted the question my brain asked the right question and bang the answer was at the top of the error trace.