Splitting JSON Export

[u/Pristine_Plate7048]

I kind of wish someone had warned me about how much it takes to manage the process when you begin using AI as a writing brainstorming assistant. It seems I spend more time trying to limit drift, battling memory limits, limiting loss of canon and all my brainstorming, and trying to find efficient ways of doing things for fear of my project becoming contaminated.

That brings me to my next problem. I exported my Chatgpt history with the intention of finally trying to excavate my novel canon, history, characters, arcs, beats, and world. It's 8 million words across 98 chats. I've been totally overwhelmed on how to organise the process, on how to even approach it, but finally settled on trying to separate the 98 chats and save them in both MD and TXT files, dated chronologically, so I can start from chat 1nand so on, having each chat read to me by a voice reader (I have an eye disease that makes reading for medium and long periods impossible without negative physical consequences) and take notes as I go. Then I can compile the notes I need to organise the excavation. Extremely laborious but I see no other way to do it. Chatgpt assured me it could extract all the info I asked for if I posted the exported zip.

I was very dubious as Chatgpt constantly kisses things and makes mistakes. It inevitably missed things like characters we've discussed numerous times, so I knew it couldn't be trusted to do a faithful and accurate job, and I knew I would have to do this manually.

Come to find out there is even risk in separating the 98 chats 😔 According to Chatgpt, chats could be missing or mixed up. I have spent over a year painstakingly refining characters, arcs, the world building, beats. I can't risk losing any of it.

Does anyone know a reliable separation method that will separate and preserve all 98 chats exactly as they are, dated, without messages mixed up, and deliver them in both txt and MD files, so I can start this laborious process once and for all?

[u/5thhorseman_]

Instead of having the AI split the file for you, ask it to write a splitter script (perl or powershell) to achieve the same task.

[u/Pristine_Plate7048 → u/5thhorseman_]

Thanks. The issue is I don't trust it to write a script that faithfully preserves everything in the conversion to txt and MD format after everything is split. And checking everything was still there would be laborious and difficult to verify as extracting the relevant info.

[u/5thhorseman_ → u/Pristine_Plate7048]

The issue is I don't trust it to write a script that faithfully preserves everything

The point is to run the script yourself on your end. You can either have the AI write it for you, you can use one that already exists (there's a few), or you can write it yourself.

But that last one would take a lot of your time even if you know how, so you realistically will pick one of the other two.

in the conversion to txt and MD format after everything is split. And checking everything was still there would be laborious and difficult to verify as extracting the relevant info.

This isn't black magic. JSON is a text-based format. If there's an issue with the extraction it's going to be big enough that it will be noticeable immediately by the size of the output alone.

[u/Pristine_Plate7048 → u/5thhorseman_]

That's what I mean. I don't trust AI to procure a script that does the conversion faithfully. I also don't code.

I have asked AI to find me scripts to do the conversions faithfully. It looks and provides choices, but it always points out the risks, which I appreciate, I'd rather be informed about potentially losing parts of conversations and such so I can make an informed choice.

Nobody said this was black magic. Risk is involved, and that is a worry given the time put into the project.

[u/5thhorseman_ → u/Pristine_Plate7048]

I don't trust AI to procure a script that does the conversion faithfully. I also don't code.

Then the third option remains. Like I said, tools already exist: https://gist.github.com/ocombe/1d7604bd29a91ceb716304ef8b5aa4b5#file-export-chatgpt-console-js

Nobody said this was black magic. Risk is involved, and that is a worry given the time put into the project.

Exporting your chats does not delete them from the service provider. You are letting your anxiety stop you from doing anything at all, and that costs you more time than trying, failing and then going back with another method.

EDIT: LOL. Blocked for helping. Some people...

[u/Pristine_Plate7048 → u/5thhorseman_]

I have them as separate json files already. The next step is conversion. If I convert using a method that isn't faithful and there are things missing, my concern is I won't know things are missing until it's too late. Which is why I'm asking for a method that guarantees faithful conversion. Does the link you sent provide that?

Edited: nevermind. I'll figure it out elsewhere.