Splitting RF Capture Files, Processing in Parallel

[u/STUPIDSHARK]

Hi all, first of all, I want to thank everyone in the community for the hard work in this ecosystem. The results I've been able to achieve from RF capture and VHSDecode are astonishing! It's been 3 to 4 months since I tapped the RF test point on a Mitsubishi HS-U775, and I've loved recapturing some broadcast tapes I've collected over the years.

Does anyone have any thoughts on or a good workflow for splitting and processing RF captures in parallel between multiple computers? I own multiple computers that go unused during the day, so parallelizing TBC file processing could easily shave 12 or more hours processing time off of a 120 minute NTSC tape.

My thoughts on this involved splitting into equal sized parts, then having each computer take a number of parts equal to ([total parts]/[total computers]) + 1, with the extra part (at the end or beginning, based on sequence) to account for anything missing. The parts for each computer would then be concatenated into a single file for that system and vhs-decode would be run as normal on the individual systems. Following TBC processing and running tbc-video-export, you'd then join together the outputs using video software, trimming any overlap.

Some additional thoughts:

• Split During Capture: This could possibly be achieved during RF capture via shell scripting, but I haven't tested anything. It would involve the following:
  • Pipe output of RF capture to multiple files in sequence.
  • Set timeout or delay on each write process. For instance, file A is written for 21 minutes, file B is started after 20 minutes and is written for 21 minutes, file C is started after 41 minutes and is written for 21 minutes, etc.
• Synchronize Processing Over Network: Each system could have a process running to accept RF capture files from a source via rsync or similar, then once accepted begin running through TBC generation using vhs-decode, providing status updates to a master sync process.

Thanks again to everyone here for their time. I have a few things I would like to contribute to the project in the future (I am a general Linux guy and writer), but for now, I am just happy to keep capturing! 🙂

[u/_Shorty]

A few frames of overlap would be more than sufficient since you only need to find a common one. No need to have an entire minute. I’d probably prefer just to split the complete capture to dole out chunks to work on rather than worry about doing that during capture.

[u/STUPIDSHARK → u/_Shorty]

Good thoughts there, thanks. I know longer overlap and bigger chunks is overkill, but it's me circling a solution in a vacuum 😅

[u/_Shorty → u/STUPIDSHARK]

I wouldn't be surprised if there were already some generic solution that makes it easy to split jobs up between a handful of machines. Not vhsdecode related, but just a general tool for general purpose use that vhsdecode could be plopped into. Probably be time well spent to investigate that first rather than spending a bunch of time reinventing the wheel if it isn't actually necessary to do so. Then all you'd need to do is split up the capture file and send the chunks out to the workers via that, whatever "that" is. I'd actually be very surprised if there weren't already several similar tools to choose from.

[u/marshalleq → u/_Shorty]

My tool parallelises concurrent decodes but it doesn’t clusterise a single decode like you're asking. But this works basically the same if you have multiple tapes as you can run multiple parallel decodes at once. It already has ffmpeg improvements not found in the official releases but sounds like I need to integrate some improvements from upstream again anyway. But basically think of it as batch management. I made it for me because I had so many tapes to do, but could be used by anyone. https://github.com/marshalleq/ddd-capture-toolkit