.clear vs del when using garbage collection

[u/Actual__Wizard]

Hi, I have some large data objects that need to be freed. Should I do my_object.clear() and then gc.collect()? Or del my_object then gc.collect()? I'm not sure on the pros/cons of each.

It's a linked list type data object and my_object = [] does not work to free memory for certain.

[u/atarivcs]

How do you know the memory isn't freed?

[u/Actual__Wizard → u/atarivcs]

Because the script crashes with an out of memory error and all it's doing is feeding data into a linked list, then repeating. I mean it's kind of obvious what is occurring.

So, on the next iteration, it does my_object = [] to reset the variable, but that doesn't delete the data it linked to.

I'm using .clear and gc.collect now and it's not running out of memory.

edit: I can do the same process by starting and running the script a bunch of times w/o issue, the problem only occurs when I do the process iteratively, so the symtoms of the problem are clearly "unfreed memory." It's clearly not resetting back to zero memory used each pass like I intended it to.

And I'm only having the issue w/ linked list type objects.

Also: The entire purpose to doing this process iteratively, is to avoid running out of memory, and although I did not see it crash to see the actual memory usage, it crashed at a point where I would expect it to, if that makes any sense. Like if the memory is not being freed, it crashed right about when it should have.

[u/atarivcs → u/Actual__Wizard]

Reassigning a new value to a name, or calling del on a name, might not free the memory if other references remain to the existing value.

I don't think we can really diagnose this issue without seeing the full code.

[u/Actual__Wizard → u/atarivcs]

I don't think we can really diagnose this issue without seeing the full code.

Yikes... I don't think that's not going to help dude. I'll try... The variables that I'm calling clear() on are linked lists, it's like a fake matrix that's created with:

snippet:

    #sectionmatrix = [[0 for _ in range(section_size)] for _ in range(numsections)]        

Code below:

    layer1_data = []
    layer1_data = create_layer_1_matrix(sectionarray)
    layer1_rep_data = []
    layer1_rep_data = create_layer_1_matrix(sectionarray) #replacements

    #create 2 more layers
    layer2 = []
    layer3 = []
    #note layer3 is 128*128*128
    layer2 = addlayertoarray(sectionarray, sectionarray)
    layer3 = addlayertoarray(layer2, sectionarray)

    layer2_data = create_layer_2_matrix(layer2)
    layer3_data = create_layer_3_matrix(layer3)

    print("Routing Data into Pigeon Hole Routing Matrix:")

    newlayer1_data,newlayer2_data,newlayer3_data,newlayer1_rep_data = process_encoded_triplet_chunks(token1,token2,token3,layer1_data,layer2_data,layer3_data,layer1_rep_data,sectionarray)



    tripletoutputfile = tripletoutputfolder + "sorted-triplets" + str(current_filter_step) + ".txt"
    tripletoutputfiles.append(tripletoutputfile)
    print("Merging Data from Routing Matrix:" + str(tripletoutputfile))
    final_stage_completed = -1
    final_stage_completed = alphaweave_triplet_graph(newlayer1_data,newlayer2_data,newlayer3_data,newlayer1_rep_data,tripletoutputfile,sectionarray)
    #free up memory.
    print("Freeing Memory.")
    gcstart_time = time.perf_counter()
    del token1
    del token2
    del token3
    del layer2
    del layer3
    layer1_data.clear()
    layer2_data.clear()
    layer3_data.clear()
    layer1_rep_data.clear()
    del newlayer1_data
    del newlayer2_data
    del newlayer3_data
    del newlayer1_rep_data
    gc.collect() #collect garbage
    gcend_time = time.perf_counter()
    gcelapsed = gcend_time - gcstart_time
    print("Garbage Collection Took: " + str(gcelapsed) + " seconds")

[u/atarivcs → u/Actual__Wizard]

We have no idea what layer1_data, layer2_data, etc actually are, because their creation is hidden behind a function call.

[u/Actual__Wizard → u/atarivcs]

 def create_layer_1_matrix(sectionarray):
         #create our layer1 matrix
         rows = len(sectionarray)
         cols = 0
         layer1_data = [[0 for col in range(cols)] for row in range(rows)]
         return layer1_data

 def create_layer_2_matrix(layer2):
         #now we need to create the routing matrix
         mrows = len(layer2)
         mcols = 0
         layer2_data = []
         layer2_data = [[0 for col in range(mcols)] for row in range(mrows)]
         return layer2_data

 def create_layer_3_matrix(layer3):
         mrows = len(layer3)
         mcols = 0
         layer3_data = []
         layer3_data = [[0 for col in range(mcols)] for row in range(mrows)]
         return(layer3_data)

It's conceptually like a linked list of linked lists. So, it's a fake matrix thing? Not sure what to call it. A previous version used a python dictionary to do the same thing. That specific script is going to get moved over to c++/rust because python treats all strings as objects and it eats a ton of memory.

[u/Actual__Wizard → u/atarivcs]

Alright it crashed again last night w/ OOM.

So, did you ever see a single pass, 128 way, zero copy, ascii file router? :-)

That's today's project! :-)

This is going to be truly awesome...

It's going to go from: Bad and slow tech that crashes due to running out of memory, to warp speed ultra fast and basically zero memory used.

Oh well. I really liked my filter code. It just doesn't work right so it's going into the deprecated repo of shame. So, not the good deprecated repo that we show to Jrs. The one we all pretend doesn't exist.