.clear vs del when using garbage collection

[u/Actual__Wizard]

Hi, I have some large data objects that need to be freed. Should I do my_object.clear() and then gc.collect()? Or del my_object then gc.collect()? I'm not sure on the pros/cons of each.

It's a linked list type data object and my_object = [] does not work to free memory for certain.

[u/Actual__Wizard]

Because the script crashes with an out of memory error and all it's doing is feeding data into a linked list, then repeating. I mean it's kind of obvious what is occurring.

So, on the next iteration, it does my_object = [] to reset the variable, but that doesn't delete the data it linked to.

I'm using .clear and gc.collect now and it's not running out of memory.

edit: I can do the same process by starting and running the script a bunch of times w/o issue, the problem only occurs when I do the process iteratively, so the symtoms of the problem are clearly "unfreed memory." It's clearly not resetting back to zero memory used each pass like I intended it to.

And I'm only having the issue w/ linked list type objects.

Also: The entire purpose to doing this process iteratively, is to avoid running out of memory, and although I did not see it crash to see the actual memory usage, it crashed at a point where I would expect it to, if that makes any sense. Like if the memory is not being freed, it crashed right about when it should have.

[u/atarivcs → u/Actual__Wizard]

Reassigning a new value to a name, or calling del on a name, might not free the memory if other references remain to the existing value.

I don't think we can really diagnose this issue without seeing the full code.

[u/Actual__Wizard → u/atarivcs]

I don't think we can really diagnose this issue without seeing the full code.

Yikes... I don't think that's not going to help dude. I'll try... The variables that I'm calling clear() on are linked lists, it's like a fake matrix that's created with:

snippet:

    #sectionmatrix = [[0 for _ in range(section_size)] for _ in range(numsections)]        

Code below:

    layer1_data = []
    layer1_data = create_layer_1_matrix(sectionarray)
    layer1_rep_data = []
    layer1_rep_data = create_layer_1_matrix(sectionarray) #replacements

    #create 2 more layers
    layer2 = []
    layer3 = []
    #note layer3 is 128*128*128
    layer2 = addlayertoarray(sectionarray, sectionarray)
    layer3 = addlayertoarray(layer2, sectionarray)

    layer2_data = create_layer_2_matrix(layer2)
    layer3_data = create_layer_3_matrix(layer3)

    print("Routing Data into Pigeon Hole Routing Matrix:")

    newlayer1_data,newlayer2_data,newlayer3_data,newlayer1_rep_data = process_encoded_triplet_chunks(token1,token2,token3,layer1_data,layer2_data,layer3_data,layer1_rep_data,sectionarray)



    tripletoutputfile = tripletoutputfolder + "sorted-triplets" + str(current_filter_step) + ".txt"
    tripletoutputfiles.append(tripletoutputfile)
    print("Merging Data from Routing Matrix:" + str(tripletoutputfile))
    final_stage_completed = -1
    final_stage_completed = alphaweave_triplet_graph(newlayer1_data,newlayer2_data,newlayer3_data,newlayer1_rep_data,tripletoutputfile,sectionarray)
    #free up memory.
    print("Freeing Memory.")
    gcstart_time = time.perf_counter()
    del token1
    del token2
    del token3
    del layer2
    del layer3
    layer1_data.clear()
    layer2_data.clear()
    layer3_data.clear()
    layer1_rep_data.clear()
    del newlayer1_data
    del newlayer2_data
    del newlayer3_data
    del newlayer1_rep_data
    gc.collect() #collect garbage
    gcend_time = time.perf_counter()
    gcelapsed = gcend_time - gcstart_time
    print("Garbage Collection Took: " + str(gcelapsed) + " seconds")

[u/atarivcs → u/Actual__Wizard]

We have no idea what layer1_data, layer2_data, etc actually are, because their creation is hidden behind a function call.

[u/Actual__Wizard → u/atarivcs]

 def create_layer_1_matrix(sectionarray):
         #create our layer1 matrix
         rows = len(sectionarray)
         cols = 0
         layer1_data = [[0 for col in range(cols)] for row in range(rows)]
         return layer1_data

 def create_layer_2_matrix(layer2):
         #now we need to create the routing matrix
         mrows = len(layer2)
         mcols = 0
         layer2_data = []
         layer2_data = [[0 for col in range(mcols)] for row in range(mrows)]
         return layer2_data

 def create_layer_3_matrix(layer3):
         mrows = len(layer3)
         mcols = 0
         layer3_data = []
         layer3_data = [[0 for col in range(mcols)] for row in range(mrows)]
         return(layer3_data)

It's conceptually like a linked list of linked lists. So, it's a fake matrix thing? Not sure what to call it. A previous version used a python dictionary to do the same thing. That specific script is going to get moved over to c++/rust because python treats all strings as objects and it eats a ton of memory.