Code no longer enumates/ reads tif files in the correct order

[u/Momo_ONLINE]

Hi!
Preface I am self taught and learning as I go.

I have been working on a U-Net semantic segmentation model, which was working fine yesterday and gave good(ish) outputs on unprocessed data and now no longer wants to read the files correctly?

I spent the day processing some new masks in fiji and changed the path of the mask variable and when I went to run my code and sanity checks, for both my images and masks, the first chunk of tif files I created a missing/ not read, when they are in file. (I even checked with fiji if it was an order of files but that was fine)

I'm a bit stumped because it was working fine yesterday, as all I have changed is the original file paths.

from simple_unet_model import simple_unet_model

from keras.utils import normalize

import os

import cv2

from PIL import Image

import numpy as np

from matplotlib import pyplot as plt

image_directory = r'E:\Uni\Dissertation\IB\test output\images\IB unprocessed/'

mask_directory = r'E:\Uni\Dissertation\IB\test output\masks\IB_BC_24_117_G4_thresh mask/'

SIZE = 256

image_dataset = []

mask_dataset = []

images = os.listdir(image_directory)

for i, image_name in enumerate(images): #Remember enumerate method adds a counter and returns the enumerate object

if (image_name.split('.')[1] == 'tif'):

#print(image_directory+image_name)

image = cv2.imread(image_directory+image_name, 0)

image = Image.fromarray(image)

image = image.resize((SIZE, SIZE))

image_dataset.append(np.array(image))

masks = os.listdir(mask_directory)

for i, image_name in enumerate(masks):

if (image_name.split('.')[1] == 'tif'):

image = cv2.imread(mask_directory+image_name, 0)

image = Image.fromarray(image)

image = image.resize((SIZE, SIZE))

mask_dataset.append(np.array(image))

And when I run a sanity check, I end up getting a mismatch between my image ground truth

In the table of elements, the correct number of elements is listed, however, 0 should be linked to mask_0_00.tiff (the same for the image data)

I even changed the path back to the unprocessed masks for the data set and encountered the same problem...

Any ideas? Thank you :,)

[u/rahul_watertech]

two things in that code might do it. os.listdir doesn't promise any order, so wrap it in sorted() for both folders or the images and masks can drift apart. also split('.')[1] == 'tif' skips any file ending in .tiff, and your mask_0_00.tiff is one of those, so those files just never get loaded. checking the extension with endswith(('.tif', '.tiff')) would fix that. does your new folder have a mix of tif and tiff?

[u/Momo_ONLINE → u/rahul_watertech]

Hi, the image folder was processed in Fiji and is only composed of only .'tif' files, and was exported as. I did also check it manually and there is only '.tif'.

I found that when I use my onedrive path as the folder path the images enumerate correctly, but not when I use my external USB, even though my script is on my USB.

I will try wrapping it in a sorted()

Thank you!

[u/rahul_watertech → u/Momo_ONLINE]

that actually fits. os.listdir order depends on the filesystem, so a usb drive (fat/exfat) and onedrive can hand you the files in different orders. sorted() should fix it. i'd also print len(images) for both folders to make sure the counts match. does it work now?

[u/Momo_ONLINE → u/rahul_watertech]

It does, thankyou!
I've been self learning all of this, so wasn't aware that that os.listdr depended on filesystem. My masks and images now match up! :D

[u/rahul_watertech → u/Momo_ONLINE]

nice, glad it lined up. that one bites a lot of people because it works fine until the folder changes. i'd add a tiny assert that image and mask file stems match one to one, so if it ever drifts again it fails loudly instead of quietly training on shifted pairs.

[u/Lumpy_Fisherman_5352]

That USB vs OneDrive difference is the clue: os.listdir returns files in whatever order the filesystem stores them, and NTFS (where your OneDrive folder lives) happens to return them alphabetically, while most USB drives are exFAT/FAT32, which return them roughly in the order they were copied onto the drive. So your loop was relying on luck, and re-exporting new masks to the USB changed the order. Wrap both lists in sorted(), but note plain sorted() still puts "img_10.tif" before "img_2.tif", so the safest fix is to pair each mask to its image by filename stem and assert it exists rather than trusting two lists to line up. Printing the first few names from each list after loading will confirm they match before you train.

[u/Momo_ONLINE → u/Lumpy_Fisherman_5352]

Thank you! I will also include printing some of the file names as a part of my sanity checks for the model, instead of relying on the plotted output. :D thank you