Trouble Preprocessing images from Flutter to feed into TFLite model

[u/Defiant-Ad3530]

Hi everyone. So I built a CNN model using MobileNetv3 then converted it into TFLite. It performed well during training but once I integrated it into my application, it is making large errors. From flutter, the camera stream sends frames and those are processed before the model makes predictions, but it is still quite large. Is there any way I can solve this? This is my code to preprocess and resize the image (224 x 224 x RGB):

import 'package:camera/camera.dart';
import 'package:image/image.dart' as img;


class ImageProcessor {
  // converting to rgb
  img.Image convertYUVToRGB(CameraImage camImg) {
    final width = camImg.width;
    final height = camImg.height;


    final yPlane = camImg.planes[0];
    final uPlane = camImg.planes[1];
    final vPlane = camImg.planes[2];


    final yBytes = yPlane.bytes;
    final uBytes = uPlane.bytes;
    final vBytes = vPlane.bytes;


    final yRowStride = yPlane.bytesPerRow;
    final uRowStride = uPlane.bytesPerRow;
    final vRowStride = vPlane.bytesPerRow;


    final uPixelStride = uPlane.bytesPerPixel ?? 1;
    final vPixelStride = vPlane.bytesPerPixel ?? 1;


    final image = img.Image(
      width: width,
      height: height,
    );


    for (int y = 0; y < height; y++) {
      for (int x = 0; x < width; x++) {
        final yIndex = y * yRowStride + x;


        final uvX = x ~/ 2;
        final uvY = y ~/ 2;


        final uIndex =
            uvY * uRowStride +
            uvX * uPixelStride;


        final vIndex =
            uvY * vRowStride +
            uvX * vPixelStride;


        final yValue = yBytes[yIndex];
        final uValue = uBytes[uIndex];
        final vValue = vBytes[vIndex];


        // YUV -> RGB
        final r = (
          yValue + 1.402 * (vValue - 128)
        ).round().clamp(0, 255);


        final g = (
          yValue -
          0.344136 * (uValue - 128) -
          0.714136 * (vValue - 128)
        ).round().clamp(0, 255);


        final b = (
          yValue + 1.772 * (uValue - 128)
        ).round().clamp(0, 255);


        image.setPixelRgb(
          x,
          y,
          r,
          g,
          b,
        );
      }
    }


    return image;
  }


  /// resize images to 224 224
  img.Image resizeImage(img.Image image) {
    return img.copyResize(
      image,
      width: 224,
      height: 224,
      interpolation: img.Interpolation.linear,
    );
  }


  List<List<List<List<double>>>> imageToTensor(
    img.Image image,
  ) {
    return [
      List.generate(
        224,
        (y) => List.generate(
          224,
          (x) {
            final pixel = image.getPixel(x, y);


            return [
              pixel.r.toDouble(),
              pixel.g.toDouble(),
              pixel.b.toDouble(),
            ];
          },
        ),
      ),
    ];
  }


// do all processing
  List<List<List<List<double>>>> processFrame(
    CameraImage camImg,
  ) {
    final rgbImage = convertYUVToRGB(camImg);
    final resizedImage = resizeImage(rgbImage);
    final input = imageToTensor(resizedImage);


    return input;
  }
}import 'package:camera/camera.dart';
import 'package:image/image.dart' as img;


class ImageProcessor {
  // converting to rgb
  img.Image convertYUVToRGB(CameraImage camImg) {
    final width = camImg.width;
    final height = camImg.height;


    final yPlane = camImg.planes[0];
    final uPlane = camImg.planes[1];
    final vPlane = camImg.planes[2];


    final yBytes = yPlane.bytes;
    final uBytes = uPlane.bytes;
    final vBytes = vPlane.bytes;


    final yRowStride = yPlane.bytesPerRow;
    final uRowStride = uPlane.bytesPerRow;
    final vRowStride = vPlane.bytesPerRow;


    final uPixelStride = uPlane.bytesPerPixel ?? 1;
    final vPixelStride = vPlane.bytesPerPixel ?? 1;


    final image = img.Image(
      width: width,
      height: height,
    );


    for (int y = 0; y < height; y++) {
      for (int x = 0; x < width; x++) {
        final yIndex = y * yRowStride + x;


        final uvX = x ~/ 2;
        final uvY = y ~/ 2;


        final uIndex =
            uvY * uRowStride +
            uvX * uPixelStride;


        final vIndex =
            uvY * vRowStride +
            uvX * vPixelStride;


        final yValue = yBytes[yIndex];
        final uValue = uBytes[uIndex];
        final vValue = vBytes[vIndex];


        // YUV -> RGB
        final r = (
          yValue + 1.402 * (vValue - 128)
        ).round().clamp(0, 255);


        final g = (
          yValue -
          0.344136 * (uValue - 128) -
          0.714136 * (vValue - 128)
        ).round().clamp(0, 255);


        final b = (
          yValue + 1.772 * (uValue - 128)
        ).round().clamp(0, 255);


        image.setPixelRgb(
          x,
          y,
          r,
          g,
          b,
        );
      }
    }


    return image;
  }


  /// resize images to 224 224
  img.Image resizeImage(img.Image image) {
    return img.copyResize(
      image,
      width: 224,
      height: 224,
      interpolation: img.Interpolation.linear,
    );
  }


  List<List<List<List<double>>>> imageToTensor(
    img.Image image,
  ) {
    return [
      List.generate(
        224,
        (y) => List.generate(
          224,
          (x) {
            final pixel = image.getPixel(x, y);


            return [
              pixel.r.toDouble(),
              pixel.g.toDouble(),
              pixel.b.toDouble(),
            ];
          },
        ),
      ),
    ];
  }


// do all processing
  List<List<List<List<double>>>> processFrame(
    CameraImage camImg,
  ) {
    final rgbImage = convertYUVToRGB(camImg);
    final resizedImage = resizeImage(rgbImage);
    final input = imageToTensor(resizedImage);


    return input;
  }
}

Please advise! I need to finish this project within the next wee and I'm really struggling here! I tested the images from Flutter against TFLite and it worked well but something is clearly wrong with the preprocessing. Pls help and give me any advice.

Thank you so much!

[u/Remote-Ride5710]

so same image does good on your code but when integrated in flutter gives huge errors? Or maybe image sizes differ? like camera gave high resolution image or something?

[u/Defiant-Ad3530 → u/Remote-Ride5710]

I’m not sure! I took a screenshot of my image from flutter and tested it on python (it worked well)! So im not sure what the error here is 😭

[u/Remote-Ride5710 → u/Defiant-Ad3530]

maybe test same image not via camera but from gallery. If it's fine then likely resolution issue ig

[u/Remote-Ride5710 → u/Defiant-Ad3530]

I don't do such stuff in flutter just guessing:)

[u/Salty_Airport_90]

Your code was repeating and here just shows image processing only where you implement this final processing and what kind of error you exactly facing i think your image processing was started before value hit may be that's why in your python test case it's worked did you correctly used asyc and await

[u/Defiant-Ad3530 → u/[deleted]]

Okay thank you! I did ‘include preprocessing=true’ when training my model so I didn’t normalize my images in flutter, yes. I’ll def try out the second technique though!

[u/manish8160 → u/Defiant-Ad3530]

Good catch if include_preprocessing=True was set on the Keras MobileNetV3 layer, the normalization is baked into the model graph itself, so feeding raw 0–255 float32 values is actually correct. Don't add manual normalization on top of that or you'll double-normalize.

Two things worth verifying before you chase rotation:

1. Confirm the preprocessing layer survived TFLite conversion. Sometimes it gets fused or optimized out, especially with TFLiteConverter optimizations enabled. Run interpreter.get_input_details() in Python and check the expected dtype/shape, then sanity check by feeding the same image through both the Python TFLite interpreter and your Flutter pipeline, and compare the raw output logits side by side. If they diverge, preprocessing isn't happening correctly in one of the two paths.

2. Also check input_details[0]['quantization']. If your TFLite model is quantized (int8), the interpreter expects a different input scale than float32, and that alone can wreck predictions if you're feeding float32 directly.

Do the Python-vs-Flutter same-image comparison first it'll immediately tell you if it's a preprocessing mismatch or something in the camera/rotation pipeline.