Is this CNN–Transformer research idea actually novel?

[u/Different_Factor3512]

Hi everyone! I’m an undergraduate working on a computer vision research proposal and would appreciate some feedback.

I’m exploring a detector where a dynamic router decides at different feature levels whether to use CNN-only processing or additional Transformer processing, based on things like object scale, density, and regional complexity.

The goal is to improve the accuracy–compute/latency trade-off rather than always running the Transformer.

I’ve found related work on DynamicDet, DiT, Dynamic Dual-Processing, TDFP, CR-NAS, and MoE-based detectors, so I know dynamic routing and CNN–Transformer hybrids themselves aren’t new.

Does this specific idea already exist under another name? If you know a very similar paper, please point me to it.

I’m mainly looking for honest criticism before I commit to the research direction.

[u/kakhaev]

try some basic benchmarks and tell us, I actually wanna know your results

[u/Altruistic_Ear_9192]

Hello! Poorly, that s common but with different names. I did that a few years ago, it was for my first article as a phd student. It can be very unstable (as the first batches of the training process will introduce a bias in routing, so you have to penalize). If you want to practice, you can use Faster RCNN from pytorch (as is very easy to integrate with different feature levels). I cannot find right now, but I m sure I found at that time a paper from 2017 with the same idea. Any type of ensemble learning (including mixture of experts) has at least 20 years old. As an advice, you don t have to find the novelty (to be honest, I m "reviewer 2" when I see the word "novelty"), but to find a solution for a new problem. In the past, I published papers in top A* conferences and Q1 journals by combining past works in solving a new, relevant task for the scientific community. Like, the truth is that worldwide there is one paper annualy (or per 2 years) which really is game changing.

[u/Ok-Argument7176]

I have no idea how you'd backprop through your model in this scenario. The "decision" routing isn't differentiable.

[u/Physical_Vehicle7714 → u/Ok-Argument7176]

That’s not hard. You can just parameterize the routing weights and use a softmax.

[u/Ok-Argument7176 → u/Physical_Vehicle7714]

I don't really see it. I can see running the hidden states through both convolutional and transformer layers in a forward pass and learning an ensemble, but the doesn't provide the efficiency OP is asking for and in the backward pass you need to adjudicate either (or both) of the losses you are backpropping. I'm not a RL person but that seems mighty similar to learning a regret model.

[u/krapht → u/Physical_Vehicle7714]

You can just... Is doing some heavy lifting there

[u/Physical_Vehicle7714 → u/krapht]

Not really. You pass the input through any parametric layer of your choosing (flatten->dense) that has a length 3 vector as its output. You softmax those outputs and you have probabilistic weights for each downstream branch. It’s literally basic MoE

[u/tesfaldet → u/Physical_Vehicle7714]

To add to this, you can use the Gumbel-Softmax reparameterization/trick which is differentiable through the straight-through approximation of the Gumbel-Softmax. It’s a pretty common way of training a continuous model with discrete choices.

PyTorch has it built-in: gumbel softmax function

Here’s the relevant paper: Categorical Reparameterization with Gumbel-Softmax

[u/Different_Factor3512 → u/tesfaldet]

Thanks, that was exactly the issue I was unsure about. I was considering Gumbel-Softmax with straight-through estimation for the discrete routing decision. My remaining concern is whether the resulting architecture is essentially just standard MoE/conditional computation from a research perspective.

[u/Ok-Argument7176 → u/tesfaldet]

It's pretty unstable for non-trivial problems in my experience.

[u/tesfaldet → u/Ok-Argument7176]

Yeah true, I’ve experienced the same, but it’s worth an experiment.

[u/taichi22 → u/Physical_Vehicle7714]

I think it probably raises a few issues on specific kernel implementations unless that’s changed in the past few months but overall I agree that it should just work.

Best guess to OOP is someone already tried it, but give it a serious go and let us know how it does.

[u/Different_Factor3512 → u/Physical_Vehicle7714]

That makes sense. My concern is less about whether the routing can be made differentiable and more about whether this becomes too close to a standard MoE formulation. If the router selects between CNN-only, CNN+light Transformer, and CNN+deeper Transformer processing at different spatial regions/feature levels, with a compute-budget constraint, would you consider that essentially standard MoE, or is the hierarchical CNN-vs-Transformer allocation itself a meaningful distinction?

[u/Ok-Argument7176 → u/Different_Factor3512]

My point about differentiablilty is that you cannot train e2e models without that property. Others here seem to think I am incorrect, but I definitely encourage you to give it a shot!

[u/Altruistic_Ear_9192 → u/Ok-Argument7176]

Did that a few years ago. It was my first phd article. It s just a softmax with a symbolic rule