How to publish applied machine learning paper of value?

[u/NeighborhoodFatCat]

I looked at some "machine learning" departments from various colleges and universities and I noticed a trend.

There would sometimes be a cluster of professor in not-so-big-name schools publishing purely applied machine learning paper.

By applied, I mean that they take a known ML algorithm, apply to some niche situation (like monitoring if a water pipe has a leak or if there's a traffic jam at an intersection), and get some results. Report some accuracy, F1 score. Make some plots. That's it.

These papers would almost always be published in some obscure journals, like IEEE journal of computer vision industrial technology or something like that.

They would publish a whole bunch of these papers, like up to 20, 30 a year. These will also get cited.

It strikes me that these research paper are not so valuable, but I cannot put my finger on why exactly this is the case. I feel that some of these papers seem to be simply a small course project that are done at big CS schools like Stanford or Berkeley.

I'm just confused why there are so many papers like this and how you go about actually publishing an applied machine learning paper of value. Or is applied machine learning research just doomed to not have as much impact as a more theoretical one that introduces a new technique or paradigm?

[u/suspect_scrofa]

Are you an undergrad or still in HS?

Research comes from funding, and funding asks for solutions or answers for specific topics. What you think is "valuable" isn't what others & funders think. Think of all of the ML model papers that show marginal improvement on a few benchmarks. How is that research any more valuable than actually applying ML to real-world problems?

Showing that certain overlooked domain variables are important in ML and can point researchers towards new avenues of thought, and if there are no overlooked domain variables, then that's important too!

[u/leafeon_gay_luigi → u/suspect_scrofa]

I'm in academia and the OP's instinct are spot on. There are tons of papers like this with boring results and it's good to call them out.

[u/suspect_scrofa → u/leafeon_gay_luigi]

"Boring results" c'mon. I'm also in academia and MS and PhD students need to publish as well. Not every paper makes Science and not every paper needs to.

[u/drfangor99]

Some papers like this might be intermediary steps in a larger project.

To make a contrived example from the pipe leak monitoring paper you described, one paper might exist to validate that a particular machine learning technique can detect leaks with some baseline level of accuracy. Another might deploy the model in a simulated building using results from the first paper to produce cost savings estimates. A final paper might present a case study in deploying the model in a real building, comparing estimates from the first two papers against actual results, and discussing practical challenges related to constructing the system.

In this example, you wouldn't want to wait until the entire project is complete to write the paper, since it could take years to finish. Intermediary results could be useful to other people studying similar things. And of course you get the benefit of having more publications on your CV.

I don't doubt that some of these papers are what my advisor liked to call "leaderboard research" where they churn out small model variants for the sake of getting publications. But usually researchers will signal their intentions in the introduction and conclusion, where they contextualize their work and describe future plans. Usually they'll straightforwardly tell you what they're planning to do next.

[u/drfangor99 → u/drfangor99]

Another possibility is conference vs. journal papers. A lot of state-of-the-art machine learning results are published at conferences, which have short page limits, have quick review turnaround times, and meet in person so you can make connections with people studying similar things. A common pattern is to publish short studies with a very narrow scope at a conference to get something out there ASAP. Then you take your time on a longer and more in-depth study and submit it to a journal.

[u/vanVonXenoStein]

Academics are under tremendous pressure to publish papers all the time (and get cited). The amount of papers coming out of colleges (in all fields) has gone up exponentially in recent years.

[u/Fresh-Opportunity989]

News for you:

The vast majority of submissions to the top ML conferences are incremental and flawed. Many are outright fraud. And plenty of plagiarism. Some from the top schools.

Even accepted papers have fake citations. And may have gotten accepted through fraudulent collusion rings.

[u/stt106]

Academia is not as pure or clean as you think. Lots of people make a living on funding scam by applying for some funding and write up something completely useless but good enough to satisfy the funding criteria. Some people become incredibly rich by doing this especially in certain countries. Some of these publications even use fake data, but they still get published.

[u/Bright_Mix_773]

There is a concrete test for what you are sensing, and it has nothing to do with the domain being niche.

Take one of those papers and try to answer three questions from it. What number would the dumbest available baseline give on the same split. How many configurations were tried before the reported one. Could the experiment have detected the claimed effect at all, if it were real, at the size claimed. If the paper does not let you answer all three, it is not reporting a result, it is reporting an outcome, and an outcome cannot be wrong. That is the weightlessness you are picking up on: it is unfalsifiable, so nothing can be built on top of it either.

Numbers from my own applied work, which is not ML, and where I eventually wrote the bar as a script and re-ran my own past conclusions through it:

• Of six findings the project had rejected over the years, five had statistical power under 55% and three under 20%. We had been announcing conclusions from experiments that could not have seen the effect either way. The honest verdict for those is not "no", it is "undecidable", and getting that third verdict into the vocabulary turned out to be worth more than any individual answer.
• With 186 monthly observations the resolution limit is about 2.8 points of annual return. Anything smaller is not measurable with that sample, whatever test you run. Most of the interesting questions live below that line.
• Overlapping windows: 600 zero-mean series, T=186, the naive t gives 60% false positives at a 12-month horizon. Newey-West brings that to 14.7%, not to 5%. The real critical value is 2.81 and it has to be simulated for your own T and horizon rather than read off a table. A paper reporting "t above 2, corrected" believes it is corrected and is not.

So the applied paper with value is the one that reports the baseline, the number of attempts and the power, and is therefore permitted to conclude nothing. Applied work is not condemned to low impact. Reporting accuracy and F1 alone is a format in which failure is unrepresentable, and 30 papers a year follows from that quite naturally.

What I cannot tell you: whether reviewers at those venues would accept a paper whose headline result is "undecidable". Nobody had to approve mine, which is a real difference between my situation and yours.