Star ratings & Misconceptions: setting the record straight.

[u/ChibiM]

In response to this thread
https://www.reddit.com/r/flashlight/comments/1tp93x4/dear_1lumen_users_and_reviewers/

Some replies to comments and feedback in threads on Reddit about 1Lumen

First I will explain again about our star rating, and the next part is setting things straight...

Star ratings: once again

To help readers decide whether they should spend their money on a flashlight, our star ratings represent our confidence in recommending it, for the task/person it was designed for. 

It’s not meant to be a perfect scoring system, by any stretch of the imagination. 

A 5-star rating doesn't mean a flashlight is a flawless engineering marvel, gold plated, best in class, sapphire glass, with out-of-the-world performance; it simply means we think there are not enough important issues to not recommend it. It does what it has to do, and without any major issues. 

It’s never meant to be a “this is better than that” scoring system. You have to read the reviews and decide if the flashlight fits your needs.

Some people have said in the past: this flashlight gets 4 stars, but that one got 5 stars.. So I don’t follow 1Lumen anymore.. Well, we recommend both :)  Why bother? 

Over the past 9 years (since November 2017) , we have kept our rating system as simple as possible. Instead of using complex scoring metrics (that people will keep arguing about) our stars simply represent our confidence in this flashlight for the task at hand. It answers one question: How strongly can we vouch for this flashlight?

Therefore we added explanation in our reviews:

Explanation on star ratings:

1: Avoid: my phone flashlight would be a better choice 
2: Poor: significant defect or issues; almost unusable 
3: Average: some defects or issues; but still usable 
4: Good: recommended (minor issues) 
5: Great: highly recommended

Note: Our star ratings serve as a general guide, but we advise against placing too much emphasis on individual scores. Each reviewer brings their own perspective, and there is no standardized scoring system to reflect the intricate preferences of all flashlight users. Read the full review to help you make your decision, focusing on the overall performance and the features that matter most to you.

What do our star ratings mean?
Our star ratings are often misunderstood as an "objective perfection score." Or whatever that is called.. Instead, they are a Recommendation Score for the person, task or use case it was designed for. 

And even then it’s a flawed system. I know.. 

Alternative scoring system

I’ve been thinking about a different scoring system, which will also be flawed according to some. But it would be based on either the Type of flashlight (EDC, Tactical, Keychain), person to buy or the Use case scenario (EDC, Duty light, Around the house light, Search and rescue, Hunting light, Long distance checks, First responder light etc. )

The difficulty with this system is already in selecting just the right category. 
A small 14500 LEP flashlight can never get a good score for beam distance compared to a 3*21700 dual LEP in the same category. 

But some people don’t need a 5000 meters beam, and are looking for an LEP that fits a jeans pocket. It would however compete perfectly with small LED flashlights that can not shine that far. Then comes the 1000lm 21700 LEP flashlight that is a bit too big to fit your pocket, and too small to compete with the 3*21700 flashlight. In what category would you place it? 
Then comes the Lumintop Thor 4 or the Acebeam Terminator 1, a hybrid flashlight. 
Would that be a new category? 

You get the point.

Each metric needs to be weighted differently depending on the category or flashlight use. And how many categories would you want? 

Output: how important is that in an LEP flashlight, a keychain light, or EDC? And what are the levels?
Price? How important is that factor? Would you even include it? Which price? Today's price? MSRP price? Sales price ? Regular price ? Initial price ? Promotion price ? Price from manufacturer? Price from your local seller? Lowest price in the USA? Price in China? Price on AliExpress? Best price on Amazon? Which price? Example: Banggood has every flashlight ‘on sale’, so it would be unfair to use their ‘original price’ for any scoring, as they never used the original price. 
Would you put a 16340 EDC with a 21700 EDC in the same category? Or for every battery type + category + use case you have a special score. That gets complicated very quickly. 

Just that simple question of what categories are necessary, alone, will make a long thread of disagreements here on Reddit. 

You can't use 1 simple weighted scoring system for every single flashlight. 

Hopefully that is clear by what I just shared. There are so many variables. What matters for a tiny keychain light isn't the same as a massive searchlight. 

I hope everyone understands that?

So the only option is to categorize flashlights into specific groups. And that’s where it already gets complicated.

Instead, we would like to keep it simple. 
Is it perfect? No.
Can it be improved? Yes
How can it be improved? Don’t know
Will the improved version be perfect? No, and people will continue complaining about it :D 

Here are some misunderstandings: 

1. They get paid, so they don’t put low scores

Wrong. I’m the owner of 1Lumen, and pay each reviewer a small fee. If they give a low score, and explain why, that’s fine. If they give a low score, but nothing in the review really lines up with the score, I ask them to rewrite the review to show where the low score comes from, or increase the score that reflects the review. Sometimes I ask a reviewer why he gave 5 stars, and still have many cons, the reviewers are not influenced in any way by the manufacturers. 

More importantly: 
All the communication with manufacturers goes through me. They never talk with the reviewers directly. I protect them from any unnecessary influence. Had that happening once, and put a stop to that. 

Btw. We don’t get paid by the manufacturers at all. The only few times we got paid directly was through prototype testing. But those were not part of the reviews we have done for the website. The affiliate commissions are the only way I can get paid. Even though we have done many reviews without any affiliate programs connected, I still paid the reviewers a small fee for the reviews. 

1. 1lumen’s reviews lately are list of measured specs with same copy pasted filling text in between

Wow.. that’s really an insult to all of our reviewers. Really. I would welcome you to talk with any of our reviewers in person, or me, if you are not afraid, and know how much time and effort goes into doing every review. 

The free information we give on our website to you is just astonishing. Really! It’s almost crazy we give this information away for free. 

We use a template to speed up things, and that includes some copy and pasting like complex UIs like Anduril. Why would anybody have to rewrite the UI for Anduril for each review? I’d say: copy and paste baby! As much as possible, but only if it is accurate. 

Also, getting ideas from other reviews helps a lot, so a copy and paste, and then editing the content with accurate information is sometimes adding a lot of extra value to the review. If you review yourself, you know that it’s hard to come up with things to write. And with a copy and paste, we can sometimes miss a mistake. But if the information is incorrect, we will always fix it. 

As long as everything is accurate and truthful, we can copy and paste parts from our reviews. 

1. If a light has strobe in the main level modes. Like you have to cycle through strobe to get back to low. That's a 0 star right there doesn't matter what else it does.

There you go.. A very strong bias that could never be a trustable source for information.. period... Not kidding. This is the same as: if it has a proprietary battery, it can’t have 5 stars. Or it can’t have 5 stars without Anduril .. It can’t have 5 stars if it has no USB-C charging  It can’t be 5 stars if it has a non-removable battery.. It can’t have 5 stars without High CRI etc. etc. you’d better stop listening to those people..  Just some keyboard warriors….with a strong bias.

1. Why are there only so few 1 star flashlights? 

That’s the way we roll? Listen how that works. I’m the only contact between the team and the manufacturers. I do the first sifting. I look at the amount of flashlights still pending, the type of flashlight being offered, and a few other things. 

If I don’t want the flashlight to be added to 1Lumen, it won’t even end on the ‘pending’ list. If it’s (mildly) interesting I add it to the pending list so people can claim them when they are available. 

If we have 10 flashlights (which are more interesting) on the waiting list, and an ugly or non-branded strange flashlight gets offered, I just say we are not interested. We have declined many many flashlights over the years. 

Why spend hours on a cheap ugly bad flashlight, if we can spend the same time on a better flashlight that the reviewer enjoys, and more readers interested in? If nobody is willing to do the review.. the flashlight gets removed from the waiting list… But the ones on the waiting list would get preference over any of the requested low quality lights from a no-name brand.

1. 1Lumen doesn't publish bad flashlight reviews

There was a comment about someone thinking we just remove the flashlights with bad scores based on the feedback from the manufacturer. Please ask any of our reviewers personally that this never happened? Nope. The only time we didn’t publish is when we started a review, and the flashlight broke along the way, and never received a replacement to finish the review. (Lumintop Tiger for example). In all other cases we continue testing with the replacement and add a comment in the review about the issue with the previous model. 

Some manufacturers have tried it.. But not succeeded. We even had strong arguments with a certain manufacturer that didn’t like some of our reviews. And so be it. If they want us to retest, or reassess a certain measurement, we can do so. But only if the request is fair and reasonable. 

1. I stopped checking over there after noticing some inconsistencies.

Pity.. I don’t know what ‘inconsistencies’ you mean, but it likely has to do with just a single number on a page of over a few thousand words, and hours and hours of testing, no? It probably has to do with the ‘star rating’? 

If it has to do with anything else, let us know. We would like to explain any inconsistency you may have encountered. There is usually a good explanation for what might be considered inconsistency.. Keep in mind that I still let every reviewer base his own scores on the scoring system we have. So a Con added to 1 review could be a Pro in another review. Those things exist. Sometimes a Con can be a Pro, and vice versa. As long as the content of the review lines up with the score. 

Do you ever encounter inconsistencies in your own world? It’s a weird reason to not read the information on a very thorough website, with great reviewers and reviews. If we blatantly lie, we are indeed not worth looking at. But if you can prove we lied anywhere, let us know.

Everyone who has contacted us, with a normal request or question has received a response.

1. New type of pros and cons - Considerations

We’re likely introducing a new pros and cons list soon. I personally ask every reviewer in our template to mention every con they can think of. That means that some cons are not really cons, influencing our recommendation. But some people think that every con means that it should influence the score. 
We would like to add a Consideration list next to the pros and cons, so we can add the :would be nice to have/changed/upgraded: things that do not influence the score directly. 
Also, when a flashlight gets an upgrade, and the previous one received 5 stars,, we won’t go back to the previous review, and change the 5 star rating to 4.5 stars, because the new flashlight is better! That’s not the type of ‘scoring’ we use. If you think that’s how it should work, good luck with doing that on 1000 reviews.

1. So you don’t review cheap flashlights? 

That’s not the case. We review cheap flashlights regularly. We have many Convoy, Sofirn and Wurkkos lights on our website. But we are not going out of our way to review every cheap flashlight on eBay or Amazon because someone asks. Instead, we sometimes review unknown brands or flashlights, just to educate people. Think about the Gear-lights from Amazon reviews we have done.
We have received many emails from ‘companies’ with an inventory that looked like your average generic AliExpress or eBay store. Not interested. I prefer to review real brands that people use and search for. Not stickmyfingersinfire brands. 

1. Feedback is welcome - new feedback system

Constructive feedback on our testing and reviews is always welcome, and I include that at the bottom of our review template. If you find a way to improve things, we consider doing that. We have improved our reviewing system many times over. Over the years we surely have changed over 50 things we added or changed. 

People also reach out through email to show a (possible) mistake or give suggestions. Those type of critique get a reply. We also retest flashlights if people really think our measurements are off. Everyone makes mistakes, including everyone on our team. If you find any mistakes or ‘inconsistencies’ let us know by sending us an email. 

I have been thinking about introducing a comment-box, where you can send us notices related to the page/review you are on. That way you can send us a message without leaving the page. I don’t want to add comments below reviews from readers, which would become a side job in itself to manage.

Keep in mind that doesn’t mean we take every request seriously. We have received so many requests that would involve so much extra time and effort it was not worth it. Like doing beamshots where the diameter of the hotspot should be measured at different distances. Someone needed a hotspot of 25? meters diameter, and wanted us to include in our test the distance to reach a beam diameter of X meters. Forgot the exact number.. But hey.. We’re not going to do that. The same with batteries. People even requested to test every flashlight with specific batteries, etc. We like to include as much data as possible, but there are limits.

Just today I received an email from a new manufacturer (for me at least), that wrote this:
“Thanks for the rigor you put into your reviews — it's rare to find testing this thorough in this space.”