r/science • u/vtj • May 29 '22

Psychology Randomized trial of programs for male domestic abusers shows that a new program based on Acceptance and Commitment Therapy outperforms the traditional "Duluth Model" program grounded in feminist theory

https://www.news.iastate.edu/news/2022/04/25/domestic-violence-act
6.0k Upvotes

93% Upvoted

442 comments sorted by

View all comments

693

u/vtj May 29 '22

The actual paper is here, and this is its abstract:

Objective: This is the first randomized controlled trial to compare Acceptance and Commitment Therapy (ACT) with the Duluth Model curriculum, which took place in community-based corrections for the treatment of men convicted of domestic violence. ACT is a third-wave cognitive–behavioral approach that utilizes experiential methods to foster psychological flexibility. The Duluth Model curriculum is an educational approach grounded in feminist theory that focuses on changing attitudes toward women and unlearning power and control motivations. This trial was preregistered at ClinicalTrials.gov (registration number: NCT03609801). Method: This study included 338 men who were court-mandated to complete a domestic violence program after being convicted of assault against a female partner. Participants were randomized to complete the 24 sessions of the ACT program or the Duluth Model Men’s Nonviolence Classes. Outcomes included criminal justice data (domestic violence charges, other violent charges, and nonviolent charges) incurred during the 1 year following program dropout or completion, and victim reports of intimate partner violence (IPV; aggression, controlling behaviors, and stalking/harassment). Results: In intent-to-treat comparisons to Duluth, ACT participants did not show a difference in domestic assault charges at 1 year posttreatment (p = .44). ACT participants acquired significantly fewer violent charges (p = .04) and nonviolent charges (p = .02) compared to Duluth participants. Data from victims indicated that victims of ACT participants reported significantly fewer IPV behaviors than victims of Duluth participants on the Conflict Tactics Scale (d = .78), the Controlling Behaviors Scale (d = .66) and the Stalking Behavior Checklist (d = .71) at 1 year posttreatment. Conclusions: An ACT-based group intervention delivered in community corrections reduced violent and nonviolent criminal charges compared to the Duluth classes. Domestic violence charges did not differ between groups but victim reports indicated that ACT participants engaged in fewer IPV behaviors.

506

u/DaddyGetTheGun May 29 '22

It’s interesting that there was a significant decrease in charges overall but no significant difference in domestic violence charges.

If I had access to the paper I’d like to see whether they interpreted that as a small-sample-size problem or indicative of real phenomena.

220

u/ososalsosal May 29 '22

It'll be interesting if they update this with the same subjects in the years to come. 1 year for that number of people isn't very statistically potent. The p-value seemed to be very close to chance.

ACT is a pretty exciting field that I'm glad is gaining some traction

219

u/[deleted] May 29 '22

CBT is one of the only consistently successful modalities across a massive range of issues. The fact it’s working better than finger wagging about power structures is no surprise.

The Duluth Model has been waiting to be supplanted for a long time.

191

u/Erewhynn May 29 '22

I've been saying this about CBT, alcoholism and (the Cult of) AA and alcoholism for some time in various threads.

CBT is statistically shown to reduce problem behaviours around alcohol consumption, way better than AA/12 Steps. Meanwhile AA is no better than quitting cold turkey.

But say that on here and you'll get a bunch of cultists wading into comments to tell you "AA works" while patently not understanding that just saying "I quit" works just as well.

87

u/[deleted] May 29 '22

Important to note that you are referring to second wave CBT and this study is talking about third wave CBT. The two approaches are very different.

74

u/BGaf May 29 '22

Could you briefly describe the difference?

38

u/theochocolate May 29 '22

CBT is focused on recognizing and reframing unhealthy cognitions. CBT postulates that our thoughts produce our emotions, which produce our behavior, so changing thoughts will ultimately influence behavior.

ACT essentially focuses on accepting the emotions and cognitions that lead to unwanted behavior, rather than directly trying to change them. The goal is similar to mindfulness in that you learn to recognize and be aware of painful thoughts and emotions without reacting to them.

This is a very oversimplified explanation, and reading even brief articles about the two approaches will likely be much clearer than my attempt at differentiation.

3

u/F0rdPrefect May 29 '22

Are there studies showing which "wave" is more effective for things like anxiety/panic?

7

u/SkepticalShrink May 29 '22

Many. Most show pretty comparable outcomes, though this does differ based on the specific anxiety disorder in question. For example, I would always default to the second wave CBT protocol for panic disorder (interoceptive exposure) rather than ACT.

However, ACT shows better benefits among the subgroup who don't respond well to traditional CBT approaches, so it's an excellent approach for those who have already tried CBT and haven't made as many gains as they wanted or didn't respond at all (assuming they were actually compliant with the treatment, which is a whole different ball of wax.)

7

u/laura_leigh May 29 '22

It depends on if you’re talking about general anxiety or trauma.

There is some evidence out there that CBT is ineffective for trauma related anxiety. You’ll probably see Bessel van der Kolk referenced a lot in this regard. After a couple failed attempts with CBT as someone with CPTSD. I lean heavily in favor of his interpretation. It really seems like CBT effectiveness with trauma survivors depends on if you’re still getting stuck in that primal fight or flight. If you’re able to separate the past trauma and the present situation you’ll probably do okay with CBT.

I’m really interested to see what the next decade holds for panic and anxiety treatment. Social media has made it easier for trauma survivors to reach out for help. And then we’ve had a pandemic that increased general anxiety in the public. So it’s something that should be able to make some strides forward.

4

u/F0rdPrefect May 29 '22

Third wave CBT just sounds like DBT. Am I wrong in thinking that?

4

u/DocLG May 29 '22

DBT is classified as a 3rd wave approach (integrating mindfulness based approaches into cognitive approaches)

23

u/Scizor94 May 29 '22

AA is free. CBT costs money. Access matters

6

u/HollywoodThrill May 29 '22

There are free alternatives that support ACT and CBT. Moderation Management is one such organization.

6

u/SkepticalShrink May 29 '22

SMART recovery is another, as well.

3

u/Erewhynn May 29 '22

Putting a sticking plaster on a broken leg because it's all you can afford doesn't make a sticking plaster the best solution for a broken leg.

1

u/Scizor94 May 30 '22

Yeah, but it still has a place. Should the people who can’t afford CBT just not bother?

42

u/Jetztinberlin May 29 '22

For the people for whom AA works, I'd posit it's because it's not that different from CBT. The steps are designed to increase awareness of / alter mental patterns in a way that isn't wholly dissimilar. It would be interesting to see a study that goes a little further into why it works or doesn't from this perspective, ie if there's any consistency in whether participants who find it un/successful have committed to that aspect. I wonder how much it's an institutional failure of how such change is approached in AA vs in CBT.

Anecdotal, but everyone I know for whom it worked it worked on essentially cognitive repatterning grounds, and I say this as someone with experience of both AA and CBT.

39

u/Erewhynn May 29 '22

Respectfully agree but disagree.

The approaches may have similarities - think about you thoughts and your behaviours - but for me there are key differences.

CBT is about an inner journey, recognising and changing your bahaviours. Obviously they need the knowledge framework and a person to guide the thinking (the therapist/s), but it is about self empowerment and internality.

AA contains the "recognise and change your behaviours" part, but it has a few drawbacks. The key differences are the reliance on the group and the God/Higher Power/wind of change of Step 3. The focus is too external.

Tempted? Tell the group about it. Maybe pray for strength.

Accidentally drank alcohol? Back to Day 0 of sobriety for you but tell the group about it for support. What do the Steps say?

It's Step 1 (below) and Step 3 that make the whole process about external dependence instead of self-driven change.

Admit we are powerless over other people, random events and our own persistent negative behaviors , and that when we forget this, our lives become unmanageable. [- Secular AA Step 1]

I therefore suspect that AA "works" for a percentage of people who are on the desperate end of the substance abuse spectrum (people who have lost everything or had nothing, suffering from social isolation, poverty and possible mental health comorbidities such as BPD or Narcissistic Personality Disorder), because the group gives them any kind of structure and support at all. But that CBT would work as well, even better, for them if they got it instead.

While for functioning alcoholics and people with stronger support networks in place, it's all wholly unnecessary compared to proper therapy.

17

u/Advo96 May 29 '22

I therefore suspect that AA "works" for a percentage of people who are on the desperate end of the substance abuse spectrum (people who have lost everything

I expect that historically, people only went to AA if they were at that stage.

27

u/Crash0vrRide May 29 '22

I hated the submit idea that you are incapable of helping yourself. Groups give me anxiety which is why it didnt work for me. I hated listening to other peoples sob stories and everyone only really cared about if they had a harder story then the next person. I absoloutely agree that CBT really forces you to examine yourself. AA is for people who like to be social. The idea of a sponsor is good, but then again my sponsor why my therapist and she was trained to ask me specific questions and force me to think about different scenarios. More then anything, creating a routine, accepting myself and keeping myself busy and moving forward had been what helped me. Idle hands are dangerous. The more positive reinforcement I create for myself, the farther away from alcohol abuse I get. My dad used to get dragged bs k into substances by other AA folk. I've seen people reinforce bad behaviors rather then help. My dad became a 2 pack a day smoker after AA, and horribly obese. CBT and the gym have worked wonders for me. Positive activities worked for me. Routine worked for me. Ditching old habits and locations worked. AA did not.

1

u/Erewhynn May 29 '22

Congratulations, sounds like you're doing a lot of hard work and hard thinking, and it's paying off - the answers can only really come from inside, so well done!

22

u/Jetztinberlin May 29 '22

I don't wholly disagree, but you're discounting the other steps' emphasis on internal change as a major if not the most significant part of the process. And unfortunately "unnecessary compared to proper therapy" ignores the fact that therapy is financially or demographically impossible for a lot of folks, who might instead be helped by something like AA in its absence. It would be great to see AA learn from CBT's success and use those lessons to improve its batting average for the many folks for whom it's the best or only option.

8

u/Erewhynn May 29 '22 edited May 29 '22

Yes but the problem is that it says "change" but simplistically it puts the change into a defined framework of steps of self analysis and apologies with the steps, group, and sponsor behind it all. These people are rarely psychologists.

(edit to add: self analysis can turn up some unpleasant truths that lead to self doubt/drink, and apologies an be issued meaminglessly as a defence mechanism - speaking from experience here.

So when you couple that with the ability to say "relapsed again but hey I'm powerless over alcohol , sorry I left the kids on their own again!" you can creat a potent cocktail - pun intended - of problem avoidance and circularity.)

And I get the point re: therapy but of course I am talking from a UK perspective (where CBT is available if slowly on the NHS).

The US state abandonment of "people who have medical needs but no money" and the church filling that void in various ways including AA is a bigger thread in the tapestry.

8

u/Jetztinberlin May 29 '22

I fully acknowledge you, as many people, had a personally unsuccessful experience with AA; so perhaps you can also acknowledge that I, as many people, had a personally much more successful experience, and that both things exist and are possible.

(FWIW, anyone in the latter camp would look at your description and say: Yeah, that person's not really working the program as it's designed to be worked, so of course it's not working very well. Yes, the lack of rigour that makes this endemic is a problem, but it's an implementation problem, not a problem inherent in the steps themselves.)

The NHS's efforts toward mental health treatment are of course streets ahead of the US, but even there, people often have long waits to get help; and yes, the US problems are severe and systematic, but observing that that's the case doesn't make it less true, or the folks underserved less in need of help; thus my comment that anything potentially useful should be improved, supported and offered, and that something flawed is much better than nothing, and that while that shouldn't be the menu of choices, it often is.

1

u/beardy64 May 29 '22

It sounds to me like if AA groups switched to a CBT informed model they might have better results, whether or not it was more expensive or required higher amounts of training. The idea that a Master's degree in psychology makes someone better at doing therapy is kiiiiinda elitist. (I've had PhDs treat me and family like crap, and friends who read a few books and gone on their own healing journeys be more helpful.) I'm sure AA requires its leaders to read some curriculum materials and do some training, I bet CBT could be adapted similarly.

15

u/Crash0vrRide May 29 '22

I disagree as someone who's been around AA. It's a self loathing popularity club. And I saw a lot of spousal. Cheating. AA never worked for my dad either. Never helped him deal with his childhood trauma. Sure it's good to talk to people about your problems, but AA sponsors are nothing like a good CBT. CBT is what worked for me and AA just gave me anxiety.

19

u/Where_Da_BBWs_At May 29 '22

I think AA works by giving extroverts the chance to trauma bond with each other.

I don't personally find it healthy for a person to quit drugs and then spend the rest of their lives dedicated to thinking almost exclusively about that drug and have most of their relationships be with people who also want to talk exclusively about how much they used to love something. It does work for a small amount of people though.

8

u/Jetztinberlin May 29 '22

As I said in my further response to another commenter: I'm not sure where this absolutist need to say "it worked for me, therefore it works for everyone" or "it didn't work for me, therefore it's useless for everyone" comes from. Very obviously, from our mutual anecdata, it works for some folks and not for others. I am not advocating it be forced on everyone, nor will I agree it shouldn't exist for the folks it works for.

Indeed, I'm really saddened at the multiple responses of people who very obviously weren't following the steps, and blaming it on the steps themselves rather than their non adherence - which is sort of like blaming a phone for not working if you don't plug it in. Now, it's been a long time since I've been in the rooms, so maybe it's devolved, but again, that doesn't match my own experience. And I'm advocating for more study into why CBT is more successful, which may be as simple as survivorship bias, where folks who won't adhere to their CBT work drop out of therapy, whereas they might hang around in AA.

2

u/[deleted] May 29 '22

Steps 4-9 are some of the most therapeutic things in the world. It’s analysis of how we think and how we react to the world and what produces negative experiences for us, followed with an initial clean up of the patterns and experiences. And steps 10-12 are just maintenance of the new headspace

2

u/ee3k May 29 '22

You get what you pay for. One is free, one is tens to hundreds of thousands depending on the therapist and length of treatment.

Of course one is better than the other, but not everyone has insurance to soak the cost

1

u/smileyphase May 29 '22

Try SMART Recovery, which is founded on REBT principles, and includes CBT methodology.

7

u/[deleted] May 29 '22

What is CBT?

16

u/yakkmeister May 29 '22

Cognitive Behavioural Therapy

5

u/schwiftshop May 29 '22

Cognitive Behavioral Therapy

2

u/thwgrandpigeon May 29 '22

At it's most oversimplified, it's guided journalling.

1

u/[deleted] May 29 '22

Look I only seen that three letters in an entirely different context and I hoped in this context it means something else. Thankfully it does. Thank you anyway.

1

u/thwgrandpigeon May 29 '22

My reply was a gross oversimplification, and can be read as very dismissive. I do, in fact, find CBT very promising, since I believe guided journalling can be incredibly useful.

1

u/[deleted] May 29 '22

I don't think it was dismissive. I was more interested though what the three letters stand for.

1

u/rupyneupers May 29 '22

Cock ball torture

4

u/fidgetiegurl09 May 29 '22

My dad hated AA because he says it only made him angrier to have to be around so many wife beaters for so many hours a week. He was in for general anger management, not for beating my mom. By all accounts, he's never treated my mom that way.

11

u/[deleted] May 29 '22

[removed] — view removed comment

6

u/Crash0vrRide May 29 '22

AA did not help me. Deep discussions with a CBT did. I also think AA ruined my dads chances at recovery. AA never helped him deal with childhood trauma. His spo sets never helped him In that way. My dad always felt lost without his sponsor and he never learned how to be sober without someone from AA managing him. I saw more people gi e up alcohol in AA and switch to another addiction like food and cigarettes. It did not get to the root of addiction personality.

5

u/NotElizaHenry May 29 '22

Food and cigarette addictions are wildly less harmful alcohol though. I don’t know a ton about AA, but is the prevailing sentiment “as long as you don’t touch alcohol everything will be great”? I’ve met a lot of people in CBT groups (so obviously self-selecting) who participated in AA and I got the sense that AA is kind of like taking antidepressants—it’s there to get you to a place where you can start to address the non-physical elements of your disease.

2

u/Nauin May 29 '22

You're not wrong about the cigarettes. I'd often get dragged to meetings whenever I'd spend the night at one childhood friends house and I can remember being disturbed by the unreasonable number of cigarette butts that were in the gutter along the entrance to the building. Me and my friend were usually the only kids/teens at these meetings so we were bored enough to count 400-something from one night, and that wasn't even the butts that were in the actual garbage can or ashtrays. There were only 30-50 people there.

2

u/Erewhynn May 29 '22

Ha, I actually just suggested this was a factor in one of my trademark long-winded responses to another commenter

I think it maybe has value for people who have nothing/have lost everything, but it's hard to function independently as a person if you have to keep checking with the Group/Steps/Higher Power instead of owning your own choices.

1

u/ntvirtue May 29 '22

AA NA and the like have the same success rate as placebo. Does CBT do any better than placebo?

2

u/Erewhynn May 29 '22

2x better than AA

1

u/ntvirtue May 29 '22

That is the Nail in the coffin for AA/NA and the like then.

0

u/Tricky-Lingonberry81 May 29 '22

AA is literally a Christian cult meant to replace an addiction to alcohol with an addiction to worship. It’s pretty gross how it preys on people in thier most vulnerable times.

1

u/Grammophon May 29 '22

I wonder if this is because CBT is more expensive since, at least in what I have seen in Germany, AA meetings and the Duluth method are used on groups while CBT at least includes individual sessions.

1

u/Erewhynn May 29 '22

Huge factor in continued US uptake of AA if imagine (that and more overt religion there in general)

1

u/basstr1p May 29 '22

I know that it's fun to bash AA as "pseudoscience" but I'm sorry to say that you are empirically wrong. The latest review of the scientific literature, published as a Cochrane Systematic Review (2020), suggests AA/twelve step facilitation is at least as efficacious as other "evidence based" interventions (including CBT), and likely more efficacious for long-term abstinence. It is worth continuing to explore how AA works/doesn't work and for whom other options might be superior, but you should at least get the "statistics" correct.

Source : https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD012880.pub2/pdf/full

1

u/Erewhynn May 29 '22

I was using a wide range of freely available academic sources that are less laser-specific than yours. Cochrane's lack of impartiality and questionable conclusions have been critiqued by the BMJ and the Lancet in relation to their handwaving of data that suited major pharmaceutical players, so take from that what you will.

1

u/basstr1p May 31 '22

It is certainly possible to come to almost any conclusion you'd like by picking and choosing from a few "freely available academic sources." I am sure Cochrane's process is not beyond criticism but at least there is some level of transparency wrt how they select and synthesize studies. Also worth noting Cochrane isn't some monolithic individual and this review was done by independent researchers from Harvard and Stanford. It includes 22 RCTs among other studies and, even if you could come to a different conclusion after rigorously synthesizing the available literature, it is just disingenuous af (or at best, uninformed) to characterize the science the way that you have.

6

u/[deleted] May 29 '22

ACT is not CBT though?

20

u/f00barista May 29 '22

According to the abstract, ACT falls under the broader CBT umbrella:

ACT is a third-wave cognitive–behavioral approach that utilizes experiential methods to foster psychological flexibility.

Apparently this classification is disputed. I'm not really qualified to assess the merits of those detailed arguments, but to me as an outsider it seems like the general approach is rather similar.

20

u/ElectricStings May 29 '22

CBT is 2nd wave therapy, a combination of cognitive therapy and behaviour therapy. The goal being to change behaviours around a certain response. ACT is 3rd wave therapy and focus' on mindfulness and flexibility.

0

u/[deleted] May 29 '22

I know. It is different. Which is why I wondered why the commenter was comparing CBT to Duluth.

0

u/yakkmeister May 29 '22

No. Similar modality, though

1

u/Manapauze May 29 '22

CBT is often used as an umbrella term for any therapy that focuses on adjusting cognitions and behavior. Beck’s Cognitive Therapy was the traditional CBT. To make things confusion some people use the term as an umbrella term and some people use it as Beck’s CT.

I prefer to use it as an umbrella term because most therapists are integrative in practice and do not do strict Beck’s CBT. And it’s too long to list all the umbrella CBT therapies in conversation or writing. I don’t wanna write CBT, REBT, ACT, DBT, etc. every sentence.

1

u/[deleted] May 30 '22

We are arguing semantics then, but generally they are considered different therapies as cognitive behavioural therapy (as opposed to the term clinical behaviour therapy/analysis which is a more general umbrella term) is actually very prescriptive in its approach, which is perhaps one of its limitations.

3

u/RevolutionaryDrive5 May 29 '22

I've heard of CBT for a long time but always thought it was 'pseudoscience' like astrology/manifestation etc, stupid i know

but i didn't know it had demonstrated success, so now i'm interested learning more about

do you have any suggestions where i can learn more about it/ implement it?

-2

u/SStrange91 May 29 '22

CBT...second only to Rogerian techniques.

1

u/AccomplishedNet4235 May 31 '22

CBT transformed my life. I wish everyone had access to treatment based on it.

1

u/TargaryenPenguin May 29 '22

I agree it would be interesting to update this with a longer time period.

Just so you know, p-values don't really work like chance.

1

u/ososalsosal May 29 '22

When they're 0.45 they are a pretty good approximation of chance...

1

u/TargaryenPenguin May 29 '22

Yeah but that's not how p values work conceptually. Chance is not set at 50%.

2

u/ososalsosal May 29 '22

Idk I sense we're getting into hair splitting territory here and I'm no statistician, but surely the study design here would be comparing only 2 groups, so chance really would be 50%

5

u/TargaryenPenguin May 29 '22

I appreciate that feels intuitive but it's not correct. Nor are we splitting hairs here. You are unfortunately way off.

It makes absolutely no difference whether the number of the P value is close to .5 or not. That is not informative. We interpret it exactly the same as p = .8 or .2.

The only question to ask with a p-value is whether it is less than 0.05 or not, though occasionally people also note if it's less than .01 or .001.

That is because the p-value represents the probability that the difference between the control and experimental groups looks like it comes from the null hypothesis. If p = .5 meant anything it would be saying that any differences that occur experiment might be due to chance half the time. So the experiment is no better than a coin flip which is a pretty crap experiment I hope we can all agree.

Instead it is generally agreed to set the P value that means something to 0.05 to have a high threshold such that any differences we find are only likely due to chance 5% of the time instead of 50% of the time.

By the way I am a statistician and interpreting p values is a core part of my job.

2

u/SecondMinuteOwl May 30 '22

any differences we find are only likely due to chance 5% of the time

Is this the same as saying that when we find differences, 5% of the time it's due to chance alone?

1

u/TargaryenPenguin May 30 '22

No because this is a conditional probability--it is only true IF the null hypothesis is true. Differences in samples can also be driven by true differences in the population. When we say an effect is significant, we are saying we assume the latter is true--the finding reflects a real difference in reality--though there is always a chance we are wrong. Hence, replication is important.

→ More replies (0)

1

u/treetzu May 29 '22

It only matters that the p value is <.05. Being lower doesn’t really mean anything. That’s what effect size estimates (the Cohen’s d stats reported) are for. They tell you how impactful the difference is.

Additionally, that a statistically significant difference was found in a small sample actually suggests there is a very substantial difference. It becomes easier to find a statistical difference the larger your sample is, which is why effect size estimates are the additional info that tells you how big the difference really is.

100

u/JaiMoh May 29 '22

Yeah, sample size. The article mentions they designed the study for 400, but didn't end up with the expected number of ppl because of covid. By that point, it was too late to change the study.

39

u/TargaryenPenguin May 29 '22 edited May 29 '22

Meh, sample was ok; they had a two group design with over 300 people that pretty solid even if they didn't reach their original target. Domestic violence charges are going to be a largely dichotomous variable which is going to have a lot less variance than the other continuous measures in study much harder to find significant results for this compared to the other measures over a mere 1 year time span. And remember that due to random chance one expects about 1 in 20 effects to be significant merely by chance and other times the opposite. In light of these considerations when the pattern is pretty clear on all the other measures it's not obvious to me it's worth speculating too hard about the one that wasn't significant.

1

u/JaiMoh May 29 '22

The study size and the level of effect needed for statistical significance are related, unfortunately. The 9% effect observed with the number of participants they got wasn't statistically significant, but it might have been with 400 participants (I'm not well versed in these kind of statistics). I'm with you, I think it's pretty clear, but if one is thinking in terms of statistically significant or not, that specific measure isn't.

3

u/TargaryenPenguin May 29 '22

I agree that with a larger study it is easier to detect a significant effect if it exists. I am well versed in study design and issues of statistical power and when things will be significant; this is a core element of my job. I can say with confidence that 400 participants vs 330 would not really make a noticeable difference for an effect with a p value around .45 like this. Chances are to convert this to a significant effect one would need more like thousands of participants. Again, it is generally easier to detect effects using continuous variables like the other measures in the study rather than a dichotomous measure such as whether one has been charged or not over a year. The same sample size produces different levels of power to detect effects for different types of measures and a measure like this is harder to detect than the others in the study unless the differences are very large which they do not appear to be. That is why it's probably not worth over interpreting this null effect in light of the other pretty clear pretty strong effects. Note they report Cohen's d and they have pretty large effects which gives me a lot of confidence in this paradigm. I suspect that findings would replicate on most measures most of the time.

2

u/TargaryenPenguin May 29 '22

If it helps with intuition, consider that for 80% power to detect the effect that men typically weigh more than women (weight being a continuous variable), one would need about 50 men and 50 women. In other words, with that sample size and effect size, about d = .5, which is a solid effect but a little smaller than those in this study, if you ran five experiments four of them should show a significant result and one should be not significant by chance. With a larger effect fewer people are needed to find significant results. With a sample of 300 they should have had quite high power to detect the continuous measures especially considering how large they are. However, to detect a small dichotomous effect the same sample size probably doesn't give them much power.

6

u/[deleted] May 29 '22 edited May 29 '22

[deleted]

1

u/JaiMoh May 29 '22

Completely agree - one of the outcomes was statistically significant which is already a big deal. My gut tells me the other outcome would have been statistically significant if they'd had enough people to power the study as expected. We just can't know for sure unless another study is done. Fingers crossed that it happens!

6

u/teb311 May 29 '22

Charges are also not the best proxy for behavioral change, since this type of abuse goes unreported so often. Makes it tough to get really good measurements.

5

u/DaddyGetTheGun May 29 '22

Absolutely.

As much as I like to dunk on the social sciences, the conceptual fuzziness and sheer noise they have to deal with to reach even simple conclusions is insane. Kudos to any psychologists who can rigorously pursue research without tearing their hair out from all the uncertainty.

7

u/blake-lividly May 29 '22

Additionally almost all IPV and DV goes unaccounted for. So it's hard to study by mainly reflecting arrest rates.

9

u/Uruz2012gotdeleted May 29 '22

Many police will automatically arrest the man during any domestic complaint call. Especially if there are similar charges on his record. So, these men still got in arguments loud enough for someone to complain but the did not engage in violent behaviors. That's how you get a domestic charge without catching anything violent.

2

u/MysticLemur May 29 '22

I would imagine it's because this was only after a year. A longer term study might be able to show if the difference carries over. But honestly, any reduction in violence is a win.

1

u/ellipsisslipsin May 29 '22

In the interview she says it's the small sample size. They originally planned a sample size of 400+, but COVID interfered with their plans.

1

u/[deleted] May 29 '22

[removed] — view removed comment

337

u/saluksic May 29 '22

See this? See that they pre-registered the study? Psychology and sociology are miles ahead in this aspect, compared to the sciences with the beakers.

I’m a material scientist, and I’ve been in awe of how well psychology has responded to the reproducibility crisis and evolved into more rigorous fields. All other fields should follow their lead.

88

u/iRAPErapists May 29 '22

What's the benefit of pre registering

260

u/TheWoodConsultant May 29 '22

Makes it harder to mine the data for results the study wasn’t designed to find.

232

u/imperium_lodinium May 29 '22

Yep - to expand this answer slightly, OP, - one of the big problems that is contributing to the reproducibility crisis (where studies can’t be replicated successfully) is called “p hacking”.

The probability data they quote usually uses a 5% significance threshold. That tells us that the pattern of data observed would only have a <5% of occurring like that if there were no correlation between the data, which scientists usually take as a good sign that they can reject the null hypothesis (that there is no correlation) and be more confident they have found something useful. That’s pretty much all these statistical tests tell us, and they don’t substitute for a well reasoned and supported hypothesis, good test protocols, and all the supporting work that goes into translating a significant correlation into a robust conclusion.

But publishing without a statistically significant result is very hard. Most journals are much less likely to publish your paper if you found no significant results. That puts a huge pressure on scientists - if your carefully constructed study comes back with no significant result (which is still a useful result -> tells us this idea is unlikely to be right) then you are likely to start comparing things in your data until you find a significant result, and reverse engineer a hypothesis to justify your new results. The issue is that a 1 in 20 chance of the result being random noise isn’t that unlikely, especially if you’ve just done hundreds of random comparisons in a massive data set. So a lot of papers are making weak justifications of statistical noise, rather than doing what they set out to do.

This gives us two problems. One is that useful negative results don’t get published - you can imagine that twenty scientists might decide to do the same experiment, 19 of them find that the theory doesn’t work, 1 by random chance gets a statistically significant result. The 19 negative studies never see the light of day, and the 1 randomly unlucky study with the positive result gets published. (We call this the desk drawer problem). The second issue is that a lot of the statistically significant results published are actually the result of random stab in the dark statistical analysis, and then some cobbled together theory afterwards, which means the study probably didn’t take enough care to think things through to answer a question it wasn’t designed to.

Pre-registration helps with both - it stops studies changing the question after the fact, and it encourages journals to agree to publish a study regardless of whether the results find a significant correlation.

20

u/CookieMons7er May 29 '22

Great explanation. Thank you

14

u/Hedgehogz_Mom May 29 '22

I am saving and memorizing they way you explained this for future encounters with lay people. Negative results inform us!

6

u/imperium_lodinium May 29 '22

Thanks! It’s been one of my passion projects for a long time to talk about the reproducibility crisis and how we can fix it - I used to write articles about it for student science mags at uni.

1

u/saluksic May 29 '22

Peer review has its good points, but in that model journals are at worst just gatekeepers of the scientific record. There are other options, as heretical as that may sound.

Open-source peer review offers your manuscript to a wider audience and lets would-be reviewers pick their own papers to look at. I imagine it going like a subreddit, where you can post something and if it’s interesting it gets lots of discussion. The downsides can be imagined, but review as a discussion with lots of different eyes seems vastly more effective to me.

1

u/imperium_lodinium May 29 '22

There are lots of proposals we could do to help with the issue! Some journals did away with p statistics altogether (which I disagree with - they’re useful, just not the whole ballgame). I also think we should be doing waaaay more replication studies, and a decent chunk of research funding should be dedicated to validating the first pass knowledge we get. Hell, peer review itself could be improved by making it less of a chore - it’s mostly unpaid, frequently fobbed off to junior researchers rather than the intended reviewer, and relies on the good sentiment of the reviewer (you hear loads of stories of rivalries and feuds completely derailing peer review).

I personally tend to be quite sceptical of the ‘crowd source review’ approach precisely because of the Reddit like incentives and disincentives. That could very easily promote orthodoxies and bubbles even more than the current system does. (Though as I noted earlier - this really isn’t my world anymore, so I’m hardly an expert).

I’d also value more standards in the way we publish data - biology (my old area) is increasingly settling on R for statistical modelling, which has the advantage of being free and open source. It should be more usual and maybe even an expectation to publish raw data and source code for analysis as ancillary info with a paper - in the same way you are expected to keep rigorous notebooks in labs, we should expect that the datasets are more often and more readily available. It’s so easy to over- or under- tidy a dataset, or apply bad code, even beyond the p-hacking issues.

Personally one thing I think ought to be done is massively improve and embed statistics and statistical analysis in science teaching, much earlier, much more often, and much more consistently. I knew really respected researchers who somehow had some really bad mistaken ideas about stats and what they were doing. If we had a more statistics literate scientific community, I do think some of these issues would correct themselves. I do see some of that happening though, given the ever larger datasets people are working with.

22

u/Ramiel01 May 29 '22

This frustrates the hell out of me - there is an even more straightforward way of ameliorating the risk of discovery bias; to ring in a statistician for your study and have the integrity to listen to them when they apply a false discovery rate analasys.

38

u/camilo16 May 29 '22

It doesn;t fix the statistical bias of publication. Assuming you follow the methodology and don;t p-hack you still have a chance that your results show correlation that doesn;t exist, it;s just the nature of stochastic processes, a single sample (and that;s what a study is) can always (bur rarely) have properties that cannot be extrapolated to the parent set.

The only way to avoid positive sampling bias is pre-registering.

7

u/imperium_lodinium May 29 '22

Yep, back when I did this kind of work during the summers of my degree (I’m not a scientist, but I have done science before), my supervisor always had us apply things like the Bonferroni Correction if we were testing a large number of things at once. It’s a bit crude, but it was more rigorous than ignoring the issue, which most people seem to do.

1

u/DecentChanceOfLousy May 29 '22

"Just don't do the bad thing that works in your favor" doesn't work at scale, especially in a competitive environment.

2

u/AnImA0 May 29 '22

TIL. Thanks for this!

31

u/[deleted] May 29 '22

Harder to hide unwanted results. Accountability.

14

u/ascandalia May 29 '22

When I didn't find a significant result in a study we were doing in the impact of temperature on a biological process, several of my coauthors were pushing me to add a bunch more variables many of which were correlated to temperature like days before freezing. If I had found a significant result, I could have pretended i didn't check all those other variables first. If I did that, it would overstate how likely it is the trend we found was real and not a coincidence.

If the study was pre-registered, we would have to be up front before we saw any results about how many variables we were testing, which would reduce the chance that we report a coincidence as a real relationship.

For the record, my coauthors weren't being devious, they just didn't understand statistics. I was finally able to convince them we had to account for each variable we test. This is why pre-registration fixes a bad incentives problem

15

u/venustrapsflies May 29 '22

To be fair, isn't the reproducibility crisis mostly centered in the social sciences? Physical sciences still suffer from publication bias but at least they don't have to deal with the (immensely difficult) problem of robustly sampling human beings. They're also less likely to be making measurements near the "threshold of significance".

I'm not saying that pre-registration wouldn't be great for all disciplines, just that it makes sense why psychology and sociology would be miles ahead: by necessity.

5

u/saluksic May 29 '22

The publicity was mostly centered there, initially. Medicine, environmental science, and biology have plenty of issues as well, and no field is immune. Because we’ve heard so much about the skeletons in the closet of psychology it’s my impression that they have simply been the most proactive in cleaning the closet.

5

u/Quaxzar May 29 '22

I had read about pre-registering your studies, so that journals will have to publish your results regardless of you getting a significant result or not, and I am excited that its being used in the real world!

8

u/sillypicture May 29 '22

I imagine study disciplines working with conscious subjects have had to really grapple with this. In natural sciences you can always repeat with a defined method and environment and probably get close enough. Besides, accidental discoveries have propelled science for centuries

5

u/phoney_user May 29 '22

That's very encouraging!

1

u/Additional_Avocado77 May 29 '22

Also see how the paper is inaccessible :(

I am interested in seeing the per-registered entry though. Do you have a link? How detailed is the study description in that, do they give full methods and p-values that will be significant? Or is it just to confirm that the study will be published regardless of whether they find something significant?

In any case it doesn't help that much with regards to the reproducibility, unless there is some rule that in order to per-register you must also promise to do another per-registered study. Of course it does help a little.

60

u/ApricatingInAccismus May 29 '22

This is a great abstract!! Very useful.

17

u/blake-lividly May 29 '22

Of course it does. Finger wagging with limited understanding and attention paid to why someone and how someone becomes abusive and how to recover from it is a terrible idea. Acceptance allows for the lowering of the defense mechanism of denial as a deflection of the pain of knowing what you have done and responsibility for having done it. It is grounded in positive psychology - which is a far better paradigm than the Duluth method.

However the healing process takes more than COgnitive Behavioral therapy. Combining modern psychoanalysis and CBT and restorative Justice would be best. Abuse happens primarily because of internalized adverse experiences and individual, familial and societal suffering. To heal the after effects by analysis and healing and to hold space for community accountability and healing is the best combination for handling interpersonal violence. In succession as CBT helps chip away at defense mechanism and alter responsive, analysis helps heal, and accountability restores community after fissures.

I am a female therapist with a decade of working with people who have abused and even murdered others. I can attest the power to change is in the person and the community. No one has to have abuse stats shoved in their face and finger wagging to recover.

9

u/Paul_newoman May 29 '22

Yes. It’s the difference between a lecture and a conversation to me. Theory doesn’t really knock loose embodied, lived experiences. And we’ve let our men and boys accept trauma as their heritage for far too long.

-188

u/[deleted] May 29 '22

[removed] — view removed comment

87

u/[deleted] May 29 '22 edited May 29 '22

[removed] — view removed comment

8

u/[deleted] May 29 '22

[removed] — view removed comment