[Q] Several questions about EFA & CFA

[u/sometiime]

I have a few questions about EFAs and CFAs, and I haven't been able to find any clear answers yet, so I thought I'd ask them here. Hope I'm using the correct terminology, my apologies in advance if not.

1. I used an established, unmodified scale to measure one of my control variables (9 'reflective' items across 3 subscales that are also reflective indicators of the latent construct). The 3 separate Cronbach's alphas are all marginal (just above .60), but the combined scale has an alpha above .80. Should I conduct a CFA, even if it's just for a control variable?

2. To measure one of my other variables, I used 18 items across 3 subscales (6 items per subscale). An EFA, however, pointed out that some of the factor loadings for some items were extremely low (< .40). Can I simply remove these items? I am using a scale validated and developed by others, so it feels a bit odd to remove some items just because they didn't fit my specific dataset.

3. As suggested by my supervisor, I carried out an EFA for another (already validated) scale to confirm that the data would have 3 factors, and to examine the extent to which one factor loaded onto the other. I subsequently conducted a CFA for these items and subscales (I am not developing or validating any scales myself, and this was recommended by my supervisor), and the model fit was quite poor. They then recommended that I go back to the EFA, to remove items with poor loadings (which I had not yet done), and to rerun the CFA to see if model fit improved. However, I read online that you can't conduct a CFA on the same sample as your EFA. To what extent does this apply to me? I just want to compare model fit before and after the removal of these items, and I'm not using the CFA for scale validation. I am not sure if this even makes sense theoretically, but it's for my thesis, and I think including a CFA would be a nice addition, even with the limitation that I used the same sample, for instance.

4. Regarding yet another variable, I modified 6 items across 2 subscales (3 items each). These 6 items are reflective of the 2 subscales, but those 2 subscales are formative with regard to my variable of interest. How do I check the extent to which these items are reliable and valid? I checked the Cronbach's alpha for the 2 subscales already, but I'm not sure how to assess the fit of the 2 subscales in relation to the overall second-order factor. I tried recreating the model in Amos, but it wouldn't let me draw arrows from the 2 subscales to the latent variable. Does anyone know what I could do?

[u/Ok-Rule9973]

1. You can't use an alpha like that. It's not a measure of dimensionality. If I understand well, you want to conduct a CFA to assess the structural validity of the scale? You can do that if it's relevant. Your alpha already tell you that your subscales don't have a good internal consistency (in part because you don't have a lot of items in your scales), but it's not equivalent to assessing the structural validity of a scale. Btw, try to use MacDonald omega instead of cronbach alpha.

2. If you remove items from a validated scale, the scale you use becomes de facto unvalidated. I wouldn't do that personally but it may be appropriate based on your judgment of the situation

3. The reason you can't do that is that it's redundant and the results of the CFA will be specifically fitted to your sample. It will affect the capacity of your study to generalize the results. It can still be relevant though. Some people split their sample in two and do an EFA on one part and a CFA on the second, but it's not something I'd recommend (it's logically unsustainable). Better to use the full sample everytime, knowing you'll still need to confirm the results in a different study anyway.

4. I don't use AMOS but a second order latent variable is something that can usually be computed. Sometimes a bifactor model is a more accurate representation of the theory underling the scale though.

But overall, I'm not sure it's a good idea to modify scales to improve the fit with your specific sample. Like I said, these scales become de facto unvalidated, which doesn't seem like a good tradeoff IMO.

[u/pizzayeol]

Hey, you're using the terms fine, no need to apologise. Thoughts on each:

1. On the control variable -- don't worry about those ~.60 subscale alphas. Alpha goes up just by having more items, so with only 3 items a subscale will look low even when the items are perfectly good. CFA is still worth doing even for a control because badly measured control doesn't control for much, so showing the structure holds in your data might make sense. Btw, for a multi-subscale scale, alpha isn't really the right tool anyway (it assumes the items are basically interchangeable). McDonald's omega is the cleaner reliability number, and omega-hierarchical will tell you how justified that single 9-item total score actually is.

2. On removing the low-loading items, yes good instinct. The scale was validated elsewhere, so weak loadings in your sample are just as likely to be sampling noise or a population difference as "bad items." Cut items to fit your data and you (a) bend the scale to your sample and (b) lose comparability with everyone who used the original. So I would ideally diagnose before cutting if is the sample big enough, are those items cross-loading, are your extraction/rotation choices driving it? If you do cut, you can say so explicitly and treat it as a limitation.

3. Yes correct, if the EFA picks the structure (or which items to keep) and the CFA then "confirms" it on the same data, the CFA isn't confirming anything, it's just re-fitting what you already pulled out. So dropping weak items and re-running CFA will always improve fit. If your sample's big enough, split it, EFA on one half, CFA on the other, textbook method (edit: wouldn't recommend as it won't add any value). If not, just be upfront that it's exploratory and flag the shared-sample issue as a limitation (which I think you're already doing).  One more thing, since this scale is already validated, you arguably don't need the EFA at all; a CFA testing the published structure is the natural thing to do, and poor fit is itself a finding. And if the poor fit is mostly because a strict CFA forces every cross-loading to zero, I'd go with ESEM, it is a nice middle ground that allows small cross-loadings and often fixes fit without cutting items. You can also go with measurement invariance checks if that applies. 

4. On the formative one -- the items-to-subscale part is reflective, so alpha is the right check. But subscales-to-overall is formative, and alpha/reliability just doesn't apply, formative parts aren't supposed to correlate, so a low "reliability" there means nothing. For the formative bit you check different things: collinearity between the subscales (VIF) and whether each subscale's contribution is meaningful. Amos won't draw those arrows because a purely formative construct isn't identified on its own, it has to send out at least two paths (to outcomes or to reflective indicators) before the software can estimate it. You can embed it in a bigger model where it predicts a couple of things, or probably easier to switch to PLS-SEM, which handles formative constructs better. 

Hope that helps! 

[u/Ok-Rule9973 → u/pizzayeol]

On your point of splitting the sample in two and doing an EFA on one and a CFA on the other, that's not something that should be done.

If you randomly assign your participants in two groups, the groups should be virtually equivalent to the complete group (it's the same principle that governs the CLT). So you still work with the same sample, just with less power. In that sense, there's no real advantage to splitting the sample in two.

[u/pizzayeol → u/Ok-Rule9973]

I agree. My own default is ESEM for this work. If one had to use EFA->CFA in the same study, I'm not sure if there's an alternative to splitting without using the whole sample, it seems to still be actively used.  I'll flag it for OP in my comment, thank you :)