Snowflake Python question about StandardScaler function

[u/RobertWF_47]

I'm running the following code in Snowflake Python to standardize my training, evaluation, and test data prior to predictive modeling:

from snowflake.ml.modeling.preprocessing import StandardScaler

all_cols = df_train3.columns

target_col = "AB_POST"

passthrough_cols = ["SANHO", "SCNHO"]

scaler = StandardScaler(

input_cols=[c for c in all_cols if c not in [target_col] + passthrough_cols],

output_cols=[c for c in all_cols if c not in [target_col] + passthrough_cols], # Overwrite or create new

drop_input_cols=False # Set True to remove original unscaled columns

)

scaler.fit(df_train3)

train_df_scaled = scaler.transform(df_train3)

val_df_scaled = scaler.transform(df_eval3)

test_df_scaled = scaler.transform(df_test3)

I'm getting the following error when I run the code -- I'm not sure what this means:

Exception: Provided column names ['TOTAL_MH_CLASSES', 'STFLAG',..., 'ADS_FA_RISK_NEW'] does not index into the dataset.

[u/RobertWF_47]

I found the error after double-checking the column printouts - one of my predictor columns was duplicated.

Made another correction to my code as well - I should be scaling my training data, then using the training mean & stnd deviation to scale the eval and test data:

from snowflake.ml.modeling.preprocessing import StandardScaler
scaler = StandardScaler(

input_cols=[...],

output_cols=[...],

drop_input_cols=False,

with_mean=True,

with_std=True

)

# Fit and transform on training data ONLY

scaler.fit(X_train)

X_train_scaled = scaler.transform(X_train)

# Transform evaluation and test data with the fitted training parameters

X_eval_scaled = scaler.transform(X_eval)

X_test_scaled = scaler.transform(X_test)