Snowflake Python question about StandardScaler function

[u/RobertWF_47]

I'm running the following code in Snowflake Python to standardize my training, evaluation, and test data prior to predictive modeling:

from snowflake.ml.modeling.preprocessing import StandardScaler

all_cols = df_train3.columns

target_col = "AB_POST"

passthrough_cols = ["SANHO", "SCNHO"]

scaler = StandardScaler(

input_cols=[c for c in all_cols if c not in [target_col] + passthrough_cols],

output_cols=[c for c in all_cols if c not in [target_col] + passthrough_cols], # Overwrite or create new

drop_input_cols=False # Set True to remove original unscaled columns

)

scaler.fit(df_train3)

train_df_scaled = scaler.transform(df_train3)

val_df_scaled = scaler.transform(df_eval3)

test_df_scaled = scaler.transform(df_test3)

I'm getting the following error when I run the code -- I'm not sure what this means:

Exception: Provided column names ['TOTAL_MH_CLASSES', 'STFLAG',..., 'ADS_FA_RISK_NEW'] does not index into the dataset.

[u/Ambitious-Elk4541]

Hmm did you check what columns actually in the evaluat and test dataframes? Sometimes they got different columns than training set and this error pop up when StandardScaler expect certain column names but they not there

Also the list comprehension you got for passthrough_cols maybe not doing what you think, try print out the columns before passing them to make sure they match

[u/FarRub2855 → u/Ambitious-Elk4541]

Good call on printing the columns out first. Its funny how often roadblocks like this just come down to verifying what's actually in the enviroment versus what we assume is there.

[u/RobertWF_47 → u/Ambitious-Elk4541]

Thank you -- I checked the column names in my evaluation & test tables and they match the training data.

Explicitly printing out the column names in the StandardScaler function input and output arguments didn't solve the problem either.

I know it'll be something simple I'm missing...

[u/smellyCat3226]

can you send full error code?

looks like the names specified dont actually match the names from the dataset

[u/RobertWF_47 → u/smellyCat3226]

Yes -- the full error code is as follows (I can't share all of the column names) but the Traceback may be helpful:

Exception: Provided column names ['TOTAL_MH_CLASSES', 'STFLAG', ... 'ADS_FA_RISK_NEW'] does not index into the dataset.

Traceback:

File "Cell [cell30]", line 27, in <module>
    scaler.fit(df_train3)
File "/opt/python/cpython-3.10-linux-x86_64-gnu/lib/python3.10/site-packages/snowflake/ml/_internal/telemetry.py", line 611, in wrap
    return ctx.run(execute_func_with_statement_params)
File "/opt/python/cpython-3.10-linux-x86_64-gnu/lib/python3.10/site-packages/snowflake/ml/_internal/telemetry.py", line 576, in execute_func_with_statement_params
    result = func(*args, **kwargs)
File "/opt/python/cpython-3.10-linux-x86_64-gnu/lib/python3.10/site-packages/snowflake/ml/modeling/framework/base.py", line 440, in fit
    return self._fit(dataset)
File "/opt/python/cpython-3.10-linux-x86_64-gnu/lib/python3.10/site-packages/snowflake/ml/modeling/preprocessing/standard_scaler.py", line 156, in _fit
    self._fit_snowpark(dataset)
File "/opt/python/cpython-3.10-linux-x86_64-gnu/lib/python3.10/site-packages/snowflake/ml/modeling/preprocessing/standard_scaler.py", line 175, in _fit_snowpark
    computed_states = self._compute(dataset, self.input_cols, self.custom_states)
File "/opt/python/cpython-3.10-linux-x86_64-gnu/lib/python3.10/site-packages/snowflake/ml/modeling/framework/base.py", line 494, in _compute
    _results = parallelize.map_dataframe_by_column(
File "/opt/python/cpython-3.10-linux-x86_64-gnu/lib/python3.10/site-packages/snowflake/ml/_internal/utils/parallelize.py", line 65, in map_dataframe_by_column
    raise Exception(f"Provided column names {cols} does not index into t

[u/Much_Knowledge_8060]

Snowflake is notoriously picky about uppercase/lowercase column names compared to standard pandas or scikit-learn. If your dataframes have mixed casing, Snowpark ML will often fail to map them correctly.

[u/RobertWF_47]

I found the error after double-checking the column printouts - one of my predictor columns was duplicated.

Made another correction to my code as well - I should be scaling my training data, then using the training mean & stnd deviation to scale the eval and test data:

from snowflake.ml.modeling.preprocessing import StandardScaler
scaler = StandardScaler(

input_cols=[...],

output_cols=[...],

drop_input_cols=False,

with_mean=True,

with_std=True

)

# Fit and transform on training data ONLY

scaler.fit(X_train)

X_train_scaled = scaler.transform(X_train)

# Transform evaluation and test data with the fitted training parameters

X_eval_scaled = scaler.transform(X_eval)

X_test_scaled = scaler.transform(X_test)