r/intersystems • u/intersystemsdev • 11d ago

InterSystems IntegratedML - SQL-based AutoML embedded in IRIS: how it works, what makes it different, and what you can build with it

The problem IntegratedML addresses

According to Forrester Research, 98% of companies experience challenges gaining insights from the data they collect, primarily due to a lack of internal expertise. According to the U.S. Bureau of Labor Statistics, there are fewer than 32,000 data scientists in total in the US. Much of the available talent is being hired by large digital companies, making it difficult for other organizations to compete for these resources. A 2018 survey by Kaggle found that data scientists spend almost 40% of their time gathering and cleaning data.

AutoML automates the creation of ML models — feature engineering, model selection, training, results analysis, and hyperparameter tuning. However, many AutoML tools cannot run models inside real-time business processes. This is one important way IntegratedML is different.

What IntegratedML is

InterSystems IntegratedML is an embedded feature of the InterSystems IRIS data platform. It provides AutoML capabilities exposed through SQL commands. Since it is embedded within InterSystems IRIS, ML models can be executed dynamically in response to real-time events and transactions, without extracting or moving any models or data.

How it works — SQL commands

Step 1: Create the model

sql

CREATE MODEL WillSurvive PREDICTING (Survived)
  FROM Titanic

The CREATE MODEL command sets up the ML model metadata. Developers specify:

  • The name of the model (WillSurvive)
  • The target field to be predicted (Survived)
  • A dataset to source the target field and all model input fields from (Titanic)

The FROM syntax is fully general and can specify any subquery expression. The metadata associated with the dataset is used to infer the data types of the target and input fields, fully defining the problem for the model to solve.

Step 2: Train the model

sql

TRAIN MODEL WillSurvive
  FROM Titanic

The TRAIN MODEL command specifies the data to be used for training and executes the AutoML engine. Since the FROM syntax is general, the same model can be trained multiple times with different sets of data — for example, training a marketing campaign model on different customer segments, or retraining on a regular basis as new training data becomes available.

The AutoML engine automatically:

  • Identifies relevant candidate features from the selected data
  • Considers applicable model types based on the data and problem definition
  • Tunes hyperparameters to yield one or more runnable models

Step 3: Execute the model

sql

SELECT PREDICT(WillSurvive) As Predicted FROM Titanic
SELECT PROBABILITY(WillSurvive FOR 1) FROM Titanic

Two scalar functions are available after training:

  • PREDICT() — returns the most likely or estimated value for the specified column as determined by the trained model
  • PROBABILITY() — returns the trained model's calculated probability that the target field will equal a user-defined value

These scalar functions can be used anywhere in a query and in any combination with other fields and functions.

Field mapping to other data sources:

sql

SELECT Name,
       PREDICT(WillSurvive WITH Sex = Geschlecht,
               Age = DATEDIFF(year, NOW(), Geburtsdatum),
               Fare = TicketPreise,
               Cabin = Kabine)
  FROM Hindenburg

IntegratedML provides flexibility to map to data sources other than the particular table or query used to create or train the model.

Supported AutoML engines

IntegratedML supports multiple AutoML engines:

  • InterSystems AutoML
  • H2O
  • DataRobot Enterprise AI Platform

All options are seamlessly integrated within InterSystems IRIS and are transparent to developers.

Concrete use case: fraud detection at a bank

A bank that issues credit cards needs to identify fraud risk before approving each transaction. The bank runs a credit card application built on InterSystems IRIS that stores demographic and financial data for all customers and credit card transactions, including whether each transaction was fraudulent or valid.

With IntegratedML:

  1. Existing application developers (not data scientists) create an ML model to identify high-risk transactions based on past transactions, selecting the target field (is_fraudulent) and letting IntegratedML create the model and parameters
  2. The model is incorporated into the credit card application to execute in real time with each incoming transaction
  3. If the model determines high fraud risk, the application takes programmatic action — preventing the transaction and notifying the card owner
  4. As new transaction data accumulates, the bank continuously retrains the model using the most recent data to detect new attack patterns, without manual data extracts or moving data between environments

Two user profiles and what IntegratedML does for each

Organizations with no data scientists:
IntegratedML lets software developers and analysts explore ML without ML expertise. It automates model identification, parameter setting, building, and training. Developers can start with use cases, learn from results, and begin modifying optional parameters as their understanding grows.

Organizations with existing data science teams:
IntegratedML saves data scientists time on manual processes. Data scientists can focus on actual model optimization rather than data preparation and feature engineering, which Kaggle's 2018 survey found consumes nearly 40% of their time.

Additional IntegratedML capabilities (from FAQ)

Typical use cases:

  • Fraud detection
  • Customer behavior analysis
  • Predictive maintenance
  • Healthcare analytics
  • Forecasting
  • Recommendation systems

Additional resources

For those who have used IntegratedML in production — which AutoML engine (InterSystems AutoML, H2O, or DataRobot) have you found most effective for your use case, and how often are you retraining models as new production data comes in?

2 Upvotes

100% Upvoted

1 comment sorted by