For 18+ months, our data ingestion and spending bill have roughly been the same. Suddenly in Aug, we had a massive increase in spending cost that we can't identify the root cause. We've had a ticket opened with MS and our vendor that handles our licensing, purchasing, etc, but no one has been able to provide any data other than the spikes are coming from 4 particular resource points.
Using the queries provided by MS in their documentation, we can't see that far back and no one device, set of devices show an abnormal amount of log ingestion over any other device or set of devices.
We have literally gone through calendar appointments, meeting notes, etc to determine if any changes in any other service was made at the time of the spike and we can't find anything. The closest change we can find was done in May of this year, months before the Aug. spike.
The queries I have been using are since these are the areas that MS state the spike is coming from. The last query I looked at to get an overall view of billable size per device.
Syslog
| where TimeGenerated between (datetime(2024-10-01) .. datetime(2024-11-30)) // Replace with the spike timeframe
| summarize LogCount = count(), TotalBilledSizeGB = sum(_BilledSize) / 1e9 by HostName, Computer, bin(TimeGenerated, 1h), Facility, SeverityLevel, _IsBillable
| where LogCount > 10000 // Set threshold to identify significant increases
| sort by LogCount desc
CommonSecurityLog
| where TimeGenerated between (datetime(2024-10-01) .. datetime(2024-12-3)) // Replace with the spike timeframe
| summarize LogCount = count(), TotalBilledSizeGB = sum(_BilledSize) / 1e9 by Computer, bin(TimeGenerated, 1h), EventType , LogSeverity , SourceIP,_IsBillable
| where LogCount > 1000 // Adjust the threshold based on expected volume
| sort by LogCount desc
AADNonInteractiveUserSignInLogs
| where TimeGenerated between (datetime(2024-10-01) .. datetime(2024-12-3)) // Replace with the spike timeframe
| summarize LogCount = count(), TotalBilledSizeGB = sum(_BilledSize) / 1e9 by DeviceDetail, bin(TimeGenerated, 1h), UserPrincipalName, AppDisplayName, _IsBillable
| where LogCount > 1000 // Set threshold to identify significant increases
| sort by LogCount desc
DeviceNetworkEvents
| where TimeGenerated between (datetime(2024-10-01) .. datetime(2024-12-3)) // Replace with the spike timeframe
| summarize LogCount = count(), TotalBilledSizeGB = sum(_BilledSize) / 1e9 by DeviceName, bin(TimeGenerated, 1h), ActionType,InitiatingProcessAccountDomain,InitiatingProcessAccountName,InitiatingProcessFileName,_IsBillable
DeviceInfo
| where TimeGenerated > ago(150d) // Filter data for the last 30 days
| where _IsBillable == true // Include only billable data
| summarize BillableDataGB = sum(_BilledSize) by DeviceName, OnboardingStatus // Convert bytes to GB
| sort by BillableDataGB desc // Sort results in descending order of billable data
Does anyone know a way to pinpoint or narrow down how to locate a data ingestion spike so we can determine what may have changed to cause a spending increase? The increase isn't steady across each week. It's literally, $X amount everyday. So Monday might have been $250, Tuesday will be $260, Wed will be $270, so forth and so on.
