azure eventhub - Stream Analytics Job - Collect output data error message associated with individual event

Question

The project that I am working on has integration between Event Hub, Stream Analytics, and SQL Server. The message/payload coming from EH has multiple records and a table in the SQL server gets populated with those records through the configured SA job.

If the records associated with the payload has invalid data type or null values, SA job records those errors in the configured diagnostic logs along with the EventCount.

The issue which I am facing is, SA job only writes the error message associated with the first item to the log and groups the remaining items part of the payload. This is not helping me to figure out the exact issues associated with the other records in the payload.

How can I get the individual events and associated error message from SA job?

Following is the KQL query that I am using to list the errors part of the SA job:

AzureDiagnostics | where ResourceProvider == "MICROSOFT.STREAMANALYTICS" and parse_json(properties_s).Type == "DataError" | project TimeGenerated, Resource, Region_s, OperationName, properties_s, Level, _ResourceId | order by TimeGenerated desc

Any help is much appreciated.

question from:https://stackoverflow.com/questions/65879347/stream-analytics-job-collect-output-data-error-message-associated-with-individ

Answer 1

Waitting for answers