SDS for Insights Data Health and Monitoring

Investigate Data Health

If SDS for Insights finds issues with your data, click on Investigate Data Health.

found issues with data.

Data Health shows you up to the last 14 runs or health for last seven days.

data health.

If your data has errors, you can click on the "Download Report" link on the right side of the screen.

Prioritizing errors for remediation

When troubleshooting errors, we recommend you prioritize validation errors with the most instances for the same entity code ahead of troubleshooting any other errors. Validation errors may result in numerous subsequent errors for each instance within the same data run. Often, remediating these validation errors first will subsequently remediate many roster-related errors that are found during validation.

Matching and Validation Rules by Data Type

This table shows the matching and validation rules by data type and how the values are stored. These rules do not apply to SDS sync profile validation and error handling.

data matching and validation chart.

*States that the record is unique across all records. Globally unique identifier, though it may or may not take the form of a universal unique identifier (UUID)

** Leading zeros '0' missing on incoming value for Grade Enum, for example '1', will match defined Enum '01' and be stored as '01'. Also, [] means an array of multiple Enums can be accepted on a single record.

** For list of supported Enums see V2.1 format page

SDS for Insights Validation Rules & Descriptions

Rule Name Rule Description Rule Type Rule action Friendly message example
Internal Application Error
An unexpected error occurred.
Catastrophic Error
Stops data run
There was an application error when trying to process: {error}
File Not Found
A required file could not be found.
Catastrophic Error
Stops data run
Unable to find {expectedFileName} in the path: {Path}. Please check the path of {Name} to ensure {expectedFileName} exists or the name of the file is {expectedFileName} and re-run sync.
Operation Canceled
A problem occurred during the data run that required it to be canceled.
Catastrophic Error
Stops data run
The operation has been canceled with the message: {cancelMessage}
Header Does Not Exist
To ensure that a properly formatted file is being sent for processing that contains the defined file headers.
Catastrophic Error
Stops data run
Unable to find {expectedHeaderName} in {fileName}. Please correct extract to add or ensure it reads as {expectedHeaderName}. Upload corrected files and re-run sync.
Duplicated Column In Header
To ensure that a properly formatted file is being sent for processing that contains the defined file header and not multiple of a same header(s).
Catastrophic Error
Stops data run
Two or more {expectedHeaderName} were found in {fileName}. Please correct extract, upload corrected files, and re-run sync.
Cross Reference Mapping Not Found
Validates that linked data is found across the associated data being provided (example: missing organization for a user, missing section for enrollments).
Validation Error
Removes record from posting
A {entityType} record could not be found for {record} [if file: in {fileName}]. Confirm the {entityType} sourcedId and {record} sourcedId is correct or update data in source system and re-run sync.
Optional Cross Reference Mapping Not Found
Validates that linked data is found across the associated data being provided (example: missing organization for a user, missing section for enrollments).
Validation Warning
Removes value from record, sends record for posting
Invalid optional reference value {value} in field {field} for {entityType} was found for {record} and the value was dropped from the record to proceed with processing. Confirm the {entityType} sourcedId and {record} sourcedId is correct or update data in the source system and re-run sync.
Date Time Format Error
To ensure data values being passed have the proper format (ISO8601).
Validation Error
Removes record from posting
The {record} in {field} [if file: in {fileName}] has a date that is not in the proper format: 'YYYY-MM-DD'.
Missing Required Data
To detect missing value in a required field for a record.
Validation Error
Removes record from posting
A required value for {record} is missing in field name: {field} [if file: in {fileName}] to create the record.
Format Value Error
To ensure data being passed in a field matches the defined formatting. Also see matching and validation rules by data type.
Validation Error
Removes record from posting
The {record} found in {field} [if file: in {fileName}] does not have a properly formatted value for {field}.
Parse Error
To ensure for each record we are able associate the data in the correct column. Records may be flagged due to single commas, carriage returns found, or missing quotes.
Validation Error
Removes record from posting
Unable to parse [if api: {apiEndpoint}] [if file: in {fileName}] to find data in columns. Ensure that the delimiter in the file is a single comma (,) and carriage returns in fields are not permitted. Fields containing commas and double-quotes must be enclosed in double-quotes. If double-quotes are used to enclose a field, then a double-quote appearing inside the field must be escaped by preceding it with another double-quote.
Unique Data Constraint Violated
To ensure that there are not two or more records found with the same source id (SIS ID, orgSourceId).
Validation Warning
Removes value from record, sends record for posting
Two or more records exist for {record} [if file: in {fileName}] with the same sourcedId {entityType}.
Max Field Length Constraint Violated
To ensure data being passed does not exceed the field length resulting in missing data.
Validation Error
Removes record from posting
The value provided for {field} in {record} exceeds the maximum supported length of {length} characters.
Invalid Or Missing Reference Code
Identifies if a reference value [Enum] being passed for a record is not found to associate the record to.
Validation Error
Removes record from posting
The value {refValue} for {field} in {record} is not found in {refCodeEntity}. Please correct source system.
Optional Data Missing Corresponding Required Value
To ensure that if a record is passing a value for optional data that any additional associated data that now requires a value to also be present is also being supplied.
Validation Warning
Removes value from record, sends record for posting
A value was provided in an optional field for {record} but is missing a corresponding value in associated field to successfully post the data.
Circular reference
To ensure that if a record has a parent association that a circular reference has not been supplied.
Validation Warning
Removes value from record, sends record for posting
{entityType} {entityId} is linked in a circular reference with {entityType}(s) {entityList}. Value for {entityParentIdField} will be dropped from the record to proceed with processing.
User Mapping Identifier Not Found
To ensure a value exists based on the user matching rules configured.
Validation Error
Removes record from posting
User mapping identifier {identifierType} was not found for user {record}.
User Mapping Multiple Matches Found
If multiple Azure Active Directory accounts are found to be a match for this user, only the first AAD account will be used for the match. Additional matches will be dropped.
Validation Error
Removes secondary records from posting
User {record} with mapping identifier {identifierType}={value} found multiple matches in Azure Active Directory. Only the first match found will be used, see Azure Active Directory ObjectId {AadObjectId}.
User Mapping Conflicting Matches Found
To prevent automated association of a different user to an existing and matched Azure Active Directory account.
Validation Error
Removes record from posting
User {record} with mapping identifier {identifierType}={value} is matched to an existing mapped Azure Active Directory ObjectId {AadObjectId}. The existing match found will be used and this match will be skipped.
General Post Data Error
When submitting the record an error was returned.
Validation Error
Removes record from posting
Unable to add {record} due to {error}. Please check or update source system to correct.

Field to data types for all non-V2/2.1 CSV formats

This table shows how the date found in the various formats are treated when processing to SDS for Insights regarding Data Type association.

data type association chart.

*Data from OneRoster CSV, PowerSchool API, OneRoster API are converted to SDS v1 CSV format before provisioning processing. The converted format is what is sent to SDS for Insights if enabled.

**If not listed here, then is treated as a String.

***For list of supported Enums for grade and course subject, see V2.1 format page( enum.

Data that is not synced with SDS for Insights

Data that is not synced with SDS for Insights chart.

*Data from OneRoster CSV, PowerSchool API, OneRoster API are converted to SDS v1 CSV format before provisioning processing. The converted format is what is sent to SDS for Insights if enabled.

**Data from section | Term fields are dropped due to inconsistencies in implementations found across customers use.

Determining data awareness and active status

A user's association to a session (academic year) is based on its role, tied to an organization.

A user's association to a class is based on its role tied to an enrollment, which also includes a link to a session.

Based on the incoming data, these rules are used to determine awareness of the record and its session status in the data store.

  • The data will reflect when a new record is presented for the first time.

    • SDS for Insights will set the first seen date (time) and last modified date (time) to current, and if appropriate, it will mark record as "is active in session" to be true.
  • The data will reflect when the same record is present in the subsequent run.

    • SDS for Insights will preserve the first seen date (time) value, set the last modified date (time) to current, and leave "is active in session" to be true.
  • The data will reflect when the same record is not present in a subsequent run.

    • SDS for Insights will preserve the first seen date (time) and the last modified date (time) values and if appropriate, mark the record as "is active in session" to false.

      • Exceptions happen when organizations, people (users) and session records persist over time and are not inactivated.

      • There will be rolling updates for "inactivated". For example, if a user record is not present, the system will preserve the existing first seen date (time) and last modified date (time) values. The system will set "is active in session" to false for the users' org/role and enrollment records.

Data Statistics

Home page

After each data run, the following statistics will be generated and shown on the home page.

  • Data health | Sync status: If any errors or warnings are found, you will see the total number of each following resulting from the last data run.

    • An error means that a record did not pass validation and the entire record was removed before processing, post validation. If a record could not be processed post validation, the count will also include that record as well.
    • A warning means that one or more portions of data on the record did not pass validation. The field that had the data for the record was removed but the rest of the record and validated data remain before processing post validation.

Data health and sync status.

  • Institutional data | Academic year:
    • Displays the currently configured academic year for the incoming flow. If SDS is your source system, you'll also see the list of linked SDS sync profiles associated to the incoming flow that is linking data to the displayed academic year.

Institutional data and records.

  • Institutional data | Institution records:
    • Displays counts for the active data found and for records that passed validation, for the following categories:

      • Organizations: number of organizations that have active user roles associated to the organization.
      • Users: number of users that have an active user role associated to an organization. If a user has more than one role association to the same or different organizations, the number may be represented for each occurrence where this is true.
      • Classes: number of classes that have an active user enrollment role associated to a class.
      • Enrollments: number of active enrollments that have an active user enrollment role tied to classes.

institutional data.

  • Institutional data | Total number of users matched to AAD:
    • Displays the count for the number of users that have an active role associated to an organization and have been matched to AAD. The breakdown will display the division between student roles, staff roles and unmatched based on organizational role associations. By selecting the division breakdown bar for one of the groups it will display the corresponding count.

institutional data.

Data Health

After each data run the following statistics will be generated and shown on the data health page based on each run. To view the associated run stats, you can expand the desired data flow run.

data health.

  • Source Data: This will display the total number of records found in the source data before advanced validation. If using SDS as the source (pulling data from your provisioned SDS sync profiles), and there are multiple sync profiles that are providing the same record, like the same school record for organizations, then the number will be counted for each instance found.

    • Organizations: Count of organizations found. Non-V2 CSV files will show school records and V2 files will show all organizational records.
    • Users: Count of users found.
    • Classes: Count of classes found.
    • Enrollments: Count of enrollments found.
  • Transformed Data: Displays the total number of records found after running data validation for the following stats. For the institutional stats based on records that passed data validation and are active, see above for more info under home page.

    • Errors: For any record that did not pass validation and the entire record was removed before processing post validation. If for some reason a record could not be processed post validation, the count will also include those records.
    • Warnings: For any record where one or more portions of data on the record did not pass validation, where the field that had the data for the record was removed, but the rest of the record and validated data remained before processing post validation.
    • Matched users: Number of users that have an active role associated to an organization and have been matched to AAD.
    • Unmatched users: Number of users that have an active role associated to an organization and a match was not found to a user in AAD.

SDS V2.1 CSV file format

How to deploy SDS using SDS V2.1 CSV files