Further investigation revealed minimal data loss during the rollback. Affected customers have been contacted directly.
Posted Jul 11, 2025 - 17:14 EDT
Update
We are continuing to monitor for any further issues.
Posted Jul 11, 2025 - 14:27 EDT
Update
We have finished restoring the database and other dependent infrastructure. All systems should be fully operational.
No customer usage data was lost and all entitlements should be up to date with all usage.
UPDATE: Any Rules, Company Overrides, and Plan Entitlements created between 11:33 AM EDT and 12:40 PM EDT were lost in the process of restoring the database to a healthy state. These will need to be recreated. We advise all customers who were making updates in Schematic around the time of the incident to double check their plans and entitlements are now configured as they expect.
Posted Jul 11, 2025 - 14:24 EDT
Monitoring
We have finished restoring the database and other dependent infrastructure. All systems should be fully operational.
No customer usage data was lost and all entitlements should be up to date with all usage.
There were some rules, company overrides, and plan entitlements created recently that were lost in the restore. We will provide a timeframe of this dataloss soon.
Posted Jul 11, 2025 - 14:15 EDT
Update
Database has been restored. Flag rules, plan entitlements, and company overrides have been restored. We are now finishing up with rebuilding cache, plan audiences, and accounting for any data loss that occurred during the outage
Posted Jul 11, 2025 - 13:20 EDT
Update
We are continuing to work on a fix for this issue.
Posted Jul 11, 2025 - 12:29 EDT
Update
We are continuing to work on a fix for this issue.
Posted Jul 11, 2025 - 12:12 EDT
Identified
All targeting rules created prior to this outage are returning their default values. Plan entitlements have been deleted. This is was the unintended result of a migration script. We are restoring the most recent db snapshot to undo this mistake.
Posted Jul 11, 2025 - 12:11 EDT
This incident affected: API, App, and Checkout Flow / Customer Portal.