Overview

A DNS issue caused downtime for the Vector Charts Enterprise-1 API instance. The Vector Charts team diagnosed the issue, which is now resolved.

Technical Details

The issue was caused by a number of cascading failures, which aligned to cause downtime.

A failure in DNS resolution in the Vector Charts enterprise-1 cluster caused an internal host to be unreachable briefly.

At the same time, automated system updates were being applied, causing an API load balancer to restart. When configuration was reloaded, the load balancer failed to resolve the downstream server, and was not configured to retry the DNS lookup.

As a result of these two issues, the load balancer marked all instances as "unhealthy" and was not able to route API requests to a live host.

Resolution and Fix

To resolve the issue, the Vector Charts team made several fixes:
- Upon discovering the downtime, restarted the server to restore API service.
- Review & reduce the number of automated system updates on the enterprise-1 cluster, and shift critical updates to hours with an active on-call team.
- Implemented secondary DNS lookups for critical DNS records used in our hosting infrastructure
- Implemented new internal DNS monitoring status checks.

Affected customers will receive a credit to their monthly bill.