Testing Enterprise Infrastructure changes with brownouts

Posted by on

Decommissioning or migrating services in enterprise environments will frequently reveal previously unknown dependencies which result in changes getting rolled back, wasting a lot of time.

To try and head these issues off you can test these changes with “brownouts”, shutting off the relevant infra temporarily (eg at a weekend) to expose any hidden dependencies.

When brownout testing, the following need to be borne in mind:

  • Dependencies may only be exposed for certain functions. Eg at startup, shutdown or other operations such as login. Thus a full test suit should be run and components restarted.
  • The effects of DNS, such as caching.
  • Regional differences.