Wednesday, April 21, 2010

Network isolation

For the second time this week, production has gone down for an extended period of time. This time it was because of this bug introduced by McAffee. It was proliferated through all of our environments (apparently) by SSCM. I'm not sure why we are putting McAffee fixes right into production without testing them in a staging environment first. Seems to me like there is a wrong process there somewhere - Either on our side, McAffee's side, or both.
Before we realized this was a McAffee bug, it seemed that this 'virus' got propagated to production from lower environments. Having the production environment on an isolated network from the rest of our development and test environments would have protected us to some degree from this kind of propagation. I have (officially) suggested this more than once, but resourcing and other priorities have gotten in the way of making it a reality.
For future reference though, here's just another reason to have a production environment on an isolated network.

No comments: