I was asked at work today why something would work great during testing, and then when you place it into production it breaks. I immediately assumed a few things and started explaining the woes of an inadequate test environment.
Turns out they were testing in production, but not live. Same environment right? Apparently not.
Testing is a giant part of getting things right, right up there with gathering requirements, documentation, design, and oh yea.. building stuff. It’s also important in just about everything we do as a society. Unfortunatly it all too often goes by the wayside.
Testing is important but how do you do it? Unless you’re trained/experienced in this subject, it’s often not as obvious as it sounds. Without understanding the concept, it’s difficult, time consuming, and often a waste of time. Done right, it’s difficult, time consuming and absolutely necessary to create a predictable outcome.
Test environments are often a big part of this, simply because no complex system is an island. You can blackbox things pretty well, but in the real world (and in software too) there’s usually enough inputs to force you consider your environment heavily. Without a controlled, predictable environment, you can’t get 100% test coverage.
Also you must make your test environment as close to reality as you can possibly make it. This is hard in some cases, but just as necessary for success. This is where my co-worker failed to realize something. Eventhough he was testing on the production server (ignoring the don’t-ever-do-this aspect of that in general) he still didn’t get the test environment right, since it broke when he finally made it live. Getting a test environment right is a tricky business. The best way is to just release, and suffer the consequences. This although being a great test environment is infeasible for many obvious reasons. In lieu of this, you’re constructing a simulation of this.
Testing outside of a 100% simulation of the real environment in which the product will be used gives you incomprehensive testing.