Today we went live with a product we’ve been working on for A Long Time, in which a team of five has put in A Lot Of Work.
During the walk-through with the customer, loading their initial data onto the production system, we had error after error when trying to insert a particular kind of record. The stored procedures were complaining about a foreign key constraint, but on a table where we’d never seen the error before.
The walk-through had to turn into a demo out of necessity, by switching to the test environment where everything worked. Obviously this was a data problem.
During the postmortem, we confirmed that the initial records we thought we needed, had been inserted. So now it came to looking at the difference in data between test and production. Shout-out to SQL Toolbelt from Red Gate for making our lives easier.
You know what’s coming. It turns out that the process for inserting a particular kind of record, which we only see for brand new records in the system (think first-client scenario), relied on the existence of a lookup value that did not appear in the production database.
In other words, for the last year we’ve been testing a product based on a false assumption.
The good news is, the customer was signed, and the problem was identified reasonably quickly. The other good news is that we had a test environment on which to fall back.
An assumption sandwich tastes a lot like humble pie.
Photo by chuttersnap on Unsplash.