1

I am just now in the midst of a strange issue. A client moved their SQL database server from Hyper-V to VMWare on Saturday. Yesterday they called me complaining about performance. In the brief time and limited access I had I was able to see a lot of indicators of poor I/O (and many other problems) so I made a few recommendations.

They restarted their SQL server to add on the VMWare tools and take care of a few other tasks. Upon restart their data reverted (from their perspective) to the moment they brought it up on the new VM. They found that the new database was pointed to the old iSCSI drives instead of the new vmdk files and wasn't writing anything at all to disk. They tried restoring their backup but it fails every time before completion.

Did the database just fill up the dirty cache, never writing to disk? How in the world did the database continue to function for five days without presenting anything more than slow I/O to the end users? A complicating factor is they didn't have any alerts set up on the database that might have warned them of these kinds of problems. The event logs on the server itself show that there were numerous write errors.

  • 1
    Sounds like SQL Server did present issues, the customer just wasn't looking for them. SQL Server can encounter a lot of problems that it will log somewhere - if you're not looking at the logs, though... this is like putting electrical tape over the check engine light – Aaron Bertrand Aug 27 '15 at 16:05
  • Perhaps the VM had a snapshot in place that was reverted automatically when the VM was restarted. SQL Server did ensure data was written to disk, what happened to that data at the VM level is anyone's guess. – Max Vernon Aug 27 '15 at 19:16
  • One other thing... Always ensure you have a tested backup strategy in place. Test backups by restoring them to another instance. Until you do that, you should have no trust in the system. – Max Vernon Aug 27 '15 at 19:17
  • Don't have a suggestion, but would like to see how you resolve it. Please update the question. Thanks. – Emacs User Aug 27 '15 at 21:30
  • Did they have an long running transaction that was never committed? – datagod Oct 26 '15 at 16:25
1

The problem was never definitively resolved. What was found was that at some point their database was completely truncated by an unknown command. Every time they tried to restore the database and roll forward, they were re-truncating their DB.

They started rolling forward a little bit at a time until they found the moment when it went bad. At that point they cut me out of the troubleshooting loop so I was not able to participate in the post-mortem. My guess is that someone did something really stupid on the database and they didn't want me to dig too deeply.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.