Skip to main content
Tweeted twitter.com/StackDBAs/status/1300538878298816512
added 1958 characters in body
Source Link
GMassDBA
  • 345
  • 2
  • 13

Update

I missed out on doing this for a while, but I'd better keep this thread up to date. @DavidSpillet has a few points in his reply. Here is a bit more information that might help people helping me out - you're all awesome.

  1. We have about 13 (SQL natively compressed) backup files, totaling around 60GB. The approximate size after restoring them is around 270GB.
  2. Our databases are separated into two systems. System A has 2 databases, System B has 11. Restores are done concurrently per system, so we will have at most 2 restores per developer running in parallel.
  3. Developers in Spain restores in the office using a wired LAN network, sitting on the same VNET (and building, and floor) the server is in.
  4. The server is a 16-core, 32 logical CPUs, 96GB RAM monster that was once used to host a shared SQL instance for our developers. Restores in there take consistently between 15-20 minutes.
  5. The server high-capacity storage (where the backups are held) is based on spinning locally attached disks - but then again, restoring locally is not a bottleneck
  6. Developers laptops use high-capacity NVMe SSD drives (at least, the new ones. The older model used spinning drives. I'm not minding those as they're being rolled out)

I also have a few more concerns due to the current world situation, namely

  • Devs are currently working anywhere, sometimes at the office, most of the time at home. Unfortunately, our client VPN endpoint is in another country, so restoring from the office is a major pain, and I'm trying to account for that, especially considering how the country is looking like (2020-07-29).
  • Some of the new laptop models arrived with lower capacity storage (256 GB) and there's nothing I can do to fix that.
  • Cannot go on purchasing commercially available software that would do the trick (for example this one) because of budget constraints.

Update

I missed out on doing this for a while, but I'd better keep this thread up to date. @DavidSpillet has a few points in his reply. Here is a bit more information that might help people helping me out - you're all awesome.

  1. We have about 13 (SQL natively compressed) backup files, totaling around 60GB. The approximate size after restoring them is around 270GB.
  2. Our databases are separated into two systems. System A has 2 databases, System B has 11. Restores are done concurrently per system, so we will have at most 2 restores per developer running in parallel.
  3. Developers in Spain restores in the office using a wired LAN network, sitting on the same VNET (and building, and floor) the server is in.
  4. The server is a 16-core, 32 logical CPUs, 96GB RAM monster that was once used to host a shared SQL instance for our developers. Restores in there take consistently between 15-20 minutes.
  5. The server high-capacity storage (where the backups are held) is based on spinning locally attached disks - but then again, restoring locally is not a bottleneck
  6. Developers laptops use high-capacity NVMe SSD drives (at least, the new ones. The older model used spinning drives. I'm not minding those as they're being rolled out)

I also have a few more concerns due to the current world situation, namely

  • Devs are currently working anywhere, sometimes at the office, most of the time at home. Unfortunately, our client VPN endpoint is in another country, so restoring from the office is a major pain, and I'm trying to account for that, especially considering how the country is looking like (2020-07-29).
  • Some of the new laptop models arrived with lower capacity storage (256 GB) and there's nothing I can do to fix that.
  • Cannot go on purchasing commercially available software that would do the trick (for example this one) because of budget constraints.
styles
Source Link
GMassDBA
  • 345
  • 2
  • 13
  • Restoring near the server will be fast, but will force developers to traverse the Atlantic when reading from what should be the local database.
  • Restoring the snapshot from EastUs to a storage account in West Europe seems to be copying the snapshot over (which is logical), but this means there's no restore performance benefit.
  • My devs are really keen on restoring daily, weekly at most to ensure they're on par with production database releases and can troubleshoot data issues that get escalated to the engineering team.
  • Manually copying files from East Us to West Europe is not a good option, since it requires daily attention from someone.

Update 2020-07-07

Update 2020-07-07

Update 2020-07-08

Update 2020-07-08

  • Restoring near the server will be fast, but will force developers to traverse the Atlantic when reading from what should be the local database.
  • Restoring the snapshot from EastUs to a storage account in West Europe seems to be copying the snapshot over (which is logical), but this means there's no restore performance benefit.
  • My devs are really keen on restoring daily, weekly at most to ensure they're on par with production database releases and can troubleshoot data issues that get escalated to the engineering team.
  • Manually copying files from East Us to West Europe is not a good option.

Update 2020-07-07

Update 2020-07-08

  • Restoring near the server will be fast, but will force developers to traverse the Atlantic when reading from what should be the local database.
  • Restoring the snapshot from EastUs to a storage account in West Europe seems to be copying the snapshot over (which is logical), but this means there's no restore performance benefit.
  • My devs are really keen on restoring daily, weekly at most to ensure they're on par with production database releases and can troubleshoot data issues that get escalated to the engineering team.
  • Manually copying files from East Us to West Europe is not a good option, since it requires daily attention from someone.

Update 2020-07-07

Update 2020-07-08

changed wording. Previous change was to add today's update
Source Link
GMassDBA
  • 345
  • 2
  • 13

Update 2020-07-08

Also tested out backing up to a different storage location. Backups are fast, but the snapshot is still taken locally to the .mdf location - this means restores are painfully slow, and reducing that lag (and the data transfer) is the whole point of this exercise. I cannot really modify the .bak pointers after the backup has been generated, so even copying that data manually across the ocean won't work.

This means I'm officially out of ideas.

Update 2020-07-08

Also tested out backing up to a different storage location. Backups are fast, but the snapshot is still taken locally to the .mdf location - this means restores are painfully slow, and reducing that lag (and the data transfer) is the whole point of this exercise. I cannot really modify the .bak pointers after the backup has been generated, so even copying that data manually across the ocean won't work.

This means I'm officially out of ideas.

Added CDN tests results.
Source Link
GMassDBA
  • 345
  • 2
  • 13
Loading
Source Link
GMassDBA
  • 345
  • 2
  • 13
Loading