Load 1TB in less than 1 hour (from SQL CAT team) - list of advices from here (quote):
Run as many load processes as you have available CPUs. If you have 32 CPUs, run 32 parallel loads. If you have 8 CPUs, run 8 parallel loads.
If you have control over the creation of your input files, make them of a size that is evenly divisible by the number of load threads you want to run in parallel. Also make sure all records belong to one partition if you want to use the switch partition strategy.
Use BULK insert instead of BCP if you are running the process on the SQL Server machine.
Use table partitioning to gain another 8-10%, but only if your input files are GUARANTEED to match your partitioning function, meaning that all records in one file must be in the same partition.
Use TABLOCK to avoid row at a time locking.
Use ROWS PER BATCH = 2500, or something near this if you are importing multiple streams into one table.
Top 10 Best Practices for Building a Large Scale Relational Data WarehouseTop 10 Best Practices for Building a Large Scale Relational Data Warehouse (from SQL CAT team) - advices (quote):
Use SIMPLE or BULK LOGGED recovery model during the initial data load.
Create the partitioned fact table with the Clustered index.
Create non-indexed staging tables for each partition, and separate source data files for populating each partition.
Populate the staging tables in parallel (Use multiple BULK INSERT, BCP or SSIS tasks)
Build a clustered index on each staging table, then create appropriate CHECK constraints.
SWITCH all partitions into the partitioned table.
Build non-clustered indexes on the partitioned table.
The Data Loading Performance GuideThe Data Loading Performance Guide (from SQL CAT team)
Loading Bulk Data into a Partitioned Table - SQL Server Best Practices Article Loading Bulk Data into a Partitioned Table - SQL Server Best Practices Article (Technet article)
SQL Server 2000 Incremental Bulk Load Case StudySQL Server 2000 Incremental Bulk Load Case Study (Technet article)
Lessons Learned and Findings from a Large Fast-Track POCLessons Learned and Findings from a Large Fast-Track POC (from SQL CAT team)
Performance Tuning Tips for SQL Server BCPPerformance Tuning Tips for SQL Server BCP (By Brad McGehee)
Performance Impact: Finding the Most Optimal Batch Size Performance Impact: Finding the Most Optimal Batch Size (By Linchi Shea)
- Managing Batches for Bulk ImportManaging Batches for Bulk Import
- Managing Bulk Copy Batch SizesManaging Bulk Copy Batch Sizes
- Bulk Copy Performance ConsiderationsBulk Copy Performance Considerations
- Guidelines for Optimizing Bulk ImportGuidelines for Optimizing Bulk Import
- Optimizing Bulk Import PerformanceOptimizing Bulk Import Performance