How can I import data from a NetCDF file into SQL Server?

Question

I'm being asked to investigate the feasibility of importing data from a NetCDF file into SQL server.

The file could be upwards of 300-400Mb and provided 4 times a day. However I'm only interested in a small percentage of the data. (Maybe as little as 3-5%). We estimate importing ~4000 rows on each run and accumulating the data.

From what I have read a NetCDF file is an XML file. Which may also include binary data.

What would be the best options for importing the data? Should I import the whole file into working table and then select what I want? Or would it be possible to parse/query the file using SQL XML features and only import the data i'm interested in.

I'd appreciate any recommendations from anyone who has first hand experience with NetCDF and what tools they used.

Answer 1

My solution was to create a SSIS Package, which has a C# script task which used Microsoft Scientific Data Set. (SDS)

The C# script then loops through the data and passes it back to SSIS, when then inserts it into SQL server.

Once I had the basics running I asked on other SE sites for help and my example code is there.

My data files contained Meterological data for all of New Zealand, but i was only interested in a small specific area. As the data was geospatial with reliable coordinates, my script could be more selective about what it returned, saving time and a lot of unnecesary I/O.

I am extremely grateful for the help people gave me. And I would like to be able to help others in the same way. If you have question DM me or add a comment here and I will try and help as best i can.