Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

Current situtation

I am working with a dozen of CSV files containing financial simulations in form of 100'000x100 matrices. Each file is around 100Mb large.

Until now, only analyse these files with R (or Matlab).

Thoughts

Since I only perform elementary operations on these matrices, I am thinking about importing them in a database and perform the same calculations with some SQL queries.

Question

Is it a bad practice to use a database server (SQL Server in my case) to perform such mathematical computations (conditional sum, product,...) ?

The server is slower than a program in R, but I find it less prone to error, specially when I have a lot of CSV files.

share|improve this question

closed as primarily opinion-based by Max Vernon, Paul White, RolandoMySQLDBA, Jon Seigel, Mark Storey-Smith Feb 7 at 10:23

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise.If this question can be reworded to fit the rules in the help center, please edit the question.

4  
There is certainly nothing wrong with using SQL Server to do math. The question is does it do it fast enough? Is there a compelling reason to use SQL Server or would some other product be better suited. Only you can answer these questions. –  Max Vernon Feb 6 at 23:24
2  
In general, I would keep them in Matlab because it's more flexible to do any mathematical computation needed now and into the future. You'll end up rolling your own algorithms in SQL for anything slightly more than trivial, which won't be fun and will be prone to error. If you're having issues with consistency of the CSVs, then consider writing some code to normalize them before doing the import. –  Jon Seigel Feb 7 at 4:01
    
@JonSeigel you know, you could move that comment as an answer :-). Looks pretty sound to me. –  Marian Feb 7 at 9:40
    
By the way, with consistency of the CSV, i mean that the number of columns change every year and hence a mapping is needed to correct this. For one or two years it isn't a problem, but when 10 different mapping are needed for 10 different years, it can become quite tricky. –  motsdujour Feb 7 at 9:54