I work for a small technology MSP. Having just completed my Bachelor's in Information Science, with a focus in data science, my boss has asked me to build a data warehouse (I think he really means a data mart from what I have read). I've been realizing that my education was very heavy on coding, math/stats, and the ML aspect of the data pipeline, but very light on the data preparation aspect, which is frequently claimed to take 80% of the time, so this is a good opportunity for me to learn how all of those tidy data sets I've been analyzing come into existence.

We have 2 main systems, which respectively run SQL Server (~60 GB), and MySQL (~80 GB), and a few other data sources which will come in as CSV. At this point, we simply want to do some simple asset reporting for our clients, and also some simple pricing analysis for some of our monthly fees (e.g. boss has no idea what it actually costs us to support a client's server per month, so our prices are essentially best guesses).

At this point, my main problem is that I don't know what I don't know. I'm currently reading "The Data Warehouse Toolkit" by Kimball, and also taking an EdX.org course in T-SQL from Microsoft. Another co-worker has a small bit of experience in this area, and he says that we should figure out what tables we will need from our 2 main systems, copy them over to another instance of SQL Server, and then use stored procedures to load the data (in the desired format) into a new database, to be used for our purposes for reporting and analysis.

I have two questions:

  1. Does my co-worker's approach seem reasonable for what we are trying to do?
  2. As far as learning what I need to know, are the two sources above sufficient for what I want to do, or am I missing anything important?
  • Could you also elaborate on what your security restrictions are, what platforms (OS, DB, maybe even cloud platform) you intend to leverage, what's your budget, how many developers are working on this, and what's your time frame? – codedawg82 Jan 20 '18 at 12:29
  • I'm not really sure what you are asking but the Kimball book is definitely a good start – Tom V Jan 20 '18 at 16:37
  • I'm not sure what you mean by 'security restrictions' in this case. Both of the mentioned systems are running on separate Hyper-V instances of Server 2012 R2. One of the systems uses SQL Server 2012, and the other uses MySQL (don't know version off the top of my head). It will mostly be me working on this. Regarding time and budget, in my discussion with my boss, he didn't really mention anything. It was sort of an informal "Hey, I'd like to do this, can you see what you can do?" I'm not really trying to find out if this is feasible, but if the proposed approach seems reasonable? – deano Jan 20 '18 at 16:51
  • Tom V, I have edited my post so that question #1 is more clear. Question #2 is just asking what resources/topics should I look into in order to complete a project like this? Thanks. – deano Jan 20 '18 at 16:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.