My company is looking to access large quantities of data from another company we have good relations with. Currently we are using a web interface to interrogate their information, however it is quite limited in its functionality (set reports), and the available data is vast. At the moment we have limited technology resources to develop ways of mining this data, and though I am pretty clued up on basic database construction, this is something totally new to me.
If the data was small enough to be manageable, I could request a CSV file to be sent to an FTP server, pull it into the software we have (Access, Tableau) and query it from there. However, there's going to be loads of it.
- So what are my options here?
From the little I know, the simplest thing would be to query source data. Are there any tools that would do that outside them using another API?
Similarly in my head I could foresee a way to send new data to a SQL server somewhere, and then query/link to it.
- Is this possible?
- Is this how SQL Server works?
- How much effort would this take on their part?
I am not looking for a full schematic on how to do this (though I wouldn't turn one down), but just some clear terms and tools I can research on my own would be enough.
Essentially, we are meeting with the company to discuss how best we would do this. They currently have a limited UI for customers, but are interested in expanding. We have very little database infrastructure but are also eager to expand and looking for tools/products that will help us. We don't have SQL Server at the moment - we don't hold much data here, hence why we rely on querying third party data. So without them building a new interface for customers, how would one go about sharing this data considering the volume is considerable?
((Update re data size: Apparently we're looking in the region of 1-2 million rows a day with, probably, about 50 fields (Mostly short number and text < 50)....I don't know exactly what that translates into in terms of Mb/day, but suspect it's going to be too much to handle over email on anything over a daily basis))