I can get some financial data from our vendor via their API. In this case API is too strong a word. Basically we can get only an array of orders for a given time range. What we paid them for the product, what they charges us for shipping, etc.
To report on and analyze this data I want to store it in a table. I have some concerns:
- What if i load 1 Jan - 31 Jan today, and then next week they change some data in that range? With thousands of products sold daily combing over that to look for necessary updates could be intense.
- How should I manage adding new data? I'm thinking to get new data mid-month for the previous month. This vendor's practices are not the best, I'm concerned they'll add some things one Monday that they forgot to enter the previous Friday. I think if I get today's data every day the likelihood of changed data in a past period rises.
Appreciate any advice from folks who've faced a similar issue. Maybe people do this all the time and I'm worried over nothing. Just would like to get some advice before I proceed.