I am looking into making use of Azure hosting services in order to develop a logging service for our company.
Azure has a number of storage options available (I have listed links below). However I am not sure which one would be best suited for my needs. I am coming from a MS SQL Server background so I am probably not as open minded as I should be, hence my question here is to try and explore other more suitable options. If any one knows of any place where I can get design help specifically for azure please let me know, as I have not been able to find any.
Background: We are looking to log messages from a variety of types of devices which the company owns. This needs to be on a mass scale. Several million entries per day. As much as I have tried to keep the design a fixed structure, due to the variety of device types out their they messages differ completely and have different elements. I would somehow like to still be able to query message specific attributes. (How one would do this, I do not know).
- All messages come into this logging service as A JSON object. The JSON object obviously varies based on the type of device which sent the message and the type of message the device is sending.
- There will be some common attributes such as deviceID, LogTime,customerId etc.
- Due to the mention of above, it seems that a document / object store technology would suit the application better. However as mentioned above I have not have much exposure to these types of technologies. Will it be able to efficiently handle the load, and queries etc.
- We will need to be able to store months of log values without slowing down daily queries to a halt.
- Data analytic's and big queries should be able to be done on the data after some time of running. I'm not sure if one would make use of a different mechanism for this? Possibly a sort of storage layer and then a big data layer. However my storage mechanism needs to keep this future features in mind.
Azure offers the following storage and database services:
- Sql Database
- Storage: Blobs, Tables,Queues,Files and Disks
- StorSimple Hybrid cloud Stroage
- SQL Data Warehouse
I am sorry, I know this question is quite general and possibly open ended. I am not looking for anyone to just give the the complete solution in my hand. I am looking for some guidance in the right direction and experience from people who have already gone down this road before.