1

I am looking into making use of Azure hosting services in order to develop a logging service for our company.

Azure has a number of storage options available (I have listed links below). However I am not sure which one would be best suited for my needs. I am coming from a MS SQL Server background so I am probably not as open minded as I should be, hence my question here is to try and explore other more suitable options. If any one knows of any place where I can get design help specifically for azure please let me know, as I have not been able to find any.

Requirements:

Background: We are looking to log messages from a variety of types of devices which the company owns. This needs to be on a mass scale. Several million entries per day. As much as I have tried to keep the design a fixed structure, due to the variety of device types out their they messages differ completely and have different elements. I would somehow like to still be able to query message specific attributes. (How one would do this, I do not know).

  • All messages come into this logging service as A JSON object. The JSON object obviously varies based on the type of device which sent the message and the type of message the device is sending.
  • There will be some common attributes such as deviceID, LogTime,customerId etc.
  • Due to the mention of above, it seems that a document / object store technology would suit the application better. However as mentioned above I have not have much exposure to these types of technologies. Will it be able to efficiently handle the load, and queries etc.
  • We will need to be able to store months of log values without slowing down daily queries to a halt.
  • Data analytic's and big queries should be able to be done on the data after some time of running. I'm not sure if one would make use of a different mechanism for this? Possibly a sort of storage layer and then a big data layer. However my storage mechanism needs to keep this future features in mind.

Azure offers the following storage and database services:

  1. Sql Database
  2. DocumentDb
  3. Storage: Blobs, Tables,Queues,Files and Disks
  4. StorSimple Hybrid cloud Stroage
  5. SQL Data Warehouse

I am sorry, I know this question is quite general and possibly open ended. I am not looking for anyone to just give the the complete solution in my hand. I am looking for some guidance in the right direction and experience from people who have already gone down this road before.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.