Preferred database solution for storing sensor data

Question

As a hobby project I am developing a system which stores data from different kind of sensors around the house. Currently it is implemented using ASP.Net MVC 6 and MongoDB for the database. (I choose MongoDB because I was curious about how it works) Every sensor peroidically posts data to the server on it's own.

For the sake of not having to store the same information possibly hundreds of times I am using references for redundant data. This data includes the type of a sensor, the data fields used by the sensor(and information about fields like it's name, etc).

Here is an example in JSON format:

// Sensors collection
{
    _id : 1,
    TypeId : 1
}

// SensorTypes
{
    _id : 1,
    GenericName : "Test sensor",
    FieldIds : [1, 2]
}

// Fields
[{
    _id : 1,
    Name : "field1",
    DecimalPlaces : 2
},
{
    _id : 2,
    Name : "field2",
    DecimalPlaces : 3
}]

Questions:

1. Is this good practice? Am I using correct and preferred way to avoid redundancy in my database?
2. "Schema" of my documents can be displayed like a graph. Does a graph database like Neo4j would be more suitable for my problem?

Thank you for your help!

Answer 1

In general you do not want to normalize the data you are stuffing into whichever NoSQL solution you are using, in other words, you want to leave the redundant information there. Otherwise you should be using a relational database system like MySQL,Postgres,SQLite, or MSSQL (Express edition is free, but has database size limits). If you use any of those systems, reducing redundant information would be beneficial.

Honestly, if you are looking to grab a bunch of sensor data, then display it as a graph at a later point, you might want to look into setting up a box with Elasticsearch + Kibana on it. You can get something up and running in a matter of minutes with VERY minimal amounts of configuration: https://www.elastic.co/products/kibana

When Neo4j calls itself a "graphing database" it is not graphs in the visual sense of the word, it is graphs in the relationship sense: https://en.wikipedia.org/wiki/Graph_database

Hope some of this helps.

Answer 2

i assume its bit outdated but i would like to share my experience with MongoDB and Sensor data. Development went really easy and fast but some drawbacks during project created too much problems that we had to change to MSSQL.

First of the problems was that Data got lost - more corrupted. We could see the data, but amount of stored documents was totally different than amount of Iterable documents. Amount of lost data was about 1/3. This is something that happens in MOngoDB and as i heard in a lot of NoSQL DBSs. However, if you are in need of all information that was aceppted by the server, than this could be a major drawback

Second was Agreggation lookup. Its basically a left outer join and it took me ages untill i was able to make any join. Couldnt place all the data in one document as the packets sizes would have been way over 16MB (when all different sensors values are writen in one document)

Third was lack of scheme: some values of field devices (sensors) got corupted during concertion from degrees Celsius to Kelvins due to Controller bug. MongoDB had no validator and i got bunch of strange values in DBS. That had arround 5 days cost, as we blieved that mistake was in DBS,

Placing validators has a large cost, firstly it is almost impossible to create it properly if you have more than 50 Key value pairs. Not to talk about costs of performance

Why we went for MSSQL instead of Mongo is basically described above. For reporting purposes Grafana has MSSQL driver and that worked just fine

Coming back to your question: if you stored redundancies in the way you did it allready, there is no quarantee, that this sensor type exists (which is something that you could easily change with relational DBS by using PK and FK)

If you are using it in such way, you will end up by using Lookup - that could take a lot of processing time, and changes of one document referenced, could cause you loosing data as mongoDB doesnt check it automatically as relational DBSs do.

BR HK