I have been developing a solution that uses MongoDB Atlas (currently M0) for storage. As I would need to provide some statistics throughout different time periods, I need to run periodic aggregations on hourly, daily, weekly, and monthly basis.
As the project is still small, I would prefer not to deploy Spark engine and not upgrade to higher Tier with map reduce functionalities. Considering this, I would be glad for any advice about how to periodically aggregate the raw documents.
MongoDB Stitch triggers enable you to execute application and database logic automatically, either in response to events or based on a pre-defined schedule.
As per MongoDB official documentation, Stitch supports three types of triggers:
Database triggers, which can automatically respond when documents are added, updated, or removed in a linked MongoDB collection.
Authentication triggers, which execute additional server-side logic when a user is created, authenticated or deleted.
Scheduled triggers, which execute functions at regular intervals according to a pre-defined schedule.
The Scheduled triggers may help you in achieving your expected behavior.
I have been developing a solution that uses MongoDB Atlas (currently M0) for storage. As I would need to provide some statistics throughout different time periods, I need to run periodic aggregations on hourly, daily, weekly, and monthly basis.
As per MongoDB atlas blog here MongoDB Atlas, the fully managed database-as-a-service solution of MongoDB, gives index recommendations based on query performances. Atlas’ Performance Advisor monitors the slow-running queries, which is a metric that can be defined by the developers — e.g., a query whose execution takes 150 ms or more — and makes suggestions on adding indexes to the collection, even providing the necessary code snippet to create the index.
Building an index on an existing, populated collection can be challenging. MongoDB offers a run-in-background option for indexing populated collections that does not affect the performance or the availability of the database.
Unused indexes create additional performance overhead and detecting when an index is no longer required is critical. Gathering statistics for each index can be hard if done manually, but MongoDB allows index statistics aggregation to monitor the indexes. These statistics are also available through MongoDB Compass, the intuitive UI solution for MongoDB.
$indexStats (aggregation) New in version 3.2.
Returns statistics regarding the use of each index for the collection. If running with access control, the user must have privileges that include indexStats action.
The $indexStats stage takes an empty document and has the following syntax:
{ $indexStats: { } }
For example to view statistics on the index use on the orders collection, run the following aggregation operation:
db.orders.aggregate( [ { $indexStats: { } } ] )
The operation returns a document that contains usage statistics for each index:
{
"name" : "item_1_quantity_1",
"key" : {
"item" : 1,
"quantity" : 1
},
"host" : "examplehost.local:27017",
"accesses" : {
"ops" : NumberLong(1),
"since" : ISODate("2015-10-02T14:31:53.685Z")
}
}
{
"name" : "_id_",
"key" : {
"_id" : 1
},
"host" : "examplehost.local:27017",
"accesses" : {
"ops" : NumberLong(0),
"since" : ISODate("2015-10-02T14:31:32.479Z")
}
}
{
"name" : "type_1_item_1",
"key" : {
"type" : 1,
"item" : 1
},
"host" : "examplehost.local:27017",
"accesses" : {
"ops" : NumberLong(1),
"since" : ISODate("2015-10-02T14:31:58.321Z")
}
}
The considerations when optimizing a query include the used indexes, query execution time, number of documents read, and number of documents returned. MongoDB optimizes queries automatically. By periodically running different query plans and keeping the empirical results as a cached query plan, MongoDB determines which indexes to use to optimize performance.
