0

I am performing some calculations on datasets and adding some new information to each collection row. I have over 8 billion records in the database.

I do not need the cache as I am only going to traverse all rows and update them. MongoDB is caching data in memory. There is a way to make it memory-optimized. Is it possible to disable cache or clear it as soon as I have done processing? I have a 64 GB computer, it's taking more than 25 GB of memory.

Edit:

The real problem now is MongoDB stops processing data after a certain amount of data is cached. It terminates the process. It does not overwrite any of the old caches. I need to restart the MongoDB service so that it runs again.

MongoDB terminates when it runs out of memory

Basically, I am adding all organizations for each person as a column. So I do not have to make any kind of join in the production server.

This error occurs after inserting 345000 into the remote MongoDB server. Locally there are two MongoDB datasets. 183.0 GB persons 16.7 GB organizations

Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception.Unhandled exception. Unhandled exception.Unhandled exception. Unhandled exception.   Unhandled exception. Unhandled exception. Unhandled exception. Unhandled exception. MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server.
 ---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (10061): No connection could be made because the target machine actively refused it. [::1]:27017
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.Sockets.Socket.<>c.<ConnectAsync>b__274_0(IAsyncResult iar)
--- End of stack trace from previous location where exception was thrown ---
   at MongoDB.Driver.Core.Connections.TcpStreamFactory.ConnectAsync(Socket socket, EndPoint endPoint, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStreamAsync(EndPoint endPoint, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.GetChannelAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableReadContext.InitializeAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableReadContext.CreateAsync(IReadBinding binding, Boolean retryRequested, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.FindOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperationAsync[TResult](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
   at MongoDB.Driver.IAsyncCursorSourceExtensions.ToListAsync[TDocument](IAsyncCursorSource`1 source, CancellationToken cancellationToken)
   at ConsoleApp20.Program.<>c.<<UpdateOrganizationReferenceAsync>b__6_0>d.MoveNext() in D:\Programming\DotNetCore\ConsoleApp20\ConsoleApp20\Program.cs:line 75
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__139_1(Object state)
   at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server.
 ---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (10061): No connection could be made because the target machine actively refused it. [::1]:27017
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.Sockets.Socket.<>c.<ConnectAsync>b__274_0(IAsyncResult iar)
--- End of stack trace from previous location where exception was thrown ---
   at MongoDB.Driver.Core.Connections.TcpStreamFactory.ConnectAsync(Socket socket, EndPoint endPoint, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStreamAsync(EndPoint endPoint, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.GetChannelAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableReadContext.InitializeAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableReadContext.CreateAsync(IReadBinding binding, Boolean retryRequested, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.FindOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperationAsync[TResult](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
   at MongoDB.Driver.IAsyncCursorSourceExtensions.ToListAsync[TDocument](IAsyncCursorSource`1 source, CancellationToken cancellationToken)
   at ConsoleApp20.Program.<>c.<<UpdateOrganizationReferenceAsync>b__6_0>d.MoveNext() in D:\Programming\DotNetCore\ConsoleApp20\ConsoleApp20\Program.cs:line 75
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__139_1(Object state)
   at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
4
  • 1
  • How are you measuring memory usage and what specific versions of MongoDB and O/S are you using? If you stop the MongoDB server, memory associated with the mongod process should be freed. You can set a limit on the WiredTiger cache size, which will be 50% of (RAM -1GB) in MongoDB 3.4+ (so probably 31.5GB if you have 64GB of RAM). If MongoDB is terminating unexpectedly, please edit your question to include more details (eg specific error message).
    – Stennie
    Mar 5, 2021 at 1:16
  • It gets freed. After stopping it and running it again. I was seeing that it was immediately taking the amount of memory again. I was thinking that it was not releasing. After restarting it, it works. Also is there any way to detect whether mongo has stopped because of a memory issue? I was thinking of programmatically run it again after it getting stopped at some point. I am keeping track of how many rows are already processed.
    – Khalil
    Mar 5, 2021 at 14:22
  • I will add exception details when it's triggered again. From what I remember it fails to get a connection after processing lots of rows.
    – Khalil
    Mar 5, 2021 at 14:33

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.