I am running a scraper to check for this one thing to change. So I have no idea why my mongo data base keeps growing when all I do is an update on this single data.

I only have one item in the database, with a post_id : 1. In my code all I am doing is updating it. (python)

client = MongoClient(DB_URL)
db = client.chatScrape

collection = db.posts

#insert scraped data
result = collection.update({'post_id' : 1}, scraped, upsert = True)


So after the first execution I get

> db.posts.find().count()
> 1

But after an hour or so I get

> db.posts.find().count()
> 2

After about a week or so, the number grows to 10+. So what is going on here. Shouldn't update only modify one item and not make more?

  • Are all of the documents with a post_id of 1 still? Jan 18 '17 at 0:03
  • Only one of the document in the collection is post_id of 1.
    – Learning
    Jan 18 '17 at 0:40
  • is post_id an auto incrementing field? Jan 18 '17 at 0:53
  • Nope. It is static. The other document do no have a post_id. The other are just documents containing data. It looks like it is extending the post_id... Which is strange because the document size is always the same. I am expecting the update to overwrite the original document. But from what I am seeing new data into post_id of 1 and moving the old stuff to another document.
    – Learning
    Jan 18 '17 at 1:17
  • Can you show us what is in the scraped variable? Feb 27 '17 at 10:10

I'd hazard a guess that your code is running the upsert command fairly quickly. It may depend on your MongoDB version, but on the latest version an update command which does not specify update modifiers will just overwrite the entire document, which involves deleting and re-writing it.

It seems likely that in between another upsert operation, a second upsert is run and when it sees no document it inserts a new one. In this case you might want to look at not updating the entire document.

You could also look into using a unique index and a little re-try logic which might work for you based on how infrequently this appears to be happening.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.