Tell me more ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.
up vote 0 down vote favorite

I have an rss reader project, in one table( feeds) I have the urls. In table feed_items, I store the items of each feed.

I have set a cron which opens the feeds one by one and parses the XML. Then it checks if they are already in the db or not. If not, they are inserted.

Every thing looks ok, but my server admin blocked my script because firstly it has a long run time and secondly they argue about the I/O which huge number of inserts will make. Th first thing I did is split the cron into 1 per url, but I still have the I/O problem. What are my solutions?

share|improve this question
How many inserts are we talking about as 'huge number of inserts', on the scale of say, 1 hour? Also can you give us an idea on what 'long run time' is? I don't want to get into a pissing match with your server admin, just want to have an idea what thresholds you have to work with. – Derek Downey Jul 24 '12 at 17:39
Also, probably more important, what engine are these tables using? – Derek Downey Jul 24 '12 at 17:44
The table engines are currently myisam, and i dont know does it affect mainly if i change them to innoDB, the number of records depends on the rss feeds' items, at least on every update i will have ten new updates which are also heavy (5-25KB) contents,when one rss url is parsed in one FOR i check if the url exist, if not exist i insert them. also the run time if no new is 1.0sec, but i have new is between 1.0sec to 90sec – user539656 Jul 25 '12 at 17:52
What IO are they worried about - (1) the script that is pulling and parsing the XML, (2) sending the parsed data to the database, or (3) storing the data on the server? Each has a different answer. – Alain Collins Nov 6 '12 at 0:17

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.