As a web developer, I've typically only worked with nice, normalized tables in SQL-based RDBMS, but I'm doing some data consolidation for a researcher who will be analyzing correlations between all items of several very long questionnaires and wants a giant CSV with data for all takers and all items. So, say, 3000 rows and 5000 columns (?).
I set up tables for each questionnaire (each has its own CSV), but we're talking tables with 1000+ columns representing item responses for each taker / pkid. Obviously I can't join up four, five different questionnaires like that based on pkid -- I'd have 5K or more columns and I'm already at the outer limits of MySQL just on the individual tables.
So do I: 1) look into NoSQL at last, 2) just programmatically write a one-off script to consolidate this data 3) convince scientist to buy Oracle or MS SQL 4) other option I as a non-data-scientist don't know about?
Thanks for the advice!