Timeline for Storing many pandas DataFrames in SQLite with metadata
Current License: CC BY-SA 4.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 3, 2020 at 18:50 | vote | accept | smcs | ||
| Mar 3, 2020 at 18:50 | comment | added | smcs | According to this, there is no one perfect way to do it and the answer there actually supports your way of adding a whole column. | |
| Mar 3, 2020 at 17:57 | comment | added | smcs | The DataFrames do not have a unique identifier at this time. With pandas, as far as I can see, the only way to add such an identifier is to add a whole column with the same value, like you did. I thought maybe since I'm moving to SQLite now that there might be a better solution, e.g. that it's possible to add "date" or "name" as a single field, as metadata basically. | |
| Mar 3, 2020 at 17:45 | comment | added | Peter Vandivier | The "append experiment" line is based on the initial stated premise of multiple csv files and was included in the event that your dataframes do not already uniquely isolate experiments within themselves. Appending values conditionally to a dataframe is likely another question (for SO, not dba.se), but it's also probably an answered question | |
| Mar 3, 2020 at 17:42 | comment | added | smcs | Thanks! So there is no good way to store "single field" values? Filling a whole column with the same entry a million times seems sort of unclean. Apart from that this is much easier to work with than what I had before. | |
| Mar 3, 2020 at 17:04 | history | answered | Peter Vandivier | CC BY-SA 4.0 |