Maybe I have too much time on my hands, and this may be a stupid idea, but...

I have recently had to download data in table-form from websites, and I got to thinking that it might have been easier, if I could define these "https://..." as tables/views that I could simply query with SELECT. Is this simply stupid?

  • Scraping is a field in itself - what happens if you have multiple tables per page (e.g. Wiki)? I used Weichbrodt's tool to good effect to get .csv from Wiki pages. The problem I see is that there is no agreed format for every web page to have its tables - some are images also. Still, I'm giving a +1 for a very interesting question and I'd like to see if there are any imaginative solutions out there. A search provides lots of stuff for Wiki, but for other sources, not so much. I would suggest that there might be a Python solution but not MySQL specific?
    – Vérace
    Feb 6, 2020 at 11:28
  • @Vérace I suspect there is no solution that will work in general, but perhaps something that would work with a largish subset of REST APIs? They would have to return things in some format that could be parsed into a table, and support selection on keyword (similar to WHERE ...). Not straightforward, but doable?
    – j4nd3r53n
    Feb 6, 2020 at 11:39
  • You can try data-miner.io but for an automated scraping, i think you must program it.
    – nbk
    Feb 6, 2020 at 16:10
  • Please provide a few sample pages with tables you want to grab. I agree that there should be two steps: "fetch and convert to CSV", then use the existing MySQL tools for CSV.
    – Rick James
    Feb 10, 2020 at 17:46


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.