1

I have a shell script which extracts data from the website. I know, I can store this data into text or csv file and then import it into database using PostgreSQL "copy" command.

My interest is not to use this text or csv file in the process. But, directly copy the data into database.

Is there anyway to do this? Any suggestion are appreciated. Thank you

  • You should be able to just break up the data extracted from the website and run simple INSERT statements, at a minimum. It would be helpful to know what programming language you're using as well. – Kassandry Feb 23 '16 at 23:13
  • I have used shell scripting to scrap the website. Specifically " xidel" command. – Swatesh Pakhare Feb 23 '16 at 23:15
1

You can open a pipe and write your data to the pipe directly. Related answer with demo code:

Or you can use COPY with a program or standard input directly:

COPY table_name [ ( column_name [, ...] ) ]
    FROM { 'filename' | PROGRAM 'command' | STDIN }
    [ [ WITH ] ( option [, ...] ) ]

...

When PROGRAM is specified, the server executes the given command and reads from the standard output of the program, or writes to the standard input of the program. The command must be specified from the viewpoint of the server, and be executable by the PostgreSQL user. When STDIN or STDOUT is specified, data is transmitted via the connection between the client and the server.

  • COPY profile FROM PROGRAM 'sh echo.sh' with DELIMITER ',' ; Here is what i tried. But i am getting an error program sh echo.sh failed. Any suggestion on this. Here, echo.sh is just a script which displays firstname, lastname, age, SportsPreference. – Swatesh Pakhare Feb 25 '16 at 22:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.