How to insert (file) data into a PostgreSQL bytea column?

Question

This question is not about bytea v. oid v. blobs v. large objects, etc.

I have a table containing a primary key integer field and a bytea field. I'd like to enter data into the bytea field. This can, presumably, be done by one of the PL/ languages, and I may look into doing this with PL/Python in the future.

As I am still testing and experimenting, I would simply like to insert data from a file (on the server) using "standard" SQL statements. I am aware that only administrators with write permission on the server would be able to insert data in the way I would like to. I'm not concerned about that at this stage as users would not be inserting bytea data at present. I have searched the various StackExchange sites, the PostgreSQL Archives and the Internet generally, but have not been able to find an answer.

Edit: This discussion from 2008 implies that what I want to do is not possible. How are bytea fields used then?

Edit: This similar question from 2005 remains unanswered.

Solved: The details provided here on the psycopg website provided the basis for a solution I've written in Python. It may also be possible to insert binary data into a bytea column using PL/Python. I don't know if this is possible using "pure" SQL.

Answer 1

as superuser:

create or replace function bytea_import(p_path text, p_result out bytea) 
                   language plpgsql as $$
declare
  l_oid oid;
begin
  select lo_import(p_path) into l_oid;
  select lo_get(l_oid) INTO p_result;
  perform lo_unlink(l_oid);
end;$$;

lo_get was introduced in 9.4 so for older versions you would need:

create or replace function bytea_import(p_path text, p_result out bytea) 
                   language plpgsql as $$
declare
  l_oid oid;
  r record;
begin
  p_result := '';
  select lo_import(p_path) into l_oid;
  for r in ( select data 
             from pg_largeobject 
             where loid = l_oid 
             order by pageno ) loop
    p_result = p_result || r.data;
  end loop;
  perform lo_unlink(l_oid);
end;$$;

then:

insert into my_table(bytea_data) select bytea_import('/my/file.name');

Answer 2

Use pg_read_file('location_of file')::bytea.

For example,

create table test(id int, image bytea);
insert into test values (1, pg_read_file('/home/xyz')::bytea);

Manual

Answer 3

This solution isn't exactly efficient in terms of runtime, but it's trivially easy compared to making your own headers for COPY BINARY. Further, it doesn't require any libraries or scripting languages outside of bash.

First, convert the file into a hexdump, doubling the size of the file. xxd -p gets us pretty close, but it throws in some annoying newlines that we have to take care of:

xxd -p /path/file.bin | tr -d '\n' > /path/file.hex

Next, import the data in PostgreSQL as a very large text field. This type holds up to one GB per field value, so we should be okay for most purposes:

CREATE TABLE hexdump (hex text); COPY hexdump FROM '/path/file.hex';

Now that our data is a gratuitously large hex string, we use PostgresQL's decode to get it into a bytea type:

CREATE TABLE bindump AS SELECT decode(hex, 'hex') FROM hexdump;

Answer 4

The answer with xxd is nice and, for small files, very fast. Below is an example script that I'm using.

xxd  -p /home/user/myimage.png | tr -d '\n' > /tmp/image.hex
echo "
    -- CREATE TABLE hexdump (hex text);
    DELETE FROM hexdump;
    COPY hexdump FROM '/tmp/image.hex';

    -- CREATE TABLE bindump (binarydump bytea);
    DELETE FROM bindump;

    INSERT INTO bindump (binarydump)  
    (SELECT decode(hex, 'hex') FROM hexdump limit 1);

    UPDATE users 
    SET image= 
    (
        SELECT decode(hex, 'hex') 
        FROM hexdump LIMIT 1
    )  
    WHERE id=15489 ;
    " | psql mydatabase

Answer 5

Here is how to do it without superuser privileges (such as on Heroku).

\lo_import '/cygdrive/c/Users/Chloe/Downloads/Contract.pdf'
update contracts set contract = lo_get(:LASTOID) where id = 77;

You can use \lo_list to see the large objects, and \lo_unlink to delete them. The contract field in my example is bytea.

     Column      |            Type             |
contract         | bytea                       |

Answer 6

Use the Postgres COPY BINARY function. This is broadly equivalent to Oracle's external tables.