3

I have a CSV (tab-delimited in fact) file with 7,590,051 lines, according to wc -l, which I want to import using COPY FROM. I removed the table's primary key, so it has no constraints.

I ran COPY customer FROM '.../customer.dsv' WITH DELIMITER E'\t' CSV HEADER; and it reported 7,588,671 rows imported and no errors, so it's missing 1,379 rows (already discounting the header line).

Since PostgreSQL didn't report any errors, how do you suggest I troubleshoot which rows are missing (and why they are missing)?

1
  • 1
    You could export the table again (using copy) and compare the two files after you have sorted them.
    – user1822
    Jul 20, 2012 at 22:45

1 Answer 1

6

It could be normal if there are newlines in certain text fields. Newlines are allowed when the value in the field is enclosed by double quotes. And obviously that makes the number of lines in the file greater than the number of records.

Example :

$ cat file.csv
1,"ab
cd",2
3,"efgh",4

$ wc -l file.csv
3 file.csv

=> create table csvtest(a int, b text, c int);
=> \copy csvtest from 'file.csv' with csv

=> select * from csvtest;
 a |   b   | c 
---+-------+---
 1 | ab    | 2
   : cd      
 3 |  efgh | 4
(2 rows)
2
  • That was it, the file contains some newlines inside some fields. They are surrounded by ", do you have any suggestion for dealing with this problem? Jul 23, 2012 at 14:47
  • Nevermind, the COPY FROM went ok, that's why no errors were reported. wc -l was wrong. Jul 23, 2012 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.