Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I have a CSV (tab-delimited in fact) file with 7,590,051 lines, according to wc -l, which I want to import using COPY FROM. I removed the table's primary key, so it has no constraints.

I ran COPY customer FROM '.../customer.dsv' WITH DELIMITER E'\t' CSV HEADER; and it reported 7,588,671 rows imported and no errors, so it's missing 1,379 rows (already discounting the header line).

Since PostgreSQL didn't report any errors, how do you suggest I troubleshoot which rows are missing (and why they are missing)?

share|improve this question
1  
You could export the table again (using copy) and compare the two files after you have sorted them. –  a_horse_with_no_name Jul 20 '12 at 22:45
add comment

1 Answer

up vote 3 down vote accepted

It could be normal if there are newlines in certain text fields. Newlines are allowed when the value in the field is enclosed by double quotes. And obviously that makes the number of lines in the file greater than the number of records.

Example :

$ cat file.csv
1,"ab
cd",2
3,"efgh",4

$ wc -l file.csv
3 file.csv

=> create table csvtest(a int, b text, c int);
=> \copy csvtest from 'file.csv' with csv

=> select * from csvtest;
 a |   b   | c 
---+-------+---
 1 | ab    | 2
   : cd      
 3 |  efgh | 4
(2 rows)
share|improve this answer
    
That was it, the file contains some newlines inside some fields. They are surrounded by ", do you have any suggestion for dealing with this problem? –  Daniel Serodio Jul 23 '12 at 14:47
    
Nevermind, the COPY FROM went ok, that's why no errors were reported. wc -l was wrong. –  Daniel Serodio Jul 23 '12 at 19:02
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.