1

I've been looking at other posts. It seems many people have had this problem, but I'm somewhat new to Postgres and tried other solutions with no luck.

I have ny_stations table:

CREATE TABLE ny_stations (
id integer primary key,
name text,
latitude numeric,
longitude numeric,
nyct2010_gid integer,
boroname text,
ntacode text,
ntaname text
);

I am trying to insert values from another table, ny_raw_trips:

INSERT into ny_stations (id, name, latitude, longitude)
SELECT DISTINCT start_station_id, start_station_name, start_station_latitude, start_station_longitude
FROM ny_raw_trips
WHERE start_station_id NOT IN (SELECT id FROM ny_stations);

Getting the error:

ERROR: duplicate key value violates unique constraint "ny_stations_pkey"
DETAIL: Key (id)=(151) already exists.

What am I doing wrong? Also,

SELECT pg_get_serial_sequence('ny_stations', 'id');

...returns nothing. Please let me know if any other info is required.

  • Comment out the INSERT line; run the rest of the query. How many lines show up for start_station_id = 151? That's the problem. DISTINCT considers all columns in the row, not just the ones you want it to. At a guess, I'd say the lat and long differ by a small fraction – Michael Green May 2 '16 at 3:11
1

The id field in your ny_stations table does not seem to be defined as a serial, so it is expected that pg_get_serial_sequence will return nothing.

The duplicate you get relates to one of the records in your SELECT DISTINCT ... FROM ny_raw_trips ... is returning two rows with the same id:

SELECT start_station_id, COUNT(*) FROM (
  SELECT DISTINCT start_station_id, start_station_name, start_station_latitude, start_station_longitude
  FROM ny_raw_trips
  WHERE start_station_id NOT IN (SELECT id FROM ny_stations)
  ) a
GROUP BY start_station_id
HAVING COUNT(*) > 1;

You could list the rows that are introducing the duplication like this:

WITH src AS (
  SELECT DISTINCT start_station_id, start_station_name, start_station_latitude, start_station_longitude
  FROM ny_raw_trips
  WHERE start_station_id NOT IN (SELECT id FROM ny_stations)
  )
SELECT *
FROM src
WHERE start_station_id IN (SELECT start_station_id FROM src GROUP BY start_station_id HAVING COUNT(*) > 1)
ORDER BY start_station_id;


Edit

Once you find the offending duplicates, should you consider that the first occurrence of each case is good enough (e.g. trivial differences in description or coordinate fields), you can use DISTINCT ON:

INSERT into ny_stations (id, name, latitude, longitude)
SELECT DISTINCT ON (start_station_id) start_station_id, start_station_name, start_station_latitude, start_station_longitude
FROM ny_raw_trips
WHERE start_station_id NOT IN (SELECT id FROM ny_stations)
ORDER BY start_station_id;
  • Excellent. Thanks.. I'm figuring out now how to delete those duplicates.. It should not have happened, but seems the import messed up some numbers in the lat and long field: ' 151;"Cleveland Pl & Spring St";40.7218158;-73.99720307 151;"Cleveland Pl & Spring St";40.722103786686034;-73.99724900722504 153;"E 40 St & 5 Ave";40.752062307;-73.9816324043 153;"E 40 St & 5 Ave";40.75206231;-73.9816324 228;"E 48 St & 3 Ave";40.7546011;-73.97187886 228;"E 48 St & 3 Ave";40.7546011026;-73.971878855 ' Any idea why/how to resolve this? – jimseeve May 2 '16 at 4:07
  • @user3426938 check my edit for a hint on how to deal with your case – Ziggy Crueltyfree Zeitgeister May 2 '16 at 5:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.