Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [bulkcopy]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2 votes
1 answer
57 views

Should I delete index before bulk-loading data with COPY?

My pet project has one table which is totally rewritten once a day. This project uses a PostgreSQL database. I decided to use COPY to insert data from 4 CSV files (about 3 million rows). At the same ...
Rash-77's user avatar
  • 21
8 votes
5 answers
1k views

Copying a table (and all of its data) from one server to another?

I have a massive table, let's say 500,000 rows. I want to copy it (schema and data) from one server to another. This is not an upsert or any kind of update; It's a one-off straight copy and paste. ...
J. Mini's user avatar
  • 421
1 vote
2 answers
5k views

Where to download bcp command line package for SQL Server 2019?

I have a Windows 10 machine where I would like to run bcp from powershell to perform bulk copy operations against SQL Server 2019. Which package do I install to get bcp support Windows 10? Would ...
user2368632's user avatar
  • 1,097
3 votes
2 answers
8k views

Copy millions of rows to another table in batches mySQL

Table A which is always getting updated (records being inserted or updated). Table A contains millions of records. I'd like to copy some of these records to a new table Table B. Table A and Table B ...
BountyHunter's user avatar
0 votes
1 answer
3k views

Most efficient method to import bulk JSON data from different sources in postgresql?

I need to import data from thousands of URLs, here is an example of the data: [{"date":"20201006T120000Z","uri":"secret","val":"1765.756"},{...
Lautaro Aguilera's user avatar
0 votes
1 answer
254 views

PostgreSQL - Best way to incremental export 1000+ tables every 5 mins

I have 6 PostgreSQL database servers(v11) hosted on some third-party vendor. I don't have access to setup pg_logical. Each server has 1 database but 1000+ tables. So I want to get the data from these ...
TheDataGuy's user avatar
  • 1,940
0 votes
1 answer
318 views

Best way to copy MS SQL (2000) database files to an external drive

We have a system running on Windows 2003, with a MS SQL server 2000 (Standard) instance as the data source. As we have sold the product, we are in the process of moving everything to the purchasing ...
JohnP's user avatar
  • 159
1 vote
1 answer
601 views

Inspecting the status of a long-running COPY statement

I need to migrate a table from one PostgreSQL database to another. There was a chance I would need to fix some data, so I exported to a CSV. Then I imported the CSV in the second database with a COPY ...
rubik's user avatar
  • 525
0 votes
1 answer
82 views

Copy result of a query to a new PostgreSQL database

I have a Postgres database A with 2 tables, and I need to copy part of these tables (recent records in 1 month) to the new database with the same structures. How to achieve this, as I know cross ...
Tiana987642's user avatar
1 vote
1 answer
885 views

Copying table subset from one database to another with same schema

I am trying to subset of a table from one table to another users(id, name, referrer_id, country_id) this table have self referential association with refferer_id as foreign key to itself. I ...
Tachyons's user avatar
  • 235
1 vote
1 answer
2k views

SELECT vs COPY (SELECT) TO STDOUT

I need to read more rows than fit in memory, resonably fast. I see two options: SELECT ... where my bindings use a cursor to stream the rows; COPY (SELECT ...) TO STDOUT (FORMAT binary) where my ...
Max's user avatar
  • 119
1 vote
2 answers
322 views

SQL Server bulk transfer help needed

We've got this very badly designed logging table that we want to add functionality to. The problem is that it's already a scalability nightmare and we want to fix the design before adding to it, but ...
user1664043's user avatar
-1 votes
1 answer
2k views

Copy in SQL with index vs without index

Recently I have received a data dump in csv format that I am trying to import in my PSQL database. The dump consists of 8 csv file each of size 7GB. I have found that using the Copy command on one ...
user163436's user avatar
1 vote
1 answer
287 views

What data is duplicated when MySQL/MariaDB BLOB columns are copied?

Let table_1 be created as follows: CREATE TABLE table_1 ( id INT AUTO_INCREMENT PRIMARY KEY, some_blob BLOB ); Let table_2 be created as follows: CREATE TABLE table_2 ( id INT ...
AjaxLeung's user avatar
  • 111
0 votes
1 answer
217 views

PostgreSQL: Non-continious replication of new data with COPY

I have a database tables with millions of measurements. New data is coming in every day. For analysis, I want to replicate the data to my laptop (local postgres db). No need for automatic replication. ...
schoettl's user avatar
  • 101
6 votes
1 answer
12k views

Keep empty string instead of null for /copy command

When using the \copy command with psql I have a file with empty values. These values are not to be null but instead always an empty string. My command: \copy agltransact FROM 'dbo_agltransact.csv' ...
Asken's user avatar
  • 317
0 votes
1 answer
165 views

How do I append all tables from one Azure Database to another?

I have two Azure databases (on different Azure instances) with the same schema. I need to append all the records from one of the databases to the other. All keys are GUIDs so should not collide. Many ...
Paul Chernoch's user avatar
-1 votes
1 answer
490 views

Move data chunk from table with 200+million rows to a new table MySQL

Our DB is hosted in AWS MySQL RDS. The data is such that each company that signs up gets its own set of tables which then receive enormous amount of data in them. Currently we want to move a chunk of ...
Asad M's user avatar
  • 1
1 vote
1 answer
3k views

How can I disable all unique index and constraints in Oracle?

I have been trying to do OracleBulkCopy on .NET, however, Oracle throws me errors. What specifically does OracleBulkCopy do, and how can I optimize its performance? (This was the post helped me alot) ...
JackTheRipper's user avatar
5 votes
1 answer
3k views

System.Data.Entity.Infrastructure.CommitFailedException: C# Multithreading & SQL Server 2012

We have a C# multi threaded (100 threads) program which reads the records from the DB and each thread picks up one record (one Entity Framework connection per thread) and update the a single DB table. ...
Balu's user avatar
  • 71
0 votes
1 answer
512 views

huge table copy avoid duplicates

I've a table with more than 10 billion records. Many NEW records (in quantum of 40-50 million records)are added regularly in a file as input part of batch process at night. The active table is majorly ...
pratikch's user avatar
  • 113
0 votes
1 answer
124 views

Trying to load a file into a database on virtual machine

I have set up a Postgres db on a linux vm and have been having no issues using a GUI to connect to it. However, I am trying to load a large, 32GB, file onto it and so am skeptical of the old way I was ...
Ron's user avatar
  • 1
0 votes
2 answers
8k views

Copy whole table to another database [closed]

I want to insert a whole table's contents to another database which has the same table name. How can I achieve this? I am using SQL Server Express 2008 R2.
Kodeeswaran Duraisamy's user avatar
3 votes
1 answer
1k views

Diagnosing the error in a Postgresql to Redshift migration through psycopg2

I am running a job class that contains the following: A Postgresql connection that can issue SQL statements A Redshift connection that can do the same an S3 connection to function as an intermediary ...
yburyug's user avatar
  • 235
1 vote
3 answers
6k views

psql: id SERIAL PRIMARY KEY not starting with 1?

This is something I don't understand, can somebody explain? I create a table CREATE TABLE cali (id SERIAL PRIMARY KEY, ALK_from char, ALK_to char, AND_from char, AND_to char); Then ...
LarsVegas's user avatar
  • 107
4 votes
3 answers
69k views

SQL Server - Copy data from one database to another - identical tables, new identity values needed

Here's the thing, I have been running identical copies of my website for two different geographical regions. The site now supports multi-currency pricing so I want to merge both sites together. All ...
Jordan's user avatar
  • 41
5 votes
1 answer
5k views

Copy few rows from one database to another

I have two PostgreSQL databases on two different machines and the network connection is slow. I would like to copy few selected rows from the first instance to the second. Unfortunately, the full dump ...
Michas's user avatar
  • 185
3 votes
1 answer
2k views

BULK INSERT Data from file but ignore the first occurence of field terminator

I have a file that has data arranged in the following style: 1 1 0.5102 0.4898 0.0000 0.0000 0.0000 1 2 0.3114 0.6886 0.0000 0.0000 0.0000 I need to load ...
sriramn's user avatar
  • 429
18 votes
1 answer
58k views

Usage of COPY FROM STDIN in postgres

I just started using Postgres and I'm trying to create a sample db to understand its functions, looking around, I found some scripts in pgfoundry.org. I understand the commands, since I previously ...
Eugenio Laghi's user avatar
4 votes
1 answer
5k views

Postgres copy data with \xYY as plain string instead of interpreting as encoded string

I have a log file full of URLs, generated by Bro IDS. When Bro logs a URL with non-ascii characters in it, it inserts \xYY where YY is the hexadecimal character code. Also, some URLs contain "\x". ...
potatohead00's user avatar
4 votes
0 answers
3k views

Can PostgreSQL's COPY create CSV with Locale-specific number formatting?

I want to create a CSV with PostgreSQL 9.1 COPY that directly opens correctly with a double-click in crazy German Excel 2010. Using ";"-Delimiter is easy. The encoding is solved, but Excel only ...
alfonx's user avatar
  • 827
10 votes
2 answers
4k views

Variable substitution in psql when using \copy

I'm using the psql command \copy and I would like to pass a variable to it from the shell (for table name) like I've done when scripting queries. I've read in the documentation for psql that: The ...
beacon_bonanza's user avatar
0 votes
1 answer
10k views

Save output from multiple SQL SELECT commands to a file

Within a Postgres function I would like to run a number of SQL SELECT commands and append the output to the same variable. After all the SELECT commands have run this output should be written to a ...
John Mitchell's user avatar
7 votes
2 answers
5k views

BCP stops after the last "1000 rows sent to SQL Server. Total sent: ..."

I ran SQL Server 2005's bulk copy program BCP like this: bcp mydb.dbo.mytbl in myfile.blk -c -S mysvr -U mylogin -P mypass And it ran and produced output like so: Starting copy... 1000 rows sent to ...
Endy Tjahjono's user avatar
3 votes
1 answer
6k views

Duplicate primary key in bulk insert after truncate in SQL Server 2008?

I'm using SSIS to dump data from my SQL Server 2008 prod database to flat files and then using bulk insert to load those flat files into the SQL Server 2008 dev database. The SSIS package is running a ...
stomcavage's user avatar
2 votes
2 answers
8k views

Connecting remote MySQL database to local MySQL database?

I want to write PHP code to be embedded in a Drupal7 module. I want to call a procedure which can copy newly generated data in the local MySQL database to a remote MySQL database. When data is ...
Shashank's user avatar
4 votes
4 answers
4k views

What options exist for copying data between servers on a regular basis?

My apologies if this has already been answered. I've searched SO and of course here on DBA, and am surprised to find no close matches. Specifically, I'm looking for a solution which survives schema ...
Jon of All Trades's user avatar
1 vote
1 answer
743 views

using COPY command and wal_buffers

since COPY command runs inside a single transaction, if its WAL data doesn't fit into wal_buffers? for example if we are bulk importing from CSV file.
Majid Azimi's user avatar
  • 2,271
2 votes
2 answers
4k views

Move data from one table to other in batch's

I have a table 18 Million records I need to add append it to another table (with the same structure) I tried the import-export wizard, the problem is that the log gets very very big and eventually it ...
Ezi's user avatar
  • 652
3 votes
2 answers
3k views

Copy huge amount from 1 table to other

I have two tables with same structure I need to copy over all tables from table A to table B, the problem is that some records from table A already exist in table B so that made the Import Fail. So I ...
Ezi's user avatar
  • 652
4 votes
3 answers
2k views

Copy 8 Million Records

I have a table in a local MS SQL Server database that has 72 columns and over 8 million records. Most of the columns are floats except for the primary key which is an INT Identity column. The table ...
Bobby Ortiz's user avatar
10 votes
4 answers
98k views

ORA-01502: index or partition of such index is in usable state problem

I have a table in my Oracle database, where select pkcol, count(*) from myTable group by pkcol having count(*) > 1; yields PKCOL COUNT(*) ------- ---------- 1 2 2 ...
bernd_k's user avatar
  • 12.2k
2 votes
1 answer
3k views

What is the purpose of sqlbulkcopy's useinternaltransaction property?

I'm just testing my PowerShellScript to do bulkcopy of multiple tables between sql-server and oracle Databases. When the destination is Oracle, than I can monitor the progress by by executing select ...
bernd_k's user avatar
  • 12.2k