Questions tagged [bulkcopy]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
1answer
28 views

Inspecting the status of a long-running COPY statement

I need to migrate a table from one PostgreSQL database to another. There was a chance I would need to fix some data, so I exported to a CSV. Then I imported the CSV in the second database with a COPY ...
0
votes
1answer
19 views

Copy result of a query to a new PostgreSQL database

I have a Postgres database A with 2 tables, and I need to copy part of these tables (recent records in 1 month) to the new database with the same structures. How to achieve this, as I know cross ...
0
votes
1answer
29 views

Copying table subset from one database to another with same schema

I am trying to subset of a table from one table to another users(id, name, referrer_id, country_id) this table have self referential association with refferer_id as foreign key to itself. I ...
0
votes
1answer
134 views

SELECT vs COPY (SELECT) TO STDOUT

I need to read more rows than fit in memory, resonably fast. I see two options: SELECT ... where my bindings use a cursor to stream the rows; COPY (SELECT ...) TO STDOUT (FORMAT binary) where my ...
1
vote
2answers
111 views

SQL Server bulk transfer help needed

We've got this very badly designed logging table that we want to add functionality to. The problem is that it's already a scalability nightmare and we want to fix the design before adding to it, but ...
-1
votes
1answer
276 views

Copy in SQL with index vs without index

Recently I have received a data dump in csv format that I am trying to import in my PSQL database. The dump consists of 8 csv file each of size 7GB. I have found that using the Copy command on one ...
1
vote
1answer
49 views

What data is duplicated when MySQL/MariaDB BLOB columns are copied?

Let table_1 be created as follows: CREATE TABLE table_1 ( id INT AUTO_INCREMENT PRIMARY KEY, some_blob BLOB ); Let table_2 be created as follows: CREATE TABLE table_2 ( id INT ...
0
votes
1answer
20 views

PostgreSQL: Non-continious replication of new data with COPY

I have a database tables with millions of measurements. New data is coming in every day. For analysis, I want to replicate the data to my laptop (local postgres db). No need for automatic replication. ...
2
votes
1answer
2k views

Keep empty string instead of null for /copy command

When using the \copy command with psql I have a file with empty values. These values are not to be null but instead always an empty string. My command: \copy agltransact FROM 'dbo_agltransact.csv' ...
0
votes
1answer
82 views

How do I append all tables from one Azure Database to another?

I have two Azure databases (on different Azure instances) with the same schema. I need to append all the records from one of the databases to the other. All keys are GUIDs so should not collide. Many ...
0
votes
1answer
144 views

Move data chunk from table with 200+million rows to a new table MySQL

Our DB is hosted in AWS MySQL RDS. The data is such that each company that signs up gets its own set of tables which then receive enormous amount of data in them. Currently we want to move a chunk of ...
0
votes
0answers
61 views

Efficiently moving Data from CSV to SQL 2014

I am working on a requirement to transfer almost 1 TB of data from a CSV file to SQL server. Which is the fastest way to do this? I am considering bulk copy and came across this link which actually ...
1
vote
1answer
1k views

How can I disable all unique index and constraints in Oracle?

I have been trying to do OracleBulkCopy on .NET, however, Oracle throws me errors. What specifically does OracleBulkCopy do, and how can I optimize its performance? (This was the post helped me alot) ...
5
votes
1answer
2k views

System.Data.Entity.Infrastructure.CommitFailedException: C# Multithreading & SQL Server 2012

We have a C# multi threaded (100 threads) program which reads the records from the DB and each thread picks up one record (one Entity Framework connection per thread) and update the a single DB table. ...
0
votes
1answer
501 views

huge table copy avoid duplicates

I've a table with more than 10 billion records. Many NEW records (in quantum of 40-50 million records)are added regularly in a file as input part of batch process at night. The active table is majorly ...
0
votes
0answers
55 views

MySQL: migration huge tables between two databases

Hello guys (and sorry for my English). I have 2 DB's - DB1 (staging) and DB2 (production), each day some contend added/changed in some tables in DB1 and should be moved to DB2. Right now it works ...
0
votes
1answer
72 views

Trying to load a file into a database on virtual machine

I have set up a Postgres db on a linux vm and have been having no issues using a GUI to connect to it. However, I am trying to load a large, 32GB, file onto it and so am skeptical of the old way I was ...
0
votes
2answers
8k views

Copy whole table to another database [closed]

I want to insert a whole table's contents to another database which has the same table name. How can I achieve this? I am using SQL Server Express 2008 R2.
3
votes
1answer
1k views

Diagnosing the error in a Postgresql to Redshift migration through psycopg2

I am running a job class that contains the following: A Postgresql connection that can issue SQL statements A Redshift connection that can do the same an S3 connection to function as an intermediary ...
1
vote
3answers
4k views

psql: id SERIAL PRIMARY KEY not starting with 1?

This is something I don't understand, can somebody explain? I create a table CREATE TABLE cali (id SERIAL PRIMARY KEY, ALK_from char, ALK_to char, AND_from char, AND_to char); Then ...
1
vote
3answers
66k views

SQL Server - Copy data from one database to another - identical tables, new identity values needed

Here's the thing, I have been running identical copies of my website for two different geographical regions. The site now supports multi-currency pricing so I want to merge both sites together. All ...
3
votes
1answer
2k views

Copy few rows from one database to another

I have two PostgreSQL databases on two different machines and the network connection is slow. I would like to copy few selected rows from the first instance to the second. Unfortunately, the full dump ...
3
votes
1answer
2k views

BULK INSERT Data from file but ignore the first occurence of field terminator

I have a file that has data arranged in the following style: 1 1 0.5102 0.4898 0.0000 0.0000 0.0000 1 2 0.3114 0.6886 0.0000 0.0000 0.0000 I need to load ...
14
votes
1answer
36k views

Usage of COPY FROM STDIN in postgres

I just started using Postgres and I'm trying to create a sample db to understand its functions, looking around, I found some scripts in pgfoundry.org. I understand the commands, since I previously ...
3
votes
1answer
4k views

Postgres copy data with \xYY as plain string instead of interpreting as encoded string

I have a log file full of URLs, generated by Bro IDS. When Bro logs a URL with non-ascii characters in it, it inserts \xYY where YY is the hexadecimal character code. Also, some URLs contain "\x". ...
4
votes
0answers
2k views

Can PostgreSQL's COPY create CSV with Locale-specific number formatting?

I want to create a CSV with PostgreSQL 9.1 COPY that directly opens correctly with a double-click in crazy German Excel 2010. Using ";"-Delimiter is easy. The encoding is solved, but Excel only ...
8
votes
2answers
3k views

Variable substitution in psql when using \copy

I'm using the psql command \copy and I would like to pass a variable to it from the shell (for table name) like I've done when scripting queries. I've read in the documentation for psql that: The ...
0
votes
1answer
8k views

Save output from multiple SQL SELECT commands to a file

Within a Postgres function I would like to run a number of SQL SELECT commands and append the output to the same variable. After all the SELECT commands have run this output should be written to a ...
7
votes
2answers
3k views

BCP stops after the last “1000 rows sent to SQL Server. Total sent: …”

I ran SQL Server 2005's bulk copy program BCP like this: bcp mydb.dbo.mytbl in myfile.blk -c -S mysvr -U mylogin -P mypass And it ran and produced output like so: Starting copy... 1000 rows sent to ...
3
votes
1answer
4k views

Duplicate primary key in bulk insert after truncate in SQL Server 2008?

I'm using SSIS to dump data from my SQL Server 2008 prod database to flat files and then using bulk insert to load those flat files into the SQL Server 2008 dev database. The SSIS package is running a ...
1
vote
1answer
7k views

Connecting remote MySQL database to local MySQL database?

I want to write PHP code to be embedded in a Drupal7 module. I want to call a procedure which can copy newly generated data in the local MySQL database to a remote MySQL database. When data is ...
4
votes
4answers
3k views

What options exist for copying data between servers on a regular basis?

My apologies if this has already been answered. I've searched SO and of course here on DBA, and am surprised to find no close matches. Specifically, I'm looking for a solution which survives schema ...
1
vote
1answer
444 views

using COPY command and wal_buffers

since COPY command runs inside a single transaction, if its WAL data doesn't fit into wal_buffers? for example if we are bulk importing from CSV file.
2
votes
2answers
3k views

Move data from one table to other in batch's

I have a table 18 Million records I need to add append it to another table (with the same structure) I tried the import-export wizard, the problem is that the log gets very very big and eventually it ...
3
votes
2answers
3k views

Copy huge amount from 1 table to other

I have two tables with same structure I need to copy over all tables from table A to table B, the problem is that some records from table A already exist in table B so that made the Import Fail. So I ...
4
votes
3answers
1k views

Copy 8 Million Records

I have a table in a local MS SQL Server database that has 72 columns and over 8 million records. Most of the columns are floats except for the primary key which is an INT Identity column. The table ...
5
votes
2answers
51k views

ORA-01502: index or partition of such index is in usable state problem

I have a table in my Oracle database, where select pkcol, count(*) from myTable group by pkcol having count(*) > 1; yields PKCOL COUNT(*) ------- ---------- 1 2 2 ...
2
votes
1answer
2k views

What is the purpose of sqlbulkcopy's useinternaltransaction property?

I'm just testing my PowerShellScript to do bulkcopy of multiple tables between sql-server and oracle Databases. When the destination is Oracle, than I can monitor the progress by by executing select ...