Questions tagged [bulkcopy]
The bulkcopy tag has no usage guidance.
41
questions
1
vote
2
answers
2k
views
Where to download bcp command line package for SQL Server 2019?
I have a Windows 10 machine where I would like to run bcp from powershell to perform bulk copy operations against SQL Server 2019. Which package do I install to get bcp support Windows 10? Would ...
- 1,077
2
votes
2
answers
5k
views
Copy millions of rows to another table in batches mySQL
Table A which is always getting updated (records being inserted or updated).
Table A contains millions of records.
I'd like to copy some of these records to a new table Table B.
Table A and Table B ...
- 23
0
votes
1
answer
2k
views
Most efficient method to import bulk JSON data from different sources in postgresql?
I need to import data from thousands of URLs, here is an example of the data:
[{"date":"20201006T120000Z","uri":"secret","val":"1765.756"},{...
0
votes
1
answer
234
views
PostgreSQL - Best way to incremental export 1000+ tables every 5 mins
I have 6 PostgreSQL database servers(v11) hosted on some third-party vendor. I don't have access to setup pg_logical. Each server has 1 database but 1000+ tables.
So I want to get the data from these ...
- 1,900
0
votes
1
answer
285
views
Best way to copy MS SQL (2000) database files to an external drive
We have a system running on Windows 2003, with a MS SQL server 2000 (Standard) instance as the data source. As we have sold the product, we are in the process of moving everything to the purchasing ...
- 159
1
vote
1
answer
303
views
Inspecting the status of a long-running COPY statement
I need to migrate a table from one PostgreSQL database to another. There was a chance I would need to fix some data, so I exported to a CSV. Then I imported the CSV in the second database with a COPY ...
- 515
0
votes
1
answer
79
views
Copy result of a query to a new PostgreSQL database
I have a Postgres database A with 2 tables, and I need to copy part of these tables (recent records in 1 month) to the new database with the same structures.
How to achieve this, as I know cross ...
- 131
1
vote
1
answer
826
views
Copying table subset from one database to another with same schema
I am trying to subset of a table from one table to another
users(id, name, referrer_id, country_id)
this table have self referential association with refferer_id as foreign key to itself.
I ...
- 225
1
vote
1
answer
2k
views
SELECT vs COPY (SELECT) TO STDOUT
I need to read more rows than fit in memory, resonably fast. I see two options:
SELECT ... where my bindings use a cursor to stream the rows;
COPY (SELECT ...) TO STDOUT (FORMAT binary) where my ...
- 119
1
vote
2
answers
289
views
SQL Server bulk transfer help needed
We've got this very badly designed logging table that we want to add functionality to. The problem is that it's already a scalability nightmare and we want to fix the design before adding to it, but ...
- 307
-1
votes
1
answer
2k
views
Copy in SQL with index vs without index
Recently I have received a data dump in csv format that I am trying to import in my PSQL database. The dump consists of 8 csv file each of size 7GB.
I have found that using the Copy command on one ...
1
vote
1
answer
265
views
What data is duplicated when MySQL/MariaDB BLOB columns are copied?
Let table_1 be created as follows:
CREATE TABLE table_1 (
id INT AUTO_INCREMENT PRIMARY KEY,
some_blob BLOB
);
Let table_2 be created as follows:
CREATE TABLE table_2 (
id INT ...
- 111
0
votes
1
answer
172
views
PostgreSQL: Non-continious replication of new data with COPY
I have a database tables with millions of measurements. New data is coming in every day. For analysis, I want to replicate the data to my laptop (local postgres db). No need for automatic replication. ...
- 101
5
votes
1
answer
11k
views
Keep empty string instead of null for /copy command
When using the \copy command with psql I have a file with empty values. These values are not to be null but instead always an empty string.
My command:
\copy agltransact FROM 'dbo_agltransact.csv' ...
- 307
0
votes
1
answer
154
views
How do I append all tables from one Azure Database to another?
I have two Azure databases (on different Azure instances) with the same schema. I need to append all the records from one of the databases to the other. All keys are GUIDs so should not collide.
Many ...
- 103
-1
votes
1
answer
436
views
Move data chunk from table with 200+million rows to a new table MySQL
Our DB is hosted in AWS MySQL RDS. The data is such that each company that signs up gets its own set of tables which then receive enormous amount of data in them.
Currently we want to move a chunk of ...
- 1
1
vote
1
answer
2k
views
How can I disable all unique index and constraints in Oracle?
I have been trying to do OracleBulkCopy on .NET, however, Oracle throws me errors. What specifically does OracleBulkCopy do, and how can I optimize its performance? (This was the post helped me alot) ...
5
votes
1
answer
3k
views
System.Data.Entity.Infrastructure.CommitFailedException: C# Multithreading & SQL Server 2012
We have a C# multi threaded (100 threads) program which reads the records from the DB and each thread picks up one record (one Entity Framework connection per thread) and update the a single DB table.
...
- 71
0
votes
1
answer
511
views
huge table copy avoid duplicates
I've a table with more than 10 billion records. Many NEW records (in quantum of 40-50 million records)are added regularly in a file as input part of batch process at night. The active table is majorly ...
- 113
0
votes
1
answer
103
views
Trying to load a file into a database on virtual machine
I have set up a Postgres db on a linux vm and have been having no issues using a GUI to connect to it. However, I am trying to load a large, 32GB, file onto it and so am skeptical of the old way I was ...
- 1
0
votes
2
answers
8k
views
Copy whole table to another database [closed]
I want to insert a whole table's contents to another database which has the same table name.
How can I achieve this?
I am using SQL Server Express 2008 R2.
3
votes
1
answer
1k
views
Diagnosing the error in a Postgresql to Redshift migration through psycopg2
I am running a job class that contains the following:
A Postgresql connection that can issue SQL statements
A Redshift connection that can do the same
an S3 connection to function as an intermediary ...
- 205
1
vote
3
answers
6k
views
psql: id SERIAL PRIMARY KEY not starting with 1?
This is something I don't understand, can somebody explain?
I create a table
CREATE TABLE cali (id SERIAL PRIMARY KEY,
ALK_from char,
ALK_to char,
AND_from char,
AND_to char);
Then ...
- 107
4
votes
3
answers
69k
views
SQL Server - Copy data from one database to another - identical tables, new identity values needed
Here's the thing, I have been running identical copies of my website for two different geographical regions. The site now supports multi-currency pricing so I want to merge both sites together.
All ...
- 41
5
votes
1
answer
4k
views
Copy few rows from one database to another
I have two PostgreSQL databases on two different machines and the network connection is slow.
I would like to copy few selected rows from the first instance to the second. Unfortunately, the full dump ...
- 185
3
votes
1
answer
2k
views
BULK INSERT Data from file but ignore the first occurence of field terminator
I have a file that has data arranged in the following style:
1 1 0.5102 0.4898 0.0000 0.0000 0.0000
1 2 0.3114 0.6886 0.0000 0.0000 0.0000
I need to load ...
- 419
17
votes
1
answer
55k
views
Usage of COPY FROM STDIN in postgres
I just started using Postgres and I'm trying to create a sample db to understand its functions, looking around, I found some scripts in pgfoundry.org.
I understand the commands, since I previously ...
- 273
4
votes
1
answer
5k
views
Postgres copy data with \xYY as plain string instead of interpreting as encoded string
I have a log file full of URLs, generated by Bro IDS.
When Bro logs a URL with non-ascii characters in it, it inserts \xYY where YY is the hexadecimal character code. Also, some URLs contain "\x".
...
- 43
4
votes
0
answers
3k
views
Can PostgreSQL's COPY create CSV with Locale-specific number formatting?
I want to create a CSV with PostgreSQL 9.1 COPY that directly opens correctly with a double-click in crazy German Excel 2010. Using ";"-Delimiter is easy.
The encoding is solved, but Excel only ...
- 787
9
votes
2
answers
4k
views
Variable substitution in psql when using \copy
I'm using the psql command \copy and I would like to pass a variable to it from the shell (for table name) like I've done when scripting queries. I've read in the documentation for psql that:
The ...
0
votes
1
answer
10k
views
Save output from multiple SQL SELECT commands to a file
Within a Postgres function I would like to run a number of SQL SELECT commands and append the output to the same variable. After all the SELECT commands have run this output should be written to a ...
- 195
7
votes
2
answers
5k
views
BCP stops after the last "1000 rows sent to SQL Server. Total sent: ..."
I ran SQL Server 2005's bulk copy program BCP like this:
bcp mydb.dbo.mytbl in myfile.blk -c -S mysvr -U mylogin -P mypass
And it ran and produced output like so:
Starting copy...
1000 rows sent to ...
- 381
3
votes
1
answer
6k
views
Duplicate primary key in bulk insert after truncate in SQL Server 2008?
I'm using SSIS to dump data from my SQL Server 2008 prod database to flat files and then using bulk insert to load those flat files into the SQL Server 2008 dev database. The SSIS package is running a ...
- 173
2
votes
2
answers
8k
views
Connecting remote MySQL database to local MySQL database?
I want to write PHP code to be embedded in a Drupal7 module.
I want to call a procedure which can copy newly generated data in the local MySQL database to a remote MySQL database.
When data is ...
- 23
4
votes
4
answers
4k
views
What options exist for copying data between servers on a regular basis?
My apologies if this has already been answered. I've searched SO and of course here on DBA, and am surprised to find no close matches. Specifically, I'm looking for a solution which survives schema ...
- 5,937
1
vote
1
answer
633
views
using COPY command and wal_buffers
since COPY command runs inside a single transaction, if its WAL data doesn't fit into wal_buffers? for example if we are bulk importing from CSV file.
- 2,161
2
votes
2
answers
4k
views
Move data from one table to other in batch's
I have a table 18 Million records I need to add append it to another table (with the same structure) I tried the import-export wizard, the problem is that the log gets very very big and eventually it ...
- 652
3
votes
2
answers
3k
views
Copy huge amount from 1 table to other
I have two tables with same structure I need to copy over all tables from table A to table B, the problem is that some records from table A already exist in table B so that made the Import Fail.
So I ...
- 652
4
votes
3
answers
2k
views
Copy 8 Million Records
I have a table in a local MS SQL Server database that has 72 columns and over 8 million records.
Most of the columns are floats except for the primary key which is an INT Identity column.
The table ...
- 141
10
votes
4
answers
89k
views
ORA-01502: index or partition of such index is in usable state problem
I have a table in my Oracle database, where
select pkcol, count(*) from myTable group by pkcol having count(*) > 1;
yields
PKCOL COUNT(*)
------- ----------
1 2
2 ...
- 12.1k
2
votes
1
answer
3k
views
What is the purpose of sqlbulkcopy's useinternaltransaction property?
I'm just testing my PowerShellScript to do bulkcopy of multiple tables between sql-server and oracle Databases.
When the destination is Oracle, than I can monitor the progress by by executing
select ...
- 12.1k