Questions tagged [archive]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
1 answer
1k views

Restoring dump(backup): ERROR 1022 (23000) at line 44: Can't write; duplicate key in table

I'm doing tests to change a current table from my InnoDB database to Archive, but I had a problem trying to restore a dump after the change. After several tests, the problem is related to the value ...
0 votes
0 answers
29 views

SQL Server database migration issues

I have a system with two servers and three client workstations. Servers were configured to work as principal/mirror servers (mirroring was configured with certificates). I have two databases in SQL ...
0 votes
1 answer
858 views

I need to create a process where anything older than six months we would write to an archive table and delete the records from the inventory table

I need to create a process where anything older than six months we would write to an archive table and delete the records from the inventory table. Our inv table is huge btw lets use (x) as the number....
3 votes
1 answer
801 views

MongoDB sharding by monotonic id with zones for archiving

Situation: I have an ever-growing collection of documents which do have monotonically increasing unique id. All queries are direct lookup for documents with specified _id (no range queries nor ...
0 votes
1 answer
481 views

Options for archiving old MySQL 5.7data

I have an web application that has a 1TB MySQL 5.7 production database. We are running into performance issues due to table data size. We are planning to archive old data from different tables. ...
0 votes
0 answers
26 views

Archiving tables with foreign key constraints in PostgreSQL

I'd like to get your thoughts on the approach to archiving data from a set of tables with foreign key constraints. Please find the tables details below. booking (Table):ID (UUID), created_date_time ...
-1 votes
1 answer
1k views

Best practice for data archiving

I am trying to figure out the best practice for data archiving. I have 1 pair of Master/Slave MariaDB servers where data is replicated from Master to slave. Since this is a very old app I am not able ...
1 vote
2 answers
4k views

Best Practices to archive or remove old data in SQL Server 2012

Our application has numerous tables that contain outdated data, some dating as far back as 2010. Despite having indexes defined on these tables, many of the index seeks retrieve old records, which is ...
0 votes
2 answers
261 views

Pt-archiver csv double quote missing for some fields

I am trying to use pt-archive to archive my database. Below is my script for the archiving process. pt-archiver --source h=localhost,D=***,t=** --user=*** --password=**** --where "dateTime<'...
1 vote
1 answer
97 views

data archival postgres

I have this code to delete rows with column "id". I want to limit the number of rows deleted in each loop. This code deletes the first 25k rows and the next loop deletes nothing since its ...
1 vote
0 answers
328 views

"Snapshot" of Historical Data for Analysis Purposes

I pull monthly aggregate data from a SQL database. I store the aggregated data in a separate table in the same database (so I can quickly grab numbers from six months ago, for instance). My issue is ...
0 votes
1 answer
1k views

Archiving SQLite DB with WAL mode

I have an SQLite database gapongasi.db configured using WAL mode and I want to have daily archive with tar(1). With WAL, there are two files being used (1) gapongasi.db-wal and (2)gapongasi.db-shm, ...
0 votes
2 answers
858 views

Database returned to CIRCULAR mode but logs are still being archived to old location

my DB2 database (v 9.7) is in Circular mode. Current Logging Type:Circular (USEREXIT) = OFF (LOGRETAIN) = OFF (LOGARCHMETH1) = OFF (LOGARCHMETH2) = OFF (LOGPRIMARY) = 10 (LOGSECOND) = 10 Path to log ...
0 votes
1 answer
1k views

understanding pg_wal and archiving

So I inherited this database. At the very beginning, there were already some files on pg_wal. Then, after some weeks, I do count SELECT COUNT(*) FROM pg_ls_dir('pg_wal') WHERE pg_ls_dir ~ '^[0-9A-F]{...
0 votes
1 answer
95 views

Why is a Move and Delete Data (approx 1/3 of the table) using (CTEs) WITH statements taking hours on a 600m row table taking 3+ hours - best approach?

Am not sure if this is the best method of moving/deleting data. The play table has 700m+ rows and game table has over 600m+ rows. I have 2 tables play and game, information from the play table is ...
0 votes
1 answer
2k views

Determining data retention period

I have an application that has generating data for the past 15 years. Currently, no retention period is configured so the data size grows day by day. I would like to add a retention period to the data ...
0 votes
1 answer
204 views

Auto-archive MariaDB10 database table

I am trying to auto-archive active database table to archive database table using stored procedure and event in MariaDB10. These are the steps I am following: Create new table everyday just before ...
1 vote
3 answers
1k views

Database archiving strategy

I work on a payment system (a bit regulated area with requirements to store really old data for audits & customer requests). We have SQL Server 2016 (but will move to 2019 soon). System is 16 ...
0 votes
1 answer
305 views

Oracle - Big Table Archiving

Version: Oracle 12cR2 EE The table contains the following column types: NUMBER, CLOB, NVARCHAR2, TIMESTAMP, VARCHAR2 Table size: 95GB Table row count: 20M Table partitioned? No Indexes on table: there ...
1 vote
1 answer
620 views

Best way for archiving old mysql records

we have a Ruby on Rails app with a MySQL DB (on AWS RDS). Some of our tables have a large number of records ~100 million and this number is rapidly increasing every month. Most of the records in these ...
user avatar
2 votes
1 answer
3k views

Best practice for removing old data in postgres table but retaining a copy of removed data

We currently have a table that has approx 20 columns (one of the columns being a timestamptz datatype) in our database server which has 834 million rows. This in terms of size is a big table (approx ...
0 votes
1 answer
72 views

When should I Switch In from Stage Table to a SQL OLTP Partitioned Table

I have an OLTP Database with a big table (30M records a day / 400TPS). I want to implement Partitioning & Sliding Window on that table with Monthly Partitions plus a temp Stage-Table. The time to ...
0 votes
1 answer
77 views

SQL Server Partitioning by Local And Remote database

We have a table related to documents(about 17m records on one table), and another one for properties of that documents. And team decided to archive old entries. I thought about partitioning with ...
0 votes
1 answer
356 views

MYSQL 5.7 Partition a table based on partition key of another related table

Please understand a little background before getting to know the problem. TABLE transaction ( t_id: AUTO INCREMENT BIGINT t_execute_on_date: DATE, timestamp: DATETIME, ... other columns ) The ...
1 vote
1 answer
504 views

Where to put standby control file in Physical standby database?

I am creating a physical standby database for which I am creating a standby control file using command = ALTER DATABASE CREATE STANDBY CONTROLFILE AS '/oracle/dbs/stbycf.ctl'; but It creates only one ...
0 votes
1 answer
2k views

MongoDB - Can I create TTL index with rolling method

Im getting stuck with archiving a huge amount of data in MongoDB 3.6 I want to delete 506 million records in a collection. I tried to remove using bulk.remove(), but that is also slow. 50 records are ...
1 vote
0 answers
97 views

How to configure partial replication of Oracle DB

Suppose we have 2 identical Oracle 11g DB schemas: PRODUCTION and ARCHIVE. The goal is to keep in PRODUCTION the minimal amount of data required for online operations, and use ARCHIVE to store all the ...
0 votes
1 answer
171 views

How to properly replicate and archive at once

I have a 300GB database that is growing around 5GB a day. It stores time-based data, and as such, I currently have around 60 days of data. My main database server has a 450 GB NVMe, so I want to get a ...
1 vote
3 answers
25k views

DB2 ARCHIVE mode - Can all archive logs be deleted after doing the FULL OFFLINE backup?

I am interested in one thing, if I understood correctly. DB2 is in Archive mode (online backups are made and archive logs keep coming which is OK). If I made offline full backup today and I know I ...
0 votes
2 answers
62 views

Analyze MySQL what needs to be archived

I'm not a DBA, I'm a DevOps guy trying to manage a MySQL database which is ever growing and desperately needs some archiving. My question is not how to archive. I think I have a pretty good idea and ...
0 votes
1 answer
1k views

mysql - How to repair & backup archive table, also stop growing .ARN extension file?

I have an MySQL table build with archive storage engine. The table has over 100k records. I have got disk full and it turned out to be a large .ARN file from an archive table. When I cleared some ...
2 votes
3 answers
2k views

Extracting Data from multiple archive tables

I am developing a report in report builder 3.0 where I am extracting data from a table dbo.odcalls. The problem is this table is archived on monthly basis to a separate database (HN_Backup), and I ...
0 votes
1 answer
229 views

Use a SP to Archive/Revision related records in several related tables?

I have a database for an application that has a complicated relational table structure. Is there a good way to create revisions of the data, or archive the data before a user makes changes? Ideally ...
0 votes
1 answer
332 views

Table partitioning on OLTP

On SQL Server 2017 Std, I have an SSIS package that runs once a day on a 24×7 mission critical operational database to archive the largest transactional tables. The data is moved to tables on a ...
-1 votes
1 answer
3k views

Postgresql - archive_command Windows, not working

I'm attempting to set up archiving on a windows server machine but when I check the logs nothing is happening. The postgresql.conf file this is my input: #wal_level = replica #archive_mode = on #...
2 votes
1 answer
35 views

accessing, updating a large MYSQL table advice and guidance

I'm after some guidance or advice really. We've built an application and its gone mental with custom since coronavirus. We are after some advice for something we have never tackled. The problem we ...
3 votes
2 answers
38k views

What is the Difference between Archiving and Purging in Oracle database?

I would like to know the main difference between Archiving and Purging in any database we use.
1 vote
1 answer
3k views

What are the options for archiving old data of mariadb tables if partitioning can not be implemented due to a restriction

We are planning to archive some old data of our MariaDB tables to some other volume(or mount point) of the linux server apart from the one the tables are on. We are using partitioning for this and ...
0 votes
1 answer
98 views

Yearly database archive design

I need to implement a DB of tracking numbers. The tracking numbers table could potentially contain billions of records during its lifetime so an archive database should be implemented to improve ...
0 votes
2 answers
828 views

Archive old data in a large table in DB2

I have a very large non-partitioned table with ~4,500,000,000 records: ODS.LARGE_TABLE. The table has stored 3 months of data with ~50,000,000 rows per day, therefore resulting total of ~4,500,000,...
0 votes
0 answers
572 views

Archiving Huge table with Foreign Key dependencies

We have a huge table of ~ 1TB (data+Indexes) which has ~7 child tables with Foreign key relation ship We have decided to archive the huge table but not the child tables(since they are not that big ) ...
-1 votes
1 answer
329 views

Archiving strategy

I have an instance with only one database which contains 4 years of data (400 gigas). I name this instance inst_live I want to implement this following archiving strategy: create an archive instance (...
6 votes
2 answers
3k views

Managing a SQL Server database with over one terabyte of data [closed]

Recently we have an audit database that went over one terabyte and, since we have storage problems, management is looking for options. My proposal is at the end of each year we take a backup and ...
0 votes
2 answers
2k views

Inserting into mysql table with archive engine "duplicate key" error

I am trying to archive some files using mysql archive engine. I am using this query to insert file contents: insert into test_table (id,arch) values (123,'FILE_CONTENT') After inserting 2 records, I ...
0 votes
0 answers
651 views

Archiving a huge SQL Server table with change data capture and foreign key constraints on multiple columns

We are trying to archive a huge table with ~170 million rows, which is partitioned, has change data capture (CDC) enabled and has ~5 foreign key constraints. We decided on the archival since the ...
1 vote
2 answers
1k views

How to move old data to cheap hard drive

I have a quite big PostgreSQL database (with timescale plugin). Right now it is consuming about 500Gb on a SSD. Most of the data is in the form of time series. In most cases data older than a few ...
2 votes
1 answer
9k views

archive command repeatly failing for one particular file

I am seeing many pg_log files full of following errors 2019-07-08 00:04:06 CEST DETAIL: The failed archive command was: copy "pg_xlog\000000010000002100000025" "D:/Programme/OmniBack/server/db80/pg/...
0 votes
1 answer
74 views

Which mechanism is best for previous data archived in oracle?

Our company has a big data server in which large data is stored, like images, video, audio, etc. Our company wants a process running with the new year, such that in 2020 the previous year's 2019 data ...
1 vote
1 answer
5k views

mongodb - monthly archival of data

Coming from a RDBMS background, this is what I have done in the past. Create a table with partitions based on a date field. There are typically three tables (partitions) - current (M0), previous ...
1 vote
2 answers
126 views

Resetting a new archiving database

I have an archiving question. We have a History DB used for archiving old records from our main system and the application knows to query it However, it becomes too big. I would like to ...