Questions tagged [aws]
Amazon Web Services.
500 questions
3
votes
1
answer
2k
views
Aurora upgrade 2 to 3 / MySql 5.7 to 8.0: schemaInconsistencyCheck
In our attempts to upgrade our Aurora 2 MySql 5.7 DB to Aurora 3 MySql 8.0, we are running into a pre-check that we are unable to resolve. The error is:
{
"id": "...
- 31
4
votes
1
answer
3k
views
Why SQL inserts are slower in AWS Aurora Serverless than in AWS RDS
We had an AWS MySQL RDS t3.xlarge instance with 2000 IOPS. Also, I have a cron job written in java which inserts data in a table: it can insert up to 12M rows daily. Since AWS RDS cost is quite high ...
- 141
0
votes
1
answer
47
views
Where to install SQL on EC2
If SQL Server and it's system databases are installed on an EC2 local drive, will they be recovered in a crash?
My understanding is nothing but temporary files and the OS should be on the local ...
1
vote
0
answers
185
views
AWS S3 CLI : Check Validate Inside Automatically?
We're using "s3api put-object" and explicitly checking Hash MD5 to make sure the file uploaded to S3 correctly (Unless the upload file is less than 5 GB)
aws s3api put-object --bucket ...
- 121
1
vote
1
answer
2k
views
AWS RDS "Maintenance Required"
I saw from the web interface that maintenance on my RDS postgresql instances is required, so I ran aws rds describe-pending-maintenance-actions, and it says that there's a "system-update" ...
- 658
1
vote
2
answers
2k
views
Reading SQL Server .sqlaudit Files Stored in S3
I set up an audit in SQL Server that saves audit logs (.sqlaudit files) to RDS, which we then copy over to S3 using a standard process. All of the documentation I can find on reading these audit files ...
2
votes
1
answer
3k
views
Connection to AWS database gives back "You may not specify an IPv4 CIDR for an existing referenced group id rule."
I try to connect to an AWS database instance from a client by following this tutorial:
https://medium.com/simplebackups/allow-a-remote-ip-to-connect-to-your-amazon-rds-mysql-instance-2d5074d601eb
I ...
1
vote
0
answers
329
views
Does AWS Aurora MySQL not support encrypted binary log replication with RDS MySQL master?
Trying to setup replication between an RDS MySQL (5.6) master with an Aurora MySQL (also 5.6) replica cluster in the same region via binary log replication for a work project. However, this doesn't ...
1
vote
2
answers
903
views
AWS Aurora Postgres 12 Setting Collation
I posted this on stack overflow as well, but didn't get much traction over there. If I am not able to get some thoughts on here, then I will probably open a support ticket with AWS.
I am trying to set ...
0
votes
1
answer
100
views
Moving entire SQL Server 2016 to a new location - best method
At my organization, we have a SQL Server (2016) running on an EC2 instance at Amazon on a Windows with SQL Server license. My boss realized we could save a ton of money by standing up a Windows-only ...
- 327
0
votes
0
answers
1k
views
Oracle Storage Full, How to release space?
i have and AWS RDS Oracle Database wich its storage is full, looks like the cause is SYSAUX tablespace with 10.2GB size
The largest object inside SYSAUX is: WRI$_ADV_OBJECTS (TABLE) checking the table ...
1
vote
1
answer
322
views
Intermittent SQLSTATE[HY000] [2003] Can't connect to MySQL server on 'server.here' (110)
I'm suddenly having intermittent 2003 connection errors and I can't find anything that has changed that can explain it. These servers have been running smoothly for years and suddenly two days ago ...
- 111
-1
votes
1
answer
468
views
How to temporarily disable foreign keys in Amazon RDS Aurora PostgreSQL? [duplicate]
Hi I am using DMS to load data from physical server to AWS Aurora PostgreSQL, but I am getting error for foreign key constraint.
I want to disable using any sp / function. Could you please help me?
1
vote
0
answers
41
views
Any thoughts on what crashed my database?
A few days ago, our database server, (see specs below), spiked to 4800% CPU usage. The database is about 100GB. For 3 hours we tried to figure out what the issue was. We finally figured out a feature ...
- 11
0
votes
2
answers
59
views
database changes are lost in the next session
We just move to Amazon RDS. Now all the CREATE, UPDATE, and INSERT commands are reverting in the next session. All the tables are InnoDB and query caching is on. What could be the problem?
- 139
1
vote
0
answers
271
views
Container is running beyond physical memory and killed on request (code 143) [closed]
I ran a query which involved a JOIN and an AVG operation:
SELECT
AVG(user_amount) AS average_user_per_game
FROM
(
SELECT
start_time
,end_time
,COUNT(DISTINCT ID) AS ...
- 111
0
votes
0
answers
52
views
Remote Nodes is not detecting in cluster validate configuration in sql server 2019
I am not a professional DBA. In my organization, we are trying to implement active active node always on clustering. We have 2 nodes, primary and secondary. After installing WSFC, and during ...
0
votes
1
answer
2k
views
RDS PostgreSQL -> aws_lambda.invoke error: 'Unable to connect to endpoint'
SELECT * FROM aws_lambda.invoke(
'arn:aws:lambda:us-west-2:123456789012:function:spark-ods-lambda-preprod-myETLTest',
'{"body": "Hello from Postgres!"}'::JSON,
'us-west-2',
...
- 51
0
votes
1
answer
692
views
Postgres Database flagged as STORAGE FULL whereas there's 45 Gb available
I will start this post by stating that I am not very experienced in database administration, please accept my apologies for any unclear explanation you may find below.
We have a replica postgresql ...
2
votes
1
answer
5k
views
How to connect Datagrip with AWS Athena using an assumed role?
I am able to make Athena queries through the AWS CLI. This requires me to assume a role first (I use awsume). The role requires MFA.
Now I want to configure Datagrip for this as well. However there ...
- 151
2
votes
3
answers
1k
views
Backup TO URL S3
Can I do a SQL backup to AWS S3 URL from an EC2 machine
I know that Azure side is possible but I do not know if it is possible AWS side
thank you for your advice
0
votes
0
answers
1k
views
AWS DMS migration not migrated Foreign keys, sequences and indexes got renamed
I have used DMS for AURORA PostgreSQL to PostgreSQL SCHEMA Migration. Migrate Data only type is used for DMS activity.
After a Full load of Data, all tables were created on the target endpoint with ...
0
votes
1
answer
933
views
delete obsolete archivelog
I have an oracle instance aws recently migrated to RDS aws
I just noticed that my database is set by default in archivelog mode and that the archivelog are not deleted
who has any idea how deleted the ...
0
votes
0
answers
184
views
Python / AWS RDS - 300 insert statements taking 10 minutes to complete
I'm using Python 3.7 with SQLAlchemy on Ubuntu 18.04. I have two MySQL Databases - a local instance hosted on bare metal and an Amazon RDS instance. Both are accessible from MySQL Workbench on my ...
- 101
0
votes
1
answer
208
views
MySQL Replication from On-Premises to AWS
I have an on-premises DB server running MySQL 5.5.37 with multiple databases.
I created database on the AWS RDS.
The idea is to set the replication from on-premises to the AWS, but for only one ...
- 111
0
votes
0
answers
136
views
Is offsite replication from MySQL AWS RDS with encrypted RDS storage possible? How?
AWS RDS for MySQL now supports encrypted storage backends. This encrypts the MySQL data and binlogs with a key within AWS's key management system, thus causing offsite replication issues because we ...
- 43
1
vote
1
answer
680
views
Clear PostgreSQL logs in AWS Ubuntu Server
I am trying to clear the logs of an postgresql under /var/log/postgresql/ but I am getting error message as below:
/var/log/postgresql$ sudo > postgresql-11-main.log
-bash: postgresql-11-main.log:...
- 113
1
vote
1
answer
2k
views
subquery must return onl one column in postgresql
I want to get the XML response after the stored procedure call in PostgreSQL. In the below query I was trying to get the output, but it throws an Error: subquery must return only one column. Can ...
0
votes
0
answers
105
views
Same query taking different execution times on equivalent RDS instances
The train table has around 60million records all of which have their updated_at timestamp in the year 2021. The schema of the table is as below.
Table "public.train"
...
- 131
0
votes
0
answers
41
views
Does it make sense to separate records in a document DB (e.g. DynamoDB) for better query performance?
I've recently started working with DynamoDB, and most of my experience is in traditional data warehousing (e.g. sql server). One common pattern which I saw when working with SQL server was that we ...
- 101
0
votes
1
answer
347
views
how to mongorestore mongoDB 4.2 to documentDB 4.0?
I need to do mongorestore from my mongodb 4.2 to DocumenDb 4.0 compatible i got the command from aws
which is this
mongorestore --ssl --host="abc.aws.com:27017" --username=user --password=...
- 69
1
vote
0
answers
171
views
SQL Always on Availability Groups Direct Seeding
Setting up a new SQL cluster and HA using SQL Availability Groups.
SQL 2019 STD 2 Node Multi Subnet Basic AG
Primary Node AWS
Secondary Node VMWare on premise
I am using the newer Automatic Seeding ...
- 431
2
votes
1
answer
145
views
QuasarDB cluster stuck in IOWAIT on AWS EC2 instances
I am experiencing a system freeze in IOWAIT on my QuasarDB cluster.
I am continuously ingesting data at 10 GBit/s using the QuasarDB batch API. After a couple of hours, I have all the qdb threads in ...
- 21
0
votes
0
answers
205
views
Querying for multiple values when an error message is present in AWS Timestream
I am using AWS Timestream so the DB structure is a bit different than normal but effectively it fairly similar to a normal structure.
What I need to do:
When an error message of a certain type (Ie ...
- 1
0
votes
1
answer
331
views
Keeping postgres connection behind load balancer alive during long-running query (vacuum)
Background:
I have a postgres instance behind a load balancer (AWS Classic Load Balancer). I want to access the database from another cluster through the load balancer and run queries. I cannot change ...
- 371
1
vote
0
answers
115
views
Seeding new data daily in an ever changing database
I've been working with a data team that imports data from suppliers daily into a MySQL DB. The data from the supplier needs to be worked on by them and formatted properly before we can use it in ...
- 111
-2
votes
2
answers
234
views
Cost of reading and sorting 10GB table at cloud providers [closed]
Say I have a 10 GB table and I need to read it and then sort it by some column. How much does such an operation cost at cloud providers, say AWS or Azure if the pricing is variable? I just need a very ...
- 817
-1
votes
1
answer
799
views
How to checkdb automatically in AWS RDS?
I'm pretty new to Amazon AWS and RDS. I have migrated my SQL Database to RDS and I have it up and running. I'm now looking to automate maintenance tasks. Anyone with experience with RDS can help me ...
- 103
2
votes
2
answers
7k
views
MySQL - Can I have quick queries in a table with 200 million rows?
THE PROBLEM
I have following trivial SELECT statement:
select * from `web_notifications` where `user_id` = 123456 order by `created_at` limit 500;
The query is pretty simple (no join statements, no ...
4
votes
1
answer
4k
views
RDS Postgres chicken-and-egg problem altering roles
I just created an RDS Postgres instance, the user postgres (and it's master password), and connected via psql as user postgres.
Then I ran the script generated from the current system using pg_dumpall ...
- 658
1
vote
1
answer
277
views
What are the risks of upgrading DB major version in replica and then promoting it as master?
Recently AWS decided to force update all MariaDB RDS under 10.3, my RDS is in MariaDB 10.1. We are trying to minimize the downtime and any risk of data out of sync.
Our plan is to create a replica and ...
- 142
1
vote
2
answers
733
views
How to enable Cloud Watch logs for AWS DMS task?
I am creating AWS DMS task for migrating data from GCP SQL server to AWS Aurora MySQL. How can I attach log group which I created?
I could able to enable cloud watch logs for the task but when I click ...
- 151
0
votes
2
answers
1k
views
How to ensure backups are successfully persisted to S3 when using Ola Hallengren solution on AWS EC2 Instances
Improving Existing Backup Approach By Using Ola Hallengren Backup Solution
Deploying a new backup approach to eliminate dependency on a vendor tool (Cloudberry) that I've leveraged before. In my ...
- 280
0
votes
1
answer
384
views
My Galera Cluster does not connect across AWS regions
I am building a MariaDB Galera Cluster, on AWS, EC2. Using MariaDB 10.5 and Galera 4.
I have two nodes in the California region, named galera_ca and galera_ca2, and a third node in Oregon name ...
- 111
0
votes
1
answer
1k
views
AWS Aurora Mysql different instance types for writer/reader
I have a db.r5.large for the master/writer.
I want to add some readers.
Do they need to be the same instance type as the master? Or can i add for example 3x db.t3.medium ?
- 101
3
votes
1
answer
5k
views
How to setup continuous DB replication from on-prem MS SQL server to PostgreSQL on AWS RDS without reconfiguring source machine?
I am looking for a way to set up continuous data replication between an on-prem Microsoft SQL Server to AWS RDS PostgreSQL. The source table is updated once every 24 hours. The challenge here is that ...
2
votes
0
answers
4k
views
DBeaver connection via SSH tunnel
Using DBeaver, I'm trying to connect to my AWS ec2 instance where I've installed a MariaDB database.
I've created a new connection as:
Connection: AWS / Maria DB
Server Host: localhost (because I'm ...
- 121
0
votes
3
answers
3k
views
SQL Server RDS Database Restore from S3 stuck in Restore Mode
Summary:
We are doing a restore of a SQL Server database from S3 on an RDS Instance. And all though the job will say the restore is complete when we go to access the database it's stuck in "...
- 55
5
votes
1
answer
2k
views
Swap Usage but high Freeable memory
We have an AWS RDS with a MySQL database. Buffer Pool size is only 144GB out of the total available 244GB RAM. However, even at such a low RAM config, we see Swap Usage happening. Does anyone have any ...
- 223
0
votes
1
answer
666
views
Problem with mysql/data folder is too big and took the whole place
for the first time I am faced with such a problem that has puzzled everyone.
I have all hosting busy (AWS lightsail) 78GB and after a few analyzes I realized that 90% of the memory is occupied in the ...
- 11