Questions tagged [aws]
Amazon Web Services.
472
questions
0
votes
1
answer
39
views
New rows are locked by select on PostgreSQL replica
have a strange trouble with getting new rows on replica on PostgreSQL on AWS.
I have master database on AWS with constant flow inserts/updates not with big rate.
And these changes are replicated ...
- 3
1
vote
0
answers
22
views
Postgres: copy from 2 different DBs for identical tables
I have two identical DBs, under the same port and host. (they both postgres DBs and on AWS) I want to create a script\write any query (even though it is not possible to create a query that use data ...
- 21
0
votes
0
answers
20
views
Is there a way to export AWS RDS postgres snapshot to on-premise server?
I am working towards exporting a snapshot of RDS Postgres to S3 and pull it to on-premise server to cater to a requirement where I can run the database locally. Has anyone tried this before? and can ...
0
votes
0
answers
18
views
Describe table from Athena data source in DataGrip fails
Whenever I attempt to Describe an AWS Athena table in DataGrip it fails. The error states that is is a permissions error but I am fairly certain it is not. In fact, I can describe tables using ...
2
votes
2
answers
114
views
Operational Error on MySQL AWS RDS Instance
The heading pretty much describes it all, here are some more information on the Amazon RDS I am using.
Engine: MySQL Community 8.0.28
Class db.r6g.large
Multi-AZ no. The data volume is very small, ...
- 73
2
votes
1
answer
67
views
AWS to Azure Degraded Performance With "Hash Match (Aggregate) Step"/"Temporary Disk" TempDB Performance Issue
Ok, so I have a series of AWS VM's, mostly using the z1d.3xlarge server class with built in SQL Server 2019, these servers each contain a NVMe of various size. (Not surprisingly) I've placed my SQL ...
- 205
0
votes
0
answers
21
views
Where clause in which the value of column2 changes based on column1
So I have a table which has
State1:
id
column1
column2
1
myboolvalue
false
1
myecogvalue
0
1
myintvalue
1700
State2
id
column1
column2
1
myboolvalue
true
1
myecogvalue
0
1
myintvalue
1700
In ...
0
votes
0
answers
22
views
RDS Postgres v14.4 CPU utilization not going below 5% when idle
I have an RDS Postgres instance on a t3.small and am trying to drive down the CPU usage of my clients connecting to it (I have a few API services connected).
I've stopped all my services connecting to ...
- 101
0
votes
0
answers
23
views
How to setup replication between a new aws rds instance and an existing aws rds instance using rds_set_external_master
I have an existing aws rds master (Source Master) and replica instance (Source Replica).
I have created a new aws rds instance (Target Instance).
Goal:
I would like to setup replication so the Target ...
- 109
0
votes
0
answers
20
views
Does AWS RDS Oracle support SSL between Standby and Read Replicas?
Does AWS RDS Oracle support TLS 1.2 between cluster members e.g. Standby (Multi-AZ deployment) and Read Replicas? I cant seem to find any documentation to confirm this.
- 101
0
votes
0
answers
27
views
Finding distance between 2 geospatial location in aws neptune with opencypher query language
I am working with amazon neptune and using openCypher query language and doing some stuff using notebook. I have latitude and longitude of 2 points and I want to calculate the distance between these 2 ...
1
vote
0
answers
11
views
How to encrypt a production unencrypted RDS Aurora MySQL 8 cluster
I have a running RDS Aurora MySQL 8.0.23 cluster running in production. The database is unencrypted and I need to enable encryption for it. As far as I understand, this is not possible to do directly. ...
- 111
1
vote
0
answers
38
views
Redshift User password change permission denied
Redshift user not able to change its own password getting below error:
ALTER USER or_user WITH PASSWORD 'md5<hash>';
ERROR:permission denied
Also tried granting 'ALTER USER' privileges using ...
- 11
1
vote
0
answers
48
views
Restart a Running Sum after Static Threshold
I've been stuck trying to solve the following problem for some time: I essentially want to restart the rolling sum if the rolling sum exceeds 3600, else continue summing. I've sort of have it working ...
0
votes
0
answers
20
views
Can we use AWS SCT (Schema Conversion Tool) with AWS DMS for ETL and migrating data from MySQL to Amazon Aurora for MySQL?
I want to migrate data from on-premises MySQL/MySQL RDS to amazon Aurora for MySQL. While migrating, I need to convert schema as well as do some ETL to get required schema and data.
In ETL I need to ...
- 21
1
vote
0
answers
20
views
Best practice to store subitems
I am going to create my first Dynamodb database table. I have 3 options in my mind:
1. Create a long table where every subitem is a new row.
Example:
user John | item 1 | 2021
user John | item 2 | ...
- 111
3
votes
2
answers
151
views
Need to keep each binlog files to 24 hrs
We have a requirement that we need to keep each binlog files(like mysql-bin-changelog.025249, mysql-bin-changelog.025250, mysql-bin-changelog.025251....) of MYSQL RDS to minimum 24 hrs.
However, i see ...
- 83
0
votes
0
answers
112
views
Getting "The provided token has expired" error message in AWS DMS
I have set up a DMS task with the following setup:
RDS MYSQL as source endpoint
and REDSHIFT as target endpoint with "full load and CDC".
setup was working fine for couple of days then ...
- 83
1
vote
0
answers
62
views
Backup data from RDS to S3 and delete from RDS once copied to S3
We have a requirement werein we have to backup data from RDS(MYSQL and Postgres) to S3 incrementally and to delete the already copied data from RDS. I thought of using aws data-pipeline for this, ...
- 83
0
votes
0
answers
26
views
Need to create DMS task thru AWS CLI
I have to migrate from MYSQL RDS to REDSHIFT thru DMS. There are 400+ tables in my source MYSQL RDS instance.
i have to create 4 DMS task with 100 tables each thru AWS CLI since thru console it will ...
- 83
0
votes
0
answers
24
views
Using AWS DMS for EC2 MariaDB instance to MariaDB RDS instance
After combing through AWS documentation, I am still unclear if DMS supports migration from EC2 instances running latest mariadb version to RDS mariadb instance?I see that MySQL is supported, but am I ...
- 95
0
votes
0
answers
119
views
Getting '.....successive recovery failures' error in DMS while using postgres read replica as source
I'm getting below error in AWS DMS while migrating from Postgres read replica to Redshift. I'm using full load only.The DMS task will run for few mins and some tables will be loaded to target and then ...
- 83
0
votes
0
answers
91
views
Getting ODBC general error in DMS while using postgres read replica as source
I'm getting below error in AWS DMS while migrating from Postgres read replica to Redshift. I'm using full load only.The DMS task will run for few mins and some tables will be loaded to target and then ...
- 83
0
votes
0
answers
196
views
How to connect PgAdmin4 to postgresql 9.5 database process running on AWS EC2 server
I have a postgres database running on a linux EC2 instance. I want to view the data inside the tables, and currently the only way I can do that is via command line on the host server by:
sudo su ...
- 1
3
votes
1
answer
158
views
Getting Error 1236 (Could not find first log file name in binary log index file) reading binlog in AWS DMS
I have created two DMS task (full load + CDC) with RDS MYSQL 'Read' replica as source endpoint and REDSHIFT as target endpoint. For both the task source and target endpoint is same. One task with only ...
- 83
0
votes
0
answers
14
views
Secure way for a public website to post requests with an AWS rest api/database
I am quite new to using rest type apis, and I am currently trying to develop a public website that I want to be able to post data to a backend (possibly via an intermediate DynamoDB on AWS or similar ...
- 101
3
votes
3
answers
53
views
How can I make a simple left join with null condition run faster?
I've got two very simple tables
CREATE TABLE `track` (
`id` char(36) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
`number` int NOT NULL DEFAULT '-1',
`title` varchar(255) CHARACTER SET ...
- 143
0
votes
1
answer
107
views
AWS RDS database uses more space after pg_dump/pg_restore
I've got an RDS Postgresql 12 instance that allocated 11,000,000 MB of io1 disk space, and there's 7,300,000 MB of Free Storage Space.
That means the database is only using 11TB - 7.3TB = 3.7TB of ...
- 559
0
votes
0
answers
53
views
Redshift from dc2 to ra3 experiences
I'm currently running a 3 x dc2.8xlarge Redshift server, it's all good but I'm getting out of space (7.7TB available).
So I have two options to maintain the current price (approx 10k/month):
4 x ra3....
- 103
1
vote
1
answer
93
views
How can I load a custom format dump from pg_dump into an RDS while ignoring constraint checks during the import?
I have a number of constraints:
I have binary custom format dump from Heroku that I need to load via pg_restore.
I need to load this onto an AWS RDS instance, where I don't have full superuser (only ...
- 113
1
vote
1
answer
276
views
How to disable binlog_format in AWS RDS?
I created RDS on AWS and created MySQL v5.7 instance. I tried to find the binlog_format settings in parameter group but it seems to me that there are only three possible settings - like the screenshot ...
- 115
0
votes
0
answers
61
views
How can I speed up INSERT INTO ... SELECT ... ORDER BY queries?
I recently learned that it is advisable to have deterministic queries when using a replication instance. Therefore, we started adding ORDER BY to our INSERT INTO (...) SELECT (...) FROM X statements. ...
- 111
0
votes
0
answers
64
views
Unable to upload an excel file to s3 bucket. getting error
My method for uploading a file to s3bucket is as:
upload_image_v1: async (files) => {
// console.log("files", files);
let res_promises = files.map(
(file) =>
new ...
- 11
0
votes
1
answer
102
views
Backups cannot be restored on AWS RDS - View creation problem. Access denied; you need (at least one of) the SUPER, SET USER privilege(s) for this
The environment is MariaDB 10.6.8, via Amazon RDS. We have all 'admin' rights but not SUPER on AWS databases.
I have a view in an application that has it's data stored in a single schema. We use non-...
- 101
1
vote
0
answers
58
views
aws rds password change triggers updates to which fields in mysql.user?
I'm using AWS RDS with MySql (5.7.mysql_aurora.2.10.2)
Due to internal policy I updated db password via AWS RDS modify DB instance menu.
What I can't grasp is, when I updated pw for dev db instance, ...
- 33
0
votes
0
answers
18
views
Is there a difference in performance loading different datatypes
Context: loading (COPY) a high volume of data to a warehouse (Redshift specifically however curious about all)
Question: Is there a performance difference in the different datatypes that I am loading? ...
0
votes
0
answers
108
views
AWS RDS: How do I enable connection for specific IP address?
I've been managing a site for over four years, and initially what I did to allow access to the database for some IP address was just editing an existing Security Group called specificip
This is what I ...
- 101
0
votes
0
answers
48
views
AWS RDS PostgreSQL suddenly consumed all free space
0
I am Migrating table from On-Prem to AWS using AWS DMS tasks.
My On-Prem SQL server total DB's sizes are below 1TB
But my RDS storage(PostgreSQL) is taking more space 8TB. still it is increasing. I ...
- 1
0
votes
0
answers
53
views
Can I generate the commands/script to create an RDS Instance from an existing one?
I have a few development and test RDS instances that I need to drop, but would like to be able to recreate them again if so needed (with the same VPC, Options, etc...).
Is there a way to, given an ...
- 133
0
votes
0
answers
304
views
Evaluating AWS RDS Proxy latency overhead
Evaluating some benchmark tests I noticed that there is latency overhead in the transactions to the database using the service provided by AWS: RDS Proxy.
The tests were performed against a Postgres ...
- 101
0
votes
0
answers
14
views
Are AWS Lambda functions actually EC2 instances under the hood?
I’ve heard that AWS security groups are essentially firewalls that can be applied to EC2 instances. Since you can apply security groups to many things in AWS, including Lambdas, I’m wondering if many ...
- 171
0
votes
2
answers
135
views
Is it possible to fetch AWS RDS MySQL transaction logs?
I have AWS RDS MySQL instance, without general query logs.
Is it possible to get all transaction logs? I understand that MySQL binlog retention time is very short, but obviously, AWS RDS keeps all ...
0
votes
0
answers
168
views
Fastest way to migrate largish (15 TB) PostgreSQL 11 instance to Aurora 14 in AWS
We have an AWS RDS PostgreSQL 11 cluster (currently version 11.13), in which we have a few databases and some 1500 tables. The RDS cluster is of size "db.r5.xlarge", with 4 vCPUs, 32 GB RAM ...
- 13
0
votes
0
answers
51
views
Migrating AWS RDS SQL Server Express to Azure SQL DB
I have a task to migrate an AWS RDS SQL Server Express instance to an Azure SQL Database. I already assessed the RDS db using Data Migration Assessment Tool and there are not compatibility issues, ...
- 109
0
votes
1
answer
69
views
AWS RDS MySQL limit connection
We have multiple databases in a single RDS instance. Because of any reason if we have load in 1 database it stops all the databases.
I'm not able to move any of the database because of following ...
- 115
0
votes
0
answers
27
views
Why is a user randomly created in AWS RDS even though the permissions from any host (%) exist?
I have a MySQL database in RDS. There are separate users which are allowed to connect from anywhere. To be more precise, using MySQL Workbench there are different users listed in "Users and ...
- 1
0
votes
0
answers
206
views
How to enable replication filters in AWS Aurora 2 (5.7.mysql_aurora.2.10.2)?
I enabled binary log replication in Aurora to replicate from one cluster to another. I went to use the CHANGE REPLICATION FILTER command but get a 1227 error saying I must have the SUPER privilege, ...
1
vote
0
answers
185
views
How to debug a huge MySQL replication lag?
We have an AWS RDS and a corresponding Replica. Once or twice a month the replication lags and does not recover from its own. Our "solution" is to create a new replica and delete the old one....
- 111
1
vote
1
answer
165
views
dsbulk unload fails on a specific row after running for nearly 2 minutes
We would like to export all data from a Cassandra Keyspace table to CSV with dsbulk.
dsbulk unload -k KEYSPACE -t TABLE -f ./application.conf > export_TABLE.csv
If we do so, it completes with 1 ...
0
votes
0
answers
125
views
Why does my query take longer when Aurora Serverless is under load?
I have a Aurora Serverless Postgres database (8 to max ACU). I'm running some stress testing against it with about 500 active connections over a period of about 20 minutes.
During this time period I'...
- 678