Questions tagged [python]

Python is a powerful and ubiquitous language. When using a Python variant or library (e.g. Pandas, SQLAlchemy, pg8000, psycopg2), please include it in the tags. Questions using this tag should restrict themselves to interactions directly with the database tier. General python scripting questions should be directed to [StackOverflow](https://stackoverflow.com/questions/tagged/python)

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views

Pymongo: Query data from specific wiredtiger (.wt) files in mongodb

I am new in pymongo. I have some wiredtiger(.wt) available in mongodb as below. I wanted to know what data are being stored in each of the following (.wt) file. All these files are belongs to one ...
0
votes
0answers
6 views

New dataframe for different groups

I have a dataframe with a column 'Country'. Every country has multiple records. I want to write a function in python using pandas so that for every country it can return a seperate dataframe. Having ...
0
votes
1answer
16 views

Speed up AWS PostgreSQL insertion via Python script

I have 120,000 csv inside my AWS EC2 instance, each containing 120,000 rows. I can't insert each of them as is into my AWS RDS postgresql DB, that will be 120,000^2 = 14,400,000,000 records. Each csv ...
1
vote
1answer
40 views

How to see unclosed PostgreSQL connection while using psycopg2? [closed]

I run python code: con = psycopg2.connect(some_dsn) cur = con.cursor() cur.execute(some_sql) res = cur.fetchone() #con.close() <-- connection is not closed Then I try: ps auxf | grep postgres ...
1
vote
0answers
17 views

How to migrate data from a large amount of csv files with time x space measurements to a database that needs to be available for analysis [closed]

I have made a simulation experiment to obtain time series measurements at different points along a power line. These measurements were done for different types of power lines in different test ...
1
vote
1answer
31 views

Storing many pandas DataFrames in SQLite with metadata

I need some help with designing a database. My aim is to persistently store a number of pandas DataFrames in a searchable way, and from what I've read SQLite is perfect for this task. Each DataFrame ...
0
votes
0answers
17 views

Populate mysql db to Shotgun server using Python

I've an internal mysql database. On each creation, modification or deletion of the content of this db, I need to run a python script to replicate those information into another software web server. ...
0
votes
2answers
25 views

inserting multiple values into SQL with %

This works: cursor = conn.execute("""Select FAMILY,F_NAME from birthday where B_MONTH = '%s' """ % (currentMonth)) I cannot seem to expand it to work using an AND clause to also include day, I get a ...
0
votes
0answers
86 views

Error: duplicate key value violates unique constraint, but the column does not have a unique constraint? postgreSQL

By the way, I'm trying to automate the below using python, but I've isolated the SQL code and tried it by itself on pgAdmin III, and it still doesn't work. So I have a table called tableName, and I ...
0
votes
3answers
17 views

Overwriting MySQL database to only store 1 month of data

We are logging data on hardware with only small memory (short on disk storage) only 4GB. We only require the data to be stored for 1 month and then be over written in a way that it overrides older ...
0
votes
1answer
27 views

Generated DDL doesn't match with CREATE TABLE statement

I'm using MySQL with InnoDB engine. I create a new table that contains a foreign key, eg: CREATE TABLE rooms ( id INTEGER NOT NULL AUTO_INCREMENT, my_id VARCHAR(15), house_id INTEGER, ...
-4
votes
1answer
75 views

how to get all the table existing in a database [closed]

import pandas as pd import sqlite3 conn = sqlite3.connect('Db-IMDB.db') imdb = pd.read_sql_query('''SELECT table_name FROM information_schema.tables''',conn) but it is giving error: DatabaseError: ...
0
votes
1answer
26 views

MongoDB | Problem inserting embedded documents into document

I am trying to insert data in a nested format. A snippet of my models (there are many, many more): class LifetimeMode(db.EmbeddedDocument): meta = {'collection': 'lifetime_mode'} career = ...
0
votes
2answers
212 views

Run SQL query with python in SQL Server raises error: SqlSatelliteCall error: Unsupported input data type in column

I am trying to run Python in SQL Server and want to define a variable which contains the result of a simple select query. exec sp_execute_external_script @language = N'python', @script = N' import ...
0
votes
0answers
124 views

SSHtunnelForwarder to Linux Box, then connect to Postgres RDS (AWS)

I'm trying to write a python script to update DDL in a Aurora RDS (postgres) on AWS. In Order to do this, I need to SSHTunnel to a linux box that sits at the edge of the VPC, then do another jump to ...
-2
votes
2answers
49 views

Python - Best database for exponentially growing database? [closed]

So I am making a Discord (social media) Bot in Python using discord.py-rewrite, and I need to save informations of lots of servers and users (such as how much money they have, what items thay have…). ...
0
votes
0answers
18 views

Neo4j: Connection closed after few minutes: defunct connection Address (Python)

I have a server (hosted in Azure) in which the connection is opened (python): self._driver: Driver = GraphDatabase.driver(credentials["uri"], auth=(credentials["user"], credentials["password"]), ...
0
votes
1answer
26 views

Inserting records into Mongodb using Pymongo is using up my RAM incrementally and is eventually exhausting all memory

I am writing Python code that parses through a huge XML document of size 17GB(Wikipedia collection) and inserts each page title along with the links in that page (storing as a list) present on the ...
0
votes
1answer
23 views

Is it necessary to connect to a postgresql database in order to create a new server?

I am writing a script in python 3.7, using psycopg2 to set up a new database from scratch. In all the tutorials I have seen, using psycopg2 starts by connecting to a database and getting a cursor ...
0
votes
1answer
135 views

CSV (UTF-8) to Postgres database (UTF-8), doesn't show äöå characters

I have csv-file that is UTF-8 encoded. I'm using python (psycopg2) to copy it into a Postgres database. My Postgres database is using UTF-8. However I have problems to show Ä, Ö and Å characters. If ...
1
vote
2answers
470 views

psycopg2 connection with multihosts [closed]

I have 3 db servers on docker containers, and i used to connect like this from my python app: psycopg2.connect(host="db-0", user="user", password="password", database="data") This works,but if i add ...
0
votes
0answers
18 views

How to increase idle time of “LaunchPad” for SQL Server external scripts?

sp_execute_external_script can take a while to initialize when not run for a short time. How do I set the idle time on the controller Launchpad process to keep it from shutting down too soon? Note I ...
0
votes
1answer
779 views

How to Bulk Insert sqlalchemy subclasses

I am trying to bulk insert SQL-Alchemy Subclasses into the parent table and their respective tables ie fruits tables -> Apple Table and so I insert a table of APPLE and it will insert both the row ...
1
vote
0answers
175 views

operand should contain 1 column Use Variables in SQL statement in Python

Trying to insert four columns and multiple rows with variables. The table in mysql has 8 columns in total. Four columns to insert is as following: report_type= tuple(("ada","asda","frfe","adaaa","...
0
votes
1answer
618 views

Psycopg2 Errors on SQL statement when trying to copy data from CSV file into PostgreSQL database

I am not a developer or PostgreSQL DB admin, so this may be basic questions. Logistics: Windows 10 server / pgAdmin 4 / Postgres 10 / Python 2.7.13 I'm using a python script to ingest external data, ...
0
votes
0answers
16 views

Reconstruct folder hierarchy in JSON format from MySQL using Python

I have a table with a structure like this: | itemID | parentFolderID | folderName 1 0 folderA 2 1 folderB 3 1 ...
0
votes
0answers
91 views

which python version is required to install plpython3u of which postgresql version?

I want to know if there is a reference table or trick to find out which python version is required by plpython3u of which version of postgresql . example python 3.4 is required for plpython3u of ...
1
vote
0answers
16 views

Custom Export from power BI service

I'm trying to make a custom button that exports filtered tables to a csv and downloads them to my computer. I am aware that you can do this with the '...' button but this functionality is for other ...
0
votes
0answers
22 views

Changing Cube Max Dimension

thanks in advance for any help. Python/postgres noob here. I need to change the default dimension for cubes in postgres. I tried to download source code, fixing the cubedata.h file to change max ...
0
votes
1answer
89 views

MySQL: columns with mixed character encodings, and finding columns which have multibyte data

I have some rather large CSV files that I am loading into my MySQL 5.7 database. The files are several gigabytes in size, several million lines long, and have large column widths that must be used in ...
0
votes
0answers
435 views

psycopg2.pool.PoolError: trying to put unkeyed connection

I'm trying to get connection pooling to work in my Python Flask web application. The code I am using is as follows: from contextlib import contextmanager import logging from psycopg2 import pool, ...
3
votes
0answers
103 views

TCP Resets when using psycopg2

I'm trying to do a simple health check for keepalived and other services via a python script and psycopg2 library. All seems to be alright, until I close the connection, at which point a packet with ...
1
vote
0answers
64 views

Designing a table with precalculated team values for fast read queries

My version of PostgreSQL is 10.7. As it's something I can't calculate on the fly due to the amount of times it might be used, I decided to store all the 11 million combinations and their result on a ...
0
votes
1answer
599 views

Comparing multiple tables in Sqlite 3 using python:

I am quite new to SQLITE3 as well as python. I am working on a project where I have one database with about 20 tables inside of it. One table is for user input and the other tables are pre-loaded with ...
0
votes
1answer
305 views

django_session table killing my MySQL server and django app

I need your help I hav a Django app that is about 7 years old and have been degrading now..every time the website goes down and back up and the issue is with mysql server Usually i need to just ...
1
vote
1answer
130 views

No output when use pymongo watch()

I try to access real-time data changes with pymongo.mongo_client.MongoClient.watch(), code example (Python 3.5.3, pymongo==3.7.2): from pymongo import MongoClient DB_CON_STR = "mongodb://10.10.10....
0
votes
1answer
192 views

write csv file only after SQL table update, insert, delete

Objective: Write csv file of updated data from sql table only after SQL table update, insert, delete. I've read from the community that I should not do triggers to execute python. I should not run ...
3
votes
1answer
319 views

When I run mssql-cli, I get “No module named mssqlcli”

I just installed mssql-cli with pip, pip3 install mssql-cli Then I try to run mssql-cli and I get $ mssql-cli /usr/bin/python: No module named mssqlcli
1
vote
1answer
38 views

How to point SQL Server Python to a specific Python install?

SQL Server 2017 installs an instance of Python to "C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\PYTHON_SERVICES" (documentation link) that it uses for its Python calls. This is installed ...
0
votes
1answer
256 views

Find and Insert Missing data in Mongodb Collection

I want to write python3 code to check and insert missing data. My MongoDB collection documents have one field named "height" which is BTC block number. I want to traverse in a range start to the ...
1
vote
0answers
279 views

PGPool II + Django w/ psycopg2 load balancing

I've a pool of applications running django 1.6 with psycopg2 on top of a PGPool II with two backends PostgreSQL servers. But all queries (read and/or write) are going to master PostgreSQL even SELECT. ...
1
vote
2answers
307 views

MySQL HA_ERR_LOCK_WAIT_TIMEOUT ignoring innodb_lock_wait_timeout (deleting most of a table)

I am issuing some long-running delete queries to some databases, and I am getting the following error: Last_Error: Could not execute Write_rows event on table DATABASE.TABLENAME; Lock wait timeout ...
1
vote
1answer
195 views

MS SQL 2017 Batch insert performance degradation

This one's a bit strange so bear with me: I'm seeing extreme insert performance degradation after an insert was interrupted in a dirty fashion (the transaction was not committed and the connection ...
0
votes
1answer
247 views

Python SQLite3 horizontal partitioning by string column

I have a 30 GB SQLite 3 database with a table containing 25 million rows. I want to group the table by a string column and create a new table for every group of records, named after the string and ...
0
votes
1answer
1k views

Query using $gte and New Date() working in MongoDB shell but not through python pymongo

When I run the following in the mongodb shell, it returns the correct number of records: (searching for the matching "SourceName" within the last 8 days) mongos> db.fdl.count({ SourceName : "SRC1"...
1
vote
0answers
153 views

Cassandra pool warning displaying continuously

I am using the Cassandra driver for python in Spyder. I am trying to fetch some data from Cassandra table. Here is my code: from cassandra.cluster import Cluster cluster=Cluster(['some_ip']) session=...
0
votes
1answer
415 views

get row/value by primary key number

I'm writing my own python wrapper on top of official mysqldb connector. I'm trying to design method that would dynamically extract row data knowing KEY ID. SELECT * FROM cameras; +----------+--------...
1
vote
0answers
66 views

MySQL Shell Python Client library

Is there a python client library available to interact with MySQL Admin API and to do similar stuff that MySQL Shell does. From the documentation of MySQL Shell, I could see that there is a --py mode ...
2
votes
2answers
79 views

Allow only one reader to a table

I have a simple Postgres table which port numbers to listen on. If one port is used it can't be used again. However due to multiple worker/threads they many times read simultaneously. I want that ...
0
votes
1answer
72 views

MySQL 'generator' not closing connections

I have the following function which is leaving connections open but I can't figure out why. The Cursor class creates a python-generator type function, allowing me to iterate over millions of rows. ...