Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [encoding]

The tag has no usage guidance.

0
votes
1answer
39 views

How can I search for a hex string in oracle?

There's a database record that is incorrect. The name field displays like this in a browser ( this is incorrect ): Thcˇodore And when I look at the record via SQL results, I see: Thc\xCB\x87odore ...
4
votes
2answers
81 views

Byte ordering for multibyte characters in SQL Server versus Oracle

I am currently in the process of migrating data from Oracle to SQL Server and I'm encountering an issue trying to validate the data post-migration. Environment Details: Oracle 12 - AL32UTF8 ...
1
vote
2answers
160 views

Bulk insert not retaining special chars of UTF-8 Encoded txt file Sql Server 2008

I have a stored procedure that bulk imports a text file and inserts it into my database. CREATE TABLE DBO.TEMP_STORE ( ID nvarchar(max), [MONTH] nvarchar(max), [...
4
votes
3answers
318 views

Unicode storage of \u202b RLE and \u202c PDE in a Unicode-aware database?

I'm building a new product for toponyms and in it the Arabic shows kinda like this: ^IArabic^I<202b>ﺰﻤﺑﺎﺑﻮﻳ<202c>^I<202b>ﺞﻫﻭﺮﻳﺓ ﺰﻤﺑﺎﺑﻮﻳ<202c>$ Actually not quite. This is a ...
1
vote
1answer
57 views

How to insert a Unicode character verbose into a varchar DB?

I need to insert this character '●' into a VARCHAR column of a MSSQL database with collation set as SQL_Latin1_General_CP1_CI_AS (or at least mock what my Python + Windows MSSQL Driver might have done)...
3
votes
2answers
883 views

SQL Server 2019 UTF-8 Support Benefits

I'm already quite comfortable with using COMPRESS() and DECOMPRESS() in an internal forum software for our company (Currently in SQL Server 2017), but trying to make the database as efficient as ...
0
votes
1answer
47 views

Users running ANSI scripts are having problems with special characers [closed]

Long story short. How can I fix users environments, to make them run our scripts using ANSI encode? The problem is, we send them scripts to run on their databases using ANSI encode. But some of ...
6
votes
1answer
1k views

Error starting SQL Server 2017 service. Error Code 3417

I have SQL Server 2017 installed on my computer. This is what SELECT @@VERSION returns: Microsoft SQL Server 2017 (RTM-GDR) (KB4293803) - 14.0.2002.14 (X64) Jul 21 2018 07:47:45 Copyright (C) ...
0
votes
1answer
155 views

How to configure DBeaver to save files without a BOM for UTF files?

I am looking for the equivalent setting as the PGADMIN setting Write BOM for UTF files which will cause saved files to be saved with/without the BOM. I've found settings in DBeaver when exporting ...
1
vote
2answers
301 views

Wrong sort order using utf8_unicode_ci and swedish characters like ö

I am confused about what encoding and collation I should use so my queries returns the correct sort order. Right now my encoding is UTF-8 Unicode and collation utf8_unicode_ci for both firstname and ...
1
vote
1answer
18 views

Table/Column Character Encoding and Indexes

Does table/column character encoding have any impact on how the MySQL optimizer chooses join indexes? I have a table that have the same columns and indexes on two servers, one server does a eq_ref ...
4
votes
1answer
2k views

Why do I get incorrect characters when decoding a Base64 string to NVARCHAR in SQL Server?

I've been looking at how to decode Base64 with SQL Server and after searching around many solutions online (some from here) seem to be based upon this sort of method. SELECT CAST(CAST('...
1
vote
1answer
35 views

Convert Character Encoding, and Check Encoding for Issues

I'm planning to convert a table from utf8 (_general_ci) to utf8mb4 (utf8mb4_unicode_ci). I want to make sure I don't have any malformed data after the conversion though so I was planning to duplicate ...
4
votes
1answer
683 views

Postgres - Encoding, Collation and CType

List of databases Name | Owner | Encoding | Collate | Ctype | -------------------------+----------+----------+---------------------------...
3
votes
2answers
425 views

Use non-printable characters in SQL Server BCP xml format file as TERMINATOR

I have a flat-text data file containing records which fields are separated by the non printable character "File Separator" (0x1c). I am trying to use SQL Server's bcp utility to load this data into my ...
2
votes
3answers
2k views

NLS_LANG and NLS_CHARACTERSET

I thought I read that NLS_CHARACTERSET has the encoding of the database and NLS_LANG the encoding of the client. Is that correct? Does that mean that the two encodings can be different? In some ...
0
votes
1answer
60 views

Change of NLS_LANG

Until not long ago, the NLS_CHARACTERSET and the NLS_LANG were both ISO88591. We deliver to production some scripts encoded in ISO88591. Since they changed their NLS_LANG in utf8, they have a problem ...
1
vote
1answer
97 views

Is there any string compaction option for PostgreSQL?

I have a table like this: CREATE TABLE BLOG( CATEGORY CHAR(32) ); Category is a column, which stored enum values. For example, I have 1 millions of rows and 10 unique values of Category. Let'...
2
votes
2answers
268 views

DB2 on Windows: Fix encoding in db2cmd and for JDBC

I'm running a DB2 v10.5 FP 7 on a Windows Server 2013 and there is a problem with the encoding setup: The windows is set to use German(Switzerland) as region setting and input language. chcp: Active ...
1
vote
2answers
1k views

Character Conversion Issue Importing From CSV File

After doing a load of CSV files, there are various words that are "written" to the database incorrectly. Some examples: Diã¡ria should be Diária Crã©dito should be Crédito Ligaã§ãµes should be ...
5
votes
2answers
2k views

SQL Server Bulk Insert properly interprets some Unicode characters but not others?

For some reason the MS SQL Server 2016 bulk insert misinterprets/translates Unicode characters: C9 (É) into 2B (+) A1 (¡) into ED (í) A0 ( ) into E1 (á) AE (®) into AB («) CB (Ë) into 2D (-) D1 (...
1
vote
0answers
66 views

Merge Replication Business Logic Handler Incorrect Encoding when connecting to SQL Server

We have a SQL Server Merge Replication configuration that uses a business logic handler to do conflict resolution on a datetimeoffset column. All we're doing is a copy of the Subscriber or Publisher ...
3
votes
0answers
601 views

Why is my UTF-8 document raising UTF-8 encoding errors in Azure Data Lake Analytics?

I have a document that was compressed in gunzip from a unknown source system. It was downloaded and decompressed using a 7zip console application. The document is a CSV file that appears to be encoded ...
1
vote
1answer
853 views

How to Create a PostgreSQL database with different Encoding?

I want to create a New Database that will store data in German(i.e Text with accents etc). Hence I tried to create it using the following SQL: CREATE DATABASE testdb WITH ENCODING='LATIN1' ...
1
vote
1answer
115 views

What is the equivalent of the SQL Server CHAR function?

SQL Server has CHAR ( integer_expression ), so I can do SELECT CHAR(65) What is the equivalent on Postgresql? I have tried SELECT E'\0x65'; but I get ERROR: invalid byte sequence for encoding "...
4
votes
1answer
401 views

Collate issues with wrong characters

Well, the problem is well known, but I'm looking for a smarter solution if there's one. For some reason the system doesn't recognize some characters and I can't compare the columns Here is an ...
2
votes
2answers
2k views

Remove characters from PostgreSQL database causing encoding errors

I have a PostgreSQL 10 database that uses WIN1252 encoding. One of my columns has values that cause conversion errors when running a select from pgAdmin 4: SELECT myfield FROM mydb.myschema."MYTABLE"...
0
votes
1answer
158 views

UTF-8: showing correct in database, but not in webpage

When i upload data from php post forums "हिन्दी" language, changed into "हरिà" in database but looking correct in webpage, and when i upload data through phpmyAdmin from csv file "हिन्दी" ...
3
votes
1answer
3k views

How to be sure that a SQL Script is running with ANSI encoding?

Maybe this is a stupid question or I'm asking it in the wrong way. How can I be sure that a script ( with thousands of lines ) is running with ANSI encoding? Let's say we created a script using ...
1
vote
1answer
71 views

Invalid characters in xml [closed]

I have just downloaded a large xml file and most of characters appear as "�". I changed the encoding but there was no improvement. Anyone knows what this might be due to? One fishy thing: the file ...
9
votes
1answer
5k views

Encoding issue with SQL Server VARCHAR column retrieved in Python

We recently had an issue with encoding related to a field that's being stored as a varchar(120) in SQL Server. In SSMS, the varchar appears as: "Who Killed JonBen‚t?" However, when it's brought ...
4
votes
1answer
1k views

How do I list all available encoding types available in postgreSQL?

SELECT datname, pg_encoding_to_char(encoding) FROM pg_database; …lists all the databases, each with its encoding type. However, I am trying to find out what are all the encoding types available in ...
1
vote
1answer
1k views

Saving into a VARCHAR(MAX) throwing error in some scenarios

We are noticing an error like this: InterfaceError: ('HY104', 0, '[Microsoft][SQL Server Native Client 11.0] Invalid precision value', 10226) It happens when we insert a certain length of characters ...
4
votes
1answer
334 views

Two tables with two different charsets

I have one table which stores input from user, so to allow national characters I've set charset to utf8mb4. Then I have second table, which can grow really large (100+ millions of rows) and I'm 100% ...
4
votes
1answer
4k views

Using HASHBYTES() yields different results for nvarchar and a variable

I'm using server-side hashing to transmit passwords, then running PBKDF2 in the database to store the hashed password + salt combination. Hashing nvarchar(max) and a @variable holding the same value ...
0
votes
1answer
3k views

How to export data in Oracle database into CSV file with UTF-8 format?

I have an CSV file with WE8ISO8859P1 encoding generated from Oracle Stored Procedure. Is there any script line to define the output of CSV file in UTF-8 encoded format? CREATE OR REPLACE PROCEDURE ...
0
votes
1answer
553 views

Postgres upgrade and converting latin1 to utf8

I'm pretty new to PostgreSQL, but I've been tasked with doing an upgrade. The current system runs on CentOS 5.11 and is Postgres 8.1.23 with a default encoding of latin1. The new server is OpenSuSE ...
4
votes
1answer
203 views

Migrating from Postgres 8.4 to 9.4 in different data centers

I'm trying to migrate a Postgres 8.4 database in a Brazilian machine to Amazon RDS Postgres 9.4 in North Virginia. The database uses encoding LATIN1, tablespace pg_default, lc_collate and lc_ctype C. ...
3
votes
2answers
462 views

Database keeps using utfmb4 instead of utf8

Using MariaDB 10.1. The default encoding is utf8mb4, but I have an existing database that uses utf8. So in order to re-use the same DB, I would have to keep using utf8 for now. So I set the server ...
4
votes
1answer
162 views

ERROR: requested character too large for encoding

I'm using PostgreSQL 9.6 on Linux. I got an error when I do a test on chr() function. postgres=# select chr(1199111); ERROR: requested character too large for encoding: 1199111 postgres=# select ...
2
votes
1answer
2k views

Character Encoding in Postgres

I have UTF8 data stored in Postgres. I can query it with a Perl script and display it properly. When I use the Postgres client, it displays like the encoding is off. I'll see characters displayed ...
1
vote
1answer
399 views

sp_send_dbmail is populating 00 bits between all characters in a result set

I'm using sp_send_dbmail in a SQL agent job triggered nightly. The job queries our database and checks for product price updates, if there are some, it will send them in an email as an attachment to ...
3
votes
2answers
3k views

Not displaying special characters imported from a file

The scenario is a SQL Server instance, a database which is data fed using BULK INSERT operations mainly, and some of the texts inserted contain special characters like ñ because I am working in a ...
1
vote
0answers
264 views

Fixing a UTF8 encoded database that was dumped using an incorrect encoding?

The question is probably a little vague so let me elucidate: System OS is Ubuntu Linux 14.10 LTS The Servers default encoding is Latin1 The database default encoding is/was UTF8 The data was ...
3
votes
1answer
4k views

Equivalent of UTF8_UNICODE_CI collation in PostgreSQL

I would like a column in a table inside a PostgreSQL database (I am using version 9.6). I know of the UTF8_UNICODE_CI collation on MySQL, so I tried: CREATE TABLE thing ( id BIGINT PRIMARY KEY ...
0
votes
1answer
107 views

psqlODBC access encoding problem

My local machine: Windows 7, 32-bit. I have installed on it the psqlODBC for 32-bit. MS Access is 32-bit as well. I have tried both Unicode and Ansi connections to... The server: a 64-bit Ubuntu, and ...
1
vote
2answers
351 views

Mysql - latin1 (with latin2 content) to utf8

I have a database (ubbthreads) encoded in latin1 with content from latin2 (polish characters). Since latin2 is compatible with latin1 it looks fine on the website, however I cannot convert it in any ...
4
votes
2answers
7k views

Nvarchar to varchar to nvarchar characters conversion

I had a table filled with some greek text rows as nvarchar(2000). Lately, I altered the type of the column to varchar(4000) but I realized that some greek characters were shown as question marks. ...
2
votes
0answers
608 views

Convert large utf8 database to utf8mb4

I have a large database (~450gb) which I need to convert from utf8 to utf8mb4. I'm aware of the steps I need to take, but is this practical on tables containing this much data? I'm mainly concerned ...
1
vote
3answers
319 views

Fix double-encoded data

I have a field defined as Question VARCHAR(1024) CHARACTER SET utf8 NOT NULL. Our application stored double-encoded data into this field. For example, é shows up as é in phpMyAdmin, and C383C2A9 when ...