Questions tagged [encoding]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
25
votes
1answer
30k views

How do I set a SQL Server Unicode / NVARCHAR string to an emoji or Supplementary Character?

I want to set a Unicode string variable to particular character based on its Unicode code point. I want to use a code point beyond 65535, but the SQL Server 2008 R2 database has a collation of ...
20
votes
3answers
25k views

SQLCMD command is not able to insert accents

I am trying to run sqlcmd.exe in order to setup a new database from command line. I am using SQL SERVER Express 2012 on Windows 7 64 bits. Here's the command I use: SQLCMD -S .\MSSQLSERVER08 -V 17 -...
17
votes
4answers
10k views

Why does the varchar datatype allow unicode values?

I have a table with a varchar column. It is allowing Trademark(™), copyright(©) and other Unicode characters as shown below. Create table VarcharUnicodeCheck ( col1 varchar(100) ) insert into ...
17
votes
4answers
1k views

How To Strip Hebrew Accent Marks

I need a Char Encoding Trick to Strip Hebrew Accent Marks. Sample Before בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ Sample After בראשית ברא אלהים את השמים ואת הארץ
16
votes
4answers
71k views

SQL Server 2005/2008 UTF-8 Collation/Charset

I can't find option(s) directly to set UTF-8 rellated Collations/Charsets in SQL Server 2005/2008, same as is possible to set in another SQL engines, but in SQL Server 2005/2008 are there only Latin ...
15
votes
3answers
43k views

Using PostgreSQL 8.4, how to convert bytea to text value in postgres?

In my application I insert data in database using C code, since the strings I receive from an untrusted source I have escaped them using PQescapeByteaConn of libpq library. Which is working perfectly ...
12
votes
1answer
4k views

Error starting SQL Server 2017 service. Error Code 3417

I have SQL Server 2017 installed on my computer. This is what SELECT @@VERSION returns: Microsoft SQL Server 2017 (RTM-GDR) (KB4293803) - 14.0.2002.14 (X64) Jul 21 2018 07:47:45 Copyright (C) ...
10
votes
1answer
8k views

Encoding issue with SQL Server VARCHAR column retrieved in Python

We recently had an issue with encoding related to a field that's being stored as a varchar(120) in SQL Server. In SSMS, the varchar appears as: "Who Killed JonBen‚t?" However, when it's brought ...
9
votes
1answer
27k views

Postgresql: set default psql client encoding

When I connect to a Postgresql DB using psql, I often get these messages: => SELECT * FROM question_view ; ERROR: character with byte sequence 0xd7 0x9e in encoding "UTF8" has no equivalent in ...
8
votes
2answers
4k views

Automatic Translation when Converting Unicode to non-Unicode / NVARCHAR to VARCHAR

Unicode code point 9619 is a character called "Dark shade": ▓ (http://unicode-table.com/en/search/?q=9619). Using the SQL_Latin1_General_CP1_CI_AS collation and 1252 code page, I would expect that ...
7
votes
1answer
20k views

MySQL: Illegal mix of collations

I've tried using a stored procedure to create indices, and got the following error: ERROR 1267 (HY000): Illegal mix of collations (utf8_general_ci,IMPLICIT) and (utf8_unicode_ci,...
5
votes
1answer
1k views

Why does MS SQL Server return a result for empty string check when Unicode string is not empty

select * from (select N'ግዜ ' as t) as t2 where t= '' the string 'ግዜ ' matches the above check, why is this?
5
votes
1answer
8k views

Convert MySQL database from latin1 to utf8mb4 - and take care of German umlauts

I am using 5.6.28 MySQL Community Server to host WordPress + ProPhoto blog in German language at CentOS 6.7 Linux server: mysql> show tables; +-----------------------+ | Tables_in_blog | +--...
5
votes
1answer
2k views

Postgres - Encoding, Collation and CType

List of databases Name | Owner | Encoding | Collate | Ctype | -------------------------+----------+----------+---------------------------...
5
votes
2answers
3k views

SQL Server Bulk Insert properly interprets some Unicode characters but not others?

For some reason the MS SQL Server 2016 bulk insert misinterprets/translates Unicode characters: C9 (É) into 2B (+) A1 (¡) into ED (í) A0 ( ) into E1 (á) AE (®) into AB («) CB (Ë) into 2D (-) D1 (...
5
votes
1answer
135 views

Query to find rows containing ASCII characters in a given range

I am using some scripts from another topic, but the accepted answer isn't working for all my data scenarios. I would have asked my question on the original How to check for Non-Ascii Characters post, ...
5
votes
2answers
2k views

Why is my UTF-8 document raising UTF-8 encoding errors in Azure Data Lake Analytics?

I have a document that was compressed in gunzip from a unknown source system. It was downloaded and decompressed using a 7zip console application. The document is a CSV file that appears to be encoded ...
5
votes
1answer
2k views

How do I list all available encoding types available in postgreSQL?

SELECT datname, pg_encoding_to_char(encoding) FROM pg_database; …lists all the databases, each with its encoding type. However, I am trying to find out what are all the encoding types available in ...
4
votes
3answers
468 views

Unicode storage of \u202b RLE and \u202c PDE in a Unicode-aware database?

I'm building a new product for toponyms and in it the Arabic shows kinda like this: ^IArabic^I<202b>ﺰﻤﺑﺎﺑﻮﻳ<202c>^I<202b>ﺞﻫﻭﺮﻳﺓ ﺰﻤﺑﺎﺑﻮﻳ<202c>$ Actually not quite. This is a ...
4
votes
2answers
8k views

Nvarchar to varchar to nvarchar characters conversion

I had a table filled with some greek text rows as nvarchar(2000). Lately, I altered the type of the column to varchar(4000) but I realized that some greek characters were shown as question marks. ...
4
votes
1answer
6k views

Using HASHBYTES() yields different results for nvarchar and a variable

I'm using server-side hashing to transmit passwords, then running PBKDF2 in the database to store the hashed password + salt combination. Hashing nvarchar(max) and a @variable holding the same value ...
4
votes
1answer
5k views

Why do I get incorrect characters when decoding a Base64 string to NVARCHAR in SQL Server?

I've been looking at how to decode Base64 with SQL Server and after searching around many solutions online (some from here) seem to be based upon this sort of method. SELECT CAST(CAST('...
4
votes
1answer
826 views

Collate issues with wrong characters

Well, the problem is well known, but I'm looking for a smarter solution if there's one. For some reason the system doesn't recognize some characters and I can't compare the columns Here is an ...
4
votes
2answers
671 views

Byte ordering for multibyte characters in SQL Server versus Oracle

I am currently in the process of migrating data from Oracle to SQL Server and I'm encountering an issue trying to validate the data post-migration. Environment Details: Oracle 12 - AL32UTF8 ...
4
votes
2answers
536 views

Different characters, same ASCII code?

I have this query that throws two results: SELECT id FROM table1 WHERE id like 'nm041033%' nm0410331 nm0410331 And this slightly different query that throws only one result: SELECT id FROM table1 ...
4
votes
1answer
866 views

Two tables with two different charsets

I have one table which stores input from user, so to allow national characters I've set charset to utf8mb4. Then I have second table, which can grow really large (100+ millions of rows) and I'm 100% ...
4
votes
1answer
6k views

Equivalent of UTF8_UNICODE_CI collation in PostgreSQL

I would like a column in a table inside a PostgreSQL database (I am using version 9.6). I know of the UTF8_UNICODE_CI collation on MySQL, so I tried: CREATE TABLE thing ( id BIGINT PRIMARY KEY ...
4
votes
1answer
4k views

Postgres copy data with \xYY as plain string instead of interpreting as encoded string

I have a log file full of URLs, generated by Bro IDS. When Bro logs a URL with non-ascii characters in it, it inserts \xYY where YY is the hexadecimal character code. Also, some URLs contain "\x". ...
4
votes
1answer
4k views

Postgres database encoding problem

I'm striving to convert badly encoded data from my table. For instance, I have a field with Nadège which should be Nadège. I tried using Postgres's functions convert, convert_from, convert_to ...
4
votes
1answer
227 views

How is encoding done in SQL database tables for pattern matching?

Are strings in table columns represented as bit patterns or Unicode? Does the database engine bring them into memory in a different representation character set from the disk-persisted ...
3
votes
1answer
7k views

“String or binary data would be truncated” error copying nvarchar(10) to char(10)

I am inserting values from one SQL table / column into another. The datatypes are different for these columns due to other reasons but I do not understand why nvarchar(10) source and char(10) ...
3
votes
2answers
2k views

SQL Server 2019 UTF-8 Support Benefits

I'm already quite comfortable with using COMPRESS() and DECOMPRESS() in an internal forum software for our company (Currently in SQL Server 2017), but trying to make the database as efficient as ...
3
votes
2answers
743 views

Use non-printable characters in SQL Server BCP xml format file as TERMINATOR

I have a flat-text data file containing records which fields are separated by the non printable character "File Separator" (0x1c). I am trying to use SQL Server's bcp utility to load this data into my ...
3
votes
1answer
6k views

How to be sure that a SQL Script is running with ANSI encoding?

Maybe this is a stupid question or I'm asking it in the wrong way. How can I be sure that a script ( with thousands of lines ) is running with ANSI encoding? Let's say we created a script using ...
3
votes
2answers
5k views

Not displaying special characters imported from a file

The scenario is a SQL Server instance, a database which is data fed using BULK INSERT operations mainly, and some of the texts inserted contain special characters like ñ because I am working in a ...
3
votes
2answers
4k views

Character Conversion Issue Importing From CSV File

After doing a load of CSV files, there are various words that are "written" to the database incorrectly. Some examples: Diã¡ria should be Diária Crã©dito should be Crédito Ligaã§ãµes should be ...
3
votes
1answer
1k views

Dutch characters in DB2 problem

Ok, first of all, I have really limited knowledge about databases. Problem that I have is next: I inserted few rows with data that contains Dutch characters, such as names of different countries. ...
3
votes
2answers
18k views

Tell MySQL to start using utf-8 encoding without `convert to`ing it

In a fairly unique situation, my team has ended up with UTF-8 bytes in a database that thinks the data is encoded as latin1. At least, I'm 85% certain that this is the situation at hand. For example,...
3
votes
1answer
238 views

Migrating from Postgres 8.4 to 9.4 in different data centers

I'm trying to migrate a Postgres 8.4 database in a Brazilian machine to Amazon RDS Postgres 9.4 in North Virginia. The database uses encoding LATIN1, tablespace pg_default, lc_collate and lc_ctype C. ...
3
votes
1answer
240 views

ERROR: requested character too large for encoding

I'm using PostgreSQL 9.6 on Linux. I got an error when I do a test on chr() function. postgres=# select chr(1199111); ERROR: requested character too large for encoding: 1199111 postgres=# select ...
3
votes
1answer
2k views

How to change database encoding in an Amazon RDS Postgresql instance?

I need to create a database with SQL_ASCII encoding in Amazon RDS PostgreSql database instance. Since template0 and template1 are both with utf8 encoding, I get the following error $ createdb -h ...
3
votes
1answer
2k views

Confused over encoding/locale in postgresql

I am a little confused over the difference between encoding and locale as it pertains to the postgresql database. When initializing the database, you can specify both an encoding and a locale. ...
3
votes
2answers
10k views

MySQL silently replaces UTF chars with literal question marks

I'm experiencing a situation similar to this SO question i.e. I'm working with a legacy database which has UTF8 content in latin1 tables (pretty ugly I know). Now I'm getting new data from a new ...
3
votes
2answers
579 views

Database keeps using utfmb4 instead of utf8

Using MariaDB 10.1. The default encoding is utf8mb4, but I have an existing database that uses utf8. So in order to re-use the same DB, I would have to keep using utf8 for now. So I set the server ...
3
votes
3answers
590 views

TSQL Script Execution Through Power shell, Special Characters Rendering Problem

We have a simple T-SQL script file (.sql), which we execute through Powershell. Everything works fine, except the script portion, where we are using some special characters as string (example: 'ô' ). ...
3
votes
0answers
2k views

Converting a SQL_ASCII database to UTF-8

I have a postgresql (version 8.1) database that has the encoding set to SQL_ASCII, combined with several apps that some are running using UTF8 and some with LATIN1. With this scope I have a database ...
2
votes
2answers
7k views

Postgres client unable to detect UTF-8 database

Context I am deploying a Vagrant box for my development team, and have encountered a problem restoring UTF-8 collated dumps. I don't believe it is related to the now-infamous "UTF8" has no equivalent ...
2
votes
3answers
5k views

NLS_LANG and NLS_CHARACTERSET

I thought I read that NLS_CHARACTERSET has the encoding of the database and NLS_LANG the encoding of the client. Is that correct? Does that mean that the two encodings can be different? In some ...
2
votes
3answers
2k views

Restoring SQL_ASCII dumps to a UTF8-encoded database

I've got a Postgres 8.4 environment where the encoding on all our databases is set to SQL_ASCII - We're finally migrating to Postgres 9.2, and I would like to migrate everything over to UTF8 encoding. ...
2
votes
1answer
670 views

How to insert a Unicode character verbose into a varchar DB?

I need to insert this character '●' into a VARCHAR column of a MSSQL database with collation set as SQL_Latin1_General_CP1_CI_AS (or at least mock what my Python + Windows MSSQL Driver might have done)...