Skip to main content

Questions tagged [unicode]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
53 votes
2 answers
6k views

How to create Unicode parameter and variable names

All of this works: CREATE DATABASE [¯\_(ツ)_/¯]; GO USE [¯\_(ツ)_/¯]; GO CREATE SCHEMA [¯\_(ツ)_/¯]; GO CREATE TABLE [¯\_(ツ)_/¯].[¯\_(ツ)_/¯]([¯\_(ツ)_/¯] NVARCHAR(20)); GO CREATE UNIQUE CLUSTERED INDEX [¯...
Brent Ozar's user avatar
35 votes
1 answer
48k views

How do I set a SQL Server Unicode / NVARCHAR string to an emoji or Supplementary Character?

I want to set a Unicode string variable to particular character based on its Unicode code point. I want to use a code point beyond 65535, but the SQL Server 2008 R2 database has a collation of ...
Riley Major's user avatar
  • 1,903
26 votes
1 answer
18k views

Why does searching for LIKE N'%�%' match any Unicode character and = N'�' match many?

DECLARE @T TABLE( Col NCHAR(1)); INSERT INTO @T VALUES (N'A'), (N'B'), (N'C'), (N'Ƕ'), (N'Ƿ'), (N'Ǹ'); SELECT * FROM @T WHERE ...
Martin Smith's user avatar
  • 84.9k
20 votes
2 answers
5k views

Can't update "CO2" to "CO₂" in table row

Given this table: CREATE TABLE test ( id INT NOT NULL, description NVARCHAR(100) COLLATE Modern_Spanish_CI_AS NOT NULL ); INSERT INTO test (id, description) VALUES (1, 'CO2'); I've realised ...
Álvaro González's user avatar
20 votes
2 answers
4k views

Why these characters are all equal in SQL Server?

I just don't get it. See this SQL query: select nchar(65217) -- ﻁ select nchar(65218) -- ﻂ select nchar(65219) -- ﻃ select nchar(65220) -- ﻄ if nchar(65217) = nchar(65218) print 'equal' if nchar(...
Saeed Neamati's user avatar
20 votes
2 answers
42k views

Latin1_General_BIN performance impact when changing the database default collation

I have set the database collation to Latin1_General_BIN, to make string comparisons case-sensitive. Will this have an impact on performance? Will it have any impact on DML or DDL operations in the ...
Rakesh's user avatar
  • 303
20 votes
1 answer
25k views

Querying non-ASCII rows from Postgres

Does [:ascii:] class work in Postgres at all? It is not listed in their help, however I see examples in the web which utilize it. I have a UTF-8 database, where collation and c_type are en_US.UTF-8, ...
Suncatcher's user avatar
20 votes
2 answers
2k views

Accent Sensitive Sort

Why do these two SELECT statements result in a different sort order? USE tempdb; CREATE TABLE dbo.OddSort ( id INT IDENTITY(1,1) PRIMARY KEY , col1 NVARCHAR(2) , col2 NVARCHAR(2) ); GO ...
Aram's user avatar
  • 203
19 votes
5 answers
22k views

Why does the varchar datatype allow unicode values?

I have a table with a varchar column. It is allowing Trademark(™), copyright(©) and other Unicode characters as shown below. Create table VarcharUnicodeCheck ( col1 varchar(100) ) insert into ...
Shiva's user avatar
  • 787
17 votes
4 answers
2k views

How To Strip Hebrew Accent Marks

I need a Char Encoding Trick to Strip Hebrew Accent Marks. Sample Before בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ Sample After בראשית ברא אלהים את השמים ואת הארץ
Decrypted's user avatar
  • 273
15 votes
4 answers
20k views

Detect if any values in NVARCHAR columns are actually unicode

I have inherited some SQL Server databases. There is one table (I'll call "G"), with about 86.7 million rows, and 41 columns wide, from a source database (I'll call "Q") on SQL Server 2014 Standard ...
John G Hohengarten's user avatar
14 votes
1 answer
1k views

Why does a comparison between 'tr' & 'tR' fail on a SQL Server with Vietnamese_CI_AI collation?

There seems to be something special about 'tR' in Vietnamese collation. Appreicate if anyone who knows about it can explain in simple terms. This issue was discovered during the installation of our ...
QFirstLast's user avatar
13 votes
1 answer
18k views

Text string stored in SQLite Integer column?

I'm a database novice looking at an SQLite database which appears to be storing text in an integer column. Here's an example session at the sqlite3 command line: sqlite> .schema mytable CREATE ...
igal's user avatar
  • 345
12 votes
2 answers
3k views

When `nvarchar/nchar` is going to be used with SQL Server 2019?

With SQL Server 2019 Microsoft introduces UTF-8 support for CHAR and VARCHAR data types and says: This feature may provide significant storage savings, depending on the character set in use. For ...
gotqn's user avatar
  • 4,026
11 votes
1 answer
11k views

PostgreSQL nondeterministic collations are not supported for LIKE

I am using Postgresql v12. I created a collation like this: CREATE COLLATION ci (provider = icu, locale = 'tr_TR', deterministic = false); I used that collation in a table: create table testtable1 ...
Banu Akkus's user avatar
9 votes
2 answers
3k views

Why does Oracle use a different byte length than java for the supplementary unicode character chipmunk?

I have java code trimming a UTF-8 string to the size of my Oracle (11.2.0.4.0) column which ends up throwing an error because java and Oracle see the string as different byte lengths. I've verified my ...
agradl's user avatar
  • 211
9 votes
0 answers
712 views

Inconsistent equality checking of Unicode text in Postgres 9.5?

Using PostgreSQL 9.5. It seems like Postgres is being inconsistent with how it compares strings similar to لى. A unique constraint is considering some strings equal that GROUP BY is considering ...
sudo's user avatar
  • 1,011
8 votes
2 answers
5k views

Automatic Translation when Converting Unicode to non-Unicode / NVARCHAR to VARCHAR

Unicode code point 9619 is a character called "Dark shade": ▓ (http://unicode-table.com/en/search/?q=9619). Using the SQL_Latin1_General_CP1_CI_AS collation and 1252 code page, I would expect that ...
Henry Lee's user avatar
  • 1,246
8 votes
2 answers
6k views

How do I list all available encoding types available in postgreSQL?

SELECT datname, pg_encoding_to_char(encoding) FROM pg_database; …lists all the databases, each with its encoding type. However, I am trying to find out what are all the encoding types available in ...
Ashwin's user avatar
  • 183
8 votes
2 answers
12k views

What is the SQL Server equivalent of MySQL's utf8mb4 character set?

We have some web applications based on databases using utf8mb4 as character set and utf8mb4_Standard as collation. We saw that we can use any character we want with this setting. In SQL Server ...
Matthias's user avatar
  • 181
7 votes
2 answers
36k views

Storing Japanese characters in a table

I am using SQL Server 2008 R2 and I want to store Japanese characters in one of the columns of my table. Say I want to store Japanese names, how can I achieve that? Is there an easy way?
hud's user avatar
  • 203
7 votes
2 answers
7k views

Cannot remove char 0x0000 from NVARCHAR column

So, I know all about the Replace function and a bug with char(0). I have a column (NVARCHAR(128)) that has some NCHAR(0x0000) characters from a bad import. I'm using SQL Server 2008 R2. The ...
ttomsen's user avatar
  • 289
7 votes
1 answer
1k views

Problem with pattern matching on unicode characters

I'm writing a piece of code that uses pattern matching on unicode characters but I'm running into an odd problem. Some characters work and some don't. DECLARE @Pattern nvarchar(50) = N'%[^a-z]%' -- ...
Kenneth Fisher's user avatar
6 votes
1 answer
4k views

What is the meaning of "-x-icu" in PostgreSQL's "collates"?

I had this query: SELECT * FROM table ORDER BY label ASC; Since the labels are not in English, they didn't get sorted in the right order (ones beginning with "ö" were not in the bottom/end). I ...
Tadhg Mocniak's user avatar
6 votes
2 answers
2k views

Byte ordering for multibyte characters in SQL Server versus Oracle

I am currently in the process of migrating data from Oracle to SQL Server and I'm encountering an issue trying to validate the data post-migration. Environment Details: Oracle 12 - AL32UTF8 ...
HandyD's user avatar
  • 9,982
6 votes
1 answer
5k views

How to find values with multiple consecutive upper case characters

For a project that I am working on, I need to identify values that have incorrect uppercase. For example, I need it to identify the following types of values: Mr JOHN Smith MR John Smith Mr John ...
Megs D's user avatar
  • 61
6 votes
2 answers
378 views

Is there a (non-binary) MySQL collation that doesn't treat different mathematical symbols as the same character?

I've run into a real headache with MySQL's collations and non-BMP characters (ones with Unicode codepoints above U+FFFF). Basically, given a table and data like: CREATE TABLE `math` ( `id` int(11) ...
morganwahl's user avatar
5 votes
2 answers
23k views

Why do I get incorrect characters when decoding a Base64 string to NVARCHAR in SQL Server?

I've been looking at how to decode Base64 with SQL Server and after searching around many solutions online (some from here) seem to be based upon this sort of method. SELECT CAST(CAST('...
Geesh_SO's user avatar
  • 153
5 votes
1 answer
2k views

Why does MS SQL Server return a result for empty string check when Unicode string is not empty

select * from (select N'ግዜ ' as t) as t2 where t= '' the string 'ግዜ ' matches the above check, why is this?
Aussie Ash's user avatar
5 votes
2 answers
11k views

SQL Server Bulk Insert properly interprets some Unicode characters but not others?

For some reason the MS SQL Server 2016 bulk insert misinterprets/translates Unicode characters: C9 (É) into 2B (+) A1 (¡) into ED (í) A0 ( ) into E1 (á) AE (®) into AB («) CB (Ë) into 2D (-) D1 (...
Dave Sims's user avatar
  • 323
5 votes
2 answers
340 views

How can I efficiently downsize some Unicode fields?

We have a SQL Server 2012 Enterprise live transactional database that is now growing over 1G per month and is becoming a size problem for us. It is currently at 23G. Character type fields are all ...
Dave's user avatar
  • 175
5 votes
2 answers
5k views

Why is my UTF-8 document raising UTF-8 encoding errors in Azure Data Lake Analytics?

I have a document that was compressed in gunzip from a unknown source system. It was downloaded and decompressed using a 7zip console application. The document is a CSV file that appears to be encoded ...
Fastidious's user avatar
5 votes
2 answers
2k views

Problematic slashes and questions marks in unique index values

What is it about the Phone values inserted below that SQL Server is treating them as identical in a unique index? CREATE TABLE Phone ( Id int identity(1, 1) primary key, Phone nvarchar(448) not ...
cocogorilla's user avatar
5 votes
0 answers
3k views

VARCHAR(MAX) taking more space than NVARCHAR(MAX) [duplicate]

I have a table in SQL Server 2012 Enterprise Edition that has ~100GB of data. Most of this data is attributed to a single NVARCHAR(MAX) column which has an average length of 4500 characters. It stores ...
LowlyDBA - John M's user avatar
4 votes
3 answers
1k views

Unicode storage of \u202b RLE and \u202c PDE in a Unicode-aware database?

I'm building a new product for toponyms and in it the Arabic shows kinda like this: ^IArabic^I<202b>ﺰﻤﺑﺎﺑﻮﻳ<202c>^I<202b>ﺞﻫﻭﺮﻳﺓ ﺰﻤﺑﺎﺑﻮﻳ<202c>$ Actually not quite. This is a ...
Evan Carroll's user avatar
  • 63.7k
4 votes
1 answer
610 views

Fix corrupted utf8 nvarchar value that was inserted without N prefix

Is it possible to restore the UTF8 data from a corrupted nvarchar, that was used without the N prefix? For example, in the following snippet, I'd like @n1_fixed to have the correct value based on @n1: ...
HeyJude's user avatar
  • 447
4 votes
1 answer
915 views

What is difference between Arabic_100_CS_AS_KS_WS_SC_UTF8 and Latin1_General_100_CS_AS_KS_WS_SC_UTF8?

Starting from SQL Server 2019, it supports UTF-8 as collation. However, according to the following queries: SELECT COLLATIONPROPERTY('Arabic_100_CS_AS_KS_WS_SC_UTF8', 'CodePage') SELECT ...
Eng.Fouad's user avatar
  • 143
4 votes
1 answer
10k views

"String or binary data would be truncated" error copying nvarchar(10) to char(10)

I am inserting values from one SQL table / column into another. The datatypes are different for these columns due to other reasons but I do not understand why nvarchar(10) source and char(10) ...
user5664489's user avatar
4 votes
2 answers
11k views

Nvarchar to varchar to nvarchar characters conversion

I had a table filled with some greek text rows as nvarchar(2000). Lately, I altered the type of the column to varchar(4000) but I realized that some greek characters were shown as question marks. ...
George Chondrompilas's user avatar
4 votes
1 answer
1k views

Why isn't Unicode character REPLACEd in some cases?

Running this results in "test", as expected: SELECT REPLACE(NCHAR(1234), NCHAR(1234), N'test'); However, running this results in "aӒa", which doesn't have "test" in it: SELECT REPLACE(N'a' +...
Riley Major's user avatar
  • 1,903
4 votes
2 answers
18k views

Remove characters from PostgreSQL database causing encoding errors

I have a PostgreSQL 10 database that uses WIN1252 encoding. One of my columns has values that cause conversion errors when running a select from pgAdmin 4: SELECT myfield FROM mydb.myschema."MYTABLE"...
Jesse Reilly's user avatar
4 votes
1 answer
13k views

inserting a special character into a table

I have a few characters that are being replaced by a ?. How do you keep the original character in the table? The character that I am trying to insert is a ɬ; Latin Small Letter L With Belt. Is it ...
kevorski's user avatar
  • 177
4 votes
1 answer
4k views

Querying special character from SQL Server

When I saved special characters in a column of type nvarchar, I did not use the prefix 'N', so the characters got saved as ??. Is there any way to retrieve the original special characters from these ...
Kashinath's user avatar
4 votes
1 answer
3k views

Importing Chinese characters via SSIS

I'm trying to import file that contains Chinese characters into a table. File encoding is Big5 (Traditional). Here is a sample file like one that I need to import: https://www.pastiebin.com/...
GriGrim's user avatar
  • 302
4 votes
1 answer
1k views

supplementary characters in FreeTDS

I have a C++ application that connects to SQL Server 2008 through FreeTDS unixODBC on an RHEL box. However, i am unable to insert supplementary characters to the database through FreeTDS unixODBC. [...
D3XT3R's user avatar
  • 73
4 votes
1 answer
521 views

How is encoding done in SQL database tables for pattern matching?

Are strings in table columns represented as bit patterns or Unicode? Does the database engine bring them into memory in a different representation character set from the disk-persisted ...
Yousef's user avatar
  • 41
4 votes
1 answer
8k views

Special Characters in MySQL using UTF8 Unicode Collation and LOAD DATA INFILE

I have several csv files in am loading into MySQl using Java. In the Description field I have several Special Characters that are causing the load to fail. I am using LOAD DATA INFILE. This is nested ...
apesa's user avatar
  • 171
3 votes
2 answers
5k views

Storing unicode text into varchar column in SQL Server

I have mistakenly stored unicode text into a varchar column in a table and now it is shown as question marks (???). Is it possible to get our main information back from the column instead of question ...
Asrar Ahmad Ehsan's user avatar
3 votes
1 answer
7k views

ERROR: "sql " is not a known variable

This must one of the dumbest questions I've ever asked here, but there must something really sick hiding in my SQL script that is blocking it from running. I'm invoking the cobertura.sql file using ...
Gonzalo Vasquez's user avatar
3 votes
1 answer
17k views

How to be sure that a SQL Script is running with ANSI encoding?

Maybe this is a stupid question or I'm asking it in the wrong way. How can I be sure that a script ( with thousands of lines ) is running with ANSI encoding? Let's say we created a script using ...
Racer SQL's user avatar
  • 7,386