Questions tagged [unicode]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
53
votes
2answers
5k views

How to create Unicode parameter and variable names

All of this works: CREATE DATABASE [¯\_(ツ)_/¯]; GO USE [¯\_(ツ)_/¯]; GO CREATE SCHEMA [¯\_(ツ)_/¯]; GO CREATE TABLE [¯\_(ツ)_/¯].[¯\_(ツ)_/¯]([¯\_(ツ)_/¯] NVARCHAR(20)); GO CREATE UNIQUE CLUSTERED INDEX [¯...
22
votes
1answer
26k views

How do I set a SQL Server Unicode / NVARCHAR string to an emoji or Supplementary Character?

I want to set a Unicode string variable to particular character based on its Unicode code point. I want to use a code point beyond 65535, but the SQL Server 2008 R2 database has a collation of ...
20
votes
2answers
3k views

Why these characters are all equal in SQL Server?

I just don't get it. See this SQL query: select nchar(65217) -- ﻁ select nchar(65218) -- ﻂ select nchar(65219) -- ﻃ select nchar(65220) -- ﻄ if nchar(65217) = nchar(65218) print 'equal' if nchar(...
20
votes
1answer
6k views

Why does searching for LIKE N'%�%' match any Unicode character and = N'�' match many?

DECLARE @T TABLE( Col NCHAR(1)); INSERT INTO @T VALUES (N'A'), (N'B'), (N'C'), (N'Ƕ'), (N'Ƿ'), (N'Ǹ'); SELECT * FROM @T WHERE ...
19
votes
2answers
453 views

Accent Sensitive Sort

Why do these two SELECT statements result in a different sort order? USE tempdb; CREATE TABLE dbo.OddSort ( id INT IDENTITY(1,1) PRIMARY KEY , col1 NVARCHAR(2) , col2 NVARCHAR(2) ); GO ...
18
votes
2answers
3k views

Can't update “CO2” to “CO₂” in table row

Given this table: CREATE TABLE test ( id INT NOT NULL, description NVARCHAR(100) COLLATE Modern_Spanish_CI_AS NOT NULL ); INSERT INTO test (id, description) VALUES (1, 'CO2'); I've realised ...
17
votes
4answers
1k views

How To Strip Hebrew Accent Marks

I need a Char Encoding Trick to Strip Hebrew Accent Marks. Sample Before בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ Sample After בראשית ברא אלהים את השמים ואת הארץ
16
votes
2answers
24k views

Latin1_General_BIN performance impact when changing the database default collation

I have set the database collation to Latin1_General_BIN, to make string comparisons case-sensitive. Will this have an impact on performance? Will it have any impact on DML or DDL operations in the ...
16
votes
4answers
7k views

Why does the varchar datatype allow unicode values?

I have a table with a varchar column. It is allowing Trademark(™), copyright(©) and other Unicode characters as shown below. Create table VarcharUnicodeCheck ( col1 varchar(100) ) insert into ...
14
votes
4answers
10k views

Detect if any values in NVARCHAR columns are actually unicode

I have inherited some SQL Server databases. There is one table (I'll call "G"), with about 86.7 million rows, and 41 columns wide, from a source database (I'll call "Q") on SQL Server 2014 Standard ...
12
votes
1answer
8k views

Querying non-ASCII rows from Postgres

Does [:ascii:] class work in Postgres at all? It is not listed in their help, however I see examples in the web which utilize it. I have a UTF-8 database, where collation and c_type are en_US.UTF-8, ...
9
votes
2answers
1k views

When `nvarchar/nchar` is going to be used with SQL Server 2019?

With SQL Server 2019 Microsoft introduces UTF-8 support for CHAR and VARCHAR data types and says: This feature may provide significant storage savings, depending on the character set in use. For ...
8
votes
2answers
3k views

Automatic Translation when Converting Unicode to non-Unicode / NVARCHAR to VARCHAR

Unicode code point 9619 is a character called "Dark shade": ▓ (http://unicode-table.com/en/search/?q=9619). Using the SQL_Latin1_General_CP1_CI_AS collation and 1252 code page, I would expect that ...
8
votes
2answers
1k views

Why does Oracle use a different byte length than java for the supplementary unicode character chipmunk?

I have java code trimming a UTF-8 string to the size of my Oracle (11.2.0.4.0) column which ends up throwing an error because java and Oracle see the string as different byte lengths. I've verified my ...
7
votes
1answer
899 views

Problem with pattern matching on unicode characters

I'm writing a piece of code that uses pattern matching on unicode characters but I'm running into an odd problem. Some characters work and some don't. DECLARE @Pattern nvarchar(50) = N'%[^a-z]%' -- ...
6
votes
1answer
3k views

How to find values with multiple consecutive upper case characters

For a project that I am working on, I need to identify values that have incorrect uppercase. For example, I need it to identify the following types of values: Mr JOHN Smith MR John Smith Mr John ...
6
votes
2answers
260 views

Is there a (non-binary) MySQL collation that doesn't treat different mathematical symbols as the same character?

I've run into a real headache with MySQL's collations and non-BMP characters (ones with Unicode codepoints above U+FFFF). Basically, given a table and data like: CREATE TABLE `math` ( `id` int(11) ...
6
votes
2answers
2k views

What is the SQL Server equivalent of MySQL's utf8mb4 character set?

We have some web applications based on databases using utf8mb4 as character set and utf8mb4_Standard as collation. We saw that we can use any character we want with this setting. In SQL Server ...
5
votes
1answer
18k views

Storing Japanese characters in a table

I am using SQL Server 2008 R2 and I want to store Japanese characters in one of the columns of my table. Say I want to store Japanese names, how can I achieve that? Is there an easy way?
5
votes
1answer
1k views

Why does MS SQL Server return a result for empty string check when Unicode string is not empty

select * from (select N'ግዜ ' as t) as t2 where t= '' the string 'ግዜ ' matches the above check, why is this?
5
votes
2answers
4k views

Cannot remove char 0x0000 from NVARCHAR column

So, I know all about the Replace function and a bug with char(0). I have a column (NVARCHAR(128)) that has some NCHAR(0x0000) characters from a bad import. I'm using SQL Server 2008 R2. The ...
5
votes
2answers
255 views

How can I efficiently downsize some Unicode fields?

We have a SQL Server 2012 Enterprise live transactional database that is now growing over 1G per month and is becoming a size problem for us. It is currently at 23G. Character type fields are all ...
5
votes
2answers
2k views

SQL Server Bulk Insert properly interprets some Unicode characters but not others?

For some reason the MS SQL Server 2016 bulk insert misinterprets/translates Unicode characters: C9 (É) into 2B (+) A1 (¡) into ED (í) A0 ( ) into E1 (á) AE (®) into AB («) CB (Ë) into 2D (-) D1 (...
5
votes
2answers
829 views

Problematic slashes and questions marks in unique index values

What is it about the Phone values inserted below that SQL Server is treating them as identical in a unique index? CREATE TABLE Phone ( Id int identity(1, 1) primary key, Phone nvarchar(448) not ...
5
votes
1answer
1k views

How do I list all available encoding types available in postgreSQL?

SELECT datname, pg_encoding_to_char(encoding) FROM pg_database; …lists all the databases, each with its encoding type. However, I am trying to find out what are all the encoding types available in ...
5
votes
0answers
1k views

VARCHAR(MAX) taking more space than NVARCHAR(MAX) [duplicate]

I have a table in SQL Server 2012 Enterprise Edition that has ~100GB of data. Most of this data is attributed to a single NVARCHAR(MAX) column which has an average length of 4500 characters. It stores ...
5
votes
0answers
435 views

Inconsistent equality checking of Unicode text in Postgres 9.5?

Using PostgreSQL 9.5. It seems like Postgres is being inconsistent with how it compares strings similar to لى. A unique constraint is considering some strings equal that GROUP BY is considering ...
4
votes
3answers
400 views

Unicode storage of \u202b RLE and \u202c PDE in a Unicode-aware database?

I'm building a new product for toponyms and in it the Arabic shows kinda like this: ^IArabic^I<202b>ﺰﻤﺑﺎﺑﻮﻳ<202c>^I<202b>ﺞﻫﻭﺮﻳﺓ ﺰﻤﺑﺎﺑﻮﻳ<202c>$ Actually not quite. This is a ...
4
votes
2answers
7k views

Nvarchar to varchar to nvarchar characters conversion

I had a table filled with some greek text rows as nvarchar(2000). Lately, I altered the type of the column to varchar(4000) but I realized that some greek characters were shown as question marks. ...
4
votes
1answer
4k views

Why do I get incorrect characters when decoding a Base64 string to NVARCHAR in SQL Server?

I've been looking at how to decode Base64 with SQL Server and after searching around many solutions online (some from here) seem to be based upon this sort of method. SELECT CAST(CAST('...
4
votes
1answer
8k views

Text string stored in SQLite Integer column?

I'm a database novice looking at an SQLite database which appears to be storing text in an integer column. Here's an example session at the sqlite3 command line: sqlite> .schema mytable CREATE ...
4
votes
2answers
370 views

Byte ordering for multibyte characters in SQL Server versus Oracle

I am currently in the process of migrating data from Oracle to SQL Server and I'm encountering an issue trying to validate the data post-migration. Environment Details: Oracle 12 - AL32UTF8 ...
4
votes
1answer
191 views

Why isn't Unicode character REPLACEd in some cases?

Running this results in "test", as expected: SELECT REPLACE(NCHAR(1234), NCHAR(1234), N'test'); However, running this results in "aӒa", which doesn't have "test" in it: SELECT REPLACE(N'a' +...
4
votes
1answer
723 views

Querying special character from SQL Server

When I saved special characters in a column of type nvarchar, I did not use the prefix 'N', so the characters got saved as ??. Is there any way to retrieve the original special characters from these ...
4
votes
1answer
862 views

supplementary characters in FreeTDS

I have a C++ application that connects to SQL Server 2008 through FreeTDS unixODBC on an RHEL box. However, i am unable to insert supplementary characters to the database through FreeTDS unixODBC. [...
4
votes
1answer
222 views

How is encoding done in SQL database tables for pattern matching?

Are strings in table columns represented as bit patterns or Unicode? Does the database engine bring them into memory in a different representation character set from the disk-persisted ...
4
votes
1answer
5k views

Special Characters in MySQL using UTF8 Unicode Collation and LOAD DATA INFILE

I have several csv files in am loading into MySQl using Java. In the Description field I have several Special Characters that are causing the load to fail. I am using LOAD DATA INFILE. This is nested ...
3
votes
1answer
7k views

“String or binary data would be truncated” error copying nvarchar(10) to char(10)

I am inserting values from one SQL table / column into another. The datatypes are different for these columns due to other reasons but I do not understand why nvarchar(10) source and char(10) ...
3
votes
2answers
175 views

Storing unicode text into varchar column in SQL Server

I have mistakenly stored unicode text into a varchar column in a table and now it is shown as question marks (???). Is it possible to get our main information back from the column instead of question ...
3
votes
1answer
4k views

How to be sure that a SQL Script is running with ANSI encoding?

Maybe this is a stupid question or I'm asking it in the wrong way. How can I be sure that a script ( with thousands of lines ) is running with ANSI encoding? Let's say we created a script using ...
3
votes
1answer
4k views

inserting a special character into a table

I have a few characters that are being replaced by a ?. How do you keep the original character in the table? The character that I am trying to insert is a ɬ; Latin Small Letter L With Belt. Is it ...
3
votes
1answer
448 views

Where can I find exact rules of a collation?

Is there any documentation anywhere for the exact sorting/equality rules defined by a collation in Windows (SQL Server), for instance "Finnish_Swedish_CI_AS"? I know about CI/CS, AI/AS etc, but I'm ...
3
votes
3answers
482 views

TSQL Script Execution Through Power shell, Special Characters Rendering Problem

We have a simple T-SQL script file (.sql), which we execute through Powershell. Everything works fine, except the script portion, where we are using some special characters as string (example: 'ô' ). ...
2
votes
1answer
2k views

ERROR: “sql ” is not a known variable

This must one of the dumbest questions I've ever asked here, but there must something really sick hiding in my SQL script that is blocking it from running. I'm invoking the cobertura.sql file using ...
2
votes
1answer
14k views

How to disable case sensitivity in SQL Server 2012?

I tried to update collation_name in sys.databases. I am trying to change 'SQL_Latin1_General_CP1_CS_AS' to 'SQL_Latin1_General_CP1_CI_AS' in order to disable case sensitivity in table. update d set d....
2
votes
1answer
6k views

ssis cannot convert between unicode and non-unicode data types even though the source and the destination columns are both unicode

I faced an error: 'SSIS cannot convert between unicode and non-unicode string data types' even though the source and the destination columns are both unicode (NVARCHAR) data type. The source db and ...
2
votes
1answer
1k views

What collation to use for Ukraine?

We have an accounting program with an SQL Server database and would like to start a Ukrainian version. I am not sure which collation to use: Ukrainian or Cyrillic_General? With a non-technical ...
2
votes
1answer
204 views

How to send Unicode symbols using SQL Server Database Mail? [closed]

How can I send Unicode symbols using SQL Server Database Mail? My goal is to send a symbol on the subject of an email. Let's say that, for example, I want to send this "🚯 Subject with symbol" on the ...
2
votes
1answer
175 views

SELECT query: filter unable to distinguish between two different characters

In a recent support case, an error was traced to the difference between two similar-looking characters: ― (Unicode U+2015) and ー (Unicode U+30FC). A select statement with a WHERE clause was unable to ...
2
votes
1answer
512 views

Why isn't the ASYMKEY_ID function returning the Asymmetric Key ID?

I have created database master key. Then I have created asymmetric key: CREATE ASYMMETRIC KEY smGK_АSymmetricKey_01 WITH ALGORITHM = RSA_512 In the documentation is said, that: PASSWORD = '...