Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [unicode]

The tag has no usage guidance.

-1
votes
1answer
103 views

Incorrect string value: '\xF0\x9F\xA4\x96<b…' for column 'body' at row 1 [closed]

I've set the column type to utf8mb4 and the collation to utf8mb4_bin. The table defaults are also utf8mb4 and utf8mb4_bin. When I use SequelPro to add the emoji "🤖" to the database manually, it ...
0
votes
1answer
28 views

Mariadb (MySQL) On Windows- problem entering non-ASCII characters in a query

MariaDB: mysql Ver 15.1 Distrib 10.3.8-MariaDB OS: Windows 7 Pro 64 bit I have my server database and client all set up (I believe) to support utf8 end to end- (**my.cnf**) # # Set everything to ...
4
votes
2answers
79 views

Byte ordering for multibyte characters in SQL Server versus Oracle

I am currently in the process of migrating data from Oracle to SQL Server and I'm encountering an issue trying to validate the data post-migration. Environment Details: Oracle 12 - AL32UTF8 ...
4
votes
3answers
318 views

Unicode storage of \u202b RLE and \u202c PDE in a Unicode-aware database?

I'm building a new product for toponyms and in it the Arabic shows kinda like this: ^IArabic^I<202b>ﺰﻤﺑﺎﺑﻮﻳ<202c>^I<202b>ﺞﻫﻭﺮﻳﺓ ﺰﻤﺑﺎﺑﻮﻳ<202c>$ Actually not quite. This is a ...
0
votes
1answer
25 views

How to import SQL file which has no N prefix

I have a large .sql file in Persian Language. The file is exported from sqlite db browser and has no N prefix for unicode fields. How can I reuse it in Sql Server? This is sample of insert query in ...
1
vote
1answer
57 views

How to insert a Unicode character verbose into a varchar DB?

I need to insert this character '●' into a VARCHAR column of a MSSQL database with collation set as SQL_Latin1_General_CP1_CI_AS (or at least mock what my Python + Windows MSSQL Driver might have done)...
2
votes
1answer
106 views

How to send Unicode symbols using SQL Server Database Mail? [closed]

How can I send Unicode symbols using SQL Server Database Mail? My goal is to send a symbol on the subject of an email. Let's say that, for example, I want to send this "🚯 Subject with symbol" on the ...
9
votes
2answers
940 views

When `nvarchar/nchar` is going to be used with SQL Server 2019?

With SQL Server 2019 Microsoft introduces UTF-8 support for CHAR and VARCHAR data types and says: This feature may provide significant storage savings, depending on the character set in use. For ...
4
votes
1answer
195 views

Querying special character from SQL Server

When I saved special characters in a column of type nvarchar, I did not use the prefix 'N', so the characters got saved as ??. Is there any way to retrieve the original special characters from these ...
4
votes
1answer
2k views

Why do I get incorrect characters when decoding a Base64 string to NVARCHAR in SQL Server?

I've been looking at how to decode Base64 with SQL Server and after searching around many solutions online (some from here) seem to be based upon this sort of method. SELECT CAST(CAST('...
4
votes
1answer
109 views

Why isn't Unicode character REPLACEd in some cases?

Running this results in "test", as expected: SELECT REPLACE(NCHAR(1234), NCHAR(1234), N'test'); However, running this results in "aӒa", which doesn't have "test" in it: SELECT REPLACE(N'a' +...
3
votes
1answer
2k views

inserting a special character into a table

I have a few characters that are being replaced by a ?. How do you keep the original character in the table? The character that I am trying to insert is a ɬ; Latin Small Letter L With Belt. Is it ...
1
vote
1answer
34 views

Can't set up Unicode support

I am creating a toy website locally with XAMPP and I want it to support Unicode. index.php contains a simple form that I can use to add a new entry (a UTF-8 string) to a table "myTable" in the DB. ...
5
votes
2answers
1k views

SQL Server Bulk Insert properly interprets some Unicode characters but not others?

For some reason the MS SQL Server 2016 bulk insert misinterprets/translates Unicode characters: C9 (É) into 2B (+) A1 (¡) into ED (í) A0 ( ) into E1 (á) AE (®) into AB («) CB (Ë) into 2D (-) D1 (...
0
votes
1answer
94 views

sp_executesql changes non standard characters to?

Am trying to use sp_executesql to run commands but I've come across an issue where in non-english characters (aka Chinese & Japanese) are changed to ??? when ran. sql server 2008R2 & 2012 ...
53
votes
2answers
4k views

How to create Unicode parameter and variable names

All of this works: CREATE DATABASE [¯\_(ツ)_/¯]; GO USE [¯\_(ツ)_/¯]; GO CREATE SCHEMA [¯\_(ツ)_/¯]; GO CREATE TABLE [¯\_(ツ)_/¯].[¯\_(ツ)_/¯]([¯\_(ツ)_/¯] NVARCHAR(20)); GO CREATE UNIQUE CLUSTERED INDEX [¯...
19
votes
2answers
357 views

Accent Sensitive Sort

Why do these two SELECT statements result in a different sort order? USE tempdb; CREATE TABLE dbo.OddSort ( id INT IDENTITY(1,1) PRIMARY KEY , col1 NVARCHAR(2) , col2 NVARCHAR(2) ); GO ...
2
votes
1answer
178 views

pgAdmin 4 including <U+FEFF> in all copied strings

I'm having trouble using copy/paste from pgAdmin 4. The symptom is that I get the character <U+FEFF> aka Zero Width No-break Space aka Byte Order Mark (BOM) prepended to any string I copy. The ...
2
votes
1answer
2k views

ERROR: “sql ” is not a known variable

This must one of the dumbest questions I've ever asked here, but there must something really sick hiding in my SQL script that is blocking it from running. I'm invoking the cobertura.sql file using ...
13
votes
4answers
5k views

Why does the varchar datatype allow unicode values?

I have a table with a varchar column. It is allowing Trademark(™), copyright(©) and other Unicode characters as shown below. Create table VarcharUnicodeCheck ( col1 varchar(100) ) insert into ...
2
votes
2answers
2k views

Remove characters from PostgreSQL database causing encoding errors

I have a PostgreSQL 10 database that uses WIN1252 encoding. One of my columns has values that cause conversion errors when running a select from pgAdmin 4: SELECT myfield FROM mydb.myschema."MYTABLE"...
0
votes
1answer
39 views

Special letter behaviour problem?

I asked question in very blurry manner because I cannot explain in short sentence what is my problem about. In general I have a set of procedures which transforms text and XML. Inside the procedures ...
1
vote
1answer
1k views

Using LIKE to search case insensitive over language specific characters (PostgreSQL)

Using a PostgreSQL database (Collation C, Encoding UTF8) we store data from various languages. This means we also get the special characters for free (e.g. Å, å, ...) Within an application we now ...
18
votes
2answers
3k views

Can't update “CO2” to “CO₂” in table row

Given this table: CREATE TABLE test ( id INT NOT NULL, description NVARCHAR(100) COLLATE Modern_Spanish_CI_AS NOT NULL ); INSERT INTO test (id, description) VALUES (1, 'CO2'); I've realised ...
3
votes
1answer
3k views

How to be sure that a SQL Script is running with ANSI encoding?

Maybe this is a stupid question or I'm asking it in the wrong way. How can I be sure that a script ( with thousands of lines ) is running with ANSI encoding? Let's say we created a script using ...
5
votes
0answers
1k views

VARCHAR(MAX) taking more space than NVARCHAR(MAX) [duplicate]

I have a table in SQL Server 2012 Enterprise Edition that has ~100GB of data. Most of this data is attributed to a single NVARCHAR(MAX) column which has an average length of 4500 characters. It stores ...
0
votes
2answers
1k views

Spool Unicode data in sqlplus to ISO-client without character set conversion

I'm currently facing the following problem: We have an Oracle 11g database with character set AL32UTF8, and an sqlplus-client with NLS_LANG=GERMAN_GERMANY.WE8ISO8859P1. Stored in the database are ...
5
votes
2answers
593 views

Problematic slashes and questions marks in unique index values

What is it about the Phone values inserted below that SQL Server is treating them as identical in a unique index? CREATE TABLE Phone ( Id int identity(1, 1) primary key, Phone nvarchar(448) not ...
4
votes
1answer
1k views

How do I list all available encoding types available in postgreSQL?

SELECT datname, pg_encoding_to_char(encoding) FROM pg_database; …lists all the databases, each with its encoding type. However, I am trying to find out what are all the encoding types available in ...
2
votes
1answer
159 views

SELECT query: filter unable to distinguish between two different characters

In a recent support case, an error was traced to the difference between two similar-looking characters: ― (Unicode U+2015) and ー (Unicode U+30FC). A select statement with a WHERE clause was unable to ...
8
votes
2answers
1k views

Why does Oracle use a different byte length than java for the supplementary unicode character chipmunk?

I have java code trimming a UTF-8 string to the size of my Oracle (11.2.0.4.0) column which ends up throwing an error because java and Oracle see the string as different byte lengths. I've verified my ...
2
votes
1answer
84 views

International Components for Unicode (ICU) Date Formatter for MySQL?

Regarding to ICU - International Components for Unicode definition: ICU is a mature, widely used set of C/C++ and Java libraries providing Unicode and Globalization support for software ...
2
votes
1answer
6k views

ssis cannot convert between unicode and non-unicode data types even though the source and the destination columns are both unicode

I faced an error: 'SSIS cannot convert between unicode and non-unicode string data types' even though the source and the destination columns are both unicode (NVARCHAR) data type. The source db and ...
21
votes
1answer
4k views

Why does searching for LIKE N'%�%' match any Unicode character and = N'�' match many?

DECLARE @T TABLE( Col NCHAR(1)); INSERT INTO @T VALUES (N'A'), (N'B'), (N'C'), (N'Ƕ'), (N'Ƿ'), (N'Ǹ'); SELECT * FROM @T WHERE ...
0
votes
1answer
494 views

How to handle the junk characters upon conversion of the data from image datatype to varchar?

We are trying to convert the column of type image to column of type varchar(max). We tried the below way to do the conversion. --col1 is of type image --col2 is of type varchar(max) set col2 = ...
13
votes
1answer
6k views

Querying non-ASCII rows from Postgres

Does [:ascii:] class work in Postgres at all? It is not listed in their help, however I see examples in the web which utilize it. I have a UTF-8 database, where collation and c_type are en_US.UTF-8, ...
13
votes
4answers
8k views

Detect if any values in NVARCHAR columns are actually unicode

I have inherited some SQL Server databases. There is one table (I'll call "G"), with about 86.7 million rows, and 41 columns wide, from a source database (I'll call "Q") on SQL Server 2014 Standard ...
1
vote
1answer
398 views

sp_send_dbmail is populating 00 bits between all characters in a result set

I'm using sp_send_dbmail in a SQL agent job triggered nightly. The job queries our database and checks for product price updates, if there are some, it will send them in an email as an attachment to ...
0
votes
1answer
291 views

Character Strings related : string_units db cfg changed from SYSTEM to CODEUNITS32

To handle few special characters, we changed string_units db cfg parameter from default SYSTEM to CODEUNITS32. Based on IBM KC documentation, there are three methods to achieve this. 1.Set ...
1
vote
0answers
75 views

How is MySQL encoding the general.log

My MySQL 5.7 is running on UTF8 mysql> show variables like 'collation%'; +----------------------+-----------------+ | Variable_name | Value | +----------------------+--------------...
0
votes
1answer
276 views

Which mysql collation is good for storing unicode strings?

For all I know it must be utf8mb4_unicode_ci, but: > create table t1 (f1 varchar(255) collate utf8mb4_unicode_ci) > insert into t1 (f1) values ('𢄂') And what I get is: ERROR 1366 (22007) at ...
4
votes
2answers
7k views

Nvarchar to varchar to nvarchar characters conversion

I had a table filled with some greek text rows as nvarchar(2000). Lately, I altered the type of the column to varchar(4000) but I realized that some greek characters were shown as question marks. ...
2
votes
1answer
415 views

Why isn't the ASYMKEY_ID function returning the Asymmetric Key ID?

I have created database master key. Then I have created asymmetric key: CREATE ASYMMETRIC KEY smGK_АSymmetricKey_01 WITH ALGORITHM = RSA_512 In the documentation is said, that: PASSWORD = '...
2
votes
1answer
695 views

Supporting non-ascii characters in Postgres full text (english)

We're implementing Postgres full-text search. While the bulk of the text is normal English, there are a number of place names which include diacriticals (ñ,á,ó, etc) More specifically, there are ...
2
votes
4answers
828 views

Query on char with selector with N prefix is very slow

I have a table with a CHAR(36) "id" column, which is a clustered primary key. It has multiple other columns, including an "imdbId" column, which is an INT. When I update using UPDATE Movies SET ...
1
vote
1answer
2k views

Storing/Retrieving Chinese/Unicode in PL/SQL

We are on Oracle9i Enterprise Edition. I tried the following in PL/SQL. MANUFACTUERER Field is defined as VARCHAR2(75). I tried similar table with NVARCHAR2(75) instead, and same result. INSERT ...
2
votes
1answer
177 views

Tablediff utility output file encoding

When the SQL Server tablediff utility generates a file with differences between source and destination, if SSIS executes the script file which contains Arabic letters, then those letters are inserted ...
4
votes
1answer
5k views

Special Characters in MySQL using UTF8 Unicode Collation and LOAD DATA INFILE

I have several csv files in am loading into MySQl using Java. In the Description field I have several Special Characters that are causing the load to fail. I am using LOAD DATA INFILE. This is nested ...
19
votes
1answer
22k views

How do I set a SQL Server Unicode / NVARCHAR string to an emoji or Supplementary Character?

I want to set a Unicode string variable to particular character based on its Unicode code point. I want to use a code point beyond 65535, but the SQL Server 2008 R2 database has a collation of ...
3
votes
3answers
397 views

TSQL Script Execution Through Power shell, Special Characters Rendering Problem

We have a simple T-SQL script file (.sql), which we execute through Powershell. Everything works fine, except the script portion, where we are using some special characters as string (example: 'ô' ). ...