Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [unicode]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
2answers
40 views

Replace function removing Ones / 1 when removing char(185) (superscript 1)

Ok as the title suggests this is occuring. When using the REPLACE function on char(185) for data in a temporary table, ones ("1") are being removed as well. SQL SERVER 2008R2 UNICODE Data DB ...
4
votes
1answer
72 views

Importing Chinese characters via SSIS

I'm trying to import file that contains Chinese characters into a table. File encoding is Big5 (Traditional). Here is a sample file like one that I need to import: https://www.pastiebin.com/...
1
vote
1answer
335 views

How to replace multiple unicode characters

I have identified a couple of unicode characters in my database by means of the following query: SELECT name FROM table WHERE name NOT LIKE '%'+CAST(name AS VARCHAR(1000))+'%' ...with which I get ...
0
votes
1answer
28 views

Reusing literal string in different parts of query

I created this PostgreSQL query while troubleshooting a problem that turned out to be due to non-printing characters in my strings: with cte as ( select 'The character between the xes is evil: xx' as ...
3
votes
2answers
326 views

Storing unicode text into varchar column in SQL Server

I have mistakenly stored unicode text into a varchar column in a table and now it is shown as question marks (???). Is it possible to get our main information back from the column instead of question ...
-1
votes
1answer
660 views

Incorrect string value: '\xF0\x9F\xA4\x96<b…' for column 'body' at row 1 [closed]

I've set the column type to utf8mb4 and the collation to utf8mb4_bin. The table defaults are also utf8mb4 and utf8mb4_bin. When I use SequelPro to add the emoji "🤖" to the database manually, it ...
0
votes
1answer
114 views

Mariadb (MySQL) On Windows- problem entering non-ASCII characters in a query

MariaDB: mysql Ver 15.1 Distrib 10.3.8-MariaDB OS: Windows 7 Pro 64 bit I have my server database and client all set up (I believe) to support utf8 end to end- (**my.cnf**) # # Set everything to ...
4
votes
2answers
495 views

Byte ordering for multibyte characters in SQL Server versus Oracle

I am currently in the process of migrating data from Oracle to SQL Server and I'm encountering an issue trying to validate the data post-migration. Environment Details: Oracle 12 - AL32UTF8 ...
4
votes
3answers
428 views

Unicode storage of \u202b RLE and \u202c PDE in a Unicode-aware database?

I'm building a new product for toponyms and in it the Arabic shows kinda like this: ^IArabic^I<202b>ﺰﻤﺑﺎﺑﻮﻳ<202c>^I<202b>ﺞﻫﻭﺮﻳﺓ ﺰﻤﺑﺎﺑﻮﻳ<202c>$ Actually not quite. This is a ...
0
votes
1answer
41 views

How to import SQL file which has no N prefix

I have a large .sql file in Persian Language. The file is exported from sqlite db browser and has no N prefix for unicode fields. How can I reuse it in Sql Server? This is sample of insert query in ...
1
vote
1answer
403 views

How to insert a Unicode character verbose into a varchar DB?

I need to insert this character '●' into a VARCHAR column of a MSSQL database with collation set as SQL_Latin1_General_CP1_CI_AS (or at least mock what my Python + Windows MSSQL Driver might have done)...
2
votes
1answer
230 views

How to send Unicode symbols using SQL Server Database Mail? [closed]

How can I send Unicode symbols using SQL Server Database Mail? My goal is to send a symbol on the subject of an email. Let's say that, for example, I want to send this "🚯 Subject with symbol" on the ...
10
votes
2answers
1k views

When `nvarchar/nchar` is going to be used with SQL Server 2019?

With SQL Server 2019 Microsoft introduces UTF-8 support for CHAR and VARCHAR data types and says: This feature may provide significant storage savings, depending on the character set in use. For ...
4
votes
1answer
973 views

Querying special character from SQL Server

When I saved special characters in a column of type nvarchar, I did not use the prefix 'N', so the characters got saved as ??. Is there any way to retrieve the original special characters from these ...
4
votes
1answer
4k views

Why do I get incorrect characters when decoding a Base64 string to NVARCHAR in SQL Server?

I've been looking at how to decode Base64 with SQL Server and after searching around many solutions online (some from here) seem to be based upon this sort of method. SELECT CAST(CAST('...
4
votes
1answer
219 views

Why isn't Unicode character REPLACEd in some cases?

Running this results in "test", as expected: SELECT REPLACE(NCHAR(1234), NCHAR(1234), N'test'); However, running this results in "aӒa", which doesn't have "test" in it: SELECT REPLACE(N'a' +...
3
votes
1answer
4k views

inserting a special character into a table

I have a few characters that are being replaced by a ?. How do you keep the original character in the table? The character that I am trying to insert is a ɬ; Latin Small Letter L With Belt. Is it ...
1
vote
1answer
37 views

Can't set up Unicode support

I am creating a toy website locally with XAMPP and I want it to support Unicode. index.php contains a simple form that I can use to add a new entry (a UTF-8 string) to a table "myTable" in the DB. ...
5
votes
2answers
3k views

SQL Server Bulk Insert properly interprets some Unicode characters but not others?

For some reason the MS SQL Server 2016 bulk insert misinterprets/translates Unicode characters: C9 (É) into 2B (+) A1 (¡) into ED (í) A0 ( ) into E1 (á) AE (®) into AB («) CB (Ë) into 2D (-) D1 (...
0
votes
1answer
152 views

sp_executesql changes non standard characters to?

Am trying to use sp_executesql to run commands but I've come across an issue where in non-english characters (aka Chinese & Japanese) are changed to ??? when ran. sql server 2008R2 & 2012 ...
53
votes
2answers
5k views

How to create Unicode parameter and variable names

All of this works: CREATE DATABASE [¯\_(ツ)_/¯]; GO USE [¯\_(ツ)_/¯]; GO CREATE SCHEMA [¯\_(ツ)_/¯]; GO CREATE TABLE [¯\_(ツ)_/¯].[¯\_(ツ)_/¯]([¯\_(ツ)_/¯] NVARCHAR(20)); GO CREATE UNIQUE CLUSTERED INDEX [¯...
5
votes
2answers
1k views

Why is my UTF-8 document raising UTF-8 encoding errors in Azure Data Lake Analytics?

I have a document that was compressed in gunzip from a unknown source system. It was downloaded and decompressed using a 7zip console application. The document is a CSV file that appears to be encoded ...
19
votes
2answers
498 views

Accent Sensitive Sort

Why do these two SELECT statements result in a different sort order? USE tempdb; CREATE TABLE dbo.OddSort ( id INT IDENTITY(1,1) PRIMARY KEY , col1 NVARCHAR(2) , col2 NVARCHAR(2) ); GO ...
2
votes
1answer
267 views

pgAdmin 4 including <U+FEFF> in all copied strings

I'm having trouble using copy/paste from pgAdmin 4. The symptom is that I get the character <U+FEFF> aka Zero Width No-break Space aka Byte Order Mark (BOM) prepended to any string I copy. The ...
2
votes
1answer
3k views

ERROR: “sql ” is not a known variable

This must one of the dumbest questions I've ever asked here, but there must something really sick hiding in my SQL script that is blocking it from running. I'm invoking the cobertura.sql file using ...
17
votes
4answers
8k views

Why does the varchar datatype allow unicode values?

I have a table with a varchar column. It is allowing Trademark(™), copyright(©) and other Unicode characters as shown below. Create table VarcharUnicodeCheck ( col1 varchar(100) ) insert into ...
2
votes
2answers
3k views

Remove characters from PostgreSQL database causing encoding errors

I have a PostgreSQL 10 database that uses WIN1252 encoding. One of my columns has values that cause conversion errors when running a select from pgAdmin 4: SELECT myfield FROM mydb.myschema."MYTABLE"...
0
votes
1answer
41 views

Special letter behaviour problem?

I asked question in very blurry manner because I cannot explain in short sentence what is my problem about. In general I have a set of procedures which transforms text and XML. Inside the procedures ...
1
vote
1answer
1k views

Using LIKE to search case insensitive over language specific characters (PostgreSQL)

Using a PostgreSQL database (Collation C, Encoding UTF8) we store data from various languages. This means we also get the special characters for free (e.g. Å, å, ...) Within an application we now ...
18
votes
2answers
3k views

Can't update “CO2” to “CO₂” in table row

Given this table: CREATE TABLE test ( id INT NOT NULL, description NVARCHAR(100) COLLATE Modern_Spanish_CI_AS NOT NULL ); INSERT INTO test (id, description) VALUES (1, 'CO2'); I've realised ...
3
votes
1answer
5k views

How to be sure that a SQL Script is running with ANSI encoding?

Maybe this is a stupid question or I'm asking it in the wrong way. How can I be sure that a script ( with thousands of lines ) is running with ANSI encoding? Let's say we created a script using ...
5
votes
0answers
2k views

VARCHAR(MAX) taking more space than NVARCHAR(MAX) [duplicate]

I have a table in SQL Server 2012 Enterprise Edition that has ~100GB of data. Most of this data is attributed to a single NVARCHAR(MAX) column which has an average length of 4500 characters. It stores ...
0
votes
2answers
2k views

Spool Unicode data in sqlplus to ISO-client without character set conversion

I'm currently facing the following problem: We have an Oracle 11g database with character set AL32UTF8, and an sqlplus-client with NLS_LANG=GERMAN_GERMANY.WE8ISO8859P1. Stored in the database are ...
5
votes
2answers
934 views

Problematic slashes and questions marks in unique index values

What is it about the Phone values inserted below that SQL Server is treating them as identical in a unique index? CREATE TABLE Phone ( Id int identity(1, 1) primary key, Phone nvarchar(448) not ...
5
votes
1answer
2k views

How do I list all available encoding types available in postgreSQL?

SELECT datname, pg_encoding_to_char(encoding) FROM pg_database; …lists all the databases, each with its encoding type. However, I am trying to find out what are all the encoding types available in ...
2
votes
1answer
180 views

SELECT query: filter unable to distinguish between two different characters

In a recent support case, an error was traced to the difference between two similar-looking characters: ― (Unicode U+2015) and ー (Unicode U+30FC). A select statement with a WHERE clause was unable to ...
8
votes
2answers
1k views

Why does Oracle use a different byte length than java for the supplementary unicode character chipmunk?

I have java code trimming a UTF-8 string to the size of my Oracle (11.2.0.4.0) column which ends up throwing an error because java and Oracle see the string as different byte lengths. I've verified my ...
2
votes
1answer
117 views

International Components for Unicode (ICU) Date Formatter for MySQL?

Regarding to ICU - International Components for Unicode definition: ICU is a mature, widely used set of C/C++ and Java libraries providing Unicode and Globalization support for software ...
2
votes
1answer
7k views

ssis cannot convert between unicode and non-unicode data types even though the source and the destination columns are both unicode

I faced an error: 'SSIS cannot convert between unicode and non-unicode string data types' even though the source and the destination columns are both unicode (NVARCHAR) data type. The source db and ...
20
votes
1answer
6k views

Why does searching for LIKE N'%�%' match any Unicode character and = N'�' match many?

DECLARE @T TABLE( Col NCHAR(1)); INSERT INTO @T VALUES (N'A'), (N'B'), (N'C'), (N'Ƕ'), (N'Ƿ'), (N'Ǹ'); SELECT * FROM @T WHERE ...
0
votes
1answer
630 views

How to handle the junk characters upon conversion of the data from image datatype to varchar?

We are trying to convert the column of type image to column of type varchar(max). We tried the below way to do the conversion. --col1 is of type image --col2 is of type varchar(max) set col2 = ...
14
votes
1answer
9k views

Querying non-ASCII rows from Postgres

Does [:ascii:] class work in Postgres at all? It is not listed in their help, however I see examples in the web which utilize it. I have a UTF-8 database, where collation and c_type are en_US.UTF-8, ...
14
votes
4answers
11k views

Detect if any values in NVARCHAR columns are actually unicode

I have inherited some SQL Server databases. There is one table (I'll call "G"), with about 86.7 million rows, and 41 columns wide, from a source database (I'll call "Q") on SQL Server 2014 Standard ...
1
vote
1answer
452 views

sp_send_dbmail is populating 00 bits between all characters in a result set

I'm using sp_send_dbmail in a SQL agent job triggered nightly. The job queries our database and checks for product price updates, if there are some, it will send them in an email as an attachment to ...
0
votes
1answer
395 views

Character Strings related : string_units db cfg changed from SYSTEM to CODEUNITS32

To handle few special characters, we changed string_units db cfg parameter from default SYSTEM to CODEUNITS32. Based on IBM KC documentation, there are three methods to achieve this. 1.Set ...
1
vote
0answers
88 views

How is MySQL encoding the general.log

My MySQL 5.7 is running on UTF8 mysql> show variables like 'collation%'; +----------------------+-----------------+ | Variable_name | Value | +----------------------+--------------...
0
votes
1answer
322 views

Which mysql collation is good for storing unicode strings?

For all I know it must be utf8mb4_unicode_ci, but: > create table t1 (f1 varchar(255) collate utf8mb4_unicode_ci) > insert into t1 (f1) values ('𢄂') And what I get is: ERROR 1366 (22007) at ...
4
votes
2answers
7k views

Nvarchar to varchar to nvarchar characters conversion

I had a table filled with some greek text rows as nvarchar(2000). Lately, I altered the type of the column to varchar(4000) but I realized that some greek characters were shown as question marks. ...
2
votes
1answer
559 views

Why isn't the ASYMKEY_ID function returning the Asymmetric Key ID?

I have created database master key. Then I have created asymmetric key: CREATE ASYMMETRIC KEY smGK_АSymmetricKey_01 WITH ALGORITHM = RSA_512 In the documentation is said, that: PASSWORD = '...
2
votes
1answer
825 views

Supporting non-ascii characters in Postgres full text (english)

We're implementing Postgres full-text search. While the bulk of the text is normal English, there are a number of place names which include diacriticals (ñ,á,ó, etc) More specifically, there are ...