Skip to main content

Questions tagged [unicode]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
1 answer
1k views

Mariadb (MySQL) On Windows- problem entering non-ASCII characters in a query

MariaDB: mysql Ver 15.1 Distrib 10.3.8-MariaDB OS: Windows 7 Pro 64 bit I have my server database and client all set up (I believe) to support utf8 end to end- (**my.cnf**) # # Set everything to ...
1 vote
0 answers
80 views

MySQL 8.0 fails to store 5.7-allowed symbols (Incorrect string value, Error 1366) [closed]

I'm upgrading MySQL 5.7 server to 8.0 with a bunch of user-input data that sometimes contain "malformed" unicode characters like x'eda0bdedb98f'. The latter seems to be 2 UTF-16 characters ...
1 vote
1 answer
864 views

sp_send_dbmail is populating 00 bits between all characters in a result set

I'm using sp_send_dbmail in a SQL agent job triggered nightly. The job queries our database and checks for product price updates, if there are some, it will send them in an email as an attachment to ...
4 votes
2 answers
11k views

Nvarchar to varchar to nvarchar characters conversion

I had a table filled with some greek text rows as nvarchar(2000). Lately, I altered the type of the column to varchar(4000) but I realized that some greek characters were shown as question marks. ...
2 votes
1 answer
4k views

How to insert a Unicode character verbose into a varchar DB?

I need to insert this character '●' into a VARCHAR column of a MSSQL database with collation set as SQL_Latin1_General_CP1_CI_AS (or at least mock what my Python + Windows MSSQL Driver might have done)...
1 vote
1 answer
968 views

Character Strings related : string_units db cfg changed from SYSTEM to CODEUNITS32

To handle few special characters, we changed string_units database configuration parameter from default SYSTEM to CODEUNITS32. Based on IBM DB2 Knowledge Center, there are three methods to achieve ...
1 vote
1 answer
1k views

How to display non-English character correctly in db2cmd?

On Db2 v11.5.7 on Linux/x86_64 I have a UTF-8 database. Executing db2 get db cfg for test1 returns: Database code page = 1208 Database code set = UTF-8 On my Windows 10 computer in ...
0 votes
1 answer
518 views

Generating MD5 Hash for NVARCHAR and VARCHAR columns across different databases

I am building a utility that generates a single MD5 hash across all CONCATENATED column values for any table row. To make this work, I have to eliminate NULL values using COALESCE and must CAST NUMBER ...
0 votes
1 answer
537 views

sp_executesql changes non standard characters to?

Am trying to use sp_executesql to run commands but I've come across an issue where in non-english characters (aka Chinese & Japanese) are changed to ??? when ran. sql server 2008R2 & 2012 ...
9 votes
0 answers
712 views

Inconsistent equality checking of Unicode text in Postgres 9.5?

Using PostgreSQL 9.5. It seems like Postgres is being inconsistent with how it compares strings similar to لى. A unique constraint is considering some strings equal that GROUP BY is considering ...
5 votes
2 answers
23k views

Why do I get incorrect characters when decoding a Base64 string to NVARCHAR in SQL Server?

I've been looking at how to decode Base64 with SQL Server and after searching around many solutions online (some from here) seem to be based upon this sort of method. SELECT CAST(CAST('...
6 votes
2 answers
2k views

Byte ordering for multibyte characters in SQL Server versus Oracle

I am currently in the process of migrating data from Oracle to SQL Server and I'm encountering an issue trying to validate the data post-migration. Environment Details: Oracle 12 - AL32UTF8 ...
3 votes
2 answers
13k views

SQL Server error JSON text is not properly formatted. Unexpected character '?' is found at position 21

I am getting below error while parsing any JSON string with special character or emoji. How to parse this? JSON text is not properly formatted. Unexpected character '?' is found at position 21. ...
4 votes
1 answer
3k views

Importing Chinese characters via SSIS

I'm trying to import file that contains Chinese characters into a table. File encoding is Big5 (Traditional). Here is a sample file like one that I need to import: https://www.pastiebin.com/...
0 votes
1 answer
444 views

Unable to translate Unicode character \\uDCC9 at index 5 to specified code page

I'm using MariaDB (and Adminer) MySQL version: 5.5.5-10.10.2-MariaDB-1:10.10.2+maria~ubu2204 through PHP extension MySQLi And this is the show create table Pages: CREATE TABLE `Pages` ( `Id` ...
4 votes
1 answer
610 views

Fix corrupted utf8 nvarchar value that was inserted without N prefix

Is it possible to restore the UTF8 data from a corrupted nvarchar, that was used without the N prefix? For example, in the following snippet, I'd like @n1_fixed to have the correct value based on @n1: ...
1 vote
1 answer
14k views

How to replace multiple unicode characters

I have identified a couple of unicode characters in my database by means of the following query: SELECT name FROM table WHERE name NOT LIKE '%'+CAST(name AS VARCHAR(1000))+'%' ...with which I get ...
35 votes
1 answer
48k views

How do I set a SQL Server Unicode / NVARCHAR string to an emoji or Supplementary Character?

I want to set a Unicode string variable to particular character based on its Unicode code point. I want to use a code point beyond 65535, but the SQL Server 2008 R2 database has a collation of ...
1 vote
1 answer
2k views

How to fix unicode REPLACEMENT CHARACTER � in postgres database

Is there a way fix the � problem in postgres database: As you can see there are names with accents and are not displayed corretly in the database. I would like to know if there is a way to fix this ...
1 vote
2 answers
2k views

unicode symbols for entity-relationship

What are the Unicode symbols that most closely relate to the symbols used in entity-relationship diagrams, such as the ones in the screenshot of the OP here: https://stackoverflow.com/questions/...
0 votes
1 answer
331 views

Detecting whether a varchar value is already encrypted by EncryptByPassPhrase() in SQL Server

Starting with a varchar(100) column named Password in a table Users with one record in it, after performing an update on the column to encrypt it with EncryptByPassPhrase() on the Password column the ...
1 vote
2 answers
3k views

Storing/Retrieving Chinese/Unicode in PL/SQL

We are on Oracle9i Enterprise Edition. I tried the following in PL/SQL. MANUFACTUERER Field is defined as VARCHAR2(75). I tried similar table with NVARCHAR2(75) instead, and same result. INSERT ...
0 votes
0 answers
2k views

Convert VARCHAR2 to UTF-8 in Oracle 12

I need to convert VARCHAR2 columns that store text in unkown character sets into a standard UTF-8 BLOB. The table has more columns, but the ones of interest to me are: ADDRESS POST ---------- ----...
0 votes
1 answer
116 views

Producing JSON in Oracle 12c1

I need to produce a JSON document with several properties from a list of Oracle 12c1 production databases, and I was given read-only access in order to generate this data. The problem is these ...
1 vote
1 answer
233 views

Datapump metadata-only import changes datalength when importing with conversion

Datapump metadata only import changes datalength when importing with conversion: (export done in WE8ISO8859P15 and import done in AL32UTF8 character set and AL16UTF16 NCHAR character set) Example ...
14 votes
1 answer
1k views

Why does a comparison between 'tr' & 'tR' fail on a SQL Server with Vietnamese_CI_AI collation?

There seems to be something special about 'tR' in Vietnamese collation. Appreicate if anyone who knows about it can explain in simple terms. This issue was discovered during the installation of our ...
4 votes
1 answer
913 views

What is difference between Arabic_100_CS_AS_KS_WS_SC_UTF8 and Latin1_General_100_CS_AS_KS_WS_SC_UTF8?

Starting from SQL Server 2019, it supports UTF-8 as collation. However, according to the following queries: SELECT COLLATIONPROPERTY('Arabic_100_CS_AS_KS_WS_SC_UTF8', 'CodePage') SELECT ...
7 votes
2 answers
36k views

Storing Japanese characters in a table

I am using SQL Server 2008 R2 and I want to store Japanese characters in one of the columns of my table. Say I want to store Japanese names, how can I achieve that? Is there an easy way?
19 votes
5 answers
22k views

Why does the varchar datatype allow unicode values?

I have a table with a varchar column. It is allowing Trademark(™), copyright(©) and other Unicode characters as shown below. Create table VarcharUnicodeCheck ( col1 varchar(100) ) insert into ...
2 votes
2 answers
2k views

How to create a new column in SELECT CASE when string contains Arabic words?

I have a problem with a select case when statement, I would like to add a new column in the select case when statement, I got the result but with ????? because I have set the column to Arabic words, I ...
3 votes
2 answers
14k views

How do I fix the Code Page in SSIS Lookup Transformation to be 65001?

I have an SQL server 2019 with databases and tables all set with the collation Latin1_General_100_CI_AS_SC_UTF8. The relevant table has code and desc columns which are both of data type varchar. In ...
3 votes
1 answer
11k views

Unicode error with \u0000 on COPY of large JSON file into Postgres

I'm using Postgres 9.4 on Ubuntu 14.04. I have a 30GB JSON file that I'm trying to COPY into Postgres. But I keep getting the following error: COPY <table>(comment_jsonb) FROM '<json file&...
0 votes
1 answer
1k views

How to pattern match against Unicode characters by PostgreSQL `SIMILAR TO` expressions?

In a PostgreSQL database, I have a column named metaid of type text whose value contains digits as specified in the third edition of XML 1.0 standard: Digit ::= [#x0030-#x0039] | [#x0660-#x0669] | [#...
8 votes
2 answers
6k views

How do I list all available encoding types available in postgreSQL?

SELECT datname, pg_encoding_to_char(encoding) FROM pg_database; …lists all the databases, each with its encoding type. However, I am trying to find out what are all the encoding types available in ...
5 votes
2 answers
5k views

Why is my UTF-8 document raising UTF-8 encoding errors in Azure Data Lake Analytics?

I have a document that was compressed in gunzip from a unknown source system. It was downloaded and decompressed using a 7zip console application. The document is a CSV file that appears to be encoded ...
3 votes
2 answers
1k views

How to convert xml entities to unicode characters (with read-only access) in Oracle?

I'm connected to an Oracle Database (11g Release 2 - 11.2.0.4), with read-only access. Into this database, some of the data is uploaded via, or rather as, XML - and quite a few entries contain ...
0 votes
1 answer
79 views

Special letter behaviour problem?

I asked question in very blurry manner because I cannot explain in short sentence what is my problem about. In general I have a set of procedures which transforms text and XML. Inside the procedures ...
11 votes
1 answer
11k views

PostgreSQL nondeterministic collations are not supported for LIKE

I am using Postgresql v12. I created a collation like this: CREATE COLLATION ci (provider = icu, locale = 'tr_TR', deterministic = false); I used that collation in a table: create table testtable1 ...
6 votes
1 answer
4k views

What is the meaning of "-x-icu" in PostgreSQL's "collates"?

I had this query: SELECT * FROM table ORDER BY label ASC; Since the labels are not in English, they didn't get sorted in the right order (ones beginning with "ö" were not in the bottom/end). I ...
8 votes
2 answers
12k views

What is the SQL Server equivalent of MySQL's utf8mb4 character set?

We have some web applications based on databases using utf8mb4 as character set and utf8mb4_Standard as collation. We saw that we can use any character we want with this setting. In SQL Server ...
0 votes
1 answer
1k views

Accent marks/symbols not displaying correctly in MySQL database

Anytime there's an accent mark (é or ') or symbol (&) in my database it displays as HTML code. The table CHARACTER SET is UTF-8 Unicode and the COLLATION is utf8mb4_general_ci, which should ...
13 votes
1 answer
18k views

Text string stored in SQLite Integer column?

I'm a database novice looking at an SQLite database which appears to be storing text in an integer column. Here's an example session at the sqlite3 command line: sqlite> .schema mytable CREATE ...
3 votes
1 answer
3k views

Is there some function in PostgreSQL to simply check if the input string is valid UTF-8 according to PG?

For several years now, I've been hunting this extremely quick-footed little bug which knows every hiding place in my metaphorical coding house. He lurks in the walls, sometimes takes months to show ...
26 votes
1 answer
18k views

Why does searching for LIKE N'%�%' match any Unicode character and = N'�' match many?

DECLARE @T TABLE( Col NCHAR(1)); INSERT INTO @T VALUES (N'A'), (N'B'), (N'C'), (N'Ƕ'), (N'Ƿ'), (N'Ǹ'); SELECT * FROM @T WHERE ...
3 votes
1 answer
1k views

Where can I find exact rules of a collation?

Is there any documentation anywhere for the exact sorting/equality rules defined by a collation in Windows (SQL Server), for instance "Finnish_Swedish_CI_AS"? I know about CI/CS, AI/AS etc, but I'm ...
1 vote
2 answers
5k views

Spool Unicode data in sqlplus to ISO-client without character set conversion

I'm currently facing the following problem: We have an Oracle 11g database with character set AL32UTF8, and an sqlplus-client with NLS_LANG=GERMAN_GERMANY.WE8ISO8859P1. Stored in the database are ...
20 votes
1 answer
25k views

Querying non-ASCII rows from Postgres

Does [:ascii:] class work in Postgres at all? It is not listed in their help, however I see examples in the web which utilize it. I have a UTF-8 database, where collation and c_type are en_US.UTF-8, ...
-1 votes
1 answer
240 views

ORACLE DMU (database migration assistant for unicode) Tool

Reading the documentation for Oracle DMU tool, when I try the step create a step for DMU Repository tablespace the document says to use the result of the query for initial size. However; I could not ...
1 vote
2 answers
242 views

Replace function removing Ones / 1 when removing char(185) (superscript 1)

Ok as the title suggests this is occuring. When using the REPLACE function on char(185) for data in a temporary table, ones ("1") are being removed as well. SQL SERVER 2008R2 UNICODE Data DB ...
15 votes
4 answers
20k views

Detect if any values in NVARCHAR columns are actually unicode

I have inherited some SQL Server databases. There is one table (I'll call "G"), with about 86.7 million rows, and 41 columns wide, from a source database (I'll call "Q") on SQL Server 2014 Standard ...