Questions tagged [unicode]
The unicode tag has no usage guidance.
104
questions
0
votes
1
answer
120
views
Generating MD5 Hash for NVARCHAR and VARCHAR columns across different databases
I am building a utility that generates a single MD5 hash across all CONCATENATED column values for any table row. To make this work, I have to eliminate NULL values using COALESCE and must CAST NUMBER ...
- 69
2
votes
2
answers
7k
views
SQL Server error JSON text is not properly formatted. Unexpected character '?' is found at position 21
I am getting below error while parsing any JSON string with special character or emoji. How to parse this?
JSON text is not properly formatted. Unexpected character '?' is found at position 21.
...
- 21
0
votes
1
answer
318
views
Unable to translate Unicode character \\uDCC9 at index 5 to specified code page
I'm using MariaDB (and Adminer)
MySQL version: 5.5.5-10.10.2-MariaDB-1:10.10.2+maria~ubu2204 through PHP extension MySQLi
And this is the show create table Pages:
CREATE TABLE `Pages` (
`Id` ...
- 1,407
4
votes
1
answer
571
views
Fix corrupted utf8 nvarchar value that was inserted without N prefix
Is it possible to restore the UTF8 data from a corrupted nvarchar, that was used without the N prefix?
For example, in the following snippet, I'd like @n1_fixed to have the correct value based on @n1:
...
- 437
1
vote
1
answer
687
views
How to display non-English character correctly in db2cmd?
On Db2 v11.5.7 on Linux/x86_64 I have a UTF-8 database.
Executing db2 get db cfg for test1 returns:
Database code page = 1208
Database code set = UTF-8
On my Windows 10 computer in ...
- 303
0
votes
0
answers
133
views
What is the unicode version of UTF8 encoding in PostgreSQL?
As mentioned in Oracle document, AL32UTF8 character set in Oracle has different unicode version compliance in diffrent database version.
But there's no such indication in PostgreSQL document for UTF8 ...
1
vote
1
answer
1k
views
How to fix unicode REPLACEMENT CHARACTER � in postgres database
Is there a way fix the � problem in postgres database:
As you can see there are names with accents and are not displayed corretly in the database.
I would like to know if there is a way to fix this ...
- 195
0
votes
1
answer
271
views
Detecting whether a varchar value is already encrypted by EncryptByPassPhrase() in SQL Server
Starting with a varchar(100) column named Password in a table Users with one record in it, after performing an update on the column to encrypt it with EncryptByPassPhrase() on the Password column the ...
- 153
0
votes
0
answers
1k
views
Convert VARCHAR2 to UTF-8 in Oracle 12
I need to convert VARCHAR2 columns that store text in unkown character sets into a standard UTF-8 BLOB.
The table has more columns, but the ones of interest to me are:
ADDRESS POST
---------- ----...
- 101
0
votes
1
answer
90
views
Producing JSON in Oracle 12c1
I need to produce a JSON document with several properties from a list of Oracle 12c1 production databases, and I was given read-only access in order to generate this data.
The problem is these ...
- 101
1
vote
1
answer
121
views
Datapump metadata-only import changes datalength when importing with conversion
Datapump metadata only import changes datalength when importing with conversion: (export done in WE8ISO8859P15 and import done in AL32UTF8 character set and AL16UTF16 NCHAR character set)
Example ...
- 978
14
votes
1
answer
1k
views
Why does a comparison between 'tr' & 'tR' fail on a SQL Server with Vietnamese_CI_AI collation?
There seems to be something special about 'tR' in Vietnamese collation. Appreicate if anyone who knows about it can explain in simple terms.
This issue was discovered during the installation of our ...
- 373
4
votes
1
answer
734
views
What is difference between Arabic_100_CS_AS_KS_WS_SC_UTF8 and Latin1_General_100_CS_AS_KS_WS_SC_UTF8?
Starting from SQL Server 2019, it supports UTF-8 as collation. However, according to the following queries:
SELECT COLLATIONPROPERTY('Arabic_100_CS_AS_KS_WS_SC_UTF8', 'CodePage')
SELECT ...
- 143
2
votes
2
answers
1k
views
How to create a new column in SELECT CASE when string contains Arabic words?
I have a problem with a select case when statement, I would like to add a new column in the select case when statement, I got the result but with ????? because I have set the column to Arabic words, I ...
- 61
3
votes
2
answers
11k
views
How do I fix the Code Page in SSIS Lookup Transformation to be 65001?
I have an SQL server 2019 with databases and tables all set with the collation Latin1_General_100_CI_AS_SC_UTF8.
The relevant table has code and desc columns which are both of data type varchar.
In ...
1
vote
2
answers
1k
views
unicode symbols for entity-relationship
What are the Unicode symbols that most closely relate to the symbols used in entity-relationship diagrams, such as the ones in the screenshot of the OP here: https://stackoverflow.com/questions/...
- 134
0
votes
1
answer
786
views
How to pattern match against Unicode characters by PostgreSQL `SIMILAR TO` expressions?
In a PostgreSQL database, I have a column named metaid of type text whose value contains digits as specified in the third edition of XML 1.0 standard:
Digit ::= [#x0030-#x0039] | [#x0660-#x0669] | [#...
- 103
3
votes
2
answers
1k
views
How to convert xml entities to unicode characters (with read-only access) in Oracle?
I'm connected to an Oracle Database (11g Release 2 - 11.2.0.4), with read-only access.
Into this database, some of the data is uploaded via, or rather as, XML - and quite a few entries contain ...
9
votes
1
answer
8k
views
PostgreSQL nondeterministic collations are not supported for LIKE
I am using Postgresql v12.
I created a collation like this:
CREATE COLLATION ci (provider = icu, locale = 'tr_TR', deterministic = false);
I used that collation in a table:
create table testtable1 ...
- 329
0
votes
1
answer
1k
views
Accent marks/symbols not displaying correctly in MySQL database
Anytime there's an accent mark (é or ') or symbol (&) in my database it displays as HTML code. The table CHARACTER SET is UTF-8 Unicode and the COLLATION is utf8mb4_general_ci, which should ...
- 25
3
votes
1
answer
2k
views
Is there some function in PostgreSQL to simply check if the input string is valid UTF-8 according to PG?
For several years now, I've been hunting this extremely quick-footed little bug which knows every hiding place in my metaphorical coding house. He lurks in the walls, sometimes takes months to show ...
- 31
5
votes
1
answer
3k
views
What is the meaning of "-x-icu" in PostgreSQL's "collates"?
I had this query:
SELECT * FROM table ORDER BY label ASC;
Since the labels are not in English, they didn't get sorted in the right order (ones beginning with "ö" were not in the bottom/end).
I ...
-1
votes
1
answer
221
views
ORACLE DMU (database migration assistant for unicode) Tool
Reading the documentation for Oracle DMU tool, when I try the step create a step for DMU Repository tablespace the document says to use the result of the query for initial size. However; I could not ...
- 35
1
vote
2
answers
230
views
Replace function removing Ones / 1 when removing char(185) (superscript 1)
Ok as the title suggests this is occuring.
When using the REPLACE function on char(185) for data in a temporary table, ones ("1") are being removed as well.
SQL SERVER 2008R2
UNICODE Data
DB ...
- 349
4
votes
1
answer
3k
views
Importing Chinese characters via SSIS
I'm trying to import file that contains Chinese characters into a table. File encoding is Big5 (Traditional).
Here is a sample file like one that I need to import: https://www.pastiebin.com/...
- 292
1
vote
1
answer
12k
views
How to replace multiple unicode characters
I have identified a couple of unicode characters in my database by means of the following query:
SELECT name FROM table WHERE name NOT LIKE '%'+CAST(name AS VARCHAR(1000))+'%'
...with which I get ...
user151179
0
votes
1
answer
396
views
Reusing literal string in different parts of query
I created this PostgreSQL query while troubleshooting a problem that turned out to be due to non-printing characters in my strings:
with cte as ( select 'The character between the xes is evil: xx' as ...
- 347
3
votes
2
answers
4k
views
Storing unicode text into varchar column in SQL Server
I have mistakenly stored unicode text into a varchar column in a table and now it is shown as question marks (???).
Is it possible to get our main information back from the column instead of question ...
-1
votes
1
answer
3k
views
Incorrect string value: '\xF0\x9F\xA4\x96<b...' for column 'body' at row 1 [closed]
I've set the column type to utf8mb4 and the collation to utf8mb4_bin.
The table defaults are also utf8mb4 and utf8mb4_bin.
When I use SequelPro to add the emoji "🤖" to the database manually, it ...
- 99
1
vote
1
answer
1k
views
Mariadb (MySQL) On Windows- problem entering non-ASCII characters in a query
MariaDB: mysql Ver 15.1 Distrib 10.3.8-MariaDB
OS: Windows 7 Pro 64 bit
I have my server database and client all set up (I believe) to support utf8 end to end-
(**my.cnf**)
#
# Set everything to ...
- 11
6
votes
2
answers
2k
views
Byte ordering for multibyte characters in SQL Server versus Oracle
I am currently in the process of migrating data from Oracle to SQL Server and I'm encountering an issue trying to validate the data post-migration.
Environment Details:
Oracle 12 - AL32UTF8 ...
- 9,842
4
votes
3
answers
1k
views
Unicode storage of \u202b RLE and \u202c PDE in a Unicode-aware database?
I'm building a new product for toponyms and in it the Arabic shows kinda like this:
^IArabic^I<202b>ﺰﻤﺑﺎﺑﻮﻳ<202c>^I<202b>ﺞﻫﻭﺮﻳﺓ ﺰﻤﺑﺎﺑﻮﻳ<202c>$
Actually not quite. This is a ...
- 61.5k
0
votes
1
answer
184
views
How to import SQL file which has no N prefix
I have a large .sql file in Persian Language. The file is exported from sqlite db browser and has no N prefix for unicode fields. How can I reuse it in Sql Server?
This is sample of insert query in ...
- 231
2
votes
1
answer
3k
views
How to insert a Unicode character verbose into a varchar DB?
I need to insert this character '●' into a VARCHAR column of a MSSQL database with collation set as SQL_Latin1_General_CP1_CI_AS (or at least mock what my Python + Windows MSSQL Driver might have done)...
- 837
2
votes
1
answer
1k
views
How to send Unicode symbols using SQL Server Database Mail? [closed]
How can I send Unicode symbols using SQL Server Database Mail? My goal is to send a symbol on the subject of an email. Let's say that, for example, I want to send this "🚯 Subject with symbol" on the ...
12
votes
2
answers
3k
views
When `nvarchar/nchar` is going to be used with SQL Server 2019?
With SQL Server 2019 Microsoft introduces UTF-8 support for CHAR and VARCHAR data types and says:
This feature may provide significant storage savings, depending on the
character set in use. For ...
- 3,741
4
votes
1
answer
4k
views
Querying special character from SQL Server
When I saved special characters in a column of type nvarchar, I did not use the prefix 'N', so the characters got saved as ??.
Is there any way to retrieve the original special characters from these ...
- 41
5
votes
2
answers
20k
views
Why do I get incorrect characters when decoding a Base64 string to NVARCHAR in SQL Server?
I've been looking at how to decode Base64 with SQL Server and after searching around many solutions online (some from here) seem to be based upon this sort of method.
SELECT CAST(CAST('...
- 153
4
votes
1
answer
1k
views
Why isn't Unicode character REPLACEd in some cases?
Running this results in "test", as expected:
SELECT
REPLACE(NCHAR(1234), NCHAR(1234), N'test');
However, running this results in "aӒa", which doesn't have "test" in it:
SELECT
REPLACE(N'a' +...
- 1,853
4
votes
1
answer
12k
views
inserting a special character into a table
I have a few characters that are being replaced by a ?. How do you keep the original character in the table? The character that I am trying to insert is a ɬ; Latin Small Letter L With Belt. Is it ...
- 177
1
vote
1
answer
213
views
Can't set up Unicode support
I am creating a toy website locally with XAMPP and I want it to support Unicode.
index.php contains a simple form that I can use to add a new entry (a UTF-8 string) to a table "myTable" in the DB. ...
- 111
5
votes
2
answers
10k
views
SQL Server Bulk Insert properly interprets some Unicode characters but not others?
For some reason the MS SQL Server 2016 bulk insert misinterprets/translates Unicode characters:
C9 (É) into 2B (+)
A1 (¡) into ED (í)
A0 ( ) into E1 (á)
AE (®) into AB («)
CB (Ë) into 2D (-)
D1 (...
- 323
0
votes
1
answer
465
views
sp_executesql changes non standard characters to?
Am trying to use sp_executesql to run commands but I've come across an issue where in non-english characters (aka Chinese & Japanese) are changed to ??? when ran.
sql server 2008R2 & 2012
...
- 2,041
53
votes
2
answers
6k
views
How to create Unicode parameter and variable names
All of this works:
CREATE DATABASE [¯\_(ツ)_/¯];
GO
USE [¯\_(ツ)_/¯];
GO
CREATE SCHEMA [¯\_(ツ)_/¯];
GO
CREATE TABLE [¯\_(ツ)_/¯].[¯\_(ツ)_/¯]([¯\_(ツ)_/¯] NVARCHAR(20));
GO
CREATE UNIQUE CLUSTERED INDEX [¯...
- 42.7k
5
votes
2
answers
5k
views
Why is my UTF-8 document raising UTF-8 encoding errors in Azure Data Lake Analytics?
I have a document that was compressed in gunzip from a unknown source system. It was downloaded and decompressed using a 7zip console application. The document is a CSV file that appears to be encoded ...
- 486
20
votes
2
answers
2k
views
Accent Sensitive Sort
Why do these two SELECT statements result in a different sort order?
USE tempdb;
CREATE TABLE dbo.OddSort
(
id INT IDENTITY(1,1) PRIMARY KEY
, col1 NVARCHAR(2)
, col2 NVARCHAR(2)
);
GO
...
- 203
2
votes
1
answer
656
views
pgAdmin 4 including <U+FEFF> in all copied strings
I'm having trouble using copy/paste from pgAdmin 4. The symptom is that I get the character <U+FEFF> aka Zero Width No-break Space aka Byte Order Mark (BOM) prepended to any string I copy.
The ...
- 347
3
votes
1
answer
7k
views
ERROR: "sql " is not a known variable
This must one of the dumbest questions I've ever asked here, but there must something really sick hiding in my SQL script that is blocking it from running.
I'm invoking the cobertura.sql file using ...
- 969
19
votes
5
answers
21k
views
Why does the varchar datatype allow unicode values?
I have a table with a varchar column. It is allowing Trademark(™), copyright(©) and other Unicode characters as shown below.
Create table VarcharUnicodeCheck
(
col1 varchar(100)
)
insert into ...
- 777
4
votes
2
answers
15k
views
Remove characters from PostgreSQL database causing encoding errors
I have a PostgreSQL 10 database that uses WIN1252 encoding.
One of my columns has values that cause conversion errors when running a select from pgAdmin 4:
SELECT myfield FROM mydb.myschema."MYTABLE"...
- 43