Questions tagged [unicode]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
1 answer
37 views

How to fix unicode REPLACEMENT CHARACTER � in postgres database

Is there a way fix the � problem in postgres database: As you can see there are names with accents and are not displayed corretly in the database. I would like to know if there is a way to fix this ...
user avatar
  • 175
0 votes
1 answer
73 views

Detecting whether a varchar value is already encrypted by EncryptByPassPhrase() in SQL Server

Starting with a varchar(100) column named Password in a table Users with one record in it, after performing an update on the column to encrypt it with EncryptByPassPhrase() on the Password column the ...
user avatar
  • 153
0 votes
0 answers
297 views

Convert VARCHAR2 to UTF-8 in Oracle 12

I need to convert VARCHAR2 columns that store text in unkown character sets into a standard UTF-8 BLOB. The table has more columns, but the ones of interest to me are: ADDRESS POST ---------- ----...
user avatar
0 votes
1 answer
20 views

Producing JSON in Oracle 12c1

I need to produce a JSON document with several properties from a list of Oracle 12c1 production databases, and I was given read-only access in order to generate this data. The problem is these ...
user avatar
1 vote
1 answer
30 views

Datapump metadata-only import changes datalength when importing with conversion

Datapump metadata only import changes datalength when importing with conversion: (export done in WE8ISO8859P15 and import done in AL32UTF8 character set and AL16UTF16 NCHAR character set) Example ...
user avatar
  • 868
14 votes
1 answer
1k views

Why does a comparison between 'tr' & 'tR' fail on a SQL Server with Vietnamese_CI_AI collation?

There seems to be something special about 'tR' in Vietnamese collation. Appreicate if anyone who knows about it can explain in simple terms. This issue was discovered during the installation of our ...
user avatar
4 votes
1 answer
295 views

What is difference between Arabic_100_CS_AS_KS_WS_SC_UTF8 and Latin1_General_100_CS_AS_KS_WS_SC_UTF8?

Starting from SQL Server 2019, it supports UTF-8 as collation. However, according to the following queries: SELECT COLLATIONPROPERTY('Arabic_100_CS_AS_KS_WS_SC_UTF8', 'CodePage') SELECT ...
user avatar
  • 143
2 votes
2 answers
593 views

How to create a new column in SELECT CASE when string contains Arabic words?

I have a problem with a select case when statement, I would like to add a new column in the select case when statement, I got the result but with ????? because I have set the column to Arabic words, I ...
user avatar
  • 61
3 votes
2 answers
4k views

How do I fix the Code Page in SSIS Lookup Transformation to be 65001?

I have an SQL server 2019 with databases and tables all set with the collation Latin1_General_100_CI_AS_SC_UTF8. The relevant table has code and desc columns which are both of data type varchar. In ...
user avatar
1 vote
2 answers
429 views

unicode symbols for entity-relationship

What are the Unicode symbols that most closely relate to the symbols used in entity-relationship diagrams, such as the ones in the screenshot of the OP here: https://stackoverflow.com/questions/...
user avatar
  • 124
0 votes
1 answer
176 views

How to pattern match against Unicode characters by PostgreSQL `SIMILAR TO` expressions?

In a PostgreSQL database, I have a column named metaid of type text whose value contains digits as specified in the third edition of XML 1.0 standard: Digit ::= [#x0030-#x0039] | [#x0660-#x0669] | [#...
user avatar
  • 103
3 votes
2 answers
714 views

How to convert xml entities to unicode characters (with read-only access) in Oracle?

I'm connected to an Oracle Database (11g Release 2 - 11.2.0.4), with read-only access. Into this database, some of the data is uploaded via, or rather as, XML - and quite a few entries contain ...
user avatar
6 votes
1 answer
4k views

PostgreSQL nondeterministic collations are not supported for LIKE

I am using Postgresql v12. I created a collation like this: CREATE COLLATION ci (provider = icu, locale = 'tr_TR', deterministic = false); I used that collation in a table: create table testtable1 ...
user avatar
0 votes
1 answer
420 views

Accent marks/symbols not displaying correctly in MySQL database

Anytime there's an accent mark (é or ') or symbol (&) in my database it displays as HTML code. The table CHARACTER SET is UTF-8 Unicode and the COLLATION is utf8mb4_general_ci, which should ...
user avatar
  • 25
2 votes
1 answer
943 views

Is there some function in PostgreSQL to simply check if the input string is valid UTF-8 according to PG?

For several years now, I've been hunting this extremely quick-footed little bug which knows every hiding place in my metaphorical coding house. He lurks in the walls, sometimes takes months to show ...
user avatar
3 votes
1 answer
961 views

What is the meaning of "-x-icu" in PostgreSQL's "collates"?

I had this query: SELECT * FROM table ORDER BY label ASC; Since the labels are not in English, they didn't get sorted in the right order (ones beginning with "ö" were not in the bottom/end). I ...
user avatar
-1 votes
1 answer
169 views

ORACLE DMU (database migration assistant for unicode) Tool

Reading the documentation for Oracle DMU tool, when I try the step create a step for DMU Repository tablespace the document says to use the result of the query for initial size. However; I could not ...
user avatar
1 vote
2 answers
172 views

Replace function removing Ones / 1 when removing char(185) (superscript 1)

Ok as the title suggests this is occuring. When using the REPLACE function on char(185) for data in a temporary table, ones ("1") are being removed as well. SQL SERVER 2008R2 UNICODE Data DB ...
user avatar
4 votes
1 answer
2k views

Importing Chinese characters via SSIS

I'm trying to import file that contains Chinese characters into a table. File encoding is Big5 (Traditional). Here is a sample file like one that I need to import: https://www.pastiebin.com/...
user avatar
  • 282
1 vote
1 answer
7k views

How to replace multiple unicode characters

I have identified a couple of unicode characters in my database by means of the following query: SELECT name FROM table WHERE name NOT LIKE '%'+CAST(name AS VARCHAR(1000))+'%' ...with which I get ...
user avatar
0 votes
1 answer
135 views

Reusing literal string in different parts of query

I created this PostgreSQL query while troubleshooting a problem that turned out to be due to non-printing characters in my strings: with cte as ( select 'The character between the xes is evil: xx' as ...
user avatar
  • 349
3 votes
2 answers
3k views

Storing unicode text into varchar column in SQL Server

I have mistakenly stored unicode text into a varchar column in a table and now it is shown as question marks (???). Is it possible to get our main information back from the column instead of question ...
user avatar
-1 votes
1 answer
2k views

Incorrect string value: '\xF0\x9F\xA4\x96<b...' for column 'body' at row 1 [closed]

I've set the column type to utf8mb4 and the collation to utf8mb4_bin. The table defaults are also utf8mb4 and utf8mb4_bin. When I use SequelPro to add the emoji "🤖" to the database manually, it ...
user avatar
  • 99
0 votes
1 answer
732 views

Mariadb (MySQL) On Windows- problem entering non-ASCII characters in a query

MariaDB: mysql Ver 15.1 Distrib 10.3.8-MariaDB OS: Windows 7 Pro 64 bit I have my server database and client all set up (I believe) to support utf8 end to end- (**my.cnf**) # # Set everything to ...
user avatar
  • 1
5 votes
2 answers
1k views

Byte ordering for multibyte characters in SQL Server versus Oracle

I am currently in the process of migrating data from Oracle to SQL Server and I'm encountering an issue trying to validate the data post-migration. Environment Details: Oracle 12 - AL32UTF8 ...
user avatar
  • 9,232
4 votes
3 answers
944 views

Unicode storage of \u202b RLE and \u202c PDE in a Unicode-aware database?

I'm building a new product for toponyms and in it the Arabic shows kinda like this: ^IArabic^I<202b>ﺰﻤﺑﺎﺑﻮﻳ<202c>^I<202b>ﺞﻫﻭﺮﻳﺓ ﺰﻤﺑﺎﺑﻮﻳ<202c>$ Actually not quite. This is a ...
user avatar
  • 55.7k
0 votes
1 answer
125 views

How to import SQL file which has no N prefix

I have a large .sql file in Persian Language. The file is exported from sqlite db browser and has no N prefix for unicode fields. How can I reuse it in Sql Server? This is sample of insert query in ...
user avatar
2 votes
1 answer
2k views

How to insert a Unicode character verbose into a varchar DB?

I need to insert this character '●' into a VARCHAR column of a MSSQL database with collation set as SQL_Latin1_General_CP1_CI_AS (or at least mock what my Python + Windows MSSQL Driver might have done)...
user avatar
  • 737
2 votes
1 answer
949 views

How to send Unicode symbols using SQL Server Database Mail? [closed]

How can I send Unicode symbols using SQL Server Database Mail? My goal is to send a symbol on the subject of an email. Let's say that, for example, I want to send this "🚯 Subject with symbol" on the ...
user avatar
12 votes
2 answers
3k views

When `nvarchar/nchar` is going to be used with SQL Server 2019?

With SQL Server 2019 Microsoft introduces UTF-8 support for CHAR and VARCHAR data types and says: This feature may provide significant storage savings, depending on the character set in use. For ...
user avatar
  • 3,609
4 votes
1 answer
3k views

Querying special character from SQL Server

When I saved special characters in a column of type nvarchar, I did not use the prefix 'N', so the characters got saved as ??. Is there any way to retrieve the original special characters from these ...
user avatar
4 votes
1 answer
14k views

Why do I get incorrect characters when decoding a Base64 string to NVARCHAR in SQL Server?

I've been looking at how to decode Base64 with SQL Server and after searching around many solutions online (some from here) seem to be based upon this sort of method. SELECT CAST(CAST('...
user avatar
  • 143
4 votes
1 answer
610 views

Why isn't Unicode character REPLACEd in some cases?

Running this results in "test", as expected: SELECT REPLACE(NCHAR(1234), NCHAR(1234), N'test'); However, running this results in "aӒa", which doesn't have "test" in it: SELECT REPLACE(N'a' +...
user avatar
  • 1,791
3 votes
1 answer
9k views

inserting a special character into a table

I have a few characters that are being replaced by a ?. How do you keep the original character in the table? The character that I am trying to insert is a ɬ; Latin Small Letter L With Belt. Is it ...
user avatar
  • 157
1 vote
1 answer
86 views

Can't set up Unicode support

I am creating a toy website locally with XAMPP and I want it to support Unicode. index.php contains a simple form that I can use to add a new entry (a UTF-8 string) to a table "myTable" in the DB. ...
user avatar
5 votes
2 answers
7k views

SQL Server Bulk Insert properly interprets some Unicode characters but not others?

For some reason the MS SQL Server 2016 bulk insert misinterprets/translates Unicode characters: C9 (É) into 2B (+) A1 (¡) into ED (í) A0 ( ) into E1 (á) AE (®) into AB («) CB (Ë) into 2D (-) D1 (...
user avatar
  • 303
0 votes
1 answer
345 views

sp_executesql changes non standard characters to?

Am trying to use sp_executesql to run commands but I've come across an issue where in non-english characters (aka Chinese & Japanese) are changed to ??? when ran. sql server 2008R2 & 2012 ...
user avatar
  • 1,951
53 votes
2 answers
6k views

How to create Unicode parameter and variable names

All of this works: CREATE DATABASE [¯\_(ツ)_/¯]; GO USE [¯\_(ツ)_/¯]; GO CREATE SCHEMA [¯\_(ツ)_/¯]; GO CREATE TABLE [¯\_(ツ)_/¯].[¯\_(ツ)_/¯]([¯\_(ツ)_/¯] NVARCHAR(20)); GO CREATE UNIQUE CLUSTERED INDEX [¯...
user avatar
  • 41.3k
5 votes
2 answers
4k views

Why is my UTF-8 document raising UTF-8 encoding errors in Azure Data Lake Analytics?

I have a document that was compressed in gunzip from a unknown source system. It was downloaded and decompressed using a 7zip console application. The document is a CSV file that appears to be encoded ...
user avatar
19 votes
2 answers
1k views

Accent Sensitive Sort

Why do these two SELECT statements result in a different sort order? USE tempdb; CREATE TABLE dbo.OddSort ( id INT IDENTITY(1,1) PRIMARY KEY , col1 NVARCHAR(2) , col2 NVARCHAR(2) ); GO ...
user avatar
  • 193
2 votes
1 answer
479 views

pgAdmin 4 including <U+FEFF> in all copied strings

I'm having trouble using copy/paste from pgAdmin 4. The symptom is that I get the character <U+FEFF> aka Zero Width No-break Space aka Byte Order Mark (BOM) prepended to any string I copy. The ...
user avatar
  • 349
3 votes
1 answer
5k views

ERROR: "sql " is not a known variable

This must one of the dumbest questions I've ever asked here, but there must something really sick hiding in my SQL script that is blocking it from running. I'm invoking the cobertura.sql file using ...
user avatar
18 votes
5 answers
16k views

Why does the varchar datatype allow unicode values?

I have a table with a varchar column. It is allowing Trademark(™), copyright(©) and other Unicode characters as shown below. Create table VarcharUnicodeCheck ( col1 varchar(100) ) insert into ...
user avatar
  • 765
3 votes
2 answers
10k views

Remove characters from PostgreSQL database causing encoding errors

I have a PostgreSQL 10 database that uses WIN1252 encoding. One of my columns has values that cause conversion errors when running a select from pgAdmin 4: SELECT myfield FROM mydb.myschema."MYTABLE"...
user avatar
0 votes
1 answer
48 views

Special letter behaviour problem?

I asked question in very blurry manner because I cannot explain in short sentence what is my problem about. In general I have a set of procedures which transforms text and XML. Inside the procedures ...
user avatar
1 vote
1 answer
2k views

Using LIKE to search case insensitive over language specific characters (PostgreSQL)

Using a PostgreSQL database (Collation C, Encoding UTF8) we store data from various languages. This means we also get the special characters for free (e.g. Å, å, ...) Within an application we now ...
user avatar
  • 13
20 votes
2 answers
4k views

Can't update "CO2" to "CO₂" in table row

Given this table: CREATE TABLE test ( id INT NOT NULL, description NVARCHAR(100) COLLATE Modern_Spanish_CI_AS NOT NULL ); INSERT INTO test (id, description) VALUES (1, 'CO2'); I've realised ...
user avatar
3 votes
1 answer
13k views

How to be sure that a SQL Script is running with ANSI encoding?

Maybe this is a stupid question or I'm asking it in the wrong way. How can I be sure that a script ( with thousands of lines ) is running with ANSI encoding? Let's say we created a script using ...
user avatar
  • 7,100
5 votes
0 answers
2k views

VARCHAR(MAX) taking more space than NVARCHAR(MAX) [duplicate]

I have a table in SQL Server 2012 Enterprise Edition that has ~100GB of data. Most of this data is attributed to a single NVARCHAR(MAX) column which has an average length of 4500 characters. It stores ...
user avatar
1 vote
2 answers
4k views

Spool Unicode data in sqlplus to ISO-client without character set conversion

I'm currently facing the following problem: We have an Oracle 11g database with character set AL32UTF8, and an sqlplus-client with NLS_LANG=GERMAN_GERMANY.WE8ISO8859P1. Stored in the database are ...
user avatar
  • 11