Questions tagged [utf-8]

Unicode variable-width character encoding (between 1 and 4 bytes per code point).

Filter by
Sorted by
Tagged with
1
vote
1answer
22 views

MySQL query show something like 0x8081 instead of special characters

When i run SELECT CHAR(128,129,130,131,132,133,134,135,136,137); I got 0x80818283848586878889 istead of Çüéâäàåçêë. Does anybody know why?, I'm using charset utf8mb4. When I run show variables like '%...
1
vote
1answer
1k views

MySQL/MariaDB connection in asp code encodes wrongly charset utf8 at insert or update in data base

I can't get a simple update or insert to save special äöü characters. Instead they are being stored as äöü. The website displays äöü characters correctly if they are already stored as äöü in the ...
0
votes
1answer
97 views

converting latin to utf8mb4 causes questionmarks

The original format of the data is unknown The new table is in utf8mb4_general_ci If I do CONVERT(BINARY CONVERT(column USING latin1) USING UTF8) as mentioned here - it fixes all text, but converts ...
3
votes
1answer
3k views

Why pgAdmin4 invoke "'utf8' codec can't decode byte 0xe9 in position 42: invalid continuation byte" error?

I'm newbie in postgreSQL ,when I'm trying to create a new local server using pgAdmin4 ,it shows me this error : 'utf8' codec can't decode byte 0xe9 in position 42: invalid continuation byte I'm ...
2
votes
1answer
144 views

Find efficient way of migration 2TB GBK table in SybaseIQ to UTF-8 files

My project need to migrate more than 2TB raw data from SybaseIQ to another Database. SybaseIQ using GBK, the target Database using UTF-8, the target database can only load from UTF-8 files. We use ...
4
votes
1answer
185 views

What is difference between Arabic_100_CS_AS_KS_WS_SC_UTF8 and Latin1_General_100_CS_AS_KS_WS_SC_UTF8?

Starting from SQL Server 2019, it supports UTF-8 as collation. However, according to the following queries: SELECT COLLATIONPROPERTY('Arabic_100_CS_AS_KS_WS_SC_UTF8', 'CodePage') SELECT ...
2
votes
2answers
2k views

How to bulk convert ISO8859 text in Text columns to UTF8 in nvarchar(max) columns?

I've been struggling for days with a problem on a MS SQL database. Here is the situation: I have an old Open Source PHP application (Limesurvey version 1.92) running on a Windows 2008R2 server, with ...
1
vote
2answers
4k views

mysqldump collation issues

I have tried my hand at building a shell script to backup our wordpress database using tar balls and chron jobs on a Debian environment. The problem I am facing now is that our wp database after ...
3
votes
1answer
750 views

How to migrate Unicode UTF-8 CLOB data from Oracle to SQL Server 2017 UTF-? [closed]

I am currently assisting in a migration effort of this application from Oracle 12c to SQL Server 2017. Initially I just performed table inserts using openquery to Oracle. I discovered that the tables ...
1
vote
2answers
161 views

Changing from utf8 to utf8mb4

MySql database at the moment has: Character Set: utf8 Default Collation Name: utf8_unicode_ci I want to change it to: utf8mb4 utf8mb4_unicode_ci I would also do the same thing for all tables in ...
2
votes
2answers
715 views

Having charset "utf8mb4" - does it mean every character takes 4 bytes, or only those that need 4 bytes?

I have a VARCHAR(80) utf8mb4 column, and I'm testing adding some ASCII & Emoji characters, and use LENGTH() and CHAR_LENGTH() to understand the differences. From reading in different places, my ...
1
vote
1answer
130 views

mysqldump dumps different data with and without --no-create-info

mysqldump dumps different representations of data when called with/without --no-create-info. Test case First, create the test table and populate it with interesting data. CREATE TABLE `test` ( `...
194
votes
2answers
424k views

Create a MySQL database with charset UTF-8

I'm new to MySQL and I would like to know: How can I create a database with charset utf-8 like I did in navicat? create mydatabase; ...seems to be using some kind of default charset.
0
votes
0answers
9 views

How to use multilinguals via UTF8 in oracle 11g

My oracle(11g) DB uses UTF8 encoding, when I insert/update a row in a different language (Greek/Latin/Japanese) the row is getting updated, but when I check for the stored record in DB, it shows just &...
3
votes
1answer
448 views

What is the difference between different utf8mb4 binary collations?

What is the difference between utf8mb4_0900_bin vs utf8mb4_bin binary collations?
0
votes
1answer
2k views

Character set 'utf8mb4' is not a compiled character set and is not specified in the '/usr/share/mysql/charsets/Index.xml' file

Full error Character set 'utf8mb4' is not a compiled character set and is not specified in the '/usr/share/mysql/charsets/Index.xml' file InvalidArgumentException: There was a problem connecting to ...
1
vote
0answers
136 views

PSQL console encoding

How can I have a working PSQL console using UTF8 encoding under Windows? I have a Windows server and client. The Postgres 12 database contains tables with content in multiple languages (ex: English, ...
3
votes
1answer
9k views

CONVERT() returns garbage with CLOBs in Oracle

I am looking to repair a table of data in Oracle, ideally via unprivileged SQL, which has had UTF-8 data inserted into a UTF-8 database, using the Latin-1 character set by mistake. The symbol β GREEK ...
0
votes
1answer
3k views

CSV (UTF-8) to Postgres database (UTF-8), doesn't show äöå characters

I have csv-file that is UTF-8 encoded. I'm using python (psycopg2) to copy it into a Postgres database. My Postgres database is using UTF-8. However I have problems to show Ä, Ö and Å characters. If ...
0
votes
1answer
242 views

Accent marks/symbols not displaying correctly in MySQL database

Anytime there's an accent mark (é or ') or symbol (&) in my database it displays as HTML code. The table CHARACTER SET is UTF-8 Unicode and the COLLATION is utf8mb4_general_ci, which should ...
0
votes
0answers
194 views

How to convert UTF8 general in UTF8 unicode?

Short: My tables have the collation utf8mb4_unicode_ci, but the data itself is utf8mb4_general_ci and now I like to know how I can convert it into unicode? Long: I used this query to convert all my ...
2
votes
1answer
561 views

Is there some function in PostgreSQL to simply check if the input string is valid UTF-8 according to PG?

For several years now, I've been hunting this extremely quick-footed little bug which knows every hiding place in my metaphorical coding house. He lurks in the walls, sometimes takes months to show ...
1
vote
2answers
1k views

Upgrade all MySQL columns, tables, and databases from utf8mb3 to utf8mb4

I have thousands of columns across hundreds of tables in about a hundred databases inside a MySQL instance that need to be upgraded from utf8mb3 to utf8mb4. Is there a way to generate ALTER ...
3
votes
2answers
3k views

SQL Server 2019 UTF-8 Support Benefits

I'm already quite comfortable with using COMPRESS() and DECOMPRESS() in an internal forum software for our company (Currently in SQL Server 2017), but trying to make the database as efficient as ...
1
vote
0answers
103 views

MariaDB REGEXP_REPLACE Invalid utf8 byte sequence

I'm working with a fairly old (15 years) old database. It started as a MySQL3, was upgraded several times, the frontends had problems with proper encoding, the whole database was converted to utf8mb4 (...
1
vote
2answers
4k views

Spool Unicode data in sqlplus to ISO-client without character set conversion

I'm currently facing the following problem: We have an Oracle 11g database with character set AL32UTF8, and an sqlplus-client with NLS_LANG=GERMAN_GERMANY.WE8ISO8859P1. Stored in the database are ...
17
votes
1answer
16k views

Querying non-ASCII rows from Postgres

Does [:ascii:] class work in Postgres at all? It is not listed in their help, however I see examples in the web which utilize it. I have a UTF-8 database, where collation and c_type are en_US.UTF-8, ...
0
votes
0answers
135 views

Will MariaDB treat ut8/utf8mb3 and utf8mb4 collations as compatible?

Background on relevant character sets MySQL and MariaDB support the deprecated utf8mb3 character set and collations. (utf8 is currently an alias for utf8mb3.) This supports all Unicode code points on ...
0
votes
1answer
84 views

Arabic language search in codeigniter error : Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE)

Arabic language search in codeigniter error : Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) In English language search is working JOIN Query to use to ...
-1
votes
1answer
2k views

Workbench Error Code 1267 - utf8_unicode and utf8_general

I've recently just swapped all of a databases tables and columns to utf8_unicode_ci from the weird collection of about 3-5 different collations they were previously (don't ask... not my database) So, ...
0
votes
0answers
52 views

ALTER TABLE query to convert table to utf8mb4_bin takes many hours - how to optimise this?

I have a table which has approx 15 million rows - I won't add the schema here but it has 96 columns (far too many in my opinion). The current table collation is latin1_swedish_ci but for full emoji ...
-1
votes
1answer
814 views

UTF-8: showing correct in database, but not in webpage

When i upload data from php post forums "हिन्दी" language, changed into "हरिà" in database but looking correct in webpage, and when i upload data through phpmyAdmin from csv file "हिन्दी" ...
2
votes
1answer
2k views

Impact of switching a column's encoding from utf8 to utf8mb4 in MySQL

Background Info Let's say I have a table in MySQL that has +1 billion records. I have a column with that contains some description of a post. We don't index, search, or sort this column in any way. ...
0
votes
1answer
117 views

Simple Linux MySQL GUI Client That Reads UTF-8? [closed]

I'm an inexperienced MySQL user running Ubuntu and I need to open a database full of Hebrew data. I prefer a GUI and I downloaded MySQL Workbench. Not only is it incredibly confusing for me, but when ...
0
votes
1answer
842 views

Insert and Update with AES_ENCRYPT is mangling character set on double byte script

DB and fields are UTF8_unicode_ci collation - MySql v 5.6 HTML charset is utf 8 Double byte script was entered successfully prior to this issue (Chinese, Japanese, plus Cyrillic etc) when using ...
0
votes
1answer
194 views

MySQL: columns with mixed character encodings, and finding columns which have multibyte data

I have some rather large CSV files that I am loading into my MySQL 5.7 database. The files are several gigabytes in size, several million lines long, and have large column widths that must be used in ...
4
votes
1answer
7k views

Special Characters in MySQL using UTF8 Unicode Collation and LOAD DATA INFILE

I have several csv files in am loading into MySQl using Java. In the Description field I have several Special Characters that are causing the load to fail. I am using LOAD DATA INFILE. This is nested ...
3
votes
2answers
49k views

How to convert a MySQL database from utf8mb4 to utf8?

I would like to convert a MySQL database, which has utf8mb4 character set and utf8mb4_unicode_cicollation, to utf8 and utf8_unicode_ci. How to do that please? Thank you in advance for your ...
15
votes
2answers
46k views

Set value of character_set_client to utf8mb4

I'm trying to convert my DB to utf8mb4 following this guide. I have set: [client] default-character-set=utf8mb4 [mysql] default-character-set=utf8mb4 [mysqld] init-connect='SET NAMES utf8mb4' ...
-1
votes
1answer
2k views

Incorrect string value: '\xF0\x9F\xA4\x96<b...' for column 'body' at row 1 [closed]

I've set the column type to utf8mb4 and the collation to utf8mb4_bin. The table defaults are also utf8mb4 and utf8mb4_bin. When I use SequelPro to add the emoji "🤖" to the database manually, it ...
12
votes
2answers
2k views

When `nvarchar/nchar` is going to be used with SQL Server 2019?

With SQL Server 2019 Microsoft introduces UTF-8 support for CHAR and VARCHAR data types and says: This feature may provide significant storage savings, depending on the character set in use. For ...
3
votes
1answer
2k views

Are utf8_bin and utf8_general_cs the same?

Here's the situation: I have come across several posts for "utf8_bin vs utf8_general_ci". Similarly, "utf8_general_cs vs utf8_general_ci". But I can't find a post for "utf8_bin vs utf8_general_cs". ...
2
votes
1answer
1k views

presentation of non-ascii characters (ñ)

I'm facing the following problem: I have a MySQL database which contains non-ascii characters in some records. Characters like "ñ", for example. What kind of setup do I need to perform at the ...
1
vote
1answer
65 views

Can't set up Unicode support

I am creating a toy website locally with XAMPP and I want it to support Unicode. index.php contains a simple form that I can use to add a new entry (a UTF-8 string) to a table "myTable" in the DB. ...
1
vote
2answers
695 views

Is it possible for a properly set up MySQL to return invalid UTF-8?

I am wondering: assuming everything is set up properly, meaning a column with charset utf8mb4 and a client connecting to the database setting the connection encoding to utf8mb4, is there any way in ...
1
vote
1answer
354 views

What is the equivalent of the SQL Server CHAR function?

SQL Server has CHAR ( integer_expression ), so I can do SELECT CHAR(65) What is the equivalent on Postgresql? I have tried SELECT E'\0x65'; but I get ERROR: invalid byte sequence for encoding "...
12
votes
1answer
13k views

Encoding issue with SQL Server VARCHAR column retrieved in Python

We recently had an issue with encoding related to a field that's being stored as a varchar(120) in SQL Server. In SSMS, the varchar appears as: "Who Killed JonBen‚t?" However, when it's brought ...
1
vote
2answers
1k views

"Incorrect string value" while migrating from MSSQL to MySQL with Workbench

I got the task to migrate our MSSQL database into a MySQL database. Therefore I am using the recent version of MySQL Workbench and FreeTDS Driver. During the migration I always get errors like this: ...
1
vote
1answer
2k views

Cannot set character_set_database and character_set_server to utf8mb4

I'm trying to convert my MySQL server and its already existing schemas to the utf8mb4 character set. Preliminary Notes: I'm using Xampp (with MySQL 5.6.24). Default options are read from the ...
12
votes
1answer
46k views

Why default character_set_server is latin1?

I am using MySQL 5.5 and when I show variables about charset, I have +--------------------------+----------------------------+ | Variable_name | Value | +--------------...