Questions tagged [utf-8]

Unicode variable-width character encoding (between 1 and 4 bytes per code point).

Filter by
Sorted by
Tagged with
0 votes
0 answers
16 views

How to Encode ascent character into UTF-8 without losing its representation? Any Suggestion

Method: public static void test() throws IOException { String input = "ABCÉ Ôpqr"; // charsetName = ISO-8859-1 String utf8String = new String(StandardCharsets.UTF_8....
ankush chauhan's user avatar
1 vote
2 answers
145 views

Can I stick with VARCHAR(255) instead of VARCHAR(191) when converting database from UTF8 to UTF8MB4?

I'm on my way to converting my UTF8 database to UTF8MB4. I've read some advice about the fact that VARCHAR(255) fields should be turned to VARCHAR(191). cf. https://mathiasbynens.be/notes/mysql-...
AlexLaforge's user avatar
0 votes
1 answer
111 views

Upgrading MySQL utf8mb3 to utf8mb4 - how to replicate documentation behaviour?

I'm looking to upgrade MySQL fields from mb3 to mb4 in a large database and from reading the documentation I understand that the biggest problem is mb3 stores stuff in 3 bytes, but mb4 stores stuff in ...
KillerKode's user avatar
0 votes
1 answer
202 views

SQL Server 2019, utf-8, replication problem

I know this is late to the game, but we're just upgrading to Sql Server 2019 after years and years on 2008. I started using the Latin1_General_100_CI_AI_SC_UTF8 collation as an experiment and at least ...
user1664043's user avatar
2 votes
1 answer
230 views

CHAR and VARCHAR lengths for multibyte characters

I have a table with ~100mln rows. One of the columns is a VARCHAR(64) with utf8mb4 encoding for storing user nicknames (others are few integers). The data in that column has MAX(LENGTH()) == 42 and ...
herhor67's user avatar
0 votes
2 answers
619 views

Converting from latin1 to utf8mb4 of ENUM and SET with accented characters on mysql

I am trying to convert tables from latin1 to utf8mb4 on MySQL 5.7. Everything works fine except the tables with ENUM and SET which contains a value with an accented character. 2 types of errors are ...
Toto's user avatar
  • 93
0 votes
0 answers
116 views

Migrate postgresql to MariaDB

I try to migrate mysql Zabbix database from Postgresql to MariaDB using MySQL Workbench migration Wizard I have compiled psqlODBC and enable it When I setup the source database in the wizard the test ...
DavJ's user avatar
  • 1
5 votes
1 answer
625 views

postgresql matching or converting utf-8 variant strings

Postgres 13 I am looking for a way to search UTF-8 text that may have variant character representations ( what is the proper term for this? ie 𝐋𝐈𝐅𝐄 vs life ) within postgresql. I am running into ...
unomi's user avatar
  • 153
2 votes
1 answer
934 views

Postgres encoding issue - find values that have no equivalent in (client) encoding

I have Postgresql database that internally uses UTF-8 encoding. Some clients that connect to the database use LATIN 2 (ISO 8859-2) client encoding when they connect to database. I can't change the ...
CyberMuz's user avatar
  • 123
4 votes
1 answer
598 views

Fix corrupted utf8 nvarchar value that was inserted without N prefix

Is it possible to restore the UTF8 data from a corrupted nvarchar, that was used without the N prefix? For example, in the following snippet, I'd like @n1_fixed to have the correct value based on @n1: ...
HeyJude's user avatar
  • 447
0 votes
1 answer
788 views

Tables character-set migration without huge downtime- mysql

We're in the process of migrating to a new character set for several tables in our database. Some tables are currently using utf8/utf8mb3 character set, which is deprecated now and we want move on ...
Retrosec6's user avatar
2 votes
1 answer
411 views

How can I detect failure in CONVERT( tbl AS utf8)?

MySQL provides the CONVERT(`tbl` AS utf8) expression to transcode text from one charset to another. Sometimes that conversion cannot succeed, because the destination charset does not include ...
Jim DeLaHunt's user avatar
2 votes
1 answer
521 views

How can I detect double-encoded MySQL columns and rows, and validate the repair?

My database provider, bless their hearts, migrated our MySQL databases to another server recently, and introduced double-encoding of UTF-8 data via Latin1 into our text data. Strings like 'emdash—here'...
Jim DeLaHunt's user avatar
0 votes
2 answers
746 views

What is the definition of latin1 character set in MySQL 8.0 and MySQL 5.7?

I am trying to fix some corruption to character fields in my database after a migration. I suspect that part of the problem is that fields with UTF-8 data were interpreted with the MySQL latin1 ...
Jim DeLaHunt's user avatar
0 votes
1 answer
772 views

Import postgresql database with Latin encoding

I'm going to upgrade from postgresql 9 to postgresql 11, the encoding used is Latin. I've exported the database using this command pg_dumpall | gzip > /tmp/db.sql.gz I extracted the exported ...
Naruto Uzumaki's user avatar
4 votes
2 answers
3k views

MySQL What charset/collation for Case insensitivity and Accent sensitivity?

I am looking for a charset/collation that would make it so when I do a SELECT * FROM table_name WHERE username = "Warrior" It only returns me the rows where username = "Warrior", ...
lyeaf's user avatar
  • 177
4 votes
1 answer
13k views

Storing special characters (German, French, Spanish) in SQL Server result in weird strings

I'm trying to store the German ß in SQL Server, but it results in ß. For example "Gleimstraße" is stored as "Gleimstraße" But this also happens for other characters: "König-...
Adam's user avatar
  • 245
1 vote
1 answer
2k views

How to fix unicode REPLACEMENT CHARACTER � in postgres database

Is there a way fix the � problem in postgres database: As you can see there are names with accents and are not displayed corretly in the database. I would like to know if there is a way to fix this ...
rainman's user avatar
  • 195
0 votes
1 answer
595 views

How can I know if there is data loss when converting mysql character set(s)?

I'm converting a large (70G) legacy mysql database that is mostly utf8 (but with a sprinkling of other encodings in fields) to one that is (as uniformly as possible) utf8mb4 (utf8mb4_unicode_ci). The ...
pixelearth's user avatar
1 vote
1 answer
742 views

MySQL query show something like 0x8081 instead of special characters

When i run SELECT CHAR(128,129,130,131,132,133,134,135,136,137); I got 0x80818283848586878889 istead of Çüéâäàåçêë. Does anybody know why?, I'm using charset utf8mb4. When I run show variables like '%...
Danh Le's user avatar
  • 11
4 votes
1 answer
878 views

What is difference between Arabic_100_CS_AS_KS_WS_SC_UTF8 and Latin1_General_100_CS_AS_KS_WS_SC_UTF8?

Starting from SQL Server 2019, it supports UTF-8 as collation. However, according to the following queries: SELECT COLLATIONPROPERTY('Arabic_100_CS_AS_KS_WS_SC_UTF8', 'CodePage') SELECT ...
Eng.Fouad's user avatar
  • 143
4 votes
2 answers
3k views

Having charset "utf8mb4" - does it mean every character takes 4 bytes, or only those that need 4 bytes?

I have a VARCHAR(80) utf8mb4 column, and I'm testing adding some ASCII & Emoji characters, and use LENGTH() and CHAR_LENGTH() to understand the differences. From reading in different places, my ...
Nuno's user avatar
  • 809
1 vote
1 answer
2k views

mysqldump dumps different data with and without --no-create-info

mysqldump dumps different representations of data when called with/without --no-create-info. Test case First, create the test table and populate it with interesting data. CREATE TABLE `test` ( `...
matt's user avatar
  • 203
5 votes
1 answer
3k views

What is the difference between different utf8mb4 binary collations?

What is the difference between utf8mb4_0900_bin vs utf8mb4_bin binary collations?
mvorisek's user avatar
  • 418
1 vote
0 answers
1k views

PSQL console encoding

How can I have a working PSQL console using UTF8 encoding under Windows? I have a Windows server and client. The Postgres 12 database contains tables with content in multiple languages (ex: English, ...
JGH's user avatar
  • 111
1 vote
2 answers
1k views

Changing from utf8 to utf8mb4

MySql database at the moment has: Character Set: utf8 Default Collation Name: utf8_unicode_ci I want to change it to: utf8mb4 utf8mb4_unicode_ci I would also do the same thing for all tables in ...
Vicky Dallas's user avatar
0 votes
1 answer
1k views

Accent marks/symbols not displaying correctly in MySQL database

Anytime there's an accent mark (é or ') or symbol (&) in my database it displays as HTML code. The table CHARACTER SET is UTF-8 Unicode and the COLLATION is utf8mb4_general_ci, which should ...
msklut's user avatar
  • 25
0 votes
0 answers
1k views

How to convert UTF8 general in UTF8 unicode?

Short: My tables have the collation utf8mb4_unicode_ci, but the data itself is utf8mb4_general_ci and now I like to know how I can convert it into unicode? Long: I used this query to convert all my ...
mgutt's user avatar
  • 133
3 votes
1 answer
3k views

Is there some function in PostgreSQL to simply check if the input string is valid UTF-8 according to PG?

For several years now, I've been hunting this extremely quick-footed little bug which knows every hiding place in my metaphorical coding house. He lurks in the walls, sometimes takes months to show ...
Arlington's user avatar
1 vote
1 answer
774 views

converting latin to utf8mb4 causes questionmarks

The original format of the data is unknown The new table is in utf8mb4_general_ci If I do CONVERT(BINARY CONVERT(column USING latin1) USING UTF8) as mentioned here - it fixes all text, but converts ...
stackbox's user avatar
1 vote
0 answers
322 views

MariaDB REGEXP_REPLACE Invalid utf8 byte sequence

I'm working with a fairly old (15 years) old database. It started as a MySQL3, was upgraded several times, the frontends had problems with proper encoding, the whole database was converted to utf8mb4 (...
mindhaq's user avatar
  • 111
10 votes
2 answers
10k views

Upgrade all MySQL columns, tables, and databases from utf8mb3 to utf8mb4

I have thousands of columns across hundreds of tables in about a hundred databases inside a MySQL instance that need to be upgraded from utf8mb3 to utf8mb4. Is there a way to generate ALTER ...
Stephen Ostermiller's user avatar
0 votes
0 answers
587 views

Will MariaDB treat ut8/utf8mb3 and utf8mb4 collations as compatible?

Background on relevant character sets MySQL and MariaDB support the deprecated utf8mb3 character set and collations. (utf8 is currently an alias for utf8mb3.) This supports all Unicode code points on ...
William Entriken's user avatar
0 votes
1 answer
205 views

Arabic language search in codeigniter error : Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE)

Arabic language search in codeigniter error : Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) In English language search is working JOIN Query to use to ...
SNS's user avatar
  • 109
0 votes
0 answers
485 views

ALTER TABLE query to convert table to utf8mb4_bin takes many hours - how to optimise this?

I have a table which has approx 15 million rows - I won't add the schema here but it has 96 columns (far too many in my opinion). The current table collation is latin1_swedish_ci but for full emoji ...
Zabs's user avatar
  • 111
0 votes
1 answer
11k views

CSV (UTF-8) to Postgres database (UTF-8), doesn't show äöå characters

I have csv-file that is UTF-8 encoded. I'm using python (psycopg2) to copy it into a Postgres database. My Postgres database is using UTF-8. However I have problems to show Ä, Ö and Å characters. If ...
Bitterbit's user avatar
2 votes
2 answers
5k views

How to bulk convert ISO8859 text in Text columns to UTF8 in nvarchar(max) columns?

I've been struggling for days with a problem on a MS SQL database. Here is the situation: I have an old Open Source PHP application (Limesurvey version 1.92) running on a Windows 2008R2 server, with ...
P Daubresse's user avatar
0 votes
1 answer
5k views

Character set 'utf8mb4' is not a compiled character set and is not specified in the '/usr/share/mysql/charsets/Index.xml' file

Full error Character set 'utf8mb4' is not a compiled character set and is not specified in the '/usr/share/mysql/charsets/Index.xml' file InvalidArgumentException: There was a problem connecting to ...
Novocaine's user avatar
  • 117
0 votes
1 answer
596 views

MySQL: columns with mixed character encodings, and finding columns which have multibyte data

I have some rather large CSV files that I am loading into my MySQL 5.7 database. The files are several gigabytes in size, several million lines long, and have large column widths that must be used in ...
blindsnowmobile's user avatar
4 votes
1 answer
1k views

How to migrate Unicode UTF-8 CLOB data from Oracle to SQL Server 2017 UTF-? [closed]

I am currently assisting in a migration effort of this application from Oracle 12c to SQL Server 2017. Initially I just performed table inserts using openquery to Oracle. I discovered that the tables ...
Geoff Dawdy's user avatar
  • 1,123
-1 votes
1 answer
4k views

Incorrect string value: '\xF0\x9F\xA4\x96<b...' for column 'body' at row 1 [closed]

I've set the column type to utf8mb4 and the collation to utf8mb4_bin. The table defaults are also utf8mb4 and utf8mb4_bin. When I use SequelPro to add the emoji "🤖" to the database manually, it ...
satnam's user avatar
  • 99
3 votes
2 answers
4k views

SQL Server 2019 UTF-8 Support Benefits

I'm already quite comfortable with using COMPRESS() and DECOMPRESS() in an internal forum software for our company (Currently in SQL Server 2017), but trying to make the database as efficient as ...
John Titor's user avatar
12 votes
2 answers
3k views

When `nvarchar/nchar` is going to be used with SQL Server 2019?

With SQL Server 2019 Microsoft introduces UTF-8 support for CHAR and VARCHAR data types and says: This feature may provide significant storage savings, depending on the character set in use. For ...
gotqn's user avatar
  • 4,016
3 votes
1 answer
4k views

Are utf8_bin and utf8_general_cs the same?

Here's the situation: I have come across several posts for "utf8_bin vs utf8_general_ci". Similarly, "utf8_general_cs vs utf8_general_ci". But I can't find a post for "utf8_bin vs utf8_general_cs". ...
Dinesh Kumar's user avatar
1 vote
1 answer
241 views

Can't set up Unicode support

I am creating a toy website locally with XAMPP and I want it to support Unicode. index.php contains a simple form that I can use to add a new entry (a UTF-8 string) to a table "myTable" in the DB. ...
what a disgrace's user avatar
2 votes
1 answer
4k views

Impact of switching a column's encoding from utf8 to utf8mb4 in MySQL

Background Info Let's say I have a table in MySQL that has +1 billion records. I have a column with that contains some description of a post. We don't index, search, or sort this column in any way. ...
Alex McKenzie's user avatar
0 votes
1 answer
1k views

Insert and Update with AES_ENCRYPT is mangling character set on double byte script

DB and fields are UTF8_unicode_ci collation - MySql v 5.6 HTML charset is utf 8 Double byte script was entered successfully prior to this issue (Chinese, Japanese, plus Cyrillic etc) when using ...
anoldermark's user avatar
3 votes
1 answer
5k views

Why pgAdmin4 invoke "'utf8' codec can't decode byte 0xe9 in position 42: invalid continuation byte" error?

I'm newbie in postgreSQL ,when I'm trying to create a new local server using pgAdmin4 ,it shows me this error : 'utf8' codec can't decode byte 0xe9 in position 42: invalid continuation byte I'm ...
Menai Ala Eddine - Aladdin's user avatar
1 vote
1 answer
776 views

What is the equivalent of the SQL Server CHAR function?

SQL Server has CHAR ( integer_expression ), so I can do SELECT CHAR(65) What is the equivalent on Postgresql? I have tried SELECT E'\0x65'; but I get ERROR: invalid byte sequence for encoding "...
Jonas Stawski's user avatar
1 vote
2 answers
1k views

Is it possible for a properly set up MySQL to return invalid UTF-8?

I am wondering: assuming everything is set up properly, meaning a column with charset utf8mb4 and a client connecting to the database setting the connection encoding to utf8mb4, is there any way in ...
deceze's user avatar
  • 93