Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [utf-8]

Unicode variable-width character encoding (between 1 and 4 bytes per code point).

0
votes
1answer
31 views

MySQL: columns with mixed character encodings, and finding columns which have multibyte data

I have some rather large CSV files that I am loading into my MySQL 5.7 database. The files are several gigabytes in size, several million lines long, and have large column widths that must be used in ...
-1
votes
1answer
312 views

Incorrect string value: '\xF0\x9F\xA4\x96<b…' for column 'body' at row 1 [closed]

I've set the column type to utf8mb4 and the collation to utf8mb4_bin. The table defaults are also utf8mb4 and utf8mb4_bin. When I use SequelPro to add the emoji "🤖" to the database manually, it ...
3
votes
2answers
1k views

SQL Server 2019 UTF-8 Support Benefits

I'm already quite comfortable with using COMPRESS() and DECOMPRESS() in an internal forum software for our company (Currently in SQL Server 2017), but trying to make the database as efficient as ...
9
votes
2answers
1k views

When `nvarchar/nchar` is going to be used with SQL Server 2019?

With SQL Server 2019 Microsoft introduces UTF-8 support for CHAR and VARCHAR data types and says: This feature may provide significant storage savings, depending on the character set in use. For ...
1
vote
1answer
225 views

Are utf8_bin and utf8_general_cs the same?

Here's the situation: I have come across several posts for "utf8_bin vs utf8_general_ci". Similarly, "utf8_general_cs vs utf8_general_ci". But I can't find a post for "utf8_bin vs utf8_general_cs". ...
1
vote
1answer
35 views

Can't set up Unicode support

I am creating a toy website locally with XAMPP and I want it to support Unicode. index.php contains a simple form that I can use to add a new entry (a UTF-8 string) to a table "myTable" in the DB. ...
0
votes
1answer
440 views

Impact of switching a column's encoding from utf8 to utf8mb4 in MySQL

Background Info Let's say I have a table in MySQL that has +1 billion records. I have a column with that contains some description of a post. We don't index, search, or sort this column in any way. ...
0
votes
1answer
234 views

Insert and Update with AES_ENCRYPT is mangling character set on double byte script

DB and fields are UTF8_unicode_ci collation - MySql v 5.6 HTML charset is utf 8 Double byte script was entered successfully prior to this issue (Chinese, Japanese, plus Cyrillic etc) when using ...
2
votes
0answers
2k views

Why pgAdmin4 invoke “'utf8' codec can't decode byte 0xe9 in position 42: invalid continuation byte” error?

I'm newbie in postgreSQL ,when I'm trying to create a new local server using pgAdmin4 ,it shows me this error : 'utf8' codec can't decode byte 0xe9 in position 42: invalid continuation byte I'm ...
1
vote
1answer
132 views

What is the equivalent of the SQL Server CHAR function?

SQL Server has CHAR ( integer_expression ), so I can do SELECT CHAR(65) What is the equivalent on Postgresql? I have tried SELECT E'\0x65'; but I get ERROR: invalid byte sequence for encoding "...
1
vote
2answers
276 views

Is it possible for a properly set up MySQL to return invalid UTF-8?

I am wondering: assuming everything is set up properly, meaning a column with charset utf8mb4 and a client connecting to the database setting the connection encoding to utf8mb4, is there any way in ...
0
votes
1answer
174 views

UTF-8: showing correct in database, but not in webpage

When i upload data from php post forums "हिन्दी" language, changed into "हरिà" in database but looking correct in webpage, and when i upload data through phpmyAdmin from csv file "हिन्दी" ...
0
votes
2answers
1k views

Spool Unicode data in sqlplus to ISO-client without character set conversion

I'm currently facing the following problem: We have an Oracle 11g database with character set AL32UTF8, and an sqlplus-client with NLS_LANG=GERMAN_GERMANY.WE8ISO8859P1. Stored in the database are ...
9
votes
1answer
6k views

Encoding issue with SQL Server VARCHAR column retrieved in Python

We recently had an issue with encoding related to a field that's being stored as a varchar(120) in SQL Server. In SSMS, the varchar appears as: "Who Killed JonBen‚t?" However, when it's brought ...
1
vote
0answers
50 views

Finding and replacing non-specific text to lowercase variant

I've been cracking my brain on a problem that I had at work. In one of my tables is a bunch of rows, with a certain column that contains XML data that has several properties like this: (simplified) &...
2
votes
1answer
5k views

CONVERT() returns garbage with CLOBs in Oracle

I am looking to repair a table of data in Oracle, ideally via unprivileged SQL, which has had UTF-8 data inserted into a UTF-8 database, using the Latin-1 character set by mistake. The symbol β GREEK ...
8
votes
2answers
1k views

Why does Oracle use a different byte length than java for the supplementary unicode character chipmunk?

I have java code trimming a UTF-8 string to the size of my Oracle (11.2.0.4.0) column which ends up throwing an error because java and Oracle see the string as different byte lengths. I've verified my ...
12
votes
1answer
7k views

Querying non-ASCII rows from Postgres

Does [:ascii:] class work in Postgres at all? It is not listed in their help, however I see examples in the web which utilize it. I have a UTF-8 database, where collation and c_type are en_US.UTF-8, ...
0
votes
1answer
776 views

Workbench Error Code 1267 - utf8_unicode and utf8_general

I've recently just swapped all of a databases tables and columns to utf8_unicode_ci from the weird collection of about 3-5 different collations they were previously (don't ask... not my database) So, ...
0
votes
1answer
1k views

ORA-12899: value too large for column COMESFROMTOWN (actual: 50, maximum: 50)

I am importing a csv file into an Oracle database using an external table. The file encoding is UTF-8. The separator is actually ;. The target table definition is identical to the external table ...
0
votes
2answers
810 views

“Incorrect string value” while migrating from MSSQL to MySQL with Workbench

I got the task to migrate our MSSQL database into a MySQL database. Therefore I am using the recent version of MySQL Workbench and FreeTDS Driver. During the migration I always get errors like this: ...
1
vote
1answer
665 views

MySQL/MariaDB connection in asp code encodes wrongly charset utf8 at insert or update in data base

I can't get a simple update or insert to save special äöü characters. Instead they are being stored as äöü. The website displays äöü characters correctly if they are already stored as äöü in the ...
2
votes
0answers
632 views

Convert large utf8 database to utf8mb4

I have a large database (~450gb) which I need to convert from utf8 to utf8mb4. I'm aware of the steps I need to take, but is this practical on tables containing this much data? I'm mainly concerned ...
1
vote
1answer
812 views

How do MariaDB and MySQL differ in character set handling?

In MySQL/MariaDB, I set the connection character set to UTF-8: SET NAMES 'utf8mb4'; SET CHARACTER SET 'utf8mb4'; There are some tables with a latin1_swedish_ci collation and latin1 encoded data in ...
1
vote
3answers
4k views

MySQL accent insensitive word search

After spending almost all of my day searching a fix to this I come to you guys. I need to do an accent insensitive word search on my db(InnoDB), but I cannot get it to work. I have t1 table, the ...
4
votes
1answer
5k views

Special Characters in MySQL using UTF8 Unicode Collation and LOAD DATA INFILE

I have several csv files in am loading into MySQl using Java. In the Description field I have several Special Characters that are causing the load to fail. I am using LOAD DATA INFILE. This is nested ...
2
votes
1answer
692 views

presentation of non-ascii characters (ñ)

I'm facing the following problem: I have a MySQL database which contains non-ascii characters in some records. Characters like "ñ", for example. What kind of setup do I need to perform at the ...
0
votes
1answer
735 views

Database convert character_set_server latin1 to utf8

I am writing about a problem on the character encoding that i have on my site with Magento. In practice, the special characters do not take utf8 and gives me the classic non-coded characters: Ã. ...
4
votes
2answers
5k views

Postgres: set client encoding default

When querying from certain tables, I'll often receive the error: character with byte sequence 0xef 0xbf 0xbd in encoding "UTF8" has no equivalent in encoding "WIN1252" Varying bytes, but you get the ...
2
votes
1answer
3k views

ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes

I have a databases with latin1 charachter set, I need to change tables character set to utf8, but I get error. the current character set is : mysql> show variables like "%character%"; +-----------...
1
vote
1answer
2k views

How large space will be occupied by mysql for a varchar utf8 column?

Tip: To save space with UTF-8, use VARCHAR instead of CHAR. Otherwise, MySQL must reserve three bytes for each character in a CHAR CHARACTER SET utf8 column because that is the maximum possible length....
0
votes
1answer
103 views

Simple Linux MySQL GUI Client That Reads UTF-8?

I'm an inexperienced MySQL user running Ubuntu and I need to open a database full of Hebrew data. I prefer a GUI and I downloaded MySQL Workbench. Not only is it incredibly confusing for me, but when ...
0
votes
2answers
74 views

Convert all the data of table (which have both encoding data) into single encoding either utf8 or latin1 [closed]

We have table which consists of two different encoding data (utf8 and latin1) which has been inserted from two different use cases of application. We are getting broken strings issue for other ...
2
votes
1answer
2k views

mysqldump collation issues

I have tried my hand at building a shell script to backup our wordpress database using tar balls and chron jobs on a Debian environment. The problem I am facing now is that our wp database after ...
1
vote
1answer
2k views

Cannot set character_set_database and character_set_server to utf8mb4

I'm trying to convert my MySQL server and its already existing schemas to the utf8mb4 character set. Preliminary Notes: I'm using Xampp (with MySQL 5.6.24). Default options are read from the ...
4
votes
1answer
3k views

Postgres database encoding problem

I'm striving to convert badly encoded data from my table. For instance, I have a field with Nadège which should be Nadège. I tried using Postgres's functions convert, convert_from, convert_to ...
4
votes
1answer
6k views

Convert MySQL database from latin1 to utf8mb4 - and take care of German umlauts

I am using 5.6.28 MySQL Community Server to host WordPress + ProPhoto blog in German language at CentOS 6.7 Linux server: mysql> show tables; +-----------------------+ | Tables_in_blog | +--...
1
vote
2answers
467 views

Which accented characters are considered the same?

In MySQL, certain characters are considered the same when performing searches, e.g. 'n' and 'ñ'; unless you specify BINARY in the condition. Is there a list somewhere of the characters that are ...
2
votes
1answer
370 views

mysqldump shows pairs of utf8 chars when dumping a utf8 database

I'll describe right away the problem, and put details at the bottom. I am dumping a database which I am fairly sure contains (almost: a few cells have wrong, latin1 encoded data) only UTF8 data. Also ...
0
votes
0answers
83 views

Database Importing Collation

Migrating from DB2 found this error message: Error message: Incorrect string value: '\xF2\xAC\x9F\xB0or...' for column 'column_name' This is when trying to import an XML to MySQL. I should have a ...
1
vote
2answers
1k views

How to identify and remove the SQL table row which contains UTF 8 BOM characters

I recently identified that some of the rows in our table contains UTF 8 BOM (Byte Order Marking) characters ().. Is there a way to identify those rows and removing the BOM characters from those ...
0
votes
1answer
874 views

Gotchas converting latin1_swedish_ci to utf8mb4_unicode_ci

I have a MySQL 5.5.31 database which has approx 220 tables - of these 220 tables, around half of them are already using utf8mb4_unicode_ci but the "older" tables are still using latin1_swedish_ci. The ...
2
votes
2answers
8k views

MySQL silently replaces UTF chars with literal question marks

I'm experiencing a situation similar to this SO question i.e. I'm working with a legacy database which has UTF8 content in latin1 tables (pretty ugly I know). Now I'm getting new data from a new ...
2
votes
2answers
33k views

How to convert a MySQL database from utf8mb4 to utf8?

I would like to convert a MySQL database, which has utf8mb4 character set and utf8mb4_unicode_cicollation, to utf8 and utf8_unicode_ci. How to do that please? Thank you in advance for your ...
1
vote
1answer
529 views

utf8 and utf8mb4

I am converting a few of my tables to utf8mb4 as well as the connection from PHP to MySQL to utf8mb4. This is so that I can support characters used for Emoji in my website. I have two questions ...
1
vote
1answer
3k views

Can't insert arabic text into mysql database using mysql prompt

I had a local table with arabic values.when exported i got the following query. DROP TABLE IF EXISTS `type`; CREATE TABLE IF NOT EXISTS `type` ( `bs_id` int(11) NOT NULL AUTO_INCREMENT, `types` ...
1
vote
1answer
121 views

Find efficient way of migration 2TB GBK table in SybaseIQ to UTF-8 files

My project need to migrate more than 2TB raw data from SybaseIQ to another Database. SybaseIQ using GBK, the target Database using UTF-8, the target database can only load from UTF-8 files. We use ...
123
votes
2answers
231k views

Create a MySQL database with charset UTF-8

I'm new to MySQL and I would like to know: How can I create a database with charset utf-8 like I did in navicat? create mydatabase; ...seems to be using some kind of default charset.
4
votes
1answer
3k views

Encoding Debug UTF8 & Latin 1

We have on our DB some international people that got some characters saved incorrectly on our DB (MS SQL-Server 2008 R2). For example we have strings like 'WilcoxonÃ'. We got this data into ...
2
votes
1answer
1k views

Remove question marks from database(latin1_general_ci table)?

How can I remove question marks(not diamond shaped) from a latin1_general_ci MyISAM table? I tried going back to utf8 and tried to apply things in https://stackoverflow.com/questions/1476356/...