Questions tagged [utf-8]
Unicode variable-width character encoding (between 1 and 4 bytes per code point).
104
questions
-3
votes
0
answers
35
views
Where/how to get a list of all characters included in utf8mb4?
Where/how to get a list of all characters included in utf8mb4?
I don't care about the programming language to get it if any is requried. A website with a list would suffice too.
Surprisingly, simply ...
0
votes
1
answer
164
views
SQL Server 2019, utf-8, replication problem
I know this is late to the game, but we're just upgrading to Sql Server 2019 after years and years on 2008.
I started using the Latin1_General_100_CI_AI_SC_UTF8 collation as an experiment and at least ...
- 396
2
votes
2
answers
4k
views
How to bulk convert ISO8859 text in Text columns to UTF8 in nvarchar(max) columns?
I've been struggling for days with a problem on a MS SQL database. Here is the situation: I have an old Open Source PHP application (Limesurvey version 1.92) running on a Windows 2008R2 server, with ...
CommunityBot
- 1
2
votes
1
answer
416
views
How can I detect double-encoded MySQL columns and rows, and validate the repair?
My database provider, bless their hearts, migrated our MySQL databases to another server recently, and introduced double-encoding of UTF-8 data via Latin1 into our text data. Strings like 'emdash—here'...
CommunityBot
- 1
1
vote
1
answer
2k
views
MySQL/MariaDB connection in asp code encodes wrongly charset utf8 at insert or update in data base
I can't get a simple update or insert to save special äöü characters. Instead they are being stored as äöü.
The website displays äöü characters correctly if they are already stored as äöü in the ...
CommunityBot
- 1
3
votes
1
answer
5k
views
Why pgAdmin4 invoke "'utf8' codec can't decode byte 0xe9 in position 42: invalid continuation byte" error?
I'm newbie in postgreSQL ,when I'm trying to create a new local server using pgAdmin4 ,it shows me this error :
'utf8' codec can't decode byte 0xe9 in position 42: invalid continuation byte
I'm ...
CommunityBot
- 1
2
votes
1
answer
102
views
CHAR and VARCHAR lengths for multibyte characters
I have a table with ~100mln rows. One of the columns is a VARCHAR(64) with utf8mb4 encoding for storing user nicknames (others are few integers).
The data in that column has MAX(LENGTH()) == 42 and ...
- 23
0
votes
1
answer
713
views
converting latin to utf8mb4 causes questionmarks
The original format of the data is unknown
The new table is in utf8mb4_general_ci
If I do CONVERT(BINARY CONVERT(column USING latin1) USING UTF8) as mentioned here - it fixes all text, but converts ...
CommunityBot
- 1
1
vote
2
answers
992
views
Changing from utf8 to utf8mb4
MySql database at the moment has:
Character Set: utf8
Default Collation Name: utf8_unicode_ci
I want to change it to:
utf8mb4
utf8mb4_unicode_ci
I would also do the same thing for all tables in ...
- 77k
3
votes
1
answer
5k
views
How large space will be occupied by mysql for a varchar utf8 column?
Tip: To save space with UTF-8, use VARCHAR instead of CHAR. Otherwise, MySQL must reserve three bytes for each character in a CHAR CHARACTER SET utf8 column because that is the maximum possible length....
- 77k
2
votes
1
answer
364
views
How can I detect failure in CONVERT( tbl AS utf8)?
MySQL provides the CONVERT(`tbl` AS utf8) expression to transcode text from one charset to another.
Sometimes that conversion cannot succeed, because the destination charset does not include ...
- 83k
0
votes
2
answers
505
views
Converting from latin1 to utf8mb4 of ENUM and SET with accented characters on mysql
I am trying to convert tables from latin1 to utf8mb4 on MySQL 5.7.
Everything works fine except the tables with ENUM and SET which contains a value with an accented character. 2 types of errors are ...
- 77k
0
votes
0
answers
100
views
Migrate postgresql to MariaDB
I try to migrate mysql Zabbix database from Postgresql to MariaDB using MySQL Workbench migration Wizard
I have compiled psqlODBC and enable it
When I setup the source database in the wizard the test ...
CommunityBot
- 1
5
votes
1
answer
507
views
postgresql matching or converting utf-8 variant strings
Postgres 13
I am looking for a way to search UTF-8 text that may have variant character representations ( what is the proper term for this? ie 𝐋𝐈𝐅𝐄 vs life ) within postgresql.
I am running into ...
- 2,809
2
votes
1
answer
750
views
Postgres encoding issue - find values that have no equivalent in (client) encoding
I have Postgresql database that internally uses UTF-8 encoding. Some clients that connect to the database use LATIN 2 (ISO 8859-2) client encoding when they connect to database. I can't change the ...
- 123
4
votes
1
answer
583
views
Fix corrupted utf8 nvarchar value that was inserted without N prefix
Is it possible to restore the UTF8 data from a corrupted nvarchar, that was used without the N prefix?
For example, in the following snippet, I'd like @n1_fixed to have the correct value based on @n1:
...
- 27.8k
0
votes
1
answer
528
views
Tables character-set migration without huge downtime- mysql
We're in the process of migrating to a new character set for several tables in our database. Some tables are currently using utf8/utf8mb3 character set, which is deprecated now and we want move on ...
1
vote
0
answers
1k
views
PSQL console encoding
How can I have a working PSQL console using UTF8 encoding under Windows?
I have a Windows server and client. The Postgres 12 database contains tables with content in multiple languages (ex: English, ...
- 101
1
vote
2
answers
2k
views
How to identify and remove the SQL table row which contains UTF 8 BOM characters
I recently identified that some of the rows in our table contains UTF 8 BOM (Byte Order Marking) characters ().. Is there a way to identify those rows and removing the BOM characters from those ...
- 101
0
votes
2
answers
529
views
What is the definition of latin1 character set in MySQL 8.0 and MySQL 5.7?
I am trying to fix some corruption to character fields in my database after a migration. I suspect that part of the problem is that fields with UTF-8 data were interpreted with the MySQL latin1 ...
- 191
0
votes
1
answer
623
views
Import postgresql database with Latin encoding
I'm going to upgrade from postgresql 9 to postgresql 11, the encoding used is Latin.
I've exported the database using this command pg_dumpall | gzip > /tmp/db.sql.gz
I extracted the exported ...
- 48k
4
votes
1
answer
11k
views
Storing special characters (German, French, Spanish) in SQL Server result in weird strings
I'm trying to store the German ß in SQL Server, but it results in ß.
For example "Gleimstraße" is stored as "Gleimstraße"
But this also happens for other characters:
"König-...
- 245
2
votes
1
answer
158
views
Find efficient way of migration 2TB GBK table in SybaseIQ to UTF-8 files
My project need to migrate more than 2TB raw data from SybaseIQ to another Database. SybaseIQ using GBK, the target Database using UTF-8, the target database can only load from UTF-8 files.
We use ...
CommunityBot
- 1
3
votes
2
answers
2k
views
MySQL What charset/collation for Case insensitivity and Accent sensitivity?
I am looking for a charset/collation that would make it so when I do a
SELECT * FROM table_name WHERE username = "Warrior"
It only returns me the rows where username = "Warrior", ...
- 77k
0
votes
1
answer
204
views
Arabic language search in codeigniter error : Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE)
Arabic language search in codeigniter error : Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE)
In English language search is working
JOIN Query to use to ...
CommunityBot
- 1
1
vote
1
answer
1k
views
How to fix unicode REPLACEMENT CHARACTER � in postgres database
Is there a way fix the � problem in postgres database:
As you can see there are names with accents and are not displayed corretly in the database.
I would like to know if there is a way to fix this ...
- 48k
0
votes
1
answer
492
views
How can I know if there is data loss when converting mysql character set(s)?
I'm converting a large (70G) legacy mysql database that is mostly utf8 (but with a sprinkling of other encodings in fields) to one that is (as uniformly as possible) utf8mb4 (utf8mb4_unicode_ci).
The ...
- 77k
1
vote
1
answer
710
views
MySQL query show something like 0x8081 instead of special characters
When i run SELECT CHAR(128,129,130,131,132,133,134,135,136,137);
I got 0x80818283848586878889 istead of Çüéâäàåçêë.
Does anybody know why?, I'm using charset utf8mb4.
When I run show variables like '%...
- 83k
4
votes
1
answer
795
views
What is difference between Arabic_100_CS_AS_KS_WS_SC_UTF8 and Latin1_General_100_CS_AS_KS_WS_SC_UTF8?
Starting from SQL Server 2019, it supports UTF-8 as collation. However, according to the following queries:
SELECT COLLATIONPROPERTY('Arabic_100_CS_AS_KS_WS_SC_UTF8', 'CodePage')
SELECT ...
- 68.3k
1
vote
2
answers
5k
views
mysqldump collation issues
I have tried my hand at building a shell script to backup our wordpress database using tar balls and chron jobs on a Debian environment.
The problem I am facing now is that our wp database after ...
- 1,402
4
votes
1
answer
1k
views
How to migrate Unicode UTF-8 CLOB data from Oracle to SQL Server 2017 UTF-? [closed]
I am currently assisting in a migration effort of this application from Oracle 12c to SQL Server 2017. Initially I just performed table inserts using openquery to Oracle. I discovered that the tables ...
CommunityBot
- 1
4
votes
2
answers
3k
views
Having charset "utf8mb4" - does it mean every character takes 4 bytes, or only those that need 4 bytes?
I have a VARCHAR(80) utf8mb4 column, and I'm testing adding some ASCII & Emoji characters, and use LENGTH() and CHAR_LENGTH() to understand the differences.
From reading in different places, my ...
- 809
1
vote
1
answer
2k
views
mysqldump dumps different data with and without --no-create-info
mysqldump dumps different representations of data when called with/without --no-create-info.
Test case
First, create the test table and populate it with interesting data.
CREATE TABLE `test` (
`...
- 182k
216
votes
2
answers
552k
views
Create a MySQL database with charset UTF-8
I'm new to MySQL and I would like to know:
How can I create a database with charset utf-8 like I did in navicat?
create mydatabase;
...seems to be using some kind of default charset.
- 1
5
votes
1
answer
3k
views
What is the difference between different utf8mb4 binary collations?
What is the difference between utf8mb4_0900_bin vs utf8mb4_bin binary collations?
- 68.3k
0
votes
1
answer
5k
views
Character set 'utf8mb4' is not a compiled character set and is not specified in the '/usr/share/mysql/charsets/Index.xml' file
Full error
Character set 'utf8mb4' is not a compiled character set and is not specified in the '/usr/share/mysql/charsets/Index.xml' file
InvalidArgumentException: There was a problem connecting to ...
CommunityBot
- 1
3
votes
1
answer
11k
views
CONVERT() returns garbage with CLOBs in Oracle
I am looking to repair a table of data in Oracle, ideally via unprivileged SQL, which has had UTF-8 data inserted into a UTF-8 database, using the Latin-1 character set by mistake.
The symbol β GREEK ...
- 9,252
0
votes
1
answer
10k
views
CSV (UTF-8) to Postgres database (UTF-8), doesn't show äöå characters
I have csv-file that is UTF-8 encoded. I'm using python (psycopg2) to copy it into a Postgres database. My Postgres database is using UTF-8. However I have problems to show Ä, Ö and Å characters.
If ...
CommunityBot
- 1
0
votes
1
answer
1k
views
Accent marks/symbols not displaying correctly in MySQL database
Anytime there's an accent mark (é or ') or symbol (&) in my database it displays as HTML code. The table CHARACTER SET is UTF-8 Unicode and the COLLATION is utf8mb4_general_ci, which should ...
- 77k
0
votes
0
answers
1k
views
How to convert UTF8 general in UTF8 unicode?
Short:
My tables have the collation utf8mb4_unicode_ci, but the data itself is utf8mb4_general_ci and now I like to know how I can convert it into unicode?
Long:
I used this query to convert all my ...
- 133
3
votes
1
answer
3k
views
Is there some function in PostgreSQL to simply check if the input string is valid UTF-8 according to PG?
For several years now, I've been hunting this extremely quick-footed little bug which knows every hiding place in my metaphorical coding house. He lurks in the walls, sometimes takes months to show ...
- 48k
10
votes
2
answers
8k
views
Upgrade all MySQL columns, tables, and databases from utf8mb3 to utf8mb4
I have thousands of columns across hundreds of tables in about a hundred databases inside a MySQL instance that need to be upgraded from utf8mb3 to utf8mb4. Is there a way to generate ALTER ...
3
votes
2
answers
4k
views
SQL Server 2019 UTF-8 Support Benefits
I'm already quite comfortable with using COMPRESS() and DECOMPRESS() in an internal forum software for our company (Currently in SQL Server 2017), but trying to make the database as efficient as ...
- 68.3k
1
vote
0
answers
293
views
MariaDB REGEXP_REPLACE Invalid utf8 byte sequence
I'm working with a fairly old (15 years) old database. It started as a MySQL3, was upgraded several times, the frontends had problems with proper encoding, the whole database was converted to utf8mb4 (...
- 111
1
vote
2
answers
5k
views
Spool Unicode data in sqlplus to ISO-client without character set conversion
I'm currently facing the following problem:
We have an Oracle 11g database with character set AL32UTF8, and an sqlplus-client with NLS_LANG=GERMAN_GERMANY.WE8ISO8859P1.
Stored in the database are ...
CommunityBot
- 1
20
votes
1
answer
24k
views
Querying non-ASCII rows from Postgres
Does [:ascii:] class work in Postgres at all? It is not listed in their help, however I see examples in the web which utilize it.
I have a UTF-8 database, where collation and c_type are en_US.UTF-8, ...
- 367
0
votes
0
answers
557
views
Will MariaDB treat ut8/utf8mb3 and utf8mb4 collations as compatible?
Background on relevant character sets
MySQL and MariaDB support the deprecated utf8mb3 character set and collations. (utf8 is currently an alias for utf8mb3.) This supports all Unicode code points on ...
- 123
-1
votes
1
answer
2k
views
Workbench Error Code 1267 - utf8_unicode and utf8_general
I've recently just swapped all of a databases tables and columns to utf8_unicode_ci from the weird collection of about 3-5 different collations they were previously (don't ask... not my database)
So, ...
CommunityBot
- 1
0
votes
0
answers
424
views
ALTER TABLE query to convert table to utf8mb4_bin takes many hours - how to optimise this?
I have a table which has approx 15 million rows - I won't add the schema here but it has 96 columns (far too many in my opinion).
The current table collation is latin1_swedish_ci but for full emoji ...
- 111
-1
votes
1
answer
1k
views
UTF-8: showing correct in database, but not in webpage
When i upload data from php post forums "हिन्दी" language, changed into "हरिà" in database but looking correct in webpage,
and
when i upload data through phpmyAdmin from csv file "हिन्दी" ...
CommunityBot
- 1