Tag Info

Hot answers tagged

10

No, there isn't. SQL Server doesn't support UTF-8. You need to define your columns as nvarchar/nchar if you want unicode data. Note, internally SQL Server stores this as UCS-2. Note that this has ben requested from MS on Connect and there is an older KB article. And some info on this blog too


8

Good question. I'll give a simplified answer. Oracle supports two character sets simultaneously, by way of different datatypes and parameters. A "normal" database-wide characterset and a "national" characterset. Now, the "normal" characterset affects the way that VARCHAR2, CHAR and CLOB data is stored. The "national" characterset affects the way that ...


7

WE8DEC is the old DEC MCS character set which stood for Digital Equipment Corporation Multinational Character Set. The WE8 prefix identifies this as a Western European 8-bit character set.


6

The reason for the truncation is quite simple. Some characters (accented ones, for example) in the WE8ISO8859P1 character set are stored as a single byte, but in AL32UTF8 they end up being stored as multiple bytes. As a result of conversion, a 4000 character string may end up actually requiring more than 4000 bytes. By way of example, this query shows you ...


5

I am a little doubtful that this is exactly what you are looking for, but host echo %nls_lang%; ENGLISH_UNITED KINGDOM.WE8ISO8859P1 shows the client nls_lang environment variable on the client. I don't think there will be a SQL query you can run to give the 'current' setting because AFAIK the server is not aware of what translation is done client-side, ...


4

Your database character set is WE8ISO8859P1, it doesn't have the € sign. Either use NVARCHAR2 or change your database character set (export, reinstall with a character set that supports the euro sign like WE8ISO8859P15 or AL32UTF8, and import). Here's an example of NVARCHAR2: SQL> SELECT * FROM nls_database_parameters WHERE parameter LIKE '%SET'; ...


4

I’ve recently written a detailed guide on how to switch from MySQL’s utf8 to utf8mb4. If you follow the steps there, everything should work correctly. Here are direct links to each individual step in the process: Step 1: Create a backup Step 2: Upgrade the MySQL server Step 3: Modify databases, tables, and columns Step 4: Check the maximum length of ...


4

The columns in the connection manager that your datasource is using need to be defined as "Unicode String (DT_WSTR)".


4

I believe this is covered by the documentation here


3

Think about it: You are storing data in the database as latin1 You are data is handled internally by mysqld as latin1 If data coming from the OS or from the connection is utf8, how is mysqld going to treat it? Rather than guessing or hoping for the best, you could change the incoming character set behavior. With the exception of information_schema and ...


3

And to get your columns to DT_WSTR as liam.confrey mentions, you will want to click on your flat file connection manager and redefine each column type from string [DT_STR] to Unicode string [DT_WSTR] Do note that if you misclick like I usually manage to do and select something like 'two-byte unsigned integer [DTUI2]', classic off by one, the ...


3

If the collation of your database is set to accent insensitive, then this should work (for most accented characters). For example in the collation Latin1_General_CI_AS, the CS means Case Insensitive, Accent Sensitive. You would want the collation to be Latin1_General_CI_AI which means Case Insensitive, Accent Insensitive Be warned though, that not all ...


3

I know what you mean. Alter table sometimes has to rebuild the whole table when you use it. Assuming that the Alter table statement is annoying you because it takes too long AND it locks your table, then you can use 2 tools to make it an "online" alter table. One if from facebook ...


3

By default MySQL server is using latin1 character set for each incoming connection. Latin1, as you might know, does not support cyrillic symbols. The simplest solution is to switch so called 'connection character set' by running SET NAMES 'utf8'; in the beginning of each connection. For example, this query should work: SET NAMES 'utf8'; CREATE TABLE ...


3

I'm not certain. I tried to start out be reproducing your problem but the alter worked fine for me. test > CREATE TABLE `bar` ( `content` text ) ENGINE=MyISAM DEFAULT CHARSET=latin1; INSERT INTO bar VALUES (0x8081828384858687898A8B8C8D8E8F909192939495969798999A9B9C9D9E9F); Query OK, 0 rows affected (0.02 sec) Query OK, 1 row affected (0.00 sec) ...


3

Assuming your source data contained only English and Greek characters, and was mis-exported with an ISO-8859-1 to UTF-8 conversion (rather than an ISO-8859-7 to UTF-8 conversion), you can get your data back by first repeating the missed conversion the other way around, and then doing the right one. You could use iconv for this (available on pretty much all ...


3

You need to take the data from UTF-8 and convert it into UCS-2LE using something like iconv. For example, using the character in your example: echo "010000: dcb3" | xxd -r -s -0x10000 | iconv -f "UTF-8" -t "UCS-2LE" | xxd 0000000: 3307 Now I'm not sure what character UTF-8 \xdcb3 is, but apparently it's correct translation to UCS-2LE is \U0733. If you ...


3

You have to specify that you are inserting wide character data: CREATE TABLE #t (id INT,c1 VARCHAR(MAX),c2 NVARCHAR(MAX)); INSERT INTO #t VALUES(1,'žđšćč žđćčžđšćčŽĐŠĆČ','žđšćč žđćčžđšćčŽĐŠĆČ'); INSERT INTO #t VALUES(2,N'žđšćč žđćčžđšćčŽĐŠĆČ',N'žđšćč žđćčžđšćčŽĐŠĆČ'); SELECT * FROM #t; DROP TABLE #t; Result:


2

Opinion It's probably better to throw an exception during your applications input checking and not pass the buck to the database. Workaround There is a "workaround" but your mileage may vary: http://forge.mysql.com/worklog/task.php?id=3780 Brute Force? You could convert your front end table VARCHAR field to a BLOB and store as binary data to cure the ...


2

I had the same issue with a migration from 9i to 11g. I decided that I did not want to take a chance that Oracle's character set conversion would do something I did not expect. Managers don't care about character sets, they just want to know that the data was migrated without any problems. I kept the WE8MSWIN1252 character set and there were no problems. ...


2

It is an attempt to store a 2-byte string into one byte Since the maximum length of TEXT is 65535, it can safely hold 32767 (65536/2 - 1) 2-byte characters without an error message. Any attempt to add 32768 2-byte characters will result in Incorrect string value: '\xC3\xA9' for column 'mycol' at row 1 because the 32768th character does not have the room to ...


2

You could try turning on the general query log, which logs every query the server receives, and then use that information to see what's being queried, and then issue those queries yourself and examine the data. To see the current setting for the general log (location and whether it's enabled): mysql> SHOW VARIABLES LIKE 'general_log%'; # shows the ...


2

You might try: Take a mysqldump w/ -e --no-create-info > source.sql Truncate all tables Alter all tables source source.sql As with all risky operations, perform in a test environment first to verify the outcome. When you "source source.sql", if literally running from the mysql prompt don't forget to set names utf8. If catting into mysql, run mysql w/ ...


2

According to the MySQL Documentation Collations have these general characteristics: Two different character sets cannot have the same collation. Each character set has one collation that is the default collation. For example, the default collation for latin1 is latin1_swedish_ci. The output for SHOW CHARACTER SET indicates which collation ...


2

You can restore your database using this command mysql -u someuser -p --default-character-set=utf8 somedatabase < backup.sql This overrides the default client character set. I would also make sure that my tables are also in utf8. So when you open up your backup file with a text editor, you should see utf8 as the DEFAULT CHARSET CREATE TABLE ...


2

The solution isn't precisely the same but this question is where I originally found direction for a similar issue and the concepts there should take you where you want to go. MySQL has a BINARY character set and from all appearances, by converting through it, you can prevent MySQL from realizing what you're actually doing and being "too helpful." Test case ...


2

MySQL's documentation for utf8 shows that it will use 1 byte for Latin characters, and only use more if the situation requires them. Therefore, if you're only using normal latin characters, both utf8_general_ci and latin1_general_ci will use between 1 and 4 bytes: one byte to store the length (0-3 characters), and then up to three bytes for the actual text. ...


2

I don't think collation is your problem as that is related to sorting and ordering (although you might get a problem later using ORDER BY and so on), but you mention you have some VARCHAR fields. Those will not accept Unicode characters (which Cyrillic certainly are): They will need to be NVARCHAR throughout to do that. Next, how are you inserting your ...


1

Found a solution. As I looked more into my first DB it had UTF-8 encoding, but all fields in tables was latin1. And data in there was windows-1257. So the solution: SELECT CONVERT(CAST(pavad as BINARY) USING cp1257) pg_zodynas


1

One solution is to have both databases using the same character set. Considering your destination is a free XE version why not do a new install? Just make sure you use EL8MSWIN1253 character set and the same UTF version.



Only top voted, non community-wiki answers of a minimum length are eligible