Questions tagged [encoding]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
1answer
431 views

How to configure DBeaver to save files without a BOM for UTF files?

I am looking for the equivalent setting as the PGADMIN setting Write BOM for UTF files which will cause saved files to be saved with/without the BOM. I've found settings in DBeaver when exporting ...
2
votes
0answers
781 views

Umlauts in DBeaver

I have a Postgres Database set to UTF-8. I am working with DBeaver 3.1.4, and it seems to use a different encoding than UTF-8 (I suppose Latin-1 or something), which leads to 1) wrong display in the ...
23
votes
1answer
29k views

How do I set a SQL Server Unicode / NVARCHAR string to an emoji or Supplementary Character?

I want to set a Unicode string variable to particular character based on its Unicode code point. I want to use a code point beyond 65535, but the SQL Server 2008 R2 database has a collation of ...
3
votes
2answers
5k views

Not displaying special characters imported from a file

The scenario is a SQL Server instance, a database which is data fed using BULK INSERT operations mainly, and some of the texts inserted contain special characters like ñ because I am working in a ...
0
votes
1answer
139 views

psqlODBC access encoding problem

My local machine: Windows 7, 32-bit. I have installed on it the psqlODBC for 32-bit. MS Access is 32-bit as well. I have tried both Unicode and Ansi connections to... The server: a 64-bit Ubuntu, and ...
2
votes
2answers
48 views

charindex thrown off by extended characters

I've got a column with a filenames stored in a nvarchar(255). I'm trying to parse out the file extension using reverse( left( reverse(filename), charindex('.', reverse(filename) ) -1 ) ) This works ...
12
votes
1answer
4k views

Error starting SQL Server 2017 service. Error Code 3417

I have SQL Server 2017 installed on my computer. This is what SELECT @@VERSION returns: Microsoft SQL Server 2017 (RTM-GDR) (KB4293803) - 14.0.2002.14 (X64) Jul 21 2018 07:47:45 Copyright (C) ...
-1
votes
1answer
342 views

UTF-8: showing correct in database, but not in webpage

When i upload data from php post forums "हिन्दी" language, changed into "हरिà" in database but looking correct in webpage, and when i upload data through phpmyAdmin from csv file "हिन्दी" ...
2
votes
0answers
31 views

Create/Edit SSIS Derived Column Task From Text/Code for Large ETL

I have to insert a large UTF-8 encoded flat file to a 1252 encoded sql server. In order to do so, I am using SSIS to type cast each column. Is there a way, other than copy pasting through all 400 ...
5
votes
1answer
96 views

Query to find rows containing ASCII characters in a given range

I am using some scripts from another topic, but the accepted answer isn't working for all my data scenarios. I would have asked my question on the original How to check for Non-Ascii Characters post, ...
4
votes
2answers
532 views

Different characters, same ASCII code?

I have this query that throws two results: SELECT id FROM table1 WHERE id like 'nm041033%' nm0410331 nm0410331 And this slightly different query that throws only one result: SELECT id FROM table1 ...
1
vote
1answer
37 views

Is there a MySQL character set and encoding that will allow for both emojis and accents?

I've got a database of terms that get added to by one group of users, and queried against by another. I was running into problems when people would query for an emoji in the database and my React app ...
5
votes
2answers
1k views

Why is my UTF-8 document raising UTF-8 encoding errors in Azure Data Lake Analytics?

I have a document that was compressed in gunzip from a unknown source system. It was downloaded and decompressed using a 7zip console application. The document is a CSV file that appears to be encoded ...
15
votes
3answers
40k views

Using PostgreSQL 8.4, how to convert bytea to text value in postgres?

In my application I insert data in database using C code, since the strings I receive from an untrusted source I have escaped them using PQescapeByteaConn of libpq library. Which is working perfectly ...
3
votes
2answers
4k views

Character Conversion Issue Importing From CSV File

After doing a load of CSV files, there are various words that are "written" to the database incorrectly. Some examples: Diã¡ria should be Diária Crã©dito should be Crédito Ligaã§ãµes should be ...
1
vote
2answers
89 views

Handling data encoding issues while loading data to SQL from Script (Notepad++)

I'm pretty sure this is not a SQL Server problem. I already asked a question HERE with an awesome explanation, BUT I still can't explain to the guys where I work that it has nothing to do with SQL ...
1
vote
3answers
798 views

PostgreSQL: difference between collations 'C' and 'C.UTF-8'

In PostgreSQL, what is the difference between collations C and C.UTF-8? Both show up in rows of pg_collation. Is it perhaps the case that C.UTF-8 is the same as C with encoding UTF-8 regardless or ...
17
votes
4answers
9k views

Why does the varchar datatype allow unicode values?

I have a table with a varchar column. It is allowing Trademark(™), copyright(©) and other Unicode characters as shown below. Create table VarcharUnicodeCheck ( col1 varchar(100) ) insert into ...
16
votes
4answers
70k views

SQL Server 2005/2008 UTF-8 Collation/Charset

I can't find option(s) directly to set UTF-8 rellated Collations/Charsets in SQL Server 2005/2008, same as is possible to set in another SQL engines, but in SQL Server 2005/2008 are there only Latin ...
4
votes
2answers
609 views

Byte ordering for multibyte characters in SQL Server versus Oracle

I am currently in the process of migrating data from Oracle to SQL Server and I'm encountering an issue trying to validate the data post-migration. Environment Details: Oracle 12 - AL32UTF8 ...
0
votes
1answer
875 views

How can I search for a hex string in oracle?

There's a database record that is incorrect. The name field displays like this in a browser ( this is incorrect ): Thcˇodore And when I look at the record via SQL results, I see: Thc\xCB\x87odore ...
1
vote
2answers
1k views

Bulk insert not retaining special chars of UTF-8 Encoded txt file Sql Server 2008

I have a stored procedure that bulk imports a text file and inserts it into my database. CREATE TABLE DBO.TEMP_STORE ( ID nvarchar(max), [MONTH] nvarchar(max), [...
4
votes
3answers
452 views

Unicode storage of \u202b RLE and \u202c PDE in a Unicode-aware database?

I'm building a new product for toponyms and in it the Arabic shows kinda like this: ^IArabic^I<202b>ﺰﻤﺑﺎﺑﻮﻳ<202c>^I<202b>ﺞﻫﻭﺮﻳﺓ ﺰﻤﺑﺎﺑﻮﻳ<202c>$ Actually not quite. This is a ...
1
vote
1answer
565 views

How to insert a Unicode character verbose into a varchar DB?

I need to insert this character '●' into a VARCHAR column of a MSSQL database with collation set as SQL_Latin1_General_CP1_CI_AS (or at least mock what my Python + Windows MSSQL Driver might have done)...
3
votes
2answers
2k views

SQL Server 2019 UTF-8 Support Benefits

I'm already quite comfortable with using COMPRESS() and DECOMPRESS() in an internal forum software for our company (Currently in SQL Server 2017), but trying to make the database as efficient as ...
0
votes
1answer
51 views

Users running ANSI scripts are having problems with special characers [closed]

Long story short. How can I fix users environments, to make them run our scripts using ANSI encode? The problem is, we send them scripts to run on their databases using ANSI encode. But some of ...
5
votes
1answer
1k views

Why does MS SQL Server return a result for empty string check when Unicode string is not empty

select * from (select N'ግዜ ' as t) as t2 where t= '' the string 'ግዜ ' matches the above check, why is this?
1
vote
2answers
597 views

Wrong sort order using utf8_unicode_ci and swedish characters like ö

I am confused about what encoding and collation I should use so my queries returns the correct sort order. Right now my encoding is UTF-8 Unicode and collation utf8_unicode_ci for both firstname and ...
3
votes
3answers
568 views

TSQL Script Execution Through Power shell, Special Characters Rendering Problem

We have a simple T-SQL script file (.sql), which we execute through Powershell. Everything works fine, except the script portion, where we are using some special characters as string (example: 'ô' ). ...
1
vote
1answer
20 views

Table/Column Character Encoding and Indexes

Does table/column character encoding have any impact on how the MySQL optimizer chooses join indexes? I have a table that have the same columns and indexes on two servers, one server does a eq_ref ...
4
votes
1answer
5k views

Why do I get incorrect characters when decoding a Base64 string to NVARCHAR in SQL Server?

I've been looking at how to decode Base64 with SQL Server and after searching around many solutions online (some from here) seem to be based upon this sort of method. SELECT CAST(CAST('...
1
vote
1answer
53 views

Convert Character Encoding, and Check Encoding for Issues

I'm planning to convert a table from utf8 (_general_ci) to utf8mb4 (utf8mb4_unicode_ci). I want to make sure I don't have any malformed data after the conversion though so I was planning to duplicate ...
5
votes
1answer
2k views

Postgres - Encoding, Collation and CType

List of databases Name | Owner | Encoding | Collate | Ctype | -------------------------+----------+----------+---------------------------...
0
votes
1answer
496 views

How to get line that causes error when loading from csv to MySQL

I'm trying to load data from CSV file, but occurs error Error Code: 1366. Incorrect integer value: 'abcdxxx' for column 'col' at row 123545. I Have file with multiple breaklines CRLF's in content, ...
2
votes
3answers
5k views

NLS_LANG and NLS_CHARACTERSET

I thought I read that NLS_CHARACTERSET has the encoding of the database and NLS_LANG the encoding of the client. Is that correct? Does that mean that the two encodings can be different? In some ...
3
votes
2answers
711 views

Use non-printable characters in SQL Server BCP xml format file as TERMINATOR

I have a flat-text data file containing records which fields are separated by the non printable character "File Separator" (0x1c). I am trying to use SQL Server's bcp utility to load this data into my ...
0
votes
1answer
108 views

Change of NLS_LANG

Until not long ago, the NLS_CHARACTERSET and the NLS_LANG were both ISO88591. We deliver to production some scripts encoded in ISO88591. Since they changed their NLS_LANG in utf8, they have a problem ...
1
vote
1answer
156 views

Is there any string compaction option for PostgreSQL?

I have a table like this: CREATE TABLE BLOG( CATEGORY CHAR(32) ); Category is a column, which stored enum values. For example, I have 1 millions of rows and 10 unique values of Category. Let'...
2
votes
2answers
493 views

DB2 on Windows: Fix encoding in db2cmd and for JDBC

I'm running a DB2 v10.5 FP 7 on a Windows Server 2013 and there is a problem with the encoding setup: The windows is set to use German(Switzerland) as region setting and input language. chcp: Active ...
5
votes
2answers
3k views

SQL Server Bulk Insert properly interprets some Unicode characters but not others?

For some reason the MS SQL Server 2016 bulk insert misinterprets/translates Unicode characters: C9 (É) into 2B (+) A1 (¡) into ED (í) A0 ( ) into E1 (á) AE (®) into AB («) CB (Ë) into 2D (-) D1 (...
2
votes
1answer
1k views

How to Create a PostgreSQL database with different Encoding?

I want to create a New Database that will store data in German(i.e Text with accents etc). Hence I tried to create it using the following SQL: CREATE DATABASE testdb WITH ENCODING='LATIN1' ...
1
vote
0answers
76 views

Merge Replication Business Logic Handler Incorrect Encoding when connecting to SQL Server

We have a SQL Server Merge Replication configuration that uses a business logic handler to do conflict resolution on a datetimeoffset column. All we're doing is a copy of the Subscriber or Publisher ...
8
votes
2answers
4k views

Automatic Translation when Converting Unicode to non-Unicode / NVARCHAR to VARCHAR

Unicode code point 9619 is a character called "Dark shade": ▓ (http://unicode-table.com/en/search/?q=9619). Using the SQL_Latin1_General_CP1_CI_AS collation and 1252 code page, I would expect that ...
1
vote
1answer
235 views

What is the equivalent of the SQL Server CHAR function?

SQL Server has CHAR ( integer_expression ), so I can do SELECT CHAR(65) What is the equivalent on Postgresql? I have tried SELECT E'\0x65'; but I get ERROR: invalid byte sequence for encoding "...
10
votes
1answer
8k views

Encoding issue with SQL Server VARCHAR column retrieved in Python

We recently had an issue with encoding related to a field that's being stored as a varchar(120) in SQL Server. In SSMS, the varchar appears as: "Who Killed JonBen‚t?" However, when it's brought ...
4
votes
1answer
770 views

Collate issues with wrong characters

Well, the problem is well known, but I'm looking for a smarter solution if there's one. For some reason the system doesn't recognize some characters and I can't compare the columns Here is an ...
2
votes
2answers
4k views

Remove characters from PostgreSQL database causing encoding errors

I have a PostgreSQL 10 database that uses WIN1252 encoding. One of my columns has values that cause conversion errors when running a select from pgAdmin 4: SELECT myfield FROM mydb.myschema."MYTABLE"...
3
votes
1answer
6k views

How to be sure that a SQL Script is running with ANSI encoding?

Maybe this is a stupid question or I'm asking it in the wrong way. How can I be sure that a script ( with thousands of lines ) is running with ANSI encoding? Let's say we created a script using ...
1
vote
1answer
85 views

Invalid characters in xml [closed]

I have just downloaded a large xml file and most of characters appear as "�". I changed the encoding but there was no improvement. Anyone knows what this might be due to? One fishy thing: the file ...
0
votes
1answer
5k views

How to export data in Oracle database into CSV file with UTF-8 format?

I have an CSV file with WE8ISO8859P1 encoding generated from Oracle Stored Procedure. Is there any script line to define the output of CSV file in UTF-8 encoded format? CREATE OR REPLACE PROCEDURE ...