SQLCMD command is not able to insert accents

Question

I am trying to run sqlcmd.exe in order to setup a new database from command line. I am using SQL SERVER Express 2012 on Windows 7 64 bits.

Here's the command I use:

SQLCMD -S .\MSSQLSERVER08 -V 17 -E -i %~dp0\aqualogyDB.sql -o %~dp0\databaseCreationLog.log 

And here's a pieceof the sql file creation script:

    CREATE DATABASE aqualogy 
    COLLATE Modern_Spanish_CI_AS
    WITH TRUSTWORTHY ON, DB_CHAINING ON;
    GO
    use aqualogy
    GO
    CREATE TABLE [dbo].[BaseLayers] (
    [Code] nchar(100) NOT NULL ,
    [Geometry] nvarchar(MAX) NOT NULL ,
    [IsActive] bit NOT NULL DEFAULT ((1)) 
    )

    EXEC sp_updateextendedproperty @name = N'MS_Description', @value = N'Capas de     cartografía base de la aplicaicón. Consideramos en Galia Móvil la cartografía(...)'
, @level0type = 'SCHEMA', @level0name = N'dbo'
, @level1type = 'TABLE', @level1name = N'BaseLayers'

Well, please check that there are some accents on the words; which is the table's description . The database is created with no problems. 'Collate' is understood by the script, as you can see in the attached screenshot. Despite of this, accents are not properly shown when examining the table.

I'd really appreciate any help. Thank you very much.

[Edit]: Hi all. Changing the SQL file encoding using Notepad++ worked fine! Thank you very much for you help: I learned something interesting with this problem!

Answer 1

As from the comments, the problem is not exactly with the table or the way SQLCMD imports the special characters. Usually, problematic imports are related to the format of the script itself.

Management Studio itself offers the option of saving with a specific encoding, which should solve the problem in the future. When saving a file for the first time (or use save as) you should click on the small arrow near the Save button, to use the option Save with encoding.

By default it saves the file in Western European (1252). Whenever I have any special characters I use UTF8 (though maybe some other restrictive encoding would fit) because it's usually the fastest fix.

I'm not sure (from the pic) that you're using SSMS, so please make sure that your own editor has the option of saving the file in a different encoding. If not, converting the file in a smart editor (like you've already tried in Notepad++) usually works. Though that might not work if you're converting from a wide encoding to a narrower one and then back to a wide one (eg: from Unicode to ANSI and back to Unicode).

Answer 2

Another option, one which I've just learned, comes from the sqlcmd documentation. You need to set the codepage for sqlcmd to match that of the file encoding. In the case of UTF-8, the codepage is 65001 so you'd want:

SQLCMD -S .\MSSQLSERVER08 -V 17 -E -i %~dp0\aqualogyDB.sql -o %~dp0\databaseCreationLog.log -f 65001

Answer 3

This sort of thing is very tricky because so much is done without telling you.

The first thing I would do is use sqlcmd to display the string. If it displays correctly in the cmd.exe window, that's one useful fact. Next, I'd select the row converting the string to varbinary to see what bytes are actually there. I think cartografía will show up as 0x636172746f67726166c3ad61, where the accented "i" is represented by the bytes c3ad, which is the UTF-8 encoding for that character. It's no good having UTF-8 in a Modern Spanish column (Windows 1252). The byte value in Windows 1252 for that character is 237 decimal (hex ED).

If the column holds misencoded data, then the error lies in how it was inserted. Perhaps removing the leading N in the string constants -- N'string' tells SQL Server to generate a Unicode string, but plain 'string' indicates the characters use the client's encoding -- would insert Modern Spanish instead of Unicode.

If the column holds correctly encoded data, then I would say you found a bug in the GUI display.

If you can't get sqlcmd to insert the data correctly (leading N or not), then you want to complain to Microsoft. When you do, being able to show the bytes as stored in the column -- using convert(colname as varbinary) -- will be critical to explaining what's going wrong.

Answer 4

This worked for me, but for some reason the command -f 65001 did not.

sqlcmd -s serverName -f i:1252 -i c:\sqlUpdate.sql -f o:65001 -o c:\sqlUpdateOutput.sql -d userDatabaseName