1

PostgreSQL has a few features related to full-text search. It knows how to parse a string into tokens (see parsers). For instance, here is how it parses Hello, friends and contributors. into tokens:

# SELECT * FROM ts_debug('Hello, friends.');
   alias   |   description   |  token  |  dictionaries  |  dictionary  | lexemes  
-----------+-----------------+---------+----------------+--------------+----------
 asciiword | Word, all ASCII | Hello   | {english_stem} | english_stem | {hello}
 blank     | Space symbols   | ,       | {}             |              | 
 asciiword | Word, all ASCII | friends | {english_stem} | english_stem | {friend}
 blank     | Space symbols   | .       | {}             |              | 
(4 rows)

On my Ubuntu machine, this can handle Arabic just as well, as you can see here. Even if you can't read Arabic, you can see that it parsed out each Arabic word into a separate Arabic word token:

# SELECT * FROM ts_debug('واحد اثنين ثلاثة');
 alias |    description    | token |  dictionaries  |  dictionary  | lexemes 
-------+-------------------+-------+----------------+--------------+---------
 word  | Word, all letters | واحد  | {english_stem} | english_stem | {واحد}
 blank | Space symbols     |       | {}             |              | 
 word  | Word, all letters | اثنين | {english_stem} | english_stem | {اثنين}
 blank | Space symbols     |       | {}             |              | 
 word  | Word, all letters | ثلاثة | {english_stem} | english_stem | {ثلاثة}
(5 rows)

But on my macOS machine, it doesn't seem to be able to parse Arabic words into separate Arabic tokens at all, it seems to think that all Arabic characters are whitespace:

# SELECT * FROM ts_debug('واحد اثنين ثلاثة');
 alias |  description  |      token       | dictionaries | dictionary | lexemes 
-------+---------------+------------------+--------------+------------+---------
 blank | Space symbols | واحد اثنين ثلاثة | {}           |            | 
(1 row)

How do I get Postgres to parse واحد اثنين ثلاثة into three word tokens (as well as whitespace tokens)?


Possible clue

The documentation does say this:

The parser's notion of a “letter” is determined by the database's locale setting, specifically lc_ctype. Words containing only the basic ASCII letters are reported as a separate token type, since it is sometimes useful to distinguish them. In most European languages, token types word and asciiword should be treated alike.

My LC_CTYPE is set like this:

# SHOW LC_CTYPE;
  lc_ctype   
-------------
 en_GB.UTF-8
(1 row)

However, this doesn't explain why on Ubuntu parsing Arabic works, but no on macOS. If I attempt to create a database with an Arabic locale on macOS, I get this error:

$ createdb --locale ar_AR.utf8 -T template0 hello
createdb: database creation failed: ERROR:  invalid locale name: "ar_AR.utf8"
9
  • use dpkg-reconfigure locales to add ar_AR.utf8 locale to your os. After that restart postgres. Finally try to create database in arabic language.
    – Sahap Asci
    Commented Jun 5, 2018 at 23:25
  • @SahapAsci This doesn't work on macOS.
    – Flimm
    Commented Jun 6, 2018 at 9:08
  • can you check the locale? Use command locale -a | grep _AR in your mac
    – Sahap Asci
    Commented Jun 6, 2018 at 9:12
  • @SahapAsci I tried this on my macOS machine, and there is no Arabic locale installed.
    – Flimm
    Commented Jun 6, 2018 at 9:14
  • Did you tried to create db after you install Arabic locale?
    – Sahap Asci
    Commented Jun 7, 2018 at 22:26

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.