4

I'm building a new product for toponyms and in it the Arabic shows kinda like this:

^IArabic^I<202b>ﺰﻤﺑﺎﺑﻮﻳ<202c>^I<202b>ﺞﻫﻭﺮﻳﺓ ﺰﻤﺑﺎﺑﻮﻳ<202c>$

Actually not quite. This is a real problem for my ASCII-spewing terminal, so I'll make an exception and screenshot text.

Arabic

My question is about those U202B "Right-To-Left Embedding" (RLE), and U202C "Pop Directional Formatting" (PDF). Do those get stored as data? My first assumption was that the characters were rendered and not in the file, but alas they are there..

360    5E03 97E6 5171 548C 56FD 000A 0009 0041 0072 0061 0062 0069 0063 0009 202B 0632    布韦共和国..Arabic..ز
.............................................................................^HERE
389    0645 0628 0627 0628 0648 064A 202C 0009 202B 064F 062C 0647 0648 0631 064A 0629    مبابوي...ُجهورية
.....................................^HERE.....^HERE
422    0020 0632 0645 0628 0627 0628 0648 064A 202C 000A 0009 004E 006F 0074 0065 0073    .زمبابوي...Notes
...............................................^HERE

When storing Arabic in a database, do you typically store \u202b, and \u202c? They seem like they're rendering characters and not technically data? I'm simply wanting to process this text to throw in a database, and wondering if these characters should be present in the database, or stripped before insert.

Background

6

Arabic (as well as Hebrew and Syriac) are right-to-left languages. Hence they display in the opposite direction that the bytes are physical stored in. Having the proper display is controlled through non-printable characters that are interpreted only by the font / rendering system. These two characters in particular are used to control this (see original Unicode spec for starters: https://www.unicode.org/charts/PDF/U2000.pdf ), especially in the context of embedding right-to-left text in the same paragraph as left-to-right text (and the other way around).

So, you must keep them stored or else attempted to display this data later will render it backwards from how the language is supposed to appear and will hence be considered data loss. These are among many formatting control characters that are non-printable / zero-width.

The "official" description of how to work with these characters, from the Unicode Consortium, is (taken from "Chapter 23: Special Areas and Format Characters" top of page 868):

As with other format control characters, bidirectional ordering controls affect the layout of the text in which they are contained but should be ignored for other text processes, such as sorting or searching. However, text processes that modify text content must maintain these characters correctly, because matching pairs of bidirectional ordering controls must be coordinated, so as not to disrupt the layout and interpretation of bidirectional text. Each instance of a lre, rle, lro, or rlo is normally paired with a corresponding pdf. Likewise, each instance of an lri, rli, or fsi is normally paired with a corresponding pdi.

Regarding the importance of keeping (not discarding) these hidden formatting code points, the "Unicode® Standard Annex #9: UNICODE BIDIRECTIONAL ALGORITHM", in section "2.7 Markup and Formatting Characters" states (emphasis mine):

The explicit formatting characters introduce state into the plain text, which must be maintained when editing or displaying the text. Processes that are modifying the text without being aware of this state may inadvertently affect the rendering of large portions of the text, for example by removing a PDF.

and:

Whenever plain text is produced from a document containing markup (ed: HTML and/or CSS), the equivalent formatting characters should be introduced, so that the correct ordering is not lost.

Further explanation is provided in the (excellent) "Understanding Bidirectional (BIDI) Text in Unicode" document by Cal Henderson (taken from the O.P.'s answer) states:

... we could disallow these explicit characters (U+202A - U+202E) which is pretty easy. This does mean that anybody who wants to use them to include Neutrals at the edges of their Arabic usernames will be out of luck - and that sucks more when it's a comment they're posting, where the period jumps to the 'beginning' of the text.

If we want to allow use of these characters, the solution is fairly simple (if hard to implement): we need to make sure that every opening marker has a paired closing marker (PDF) so that the state stack coming out of the string is at the same state as when we went in. We also need to be careful that we don't allow any PDFs to be used without accompanying push markers, else we can't use any ourselves outside of the block.

So, even if the text of a particular cell is supposed to be entirely a right-to-left language, removing these markers could alter the placement of neutral characters (such as punctuation). For example (using SQL Server):

SELECT NCHAR(0x0671) + NCHAR(0x0679) +  N'!';
-- ٱٹ!

SELECT NCHAR(0x202B) + NCHAR(0x0671) + NCHAR(0x0679) + N'!' + NCHAR(0x202C);
-- ‫ٱٹ!‬

Planning to add them back in later, or having a client app add them back in, won't work because there is no inherent means of knowing that they were even being used, and if so, where they were placed.

The safest approach is to keep these characters

For example, you are attempting to include some of this text at the top of the question:

^IArabic^I<202b>ﺰﻤﺑﺎﺑﻮﻳ<202c>^I<202b>ﺞﻫﻭﺮﻳﺓ ﺰﻤﺑﺎﺑﻮﻳ<202c>$

but clearly that's not displaying in the correct order. The bytes, however, are in the correct order:

Looking at just the first <202b>...<202c> section (again, using SQL Server, hence it's Little Endian):

SELECT CONVERT(VARBINARY(MAX), N'<202b>ﺰﻤﺑﺎﺑﻮﻳ<202c>');

the bytes are:

3C00 3200 3000 3200 6200 3E00 B0FE E4FE 91FE 8EFE 91FE EEFE F3FE 3C00 3200 3000 3200 6300 3E00
<    2    0    2    b    >    .    .    .    .    .    .    .    <    2    0    2    c    >

As you can see, there are no additional formatting characters. Because the Arabic characters are strong right-to-left, the characters that follow – <202 – which are neutral (<) and weak (202), continue to display heading to the left (even turning the < into >). And to be clear, the 202 itself displays left-to-right, which would be clearer if the number wasn't a palindrome. If the number was 203, then it would still show as 203 and not 302. But the c is strong left-to-right so it (and the characters that follow) display as expected.

How to fix? Just add the implicit left-to-right markers just after the Arabic to indicate that the right-to-left directionality should end at that point. If we add Code Point U+200E after the last Arabic character (and just before the <) in each of those two segments, we get the following:

^IArabic^I<202b>ﺰﻤﺑﺎﺑﻮﻳ‎<202c>^I<202b>ﺞﻫﻭﺮﻳﺓ ﺰﻤﺑﺎﺑﻮﻳ‎<202c>$

Now, if StackOverflow were to strip out the formatting, then it would revert to the incorrect display, and there is no indication as to the intention of what is desired here that can be discovered programmatically.

If you want to strip out the formatting and add it back in later, are you 100% certain as to the intention of why those characters are there? They aren't always used, so how do you know why they are used when they are present? Don't think there will be any non-Arabic characters? Ok, so how do you classify <202>? I left out the "b" and "c" because there could be punctuation and numbers without any Latin characters and still be "fully Arabic".

This is why I said that keeping them in was the "safest" route to go. Not the only route. But if you don't control the input values, then I don't see how you can guarantee never accidentally altering the meaning of the data.

4

Your situation is more profound than just a matter of handling U+202B and U+202C characters. I'll answer your proximate questions first, then get to the more important answers.

When storing Arabic in a database, should you typically store U+202B "Right-To-Left Embedding" (RLE) and U+202C "Pop Directional Formatting" (PDF) characters? No. Store the plain text of the data value in a language-independent way. Store U+202B and U+202C only if they are within a data value which mixes characters of different directionality.

Why are these characters present in your input data? It seems like your input data is extracted from a PDF file, so I'll bet the PDF file was authored only to display correctly in a PDF viewer. If you find the extracted text usable for your re-purposing, you are fortunate. Expect to have to clean up text extracted from PDF files.

Note that the extracted text mixes Latin script text, which has left-to-right directionality, with Arabic script text, which has right-to-left directionality. Whatever software authored that text seems to have found the U+202B and U+202C characters useful for its purposes. It doesn't follow that the characters will be useful for your purposes. Just like you filter out the U+0009 TAB characters, filter out the directionality format characters wisely.

Check the Arabic text, to confirm that it is stored in the file in reading order. Since the text is extracted from a PDF file, it might be stored in display order. That is, the Arabic characters might be in reverse order in your extract.

Now, some answers to important questions you didn't ask.

Are these characters required for display and handling of all Arabic text? No. But they are often necessary.

A great place to start understanding this is UAX #9 Unicode Bidirectional Algorithm. Arabic orthography combines Arabic letters which are displayed right-to-left with digits and Latin letters which are displayed left-to-right, so Arabic text is referred to as bidirectional.

The Unicode Standard defines properties for characters. Among these properties is bidirectional type. A character can have left-to-right or right-to-left type, or weak type, or neutral type. Characters are stored in reading order, regardless of their bidirectional type. The Unicode Bidirectional Algorithm specifies how to display characters, starting from reading order, in the correct combination of directions. The bidirectional formatting characters, like U+202B and U+202C, help the algorithm get the right results when bidirectional types by themselves aren't enough.

Why did pasting the line from your input into the StackExchange web form give different results than did your "ASCII-spewing terminal"? Because the browser which renders the StackExchange web form applies the Unicode Bidirectional Algorithm. I don't read Arabic, so I'm not sure, but I suspect that the "<202c>" after the Arabic text is being split into a "<","2","0","2" part displayed right-to-left, and then a "c", ">" part displayed left-to-right. And the "<" is displayed right-to-left as ">". Meanwhile, your terminal is likely displaying each character independently, in storage order, without applying the Unicode Bidirectional Algorithm.

If you want place names in local languages, are you stuck scraping text from UN-authored PDF files? No. Two other sources for localised place names are Wikidata and the Common Locale Data Repository. Wikidata has a page on Zambia, with localised names of the country in various languages. There is a query for extracting this data in machine-readable form. The CLDR easily shows names of many countries in Arabic. By querying this data for every language, you can refactor it to have the name of Zambia, or any country, in every available language.

If you just figure out how to store this Arabic text in your database field, will you likely produce good results? I fear not.

I suspect your database and the web page or application which presents data from the database has handled only left-to-right text up till now. By adding Arabic text, you are confronting not just the database, but the application, with bidirectional text. You have have new issues related to bidirectionality which your database and app needs to resolve. How will you determine the directionality context when you display a data element? What piece of software in your system will perform the Unicode Bidirectional Algorithm? How will you select the proper font for a data item, based on which language and characters it uses? Are you going to add the capability for your app or web page to display with a right-to-left layout?

I hope that just stripping off the direction format characters and throwing the Arabic text in your database gives you adequate results. Be prepared that it might not — that including this text requires you to face questions about bidirectionality which you have not needed to worry about until now.

  • The way I read this, you're in agreement with me. To answer the questions about implementation I'm building a product that with a translation table so I intend to send directionality back to the client on the basis of the language marker in the translation table. – Evan Carroll Dec 31 '18 at 13:30
3

There are apparently three approaches for doing this and there is no right answer,

  1. First-strong. Turns out Unicode has strong, weak, and neutral characters. Arabic characters are "Strong". In this scheme you look for the first Strong character and you assume that's the default-directionality. No special formatting characters are stored in the database.
    • Augmenting First-strong with Markers. In this scheme you store things like RLM and LRM which override a base-formatting with a high degree of locality.
  2. Paired formatting. Indicates concrete base-direction in the string itself using control-points. You may see this go by "Explicit Directional Embedding", "Explicit Directional Overrides". You essentially push a new directionality while reflowing the text, and then you pop back to the original directionality. So you can have something like THIS IS ENGLISH <tempchange>this is something a non-ASCII spewing person would write</tempchange> MORE ENGLISH. In the above when MORE ENGLISH is reached it will return to the directional of THIS IS ENGLISH. This includes the PDF above.
  3. Meta-data. Store the meta-data of the language and let a smart client (templete or later consumer) render the string with that meta data. This is how a translation table sounds like it should work.

If none of that is convincing consider this direct quote from the Unicode spec,

The best practice for internationalization is to store and communicate language-neutral data, and format that data for the client. This formatting can take place on any of a number of the components in a system; a server might format data based on the user's locale, or it could be that a client machine does the formatting. The same goes for parsing data, and locale-sensitive analysis of data.

And again,

Localize that data as "close" to the end-user as possible.

From my perspective the way I read this the general rule is that if your cell is not-bidirectional, that is to the cell is either fully LTR, or RTL and that can be inferred from the language of the characters (they're Strong like in English and Arabic), then let the client do its thing and strip these characters which specify directionality before storing them. If you need to render text outside of the base-language of the document you'll have to add them back to prevent trailing neutral characters from jumping (as mentioned in Jim's answer).

For a thought experiment, there is a LRM-mark. So technically you can store an English-only phrase and explicitly declare directionality; but, if you saw that in a database you'd strip those control characters too. Because the directionality is implicit with the Latin alphabet.

See Also

  • The article by Cal Henderson is quite good. But I don't see how he is recommending what you are advocating. And I believe you quoted the wrong Unicode spec regarding the "store and communicate language-neutral data, and format that data for the client" quote. I have updated my answer to include what should be the more relevant quotes / recommendations. – Solomon Rutzky Dec 31 '18 at 9:40
  • 1
    This answer would be even better if it omitted the link to old and non-authoritative documents like "Davis-Bidi-draft5-1990.pdf", and included links to current and authoritative documents like "UAX #9 Unicode Bidirectional Algorithm". And I think it misses a chance by saying there is "no right answer". There are right answers but they require making judgments in a broader scope than just what the contents of a database field should be. One has to consider what the application display stack does, and what the requirements are. – Jim DeLaHunt Dec 31 '18 at 20:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.