Locales and Localisation
- Michael Luecke
- 1 Overview
- 2 Locales
- 3 Localisation
- 3.1 Localisation in Angular
- 3.2 Localisation in Server Scripts
- 3.3 Generating Localised Messages using a Translation Service
- 3.3.1 Configuration
- 3.3.2 Generation
- 3.3.2.1 Translate a message
- 3.3.2.2 List Missing Translations
- 3.3.2.3 Generate Missing Translations
- 3.3.2.4 Generate Localised Message
- 3.3.2.5 Replace Localised Message
- 3.3.2.6 Patch Localised Message
- 3.3.2.7 Copy Localised Message
- 3.3.2.8 Delete Translation
- 4 Dynamic Translation
Overview
Servicely supports multiple locales and localisation of messages into multiple languages, dynamic translation of fields and the ability to install language packs (a kind of specialised application) that contain LocalizedMessage records for a specific language. All of this is configurable using application properties.
Firstly, note that Servicely refers to locales using language_regions. It also refers to languages using the locale name of the primary region rather than the language name itself, so for example the English language is referred to as en_US, rather than as en, in script environments and platform code.
It is important to note at the outset that a locale is not the same as a language. A locale is rather a set of standards for a region for items such as presentation of dates, times and numbers, names of days of the week, whether the week starts on Sunday or Monday, whether time is presented in the 12 or 24 hour clock and so on. A language (and by extension a language pack) is a set of localisations of the messages of the application into a target language. Some locales share the same language for convenience and only differ in the other standards. For example United States English (en_US) and Australian English (en_AU) are different locales that share the same language (also known as en_US) so there is only one localisation for each message. One can still provide specific localisations for either locale, for example the message key "trash.bin.name” could be localised as Trash in en_US and Rubbish Bin in en_AU but the base product doesn’t include them by default.
The default locale for Servicely is en_US and the default language is en_US. LocalizedMessage records for en_US are stored directly in the applications themselves. Other languages should have their LocalizedMessage records in a language pack (a kind of application) which is not installed by default.
Locales
Servicely now supports the Unicode CLDR standard https://cldr.unicode.org/ for all locale information and use of the LocaleSettings table has been discontinued. A new LocaleCustomization table has been added to allow custom definitions for date and time formats in CLDR format if the default ones are not suitable.
Supported Locales
With the introduction of Release 1.8 Servicely support a number of locales in the base product. A number of them share the same language as detailed in the following table:
Locale | Name | Default Language |
|---|---|---|
af_ZA | Afrikaans | af_ZA |
de_DE | German | de_DE |
en_AU | English (Australia) | en_US |
en_GB | English (United Kingdom) | en_US |
en_US | English (United States) | en_US |
en_ZA | English (South Africa) | en_US |
es_CL | Spanish (Chile) | es_ES |
es_ES | Spanish (Spain) | es_ES |
fr_FR | French (France) | fr_FR |
hy_AM | Armenian (Armenia) | hy_AM |
id_ID | Indonesian (Indonesia) | id_ID |
lo_LA | Laotian (Laos) | lo_LA |
nl_NL | Dutch (Netherlands) | nl_NL |
pt_BR | Portuguese (Brazil) | pt_BR |
pt_PT | Portuguese (Portugal) | pt_PT |
ru_RU | Russian (Russia) | ru_RU |
th_TH | Thai (Thailand) | th_TH |
tr_TR | Turkish (Türkiye) | tr_TR |
uk_UK | Ukrainian (Ukraine) | uk_UK |
zh_CN | Chinese (China) | zh_CN |
CLDR information for each locale is included in the UI, along with localisations for many third party UI components. New locales can only be added as part of a new release of the base platform.
An installation of Servicely can determine whether that want to support multilingual capabilities and for which languages and locales by setting application properties appropriately. A list of properties and their meaning is in the following table:
Property | Default value | Meaning |
|---|---|---|
system.multilingual | false | Whether to enable multiple language support in the UI including the language switcher and localised message fields certain tables |
system.supported.locales | en_US, en_AU, en_GB, en_ZA, fr_FR, de_DE, nl_NL, es_ES, es_CL, pt_BR, pt_PT, af_ZA, id_ID, ru_RU, hy_AM, uk_UA, th_TH, lo_LA, tr_TR, zh_CN | Locales to display in the language switcher and the PreferredLanguage field on a user record |
system.supported.languages | en_US, fr_FR, de_DE, nl_NL, es_ES, pt_BR, pt_PT, af_ZA, id_ID, ru_RU, hy_AM, uk_UA, th_TH, lo_LA, tr_TR, zh_CN | Languages that are displayed in UI tools for localised message generation. |
Note that a multilingual installation is still possible even if system.multilingual is set to false. This is because a role of language_editor is available which enables it for users with that role. They can generate localisations, edit localised fields and switch languages but for users without that role they will see a monolingual installation but with the localisations determined by their PreferredLangage setting (which will be read only).
A installation should determine what locales and languages it wants to support by editing the application properties. The default locale and language en_US is supported even if the properties are empty or missing.
Locale Switching
If system.multilingual is true then a locale switcher component is available in the menu bar. It displays the list of locales that are configured in the installation, in the language of the current locale. By choosing a locale off the list the current users PreferredLanguage is changed to the selection and the page reloaded. A flag is displayed in the menu bar to indicate the current locale. In this sequence of images the locale is changed from United States English to French:
Localisation fields
Some tables were developed before their localisation needs were apparent. These tables have been enhanced with additional localised message fields as a complement to the existing monolingual fields, and all display templates have been modified to use the localised message fields (if they have a value) before using the monolingual fields. The localised message fields are hidden if system.multilingual is false so as to not cause confusion. The localised message field names are based on the monolingual field names with the word Localized as a prefix, so the complementary field for Description is LocalizedDescription. On a form they have the word Display as a prefix in the title and immediately follow the monolingual field as in this example of a CatalogItem:
The localised message fields visible are Display name, Display summary and Display button text.
The tables affected by this are:
Table | Localised Message Fields | Child Tables |
|---|---|---|
TemplateReport | LocalizedName LocalizedDescription |
|
QuestionItem | LocalizedName LocalizedDescription LocalizedSummary LocalizedButtonText | CatalogItem RiskAssessmentItem |
Report | LocalizedName LocalizedDescription LocalizedTitle |
|
Dashboard | LocalizedName LocalizedDescription | ReportDashboard PortalPage |
Question | LocalizedSectionTitle |
|
CatalogCategory | LocalizedName LocalizedSummary |
|
Locale Customisations
Prior to Release 1.8 Servicely used the LocaleSettings table to store the definitions of locale properties, such as date and time formats and some number formats. With the introduction of the CLDR this table is no longer necessary and is no longer used. Some existing installations did use it though for customising the date and time formats from the Servicely defaults. To support those customers a new table LocaleCustomization has been introduced that only contains date and time customisation fields in CLDR format.
The CLDR has the concept of skeleton and custom formats. A skeleton format is a name like short or long that is an abstraction that can be applied in any locale and reduces the need to explicitly specify in detail what the format contains. A custom format contains the full detail including placement of the various elements of a date or time and other indictors such as AM/PM.
The Servicely default formats are
Type | Format | Is Skeleton | Notes |
|---|---|---|---|
Short Date | short | Y |
|
Long Date | yMMMEd | Y |
|
Short Time | short | Y |
|
Long Time | medium | Y |
|
Short Date Time | short | Y | Commonly replaced by Short Date + “ “ + Short Time |
Long Date Time | medium | Y | Commonly replaced by Long Date + “ “ + Long Time |
The LocaleCustomization table allows the override of any of these values for a locale and allows either skeleton or custom formats:
If you want to use a 24 hour format, you can update the short and long time format to be HH:mm:ss , which will be 23:50:30 for example, and if you add an a, such as HH:mm:ss a, it will also include PM/AM.
Localisation Packs
Localisation packs for all supported languages are available as special applications listed in the Application languages section. They only contain LocalizedMessage records. If they are installed using the standard procedure then those messages are available when a locale using that language is selected using the locale switcher.
Currently only the French (translations-fr_fr) and German (translations-de_de) localisation packs are populated with messages by Servicely for out of the box use. We will be continually improving the translations in the packs to make them more accurate. The translations language pack is empty but deployed and is used when experimenting with localisation generation in order to not modify the default ones. There is no English localisation pack because those localisations are directly included in the default installation.
If a customer is generating localisations for a specific language in bulk then it is recommended that the appropriate language pack be installed and the messages generated into there. This allows management of the localisations to be easier.
Installation of a localisation pack is a short procedure and usually takes less than a minute. They take effect immediately.
Localisation
Servicely provides a number of mechanisms for localisation entry points, i.e. places where a localised message can be substituted for a base message, or for a localisation message key. Some of the mechanisms are based on the internationalisation tools that are part of the Angular web framework - because the majority of the Servicely UI is Angular, including Catalog, Portal and Modal components.
Localisation in Angular
Angular provides the means to localise messages in the HTML template and the TypeScript code of a component. Servicely uses the HTML template approach as-is, but provides a variation on the standard TypeScript approach that provides more flexibility and allows for development of components without needing to initially create LocalizedMessage records. The official documentation can be found at https://angular.io/guide/i18n-overview
Localisation in an Angular Template
An Angular template can indicate localisation points by using custom HTML tags. The i18n tag indicates that the element contents will be replaced with the localised message, while the i18n-XXX tag indicates that the XXX attribute will be replaced with the localised message. This is useful for replacing text such as the href of an anchor tag. Angular message keys have a marker prefix of “@@” but those characters are not used in the LocalizedMessage table, for example the message key "@@ng.AAA.BBB" is what would be used in the template while "ng.AAA.BBB" would be the message key for the record.
Here is an example of markup from a template:
<a i18n-href="@@ng.0a00000a72811a1181728242dbb22f81.PortalComponent.Template.demo.href"
href="http://default.place.com">
<span i18n="@@ng.0a00000a72811a1181728242dbb22f81.PortalComponent.Template.demo.label">
The default place is {{place}}
</span>
</a>The body of the span and the href of the anchor will both be replaced by a localised message if one exists. Note that it is necessary to supply a default value for anything that is to be localised otherwise the substitution will not be made.
The default value is also used to derive the structure of the message format itself in the LocalizedMessage table. Note that the default message is The default place is {{place}} which has an embedded Angular expression {{place}} This indicates that a variable is to be substituted in that part of the string - usually a member variable of the component itself. Because it has a substitution point the messages in the LocalizedMessage table should also have a substitution point, and they will look like The default place is {0}, or Le lieu par défaut est {0} This allows messages to be built up using the natural order of the localisation, for example one could imagine a language where the natural order would be XXX {0} YYY ZZZ, and the substitution of ‘place’ will still be at the {0} position. Messages can have an arbitrary number of substitution points indicated by {0}, {1} etc. Message formats in general also support named substitution points and number and plural substitution points but Servicely does not at the moment.
The message key is designed to allow support for automated tooling to extract messages, which will be developed in a future release. The format of the key is as follows, where name-suffix can be anything you like but without spaces:
ng.ID.TableName.FieldName.name-suffixThese can be generated easily inside a Servicely code editor using the context menu or the key sequence Central-F12 (or Cmd-F12 on macOS):
If there is no localisation for the current locale (i.e. the users PreferredLanguage) then the default message value is used instead. This allows components to be built without having to add the en_US message immediately.
Localisation in an Angular Component
In component code we can use the $translate function to perform localisation on template literals. A template literal is a string wrapped in backticks (`) that can have substitution points in a similar fashion to a default message in an Angular template. $translate is a function that operates on template literals and knows how to localise them, or how to construct the default message if there is no localisation available. An example of such a function call would be:
let msg = $translate('ng.0a00000a719518168171966aa7a50a19.PortalComponent.Template.approve-error')`Error approving: ${response.responseText}`;$translate takes a parameter which is the message key and returns a function that can operate on the string `Error approving: ${response.responseText}` which is a template literal with one substitution point. An equivalent LocalizedMessage would be Erreur d'approbation : {0} The return value of the $translate function call will be the localised message. The default message is constructed from the parts of the template literal and the substituted strings if it is necessary to use that instead.
It is also possible to just supply a key in which case the call would be:
let msg = $translateKey('ng.0a00000a719518168171966aa7a50a19.PortalComponent.Template.approve-error');Localisation in Server Scripts
Server scripts (and Script Libraries) can use $translate and $translateKey in a similar fashion to an Angular component.
It is also possible to use an older form like this:
let msg = i18n.message("message.key");Generating Localised Messages using a Translation Service
It is possible to generate localised messages using a cloud translation service if the appropriate system properties are set and an access key is supplied as a Bearer token. Servicely supports two translation services for release 1.8: Google Translate or OpenAI.
Configuration
The relevant application properties are
Property | Default value | Meaning |
|---|---|---|
system.translation.service | Whether to use Google Translate ('google') or OpenAI ('openai') for generated translations | |
ai.openai.token.name | OpenAI | Name of the Bearer Token used to access OpenAI |
ai.openai.organization.id |
| Id of the organisation that supplies the Bearer Token |
ai.openai.model.id | gpt-4 | OpenAI model used to generate translations. Release 1.8 uses gpt-4 |
system.translation.openai.prompt | Varies depending on customer… | Prompt sent to OpenAI to fine-tune translations for the Servicely application. |
There should be at least one SystemAPIOutboundToken record containing the Bearer token for the translation service that is being used.
Name | Type | Meaning |
|---|---|---|
google.api.translation.key | Bearer | Bearer token for Google Translate |
OpenAI | Bearer | Bearer token for OpenAI |
Generation
Localised messages can be generated using a Server Script. They are currently done manually on the standard Server Script page, e.g. https://<instance>.servicely.ai/#/View/ServerScript
The common approach is to generate missing localised messages for a target locale relative to a source locale, usually en_US. The messages are generated into the localisation packs mentioned above so that they are all contained in one place for ease of management. In this way the entire application can be localised in a short period of time. Note that if more components are developed then their messages can be generated in the same way (i.e. only doing the missing ones) without affecting already generated ones.
All the commands are part of the Translation server script library.
Translate a message
Note that this does not create a LocalizedMessage record, it is for testing the integration with a translation service. If a fromLocale is not supplied the translation service will assume en_US.
Translation.translate(toLocale: String, text: String): String?
Translation.translate(fromLocale: String, toLocale: String, text: String): String?
Translation.translate(fromLocale: String, toLocale: String, text: List<String>): List<String?>Translation.translate("fr_FR", "I am an elk");
// will return
"je suis un élan"List Missing Translations
This will list all the messages that are in fromLocale but not in toLocale for the application identified by applicationNameOrId. If fromLocale is not supplied then en_US is assumed. If applicationNameOrId is "*" then all messages for all applications are listed.
Translation.listMissingTranslations(toLocale: String, applicationNameOrId: String): List<LocalizedMessageRec>
Translation.listMissingTranslations(fromLocale: String, toLocale: String, applicationNameOrId: String): List<LocalizedMessageRec>
let missing = Translation.listMissingTranslations("fr_FR", "*");
// will return about 7500 records for a locale that has not had any messages translatedGenerate Missing Translations
This will generate all the missing messages using the translation service. Generally takes about two minutes to do a complete set of 7500 messages. If fromLocale is not supplied then en_US is assumed. If fromApplicationNameOrId is "*" then all messages for all applications are checked. The target application is toApplicationNameOrId and should be one of the localisation packs.
Translation.generateMissingTranslations(toLocale: String, fromApplicationNameOrId: String, toApplicationNameOrId: String): List<LocalizedMessageRec?>
Translation.generateMissingTranslations(fromLocale: String, toLocale: String, fromApplicationNameOrId: String, toApplicationNameOrId: String): List<LocalizedMessageRec?>
Translation.generateMissingTranslations("th_TH", "*", "translations-th_th");
// will generate about 7500 records into the Thai localisation pack.Generate Localised Message
Generate localised messages for the supplied keys. If fromLocale is not supplied then en_US is assumed.
Translation.generateLocalizedMessage(toLocale: String, fromApplicationNameOrId: String, toApplicationNameOrId: String, key: String): LocalizedMessageRec?
Translation.generateLocalizedMessage(fromLocale: String, toLocale: String, fromApplicationNameOrId: String, toApplicationNameOrId: String, key: String): LocalizedMessageRec?
Translation.generateLocalizedMessage(fromLocale: String, toLocale: String, fromApplicationNameOrId: String, toApplicationNameOrId: String, key: List<String>): List<LocalizedMessageRec?>Translation.generateLocalizedMessage("nl_NL", "platform", "translations-nl_nl", "common.button.create");
// Makes a single Dutch localised message in the correct localisation packReplace Localised Message
Replaces an existing message with a better translation.
Translation.replaceLocalizedMessage(locale: String, applicationNameOrId: String, text: String, replacement: String): List<LocalizedMessageRec>Translation.replaceLocalizedMessage("fr_FR", "translations-fr_fr", "NON", "Non");
// Sometimes Google Translate make words all uppercase.Patch Localised Message
Replace a substring of a message with a better substring.
Translation.patchLocalizedMessage(locale: String, applicationNameOrId: String, text: String, replacement: String): List<LocalizedMessageRec> Translation.patchLocalizedMessage("fr_FR", "translations-fr_fr", "Jacques", "Pierre");
// Renames all the peopleCopy Localised Message
Copy messages from one locale to another. This is used when the translations have be the same in both locales and you don’t want them picked up by the missing translation detector. If fromLocale is not supplied then en_US is assumed.
Translation.copyLocalizedMessage(fromLocale: String, toLocale: String, fromApplicationNameOrId: String, toApplicationNameOrId: String, key: String): LocalizedMessageRec?
Translation.copyLocalizedMessage(locale: String, fromApplicationNameOrId: String, toApplicationNameOrId: String, key: String): LocalizedMessageRec?
Translation.copyLocalizedMessage(fromLocale: String, toLocale: String, fromApplicationNameOrId: String, toApplicationNameOrId: String, key: List<String>): List<LocalizedMessageRec?>
Translation.copyLocalizedMessage(toLocale: String, fromApplicationNameOrId: String, toApplicationNameOrId: String, key: List<String>): List<LocalizedMessageRec?> Translation.copyLocalizedMessage("es_ES", "itsm", "translations-es_es", "table.incident.name");
// We want the Incident table in Spanish to also be called Incident.Delete Translation
Deletion one or more translations.
Translation.deleteTranslation(locale: String, applicationNameOrId: String, key: String): List<String?>
Translation.deleteTranslation(locale: String, applicationNameOrId: String, key: List<String>): List<String?>
Translation.deleteTranslation(locale: String, applicationNameOrId: String): List<String?>
Translation.deleteTranslation(locale: String): List<String?>Translation.deleteTranslation("th_TH");
// Delete all Thai localised messagesDynamic Translation
If you have a translation service set up with an access key it is possible to mark specific fields as supporting dynamic translation.
An application property needs to be enabled first:
Property | Default value | Meaning |
|---|---|---|
system.dynamic.translation | false | Whether to enable dynamic translation |
By setting the field level renderer property “dynamic.translation" to true a translate button will appear against that field. When clicked it will place the translation alongside or below the field itself. Translations are also possible for standard journal entries (but not for audit history entries).
Note that The renderer property is at the field level not the view definition level and is accessed from the context menu on the field:
The translate button is a small multilingual icon:
When clicked a translation into the current locale is performed using the cloud translation service
The same applies for journal entries:
Refreshing the form will clear the translations being displayed.
Servicely Documentation