Translation on Guardian Project

Easy translation workflows and the risks of translating in the cloud

Mon, 08 Jun 2020 00:00:00 +0000

Crowdsourced translation has opened up software and websites to whole new languages, regions, and uses. Making translating easier has brought in more contributors, and deploying those languages requires less work. A number of providers now offer “live”, integrated translation, speeding up the process of delivering translated websites. On the surface, this looks like a big win. Unfortunately, the way such services have been implemented opens up a big can of worms. Third parties must be trusted with user data. The translators cannot work without being tracked. Displaying the translation requires JavaScript. The security profile is more complicated and harder to defend.

For projects with elevated security and privacy requirements, these are deal breakers. Translators can get in trouble for working on publishing software or translating censorship circumvention documentation. Minority languages are suppressed in many places around the world, even publicly working in some languages can get people into trouble.

We have been working for many years now to help make software and website translation easier, more accessible, and more efficient. We also balance that with privacy and security concerns. This post outlines how to balance all those concerns.

Live, integrated translation systems

Using this new class of translation services means that the translations are not shipped with the website or app, but instead dynamically downloaded and delivered. These translation services require that third party code is integrated into the website or app to deliver the translations. All of the regular privacy and security concerns of dynamic web services apply here.

For example, Transifex Live is based around JavaScript, it uses JavaScript to dynamically load the translated strings from the Transifex CDN hosted on Amazon AWS. They also provide an iFrame option, which has its own security concerns. Crowdin provides “in-context localization“ which lets translators see the translations live in the website as they are working. This is a huge benefit to the translators, but it has all the same issues as Transifex Live. Their code must run in your site. They use other providers like Amazon to provide their own service. All of these third-parties have to be trusted to provide security and privacy. On top of that, Crowdin and Transifex are monolithic, proprietary offerings, it is not possible run your own instance. Like many cloud-based services, it is all-or-nothing: accept all the tracking, the privacy concerns, and security issues, or do not use such a service

Lastly, the data from the translation contributors must be considered. These live services provide the translators a direct channel to feed data into the website. A malicious translator could feed an exploit to the website using this channel. Such a setup relies entirely on any automated checks that the translation platform provides. These checks are optional, and often disabled by default. Also, attackers regularly find ways around even the best checkers and sanitizers, like Mozilla Bleach or Ruby loofah.

For all these reasons, projects like SecureDrop and Tails self-host Weblate to provide crowdsourced translation..

Static sites with live previews

Static sites built with tools like Jekyll and Hugo offer big benefits in terms of privacy, security, and cost of operation. But they generally require more technical skills to operate, and have restricted possibilities in terms of dynamic interaction. There is a lot that still can be done, and things are improving fast. The dream of live localization and in-context translation workflows without privacy and security concerns is within reach.

Live display of translations is not possible with a fully static site. Live translation requires front end JavaScript or a dynamic backend server. The vast majority of web tracking and security vulnerabilities come from JavaScript, sites that serve high risk communities should use static sites that work without JavaScript enabled.

Translation updates can be highly automated with a static site. This means new translations can be reviewed and deployed within minutes.

Jekyll and Hugo can also provide live previews while editing the source pages and translations. Unfortunately, using these features requires base level familiarity with technical things like working in the terminal. When Jekyll or Hugo is installed locally on the translators computer, jekyll serve and hugo serve generate the whole website on the fly, and the browser will automatically refresh the page with each change.

Wordpress, Translation, and Static Sites

Wordpress remains a popular option for running websites, especially for small and non-technical organizations. It provides intuitive editing and publishing tools combined with a wide array of attractive templates to build on. It is free software that can be self-hosted, and it can even be used as a static site generator. Even with the rise of Jekyll, Hugo, and so many other static site generators, Wordpress remains a good option for small organizations with privacy and security concerns, given that it is used with the static HTML output plugin. The one missing piece is crowdsourced translation that fits in with all that.

Poedit provides an alternate approach that is self-hosted and free software, but is not entirely a typical crowdsourced translation workflow. It is an editor app that runs locally on the translator’s own machine. It supports translating Wordpress directly via its API. Then the results are included when Wordpress generates the static HTML output.

Using self-hosted Weblate, the full website and translation workflow can be as private as needed. The static HTML output can be fed directly to Weblate or use po4a to set up an automated workflow that is tailored to your needs.

If self-hosting the translation platform is not a requirement, then Crowdin and Transifex are options for translating the static HTML that comes from Wordpress. It is important to consider that both of these will send data to many different companies, so they cannot be considered private. Using Crowdin sends data to Amazon, Google, and Sentry. Using Transifex sends data to Amazon, ChurnZero, Google, jsDelivr, New Relic, Sentry, Stripe, Adobe (Typekit), and VWO. Both can potentially also send data to Facebook, GitHub, GitLab, LinkedIn, and Twitter since those can be used for signing in.

Two good patterns for setting up the languages are the hosting each language on a subdomain like how wikipedia does it; or, use path segments for each language. With GitHub Pages and GitLab Pages, each language can be a project, then each language will be deployed to a sub-directory, e.g.:

https://mysite.gitlab.io/en comes from https://gitlab.com/mysite/en
the main site is the language chooser, e.g. https://mysite.gitlab.io comes from https://gitlab.com/mysite/mysite.gitlab.io

One idea further improve the Wordpress workflow is to combine the Transifex Wordpress Plugin with the [Wordpress Static HTML Output Plugin](https://wordpress.org/plugins/static-html-output-plugin/ to customize and streamline the whole process. This could work with Crowdin, Transifex, and Weblate, since they all provide APIs to integrate with.

Figuring Out Crowdsourced Translation of Websites

Thu, 23 Apr 2020 00:00:00 +0000

Crowdsourced translation platforms like Weblate, Transifex, Crowdin, etc. have proven to be a hugely productive way to actively translate apps and desktop software. Long form texts like documentation and websites remain much more work to translate and keep translated. Many translation services currently support Markdown and HTML, but very basically, which means much more work for translators and webmasters. Translators can inadvertently break things, either with a typo or because of a lack of knowledge of a specific syntax. This can make the whole page layout break. Webmasters and documentation maintainers must carefully check the process to ensure everything is working smoothly. With the spread of Markdown as a standard format, there is now hope! Software developers can focus efforts on the Markdown translation workflow, and Markdown is more tolerant of syntax errors than HTML.

Guardian Project has been working for a decade on making localization and the long term maintenance of translation a lot easier for developers and translators. We call this ongoing collaborative effort “Linguine” after a name chosen in a multi-project meeting. One of the biggest usability barriers in the world of software and the internet is language. Most of the world does not speak English, yet most of the internet is written in English.

The App Workflow

The developer of an Android or iOS app does not need to know much about localization or translation in order to make their apps fully translated. Both supported translation from the beginning as a native feature. Software strings are already broken up into small chunks, which fit nicely into crowdsourced translation software. The progress bars and step-by-step user experiences map easily to these strings. The string file formats also allow for all sorts of valuable metadata the give translators context and guidance, as well as to enable automated checks. The strings of an app also do not to be 100% translated to be useful. For example, if all strings are translated except for those in a new feature, most users will never even notice the missing translations. If only the most commonly seen strings are translated, that translation can still be useful since it will provide the user with enough to navigate all the features of the app. Here are three examples of app strings for translation:

Android strings.xml:

<string name="app_details__no_versions__show_incompat_versions">To show incompatible versions here anyway, enable the \"%1$s\" setting.</string>
<string name="app_details__no_versions__no_compatible_signatures">No versions with compatible signature</string>
<string name="app_details__no_versions__none_compatible_with_device">No versions compatible with device</string>

Python gettext .po:

#: ../fdroidserver/build.py
msgid "Don't create a source tarball, useful when testing a build"
msgstr "不创建源码 tarball 文件，便于内部版本测试"

#: ../fdroidserver/stats.py
msgid "Don't do anything logs-related"
msgstr "请勿做任何日志相关的操作"

#: ../fdroidserver/build.py
msgid "Don't refresh the repository, useful when testing a build with no internet connection"
msgstr "不刷新资源库，便于没有互联网时的内部版本测试"

i18next JSON:

{
    "package": {
        "anti_features": "Anti-Features",
        "added_on": "Added on {{ package_added_date }}",
        "signed_by_fdroid": "It is built and signed by F-Droid, and guaranteed to correspond to <a href=\"{{ source_code_url }}\">this source tarball</a>.",
        "suggest_downloading_via_fdroid": "Although APK downloads are available below to give you the choice, you should be aware that by installing that way you will not receive update notifications and it's a less secure way to download. We recommend that you install the F-Droid client and use that.",
    }
}

Long Form Text

With documentation, blog posts, and other long form text, it is a very different story. To start with, there are lots of debates of how best to break up a text: paragraphs, sentences, or whole pages? Does each bullet point count as a paragraph or a sentence? Do these rules work for both blog posts, which rarely change after publication, as well as documentation, which must be kept updated? Also, measuring the translation “completeness” of a long text is harder. If a blog post is 80% translated, it will be very disconcerting for readers when sections switch to English and back. A documentation page that is 20% translated is not useful.

With websites, there is also more varied context than with software strings: paragraphs, titles, bullet points, code blocks, YAML Front Matter, etc. There is currently no way to add translation metadata directly to the Markdown or HTML document, so that context is often missing, or it must be manually entered in the translation platform. One really nice property of Markdown is that the plain text version looks good, so if the writer messes up the markup syntax, all is not lost. Translation software can be made to fully “understand” Markdown (e.g. the AST), and provide automated syntax checks. Such an approach might totally fail if someone messes up the markup in the source document. In that case, a more naive direct representation might work better. Sites that run linters will want the AST approach, sites with non-technical writers might not. In short: Crowdin, Transifex, and Weblate currently support Markdown, but in a limited way. And these limitations mean much more work for translators, developers, and webmasters.

If the translators are technically savvy and know the topic and syntaxes well, then the webmaster doesn’t need to worry too much. The translation platform will make it easier for the translators to keep the translations maintained. Another key benefit of crowdsourced translation is that translators don’t need as much technical skill to meaningfully contribute, as long as the project fits well into the translation workflow. Due to the current complexity of websites and limitations of the tooling, there is still no single translation workflow for long form text that works well for the majority of websites and documentation. That means it is important to know available workflows and understand which one fits best to your project.

User Experience Breakdown

Website localization is also not nearly as standardized as apps. With apps, a single translation workflow will work for basically all cases. For websites, that is not possible since there are so many different kinds of texts and requirements. For some documentation, it is acceptable to have 20% of the text slightly out of date in the translation. For other documentation, each page needs to be 100% complete, otherwise the original English should be shown. A good translation experience therefore requires customization.

The workflow will also vary based on the content and the skills that the translators have. If it is deeply technical documentation, then the translators will be comfortable markup and code syntax since they must have technical knowledge in order to translate the materials. The translators will prefer being closer to the source format. If the content is blog posts from an organization advocating good governance, then the translators are mostly going to be non-technical, and the workflow should minimize exposure to technical details like YAML Front Matter blocks.

One large, complicated example is the F-Droid website, which is built with Jekyll. It has five different kinds of materials that need to be translated, each with its own distinct requirements: chrome strings, pages, docs, blog posts, and tutorials. In order to group the strings by these requirements, the site uses i18next JSON for use in Jekyll Liquid tags and custom po4a sections to handle the Markdown and YAML Front Matter. po4a makes it possible to group the translation levels per-section, then add requirements: the entire “pages” section must be ~95% translated for inclusion, while “docs” and “blog posts” sections can be untranslated. Each individual blog post must be completely translated to be published. Each documentation page must be 90% translated to be published. It is not 100% to allow for some time lag between when the original is updated, and the translated is updated. Ideally, the source string edits could be marked as “typo correction” vs “update” to aid making better automatic decisions about which translations should still be published.

Process and People

If there are people fully engaged in all parts of the translation process, then it is possible to tightly coordinate the updates to the source and the translations. Other projects rely on crowdsourced translations because it is easier for them to acquire volunteer contributions than increasing translation budgets. So the advantages and disadvantages of various workflows have to be weighed:

Delay publishing the source update until the translations are done.
Translated pages are reverted to English unless they are 100% updated.
Translated pages show outdated parts until they are updated.
Translated pages show English parts until they are updated.

There are other trade-offs to consider in terms of the relationship between the translator and the project:

Only trusted translators who can directly update the translations:
- Rapid time to publish
- Less work to translate
- Much fewer languages supported
Trusted reviewers with any contributions from any translator:
- More language coverage, quality control from multiple eyeballs on every string
- Asynchronous, unpredictable workflow
Contributions from any translator:
- Rapid maintenance updates and even more languages
- More errors in translation, including potentially vandalous or malicious contributions
- Skilled translators will be less interested in contributing

When it is possible to achieve 100% translation, like on a small site or when there is a budget to pay for all the documentation to be translated, then the workflow is a lot simpler. That allows the tools to have a more direct user experience. Volunteer-based, “best effort” translation is a different story. Then it is important to clearly represent which strings and pages should be translated first, and which are non-essential. Crowdin, for example, can show all the documents in a familiar looking file browser, with nice progress bars. There is no indication of priority. This is great if someone is paid to complete the full translation, but is much less clear for volunteers to jump in and contribute to the more important parts.

Crowdin project file navigation

Unfortunately, there is no shining example of a user experience that covers this second scenario well. Weblate provides one good tool, where the strings are presented to the translator in order of the priority set in the metadata (e.g. like XLIFF’s priority). The translator just needs to start translating the first strings Weblate presents.

One hack to add priority to Markdown for Crowdin

Mass Changes versus Merge Conflicts

Sometimes you have to make a small change to some of the coding of the website. The website has suddenly gotten a lot more popular, so you need to work on making the caching work a lot better to reduce the load. That means changing all of file references to include a version or hash value. Those links are also in the source strings that are fed to the translation system. And each change to the source strings prompts all of the translators to update their translations. Normally, this is exactly what should happen. But in this scenario, the translation should not change since it is just the links that changed. To handle this, the webmaster or developer needs to be able to do mass edits on the source and translation while having a way to tell the translation system to ignore this change. This can easily cause merge conflicts if a translator is working at the same time.

If these small changes only happen in source, the translations will be marked as needing an update. Some platforms try to be smart about detecting changes, and will even mark strings as changed if both the source and translation were updated together. That produces another scenario: The website fully translated into Tibetan, but now the translator has disappeared. So those trivial changes are bumping more and more strings out of the translation, and soon it will be death by a thousand paper cuts and the Tibetan translation gets kicked out even though the underlying text is current. It is just the inline code updates that are marking things as out of date.

Automation and Checks

One essential part of any translation workflow is automated checks to make sure that the syntax has not been broken in the translation process. Translation platforms must provide clear errors to the translators, otherwise all of these problems will be pushed to the webmaster or developer who imports new translations. Or worse, they are overlooked and cause crashes or break layouts. Good metadata is a key piece of that process. For example, i18next {{ interpolation }} is not the same as Python {format_strings}, though they look quite similar. The metadata should include the file type and any relevant syntaxes. Most developers are not aware of all of the details of the translation process, so these checks should be enabled automatically, as much as possible. Manually enabling checks can be very hard to understand, since you have to understand both the developer’s and translators points of view. These checks really pay off, like when technical contributors can go straight to strings with failing checks and fix the syntax problems, without any knowledge of the languages involved. This is in contrast to a project’s Continuous Integration (CI) tests, which generally only the software’s maintainers follow. The barrier to entry for fixing failing checks is much lower since they are standardized across every project on the translation platform.

Weblate highlights failing checks, the error counts are clickable to edit them all.

Reviewing a single failing check in Weblate.

Context is very important for translators, and a good translating experience will provide as much useful context as possible. That said, it cannot replace the real thing. Having a automatic staging website will help catch errors before publication. It also gives translators the feeling that they are building something real, when they are working their way up to being accepted for final publication. Tails and F-Droid both provide complete staging websites that include all of the the languages, not only the complete ones. Static site generators like Hugo and Jekyll let more technically skilled contributors rapidly build a local copy, and services like GitLab Pages and GitHub Pages can host a staging website for each fork.

Translation Setup with po4a

po4a is a tool for customizing the representation of text in the translation workflow. It is quite powerful, but also not simple to get started. Each translation platform has its own documentation for directly using Markdown, so follow those docs for a simpler setup. There is a lot to be said for having the translation platform directly handle the files that are being translated, without an intermediate localization format like XLIFF, gettext PO, or JSON. Sometimes other requirements mean that using po4a in the middle will save a lot of effort, like the F-Droid example above.

The po4a approach does work quite differently: the translations are stored in the .po files rather than Markdown documents. This feels a bit unnatural at first, as compared to the regular workflow of a static site generator. Translated pages are then converted from .po to .md just for building the site. It gives a lot in return, especially for crowdsourced translation with specific requirements. For example, it makes publishing and unpublishing translated documents happen automatically, based on the pre-configured specifications, like whether it is 100% translated.

po4a gives fine grained control over how documents are grouped, and what percentage must be translated before the translated version is published. The direct version means each document is represented by a single translation component. Many websites are organized into three common buckets, folders of Markdown documents should count as a single translation component:

The base pages like “about”, “contact, etc.
Documentation
Blog posts

Projects using po4a

Debian is using po4a to translate the man pages.
The extensive Tails documentation uses po4a to make Markdown pages translatable in ikiwiki.
The Fedora project is using po4a to translate the documentation in their own Weblate instance
Stellarium Astronomy uses po4a to translate “sky culture descriptions”.

Coming Soon:

https://reproducible-builds.org/ (built with Jekyll): https://salsa.debian.org/reproducible-builds/reproducible-website/-/merge_requests/56
https://reuse.software (built with Hugo): https://github.com/fsfe/reuse-website/issues/26

Improving Crowdsourced Translation of Long Form Text

Thu, 05 Mar 2020 00:00:00 +0000

We are happy to announce the start of work on another step in improving crowdsourced localization, funded by the ISC Project. This is the second part of our ongoing “Linguine” collaboration to move crowdsourced translation to privacy-respecting free software.

Crowdsourced translation has proven enormously successful getting apps and website software translated into many languages. Using tools like Weblate or Transifex, developers can quite easily incorporate translated app strings into their mobile apps and websites. Any kinds of text that is easily broken down into phrases and sentences will fit easily into the crowdsourced workflow. Localization Lab enables a wide range of volunteers to contribute to the most important projects in a wide array of languages.

For long form text, from blogs to documentation, large technical hurdles prevent the same fluid workflow. One clear example: Tor Browser is available in 25 languages, but the documentation is still largely just in English. The standard workflow is to first complete the English version, then submit that to translators, then wait for a complete translation. For the regular, small changes that come with maintaining documentation, that workflow makes the update cycle heavy and slow.

Few webmasters can review translations. Tools like Weblate establish a review process, then developers need only to run an automatic import to get approved translations from trusted translators.

The ideal workflow starts with the traditional method of translating in one complete unit, but now, that work can easily be shared by multiple contributors. Whenever the original documents are updated, those translations can then be maintained via the crowdsourcing. Even for organizations which directly cover their own translation needs, like a blogger who writes in both English and Tibetan, the workflow provided by tools like Weblate makes managing many small changes to documents drastically easier.

While blog posts generally do not need frequent updates, a smooth path to crowdsourced translations means, given the same effort, a larger audience can be reached since the information will be available in more languages.

How we are doing this

There are already many key building blocks in place:

Weblate and Transifex already have basic support for directly translating long form text
po4a provides good format conversion between Markdown and other text formats
Markdown is a widely adopted mark-up format for documentation and blog posts
Weblate already supports validating Markdown
po4a integration with Jekyll
Static Site Generators like Jekyll and Hugo have existing methods of supporting localization

What needs doing is fixing lots of little issues here and there to complete the whole workflow. For example:

Code blocks are quite common in technical documentation, and they often contain texts that should not be translated, e.g. the words from a programming language. po4a can be made to automatically recognize these blocks, and mark them as “do not translate” using standard tags, which Weblate communicates to translators.
Most translators do not have strong technical skills, so specialized syntax like Markdown or configuration examples need to be validated to ensure that the translator did not inadvertently break something with a typo. There are existing methods and tools for validating a wide variety of markup, code, and configuration formats. These will be integrated into the translation review process to ensure that translations will not break the website layout or generation.
When using translation sites that do not have good support for long form text, po4a can be integrated via “plugins” to static site generators like Jekyll or Hugo.
Not all workflows allow for transparent integration, for example, Hugo to Weblate. In these cases, there needs to be some “glue” tools to make it work. We will produce and distribute tools for gluing together some common setups to work smoothly with Weblate and/or Transifex.

Since this project is about getting lots of pieces to integrate nicely, all the code generated will be included in all the relevant projects. That is also where we will be looking for feedback, so file issues with Weblate, f-droid.org, guardianproject.info, etc. and we will track them there!

Good translations are essential to usability

Wed, 09 Dec 2015 17:20:15 -0400

All too often, translation of an app are treated as an afterthought. It is not something that the app developers see, since they create the software in languages that work best for them. So the software looks complete to the developers. But for anyone using the software in a different language, translation is essential in order for the app to be useful. If you can’t understand the words that you see in the app’s interface, it is going to be difficult or impossible to use that app.

The part of this question that is still open is how best to manage translating software. From the point of view of the app’s developer, it is not possible to check all of the translations since no one speaks that many languages fluently. Many of our apps are translated into 30 or more languages, represented by both pictograms and more than 10 alphabets. No one could read all of those writing systems, let alone understand all the words written in them. So inevitably, developers must trust many other people to do accurate translations, and to not slip in false or misleading information.

We receive substantial translations from random internet users who come along and contribute their time into translating our apps. We aim to make that process as easy as possible by posting clean source files to Transifex, the web service we currently use. We also work a lot with the OTF Translation Hub, which organizes the chaos of all those apps and contributors into regular events and completed translations. Organized community contributions work very well for building up baseline translations and keeping polished translations updated when the software changes. It is both a valuable and cost-effective resource, and I think its a model that can be emulated for other collections of software.

Nevertheless, for certain target languages like Tibetan or Belarusian, community contributions are rare. We get lots of contributions for many languages, like French, German, and Spanish, but other languages, like Arabic, Chinese, Persian, and Burmese get very few contributions from volunteers. Also, in order to get completed, polished translations, it is necessary to pay translators so they can devote a solid, concentrated chunk of time on making sure the whole translation works. This person can then spend time thinking about the finer points which can have a large impact on the user experience: the tone and feeling of the language, whether it is funny, serious, cute, matter of fact, etc.

So really, in order to deliver software that is translated into many languages, opening up the source files to get as many contributions as possible will get a lot of work done. But cultivating relationships with translators to oversee the whole process is essential in order to have the translations match the quality of the software.