Bug Reports/Observations with the new (completely rewritten) translation engine December 2019

ICANN IT embarked on a complete rewrite of the translation tool in August 2019 and is ready to deploy the new tool around December 9 2019, unless showstopper bugs are identified.

The new tool is deployed on the testing mailing lists new-transbot-en and new-transbot-es

new-transbot-EN : http://mm.icann.org/pipermail/new-transbot-en/

new-transbot-ES :  http://mm.icann.org/pipermail/new-transbot-es/

Bug Reports/Observations with the new translation engine (updated January 2019)

Below are some of the bug reports/observations noted during testing.

	Description of Issue	Noted by	Date Added	Status	Additional Notes on testing, fixes
1	Missing emails from one list to another In 2018, it was noticed that many emails are being silently dropped from one list, mainly emails from lacralo-es not being sent to lacralo-en Comparing :lacralo-en archives at https://atlarge-lists.icann.org/pipermail/lac-discuss-en/ and lacralo-es archive at https://atlarge-lists.icann.org/pipermail/lac-discuss-es/ Month/Year Number of lacralo-en emails Number of lacralo-es emails Missing emails Jan 2019 65 92 27 Dec 2018 63 71 8 Nov 2018 67 61 6 Oct 2018 63 70 7 Sept 2018 61 54 7 Aug 2018 76 69 7 Jul 2018 111 120 9 Jun 2018 66 77 11 May 2018 179 229 50 Apr 2018 117 148 31 Mar 2018 56 80 24 Feb 2018 61 66 5 Jan 2018 83 104 21	Dev Anand Teelucksingh	14 Jan 2019
2	Better identification of which email gives the transbot problems, what and where in the email gives the transbot problems in its error response emails. When the translation tool has issues with the email, an email is sent from transbot-no-reply@icann.org with the static subject line "Unable to translate your email to ICANN lists" and a template text like Dear <sender> Thank you for your participation in the ICANN email list new-transbot-en. You are getting this email because we were unable to translate your post automatically. It violated one or more of the formatting rules we must impose to make translation possible. A complete description of the formatting rules is available at: https://community.icann.org/x/aYtEAg In preparing your post for translation, we found the following format violations in your message:     <issue, usually Sentence punctuation must be followed by a space> Please edit your post and send it again. Thank you. The format violations message doesn't say WHERE the error is in the email. If its a long email, then how can users identify and correct such issues. Persons getting these messages and not easily seeing where the problem is aren't likely to understand how to do future emails better and the warning messages becomes more of an hindrance. Perhaps a workaround is to have the text of the email in the error email and identify what section the transbot has issues with. Other questions : Why is the test required? Are the reasons for this test still valid ( translation tool can only send a limited amount of characters to the Google API) How are domain names handled? Since domain names can't have spaces, do domain names without beginning with http:// trigger the error?	Dev Anand Teelucksingh	12 Sep 2017  updated 14 Jan 2019		Tool now identifies the subject line of the problem email in the body of the email
3	<DNT> tag isn't case sensitive. Using <DNT> tag works to not translate text, but <dnt> does not. The tool should be able to treat the DNT tag, regardless of case.	Dev Anand Teelucksingh	07/03/2017		Fixed in November 2019 version of tool
3	The <DNT> tag will be seen in the original email but not in the translated one. See EN and ES messages. Q: Should the <DNT> tags be removed in the original email? I'm thinking it should	Dev Anand Teelucksingh	07/03/2017		To achieve this, it would be necessary to re-architecture the transbot code. Upon investigating the issue it was discovered this is a known limitation, rather than a bug in the existing code. The request has been recorded as something to consider in a future iteration of the translation tool.
4	ES Translated emails seem to remove several line breaks from the EN See EN and ES messages	Dev Anand Teelucksingh	07 Mar 2017
5	Re: attachments, new-transbot lists have a message and attachment limit of 200K Given many PDFs will be larger, will be hard to test unless message size limit is raised Tested with a smaller attachment, the attachment does go through. See EN and ES	Dev Anand Teelucksingh	08/03/2017		The new-transbot list email size limit has been increased to 400K. This is enforced for the entire email, including text and attachment.
6	Handling an email sent to both new-transbot-en and new-transbot-es lists at the same time. When such an email to both lists happens, some emails don't get translated.	Dev Anand Teelucksingh, based on testing from Alfredo Calderon	09/03/2017

Fixed Bug Reports/Observations with the new translation engine (updated January 2019)

Best practices and other notes on the new translation engine (March 2017)

• As a best practice, emails in Spanish should be sent to the -es list only, and emails in English should be sent to the -en list only. Unusual behavior has been observed when an email is sent to both the -es and en list at the same time. In some cases, this results in one list receiving an un-translated and two translated emails; in other cases, the lists receive only un-translated emails. Until we have a solution for this, sending emails to only one list will reduce the risk of errors.
• Formatting changes by email clients have been observed in translated emails. In these cases, the email engine delivered the email to the translation tool in a different format than seen in the original. Format changes have been inconsistent and generally minor, not impacting the email content. Avoiding extensive formatting and using plain text in the original email has shown to minimize impact to format.

Background

A custom machine translation tool for the Latin American and Caribbean Regional At-Large Organisation (LACRALO) mailing lists was implemented around early 2011. However several issues or factors continue to negatively impact the working of the machine translation tool and in turn, has lead to great difficulty in communication and collaboration with the English and Spanish speaking communities in the LAC region.

See presentation by the At-Large Technology Taskforce Working Group for the ICANN53 meeting which has worked to identify the issues and continues to follow this issue with ICANN Staff.

To recap, LACRALO has two mailing lists

• LACRALO list in English: http://atlarge-lists.icann.org/pipermail/lac-discuss-en/
• LACRALO list in Spanish: http://atlarge-lists.icann.org/pipermail/lac-discuss-es/

Emails in english sent to lac-discuss-en@atlarge-lists.icann.org are machine translated via your custom tool using Google Translate and posted to lac-discuss-es@atlarge-lists.icann.org.

Similarly, emails in Spanish sent to the lac-discuss-es@atlarge-lists.icann.org are translated and posted to lac-discuss-en@atlarge-lists.icann.org.

Issues identified since 2012

(this section is from a June 18 2012 email)

1) Attachments in emails sent to a list are not received on the other list.

When an email with attachments such as PDFs is sent to one list, the subject line and body of the email is translated and sent to the other list BUT without the attachment.

2) Subject lines of translated emails from ES to EN become garbled.

The subject line of translated emails (seemingly) from the lac-discuss-ES list to the lac-discuss-EN list often translated to garbled text.

Examples abound from a review of the archives.

Such gibberish in the subject lines can get even worse if someone responds on the EN list and the translation further scrambles the subject line on the other list.

3) Missing [lac-discuss-es] in subject lines of translated emails posted to the lac-discuss-es list

Consider example #1 again -
First email posted to lac-discuss-en list :
Email: http://atlarge-lists.icann.org/pipermail/lac-discuss-en/2012/005932.html
Subject line: [lac-discuss-en] ICANN full list of applied for gTLD strings

which is translated and posted to lac-discuss-es list as:
Email: http://atlarge-lists.icann.org/pipermail/lac-discuss-es/2012/004552.html
Subject line: Lista completa de la ICANN solicitó cadenas de gTLD

The email at http://atlarge-lists.icann.org/pipermail/lac-discuss-es/2012/004552.html shows that the subject line is missing the [lac-discuss-es]. This hampers filtering by ES users and makes it difficult to track threaded conversations.

4) Unusual superscript and other odd characters in translated emails

There have been numerous complaints about the quality of the translation of the actual body of emails with strange characters, some of which are superscript characters appearing in the translated version.
Examples aboud (repeating phrase, I'm afraid) on the LACRALO list archives, here is one example:
Email to lac-discuss-en : http://atlarge-lists.icann.org/pipermail/lac-discuss-en/2012/005858.html
got translated to this on the lac-discuss-es list: http://atlarge-lists.icann.org/pipermail/lac-discuss-es/2012/004483.html

As you can see,
* a character like a double quote " is translated to "
* a word like "organisation" is translated to organización
* a sentence like "The highest decision making body in any organisation is also subject to rules." is translated to
"El más alto órgano de decisión en cualquier la organización también está sujeto a reglas."

History of machine translation of LACRALO mailing lists

• The LACRALO list in English: http://atlarge-lists.icann.org/pipermail/lac-discuss-en/ and LACRALO list in Spanish: http://atlarge-lists.icann.org/pipermail/lac-discuss-es/ were machine translated around September 2008. The translation engine was based on Systran.
• The quality of translation left a lot to be desired in terms of the language translation. Also, there were technical issues such as the subject lines of translated emails becoming garbled.
• During the LACRALO meeting in Cartagena in 2010, there were intense discussions regarding the quality of mailing list translations. Some wanted to disable the translation entirely. Others wanted to keep the imperfect translation for mobile  users and disabling translation would send the wrong signal to ICANN when ICANN should be giving language services a higher priority. ICANN staff acknowledged the issues and stressed that IT staff were aware of the issues and would be looking to solve the translation problems.
• Around May 2011, the translation engine was switched from Systran to Google Translate.
• in May 2011, Google announced that it would be depreciating the (then) free Google Translate API (http://googlecode.blogspot.com/2011/05/spring-cleaning-for-some-of-our-apis.html) in a few months but later (due to public outcry) announced that a paid version of the translate API would be offered.
• Other issues noted around June 2011 were identified, one was the translation engine would stop translating emails when they became too big. This meant that a conversation thread would stop being translated leading to missing emails from one list not being translated to the other list. See http://atlarge-lists.icann.org/pipermail/lac-discuss-en/2011/004308.html for the solution to this (translation tool would stop translating quoted text)
• Around August 2011, the translation engine was switched back to Systran due to Google rejecting calls to the Google Translate API. See http://atlarge-lists.icann.org/pipermail/lac-discuss-en/2011/004368.html
• Update to the translation engine: September 2011, http://atlarge-lists.icann.org/pipermail/lac-discuss-en/2011/004498.html
• June 2012, detailed analysis of translation issues (as noted in the introduction above)

Testing of new translation tool for LACRALO mailing lists

ICANN Staff have created two mailing lists (New-transbot-en and New-transbot-es) with a select number of persons on those lists for testing purposes.

Some of the key changes implemented in the new translation tool.

• The lack of punctuation was identified as a key issue for the poor translation of emails. This is because the translate tool can only send a certain amount of characters to the Google Translate API. Without punctuation, the translation tool would have to send text mid sentence.
  One of the outcomes from the LACRALO translation WG was the Proposed Notice when email is not translated message which would be sent to the user if the email had formatting issues.
• Subject lines would not be translated to ensure the conversation thread would not be lost and reduce the chance of garbled subject lines.  

Testers of new translation tool for LACRALO mailing lists

Previous bug reports against older version of the translation tool.

	Status	Date Added	Description	Additional Notes
(Noted by Dev Anand Teelucksingh and Satish Babu)		28 Feb 2016	Subject line in body of translated email has the sender and first line of the email on the same line, when the sender and first line should be on separate lines. Two examples: EN (original): http://mm.icann.org/pipermail/new-transbot-en/2016-February/000080.html ES (translated): http://mm.icann.org/pipermail/new-transbot-es/2016-February/000068.html EN (original) : http://mm.icann.org/pipermail/new-transbot-en/2016-February/000083.html ES (translated): http://mm.icann.org/pipermail/new-transbot-es/2016-February/000071.html	The converted body text was included immediately after the subject and from line in translated emails. A new line character was inserted between sender name and first body text line to separate them. Initial testing complete; additional testing in progress. Example: https://community.icann.org/x/AofDAw
(Noted by Satish Babu)		28 Feb 2016	There is an empty space in the beginning of most (but not all) lines in the translated email EN (original): http://mm.icann.org/pipermail/new-transbot-en/2016-February/000080.html ES (translated): http://mm.icann.org/pipermail/new-transbot-es/2016-February/000068.html	After investigation it appears empty spaces may have been added by the email client or by Google Translate API. Space removed before all translated lines in mailing list emails to resolve the issue. Tested with mail IDs Outlook, Yahoo, Gmail. Verified that translated emails are not indented; empty spaces are not appearing at the beginning of lines. Example: https://community.icann.org/x/vYPDAw
(Noted by Satish Babu)		28 February 2016	At the end of the message, the sender's name starts with a lower case ('dev Anand', although it is 'Dev Anand' in the original message EN (original): http://mm.icann.org/pipermail/new-transbot-en/2016-February/000080.html ES (translated): http://mm.icann.org/pipermail/new-transbot-es/2016-February/000068.html	Google Translate understands 'Dev' as an abbreviation and as a rule converts it to all lowercase. Name was hardcoded to fix 'Dev' to begin with a capital letter. Tested with mail IDs Outlook, Yahoo, Gmail. Verified that name is appearing correctly with first letter capitalized. Example: https://community.icann.org/x/ooPDAw
(Noted by Satish Babu)		28 February 2016	'Transbot' is mis-spelled as 'tansbot' (third line from the bottom) EN (original): http://mm.icann.org/pipermail/new-transbot-en/2016-February/000080.html ES (translated): http://mm.icann.org/pipermail/new-transbot-es/2016-February/000068.html	Misspelling was hardcoded. Applied hardcode fix to correct spelling of 'tansbot' to 'transbot' in translated emails. Tested with mail IDs Outlook, Yahoo, Gmail. Verified that spelling is now appearing correctly as 'transbot.' Example: https://community.icann.org/x/oILDAw
(Noted by Dev Anand Teelucksingh		April 17 2016	the transbot can't handle cedilla - as At-Large Staff signature lists a staff member with a cedilla in her name, any message from At-Large Staff will result in a message not translated See Email with cedilla  EN: http://mm.icann.org/pipermail/new-transbot-en/2016-April/000085.html  ; ES : http://mm.icann.org/pipermail/new-transbot-es/2016-April/000073.html Email with cedilla removed : EN : http://mm.icann.org/pipermail/new-transbot-en/2016-April/000094.html ; ES : http://mm.icann.org/pipermail/new-transbot-es/2016-April/000082.html The April thread with the subject line "CALL FOR MEMBERS: At-Large Public Interest Working Group" on EN : http://mm.icann.org/pipermail/new-transbot-en/2016-April/thread.html and ES : http://mm.icann.org/pipermail/new-transbot-es/2016-April/thread.html showed how the issue was isolated after several variations of the original email were tried. This is a critical bug, as any cedilla in any word in an email would result in the email not being translated.	The issue has been resolved specific to the reported cases of broken emails caused by the cedilla character in the AL Staff signature. As a larger issue, it is still in progress. Efforts around the reported case led to wider investigation into how the transbot and email applications handle Unicode characters. This is important UTF-8 compliance work and requires extensive testing. Recent tests with a wider set of characters using Outlook have been successful. Tests with those same characters have been inconsistent with Gmail and Yahoo. The team is continuing to research, test, and make progress. Examples: https://community.icann.org/x/pYrDAw
Noted by Dev Anand Teelucksingh		April 17 2016	The phrase "This Working Group is open to interested members of the At-Large community." gets translated to "Este grupo de trabajo está abierto a los miembros interesados \u200b\u200bde la comunidad de alcance." Not sure why it repeatedly happens for that phrase: See EN: http://mm.icann.org/pipermail/new-transbot-en/2016-April/000090.html ES: http://mm.icann.org/pipermail/new-transbot-es/2016-April/000078.html	The issue was related to zero-width space, which was being injected by Google Translate API. Zero-width space is used after characters that aren't followed by a visible space, but after which there may be a line break (source). It was encoded into the Unicode and was appearing as /u200b. To fix, it was replaced in the translated text with no space. Tested with mail IDs Outlook, Yahoo, Gmail. Verified that phrase is translated correctly without additional characters. Example: https://community.icann.org/x/t4PDAw
			Email subject lines can get jumbled and distorted along threads of translated emails	This issue is related to extra spaces in subject lines, and research shows it is a known issue with Microsoft Office/Mail-man server. There is not a known resolution at this time. As a workaround solution, the new test lists were designed so that subject lines aren't translated and original subject are retained.
			Attachments are not retained on translated emails	Changes were made to support attachments on translated emails. The file formats that will be retained between lists are TXT, PDF, WORD, JPEG, PPT, PNG, GIF