feat(i18n): do-not-translate registry + MACHINE_READ marker standard#41651
feat(i18n): do-not-translate registry + MACHINE_READ marker standard#41651rusackas wants to merge 9 commits into
Conversation
The backfill only checked for an empty msgstr, so it filled entries a human translator had deliberately left blank — e.g. the ru catalog's "# Не переводить" (do-not-translate) markers on literal tokens. This translated strings that must stay identical to the source: icon names (`bolt` -> the ⚡ Explore control icon), d3 enum values (`step-after`/`step-before`), API field names (`error_message`), SQL keywords, code constants, and example placeholders. Add `_is_do_not_translate`, consulted when collecting untranslated entries. An entry is skipped if its msgid is in the curated `DO_NOT_TRANSLATE` set (seeded from the ru translator's markers; language-independent) or if it carries an explicit do-not-translate translator comment in any catalog. Skipped entries are reported and left untranslated (source-token fallback). Adds unit tests for the curated set, translator-comment honoring (incl. the Cyrillic marker), and the normal-entry pass-through. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rusackas
requested review from
betodealmeida,
dpgaspar,
eschutho,
geido,
hainenber,
kgabryje,
mistercrunch,
nytai,
sadpandajoe,
sha174n and
villebro
as code owners
github-actions
Bot
added
doc
![codeant-ai-for-open-source[bot]](https://avatars.githubusercontent.com/u/151810221?s=60&v=4){kind=small}
✅ Deploy Preview for superset-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
![bito-code-review[bot]](https://avatars.githubusercontent.com/in/1061978?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review Agent Run #502ef6
Actionable Suggestions - 1
-
scripts/translations/backfill_po.py - 1
- Missing integration test for DNT filtering · Line 704-706
Review Details
-
Files reviewed - 2 · Commit Range:
fb8af9c..fb8af9c- scripts/translations/backfill_po.py
- tests/unit_tests/scripts/translations/backfill_po_test.py
-
Files skipped - 1
- docs/developer_docs/contributing/howtos.md - Reason: Filter setting
-
Tools
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
Bito Usage Guide
Commands
Type the following command in the pull request comment and save the comment.
-
/review- Manually triggers a full AI review. -
/pause- Pauses automatic reviews on this pull request. -
/resume- Resumes automatic reviews. -
/resolve- Marks all Bito-posted review comments as resolved. -
/abort- Cancels all in-progress reviews.
Refer to the documentation for additional commands.
Configuration
This repository uses Superset You can customize the agent settings here or contact your Bito workspace admin at evan@preset.io.
Documentation & Help
AI Code Review powered by
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #41651 +/- ##
==========================================
+ Coverage 64.56% 64.61% +0.05%
==========================================
Files 2673 2685 +12
Lines 147634 148542 +908
Branches 34090 34209 +119
==========================================
+ Hits 95314 95976 +662
- Misses 50591 50799 +208
- Partials 1729 1767 +38
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the hardcoded DO_NOT_TRANSLATE set with a canonical registry file (superset/translations/do-not-translate.txt) and a propagating, gettext-native marker, so do-not-translate status is consistent across every catalog and maintained in one place. - superset/translations/do-not-translate.txt: the msgid registry. - scripts/translations/apply_do_not_translate.py: stamps each registry msgid in messages.pot with a `#. MACHINE_READ-DO_NOT_TRANSLATE` extracted comment via format-preserving text edit (idempotent). Wired into babel_update.sh after extraction; `pybabel update` then propagates the marker into every .po. - backfill_po.py: loads the registry (single source of truth) and honors the `#. MACHINE_READ-DO_NOT_TRANSLATE` marker (extracted comment) as well as the legacy `# Не переводить` translator comment, in the extracted OR translator comment. - UPDATING.md + howtos: document the new standard. There is no official gettext "do not translate" flag; an extracted comment propagated from the .pot is the closest native, cross-language mechanism. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rusackas
force-pushed
the
chore/i18n-skip-do-not-translate
branch
from
bd03dc7 to
def156d
Compare
rusackas
changed the title
|
cc @PolinaFam @goldjee — heads up, and thank you for curating the Since gettext has no official "do not translate" flag and translator comments don't propagate across catalogs, this PR promotes that idea to a repo-wide standard: a Your existing |
Code Review Agent Run #04d7fbActionable Suggestions - 0Additional Suggestions - 1
Review Details
Bito Usage GuideCommands Type the following command in the pull request comment and save the comment.
Refer to the documentation for additional commands. Configuration This repository uses Documentation & Help AI Code Review powered by |
…e tokens)
Follow-up to the review pass on the Persian catalog:
- 'Unpin' was translated 'اجرا کردن' ('execute'), which is wrong and
inconsistent with the Unpin action family. Corrected to 'جداکردن' (detach),
matching 'Unpin from the result panel'/'Unpin from top'.
- 'deck.gl Geojson', 'dttm', 'p1', 'p5', 'p95', 'p99' were identity
passthroughs (msgstr == English). These are do-not-translate tokens (proper
nouns, a datetime column abbreviation, percentile statistic labels), so they
are emptied to fall back to the source literal, and added to the
do-not-translate registry (#41651).
Left as-is: the 'Delayed (missed %s refresh)' plural has identical singular/
plural forms, which is correct Persian (nouns stay singular after a numeral;
18 of 39 existing plural entries do the same).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…late registry These surfaced during the Persian review as identity passthroughs (machine 'translations' equal to the English source): the percentile statistic labels p1/p5/p95/p99, the datetime column abbreviation dttm, and the proper-noun viz name 'deck.gl Geojson'. They must stay literal, so add them to the registry; apply_do_not_translate.py stamps the MACHINE_READ-DO_NOT_TRANSLATE marker into messages.pot, which propagates to every catalog. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Code Review Agent Run #1ec8cbActionable Suggestions - 0Review Details
Bito Usage GuideCommands Type the following command in the pull request comment and save the comment.
Refer to the documentation for additional commands. Configuration This repository uses Documentation & Help AI Code Review powered by |
Addresses code-review feedback on apply_do_not_translate.py: - Strip each registry line before the blank/comment check, so trailing whitespace or an indented comment can't yield a msgid that fails to match the .pot (Critical). - Fail fast with a non-zero exit when the registry file is missing, rather than silently producing catalogs without do-not-translate markers; an existing but empty registry remains a valid no-op (Major). - Add explicit type annotations to the module-level constants. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per review, add the explicit `Path` annotation on the module-level registry path constant, matching the sibling constants. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Code Review Agent Run #32bbc8Actionable Suggestions - 0Review Details
Bito Usage GuideCommands Type the following command in the pull request comment and save the comment.
Refer to the documentation for additional commands. Configuration This repository uses Documentation & Help AI Code Review powered by |
…po, annotate locals Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Code Review Agent Run #cfa57cActionable Suggestions - 0Review Details
Bito Usage GuideCommands Type the following command in the pull request comment and save the comment.
Refer to the documentation for additional commands. Configuration This repository uses Documentation & Help AI Code Review powered by |
| # "# Не переводить"). Honored so a human's deliberate decision is never | ||
| # overridden even if a msgid is missing from the registry. | ||
| _DO_NOT_TRANSLATE_COMMENT: re.Pattern[str] = re.compile( | ||
| r"machine_read-do_not_translate|не\s+переводить" |
There was a problem hiding this comment.
Can we switch the legacy Russian translator comment to use the standard marker? Or it should be on a follow up PR?
There was a problem hiding this comment.
Checked the ru catalog — all 21 # Не переводить msgids are already in the registry, so the standard marker lands there automatically on the next babel_update.sh run. Keeping catalog regeneration out of this PR; the legacy comments stay honored by backfill_po.py regardless.
|
I'd rather to have |
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Code Review Agent Run #0422a3Actionable Suggestions - 0Review Details
Bito Usage GuideCommands Type the following command in the pull request comment and save the comment.
Refer to the documentation for additional commands. Configuration This repository uses Documentation & Help AI Code Review powered by |
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Code Review Agent Run #2d4994Actionable Suggestions - 0Review Details
Bito Usage GuideCommands Type the following command in the pull request comment and save the comment.
Refer to the documentation for additional commands. Configuration This repository uses Documentation & Help AI Code Review powered by |
3 participants
SUMMARY
Establishes a standardized, cross-language do-not-translate convention for the translation catalogs, and teaches the AI backfill (
backfill_po.py) to honor it.Background. Some catalog entries must stay identical to the source or things break — icon names (
bolt→ the ⚡ Explore control icon), d3 enum values (step-after/step-before), API field names (error_message), SQL keywords, code constants, and example placeholders. A Russian translator had been marking these with a# Не переводитьtranslator comment (thanks @PolinaFam, @goldjee!), but translator comments are per-catalog and don't propagate, and there is no official gettext "do not translate" flag. The AI backfill, which only checked for an emptymsgstr, translated several of them (CodeAnt flagged them on #41649).The standard. gettext extracted comments (
#.) do propagate frommessages.potinto every.poonpybabel update. So:superset/translations/do-not-translate.txt— a canonical registry of do-not-translate msgids (one source of truth).scripts/translations/apply_do_not_translate.py— stamps each registry msgid inmessages.potwith a#. MACHINE_READ-DO_NOT_TRANSLATEextracted comment (format-preserving text edit; idempotent). Wired intobabel_update.shright after extraction, sopybabel updatethen propagates the marker to every catalog — consistent across languages.backfill_po.py— loads the registry and honors the#. MACHINE_READ-DO_NOT_TRANSLATEmarker (and the legacy# Не переводитьtranslator comment, for back-compat), leaving these entries untranslated (source fallback).UPDATING.md+ contributing docs — document the new standard.To mark a new string do-not-translate going forward: add its msgid to the registry. No per-catalog annotation needed.
The
messages.potin this PR already carries the 21 initial markers; they propagate to each.pothe next timebabel_update.sh/pybabel updateruns.BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
N/A — tooling + build.
TESTING INSTRUCTIONS
ADDITIONAL INFORMATION