Merge pull request #4664 from pangeachat/copilot/clean-out-unused-intl-keys

Add scripts to identify and remove unused translation keys in intl files

.gitignore

@@ -81,6 +81,10 @@ olm
 needed-translations.txt
 .venv
 
+# Generated files from find_unused_intl_keys.py
+scripts/unused_intl_keys_report.txt
+scripts/unused_intl_keys.json
+
 docs/node_modules/.package-lock.json
 docs/node_modules/.bin/detect-libc
 docs/node_modules/.bin/jiti

scripts/FIND_UNUSED_INTL_KEYS.md

@@ -0,0 +1,118 @@
+# Find Unused Translation Keys Script
+
+This Python script helps identify unused translation keys in the `intl_en.arb` file that are not referenced anywhere in the codebase.
+
+## Purpose
+
+The script was created to clean up the internationalization (i18n) files by finding translation keys that are defined but never used. This helps maintain a cleaner codebase and reduces translation overhead.
+
+## How It Works
+
+1. **Extracts Keys**: Reads `lib/l10n/intl_en.arb` and extracts all translation keys after line 3243 (configurable)
+2. **Filters Metadata**: Automatically excludes keys starting with `@` (metadata keys)
+3. **Filters Placeholders**: Excludes nested placeholder keys inside metadata objects (e.g., `l1`, `l2`, `type`, `placeholders`)
+4. **Searches Repository**: Uses `git grep` to efficiently search for each key in the repository
+5. **Filters Results**: Excludes matches found only in `.arb` files (other language files)
+6. **Reports Findings**: Generates a JSON file with the list of unused keys
+
+## Usage
+
+```bash
+# Run from repository root
+python3 scripts/find_unused_intl_keys.py
+```
+
+## Output
+
+The script provides two types of output:
+
+1. **Console Output**: Real-time progress and summary of findings
+2. **JSON File**: List of unused keys saved to `scripts/unused_intl_keys.json`
+
+### Sample Console Output
+
+```
+Extracting keys from /path/to/intl_en.arb after line 3243...
+Found 1869 translation keys to check.
+
+Searching repository for key references...
+  Checked 10/1869 keys...
+  ...
+  
+Search complete!
+Total keys checked: 1869
+Used keys: 1381
+Unused keys: 488
+
+================================================================================
+RESULTS
+================================================================================
+
+Found 488 unused keys (not referenced in any .dart files):
+--------------------------------------------------------------------------------
+  - aaDisplayName
+  - abDisplayName
+  - acceptSelection
+  ...
+```
+
+### Sample JSON Output
+
+```json
+{
+  "unused_keys": [
+    "aaDisplayName",
+    "abDisplayName",
+    "acceptSelection",
+    ...
+  ],
+  "count": 488,
+  "source_file": "/path/to/lib/l10n/intl_en.arb",
+  "start_line": 3243
+}
+```
+
+## Understanding the Results
+
+- **Unused keys**: Translation keys that appear only in `.arb` files and nowhere else in the codebase
+- **Metadata keys** (starting with `@`) are automatically excluded from the analysis
+- **Placeholder keys** (nested inside metadata objects like `placeholders`) are automatically excluded
+
+## Notes
+
+- Keys starting with `@` are metadata and are automatically skipped
+- Nested keys inside metadata objects (like `l1`, `l2` in placeholders) are automatically filtered out
+- Only top-level translation keys are analyzed
+- The script searches only for exact key matches in the repository
+- False positives are possible if keys are constructed dynamically (e.g., using string interpolation)
+- Always review the unused keys list before removing them from the translation files
+
+## Customization
+
+To check from a different line number, modify the `start_line` parameter in the `main()` function:
+
+```python
+unused_keys = find_unused_keys(str(arb_file_path), str(repo_path), start_line=3243)
+```
+
+## Requirements
+
+- Python 3.x
+- Git (for `git grep` command)
+- Repository must be a git repository
+
+## Next Steps
+
+After identifying unused keys:
+
+1. Review the unused keys list in `scripts/unused_intl_keys.json`
+2. Determine which keys can be safely removed
+3. Remove unused keys from `intl_en.arb`
+4. Run the script again to verify
+5. Consider removing the same keys from other language `.arb` files
+
+## Related Files
+
+- Source translation file: `lib/l10n/intl_en.arb`
+- Other language files: `lib/l10n/intl_*.arb`
+- Generated JSON output: `scripts/unused_intl_keys.json`

scripts/REMOVE_UNUSED_INTL_KEYS.md

@@ -0,0 +1,93 @@
+# Remove Unused Translation Keys Script
+
+This Python script removes unused translation keys from all `.arb` files in the repository.
+
+## Purpose
+
+After identifying unused keys with `find_unused_intl_keys.py`, this script automates the removal of those keys from all language files. It removes both the key-value pairs and their corresponding metadata entries.
+
+## How It Works
+
+1. **Loads Unused Keys**: Reads the list of unused keys from `unused_intl_keys.json`
+2. **Processes Each File**: For each `.arb` file in `lib/l10n/`:
+   - Loads the file as JSON
+   - Identifies keys to remove (both base keys and their `@key` metadata)
+   - Removes those keys while preserving order
+   - Writes the cleaned JSON back to the file
+3. **Reports Results**: Shows how many keys were removed from each file
+
+## Usage
+
+```bash
+# First, generate the list of unused keys
+python3 scripts/find_unused_intl_keys.py
+
+# Then, remove those keys from all .arb files
+python3 scripts/remove_unused_intl_keys.py
+```
+
+## Output
+
+The script provides progress output showing how many keys were removed from each file:
+
+```
+Loading unused keys from JSON file...
+Found 488 unused keys to remove.
+
+Found 54 .arb files to process.
+
+Processing .arb files...
+================================================================================
+intl_en.arb: Removed 488 keys/metadata entries
+intl_es.arb: Removed 557 keys/metadata entries
+intl_fr.arb: Removed 950 keys/metadata entries
+...
+================================================================================
+
+Total keys/metadata entries removed: 50015
+Processed 54 .arb files successfully.
+```
+
+## What Gets Removed
+
+For each unused key (e.g., `accountInformation`), the script removes:
+1. The key itself: `"accountInformation": "Account information"`
+2. Its metadata (if present): `"@accountInformation": { ... }`
+
+## Important Notes
+
+- **Backup recommended**: The script modifies files in place. Consider committing your work or creating a backup before running.
+- **JSON parsing**: The script uses Python's JSON library, which:
+  - Preserves the order of keys (using OrderedDict)
+  - May reformat indentation to 2 spaces
+  - Resolves duplicate keys by keeping the last value
+- **Validation**: After running, verify the files are still valid JSON and that the application still works correctly.
+
+## Requirements
+
+- Python 3.x
+- Input file: `scripts/unused_intl_keys.json` (generated by `find_unused_intl_keys.py`)
+
+## Example
+
+```bash
+# Full workflow
+cd /path/to/repository
+
+# Step 1: Find unused keys
+python3 scripts/find_unused_intl_keys.py
+
+# Step 2: Review the list in scripts/unused_intl_keys.json
+
+# Step 3: Remove the unused keys
+python3 scripts/remove_unused_intl_keys.py
+
+# Step 4: Verify the changes
+git diff lib/l10n/
+```
+
+## Related Files
+
+- Input: `scripts/unused_intl_keys.json`
+- Modified: All `lib/l10n/intl_*.arb` files
+- Related: `scripts/find_unused_intl_keys.py` (generates the input file)

scripts/find_unused_intl_keys.py

@@ -0,0 +1,205 @@
+#!/usr/bin/env python3
+"""
+Script to find unused translation keys in intl_en.arb after line 3243.
+
+This script:
+1. Reads intl_en.arb and extracts all translation keys after line 3243
+2. Filters out metadata keys (those starting with @)
+3. Searches the repository for references to each key
+4. Returns a JSON file with unused keys
+
+Usage:
+    python3 scripts/find_unused_intl_keys.py
+
+Output:
+    scripts/unused_intl_keys.json - JSON file containing the list of unused keys
+"""
+
+import json
+import os
+import re
+import subprocess
+from pathlib import Path
+from typing import Set, List
+
+
+def extract_keys_after_line(arb_file_path: str, start_line: int = 3243) -> List[str]:
+    """
+    Extract translation keys from .arb file after a specific line.
+    
+    ARB files are JSON files where keys starting with @ are metadata.
+    We only want the actual translation keys (non-@ keys), not placeholder
+    keys or other nested metadata fields.
+    
+    This function extracts only TOP-LEVEL keys that first appear after the
+    specified line number. Keys that appear as placeholders are ignored.
+    
+    Args:
+        arb_file_path: Path to the .arb file
+        start_line: Line number to start extracting from (1-indexed)
+    
+    Returns:
+        List of translation key names
+    """
+    # Load the entire JSON to get proper structure
+    with open(arb_file_path, 'r', encoding='utf-8') as f:
+        data = json.load(f)
+    
+    # Read file again to get line numbers for each key
+    with open(arb_file_path, 'r', encoding='utf-8') as f:
+        lines = f.readlines()
+    
+    keys = []
+    
+    # Extract only top-level keys (not nested keys inside metadata)
+    for key in data.keys():
+        # Skip metadata keys (those starting with @)
+        if key.startswith('@'):
+            continue
+        
+        # Find the FIRST occurrence of this key as a top-level definition
+        # A top-level key appears at the start of a line (after whitespace)
+        # with the pattern: "keyName": (not nested inside another object)
+        for line_num, line in enumerate(lines, start=1):
+            # Match key at the beginning of a line (indentation level 1)
+            # This ensures we're matching top-level keys, not nested ones
+            if re.match(r'^  "' + re.escape(key) + r'":\s*', line):
+                # Only include keys that appear after the specified line
+                if line_num > start_line:
+                    keys.append(key)
+                break
+    
+    return keys
+
+
+def search_key_in_repository(key: str, repo_path: str, exclude_dirs: Set[str]) -> bool:
+    """
+    Search for a key in the repository using git grep for efficiency.
+    
+    Args:
+        key: Translation key to search for
+        repo_path: Path to the repository root
+        exclude_dirs: Set of directory names to exclude from search
+    
+    Returns:
+        True if the key is found, False otherwise
+    """
+    try:
+        # Use git grep for fast searching, excluding the l10n directory
+        # We search for the key name as it would appear in Dart code
+        result = subprocess.run(
+            ['git', 'grep', '-q', key],
+            cwd=repo_path,
+            capture_output=True,
+            text=True
+        )
+        
+        # git grep returns 0 if found, 1 if not found
+        if result.returncode == 0:
+            # Found the key, but we need to verify it's not just in the .arb files
+            # Run again with output to check the files
+            result_with_output = subprocess.run(
+                ['git', 'grep', '-l', key],
+                cwd=repo_path,
+                capture_output=True,
+                text=True
+            )
+            
+            # Check if any non-.arb files contain the key
+            files = result_with_output.stdout.strip().split('\n')
+            for file in files:
+                if not file.endswith('.arb'):
+                    return True
+            
+            return False
+        else:
+            return False
+            
+    except subprocess.CalledProcessError:
+        return False
+
+
+def find_unused_keys(arb_file_path: str, repo_path: str, start_line: int = 3243) -> List[str]:
+    """
+    Find unused translation keys in the repository.
+    
+    Args:
+        arb_file_path: Path to the .arb file
+        repo_path: Path to the repository root
+        start_line: Line number to start checking from
+    
+    Returns:
+        List of unused keys
+    """
+    # Directories to exclude from search
+    exclude_dirs = {'.git', 'build', 'node_modules', '.dart_tool', 'l10n'}
+    
+    print(f"Extracting keys from {arb_file_path} after line {start_line}...")
+    keys = extract_keys_after_line(arb_file_path, start_line)
+    print(f"Found {len(keys)} translation keys to check.\n")
+    
+    unused_keys = []
+    used_count = 0
+    
+    print("Searching repository for key references...")
+    for i, key in enumerate(keys, 1):
+        # Print progress every 10 keys
+        if i % 10 == 0:
+            print(f"  Checked {i}/{len(keys)} keys...")
+        
+        if search_key_in_repository(key, repo_path, exclude_dirs):
+            used_count += 1
+        else:
+            unused_keys.append(key)
+    
+    print(f"\nSearch complete!")
+    print(f"Total keys checked: {len(keys)}")
+    print(f"Used keys: {used_count}")
+    print(f"Unused keys: {len(unused_keys)}")
+    
+    return unused_keys
+
+
+def main():
+    """Main function to run the unused key finder."""
+    # Get repository root
+    repo_path = Path(__file__).parent.parent.absolute()
+    arb_file_path = repo_path / 'lib' / 'l10n' / 'intl_en.arb'
+    
+    if not arb_file_path.exists():
+        print(f"Error: Could not find {arb_file_path}")
+        return 1
+    
+    # Find unused keys starting from line 3243
+    unused_keys = find_unused_keys(str(arb_file_path), str(repo_path), start_line=3243)
+    
+    # Print results
+    print("\n" + "="*80)
+    print("RESULTS")
+    print("="*80)
+    
+    if unused_keys:
+        print(f"\nFound {len(unused_keys)} unused keys (not referenced in any .dart files):")
+        print("-" * 80)
+        for key in sorted(unused_keys):
+            print(f"  - {key}")
+    else:
+        print("\nNo unused keys found! All keys are referenced in the codebase.")
+    
+    # Save results to JSON file
+    output_file = repo_path / 'scripts' / 'unused_intl_keys.json'
+    with open(output_file, 'w', encoding='utf-8') as f:
+        json.dump({
+            'unused_keys': sorted(unused_keys),
+            'count': len(unused_keys),
+            'source_file': str(arb_file_path),
+            'start_line': 3243
+        }, f, indent=2, ensure_ascii=False)
+    
+    print(f"\nJSON output saved to: {output_file}")
+    
+    return 0
+
+
+if __name__ == '__main__':
+    exit(main())

scripts/remove_unused_intl_keys.py

@@ -0,0 +1,135 @@
+#!/usr/bin/env python3
+"""
+Script to remove unused translation keys from all .arb files.
+
+This script:
+1. Reads the unused keys from unused_intl_keys.json
+2. Removes those keys and their metadata entries from all .arb files
+3. Preserves the overall order and structure of the files
+
+Usage:
+    python3 scripts/remove_unused_intl_keys.py
+
+Input:
+    scripts/unused_intl_keys.json - JSON file containing the list of unused keys
+    
+Output:
+    Updates all .arb files in lib/l10n/ by removing unused keys and their metadata
+"""
+
+import json
+from pathlib import Path
+from typing import List, Set
+from collections import OrderedDict
+
+
+def load_unused_keys(json_path: str) -> List[str]:
+    """
+    Load the list of unused keys from the JSON file.
+    
+    Args:
+        json_path: Path to the unused_intl_keys.json file
+    
+    Returns:
+        List of unused key names
+    """
+    with open(json_path, 'r', encoding='utf-8') as f:
+        data = json.load(f)
+    return data['unused_keys']
+
+
+def remove_keys_from_arb_file(arb_file_path: str, keys_to_remove: Set[str]) -> int:
+    """
+    Remove specified keys and their metadata from an .arb file.
+    
+    This function removes both the key-value pairs and their corresponding
+    metadata entries (which are prefixed with @).
+    
+    Args:
+        arb_file_path: Path to the .arb file
+        keys_to_remove: Set of key names to remove
+    
+    Returns:
+        Number of keys removed from this file
+    """
+    # Read the JSON file
+    with open(arb_file_path, 'r', encoding='utf-8') as f:
+        data = json.load(f, object_pairs_hook=OrderedDict)
+    
+    # Track what we remove
+    removed_count = 0
+    keys_to_delete = []
+    
+    # Identify all keys to remove (including metadata keys)
+    for key in data.keys():
+        # Check if this is a metadata key (starts with @)
+        if key.startswith('@'):
+            # Get the base key name
+            base_key = key[1:]
+            # Remove if the base key is in our removal list
+            if base_key in keys_to_remove:
+                keys_to_delete.append(key)
+                removed_count += 1
+        else:
+            # Remove if the key itself is in our removal list
+            if key in keys_to_remove:
+                keys_to_delete.append(key)
+                removed_count += 1
+    
+    # Remove the keys
+    for key in keys_to_delete:
+        del data[key]
+    
+    # Write back to file with proper formatting
+    with open(arb_file_path, 'w', encoding='utf-8') as f:
+        json.dump(data, f, indent=2, ensure_ascii=False)
+        f.write('\n')  # Add trailing newline
+    
+    return removed_count
+
+
+def main():
+    """Main function to remove unused keys from all .arb files."""
+    # Get repository root
+    repo_path = Path(__file__).parent.parent.absolute()
+    json_path = repo_path / 'scripts' / 'unused_intl_keys.json'
+    l10n_dir = repo_path / 'lib' / 'l10n'
+    
+    if not json_path.exists():
+        print(f"Error: Could not find {json_path}")
+        print("Please run find_unused_intl_keys.py first to generate the list of unused keys.")
+        return 1
+    
+    # Load unused keys
+    print("Loading unused keys from JSON file...")
+    unused_keys = load_unused_keys(str(json_path))
+    keys_to_remove = set(unused_keys)
+    print(f"Found {len(keys_to_remove)} unused keys to remove.\n")
+    
+    # Get all .arb files
+    arb_files = sorted(l10n_dir.glob('*.arb'))
+    print(f"Found {len(arb_files)} .arb files to process.\n")
+    
+    if not arb_files:
+        print(f"Error: No .arb files found in {l10n_dir}")
+        return 1
+    
+    # Process each .arb file
+    total_removed = 0
+    print("Processing .arb files...")
+    print("=" * 80)
+    
+    for arb_file in arb_files:
+        removed = remove_keys_from_arb_file(str(arb_file), keys_to_remove)
+        total_removed += removed
+        print(f"{arb_file.name}: Removed {removed} keys/metadata entries")
+    
+    print("=" * 80)
+    print(f"\nTotal keys/metadata entries removed: {total_removed}")
+    print(f"Processed {len(arb_files)} .arb files successfully.")
+    
+    return 0
+
+
+if __name__ == '__main__':
+    exit(main())