## Curation Note
This skill gained rapid adoption in the iOS developer community after Claude Code's agentic capabilities expanded. Traditional pixel-based automation breaks when UI changes happen, but this semantic approach uses accessibility APIs to find elements by meaning. The 96% token reduction claim comes from comparing raw simulator output against structured script responses. Particularly valuable for CI/CD pipelines where flaky tests waste engineering time.
## Core Philosophy: Semantic Navigation
Traditional automation:
```bash
idb ui tap 320 400
```
Semantic automation:
```bash
python scripts/navigator.py --find-text "Login" --tap
```
The key difference: Elements are discovered by their accessibility labels, text content, or element type rather than screen coordinates. This survives UI redesigns and works across different screen sizes.
## Prerequisites
- macOS with Xcode installed
- iOS Simulator (comes with Xcode)
- Python 3.8+ for automation scripts
- idb (iOS Development Bridge) or simctl
## Script Categories
### Build & Development
```bash
xcodebuild -scheme MyApp -destination 'platform=iOS Simulator,name=iPhone 15' build
xcrun simctl install booted ./build/MyApp.app
xcrun simctl launch booted com.company.myapp
```
### Navigation & Interaction
```python
import subprocess
import json
def find_element_by_text(text):
"""Find element by accessibility label or visible text."""
result = subprocess.run(
['xcrun', 'simctl', 'ui', 'booted', 'describe'],
capture_output=True, text=True
)
pass
def tap_element(element_id):
"""Tap element by its identifier."""
subprocess.run(['xcrun', 'simctl', 'io', 'booted', 'tap', element_id])
def type_text(text):
"""Type text into focused input."""
subprocess.run(['xcrun', 'simctl', 'io', 'booted', 'type', text])
```
### Testing & Analysis
```python
def capture_screenshot(output_path):
"""Capture current simulator screen."""
subprocess.run([
'xcrun', 'simctl', 'io', 'booted',
'screenshot', output_path
])
def visual_diff(baseline, current):
"""Compare screenshots for visual regressions."""
pass
```
### Device Lifecycle
```bash
xcrun simctl list devices
xcrun simctl boot "iPhone 15 Pro"
xcrun simctl shutdown booted
xcrun simctl erase booted
xcrun simctl delete unavailable
```
## Common Workflows
### Workflow 1: Login Flow Testing
```python
def test_login_flow():
"""Test complete login flow."""
launch_app("com.company.myapp")
wait_for_element("Welcome")
tap_element_by_text("Sign In")
wait_for_element("Email")
tap_element_by_text("Email")
type_text("test@example.com")
tap_element_by_text("Password")
type_text("securepassword123")
tap_element_by_text("Log In")
assert wait_for_element("Dashboard", timeout=10)
capture_screenshot("/tmp/login_success.png")
```
### Workflow 2: Accessibility Audit
```python
def audit_accessibility():
"""Check accessibility compliance."""
result = subprocess.run([
'xcrun', 'simctl', 'ui', 'booted', 'describe',
'--format', 'json'
], capture_output=True, text=True)
tree = json.loads(result.stdout)
issues = []
for element in traverse_tree(tree):
if element.get('isAccessibilityElement') and not element.get('label'):
issues.append(f"Missing label: {element}")
return issues
```
### Workflow 3: Visual Regression Testing
```python
def visual_regression_test(test_name):
"""Compare current screenshot against baseline."""
baseline_path = f"baselines/{test_name}.png"
current_path = f"/tmp/{test_name}_current.png"
capture_screenshot(current_path)
if os.path.exists(baseline_path):
diff = compare_images(baseline_path, current_path)
if diff > 0.01: # More than 1% difference
save_diff_image(baseline_path, current_path, f"diffs/{test_name}.png")
raise AssertionError(f"Visual regression detected: {diff*100:.2f}% difference")
else:
shutil.copy(current_path, baseline_path)
```
## Output Optimization
Minimize token usage by returning structured, concise output:
```python
result = subprocess.run(['xcrun', 'simctl', 'list'], capture_output=True)
return result.stdout # Could be thousands of lines
def get_devices():
result = subprocess.run(['xcrun', 'simctl', 'list', '-j'], capture_output=True)
data = json.loads(result.stdout)
return [
{"name": d["name"], "udid": d["udid"], "state": d["state"]}
for d in data["devices"].values()
if d["isAvailable"]
]
```
## Troubleshooting
### Environment Issues
```bash
xcode-select -p
sudo xcodebuild -license accept
xcrun simctl shutdown all
xcrun simctl erase all
```
### Element Not Found
1. Check if element has accessibility label
2. Wait for animations to complete
3. Verify element is on screen (not scrolled out of view)
4. Check if element is inside a container with clipped content
### Simulator Not Responding
```bash
killall Simulator
rm -rf ~/Library/Developer/Xcode/DerivedData
```
## Best Practices
1. **Always use semantic selectors** - Never rely on coordinates
2. **Add explicit waits** - UI renders are not instant
3. **Reset state between tests** - Use `simctl erase` for clean slate
4. **Capture screenshots on failure** - Essential for debugging
5. **Use JSON output** - Parse structured data, not text
6. **Batch simulator operations** - Boot once, run many tests
7. **Check accessibility first** - Missing labels break automation
## Related Resources
- [Apple Simulator Help](https://help.apple.com/simulator/mac/current/)
- [xcrun simctl documentation](https://nshipster.com/simctl/)
- [Accessibility Programming Guide](https://developer.apple.com/accessibility/)