Add Dynamic Home Assistant Compatibility Testing #1669
Conversation
|
Warning Rate limit exceeded@robbrad has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 11 minutes and 48 seconds before requesting another review. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📒 Files selected for processing (2)
WalkthroughAdds GitHub Actions workflows for matrix-driven Home Assistant compatibility testing and a quick compatibility job, a compatibility-check script, new pytest tests for the uk_bin_collection component, compatibility documentation, and a .gitignore change to ensure Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant Trigger as Push/PR
participant MatrixGen as generate-matrix
participant PerVersion as test-ha-compatibility (per entry)
participant QuickCheck as hacs_validation:Quick Compatibility Check
participant Report as compatibility-report
participant Summary as Job Summaries
Trigger->>MatrixGen: start workflow
activate MatrixGen
MatrixGen->>MatrixGen: query PyPI & build HA↔Python matrix (incl. dev)
MatrixGen-->>PerVersion: emit matrix entries
deactivate MatrixGen
par Per-version runs
PerVersion->>PerVersion: setup Python & env
PerVersion->>PerVersion: install target HA (release or git dev)
PerVersion->>PerVersion: install component & deps
PerVersion->>PerVersion: boot HA headless, monitor logs
PerVersion->>PerVersion: run import/manifest checks
PerVersion->>Summary: upload logs/artifacts & append per-version result
end
Trigger->>QuickCheck: optional quick run
activate QuickCheck
QuickCheck->>QuickCheck: setup Python 3.12 & run check script
QuickCheck->>Summary: append quick-check result
deactivate QuickCheck
Report->>Report: aggregate summaries into final report
Report->>Summary: append final compatibility report
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1669 +/- ##
=======================================
Coverage 86.79% 86.79%
=======================================
Files 9 9
Lines 1136 1136
=======================================
Hits 986 986
Misses 150 150 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
custom_components/uk_bin_collection/tests/test_component.py (1)
79-95: Unused patch variable can be prefixed with underscore.The test correctly validates config entry setup. The
mock_appvariable is intentionally unused (it prevents actual instantiation), but this could be made clearer by prefixing it with an underscore to indicate it's intentionally unused.Apply this diff to clarify the intent:
- with patch('custom_components.uk_bin_collection.UKBinCollectionApp') as mock_app, \ + with patch('custom_components.uk_bin_collection.UKBinCollectionApp') as _mock_app, \ patch('custom_components.uk_bin_collection.HouseholdBinCoordinator') as mock_coordinator:
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
.github/workflows/ha_compatibility_test.yml(1 hunks).github/workflows/hacs_validation.yml(2 hunks).github/workflows/validate-release-ready.yml(1 hunks).gitignore(1 hunks)COMPATIBILITY.md(1 hunks)custom_components/uk_bin_collection/tests/test_component.py(1 hunks)scripts/check_ha_compatibility.py(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
custom_components/uk_bin_collection/tests/test_component.py (2)
custom_components/uk_bin_collection/__init__.py (2)
async_setup(28-84)build_ukbcd_args(282-309)custom_components/uk_bin_collection/tests/test_init.py (3)
async_register(79-80)async_forward_entry_setups(84-86)async_forward_entry_unload(88-90)
🪛 actionlint (1.7.8)
.github/workflows/hacs_validation.yml
31-31: the runner of "actions/setup-python@v4" action is too old to run on GitHub Actions. update the action's version to fix this issue
(action)
.github/workflows/ha_compatibility_test.yml
44-44: the runner of "actions/setup-python@v4" action is too old to run on GitHub Actions. update the action's version to fix this issue
(action)
🪛 GitHub Check: CodeQL
custom_components/uk_bin_collection/tests/test_component.py
[failure] 56-56: Incomplete URL substring sanitization
The string https://example.com may be at an arbitrary position in the sanitized URL.
🪛 LanguageTool
COMPATIBILITY.md
[uncategorized] ~17-~17: The official name of this software platform is spelled with a capital “H”.
Context: ... Home Assistant Compatibility Test (.github/workflows/ha_compatibility_test.yml) ...
(GITHUB)
[uncategorized] ~23-~23: The official name of this software platform is spelled with a capital “H”.
Context: ...eaking changes 2. HACS Validation (.github/workflows/hacs_validation.yml) - In...
(GITHUB)
🪛 Ruff (0.14.0)
scripts/check_ha_compatibility.py
1-1: Shebang is present but file is not executable
(EXE001)
30-30: Do not catch blind exception: Exception
(BLE001)
50-50: Consider moving this statement to an else block
(TRY300)
51-51: Do not catch blind exception: Exception
(BLE001)
60-60: Consider moving this statement to an else block
(TRY300)
63-63: Do not catch blind exception: Exception
(BLE001)
71-71: Consider moving this statement to an else block
(TRY300)
74-74: Do not catch blind exception: Exception
(BLE001)
custom_components/uk_bin_collection/tests/test_component.py
84-84: Local variable mock_app is assigned to but never used
Remove assignment to unused variable mock_app
(F841)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Run Integration Tests (3.12, 1.8.4)
🔇 Additional comments (15)
.gitignore (1)
47-47: LGTM!The explicit inclusion of the
scriptsdirectory is necessary to track the new compatibility checker script introduced in this PR..github/workflows/validate-release-ready.yml (1)
4-4: LGTM!The
workflow_dispatchtrigger enables manual execution of the release-readiness validation, which complements the automated PR-triggered runs and provides operational flexibility.COMPATIBILITY.md (1)
1-122: LGTM!The compatibility documentation is comprehensive and well-structured, covering supported versions, testing workflows, troubleshooting, and developer guidance. This provides clear value for both users and maintainers.
custom_components/uk_bin_collection/tests/test_component.py (6)
16-37: LGTM!The test fixtures are well-structured and provide appropriate mocks for Home Assistant's
ConfigEntryandHomeAssistantinstances with the necessary attributes for component testing.
40-64: LGTM!The test comprehensively validates argument building for
UKBinCollectionApp, including verification that excluded fields liketimeoutandnameare properly omitted from the argument list.
67-76: LGTM!The test correctly verifies that
async_setupinitializes the domain inhass.dataand registers the manual refresh service.
98-109: LGTM!The test correctly verifies that
async_unload_entryremoves the entry fromhass.data[DOMAIN]and properly forwards the unload request.
112-121: LGTM!The import test ensures all required component modules are importable, which is crucial for compatibility validation across different Home Assistant versions.
124-146: LGTM!The manifest validation test thoroughly checks the structure and required fields, including domain match and non-empty requirements list.
.github/workflows/hacs_validation.yml (1)
10-25: LGTM!The existing validation jobs remain intact with only minor formatting adjustments.
scripts/check_ha_compatibility.py (2)
16-86: LGTM!The compatibility check functions are well-designed with appropriate error handling. The broad exception catching is intentional and suitable for a compatibility checker that needs to gracefully handle and report all types of errors.
89-116: LGTM!The main function provides clear output with emoji indicators and appropriate exit codes for CI/CD integration.
.github/workflows/ha_compatibility_test.yml (3)
18-39: LGTM!The dynamic matrix generation from PyPI is well-designed. The jq pipeline correctly filters stable releases, groups by major version, selects the latest patch for each, and maps appropriate Python versions based on Home Assistant version requirements.
75-112: LGTM!The test validation steps appropriately combine pytest for comprehensive tests with inline Python scripts for focused import and manifest validation checks.
124-140: LGTM!The compatibility report job appropriately uses
if: always()to ensure the report is generated regardless of test outcomes, providing visibility into which versions were tested.
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Set up Python | ||
| uses: actions/setup-python@v4 | ||
| with: | ||
| python-version: ${{ matrix.python_version }} | ||
|
|
||
| - name: Install dependencies | ||
| run: | | ||
| python -m pip install --upgrade pip | ||
| pip install pytest pytest-asyncio pytest-homeassistant-custom-component | ||
| - name: Install Home Assistant ${{ matrix.ha_version }} | ||
| run: | | ||
| if [ "${{ matrix.ha_version }}" = "dev" ]; then | ||
| pip install git+https://github.com/home-assistant/core.git | ||
| else | ||
| pip install homeassistant==${{ matrix.ha_version }} | ||
| fi | ||
| - name: Install UK Bin Collection package | ||
| run: | | ||
| pip install -e . | ||
There was a problem hiding this comment.
Update action version to v6.
The test setup is well-structured with conditional installation for dev vs. stable HA versions. However, actions/setup-python@v4 should be updated to v6 for compatibility with current GitHub Actions runners.
Apply this diff:
- name: Set up Python
- uses: actions/setup-python@v4
+ uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python_version }}🤖 Prompt for AI Agents
In .github/workflows/ha_compatibility_test.yml around lines 50 to 74, update the
GitHub Action reference for setting up Python from actions/setup-python@v4 to
actions/setup-python@v6; keep the existing with: python-version: ${{
matrix.python_version }} block and no other changes so the step remains
functionally identical but uses the current supported action version.
| compatibility_check: | ||
| name: Quick Compatibility Check | ||
| runs-on: "ubuntu-latest" | ||
| steps: | ||
| - uses: "actions/checkout@v5" | ||
| - uses: actions/setup-python@v4 | ||
| with: | ||
| python-version: "3.12" | ||
| - name: Install dependencies | ||
| run: | | ||
| pip install homeassistant | ||
| pip install -e . | ||
| - name: Run compatibility check | ||
| run: python scripts/check_ha_compatibility.py |
There was a problem hiding this comment.
Update action version to v6.
The compatibility check job is well-structured. However, actions/setup-python@v4 is too old for current GitHub Actions runners and should be updated to v6 for compatibility.
Apply this diff:
- - uses: actions/setup-python@v4
+ - uses: actions/setup-python@v6
with:
python-version: "3.12"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| compatibility_check: | |
| name: Quick Compatibility Check | |
| runs-on: "ubuntu-latest" | |
| steps: | |
| - uses: "actions/checkout@v5" | |
| - uses: actions/setup-python@v4 | |
| with: | |
| python-version: "3.12" | |
| - name: Install dependencies | |
| run: | | |
| pip install homeassistant | |
| pip install -e . | |
| - name: Run compatibility check | |
| run: python scripts/check_ha_compatibility.py | |
| compatibility_check: | |
| name: Quick Compatibility Check | |
| runs-on: "ubuntu-latest" | |
| steps: | |
| - uses: "actions/checkout@v5" | |
| - uses: actions/setup-python@v6 | |
| with: | |
| python-version: "3.12" | |
| - name: Install dependencies | |
| run: | | |
| pip install homeassistant | |
| pip install -e . | |
| - name: Run compatibility check | |
| run: python scripts/check_ha_compatibility.py |
🧰 Tools
🪛 actionlint (1.7.8)
31-31: the runner of "actions/setup-python@v4" action is too old to run on GitHub Actions. update the action's version to fix this issue
(action)
🤖 Prompt for AI Agents
.github/workflows/hacs_validation.yml lines 26-39: the workflow uses
actions/setup-python@v4 which is outdated; update the action reference to
actions/setup-python@v6 by changing the "uses" value on the setup-python step to
the v6 tag, keeping the existing with: python-version: "3.12" block intact so
the runner uses the same Python version.
| @@ -0,0 +1,116 @@ | |||
| #!/usr/bin/env python3 | |||
There was a problem hiding this comment.
Make the script executable.
The shebang indicates this script should be executable, but the file lacks execute permissions. Either make the file executable or remove the shebang.
Run this command to make the file executable:
chmod +x scripts/check_ha_compatibility.pyAlternatively, if the script is only intended to be run as python scripts/check_ha_compatibility.py, remove the shebang line.
🧰 Tools
🪛 Ruff (0.14.0)
1-1: Shebang is present but file is not executable
(EXE001)
🤖 Prompt for AI Agents
In scripts/check_ha_compatibility.py around line 1 the file contains a shebang
but is not executable; either add execute permission to the file or remove the
shebang. To fix, run chmod +x scripts/check_ha_compatibility.py to make it
executable (preferred if intended to be invoked directly), or delete the
"#!/usr/bin/env python3" line if you only want it invoked via "python
scripts/check_ha_compatibility.py".
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (1)
.github/workflows/ha_compatibility_test.yml (1)
59-62: Verify Python action version is intentional.The past review recommended updating
actions/setup-python@v4to v6. The current code shows v5, which appears to be a partial update. Confirm whether v5 is acceptable or if v6 should be the target version.
🧹 Nitpick comments (1)
.github/workflows/ha_compatibility_test.yml (1)
65-71: Consider adding timeout or retry logic for dev HA installation.The dev HA installation (line 68) uses
pip install git+...without explicit timeout or retry configuration. While pip has built-in defaults, adding explicit resilience (e.g.,pip install --default-timeout=30) could improve reliability in the CI environment.This is a nice-to-have improvement; the current approach with pip defaults is acceptable for an optional dev test.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
.github/workflows/ha_compatibility_test.yml(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (11)
- GitHub Check: Run Unit Tests (3.12, 1.8.4)
- GitHub Check: Test HA dev
- GitHub Check: Test HA 2025.10.3
- GitHub Check: Test HA 2025.9.4
- GitHub Check: Test HA 2025.6.3
- GitHub Check: Test HA 2025.3.4
- GitHub Check: Test HA 2025.8.3
- GitHub Check: Test HA 2025.4.4
- GitHub Check: Test HA 2025.7.4
- GitHub Check: Test HA 2025.5.3
- GitHub Check: Analyze (python)
🔇 Additional comments (5)
.github/workflows/ha_compatibility_test.yml (5)
27-43: Verify Python version mapping for HA versions.The matrix generates Python versions based on a heuristic (line 39): HA 2025.2+ uses Python 3.13; earlier versions use 3.12. Confirm this mapping is accurate for the HA releases being tested, particularly for edge cases (e.g., HA 2024.12, HA 2025.1, dev branch).
101-152: Boot sequence logic is well-structured.The HA boot process correctly implements timeout handling, log monitoring for initialization, early failure detection on errors, and safe cleanup. The 150-second timeout is reasonable for CI, and the error detection captures component-specific issues.
185-198: Test result summary properly fixed.The summary logic now correctly evaluates step outcomes (
steps.boot.outputs.boot_failedandsteps.manifest.outcome) instead of shell exit codes, ensuring the summary reflects actual test results. This properly addresses the critical issue flagged in the previous review.
168-183: Manifest validation is sound.The script correctly validates required fields and provides clear pass/fail output with diagnostic information.
1-52: Workflow structure and orchestration are well-designed.Matrix generation, dependency management, and job orchestration follow GitHub Actions best practices. Path-based triggers and
fail-fast: falseappropriately balance efficiency and coverage.
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
.github/workflows/ha_compatibility_test.yml (1)
58-62: Updateactions/setup-pythonto v6.The setup-python action is pinned to v5, but v6 is available and recommended for current GitHub Actions runners.
Apply this diff:
- name: Set up Python - uses: actions/setup-python@v5 + uses: actions/setup-python@v6 with: python-version: ${{ matrix.python_version }} cache: 'pip'
🧹 Nitpick comments (1)
.github/workflows/ha_compatibility_test.yml (1)
27-43: Simplify and document the matrix generation jq logic.The PyPI version filtering is correct but dense. Consider breaking it into logical steps with comments or using separate steps for clarity, especially as HA versions evolve. The current approach works but is challenging to maintain or audit.
Example refactored approach:
MATRIX=$(curl -s https://pypi.org/pypi/homeassistant/json | jq -c ' # Step 1: Filter only semantic versions (x.y.z) .releases | to_entries | map(select(.key | test("^[0-9]+\\.[0-9]+\\.[0-9]+$"))) # Step 2: Group by major.minor, keep latest patch per series | group_by(.key | (split(".")[:2] | join("."))) | map(max_by(.key | (split(".")[2] | tonumber)) | .key) # Step 3: Sort and select latest 8 releases | sort_by(split(".") | map(tonumber)) | .[-8:] # Step 4: Map to matrix with Python versions per HA series | map({ha_version: ., python_version: (if (split(".")[0] == "2025" and (split(".")[1]|tonumber) >= 2) then "3.13" else "3.12" end)}) # Step 5: Add dev entry for edge testing | . + [{ha_version: "dev", python_version: "3.13"}] | {include: .} ')
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
.github/workflows/ha_compatibility_test.yml(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (11)
- GitHub Check: Run Integration Tests (3.12, 1.8.4)
- GitHub Check: Run Unit Tests (3.12, 1.8.4)
- GitHub Check: Test HA 2025.10.3
- GitHub Check: Test HA dev
- GitHub Check: Test HA 2025.5.3
- GitHub Check: Test HA 2025.4.4
- GitHub Check: Test HA 2025.7.4
- GitHub Check: Test HA 2025.6.3
- GitHub Check: Test HA 2025.9.4
- GitHub Check: Test HA 2025.3.4
- GitHub Check: Test HA 2025.8.3
🔇 Additional comments (2)
.github/workflows/ha_compatibility_test.yml (2)
99-150: Robust HA boot verification with clear failure detection.The boot and log monitoring logic is well-structured: it uses
set -euo pipefail, captures the process PID, polls for the initialization marker, detects component-specific failures, handles timeouts gracefully, and ensures clean shutdown. The 150s timeout and error checks are sensible. ✅
183-196: Test result summary logic is now correct.The summary step properly evaluates both the boot step output (
steps.boot.outputs.boot_failed) and the manifest step outcome (steps.manifest.outcome) to determine pass/fail, rather than relying on the preceding command's exit code. ✅
| | sort_by(split(".") | map(tonumber)) | ||
| | .[-8:] | ||
| # pick python version per HA series; adjust as needed | ||
| | map({ha_version: ., python_version: (if (split(".")[0] == "2025" and (split(".")[1]|tonumber) >= 2) then "3.13" else "3.12" end)}) |
There was a problem hiding this comment.
🧩 Analysis chain
Verify Python version mapping aligns with Home Assistant requirements.
Line 39 hardcodes Python 3.13 for HA 2025.2+, and 3.12 for all others. Confirm this matches Home Assistant's official compatibility matrix, especially as new releases are tested dynamically. If HA version requirements shift, this mapping may need updates.
🌐 Web query:
Home Assistant 2025 Python version requirements compatibility matrix
💡 Result:
Summary compatibility matrix — Home Assistant releases vs. required Python (concise):
- 2023.3 — Adds Python 3.11 support. (home-assistant.io)
- 2023.8 — Python 3.10 support removed; Home Assistant requires Python 3.11. (home-assistant.io)
- 2024.2 — Upgraded to Python 3.12 (ships running on 3.12). (home-assistant.io)
- 2024.4 — Confirms removal of 3.11 support; Home Assistant requires Python 3.12. (home-assistant.io)
- 2024.12 — Upgraded to Python 3.13. (home-assistant.io)
- 2025 (current development / pyproject) — package metadata shows requires-python >= 3.13.2 (Home Assistant 2025.x development). (deepwiki.com)
If you want, I can expand this into a downloadable table (CSV/Markdown) or look up the exact Home Assistant release that first began requiring a specific exact patch (e.g., 3.13.0 vs 3.13.2) for any particular release.
Citations:
- 1: https://www.home-assistant.io/blog/2023/03/01/release-20233/?utm_source=openai
- 2: https://www.home-assistant.io/blog/2023/08/02/release-20238/?utm_source=openai
- 3: https://www.home-assistant.io/blog/2024/02/07/release-20242/?utm_source=openai
- 4: https://www.home-assistant.io/blog/2024/04/03/release-20244/?utm_source=openai
- 5: https://www.home-assistant.io/blog/2024/12/04/release-202412/?utm_source=openai
- 6: https://deepwiki.com/home-assistant/core/1.1-build-system-and-dependencies?utm_source=openai
Fix Python version mapping: 2024.12+ requires Python 3.13, not 3.12
Home Assistant 2024.12 upgraded to Python 3.13, but the current mapping only assigns Python 3.13 for HA 2025.2+. This leaves 2024.12 incorrectly mapped to Python 3.12.
Update the condition to map Python 3.13 for HA 2024.12+ (or earlier if applicable), not just 2025.2+.
🤖 Prompt for AI Agents
.github/workflows/ha_compatibility_test.yml around line 39: the current JQ
mapping only assigns Python 3.13 for HA 2025.2+ which misclassifies HA 2024.12;
update the conditional so any HA version >= 2024.12 maps to Python 3.13.
Concretely, change the version check to test major and minor components (e.g.,
major > 2024 OR major == 2024 and minor >= 12) and return "3.13" for those
cases, otherwise return "3.12".
Problem
Users upgrading Home Assistant don't know if the UK Bin Collection custom component will work with the latest version. The project had no automated way to test compatibility across different HA versions, leading to potential breakage and user frustration.
Solution
Implemented comprehensive, dynamic compatibility testing that automatically validates the custom component against multiple Home Assistant versions.
Changes
Dynamic version matrix: Automatically fetches latest 8 HA releases from PyPI
No manual updates needed: Versions update automatically on each run
Multi-version testing: Tests against stable releases + dev branch
Comprehensive validation: Checks imports, manifest, and component functionality
Weekly schedule: Runs every Monday to catch breaking changes early
Tests component setup/teardown lifecycle
Validates manifest structure
Checks module imports
Mock-based testing for HA integration
Users can run locally before upgrading: python scripts/check_ha_compatibility.py
Validates manifest, imports, HA version, and dependencies
Clear pass/fail output with actionable messages
Added compatibility check to existing workflow
Ensures basic functionality alongside HACS validation
Compatibility matrix showing supported versions
Troubleshooting guide
Developer instructions for testing
Benefits
✅ Early detection of compatibility issues before users upgrade
✅ Automated testing on every code change
✅ User-friendly local testing script
✅ Zero maintenance - versions auto-update from PyPI
✅ Comprehensive coverage - tests multiple HA versions simultaneously
Testing
The workflow dynamically generates a test matrix by:
Querying PyPI's JSON API for all HA releases
Filtering for major releases (e.g., 2023.10.0, 2024.1.0)
Selecting the latest 8 versions + dev branch
Running full compatibility tests against each version
Usage
For users:
python scripts/check_ha_compatibility.py
Copy
bash
For developers:
The workflow runs automatically on:
Push to master/main
Pull requests
Weekly schedule (Mondays at 6 AM UTC)
Summary by CodeRabbit
Tests
Documentation
Chores