-
Notifications
You must be signed in to change notification settings - Fork 6
Update to work with Pydantic upgrade of mt-metadata #389
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
kujaku11
wants to merge
57
commits into
patches
Choose a base branch
from
pydantic
base: patches
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
local tests show: 30 failed, 46 passed, 41 warnings, 1 error in 360.48s (0:06:00)
Introduces a minimal conftest.py with fixtures for creating and cleaning up synthetic MTH5 files, and configures pytest to filter noisy warnings. Adds a pytest-based test that writes a TF object to a zrr file, reads it back, and asserts array equality, ensuring xdist safety and proper cleanup.
Added new pytest-based synthetic tests for Aurora and MTH5 processing, including feature weighting, Fourier coefficients, decimation, STFT agreement, and frequency band definition. Enhanced conftest.py with fixtures for synthetic test paths, file cleanup, and monkeypatching to sanitize provenance comments, improving test isolation and reliability.
Updated tests to use the same num_samples_window value when manually specifying band_edges, ensuring alignment with FFT harmonics and consistent transfer function results. Also removed unnecessary skip/xfail markers from pytest-based tests.
Introduces worker-safe pytest fixtures for synthetic MTH5 test files, replacing direct calls to file creation functions in tests. Updates processing helpers and all synthetic tests to accept or use these fixtures, improving test isolation, parallelism, and reliability. Also adds support for passing custom MTH5 file paths to processing helpers.
Removed unittest-based synthetic test modules and replaced them with pytest equivalents for metadata and multi-run tests. This improves test maintainability and integration with modern Python testing workflows.
Removed the unittest-based transfer function kernel test and added comprehensive pytest suites for ApodizationWindow and WindowingScheme classes. The new tests cover window generation, properties, taper families, sliding window operations, FFT, edge cases, and integration workflows, improving coverage and compatibility with pytest-xdist.
Introduces comprehensive tests for nan_to_mean, handle_nan, and time_axis_match functions in aurora.time_series.xarray_helpers. Covers edge cases, multiple channels, time axis mismatches, and data integrity, optimized for pytest-xdist parallel execution.
Deleted test_apodization_window.py, test_windowing_scheme.py, and test_xarray_helpers.py from the tests/time_series directory. These files contained unit tests for apodization windows, windowing schemes, and xarray helpers, respectively.
Introduces a comprehensive pytest test suite for the aurora.transfer_function.cross_power module. Tests cover channel name generation, transfer function computation, mathematical properties, edge cases, data integrity, numerical stability, return value characteristics, and consistency across calls. Optimized for parallel execution with pytest-xdist.
Added a comprehensive pytest suite for aurora.transfer_function.regression.helper_functions covering rme_beta, simple_solve_tf, and direct_solve_tf, including edge cases, mathematical properties, and data integrity. Removed the unittest-based cross_power test file to focus on regression testing for helper functions.
Introduces a comprehensive pytest suite for the RegressionEstimator base class, covering initialization, OLS estimation, QR decomposition, underdetermined systems, input type handling, xarray conversion, data validation, numerical stability, edge cases, data integrity, deterministic behavior, mathematical properties, and return value checks. These tests ensure correctness, robustness, and compatibility with various data types and scenarios.
Deleted test_base.py and test_helper_functions.py from tests/transfer_function/regression. These files contained unit tests for regression estimators and helper functions, possibly as part of a test suite cleanup or migration.
Introduces a new, fully refactored Parkfield test suite in tests/parkfield/test_parkfield_pytest.py, organized into multiple test classes with 25+ focused tests covering calibration, single-station and remote-reference processing, data integrity, and numerical validation. Adds extensive reusable fixtures to tests/conftest.py for efficient resource management and pytest-xdist compatibility. Includes a detailed REFACTORING_SUMMARY.md documenting the migration from three monolithic test files to a single, maintainable, and parallelizable suite with improved coverage and maintainability.
Updated attribute names from station/ch1/ch2 to station_1/channel_1/channel_2 in feature_weights.py and related test code for consistency. Improved logging for feature type and validation. Adjusted test deserialization logic to handle nested dicts and removed xfail marker from feature weighting test.
Updated docs/examples/dataset_definition.ipynb to use Windows-style paths, added 'channel_nomenclature.keyword', replaced nulls with empty strings, and changed 'units' from 'counts' to 'digital counts'. Also updated import paths, output examples, and warning messages for better Windows compatibility and current metadata conventions. Dropped Python 3.9 from test matrix in .github/workflows/tests.yaml.
Ensure CONFIG_PATH directory exists before saving JSON configs in make_processing_configs.py. Update test_decimation_methods_agree and test_stft_methods_agree to accept synthetic_test_paths argument for improved test setup.
Removed unnecessary close_open_files calls from test fixtures and helpers. Updated windowing scheme fixture to use actual data length. Improved exception handling in single-station processing test. Skipped EMTFXML export test due to known bug and clarified skip reason. Updated kernel dataset structure tests to check DataFrame contents instead of attributes. Refined numerical validation tests to check only impedance elements and verify transfer function shape using DataArray dimensions. Minor docstring and comment improvements for clarity.
Refactored comparison_plots.py to always show plots and close figures after saving, with improved logging. Commented out warning filters in conftest.py to allow all warnings during tests. Added a note in test_parkfield_pytest.py to implement impedance comparison tests.
Set matplotlib to non-interactive 'Agg' backend in test configuration to prevent blocking during tests. Refactor Parkfield MTH5 test fixtures to create and cache a master file once per session, then copy it to worker-specific directories for parallel test execution, reducing redundant downloads and avoiding file handle conflicts.
Introduced class-scoped pytest fixtures to process transfer functions once per test class and reuse them across multiple tests. This reduces redundant processing and significantly speeds up test execution, especially for expensive operations. Updated all relevant tests to use the new fixtures instead of reprocessing data.
Changed the scope of the 'parkfield_kernel_dataset_ss' and 'parkfield_kernel_dataset_rr' pytest fixtures to 'class' to optimize test setup and teardown for these resources.
Deleted test_calibrate_parkfield.py, test_process_parkfield_run.py, and test_process_parkfield_run_rr.py as part of test suite cleanup. Updated test_parkfield_pytest.py to set fixture scope to 'class' for config_ss and config_rr to improve test performance.
Refactored tests in test_processing_pytest.py and test_multi_run_pytest.py to use class-scoped pytest fixtures, caching expensive processing calls and reducing redundant computation. This significantly decreases CI runtime by sharing processed results across related tests, while maintaining test coverage and compatibility with parallel execution. Added documentation and comments to clarify which tests cannot be optimized due to inherent requirements.
Introduces methods to numerically compare transfer functions, sigma_e, and sigma_s between ZFile objects, including interpolation for mismatched periods. Updates Parkfield test to assert transfer function similarity using the new comparison utility. There is still an issue with the filters and channel metadata that is different causing the transfer function to be incorrect.
if the min_num_stft_windows is set to None the the value become 0, and therefore logic allows for 0 windows per decimation level which raises an error. Set default to 1, which eliminates 0 windows.
Update the parkfield_h5_master fixture to cache the Parkfield MTH5 file in a persistent directory (~/.cache/aurora/parkfield) instead of a temporary directory. This avoids repeated downloads across test sessions and improves test efficiency. The parkfield_h5_path fixture is updated to reflect this persistent caching approach.
Introduces a vectorized implementation of the pass_band function in mt_metadata/timeseries/filters/filter_base.py using numpy stride tricks for significant performance improvement. Adds detailed profiling, analysis, and optimization documentation, including benchmarking scripts, performance summaries, and an automated application script under tests/parkfield/. Also includes profiling data and supporting files to validate and communicate the optimization impact.
Updated the TransferFunctionKernel to set survey metadata only if not already present and to use the Survey object from the dataset. Also changed the way runs_processed is determined, now using unique runs from the dataset dataframe. Minor formatting and comment improvements were made in process_mth5.py.
Introduces tests/synthetic/test_fourier_coefficients_discrete.py with comprehensive discrete tests for the Fourier Coefficients workflow, including file validation, FC creation, storage, and processing for various synthetic MTH5 test files. Also updates test_fourier_coefficients_pytest.py to comment out test1 from the test file paths.
Expanded the synthetic Fourier Coefficient test to include detailed subtests for file validation, RunSummary, KernelDataset, config creation, FC addition, readback, and processing. Added error logging and explicit KeyError in transfer_function_kernel.py for missing channel components. Updated triage utility to also triage processed_date for more robust TF comparison.
Deleted test_fourier_coefficients_discrete.py and replaced its coverage by refactoring test_fourier_coefficients_pytest.py. The new test uses pytest parameterization to run the Fourier Coefficient workflow for each synthetic MTH5 file, improving maintainability and enabling parallel execution. All validation and workflow steps are now consolidated in a single, parameterized test.
Introduces a new pytest-based test suite for the MATLAB Z-file reader, including fixtures, parameterized tests, and integration tests. Also adds a sample .zrr test file for use in testing. Minor docstring formatting fix in triage.py.
Updated the test workflow to run pytest with automatic parallelization using pytest-xdist. Added pytest-xdist, pytest-subtests, and pytest-benchmark to the test dependencies in pyproject.toml to support parallel testing, subtests, and benchmarking.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
mt-metadatahas been updated to use Pydantic under the hood. This PR will update Aurora to operate using the updated versions of mt-metadata and mth5.