Tags: llnl/merlin
Tags
Version/1.13.0 (#580) * Release/1.13.0 (#549) * Refactor/distributed-tests (#493) * remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * Include celerymanager and update celeryadapter to check the status of celery workers. * Fixed issue where the update status was outside of if statement for checking workers * Include worker status stop and add template for merlin restart * Added comment to the CeleryManager init * Increment db_num instead of being fixed * Added other subprocess parameters and created a linking system for redis to store env dict * Implemented stopping of celery workers and restarting workers properly * Update stopped to stalled for when the worker doesn't respond to restart * Working merlin manager run but start and stop not working properly * Made fix for subprocess to start new shell and fixed manager start and stop * Added comments and update changelog * Include style fixes * Fix style for black * Revert launch_job script that was edited when doing automated lint * Move importing of CONFIG to be within redis_connection due to error of config not being created yet * Added space to fix style * Revert launch_jobs.py: * Update import of all merlin.config to be in the function * suggested changes plus beginning work on monitor/manager collab * move managers to their own folder and fix ssl problems * final PR touch ups * Fix lint style changes * Fixed issue with context manager * Reset file that was incorrect changed * Check for ssl cert before applying to Redis connection * Comment out Active tests for celerymanager * split up create_server_config and write tests for it * add tests for config_merlin_server function * Fix lint issue with unused import after commenting out Active celery tests * Fixed style for import * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * Fixed kwargs being modified when making a copy for saving to redis worker args. * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * Added password check and omit if a password doesn't exist * update CHANGELOG * change testing log level to debug * add debug statement for redis_connection * change debug log to info so github ci will display it * attempt to fix password missing from Namespace error * run checks for all necessary configurations * convert stop-workers tests to pytest format * update github wf and comment out stop-workers tests in definitions.py * add missing key to GH wf file * fix invalid syntax in definitions.py * comment out stop_workers tests * playing with new caches for workflow CI * fix yaml syntax error * fix typo for getting runner os * fix test and add python version to CI cache * add in common-setup step again with caches this time * run fix-style * update CHANGELOG * fix remaining style issues * run without caches to compare execution time of test suite * allow redis config to not use ssl * remove stop-workers and query-workers tests from definitions.py * create helper_funcs file with common testing functions * move query-workers to pytest and add base class w/ stop-workers tests * update CHANGELOG * final changes for the stop-workers & query-workers tests * run fix-style * move stop and query workers tests to the same file * run fix-style * go back to original cache setup * try new cache for singularity install * fix syntax issue in github workflow * attempt to fix singularity cache * remove ls statement that breaks workflow * revert back to no common setup * remove unnecessary dependency * update github actions versions to use latest * update action versions that didn't save * run fix-style * move distributed test suite actions back to v2 * add 'merlin run' tests and port existing ones to pytest * update CHANGELOG * add aliased fixture types for typehinting * add tests for the purge command * update CHANGELOG * update run command tests to use conditions when appropriate * start work on adding workflow tests * create function and class scoped config fixtures * add Tuple fixture type * get e2e test of feature_demo workflow running * add check for proper variable substitution in e2e test * generalize functionality to run workflows * add create_testing_dir fixture * port chord error workflow to pytest * create dataclasses to house common fixtures and reduce fixture import requirements * fix lint issues * remove hard requirement of Annotated type for python 3.7 and 3.8 * remove distributed test CI and add unit test CI * fix typo in fixture_types and fix lint issues * run fix-style * add check for python2 before adding that condition check * convert local run test to use StepFinishedFilesCount condition * update CHANGELOG.md * fix problem created by merge conflict when mergin develop * remove manager functionality from this PR * update README for test suite * change SIGTERM to SIGKILL * update Makefile to include new changes to test suite --------- Co-authored-by: Ryan Lee <lee1098@llnl.gov> Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> * Drop Python 3.7 and Add Python 3.12 & 3.13 (#495) * drop support for py 3.7, add support for py 3.12 and 3.13 * fix docs build issue * remove py 3.7 and add py 3.12/3.13 to integration tests * update Makefile to use target_version variable and update spellbook requirements in examples * Requirements fixes (#501) * add fix for broken deepdiff dependency in py38 and remove py37 specific requirements that were missed * remove try/except imports that are no longer necessary * update CHANGELOG * docs/api_docs (#496) * add/update docstrings for top-level files and enable API docs * remove Generator type hint * fix invalid escape sequences * add docstrings for the common/ directory * fix styling and add in cross-references to portions of Merlin codebase * give code blocks types for formatting * add api docs for half of the study directory * add API docs for study.py * ignore the data path for API docs * finish API docs for study directory * finish api docs for spec/ directory * update CHANGELOG * final cleanup of API docs for common, spec, and study folders * finish API docs for examples folder * began work on API docs for config folder * write API docs for server utils * finish api docs for the server directory * finish API docs for config directory * final cleanup * fix doc build issues * add section explaining API docs to the docs' README file * run fix-style * update readthedocs to build with latest python * fix too few arguments to generator issue * rename in MerlinStepRecord so that it's hidden again * add most of Charles' suggestions * remove unused openfilelist.py, opennpylib.py, and merlin_templates.py files, and add remainder of Charles suggestions * run fix-style * rename load_default_user_names to set_username_and_vhost * Update README.md to remove lgtm banner (#488) * Update README.md to remove lgtm banner * Update CHANGELOG.md * Delete .lgtm.yml * move changelog update to the unreleased section * remove lgtm.yml --------- Co-authored-by: Brian Gunnarson <brianfunnarson14@gmail.com> * Refactor/simplified-ci (#504) * split python and singularity setup into individual actions * add shell to new actions * try to fix shell issue * remove install-deps * reorder cache step and check pip version in the output * update CHANGELOG and make path to definitions.py relative * fix style issues * move cache check to setup-python action * fix an issue with docs build on python 3.9 (#503) * fix an issue with docs build on python 3.9 * adding ChatGPT suggestion to use get-pip.py script * add python version check to reinstall pip CI step * add reinstall pip step to all jobs that need it * save get-pip.py to tmp folder instead of directly to the repo * update CHANGELOG * fix .wci.yml links (#505) * Refactor/config (#498) * update config to use launchit as default and add ability to update from cli * add tests for config broker/backend and update github action * update CHANGELOG and run fix-style * update docs for the config broker/backend update * add new debug step to local-test-suite * Refactor/config-defaults (#497) * update config to use launchit as default and add ability to update from cli * add tests for config broker/backend and update github action * update CHANGELOG and run fix-style * update docs for the config broker/backend update * add new debug step to local-test-suite * remove debug step from github ci * add user permission check to local test suite * create a CI workflow for push and pull_request_target so secrets can be accessed * add link to article where I found this solution * add a merlin info test so we can see what server CI is connecting to * add unit tests for new config commands * add MerlinConfigManager class to replace config-related functions * add unit tests for MerlinConfigManager and update integration tests * comment out the pr-target workflow since it's not working * run fix-style * rewrite tests for find_config_file * remove uses of CONFIGFILE_DIR and chdir from configfile unit tests * run fix-style * re-order the find_config_file function to be local, config_path, merlin home * remove pr-target workflow, add missing docstrings, and run fix-style * add create and use commands, update MerlinConfigManager as necessary * update test suite for config command * update all documentation where 'merlin config' is mentioned * change merlin config to merlin config create * update CHANGELOG * run fix-style * update GitHub workflow to use 'merlin config create' * try to fix f-string issue * add Charles' suggestions * fix broken tests and security issue with new log statement * remove app.yaml file that was accidentally pushed * fix config file validation as Charles suggested * Refactor/server-config (#506) * have server init create an app.yaml in merlin_server folder * add str and repr methods to server config classes and fix bug with server start * update docs for new server refactor * update CHANGELOG * run fix-style * change print to log statement * bugfix/local-config (#507) * add local mode config initialization * have server init create an app.yaml in merlin_server folder * add str and repr methods to server config classes and fix bug with server start * update docs for new server refactor * update CHANGELOG * run fix-style * add unit tests for new functions in configfile.py * run fix-style and update CHANGELOG * change defaults to not use getpass * run fix-style * add changes that Charles suggested * Update copyright (#510) * add COPYRIGHT file, update LICENSE, and change copyright header in all files * update Makefile commands related to copyright/copyright headers * Update CHANGELOG * add a check for the copyright header in 'make check-style' * update CHANGELOG * Potential fix for code scanning alert no. 23: Workflow does not contain permissions Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * codeql-fixes (#513) * fix logging security vulnerabilities * update CHANGELOG * refactor/main-module (#515) * move each command's entry point logic to their own files * add missing docstrings * add copyright to the top of all new files * fix lint issues * rename backend utils test file * begin work on adding unit tests to the new cli folder * add common create_parser fixture * add tests for example and info * fix docstrings and move conftest file up a directory * add unit tests for 3 more commands * add tests for queue-info, restart, and run-workers * add tests for the remaining commands * remove parser fixtures when only one test uses it * run fix-style * add the no_restart functionality that got wiped from the merge * update CHANGELOG * docs/feature-demo-example (#519) * add a page for the feature_demo example to the docs * update CHANGELOG * feature/tests-spec-folder (#521) * add tests for expansion.py and fix bug * add docstrings and remove comments * add tests for override.py * added merlin and user attributes to init * add initial tests for MerlinSpec * finish writing unit tests for MerlinSpec * run fix-style * update CHANGELOG * run fix-style * Feature/allow-none-in-batch (#552) * allow None and empty strings to be in batch block * update CHANGELOG * Bugfix/abspath-required (#550) * make broker and backend use same function for reading pass file * fix tests * update changelog * run fix-style * change test log level to debug so I can see more output on gh ci * standardize password resolution across broker and backend funcs * fix tests * run fix-style * fix test and update another test * add check for local mode in celery.py file * fix style * Potential fix for code scanning alert no. 37: Clear-text logging of sensitive information Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Bugfix/caps-required (#551) * normalize for uppercase in detailed-status filters * update CHANGELOG * run fix-style * update version to 1.13.0 --------- Co-authored-by: Ryan Lee <lee1098@llnl.gov> Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: Wout De Nolf <wout.de_nolf@esrf.eu> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * bugfix/retry-on-store-error (#577) * add in BackendStoreError to be auto-retried * run fix style and update CHANGELOG * fix another styling issue * Feature/update-celery-config (#578) * update Celery settings to be more resilient * update CHANGELOG * Bugfix/workspace-ordering (#579) * sort potential studies alphabetically * update CHANGELOG * run fix-style --------- Co-authored-by: Ryan Lee <lee1098@llnl.gov> Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: Wout De Nolf <wout.de_nolf@esrf.eu> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Version/1.13.0b2 (#518) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * remove github text that was causing errors * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * docs/conversion-to-mkdocs (#460) * remove a merge conflict statement that was missed * add base requirements for mkdocs * set up configuration for API docs * start work on porting user guide to mkdocs * add custom styling and contact page * begin work on porting tutorial to mkdocs * add new examples page * move old sphinx docs to their own folder (*delete later*) * modify some admonitions to be success * modify hello examples page and port step 3 of tutorial to mkdocs * fix typo in hello example * finish porting step 4 of tutorial to mkdocs * port part 5 of the tutorial to mkdocs * copy faq and contributing from old docs * port step 6 of tutorial to mkdocs * remove unused prereq * port step 7 of tutorial to mkdocs * add more detailed instructions on contributing * move venv page into installation and add spack instructions too * add configuration docs * add content to user guide landing page * port celery page to mkdocs * rearrange configuration pages to add in merlin server configuration instructions * port command line page to mkdocs * finish new landing page * change size of merlin logo * port variables page to mkdocs * fix broken links to configuration page * port FAQ to mkdocs * fix incorrect requirement name * update CHANGELOG * attempt to get docs to build through readthedocs * port docker page to mkdocs * port contributing guide to mkdocs * add new 'running studies' page * add path changes to images * add a page on how to interpret study output * add page on the spec file * remove old sphinx docs that are no longer needed * added README to docs and updated CHANGELOG * fix copyright and hello_samples tree * rearrange images/stylesheets and statements that use them * add suggestions from Luc and Joe * add tcsh instructions for venv activation * add Charle's suggestions for the landing page * change tcsh mentions to csh * openfoam tutorial modifications (#463) * feature/revamped status (#464) * feature/new-status (#442) * add backend functionality for merlin status * add frontend functionality for merlin status * add tests for merlin status * run fix-style and remove import of deprecated function * update CHANGELOG * add more logging statements, make better use of glob * run fix-style * clean up test files a bit * fix test suite after step_name_map mod * add avg/std dev run time calculations to status * modify status tests to accommodate new avg/std dev calculations * fix linter issues * fix lint issue and add test for avg/std dev calc * feature/detailed-status (#451) * Version/1.11.0 (#449) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * add functionality for the detailed-status command * add tests for detailed-status * fix linter issues * update changelog * general cleanup and add log statements * slightly modify two tests * default status renderer now uses json status format * remove inaccurate comment * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * add Jeremy's suggestion to change vars option to output-path * remove unnecessary lines from CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * feature/queue info (#461) * remove a merge conflict statement that was missed * add queue-info functionality * add tests for queue-info * update CHANGELOG * add try/except for forceful termination of test workers * change github workflow to use py38 with black instead of py36 * run fix-style with py 3.12 and fix a typo in a test * add filetype check for dump option * add banner print statement * docs/revamped status (#462) * fix broken image link in README * add new commands to the command line page * add monitoring docs layout and complete status cmds page * fix bug with dumping queue-info to files * add docs for queue-info * add documentation for 'query-workers' * add reference to new query-workers docs and split a paragraph * fix small bug with --steps option of monitor * add documentation for monitor command * update CHANGELOG * fix dump-csv image for queue-info --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * release/1.12.0 (#465) * remove a merge conflict statement that was missed * bump version to 1.12.0 * feature/retry_priority (#468) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * add new retry priority as highest task priority * update CHANGELOG * add in MID priority * change default priority to use priority map MID value * docs/server-cross-node (#470) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * rename the merlin server config page * add instructions for running a cross-node workflow w/ containerized server * update CHANGELOG * bugfix/initial-status-issues (#471) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * fix bug with dry run status * set MANPAGER for detailed-status * fix bug with 1 sample removing the status file * add support for multiple workers on one step in status files * update test suite to accommodate changes to workers in status files * add catch and potential fix for JSONDecodeError * fix docstring of a test * update CHANGELOG.md * run fix style and add Luc's suggestions * run fix-style with python 3.12 * added additional check for status file while condensing * add try/except to catch an error for dumping statuses * release/1.12.1 (#472) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * bump version to 1.12.1 * fix a lint issue that somehow slipped through the cracks * Fix filenames for OpenFoam tutorial (#475) * bugfix/deep-merge-existing-keys (#476) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * remove DeepMergeException and add conflict_handler to dict_deep_merge * add conflict handler to dict_deep_merge * fix broken tests for detailed-status * use caplog fixture rather than IO stream * add ability to define module-specific fixtures * add tests for read/write status files and conlict handling * add caplog explanation to docstrings * update CHANGELOG * run fix-style * add pytest-mock as dependency for test suite * clean up input check in dict_deep_merge * Improved Info (#477) * Add merlin version to banner * Add python package info to and clean up 'merlin info' * Add some unit tests * Force GitHub runner checkout to grab the whole history, fixing CHANGELOG test bug * Update CHANGELOG to show bugfix to CHANGELOG test * Target is in source's history (#478) * New github action test to make sure target has been merged into source * Fix link to merlin banner image (#479) * bugfix/status_nested_workspace (#480) * remove a merge conflict statement that was missed * have status ignore nested workspaces and modify merge rules * update CHANGELOG * fixed issue with escape sequences in ascii art * apply Luc's suggestion * add setuptools as a requirement since python 3.12 doesn't have it natively * modify unit tests for status to use pytest rather than unittest * update CHANGELOG * add fixtures for status testing and add nested workflow test * update CHANGELOG * bugfix/celery-chord-error (#481) * remove a merge conflict statement that was missed * add celery results backend patch to stop ChordErrors * add MERLIN_RAISE_ERROR return code * add tests to ensure chord error isn't raised * add RAISE_ERROR to docs * update CHANGELOG * fix lint issues * up the sleep time on the chord error test * add new steps to the chord err test spec * add tree statement to the new test for debugging * upping sleep time to see if that fixes github action for python 3.7 * change sleep time for new test based on python version * run fix style * remove specific sleep time for diff python versions * release/1.12.2b1 (#482) * remove a merge conflict statement that was missed * bump version to 1.12.2b1 * bugfix/flux-nodes (#484) * remove a merge conflict statement that was missed * fix flux node allocation issue * allow for vars to be used with nodes settings of workers/batch * add tests for var usage with nodes * update CHANGELOG * run fix-style * bugfix/flux-nodes-prior-versions (#487) * add a version check for flux when getting node count * update CHANGELOG * add major version check for flux * Change Task ID to directory path (#486) * Modifying task id to include directory * Adding Several New Unit Tests (#490) * remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * split up create_server_config and write tests for it * add tests for config_merlin_server function * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * update CHANGELOG * add Ryan and Joe's suggestions * update tests to use newly named functions * fix linter issue * release/1.12.2 (#491) * update version to 1.12.2 * fix style issue * Refactor/distributed-tests (#493) * remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * Include celerymanager and update celeryadapter to check the status of celery workers. * Fixed issue where the update status was outside of if statement for checking workers * Include worker status stop and add template for merlin restart * Added comment to the CeleryManager init * Increment db_num instead of being fixed * Added other subprocess parameters and created a linking system for redis to store env dict * Implemented stopping of celery workers and restarting workers properly * Update stopped to stalled for when the worker doesn't respond to restart * Working merlin manager run but start and stop not working properly * Made fix for subprocess to start new shell and fixed manager start and stop * Added comments and update changelog * Include style fixes * Fix style for black * Revert launch_job script that was edited when doing automated lint * Move importing of CONFIG to be within redis_connection due to error of config not being created yet * Added space to fix style * Revert launch_jobs.py: * Update import of all merlin.config to be in the function * suggested changes plus beginning work on monitor/manager collab * move managers to their own folder and fix ssl problems * final PR touch ups * Fix lint style changes * Fixed issue with context manager * Reset file that was incorrect changed * Check for ssl cert before applying to Redis connection * Comment out Active tests for celerymanager * split up create_server_config and write tests for it * add tests for config_merlin_server function * Fix lint issue with unused import after commenting out Active celery tests * Fixed style for import * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * Fixed kwargs being modified when making a copy for saving to redis worker args. * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * Added password check and omit if a password doesn't exist * update CHANGELOG * change testing log level to debug * add debug statement for redis_connection * change debug log to info so github ci will display it * attempt to fix password missing from Namespace error * run checks for all necessary configurations * convert stop-workers tests to pytest format * update github wf and comment out stop-workers tests in definitions.py * add missing key to GH wf file * fix invalid syntax in definitions.py * comment out stop_workers tests * playing with new caches for workflow CI * fix yaml syntax error * fix typo for getting runner os * fix test and add python version to CI cache * add in common-setup step again with caches this time * run fix-style * update CHANGELOG * fix remaining style issues * run without caches to compare execution time of test suite * allow redis config to not use ssl * remove stop-workers and query-workers tests from definitions.py * create helper_funcs file with common testing functions * move query-workers to pytest and add base class w/ stop-workers tests * update CHANGELOG * final changes for the stop-workers & query-workers tests * run fix-style * move stop and query workers tests to the same file * run fix-style * go back to original cache setup * try new cache for singularity install * fix syntax issue in github workflow * attempt to fix singularity cache * remove ls statement that breaks workflow * revert back to no common setup * remove unnecessary dependency * update github actions versions to use latest * update action versions that didn't save * run fix-style * move distributed test suite actions back to v2 * add 'merlin run' tests and port existing ones to pytest * update CHANGELOG * add aliased fixture types for typehinting * add tests for the purge command * update CHANGELOG * update run command tests to use conditions when appropriate * start work on adding workflow tests * create function and class scoped config fixtures * add Tuple fixture type * get e2e test of feature_demo workflow running * add check for proper variable substitution in e2e test * generalize functionality to run workflows * add create_testing_dir fixture * port chord error workflow to pytest * create dataclasses to house common fixtures and reduce fixture import requirements * fix lint issues * remove hard requirement of Annotated type for python 3.7 and 3.8 * remove distributed test CI and add unit test CI * fix typo in fixture_types and fix lint issues * run fix-style * add check for python2 before adding that condition check * convert local run test to use StepFinishedFilesCount condition * update CHANGELOG.md * fix problem created by merge conflict when mergin develop * remove manager functionality from this PR * update README for test suite * change SIGTERM to SIGKILL * update Makefile to include new changes to test suite --------- Co-authored-by: Ryan Lee <lee1098@llnl.gov> Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> * Drop Python 3.7 and Add Python 3.12 & 3.13 (#495) * drop support for py 3.7, add support for py 3.12 and 3.13 * fix docs build issue * remove py 3.7 and add py 3.12/3.13 to integration tests * update Makefile to use target_version variable and update spellbook requirements in examples * Requirements fixes (#501) * add fix for broken deepdiff dependency in py38 and remove py37 specific requirements that were missed * remove try/except imports that are no longer necessary * update CHANGELOG * docs/api_docs (#496) * add/update docstrings for top-level files and enable API docs * remove Generator type hint * fix invalid escape sequences * add docstrings for the common/ directory * fix styling and add in cross-references to portions of Merlin codebase * give code blocks types for formatting * add api docs for half of the study directory * add API docs for study.py * ignore the data path for API docs * finish API docs for study directory * finish api docs for spec/ directory * update CHANGELOG * final cleanup of API docs for common, spec, and study folders * finish API docs for examples folder * began work on API docs for config folder * write API docs for server utils * finish api docs for the server directory * finish API docs for config directory * final cleanup * fix doc build issues * add section explaining API docs to the docs' README file * run fix-style * update readthedocs to build with latest python * fix too few arguments to generator issue * rename in MerlinStepRecord so that it's hidden again * add most of Charles' suggestions * remove unused openfilelist.py, opennpylib.py, and merlin_templates.py files, and add remainder of Charles suggestions * run fix-style * rename load_default_user_names to set_username_and_vhost * Update README.md to remove lgtm banner (#488) * Update README.md to remove lgtm banner * Update CHANGELOG.md * Delete .lgtm.yml * move changelog update to the unreleased section * remove lgtm.yml --------- Co-authored-by: Brian Gunnarson <brianfunnarson14@gmail.com> * Refactor/simplified-ci (#504) * split python and singularity setup into individual actions * add shell to new actions * try to fix shell issue * remove install-deps * reorder cache step and check pip version in the output * update CHANGELOG and make path to definitions.py relative * fix style issues * move cache check to setup-python action * fix an issue with docs build on python 3.9 (#503) * fix an issue with docs build on python 3.9 * adding ChatGPT suggestion to use get-pip.py script * add python version check to reinstall pip CI step * add reinstall pip step to all jobs that need it * save get-pip.py to tmp folder instead of directly to the repo * update CHANGELOG * fix .wci.yml links (#505) * Refactor/config (#498) * update config to use launchit as default and add ability to update from cli * add tests for config broker/backend and update github action * update CHANGELOG and run fix-style * update docs for the config broker/backend update * add new debug step to local-test-suite * Refactor/config-defaults (#497) * update config to use launchit as default and add ability to update from cli * add tests for config broker/backend and update github action * update CHANGELOG and run fix-style * update docs for the config broker/backend update * add new debug step to local-test-suite * remove debug step from github ci * add user permission check to local test suite * create a CI workflow for push and pull_request_target so secrets can be accessed * add link to article where I found this solution * add a merlin info test so we can see what server CI is connecting to * add unit tests for new config commands * add MerlinConfigManager class to replace config-related functions * add unit tests for MerlinConfigManager and update integration tests * comment out the pr-target workflow since it's not working * run fix-style * rewrite tests for find_config_file * remove uses of CONFIGFILE_DIR and chdir from configfile unit tests * run fix-style * re-order the find_config_file function to be local, config_path, merlin home * remove pr-target workflow, add missing docstrings, and run fix-style * add create and use commands, update MerlinConfigManager as necessary * update test suite for config command * update all documentation where 'merlin config' is mentioned * change merlin config to merlin config create * update CHANGELOG * run fix-style * update GitHub workflow to use 'merlin config create' * try to fix f-string issue * add Charles' suggestions * fix broken tests and security issue with new log statement * remove app.yaml file that was accidentally pushed * fix config file validation as Charles suggested * Refactor/server-config (#506) * have server init create an app.yaml in merlin_server folder * add str and repr methods to server config classes and fix bug with server start * update docs for new server refactor * update CHANGELOG * run fix-style * change print to log statement * bugfix/local-config (#507) * add local mode config initialization * have server init create an app.yaml in merlin_server folder * add str and repr methods to server config classes and fix bug with server start * update docs for new server refactor * update CHANGELOG * run fix-style * add unit tests for new functions in configfile.py * run fix-style and update CHANGELOG * change defaults to not use getpass * run fix-style * add changes that Charles suggested * feature/monitor-auto-restart (#494) * add classes for backends to help abstract them * establish templates and basic functionality for database interactions * flush out creation and retrieval of studies and runs * finish study/run storage, retrieval, and deletion to db functionality * add StudyNotFoundError and RunNotFoundError * add ability to dump run metadata to merlin_info directory * add new Celery signature to mark a run as complete * create merlin database commands * integrate database changes with monitor command * run fix-style and fix lint issues * update CHANGELOG and run fix-style with python 12 * last run of style fixes before push * run fix style with python 3.13 * create foundation for monitor auto-restart test * move get_backend_password import inside RedisBackend class * fix lint issue and comment out monitor test for now * fix redis connection so it works for ssl and non-ssl connections * add test for workflow auto-restart functionality * run linter * clean up print for DatabaseStudy and add -k option to delete all-studies * add docs for new monitor and database command * run fix-style * update built-in celery settings and retry exceptions to better handle timeouts * add new Monitor classes to handle the monitor command * update CHANGELOG and fix monitor test * fix lint issues * add worker entries and interactions with the database * add ABC class for database entities * add multi-run and multi-study queries, queries by study id, and queries by run workspace * add ability to delete multiple runs, studies, and workers. Also add ability to delete study by id and run by workspace * update docstrings for db and backend files * run fix-style * add repr method to HasRegex to make parsing GitHub CI easier * add ability to delete everything from db and retrieve connection string from MerlinDatabase * rename DatabaseRun and DatabaseStudy to more appropriate RunEntity and StudyEntity * split WorkerModel into LogicalWorkerModel and PhysicalWorkerModel * split redis backend class up into store classes for different entities. Removes db worker interaction temporarily * add logical worker functionality to database * add CLI functionality for physical workers and clean up MerlinDatabase * add physical worker db entry creation on celery worker startup * comment out code that's breaking worker launch * fix bug with get_all_runs method not existing anymore * run fix-style * fix bug with load/delete of physical workers and add functionality to set worker status to stopped * fix issue where ids were passed to wait_for_workers instead of names * move stop-worker celery functionality to a signal handler * add Logical Worker with ID 7620359f-6deb-0056-8956-39495dba8e59 ------------------------------------------------ Name: worker2 Runs: ['2e12e7cd-e7b9-468b-92c7-1ba1dffba852'] Queues: ['[merlin]_sim_queue', '[merlin]_seq_queue'] Physical Workers: ['dc863387-6b54-4a93-8a27-507bf0fd789e', '41a28c9e-f58d-49da-87bf-cea1569398b9'] Additional Data: {} Physical Worker with ID 41a28c9e-f58d-49da-87bf-cea1569398b9 ------------------------------------------------ Name: celery@worker2.%rzadams1011 Logical Worker ID: 7620359f-6deb-0056-8956-39495dba8e59 Launch Command: None Args: {} Process ID: None Status: WorkerStatus.RUNNING Last Heartbeat: 2025-04-14 12:40:05.786023 Last Spinup: 2025-04-14 12:40:05.786028 Host: rzadams1011 Restart Count: 0.0 Additional Data: {} Physical Worker with ID dc863387-6b54-4a93-8a27-507bf0fd789e ------------------------------------------------ Name: celery@worker2.%rzadams1010 Logical Worker ID: 7620359f-6deb-0056-8956-39495dba8e59 Launch Command: None Args: {} Process ID: None Status: WorkerStatus.RUNNING Last Heartbeat: 2025-04-14 12:40:05.027347 Last Spinup: 2025-04-14 12:40:05.027353 Host: rzadams1010 Restart Count: 0.0 Additional Data: {} Run with ID 2e12e7cd-e7b9-468b-92c7-1ba1dffba852 ------------------------------------------------ Workspace: /usr/WS1/gunny/scalability_testing/studies/long_running_wf_20250414-124613 Study ID: 64def1c2-8ee8-40a4-9f74-36d5b4872b9e Queues: ['[merlin]_seq_queue', '[merlin]_sim_queue'] Workers: ['7620359f-6deb-0056-8956-39495dba8e59'] Parent: None Child: None Run Complete: False Additional Data: {} Study with ID 64def1c2-8ee8-40a4-9f74-36d5b4872b9e ------------------------------------------------ Name: long_running_wf Runs: ['2e12e7cd-e7b9-468b-92c7-1ba1dffba852'] Additional Data: {} command * factor out common code from entity classes into mixin classes * run fix-style * remove process kill from celery shutdown * write tests for high-level modules of backends folder * add tests for the RedisBackend class * write unit tests for redis_logical_worker_store.py * add unit tests for RedisPhysicalWorkerStore * add tests for RedisRunStore * move db entities to their own folder * update version to 1.13.0a1 * add init.py file to backends/redis folder * change the monitor test so that it has more samples and purges tasks rather than stopping workers * run fix-style * fix error from merging develop * fix import issue with enums * condense store logic into base class and mixin class * add tests for new combined logic, remove tests for separated logic * run fix-style * fix imports and string representations of entities * update logical worker queue entry to be a set instead of list * make worker queues a set and fix str representations * update docs for database command * run fix style * change some info logs to debug * fix the majority of the docs warnings * fix docs warnings and run fix style * finish the database command docs * fix log messages and bug with deleting physical worker entry * add log files per Charles' suggestion * refactor db entities to share additional common logic * fix bug when deleting runs by workspace after workspace has been removed * add unit tests and some integration tests for data_models and db_commands * add unit tests for entity and mixin classes * refactor MerlinDatabase to use newly added entity managers * move mixins folder into entities folder * run fix-style * move mixin tests and remove todos from physical worker test file * add tests for the merlin_db module * add tests for entity managers * run fix-style * update CHANGELOG * fix unit tests for python 3.8 * fix typo * log warning if workspace doesn't exist rather than error * remove a function that was added by accident in merge from develop * fix import issue * reorganize redis_utils as they're no longer needed * move serialize and deserialize to common utils * add base store and move save, retrieve, delete methods up to ResultsBackend * initial work on adding sqlite support for local runs * finish implementing sqlite stores * update docstrings for the backend package * move existing tests of redis/results backend after refactor * add unit tests for sqlite modules and move around fixtures * add tests for StoreBase * fix unit tests for configfile * remove duplicate fixture * add some docs about sqlite and move database docs out of monitoring section * first attempt at fixing port issue * fix lint issues * second attempt at fixing the tests and style issues * implement Charles comments * update CHANGELOG * Update copyright (#510) * add COPYRIGHT file, update LICENSE, and change copyright header in all files * update Makefile commands related to copyright/copyright headers * Update CHANGELOG * add a check for the copyright header in 'make check-style' * update CHANGELOG * release/1.13.0b1 (#511) * Potential fix for code scanning alert no. 23: Workflow does not contain permissions Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * codeql-fixes (#513) * fix logging security vulnerabilities * update CHANGELOG * feature/disable-auto-restart (#514) * add no-restart option * run fix-style * update docstrings * add documentation about the no-restart option * update CHANGELOG * add Luc's suggestions * refactor/main-module (#515) * move each command's entry point logic to their own files * add missing docstrings * add copyright to the top of all new files * fix lint issues * rename backend utils test file * begin work on adding unit tests to the new cli folder * add common create_parser fixture * add tests for example and info * fix docstrings and move conftest file up a directory * add unit tests for 3 more commands * add tests for queue-info, restart, and run-workers * add tests for the remaining commands * remove parser fixtures when only one test uses it * run fix-style * add the no_restart functionality that got wiped from the merge * update CHANGELOG * feature/monitor-tests (#516) * split while loop inside monitor into smaller pieces for testing * add tests for task_server_monitor and celery_monitor * fix typo in list comprehension that broke a test * add tests for the Monitor class and MonitorFactory * update CHANGELOG * run fix-style and add missing docstrings/copyright headers * rename test_monitor to test_monitor_entry_point * release/1.13.0b2 (#517) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Jane Herriman <xorJane@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: Ryan Lee <lee1098@llnl.gov> Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> Co-authored-by: Wout De Nolf <wout.de_nolf@esrf.eu> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Version/1.13.0b1 (#512) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * remove github text that was causing errors * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * docs/conversion-to-mkdocs (#460) * remove a merge conflict statement that was missed * add base requirements for mkdocs * set up configuration for API docs * start work on porting user guide to mkdocs * add custom styling and contact page * begin work on porting tutorial to mkdocs * add new examples page * move old sphinx docs to their own folder (*delete later*) * modify some admonitions to be success * modify hello examples page and port step 3 of tutorial to mkdocs * fix typo in hello example * finish porting step 4 of tutorial to mkdocs * port part 5 of the tutorial to mkdocs * copy faq and contributing from old docs * port step 6 of tutorial to mkdocs * remove unused prereq * port step 7 of tutorial to mkdocs * add more detailed instructions on contributing * move venv page into installation and add spack instructions too * add configuration docs * add content to user guide landing page * port celery page to mkdocs * rearrange configuration pages to add in merlin server configuration instructions * port command line page to mkdocs * finish new landing page * change size of merlin logo * port variables page to mkdocs * fix broken links to configuration page * port FAQ to mkdocs * fix incorrect requirement name * update CHANGELOG * attempt to get docs to build through readthedocs * port docker page to mkdocs * port contributing guide to mkdocs * add new 'running studies' page * add path changes to images * add a page on how to interpret study output * add page on the spec file * remove old sphinx docs that are no longer needed * added README to docs and updated CHANGELOG * fix copyright and hello_samples tree * rearrange images/stylesheets and statements that use them * add suggestions from Luc and Joe * add tcsh instructions for venv activation * add Charle's suggestions for the landing page * change tcsh mentions to csh * openfoam tutorial modifications (#463) * feature/revamped status (#464) * feature/new-status (#442) * add backend functionality for merlin status * add frontend functionality for merlin status * add tests for merlin status * run fix-style and remove import of deprecated function * update CHANGELOG * add more logging statements, make better use of glob * run fix-style * clean up test files a bit * fix test suite after step_name_map mod * add avg/std dev run time calculations to status * modify status tests to accommodate new avg/std dev calculations * fix linter issues * fix lint issue and add test for avg/std dev calc * feature/detailed-status (#451) * Version/1.11.0 (#449) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * add functionality for the detailed-status command * add tests for detailed-status * fix linter issues * update changelog * general cleanup and add log statements * slightly modify two tests * default status renderer now uses json status format * remove inaccurate comment * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * add Jeremy's suggestion to change vars option to output-path * remove unnecessary lines from CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * feature/queue info (#461) * remove a merge conflict statement that was missed * add queue-info functionality * add tests for queue-info * update CHANGELOG * add try/except for forceful termination of test workers * change github workflow to use py38 with black instead of py36 * run fix-style with py 3.12 and fix a typo in a test * add filetype check for dump option * add banner print statement * docs/revamped status (#462) * fix broken image link in README * add new commands to the command line page * add monitoring docs layout and complete status cmds page * fix bug with dumping queue-info to files * add docs for queue-info * add documentation for 'query-workers' * add reference to new query-workers docs and split a paragraph * fix small bug with --steps option of monitor * add documentation for monitor command * update CHANGELOG * fix dump-csv image for queue-info --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * release/1.12.0 (#465) * remove a merge conflict statement that was missed * bump version to 1.12.0 * feature/retry_priority (#468) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * add new retry priority as highest task priority * update CHANGELOG * add in MID priority * change default priority to use priority map MID value * docs/server-cross-node (#470) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * rename the merlin server config page * add instructions for running a cross-node workflow w/ containerized server * update CHANGELOG * bugfix/initial-status-issues (#471) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * fix bug with dry run status * set MANPAGER for detailed-status * fix bug with 1 sample removing the status file * add support for multiple workers on one step in status files * update test suite to accommodate changes to workers in status files * add catch and potential fix for JSONDecodeError * fix docstring of a test * update CHANGELOG.md * run fix style and add Luc's suggestions * run fix-style with python 3.12 * added additional check for status file while condensing * add try/except to catch an error for dumping statuses * release/1.12.1 (#472) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * bump version to 1.12.1 * fix a lint issue that somehow slipped through the cracks * Fix filenames for OpenFoam tutorial (#475) * bugfix/deep-merge-existing-keys (#476) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * remove DeepMergeException and add conflict_handler to dict_deep_merge * add conflict handler to dict_deep_merge * fix broken tests for detailed-status * use caplog fixture rather than IO stream * add ability to define module-specific fixtures * add tests for read/write status files and conlict handling * add caplog explanation to docstrings * update CHANGELOG * run fix-style * add pytest-mock as dependency for test suite * clean up input check in dict_deep_merge * Improved Info (#477) * Add merlin version to banner * Add python package info to and clean up 'merlin info' * Add some unit tests * Force GitHub runner checkout to grab the whole history, fixing CHANGELOG test bug * Update CHANGELOG to show bugfix to CHANGELOG test * Target is in source's history (#478) * New github action test to make sure target has been merged into source * Fix link to merlin banner image (#479) * bugfix/status_nested_workspace (#480) * remove a merge conflict statement that was missed * have status ignore nested workspaces and modify merge rules * update CHANGELOG * fixed issue with escape sequences in ascii art * apply Luc's suggestion * add setuptools as a requirement since python 3.12 doesn't have it natively * modify unit tests for status to use pytest rather than unittest * update CHANGELOG * add fixtures for status testing and add nested workflow test * update CHANGELOG * bugfix/celery-chord-error (#481) * remove a merge conflict statement that was missed * add celery results backend patch to stop ChordErrors * add MERLIN_RAISE_ERROR return code * add tests to ensure chord error isn't raised * add RAISE_ERROR to docs * update CHANGELOG * fix lint issues * up the sleep time on the chord error test * add new steps to the chord err test spec * add tree statement to the new test for debugging * upping sleep time to see if that fixes github action for python 3.7 * change sleep time for new test based on python version * run fix style * remove specific sleep time for diff python versions * release/1.12.2b1 (#482) * remove a merge conflict statement that was missed * bump version to 1.12.2b1 * bugfix/flux-nodes (#484) * remove a merge conflict statement that was missed * fix flux node allocation issue * allow for vars to be used with nodes settings of workers/batch * add tests for var usage with nodes * update CHANGELOG * run fix-style * bugfix/flux-nodes-prior-versions (#487) * add a version check for flux when getting node count * update CHANGELOG * add major version check for flux * Change Task ID to directory path (#486) * Modifying task id to include directory * Adding Several New Unit Tests (#490) * remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * split up create_server_config and write tests for it * add tests for config_merlin_server function * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * update CHANGELOG * add Ryan and Joe's suggestions * update tests to use newly named functions * fix linter issue * release/1.12.2 (#491) * update version to 1.12.2 * fix style issue * Refactor/distributed-tests (#493) * remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * Include celerymanager and update celeryadapter to check the status of celery workers. * Fixed issue where the update status was outside of if statement for checking workers * Include worker status stop and add template for merlin restart * Added comment to the CeleryManager init * Increment db_num instead of being fixed * Added other subprocess parameters and created a linking system for redis to store env dict * Implemented stopping of celery workers and restarting workers properly * Update stopped to stalled for when the worker doesn't respond to restart * Working merlin manager run but start and stop not working properly * Made fix for subprocess to start new shell and fixed manager start and stop * Added comments and update changelog * Include style fixes * Fix style for black * Revert launch_job script that was edited when doing automated lint * Move importing of CONFIG to be within redis_connection due to error of config not being created yet * Added space to fix style * Revert launch_jobs.py: * Update import of all merlin.config to be in the function * suggested changes plus beginning work on monitor/manager collab * move managers to their own folder and fix ssl problems * final PR touch ups * Fix lint style changes * Fixed issue with context manager * Reset file that was incorrect changed * Check for ssl cert before applying to Redis connection * Comment out Active tests for celerymanager * split up create_server_config and write tests for it * add tests for config_merlin_server function * Fix lint issue with unused import after commenting out Active celery tests * Fixed style for import * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * Fixed kwargs being modified when making a copy for saving to redis worker args. * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * Added password check and omit if a password doesn't exist * update CHANGELOG * change testing log level to debug * add debug statement for redis_connection * change debug log to info so github ci will display it * attempt to fix password missing from Namespace error * run checks for all necessary configurations * convert stop-workers tests to pytest format * update github wf and comment out stop-workers tests in definitions.py * add missing key to GH wf file * fix invalid syntax in definitions.py * comment out stop_workers tests * playing with new caches for workflow CI * fix yaml syntax error * fix typo for getting runner os * fix test and add python version to CI cache * add in common-setup step again with caches this time * run fix-style * update CHANGELOG * fix remaining style issues * run without caches to compare execution time of test suite * allow redis config to not use ssl * remove stop-workers and query-workers tests from definitions.py * create helper_funcs file with common testing functions * move query-workers to pytest and add base class w/ stop-workers tests * update CHANGELOG * final changes for the stop-workers & query-workers tests * run fix-style * move stop and query workers tests to the same file * run fix-style * go back to original cache setup * try new cache for singularity install * fix syntax issue in github workflow * attempt to fix singularity cache * remove ls statement that breaks workflow * revert back to no common setup * remove unnecessary dependency * update github actions versions to use latest * update action versions that didn't save * run fix-style * move distributed test suite actions back to v2 * add 'merlin run' tests and port existing ones to pytest * update CHANGELOG * add aliased fixture types for typehinting * add tests for the purge command * update CHANGELOG * update run command tests to use conditions when appropriate * start work on adding workflow tests * create function and class scoped config fixtures * add Tuple fixture type * get e2e test of feature_demo workflow running * add check for proper variable substitution in e2e test * generalize functionality to run workflows * add create_testing_dir fixture * port chord error workflow to pytest * create dataclasses to house common fixtures and reduce fixture import requirements * fix lint issues * remove hard requirement of Annotated type for python 3.7 and 3.8 * remove distributed test CI and add unit test CI * fix typo in fixture_types and fix lint issues * run fix-style * add check for python2 before adding that condition check * convert local run test to use StepFinishedFilesCount condition * update CHANGELOG.md * fix problem created by merge conflict when mergin develop * remove manager functionality from this PR * update README for test suite * change SIGTERM to SIGKILL * update Makefile to include new changes to test suite --------- Co-authored-by: Ryan Lee <lee1098@llnl.gov> Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> * Drop Python 3.7 and Add Python 3.12 & 3.13 (#495) * drop support for py 3.7, add support for py 3.12 and 3.13 * fix docs build issue * remove py 3.7 and add py 3.12/3.13 to integration tests * update Makefile to use target_version variable and update spellbook requirements in examples * Requirements fixes (#501) * add fix for broken deepdiff dependency in py38 and remove py37 specific requirements that were missed * remove try/except imports that are no longer necessary * update CHANGELOG * docs/api_docs (#496) * add/update docstrings for top-level files and enable API docs * remove Generator type hint * fix invalid escape sequences * add docstrings for the common/ directory * fix styling and add in cross-references to portions of Merlin codebase * give code blocks types for formatting * add api docs for half of the study directory * add API docs for study.py * ignore the data path for API docs * finish API docs for study directory * finish api docs for spec/ directory * update CHANGELOG * final cleanup of API docs for common, spec, and study folders * finish API docs for examples folder * began work on API docs for config folder * write API docs for server utils * finish api docs for the server directory * finish API docs for config directory * final cleanup * fix doc build issues * add section explaining API docs to the docs' README file * run fix-style * update readthedocs to build with latest python * fix too few arguments to generator issue * rename in MerlinStepRecord so that it's hidden again * add most of Charles' suggestions * remove unused openfilelist.py, opennpylib.py, and merlin_templates.py files, and add remainder of Charles suggestions * run fix-style * rename load_default_user_names to set_username_and_vhost * Update README.md to remove lgtm banner (#488) * Update README.md to remove lgtm banner * Update CHANGELOG.md * Delete .lgtm.yml * move changelog update to the unreleased section * remove lgtm.yml --------- Co-authored-by: Brian Gunnarson <brianfunnarson14@gmail.com> * Refactor/simplified-ci (#504) * split python and singularity setup into individual actions * add shell to new actions * try to fix shell issue * remove install-deps * reorder cache step and check pip version in the output * update CHANGELOG and make path to definitions.py relative * fix style issues * move cache check to setup-python action * fix an issue with docs build on python 3.9 (#503) * fix an issue with docs build on python 3.9 * adding ChatGPT suggestion to use get-pip.py script * add python version check to reinstall pip CI step * add reinstall pip step to all jobs that need it * save get-pip.py to tmp folder instead of directly to the repo * update CHANGELOG * fix .wci.yml links (#505) * Refactor/config (#498) * update config to use launchit as default and add ability to update from cli * add tests for config broker/backend and update github action * update CHANGELOG and run fix-style * update docs for the config broker/backend update * add new debug step to local-test-suite * Refactor/config-defaults (#497) * update config to use launchit as default and add ability to update from cli * add tests for config broker/backend and update github action * update CHANGELOG and run fix-style * update docs for the config broker/backend update * add new debug step to local-test-suite * remove debug step from github ci * add user permission check to local test suite * create a CI workflow for push and pull_request_target so secrets can be accessed * add link to article where I found this solution * add a merlin info test so we can see what server CI is connecting to * add unit tests for new config commands * add MerlinConfigManager class to replace config-related functions * add unit tests for MerlinConfigManager and update integration tests * comment out the pr-target workflow since it's not working * run fix-style * rewrite tests for find_config_file * remove uses of CONFIGFILE_DIR and chdir from configfile unit tests * run fix-style * re-order the find_config_file function to be local, config_path, merlin home * remove pr-target workflow, add missing docstrings, and run fix-style * add create and use commands, update MerlinConfigManager as necessary * update test suite for config command * update all documentation where 'merlin config' is mentioned * change merlin config to merlin config create * update CHANGELOG * run fix-style * update GitHub workflow to use 'merlin config create' * try to fix f-string issue * add Charles' suggestions * fix broken tests and security issue with new log statement * remove app.yaml file that was accidentally pushed * fix config file validation as Charles suggested * Refactor/server-config (#506) * have server init create an app.yaml in merlin_server folder * add str and repr methods to server config classes and fix bug with server start * update docs for new server refactor * update CHANGELOG * run fix-style * change print to log statement * bugfix/local-config (#507) * add local mode config initialization * have server init create an app.yaml in merlin_server folder * add str and repr methods to server config classes and fix bug with server start * update docs for new server refactor * update CHANGELOG * run fix-style * add unit tests for new functions in configfile.py * run fix-style and update CHANGELOG * change defaults to not use getpass * run fix-style * add changes that Charles suggested * feature/monitor-auto-restart (#494) * add classes for backends to help abstract them * establish templates and basic functionality for database interactions * flush out creation and retrieval of studies and runs * finish study/run storage, retrieval, and deletion to db functionality * add StudyNotFoundError and RunNotFoundError * add ability to dump run metadata to merlin_info directory * add new Celery signature to mark a run as complete * create merlin database commands * integrate database changes with monitor command * run fix-style and fix lint issues * update CHANGELOG and run fix-style with python 12 * last run of style fixes before push * run fix style with python 3.13 * create foundation for monitor auto-restart test * move get_backend_password import inside RedisBackend class * fix lint issue and comment out monitor test for now * fix redis connection so it works for ssl and non-ssl connections * add test for workflow auto-restart functionality * run linter * clean up print for DatabaseStudy and add -k option to delete all-studies * add docs for new monitor and database command * run fix-style * update built-in celery settings and retry exceptions to better handle timeouts * add new Monitor classes to handle the monitor command * update CHANGELOG and fix monitor test * fix lint issues * add worker entries and interactions with the database * add ABC class for database entities * add multi-run and multi-study queries, queries by study id, and queries by run workspace * add ability to delete multiple runs, studies, and workers. Also add ability to delete study by id and run by workspace * update docstrings for db and backend files * run fix-style * add repr method to HasRegex to make parsing GitHub CI easier * add ability to delete everything from db and retrieve connection string from MerlinDatabase * rename DatabaseRun and DatabaseStudy to more appropriate RunEntity and StudyEntity * split WorkerModel into LogicalWorkerModel and PhysicalWorkerModel * split redis backend class up into store classes for different entities. Removes db worker interaction temporarily * add logical worker functionality to database * add CLI functionality for physical workers and clean up MerlinDatabase * add physical worker db entry creation on celery worker startup * comment out code that's breaking worker launch * fix bug with get_all_runs method not existing anymore * run fix-style * fix bug with load/delete of physical workers and add functionality to set worker status to stopped * fix issue where ids were passed to wait_for_workers instead of names * move stop-worker celery functionality to a signal handler * add Logical Worker with ID 7620359f-6deb-0056-8956-39495dba8e59 ------------------------------------------------ Name: worker2 Runs: ['2e12e7cd-e7b9-468b-92c7-1ba1dffba852'] Queues: ['[merlin]_sim_queue', '[merlin]_seq_queue'] Physical Workers: ['dc863387-6b54-4a93-8a27-507bf0fd789e', '41a28c9e-f58d-49da-87bf-cea1569398b9'] Additional Data: {} Physical Worker with ID 41a28c9e-f58d-49da-87bf-cea1569398b9 ------------------------------------------------ Name: celery@worker2.%rzadams1011 Logical Worker ID: 7620359f-6deb-0056-8956-39495dba8e59 Launch Command: None Args: {} Process ID: None Status: WorkerStatus.RUNNING Last Heartbeat: 2025-04-14 12:40:05.786023 Last Spinup: 2025-04-14 12:40:05.786028 Host: rzadams1011 Restart Count: 0.0 Additional Data: {} Physical Worker with ID dc863387-6b54-4a93-8a27-507bf0fd789e ------------------------------------------------ Name: celery@worker2.%rzadams1010 Logical Worker ID: 7620359f-6deb-0056-8956-39495dba8e59 Launch Command: None Args: {} Process ID: None Status: WorkerStatus.RUNNING Last Heartbeat: 2025-04-14 12:40:05.027347 Last Spinup: 2025-04-14 12:40:05.027353 Host: rzadams1010 Restart Count: 0.0 Additional Data: {} Run with ID 2e12e7cd-e7b9-468b-92c7-1ba1dffba852 ------------------------------------------------ Workspace: /usr/WS1/gunny/scalability_testing/studies/long_running_wf_20250414-124613 Study ID: 64def1c2-8ee8-40a4-9f74-36d5b4872b9e Queues: ['[merlin]_seq_queue', '[merlin]_sim_queue'] Workers: ['7620359f-6deb-0056-8956-39495dba8e59'] Parent: None Child: None Run Complete: False Additional Data: {} Study with ID 64def1c2-8ee8-40a4-9f74-36d5b4872b9e ------------------------------------------------ Name: long_running_wf Runs: ['2e12e7cd-e7b9-468b-92c7-1ba1dffba852'] Additional Data: {} command * factor out common code from entity classes into mixin classes * run fix-style * remove process kill from celery shutdown * write tests for high-level modules of backends folder * add tests for the RedisBackend class * write unit tests for redis_logical_worker_store.py * add unit tests for RedisPhysicalWorkerStore * add tests for RedisRunStore * move db entities to their own folder * update version to 1.13.0a1 * add init.py file to backends/redis folder * change the monitor test so that it has more samples and purges tasks rather than stopping workers * run fix-style * fix error from merging develop * fix import issue with enums * condense store logic into base class and mixin class * add tests for new combined logic, remove tests for separated logic * run fix-style * fix imports and string representations of entities * update logical worker queue entry to be a set instead of list * make worker queues a set and fix str representations * update docs for database command * run fix style * change some info logs to debug * fix the majority of the docs warnings * fix docs warnings and run fix style * finish the database command docs * fix log messages and bug with deleting physical worker entry * add log files per Charles' suggestion * refactor db entities to share additional common logic * fix bug when deleting runs by workspace after workspace has been removed * add unit tests and some integration tests for data_models and db_commands * add unit tests for entity and mixin classes * refactor MerlinDatabase to use newly added entity managers * move mixins folder into entities folder * run fix-style * move mixin tests and remove todos from physical worker test file * add tests for the merlin_db module * add tests for entity managers * run fix-style * update CHANGELOG * fix unit tests for python 3.8 * fix typo * log warning if workspace doesn't exist rather than error * remove a function that was added by accident in merge from develop * fix import issue * reorganize redis_utils as they're no longer needed * move serialize and deserialize to common utils * add base store and move save, retrieve, delete methods up to ResultsBackend * initial work on adding sqlite support for local runs * finish implementing sqlite stores * update docstrings for the backend package * move existing tests of redis/results backend after refactor * add unit tests for sqlite modules and move around fixtures * add tests for StoreBase * fix unit tests for configfile * remove duplicate fixture * add some docs about sqlite and move database docs out of monitoring section * first attempt at fixing port issue * fix lint issues * second attempt at fixing the tests and style issues * implement Charles comments * update CHANGELOG * Update copyright (#510) * add COPYRIGHT file, update LICENSE, and change copyright header in all files * update Makefile commands related to copyright/copyright headers * Update CHANGELOG * add a check for the copyright header in 'make check-style' * update CHANGELOG * release/1.13.0b1 (#511) * Potential fix for code scanning alert no. 23: Workflow does not contain permissions Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * codeql-fixes (#513) * fix logging security vulnerabilities * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Jane Herriman <xorJane@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: Ryan Lee <lee1098@llnl.gov> Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> Co-authored-by: Wout De Nolf <wout.de_nolf@esrf.eu> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Version/1.12.2 (#492) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * remove github text that was causing errors * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * docs/conversion-to-mkdocs (#460) * remove a merge conflict statement that was missed * add base requirements for mkdocs * set up configuration for API docs * start work on porting user guide to mkdocs * add custom styling and contact page * begin work on porting tutorial to mkdocs * add new examples page * move old sphinx docs to their own folder (*delete later*) * modify some admonitions to be success * modify hello examples page and port step 3 of tutorial to mkdocs * fix typo in hello example * finish porting step 4 of tutorial to mkdocs * port part 5 of the tutorial to mkdocs * copy faq and contributing from old docs * port step 6 of tutorial to mkdocs * remove unused prereq * port step 7 of tutorial to mkdocs * add more detailed instructions on contributing * move venv page into installation and add spack instructions too * add configuration docs * add content to user guide landing page * port celery page to mkdocs * rearrange configuration pages to add in merlin server configuration instructions * port command line page to mkdocs * finish new landing page * change size of merlin logo * port variables page to mkdocs * fix broken links to configuration page * port FAQ to mkdocs * fix incorrect requirement name * update CHANGELOG * attempt to get docs to build through readthedocs * port docker page to mkdocs * port contributing guide to mkdocs * add new 'running studies' page * add path changes to images * add a page on how to interpret study output * add page on the spec file * remove old sphinx docs that are no longer needed * added README to docs and updated CHANGELOG * fix copyright and hello_samples tree * rearrange images/stylesheets and statements that use them * add suggestions from Luc and Joe * add tcsh instructions for venv activation * add Charle's suggestions for the landing page * change tcsh mentions to csh * openfoam tutorial modifications (#463) * feature/revamped status (#464) * feature/new-status (#442) * add backend functionality for merlin status * add frontend functionality for merlin status * add tests for merlin status * run fix-style and remove import of deprecated function * update CHANGELOG * add more logging statements, make better use of glob * run fix-style * clean up test files a bit * fix test suite after step_name_map mod * add avg/std dev run time calculations to status * modify status tests to accommodate new avg/std dev calculations * fix linter issues * fix lint issue and add test for avg/std dev calc * feature/detailed-status (#451) * Version/1.11.0 (#449) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * add functionality for the detailed-status command * add tests for detailed-status * fix linter issues * update changelog * general cleanup and add log statements * slightly modify two tests * default status renderer now uses json status format * remove inaccurate comment * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * add Jeremy's suggestion to change vars option to output-path * remove unnecessary lines from CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * feature/queue info (#461) * remove a merge conflict statement that was missed * add queue-info functionality * add tests for queue-info * update CHANGELOG * add try/except for forceful termination of test workers * change github workflow to use py38 with black instead of py36 * run fix-style with py 3.12 and fix a typo in a test * add filetype check for dump option * add banner print statement * docs/revamped status (#462) * fix broken image link in README * add new commands to the command line page * add monitoring docs layout and complete status cmds page * fix bug with dumping queue-info to files * add docs for queue-info * add documentation for 'query-workers' * add reference to new query-workers docs and split a paragraph * fix small bug with --steps option of monitor * add documentation for monitor command * update CHANGELOG * fix dump-csv image for queue-info --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * release/1.12.0 (#465) * remove a merge conflict statement that was missed * bump version to 1.12.0 * feature/retry_priority (#468) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * add new retry priority as highest task priority * update CHANGELOG * add in MID priority * change default priority to use priority map MID value * docs/server-cross-node (#470) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * rename the merlin server config page * add instructions for running a cross-node workflow w/ containerized server * update CHANGELOG * bugfix/initial-status-issues (#471) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * fix bug with dry run status * set MANPAGER for detailed-status * fix bug with 1 sample removing the status file * add support for multiple workers on one step in status files * update test suite to accommodate changes to workers in status files * add catch and potential fix for JSONDecodeError * fix docstring of a test * update CHANGELOG.md * run fix style and add Luc's suggestions * run fix-style with python 3.12 * added additional check for status file while condensing * add try/except to catch an error for dumping statuses * release/1.12.1 (#472) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * bump version to 1.12.1 * fix a lint issue that somehow slipped through the cracks * Fix filenames for OpenFoam tutorial (#475) * bugfix/deep-merge-existing-keys (#476) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * remove DeepMergeException and add conflict_handler to dict_deep_merge * add conflict handler to dict_deep_merge * fix broken tests for detailed-status * use caplog fixture rather than IO stream * add ability to define module-specific fixtures * add tests for read/write status files and conlict handling * add caplog explanation to docstrings * update CHANGELOG * run fix-style * add pytest-mock as dependency for test suite * clean up input check in dict_deep_merge * Improved Info (#477) * Add merlin version to banner * Add python package info to and clean up 'merlin info' * Add some unit tests * Force GitHub runner checkout to grab the whole history, fixing CHANGELOG test bug * Update CHANGELOG to show bugfix to CHANGELOG test * Target is in source's history (#478) * New github action test to make sure target has been merged into source * Fix link to merlin banner image (#479) * bugfix/status_nested_workspace (#480) * remove a merge conflict statement that was missed * have status ignore nested workspaces and modify merge rules * update CHANGELOG * fixed issue with escape sequences in ascii art * apply Luc's suggestion * add setuptools as a requirement since python 3.12 doesn't have it natively * modify unit tests for status to use pytest rather than unittest * update CHANGELOG * add fixtures for status testing and add nested workflow test * update CHANGELOG * bugfix/celery-chord-error (#481) * remove a merge conflict statement that was missed * add celery results backend patch to stop ChordErrors * add MERLIN_RAISE_ERROR return code * add tests to ensure chord error isn't raised * add RAISE_ERROR to docs * update CHANGELOG * fix lint issues * up the sleep time on the chord error test * add new steps to the chord err test spec * add tree statement to the new test for debugging * upping sleep time to see if that fixes github action for python 3.7 * change sleep time for new test based on python version * run fix style * remove specific sleep time for diff python versions * release/1.12.2b1 (#482) * remove a merge conflict statement that was missed * bump version to 1.12.2b1 * bugfix/flux-nodes (#484) * remove a merge conflict statement that was missed * fix flux node allocation issue * allow for vars to be used with nodes settings of workers/batch * add tests for var usage with nodes * update CHANGELOG * run fix-style * bugfix/flux-nodes-prior-versions (#487) * add a version check for flux when getting node count * update CHANGELOG * add major version check for flux * Change Task ID to directory path (#486) * Modifying task id to include directory * Adding Several New Unit Tests (#490) * remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * split up create_server_config and write tests for it * add tests for config_merlin_server function * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * update CHANGELOG * add Ryan and Joe's suggestions * update tests to use newly named functions * fix linter issue * release/1.12.2 (#491) * update version to 1.12.2 * fix style issue --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Jane Herriman <xorJane@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov>
Version/1.12.2b1 (#483) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * docs/conversion-to-mkdocs (#460) * remove a merge conflict statement that was missed * add base requirements for mkdocs * set up configuration for API docs * start work on porting user guide to mkdocs * add custom styling and contact page * begin work on porting tutorial to mkdocs * add new examples page * move old sphinx docs to their own folder (*delete later*) * modify some admonitions to be success * modify hello examples page and port step 3 of tutorial to mkdocs * fix typo in hello example * finish porting step 4 of tutorial to mkdocs * port part 5 of the tutorial to mkdocs * copy faq and contributing from old docs * port step 6 of tutorial to mkdocs * remove unused prereq * port step 7 of tutorial to mkdocs * add more detailed instructions on contributing * move venv page into installation and add spack instructions too * add configuration docs * add content to user guide landing page * port celery page to mkdocs * rearrange configuration pages to add in merlin server configuration instructions * port command line page to mkdocs * finish new landing page * change size of merlin logo * port variables page to mkdocs * fix broken links to configuration page * port FAQ to mkdocs * fix incorrect requirement name * update CHANGELOG * attempt to get docs to build through readthedocs * port docker page to mkdocs * port contributing guide to mkdocs * add new 'running studies' page * add path changes to images * add a page on how to interpret study output * add page on the spec file * remove old sphinx docs that are no longer needed * added README to docs and updated CHANGELOG * fix copyright and hello_samples tree * rearrange images/stylesheets and statements that use them * add suggestions from Luc and Joe * add tcsh instructions for venv activation * add Charle's suggestions for the landing page * change tcsh mentions to csh * openfoam tutorial modifications (#463) * feature/revamped status (#464) * feature/new-status (#442) * add backend functionality for merlin status * add frontend functionality for merlin status * add tests for merlin status * run fix-style and remove import of deprecated function * update CHANGELOG * add more logging statements, make better use of glob * run fix-style * clean up test files a bit * fix test suite after step_name_map mod * add avg/std dev run time calculations to status * modify status tests to accommodate new avg/std dev calculations * fix linter issues * fix lint issue and add test for avg/std dev calc * feature/detailed-status (#451) * Version/1.11.0 (#449) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * add functionality for the detailed-status command * add tests for detailed-status * fix linter issues * update changelog * general cleanup and add log statements * slightly modify two tests * default status renderer now uses json status format * remove inaccurate comment * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * add Jeremy's suggestion to change vars option to output-path * remove unnecessary lines from CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * feature/queue info (#461) * remove a merge conflict statement that was missed * add queue-info functionality * add tests for queue-info * update CHANGELOG * add try/except for forceful termination of test workers * change github workflow to use py38 with black instead of py36 * run fix-style with py 3.12 and fix a typo in a test * add filetype check for dump option * add banner print statement * docs/revamped status (#462) * fix broken image link in README * add new commands to the command line page * add monitoring docs layout and complete status cmds page * fix bug with dumping queue-info to files * add docs for queue-info * add documentation for 'query-workers' * add reference to new query-workers docs and split a paragraph * fix small bug with --steps option of monitor * add documentation for monitor command * update CHANGELOG * fix dump-csv image for queue-info --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * release/1.12.0 (#465) * remove a merge conflict statement that was missed * bump version to 1.12.0 * feature/retry_priority (#468) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * add new retry priority as highest task priority * update CHANGELOG * add in MID priority * change default priority to use priority map MID value * docs/server-cross-node (#470) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * rename the merlin server config page * add instructions for running a cross-node workflow w/ containerized server * update CHANGELOG * bugfix/initial-status-issues (#471) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * fix bug with dry run status * set MANPAGER for detailed-status * fix bug with 1 sample removing the status file * add support for multiple workers on one step in status files * update test suite to accommodate changes to workers in status files * add catch and potential fix for JSONDecodeError * fix docstring of a test * update CHANGELOG.md * run fix style and add Luc's suggestions * run fix-style with python 3.12 * added additional check for status file while condensing * add try/except to catch an error for dumping statuses * release/1.12.1 (#472) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * bump version to 1.12.1 * fix a lint issue that somehow slipped through the cracks * Fix filenames for OpenFoam tutorial (#475) * bugfix/deep-merge-existing-keys (#476) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * remove DeepMergeException and add conflict_handler to dict_deep_merge * add conflict handler to dict_deep_merge * fix broken tests for detailed-status * use caplog fixture rather than IO stream * add ability to define module-specific fixtures * add tests for read/write status files and conlict handling * add caplog explanation to docstrings * update CHANGELOG * run fix-style * add pytest-mock as dependency for test suite * clean up input check in dict_deep_merge * Improved Info (#477) * Add merlin version to banner * Add python package info to and clean up 'merlin info' * Add some unit tests * Force GitHub runner checkout to grab the whole history, fixing CHANGELOG test bug * Update CHANGELOG to show bugfix to CHANGELOG test * Target is in source's history (#478) * New github action test to make sure target has been merged into source * Fix link to merlin banner image (#479) * bugfix/status_nested_workspace (#480) * remove a merge conflict statement that was missed * have status ignore nested workspaces and modify merge rules * update CHANGELOG * fixed issue with escape sequences in ascii art * apply Luc's suggestion * add setuptools as a requirement since python 3.12 doesn't have it natively * modify unit tests for status to use pytest rather than unittest * update CHANGELOG * add fixtures for status testing and add nested workflow test * update CHANGELOG * bugfix/celery-chord-error (#481) * remove a merge conflict statement that was missed * add celery results backend patch to stop ChordErrors * add MERLIN_RAISE_ERROR return code * add tests to ensure chord error isn't raised * add RAISE_ERROR to docs * update CHANGELOG * fix lint issues * up the sleep time on the chord error test * add new steps to the chord err test spec * add tree statement to the new test for debugging * upping sleep time to see if that fixes github action for python 3.7 * change sleep time for new test based on python version * run fix style * remove specific sleep time for diff python versions * release/1.12.2b1 (#482) * remove a merge conflict statement that was missed * bump version to 1.12.2b1 * bugfix/flux-nodes (#484) * remove a merge conflict statement that was missed * fix flux node allocation issue * allow for vars to be used with nodes settings of workers/batch * add tests for var usage with nodes * update CHANGELOG * run fix-style * bugfix/flux-nodes-prior-versions (#487) * add a version check for flux when getting node count * update CHANGELOG * add major version check for flux * Change Task ID to directory path (#486) * Modifying task id to include directory --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Jane Herriman <xorJane@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov>
Version/1.12.1 (#473) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * docs/conversion-to-mkdocs (#460) * remove a merge conflict statement that was missed * add base requirements for mkdocs * set up configuration for API docs * start work on porting user guide to mkdocs * add custom styling and contact page * begin work on porting tutorial to mkdocs * add new examples page * move old sphinx docs to their own folder (*delete later*) * modify some admonitions to be success * modify hello examples page and port step 3 of tutorial to mkdocs * fix typo in hello example * finish porting step 4 of tutorial to mkdocs * port part 5 of the tutorial to mkdocs * copy faq and contributing from old docs * port step 6 of tutorial to mkdocs * remove unused prereq * port step 7 of tutorial to mkdocs * add more detailed instructions on contributing * move venv page into installation and add spack instructions too * add configuration docs * add content to user guide landing page * port celery page to mkdocs * rearrange configuration pages to add in merlin server configuration instructions * port command line page to mkdocs * finish new landing page * change size of merlin logo * port variables page to mkdocs * fix broken links to configuration page * port FAQ to mkdocs * fix incorrect requirement name * update CHANGELOG * attempt to get docs to build through readthedocs * port docker page to mkdocs * port contributing guide to mkdocs * add new 'running studies' page * add path changes to images * add a page on how to interpret study output * add page on the spec file * remove old sphinx docs that are no longer needed * added README to docs and updated CHANGELOG * fix copyright and hello_samples tree * rearrange images/stylesheets and statements that use them * add suggestions from Luc and Joe * add tcsh instructions for venv activation * add Charle's suggestions for the landing page * change tcsh mentions to csh * openfoam tutorial modifications (#463) * feature/revamped status (#464) * feature/new-status (#442) * add backend functionality for merlin status * add frontend functionality for merlin status * add tests for merlin status * run fix-style and remove import of deprecated function * update CHANGELOG * add more logging statements, make better use of glob * run fix-style * clean up test files a bit * fix test suite after step_name_map mod * add avg/std dev run time calculations to status * modify status tests to accommodate new avg/std dev calculations * fix linter issues * fix lint issue and add test for avg/std dev calc * feature/detailed-status (#451) * Version/1.11.0 (#449) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * add functionality for the detailed-status command * add tests for detailed-status * fix linter issues * update changelog * general cleanup and add log statements * slightly modify two tests * default status renderer now uses json status format * remove inaccurate comment * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * add Jeremy's suggestion to change vars option to output-path * remove unnecessary lines from CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * feature/queue info (#461) * remove a merge conflict statement that was missed * add queue-info functionality * add tests for queue-info * update CHANGELOG * add try/except for forceful termination of test workers * change github workflow to use py38 with black instead of py36 * run fix-style with py 3.12 and fix a typo in a test * add filetype check for dump option * add banner print statement * docs/revamped status (#462) * fix broken image link in README * add new commands to the command line page * add monitoring docs layout and complete status cmds page * fix bug with dumping queue-info to files * add docs for queue-info * add documentation for 'query-workers' * add reference to new query-workers docs and split a paragraph * fix small bug with --steps option of monitor * add documentation for monitor command * update CHANGELOG * fix dump-csv image for queue-info --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * release/1.12.0 (#465) * remove a merge conflict statement that was missed * bump version to 1.12.0 * feature/retry_priority (#468) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * add new retry priority as highest task priority * update CHANGELOG * add in MID priority * change default priority to use priority map MID value * docs/server-cross-node (#470) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * rename the merlin server config page * add instructions for running a cross-node workflow w/ containerized server * update CHANGELOG * bugfix/initial-status-issues (#471) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * fix bug with dry run status * set MANPAGER for detailed-status * fix bug with 1 sample removing the status file * add support for multiple workers on one step in status files * update test suite to accommodate changes to workers in status files * add catch and potential fix for JSONDecodeError * fix docstring of a test * update CHANGELOG.md * run fix style and add Luc's suggestions * run fix-style with python 3.12 * added additional check for status file while condensing * add try/except to catch an error for dumping statuses * release/1.12.1 (#472) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * bump version to 1.12.1 * fix a lint issue that somehow slipped through the cracks --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Jane Herriman <xorJane@users.noreply.github.com>
PreviousNext