Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
7ace8af
adjusted github issue templates
Jun 16, 2025
ae98716
adjust template
Jun 16, 2025
9857432
remove ja templates
Jun 17, 2025
b5aec2f
Merge pull request #844 from arayabrain/hotfix/adjust-issue-templates
itutu-tienday Jun 17, 2025
fe2f226
Update to correctly read snakemake.yaml nwb_template format with run_…
milesAraya Aug 1, 2025
1d15daf
Update to _TMP_DIR creation so windows run_cluster saves correctly
milesAraya Aug 1, 2025
1f82434
update file copy to simpler method used elsewhere
milesAraya Aug 4, 2025
4d7570c
Use SmkConfigReader instead of ConfigReader
milesAraya Aug 4, 2025
b85a17d
Minor refactoring (avoid direct reference to DIRPATH.SNAKEMAKE_CONFIG…
Aug 4, 2025
3366e4f
Fix path error that occured because of post_process in nwb_template
milesAraya Aug 6, 2025
7e733b5
Updates to get run_cluster working, to merge with optinist
milesAraya Aug 7, 2025
faaf557
Merge pull request #900 from arayabrain/fix/run_cluster-for-main-hotfix
milesAraya Aug 7, 2025
20d59d8
build for release 2.2.2
milesAraya Aug 8, 2025
88d3076
Merged optinist/develop-main into release branch 2.2.2, to resolve co…
milesAraya Aug 8, 2025
e65fd62
Linter fix after merge conflict resolution
milesAraya Aug 8, 2025
1f91389
Remove code related to post_process as this is a cloud function
milesAraya Aug 8, 2025
f3144f2
Rollback change to runner.py
milesAraya Aug 8, 2025
f0ca77c
Update runner.py to use a system closer to main than develop-main, an…
milesAraya Aug 12, 2025
70562a7
build for release 2.2.2.
milesAraya Aug 12, 2025
92d4b8a
Use main branch frontend
milesAraya Aug 12, 2025
1b67e73
Use main branch frontend
milesAraya Aug 12, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 0 additions & 39 deletions .github/ISSUE_TEMPLATE/bug_report_jp.md

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
name: Additional Feature
name: (developer) Additional Feature
about: Issue for Additional Feature
title: ''
labels: 'enhancement'
Expand All @@ -8,10 +8,13 @@ assignees: ''
---

### Overview

> *Describes an overview of the feature

### References

> *Links to design documentation and reference information

### Others

> *Other comments etc.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
name: Problem
name: (developer) Problem
about: Issue for Problem
title: ''
labels: ''
Expand All @@ -8,10 +8,13 @@ assignees: ''
---

### Problem

> *Describes a summary of the problem

### Measures

> *Describes expected measures

### Others

> *Other comments etc.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
name: Task
name: (developer) Task
about: Issue for Task
title: ''
labels: ''
Expand All @@ -8,13 +8,17 @@ assignees: ''
---

### Summary

> *Describes a summary of the task

### Tasks

> *Describes expected tasks

### References

> *Links to reference information

### Others

> *Other comments etc.
4 changes: 4 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,15 @@
### Content

> *Describes a content of the pull request

### References

> *Links to reference information

### Testcase

> *Testcase of the pull request

### Others

> *Other comments etc.
2 changes: 1 addition & 1 deletion codemeta.json
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@
"Docker",
"Python 3.9"
],
"version": "2.2.0",
"version": "2.2.2",
"developmentStatus": "active",
"funding": "Brain/MINDS",
"issueTracker": "https://github.com/oist/optinist/issues",
Expand Down
2 changes: 1 addition & 1 deletion docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
project = "OptiNiSt"
copyright = f"{datetime.today().year}, OIST"
author = ""
release = "2.2.1"
release = "2.2.2"

# -- readthedocs -------------------------------------------------------------
on_rtd = os.environ.get("READTHEDOCS", None) == "True"
Expand Down
2 changes: 1 addition & 1 deletion docs/specifications/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Following is the summary of directory structure.
- `.env`: Application environment variable definition file
- Depending on usage, the following .env files are available
- `.env`: Loads for all environments
- `.env.development`: Used during development (yarn start)
- `.env.development`: Used during development (yarn start)
- `.env.production`: Used during production builds (yarn build)
- [Learn more](https://create-react-app.dev/docs/adding-custom-environment-variables/#what-other-env-files-can-be-used)

Expand Down
8 changes: 4 additions & 4 deletions docs/specifications/data_nodes.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,10 @@ OptiNiSt accepts a variety of data types. Data are added in "nodes", depending o
- Can be transposed if specified in parameters
- File Extension: ".csv"
#### HDF5 / [NWB](https://nwb-schema.readthedocs.io/en/latest/format_description.html)
- Format: Contains Groups, Datasets, Attributes (metadata).
- Format: Contains Groups, Datasets, Attributes (metadata).
- A Group is similar to a folder and may contain an arbitrary number of other groups and datasets,
- A Dataset describes an n-dimensional array and provides the primary means for storing data,
- An Attribute* is a small dataset that is attached to a specific group or dataset and is typically used to store metadata specific to the object they are associated with.
- An Attribute* is a small dataset that is attached to a specific group or dataset and is typically used to store metadata specific to the object they are associated with.
- ".hdf5", ".nwb", ".HDF5", ".NWB"
#### Matlab
- Format:
Expand Down Expand Up @@ -89,7 +89,7 @@ Defined in bar.py
Expects 1D or 2D numpy array
If 1D, converts to 2D with shape (1, length)

LineData:
LineData:
Defined in line.py
Expects 2D numpy array

Expand All @@ -104,4 +104,4 @@ Expects 2D numpy array

HTMLData:
Defined in html.py
Expects string HTML content -->
Expects string HTML content -->
2 changes: 1 addition & 1 deletion frontend/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "optinist",
"version": "2.2.1",
"version": "2.2.2",
"engines": {
"node": ">=20.0.0"
},
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[tool.poetry]
name = "optinist"
description = "Calcium Imaging Pipeline Tool"
version = "2.2.1"
version = "2.2.2"
license = "GPL-3.0"
authors = ["OIST <email@example.com>"]
readme = "README.md"
Expand Down
78 changes: 68 additions & 10 deletions run_cluster.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@

from snakemake import snakemake

from studio.app.common.core.experiment.experiment import ExptOutputPathIds
from studio.app.common.core.utils.config_handler import ConfigReader
from studio.app.common.core.utils.filepath_creater import join_filepath
from studio.app.dir_path import DIRPATH

Expand All @@ -18,6 +20,7 @@ def main(args):
print(f"Temporary work directory created at: {temp_workdir}")

config_file_path = None
temp_config_file_path = None

if args.config is None:
raise FileNotFoundError(
Expand All @@ -34,12 +37,17 @@ def main(args):
if not Path(config_file_path).exists():
raise FileNotFoundError(f"Config file not found: {args.config}")

shutil.copyfile(
config_file_path,
join_filepath([str(temp_workdir), str(Path(config_file_path).name)]),
temp_config_file_path = join_filepath(
[str(temp_workdir), Path(config_file_path).name]
)
print(f"Config file copied from {config_file_path} to {temp_workdir}")

shutil.copyfile(config_file_path, temp_config_file_path)

print(
f"Config file copied from {config_file_path} to {temp_config_file_path}"
)

# Main snakemake execution
snakemake_args = {
"snakefile": DIRPATH.SNAKEMAKE_FILEPATH,
"forceall": args.forceall,
Expand All @@ -49,19 +57,69 @@ def main(args):
}

if config_file_path is not None:
snakemake_args["configfiles"] = [str(config_file_path)]
snakemake_args["configfiles"] = [str(temp_config_file_path)]

if args.use_conda:
snakemake_args["conda_prefix"] = DIRPATH.SNAKEMAKE_CONDA_ENV_DIR

print(f"Snakemake arguments: {snakemake_args}")

result = snakemake(**snakemake_args)
try:
result = snakemake(**snakemake_args)

if result:
print("snakemake execution succeeded.")
# Copy config file back to output directory for future reference
try:
# Find the output directory from the last_output in config
config_data = ConfigReader.read(config_file_path)
if config_data and "last_output" in config_data:
# Extract the last output path
last_output_path = config_data["last_output"][0]
# Construct absolute path and extract directory
absolute_output_path = join_filepath(
[DIRPATH.OUTPUT_DIR, last_output_path]
)
# Extract the workspace_id and unique_id
path_ids = ExptOutputPathIds(
os.path.dirname(absolute_output_path)
)

# Create the output directory path to copy the config file
output_config_dir = join_filepath(
[
DIRPATH.OUTPUT_DIR,
path_ids.workspace_id,
path_ids.unique_id,
]
)
os.makedirs(output_config_dir, exist_ok=True)
output_config_path = join_filepath(
[output_config_dir, DIRPATH.SNAKEMAKE_CONFIG_YML]
)

shutil.copyfile(config_file_path, output_config_path)
print(
f"Config copied to output directory: {output_config_path}"
)
else:
print(
"Warning: No output path found in config, "
"skipping config copy"
)
except Exception as e:
print(f"Warning: Failed to copy config to output directory: {e}")
else:
print("snakemake execution failed.")
print("Check for errors above in the snakemake output.")

if result:
print("snakemake execution succeeded.")
else:
print("snakemake execution failed.")
except Exception as e:
print(f"snakemake execution failed with exception: {e}")
import traceback

print("Full traceback:")
traceback.print_exc()
result = False

return result

Expand Down
2 changes: 1 addition & 1 deletion studio/app/common/core/rules/data.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ def main():

rule_config = RuleConfigReader.read(snakemake.params.name)

rule_config = SmkUtils.resolve_nwbfile_reference(rule_config)
rule_config = SmkUtils.resolve_nwbfile_reference(rule_config, snakemake.config)

if NodeTypeUtil.check_nodetype_from_filetype(rule_config.type) == NodeType.DATA:
if rule_config.type in [FILETYPE.IMAGE]:
Expand Down
3 changes: 3 additions & 0 deletions studio/app/common/core/rules/func.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
def main():
try:
from studio.app.common.core.rules.runner import Runner
from studio.app.common.core.snakemake.smk_utils import SmkUtils
from studio.app.common.core.snakemake.snakemake_reader import RuleConfigReader
from studio.app.common.core.utils.filepath_creater import join_filepath
from studio.app.dir_path import DIRPATH
Expand All @@ -28,6 +29,8 @@ def main():

rule_config = RuleConfigReader.read(snakemake.params.name)

rule_config = SmkUtils.resolve_nwbfile_reference(rule_config, snakemake.config)

rule_config.input = snakemake.input
rule_config.output = snakemake.output[0]
run_script_path = sys.argv[0]
Expand Down
Loading