Introduce iterations & data feed (CSV/JSON) to address #5324, #5163 #5557

Manohar-ford · 2025-11-04T04:56:57Z

Closes #5324, Closes #5163

This Pull Request introduces significant enhancements to the Hoppscotch Collection Runner, enabling users to execute collections with simple iterations and drive request parameters using external data files. This allows for robust data-driven testing and automation of repetitive tasks, directly addressing user demands for more powerful collection execution capabilities.

What's changed

This PR implements two major features within the Collection Runner: Simple Iteration and Data Feed (CSV/JSON).

Collection Runner - Simple Iteration:
- A new input field has been added to the Collection Runner UI, allowing users to specify a fixed number of times a collection (or selected requests within it) should be executed.
- The runner will now loop through the entire collection N times, providing a straightforward way to repeat tests or actions.
- Each iteration will run with a fresh state, allowing for consistent testing across multiple runs.
Collection Runner - Data Feed (CSV/JSON):
- Users can now upload external data files (CSV or JSON format) to drive variables within their requests during collection runs.
- A new section in the Collection Runner UI allows for selecting and previewing the uploaded data file.
- For each row in the data file (or each object in a JSON array), a new iteration of the collection is executed.
- Data from the current row/object is automatically injected as environment variables for that specific iteration. For example, a CSV column username would be accessible as {{username}} in requests during that iteration.
- This enables dynamic population of request URLs, headers, body, and query parameters, facilitating data-driven testing scenarios.
- Robust parsing for both CSV and JSON formats is implemented, with clear feedback for malformed files.
Integration and UI/UX:
- The Collection Runner interface has been updated to include controls for both iteration count and data file upload.
- The execution logic of the Collection Runner has been refactored to manage the iteration loop and variable injection from data feeds.
- Improved feedback mechanisms during the run to indicate current iteration number and data context.

Notes to reviewers

Motivation: This feature directly responds to the community's need for advanced testing capabilities, as highlighted in issues [feature]: Support setting number of iterations in Collection Runner UI #5324 and [feature]: Collection runner via GUI with ability to import files and choose which requests should be run #5163. It allows Hoppscotch to compete more effectively with other API clients in terms of automated testing and data-driven workflows.
Testing Strategy:
- Manual Testing: Extensive manual testing was performed to cover:
  - Running collections with a specified number of iterations (e.g., 1, 5, 10).
  - Uploading valid CSV and JSON files, ensuring correct variable injection into requests.
  - Testing with various request types (GET, POST, PUT) and variable locations (URL, headers, body).
  - Verifying error handling for malformed data files or missing variables.
  - Confirming that environments are correctly handled per iteration.
- Unit/Integration Tests: Relevant unit and integration tests have been added/updated for the new iteration and data parsing logic.
Edge Cases Considered:
- Empty data files: Handled gracefully, resulting in zero iterations or an appropriate error message.
- Missing variables in data: Variables not found in the data file will fall back to existing environment/global variables or remain unresolved, consistent with current Hoppscotch behavior.
- Large data files: Performance considerations were made for parsing, though extremely large files might still impact initial load time.
- Data type handling: All data from CSV/JSON is treated as strings for variable substitution, allowing users to perform type conversions within their request scripts if needed.
Future Enhancements (Out of Scope for this PR):
- Advanced reporting for data-driven runs (e.g., pass/fail statistics per iteration).
- Support for additional data formats (e.g., Excel).
- More sophisticated iteration control (e.g., running specific rows, conditional iterations).
- Visual mapping of data file columns to request variables.
Backward Compatibility: This feature is fully backward compatible. Existing collection runs without iteration counts or data files will function as before.
Security Considerations: Data files are processed client-side. Users are responsible for the content of their uploaded files. No new external network requests are made as part of the data parsing.

Checklist

Linked issues ([feature]: Support setting number of iterations in Collection Runner UI #5324, [feature]: Collection runner via GUI with ability to import files and choose which requests should be run #5163)
Backward compatible
New UI elements for iteration count and data file upload
Collection Runner logic updated for iteration and data feed
CSV data parsing implemented
JSON data parsing implemented
Data variables injected into request context for each iteration
Manual tests performed
Automated tests (where applicable) added/updated

…load

Copilot

Pull Request Overview

This PR adds data-driven testing capabilities to the test runner, allowing users to run collection tests with iterations using CSV or JSON dataset files. The key changes include implementing dataset parsing, iteration management, and UI components for dataset upload and configuration.

Added dataset file upload support (CSV/JSON) with validation and preview capabilities
Implemented iteration logic that cycles through dataset rows and runs tests multiple times
Enhanced test runner to accumulate results across iterations instead of overwriting them

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
test-runner.service.ts	Refactored test execution to support iterations with dataset injection, modified collection/request handling to append results instead of replacing them
document.ts	Added dataset configuration type to TestRunnerConfig for storing dataset metadata and content
parser.ts	New utility file for parsing and validating CSV/JSON dataset files
RequestRunner.ts	Enhanced request runner to inject iteration data as environment variables during test execution
RunnerModal.vue	Added UI for dataset file upload, preview, download, and iteration configuration with visual feedback
en.json	Added localization strings for dataset-related UI text and error messages

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-06T08:00:49Z

packages/hoppscotch-common/src/helpers/import-export/dataset/parser.ts

+
+    for (let i = 1; i < lines.length; i++) {
+        const obj: Record<string, string> = {}
+        const currentLine = lines[i].split(",")


The CSV parser uses a naive comma split that will break when values contain commas (e.g., quoted strings like 'Hello, World'). This fails to handle standard CSV features like quoted fields, escaped quotes, or commas within values. Consider using a proper CSV parsing library or implementing RFC 4180 compliant parsing.

Copilot · 2025-11-06T08:00:49Z

packages/hoppscotch-common/src/components/http/test/RunnerModal.vue

+                      t("collection_runner.dataset_iterations_info", {
+                        iterations: config.iterations,
+                        rows: datasetRowCount,
+                        total: config.iterations * datasetRowCount,


The calculation for total iterations is incorrect. When dataset is enabled, the total should be just config.iterations (each iteration uses one row), not config.iterations * datasetRowCount. This misleads users about the actual number of test runs.

packages/hoppscotch-common/src/components/http/test/RunnerModal.vue

packages/hoppscotch-common/src/helpers/import-export/dataset/parser.ts

packages/hoppscotch-common/src/helpers/RequestRunner.ts

packages/hoppscotch-common/src/components/http/test/RunnerModal.vue

packages/hoppscotch-common/locales/en.json

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…l.vue Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…arser.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…l.vue Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…l.vue Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

cubic-dev-ai

1 issue found across 6 files

Prompt for AI agents (all 1 issues)


Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="packages/hoppscotch-common/src/helpers/import-export/dataset/parser.ts">

<violation number="1" location="packages/hoppscotch-common/src/helpers/import-export/dataset/parser.ts:48">
Convert JSON dataset values to strings so CSV/JSON feeds behave consistently during variable injection.</violation>
</file>

Since this is your first cubic review, here's how it works:

cubic automatically reviews your code and comments on bugs and improvements
Teach cubic by replying to its comments. cubic learns from your replies and gets better over time
Ask questions if you need clarification on any suggestion

_{Reply to cubic to teach it or ask questions. Re-run a review with @cubic-dev-ai review this PR}

packages/hoppscotch-common/src/helpers/import-export/dataset/parser.ts

cubic-dev-ai

1 issue found across 6 files

Prompt for AI agents (all 1 issues)


Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="packages/hoppscotch-common/src/helpers/import-export/dataset/parser.ts">

<violation number="1" location="packages/hoppscotch-common/src/helpers/import-export/dataset/parser.ts:43">
P2: The function wraps any non-array value without checking if it&#39;s actually an object. If the JSON contains a primitive (e.g., `&quot;string&quot;`, `123`, `null`) or an array of primitives, the return type `Array&lt;Record&lt;string, any&gt;&gt;` is violated. Consider validating the parsed value is an object before wrapping, and validating array items.</violation>
</file>

_{Reply to cubic to teach it or ask questions. Re-run a review with @cubic-dev-ai review this PR}

packages/hoppscotch-common/src/helpers/import-export/dataset/parser.ts

…arser.ts Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

cubic-dev-ai

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (all 1 issues)


Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="packages/hoppscotch-common/src/helpers/import-export/dataset/parser.ts">

<violation number="1" location="packages/hoppscotch-common/src/helpers/import-export/dataset/parser.ts:51">
P2: The validation allows arrays to pass as valid items since `typeof [] === &quot;object&quot;` in JavaScript. Input like `[[1,2], [3,4]]` would incorrectly pass validation. Add `&amp;&amp; !Array.isArray(item)` to ensure only plain objects are accepted.</violation>
</file>

_{Reply to cubic to teach it or ask questions. Re-run a review with @cubic-dev-ai review this PR}

packages/hoppscotch-common/src/helpers/import-export/dataset/parser.ts

…arser.ts Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

Manohar-ford added 3 commits October 15, 2025 12:09

feat: add support for iterations and validation in collection runner

b630b1c

feat: add dataset support for collection runner with CSV/JSON file up…

86d4259

…load

Merge branch 'hoppscotch:main' into update-2025.9.0

55d0a94

jamesgeorge007 added the future Scheduled to near future label Nov 5, 2025

jamesgeorge007 requested a review from Copilot November 6, 2025 07:57

Copilot AI reviewed Nov 6, 2025

View reviewed changes

Manohar-ford and others added 7 commits November 7, 2025 17:53

Update packages/hoppscotch-common/locales/en.json

b35401a

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update packages/hoppscotch-common/src/components/http/test/RunnerModa…

a9f6040

…l.vue Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update packages/hoppscotch-common/src/helpers/import-export/dataset/p…

e189281

…arser.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update packages/hoppscotch-common/src/components/http/test/RunnerModa…

f1c92a9

…l.vue Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update packages/hoppscotch-common/src/helpers/RequestRunner.ts

d431e84

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update packages/hoppscotch-common/src/components/http/test/RunnerModa…

6331761

…l.vue Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Merge branch 'main' into update-2025.9.0

394069d

cubic-dev-ai bot reviewed Nov 25, 2025

View reviewed changes

packages/hoppscotch-common/src/helpers/import-export/dataset/parser.ts Show resolved Hide resolved

Merge branch 'main' into update-2025.9.0

ff96205

cubic-dev-ai bot reviewed Dec 8, 2025

View reviewed changes

packages/hoppscotch-common/src/helpers/import-export/dataset/parser.ts Show resolved Hide resolved

Update packages/hoppscotch-common/src/helpers/import-export/dataset/p…

ec30770

…arser.ts Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

cubic-dev-ai bot reviewed Dec 8, 2025

View reviewed changes

packages/hoppscotch-common/src/helpers/import-export/dataset/parser.ts Outdated Show resolved Hide resolved

Update packages/hoppscotch-common/src/helpers/import-export/dataset/p…

05dcdd1

…arser.ts Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Introduce iterations & data feed (CSV/JSON) to address #5324, #5163 #5557

Introduce iterations & data feed (CSV/JSON) to address #5324, #5163 #5557

Uh oh!

Manohar-ford commented Nov 4, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Nov 6, 2025

Uh oh!

Copilot AI Nov 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai bot left a comment

Uh oh!

Uh oh!

cubic-dev-ai bot left a comment

Uh oh!

Uh oh!

cubic-dev-ai bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Introduce iterations & data feed (CSV/JSON) to address #5324, #5163 #5557

Are you sure you want to change the base?

Introduce iterations & data feed (CSV/JSON) to address #5324, #5163 #5557

Uh oh!

Conversation

Manohar-ford commented Nov 4, 2025

What's changed

Notes to reviewers

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants