Skip to content

Conversation

tzaffi
Copy link
Contributor

@tzaffi tzaffi commented Jul 19, 2023

Summary

The delete task wasn't running because the exporter config is wasn't actually being set in the exporter struct.

This PR includes:

  • The bugfix in conduit/plugins/exporters/postgresql/postgresql_exporter.go and conduit/plugins/exporters/postgresql/util/prune.go
  • End-to-end test enhancement to verify the fix (pyproject.toml and *.py). This required providing the capability to query the indexer database.
  • Changing the CLI descriptions for the conduit init command to be more in line with the other such descriptions (pkg/cli/internal/initialize/init.go)
  • Miscellaneous lint errors that I was experiencing locally, but that don't seem to be happening in CI.
  • Fixing TestMetrics in conduit/pipeline/pipeline_test.go which started failing mysteriously.

TODO

Issues

#117

Test Plan

A new E2E test is introduced to ensure that the delete task works as expected. I tried running this new test against the original code, and it failed in the expected manner.

See: #121

@codecov
Copy link

codecov bot commented Jul 19, 2023

Codecov Report

Merging #119 (af2eb5a) into master (442791a) will increase coverage by 2.69%.
The diff coverage is 76.61%.

@@            Coverage Diff             @@
##           master     #119      +/-   ##
==========================================
+ Coverage   67.66%   70.36%   +2.69%     
==========================================
  Files          32       36       +4     
  Lines        1976     2500     +524     
==========================================
+ Hits         1337     1759     +422     
- Misses        570      646      +76     
- Partials       69       95      +26     
Impacted Files Coverage Δ
conduit/data/block_export_data.go 100.00% <ø> (+92.30%) ⬆️
conduit/metrics/metrics.go 100.00% <ø> (ø)
conduit/pipeline/metadata.go 69.11% <ø> (ø)
conduit/plugins/config.go 100.00% <ø> (ø)
...duit/plugins/exporters/filewriter/file_exporter.go 81.63% <ø> (-1.06%) ⬇️
conduit/plugins/importers/algod/metrics.go 100.00% <ø> (ø)
...gins/processors/filterprocessor/fields/searcher.go 77.50% <ø> (ø)
...ins/processors/filterprocessor/filter_processor.go 83.82% <ø> (+3.54%) ⬆️
...plugins/processors/filterprocessor/gen/generate.go 34.28% <ø> (ø)
conduit/plugins/processors/noop/noop_processor.go 64.70% <ø> (+6.81%) ⬆️
... and 17 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

found++
// 1 hour in seconds
assert.Contains(t, stat.String(), "sample_count:1 sample_sum:3600")
assert.Regexp(t, regexp.MustCompile("sample_count:1 +sample_sum:3600"), stat.String())
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@winder
For mysterious reasons, this tests started failing on me and I had to be more lenient on the assertions in order to pass.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would guess it's related to upgrading the prometheus versions.

Am I reading the regex correctly, that there is additional whitespace and some control character changes (quotes instead of angles?) but the data is otherwise the same?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. We now have { instead of < and unpredictable white space.

Copy link
Contributor Author

@tzaffi tzaffi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestions

@tzaffi tzaffi added the Bug-Fix PR proposing to fix a bug label Jul 20, 2023
@tzaffi tzaffi marked this pull request as ready for review July 20, 2023 13:35
@tzaffi tzaffi requested a review from a team July 20, 2023 13:36
accumulated_config: dict = field(default_factory=dict)
conduit_dir: str = ""

def get_validation_errors(self) -> list[str]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we actually run accounting validation in indexer e2e. we should probably create a ticket to migrate that to conduit.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. I created issue #123

@tzaffi tzaffi requested a review from a team July 20, 2023 17:49
Copy link
Contributor

@algochoi algochoi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't look closely at the tests but the lint + fix LGTM

Copy link
Contributor

@winder winder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix look good.

The one thing I'd like to see changed is the error messages for the integration test. If the test were to fail, the current error messages are not much to work with.

@winder winder requested review from shiqizng and algochoi July 21, 2023 15:41
@tzaffi tzaffi merged commit 98efe5a into master Jul 21, 2023
@tzaffi tzaffi deleted the delete-task-bugfix branch July 21, 2023 15:56
@tzaffi tzaffi mentioned this pull request Jul 31, 2023
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug-Fix PR proposing to fix a bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants