Skip to content

Allow for rebucketing #4

Open
Open

Description

@tdozat

Currently, once you construct a Parseset object from a training file, you can't reuse it with a different file. This means that when parsing thousands of files, you'll create thousands of new Parseset objects with their own TensorFlow graphs, each only being used once. This could slow down parsing, and if for some reason they don't get garbage collected it could result in memory overflow.

Solution: only construct one Parseset when parsing files, and let the model reuse it by deleting the bucket data without overwriting the bucket TensorFlow objects.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions