Skip to content

New dataset model: handle.py + arbitrary files. - #6

Open
colah wants to merge 2 commits into
masterfrom
smart-data
Open

New dataset model: handle.py + arbitrary files.#6
colah wants to merge 2 commits into
masterfrom
smart-data

Conversation

@colah

@colah colah commented Nov 18, 2013

Copy link
Copy Markdown
Owner

We move to a new model of datasets, where the only requirement is
that a dataset provide a handle.py file describing how to use it.

This model, along similar ideas as Kolmogorov Complexity, allows
us to support the great breadth of data used in the sciences. It
simply isn't practical to coerce all data into a single format.

To avoid a hundred implementations of importers for common formats,
we introduce "helpers". These are available to handle.py at load
time, and can handle importing common data types.

This commit has had some basic testing. Datasets in datasets.py are
not moved to the new model at the time of this commit. There is
basic documentation of the new model, and a template directory with
a template handle.py.

We move to a new model of datasets, where the only requirement is
that a dataset provide a `handle.py` file describing how to use it.

This model, along similar ideas as Kolmogorov Complexity, allows
us to support the great breadth of data used in the sciences. It
simply isn't practical to coerce all data into a single format.

To avoid a hundred implementations of importers for common formats,
we introduce "helpers". These are available to `handle.py` at load
time, and can handle importing common data types.

This commit has had some basic testing. Datasets in datasets.py are
not moved to the new model at the time of this commit. There is
basic documentation of the new model, and a template directory with
a template `handle.py`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant