Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can not query multiple symbols if their columns are stored with different data types #402

Open
briancappello opened this issue Dec 6, 2020 · 0 comments

Comments

@briancappello
Copy link
Contributor

This is perhaps more of a limitation than a bug?

Let's say you create two TBKs, one stored with 64bit values and the other 32bit:

from pymarketstore import Client, DataShape
client.create('AMD/1D/OHLCV', DataShape([
  ('Open', 'float64'),
  ('High', 'float64'),
  ('Low', 'float64'),
  ('Close', 'float64'),
  ('Volume', 'int64'),
]))
client.create('NVDA/1D/OHLCV', DataShape([
  ('Open', 'float32'),
  ('High', 'float32'),
  ('Low', 'float32'),
  ('Close', 'float32'),
  ('Volume', 'int32'),
]))

If you then query for these together:

client.query('AMD,NVDA/1D/OHLCV')

The response format does not support this. (pymarketstore will blow up attempting to decode the raw response data.) The reason being, there's an implicit assumption built into NumpyMultiDataset that all datatypes must be the same:

NumpyDataset:
  ColumnNames: ['Open', 'Volume']
  ColumnTypes: ['float32', 'int32']
  ColumnData:
    Open: [AMD opens] + [NVDA opens]
    Volume: [AMD volumes] + [NVDA volumes]
StartIndex: indexes into ColumnData for each symbol
Lengths: number of bars in ColumnData for each symbol

ColumnData is simply bytes, so marketstore will happily return 32bit values and 64bit values in sequence, however, the client has no way to know that the data types vary by symbol, and as such fails to correctly index into the ColumnData arrays.

Querying for the symbols individually works fine, with marketstore returning the correct data types for each.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant