What happened?

I am not sure if what I report is a bug; however, it is certainly not what I expect from a careful reading of the documentation, and I wonder if it is leading to some issues I describe below.

I am working with a large dataset produced by merging the output of runs made by different MPI processes. There are two coordinates, ("trajectory","obs"). All of the "obs" in the dataset are in order, but the "trajectory" coordinate is not in order. I made a smaller dataset that illustrates the issue below by reducing the number of "obs" from 250 to 2; this dataset can be found at http://oxbow.sr.unh.edu/data/smallExample.zarr.zip . This dataset looks like:

Dimensions:     (trajectory: 39363539, obs: 2)
Coordinates:
  * obs         (obs) int32 0 1
  * trajectory  (trajectory) int64 100 210 227 ... 39363210 39363255 39363379
Data variables:
    age         (trajectory, obs) float32 dask.array<chunksize=(50000, 2), meta=np.ndarray>
    lat         (trajectory, obs) float64 dask.array<chunksize=(50000, 2), meta=np.ndarray>
    lon         (trajectory, obs) float64 dask.array<chunksize=(50000, 2), meta=np.ndarray>
    time        (trajectory, obs) datetime64[ns] dask.array<chunksize=(40625, 2), meta=np.ndarray>
    z           (trajectory, obs) float64 dask.array<chunksize=(50000, 2), meta=np.ndarray>

Note that the trajectory coordinate is not in order; this is due to how the problem is partitioned into MPI jobs.

If I want an ordered set of trajectories, say trajectories [1,2,3,4,5,6,7,8,9,10], and I do this with subSet=dataIn.sel(trajectory=arange(1,11)), I get what I would expect from the documentation: The first through 10th trajectories, in order:

<xarray.Dataset>
Dimensions:     (trajectory: 10, obs: 2)
Coordinates:
  * obs         (obs) int32 0 1
  * trajectory  (trajectory) int64 1 2 3 4 5 6 7 8 9 10
Data variables:
    age         (trajectory, obs) float32 dask.array<chunksize=(1, 2), meta=np.ndarray>
....

But if I use the slice operator to specify what I want, I get something very different: dataIn.sel(trajectory=slice(1,11)) returns 2567339 trajectories, starting with the location of trajectory coordinate 1 in dataIn and extending to the location of trajectory coordinate 10 in dataIn:

<xarray.Dataset>
Dimensions:     (trajectory: 2567339, obs: 2)
Coordinates:
  * obs         (obs) int32 0 1
  * trajectory  (trajectory) int64 1 27 57 59 ... 39363486 39363495 39363528 11
Data variables:
    age         (trajectory, obs) float32 dask.array<chunksize=(17944, 2), meta=np.ndarray>
...

This is not what I expect -- as I understand the documentation, .sel should work in coordinate space, and I would expect dataIn.sel(trajectory=slice(1,11)) and subSet=dataIn.sel(trajectory=arange(1,11)) to return the same thing. If I am wrong in this interpretation, perhaps a documentation update would be helpful.

I have had all sorts of issues with the full dataset, including .to_zarr(dataset, compute=False) failing to return a delayedObject because it used all the memory, .sortby(['trajectory']) failing by memory exhaustion, etc. I wonder if the issue reported here can be at the root of many of these issues? On a side note, the failure of .sortby(['trajectory']) makes re-ordering the dataset difficult, and I would be happy to hear any suggestions on that front.

What did you expect to happen?

See above for full descriptions

Minimal Complete Verifiable Example

#get data from http://oxbow.sr.unh.edu/data/smallExample.zarr.zip
dataIn=xr.open_zarr('smallExample.zarr')
print(dataIn.sel(trajectory=arange(1,11)))
print(dataIn.sel(trajectory=slice(1,11)))

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment