remove unexpected key from datum_kwargs #655

prjemian · 2022-05-30T03:37:26Z

Close unexpected key in datum_kwargs #651

prjemian · 2022-05-30T03:50:30Z

Test this locally before merging

prjemian · 2022-05-31T06:32:16Z

CI fails with this error:

E           event_model.UndefinedAssetSpecification: "Resource document with uid b57d7481-d402-4e83-af3d-14b0414c59fa refers to spec 'AD_HDF5' which is not defined in the Filler's handler registry."

That is because the area-detector-handlers package is not installed.

prjemian · 2022-05-31T06:49:53Z

Another CI failure now:

E           event_model.EventModelError: Error instantiating handler class <class 'area_detector_handlers.handlers.AreaDetectorHDF5Handler'> with Resource document {'spec': 'AD_HDF5', 'root': '/', 'resource_path': 'tmp/images/test_image_0000.h5', 'resource_kwargs': {'frame_per_point': 1}, 'path_semantics': 'posix', 'uid': '571cfd77-0ff1-42e8-a694-2f18b851865c', 'run_start': '0ee9f843-16ae-4851-a4ff-dbba41ae3d2f'}. Its 'root' field / was *not* modified by root_map.

prjemian · 2022-05-31T06:53:35Z

E               event_model.UnresolvableForeignKeyError: ('8eba8e30-6f72-4907-b1e1-1bac34a3db78/0', 'Event with uid 9123f415-941d-4bef-846c-abaca85af0e8 refers to unknown Datum datum_id 8eba8e30-6f72-4907-b1e1-1bac34a3db78/0')

gfabbris · 2022-05-31T12:18:10Z

Every time I've seen these last two error was because it could not find the image because the name or path was wrong (here /tmp/images/test_image_0000.h5).

prjemian · 2022-05-31T14:27:59Z

Every time I've seen these last two error was because it could not find the image because the name or path was wrong (here /tmp/images/test_image_0000.h5).

@gfabbris : Thanks! The test passes locally so I'm thinking this must be associated with conda environment, where the dockers are mounted in the GitHub CI, or in how the path names are constructed. The trick here is to get the CI to show the names in the various parts of the process.

prjemian · 2022-05-31T17:31:58Z

@danielballan: I'm missing a detail here and the area detector workflow is failing Lots of diagnostics have been added to the workflow to track down this pesky varmint.

E           event_model.EventModelError: Error instantiating handler class <class 'area_detector_handlers.handlers.AreaDetectorHDF5Handler'> with Resource document {'spec': 'AD_HDF5', 'root': '/', 'resource_path': 'tmp/images/test_image_0000.h5', 'resource_kwargs': {'frame_per_point': 1}, 'path_semantics': 'posix', 'uid': '0615d2ee-6258-41a3-983c-b006de6bd8ed', 'run_start': 'fc5a3756-b98a-435a-beaf-57fd27c93801'}. Its 'root' field / was *not* modified by root_map.

We're unit testing custom (EPICS PV-directed) area detector file names with bluesky's bp.count() plan. It passes these tests at home but not on GitHub Actions. Here's the relevant section (so I believe) of the conda environment requirements:

apstools/environment.yml

Lines 9 to 12 in ace1eed

    
           - area-detector-handlers 
        
           - bluesky >=1.6.7 
        
           - databroker =1 
        
           - databroker-pack

prjemian · 2022-05-31T17:36:42Z

The custom detector classes are:

apstools/apstools/devices/area_detector_support.py

Lines 224 to 415 in ace1eed

    
           class AD_EpicsHdf5FileName(FileStorePluginBase):  # lgtm [py/missing-call-to-init] 
        
               """ 
        
               custom class to define image file name from EPICS 
        
               .. index:: Ophyd Device Support; AD_EpicsHdf5FileName 
        
               .. caution:: *Caveat emptor* applies here.  You assume expertise! 
        
               Replace standard Bluesky algorithm where file names 
        
               are defined as UUID strings, virtually guaranteeing that 
        
               no existing images files will ever be overwritten. 
        
               Also, this method decouples the data files from the databroker, 
        
               which needs the files to be named by UUID. 
        
               .. autosummary:: 
        
                   ~make_filename 
        
                   ~generate_datum 
        
                   ~get_frames_per_point 
        
                   ~stage 
        
               To allow users to control the file **name**, 
        
               we override the ``make_filename()`` method here 
        
               and we need to override some intervening classes. 
        
               To allow users to control the file **number**, 
        
               we override the ``stage()`` method here 
        
               and triple-comment out that line, and bring in 
        
               sections from the methods we are replacing here. 
        
               The image file name is set in `FileStoreBase.make_filename()` 
        
               from `ophyd.areadetector.filestore_mixins`.  This is called 
        
               (during device staging) from `FileStoreBase.stage()` 
        
               EXAMPLE: 
        
               To use this custom class, we need to connect it to some 
        
               intervening structure.  Here are the steps: 
        
               #. override default file naming 
        
               #. use to make your custom iterative writer 
        
               #. use to make your custom HDF5 plugin 
        
               #. use to make your custom AD support 
        
               imports:: 
        
                   from bluesky import RunEngine, plans as bp 
        
                   from ophyd.areadetector import SimDetector, SingleTrigger 
        
                   from ophyd.areadetector import ADComponent, ImagePlugin, SimDetectorCam 
        
                   from ophyd.areadetector import HDF5Plugin 
        
                   from ophyd.areadetector.filestore_mixins import FileStoreIterativeWrite 
        
               override default file naming:: 
        
                   from apstools.devices import AD_EpicsHdf5FileName 
        
               make a custom iterative writer:: 
        
                   class myHdf5EpicsIterativeWriter(AD_EpicsHdf5FileName, FileStoreIterativeWrite): pass 
        
               make a custom HDF5 plugin:: 
        
                   class myHDF5FileNames(HDF5Plugin, myHdf5EpicsIterativeWriter): pass 
        
               define support for the detector (simulated detector here):: 
        
                   class MySimDetector(SingleTrigger, SimDetector): 
        
                       '''SimDetector with HDF5 file names specified by EPICS''' 
        
                       cam = ADComponent(SimDetectorCam, "cam1:") 
        
                       image = ADComponent(ImagePlugin, "image1:") 
        
                       hdf1 = ADComponent( 
        
                           myHDF5FileNames, 
        
                           suffix = "HDF1:", 
        
                           root = "/", 
        
                           write_path_template = "/", 
        
                           ) 
        
               create an instance of the detector:: 
        
                   simdet = MySimDetector("13SIM1:", name="simdet") 
        
                   if hasattr(simdet.hdf1.stage_sigs, "array_counter"): 
        
                       # remove this so array counter is not set to zero each staging 
        
                       del simdet.hdf1.stage_sigs["array_counter"] 
        
                   simdet.hdf1.stage_sigs["file_template"] = '%s%s_%3.3d.h5' 
        
               setup the file names using the EPICS HDF5 plugin:: 
        
                   simdet.hdf1.file_path.put("/tmp/simdet_demo/")    # ! ALWAYS end with a "/" ! 
        
                   simdet.hdf1.file_name.put("test") 
        
                   simdet.hdf1.array_counter.put(0) 
        
               If you have not already, create a bluesky RunEngine:: 
        
                   RE = RunEngine({}) 
        
               take an image:: 
        
                   RE(bp.count([simdet])) 
        
               INTERNAL METHODS 
        
               """ 
        
               def __init__(self, *args, **kwargs): 
        
                   FileStorePluginBase.__init__(self, *args, **kwargs) 
        
                   self.filestore_spec = "AD_HDF5"  # spec name stored in resource doc 
        
                   self.stage_sigs.update( 
        
                       [ 
        
                           ("file_template", "%s%s_%4.4d.h5"), 
        
                           ("file_write_mode", "Stream"), 
        
                           ("capture", 1), 
        
                       ] 
        
                   ) 
        
               def make_filename(self): 
        
                   """ 
        
                   overrides default behavior: Get info from EPICS HDF5 plugin. 
        
                   """ 
        
                   # start of the file name, file number will be appended per template 
        
                   filename = self.file_name.get() 
        
                   # this is where the HDF5 plugin will write the image, 
        
                   # relative to the IOC's filesystem 
        
                   write_path = self.file_path.get() 
        
                   # this is where the DataBroker will find the image, 
        
                   # on a filesystem accessible to Bluesky 
        
                   read_path = write_path 
        
                   return filename, read_path, write_path 
        
               # def generate_datum(self, key, timestamp, datum_kwargs): 
        
               #     """Generate a uid and cache it with its key for later insertion.""" 
        
               #     template = self.file_template.get() 
        
               #     filename, read_path, write_path = self.make_filename() 
        
               #     file_number = self.file_number.get() 
        
               #     hdf5_file_name = template % (read_path, filename, file_number) 
        
               #     # inject the actual name of the HDF5 file here into datum_kwargs 
        
               #     datum_kwargs["HDF5_file_name"] = hdf5_file_name 
        
               #     logger.debug("make_filename: %s", hdf5_file_name) 
        
               #     logger.debug("write_path: %s", write_path) 
        
               #     return super().generate_datum(key, timestamp, datum_kwargs) 
        
               def get_frames_per_point(self): 
        
                   """overrides default behavior""" 
        
                   return self.num_capture.get() 
        
               def stage(self): 
        
                   """ 
        
                   overrides default behavior 
        
                   Set EPICS items before device is staged, then copy EPICS 
        
                   naming template (and other items) to ophyd after staging. 
        
                   """ 
        
                   # Make a filename. 
        
                   filename, read_path, write_path = self.make_filename() 
        
                   # Ensure we do not have an old file open. 
        
                   set_and_wait(self.capture, 0) 
        
                   # These must be set before parent is staged (specifically 
        
                   # before capture mode is turned on. They will not be reset 
        
                   # on 'unstage' anyway. 
        
                   set_and_wait(self.file_path, write_path) 
        
                   set_and_wait(self.file_name, filename) 
        
                   # set_and_wait(self.file_number, 0) 
        
                   # get file number now since it is incremented during stage() 
        
                   file_number = self.file_number.get() 
        
                   # Must avoid parent's stage() since it sets file_number to 0 
        
                   # Want to call grandparent's stage() 
        
                   # super().stage()     # avoid this - sets `file_number` to zero 
        
                   # call grandparent.stage() 
        
                   FileStoreBase.stage(self) 
        
                   # AD does the file name templating in C 
        
                   # We can't access that result until after acquisition 
        
                   # so we apply the same template here in Python. 
        
                   template = self.file_template.get() 
        
                   self._fn = template % (read_path, filename, file_number) 
        
                   self._fp = read_path 
        
                   if not self.file_path_exists.get(): 
        
                       raise IOError(f"Path {self.file_path.get()} does not exist on IOC.") 
        
                   self._point_counter = itertools.count() 
        
                   # from FileStoreHDF5.stage() 
        
                   res_kwargs = {"frame_per_point": self.get_frames_per_point()} 
        
                   self._generate_resource(res_kwargs)

and

apstools/apstools/devices/tests/test_issue655.py

Lines 25 to 51 in ace1eed

    
           class myHdf5EpicsIterativeWriter(AD_EpicsHdf5FileName, FileStoreIterativeWrite): 
        
               """Build a custom iterative writer using the override.""" 
        
               pass 
        
           class myHDF5FileNames(HDF5Plugin, myHdf5EpicsIterativeWriter): 
        
               """Make a custom HDF5 plugin using the override.""" 
        
               pass 
        
           # define support for the detector (ADSimDetector here): 
        
           class MySimDetector(SingleTrigger, SimDetector): 
        
               """SimDetector with HDF5 file names specified by EPICS.""" 
        
               _default_read_attrs = ["hdf1"] 
        
               cam = ADComponent(SimDetectorCam, "cam1:") 
        
               image = ADComponent(ImagePlugin, "image1:") 
        
               hdf1 = ADComponent( 
        
                   myHDF5FileNames, 
        
                   suffix="HDF1:", 
        
                   write_path_template=WRITE_PATH_TEMPLATE, 
        
                   read_path_template=READ_PATH_TEMPLATE, 
        
               )

and implementation here:

apstools/apstools/devices/tests/test_issue655.py

Lines 64 to 92 in ace1eed

    
           camera = MySimDetector(IOC, name="camera") 
        
           assert isinstance(camera, MySimDetector) 
        
           if hasattr(camera.hdf1.stage_sigs, "array_counter"): 
        
               # remove this so array counter is not set to zero each staging 
        
               del camera.hdf1.stage_sigs["array_counter"] 
        
           camera.cam.stage_sigs["num_images"] = 1 
        
           camera.hdf1.stage_sigs["num_capture"] = 1 
        
           camera.hdf1.stage_sigs["file_template"] = "%s%s_%3.3d.h5" 
        
           camera.hdf1.stage_sigs["compression"] = "zlib" 
        
           # ensure the capture is always staged last 
        
           camera.hdf1.stage_sigs["capture"] = camera.hdf1.stage_sigs.pop("capture") 
        
           camera.wait_for_connection() 
        
           assert camera.connected 
        
           camera.hdf1.warmup() 
        
           file_name = "test_image" 
        
           file_number = camera.hdf1.file_number.get() 
        
           file_path = AD_IOC_MOUNT_PATH / IMAGE_DIR 
        
           file_template = "%s%s_%4.4d.h5" 
        
           camera.hdf1.create_directory.put(-5) 
        
           camera.hdf1.file_name.put(file_name) 
        
           camera.hdf1.file_path.put(str(file_path)) 
        
           camera.hdf1.stage_sigs["file_template"] = file_template

prjemian · 2022-05-31T20:01:32Z

This is the initial exception (as shown in the logs):

2022-05-31T16:55:50.2489721Z             # Look up the cached Datum doc.
2022-05-31T16:55:50.2490037Z             try:
2022-05-31T16:55:50.2490261Z >               datum_doc = self._datum_cache[datum_id]
2022-05-31T16:55:50.2490599Z E               KeyError: '7677d5cb-8e93-4285-9117-87045e916754/0'

prjemian · 2022-05-31T22:39:30Z

Passes locally, fails here.

prjemian · 2022-06-01T15:06:39Z

tests

image file exists as acquired

apstools/apstools/devices/tests/test_issue651.py

Lines 106 to 121 in 8e4b657

    
           # Can bluesky see the new image file? 
        
           ioc_file_name = (file_template % (file_path, "/" + file_name, file_number)) 
        
           assert camera.hdf1.full_file_name.get() == ioc_file_name 
        
           ioc_tmp = pathlib.Path(BLUESKY_MOUNT_PATH).parent 
        
           assert ioc_tmp.exists(), ioc_tmp 
        
           # get the full file name in terms of the bluesky file directory 
        
           _n = len(str(AD_IOC_MOUNT_PATH.parent)) 
        
           resource_file_name = ioc_file_name[_n:] 
        
           image_file = ioc_tmp / resource_file_name.lstrip("/") 
        
           assert image_file.exists(), ( 
        
               f"ioc_tmp={ioc_tmp}" 
        
               f" resource_file_name={resource_file_name}" 
        
               f" image_file={image_file}" 
        
           )

document content in databroker v1

apstools/apstools/devices/tests/test_issue651.py

Lines 123 to 147 in 8e4b657

    
           # verify that the problem key in datum_kwargs is not found 
        
           for uid in uids: 
        
               uid_start = None 
        
               uid_resource = None 
        
               id_datum = None 
        
               for doc_type, doc in cat.v1[uid].documents(): 
        
                   if doc_type == "start": 
        
                       uid_start = doc["uid"] 
        
                   elif doc_type == "datum": 
        
                       id_datum = f"{uid_resource}/0" 
        
                       assert doc["_name"] == "Datum", doc 
        
                       assert doc["datum_id"] == id_datum, doc 
        
                       # "HDF5_file_name" is an unexpected key in datum_kwargs 
        
                       assert doc["datum_kwargs"] == {"point_number": 0}, doc 
        
                       assert doc["resource"] == uid_resource, doc 
        
                   elif doc_type == "resource": 
        
                       uid_resource = doc["uid"] 
        
                       assert doc["path_semantics"] == "posix", doc 
        
                       assert doc["resource_kwargs"] == {"frame_per_point": 1}, doc 
        
                       assert doc["resource_path"] == ioc_file_name.lstrip("/"), doc 
        
                       assert doc["root"] == "/", doc 
        
                       assert doc["run_start"] == uid_start, doc 
        
                       assert doc["spec"] == "AD_HDF5", doc

databroker v2 view of the resource

apstools/apstools/devices/tests/test_issue651.py

Lines 149 to 164 in 8e4b657

    
           run = cat.v2[uid] 
        
           assert run is not None 
        
           primary = run.primary 
        
           rsrc = primary._resources 
        
           assert isinstance(rsrc, list), rsrc 
        
           assert len(rsrc) == 1 
        
           rsrc = rsrc[0].to_dict() 
        
           assert isinstance(rsrc, dict), rsrc 
        
           assert rsrc["path_semantics"] == "posix", rsrc 
        
           assert rsrc["resource_kwargs"] == {"frame_per_point": 1}, rsrc 
        
           assert rsrc["resource_path"] == ioc_file_name.lstrip("/"), rsrc 
        
           assert rsrc["root"] == "/", rsrc 
        
           assert rsrc["run_start"] == uid_start, rsrc 
        
           assert rsrc["spec"] == "AD_HDF5", rsrc 
        
           assert rsrc["uid"] == uid_resource, rsrc

image file exists

apstools/apstools/devices/tests/test_issue651.py

Lines 166 to 170 in 8e4b657

    
           # Could databroker find this file? 
        
           _n = len(str(AD_IOC_MOUNT_PATH.parent)) 
        
           resource_file_name = ioc_file_name[_n:] 
        
           image_file = ioc_tmp / resource_file_name.lstrip("/") 
        
           assert image_file.exists()

image content found in file

apstools/apstools/devices/tests/test_issue651.py

Lines 172 to 178 in 8e4b657

    
           # Could databroker read this image? 
        
           with h5py.File(str(image_file), "r") as h5: 
        
               key = "/entry/data/data" 
        
               assert key in h5 
        
               data = h5[key] 
        
               assert data is not None 
        
               assert data.shape == (1, 1024, 1024)

failing test

apstools/apstools/devices/tests/test_issue651.py

Line 180 in 8e4b657

# If this errors, one cause is issue 651:

prjemian · 2022-06-01T15:24:27Z

Consider these lines in make_filename() from the super class: https://github.com/bluesky/ophyd/blob/aa17c1e896aa3874ff722d72c8f6d9aa59c3ebaa/ophyd/areadetector/filestore_mixins.py#L417-L421

        filename = new_short_uid()
        formatter = datetime.now().strftime
        write_path = formatter(self.write_path_template)
        read_path = formatter(self.read_path_template)
        return filename, read_path, write_path

MNT #651 comment out generate_datum() method

b44b5f3

prjemian added the bug label May 30, 2022

prjemian added this to the 1.6.2 milestone May 30, 2022

prjemian self-assigned this May 30, 2022

prjemian added 2 commits May 31, 2022 01:14

TST #655 ignore DeprecationWarnings in tests

2aa8ebf

TST #655 test the unexpected key

1c91698

PKG #655 require area-detector-handlers

a80961c

prjemian added 4 commits May 31, 2022 10:32

TST #651 document contents

a518129

STY #651

3f3ca03

TST #651 Can BS see image file?

11e869b

TST #651

ace1eed

prjemian added 5 commits May 31, 2022 15:57

TST #651 review databroker view

0a79f7d

TST #651 ignore warning from upstream

bfd1209

TST #651

264037d

TST #651 test the image file content directly

cc9771c

TST #651 test the templates

8e4b657

prjemian added 2 commits June 1, 2022 16:50

MNT #651 override make_filename()

eca9d98

TST #651 revise per discussion

fca0239

prjemian added 3 commits June 1, 2022 17:17

DOC #651 cleanup

e9a75a7

DOC #651

e4b8b95

TST #651 no () here

09f12d6

prjemian merged commit 0b99a71 into main Jun 1, 2022

prjemian deleted the 651-datum_kwargs branch June 1, 2022 23:26

This was referenced Jun 2, 2022

package myHDF5FileNames() class #656

Closed

Can databroker find the images in custom EPICS HDF5 file names? #647

Closed

prjemian mentioned this pull request Jul 2, 2022

Revise JPEG support as HDF5 was changed #664

Closed

remove unexpected key from datum_kwargs #655

remove unexpected key from datum_kwargs #655

Uh oh!

Conversation

prjemian commented May 30, 2022

Uh oh!

prjemian commented May 30, 2022

Uh oh!

prjemian commented May 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

prjemian commented May 31, 2022

Uh oh!

prjemian commented May 31, 2022

Uh oh!

gfabbris commented May 31, 2022

Uh oh!

prjemian commented May 31, 2022

Uh oh!

prjemian commented May 31, 2022

Uh oh!

prjemian commented May 31, 2022

Uh oh!

prjemian commented May 31, 2022

Uh oh!

prjemian commented May 31, 2022

Uh oh!

prjemian commented Jun 1, 2022

tests

image file exists as acquired

document content in databroker v1

databroker v2 view of the resource

image file exists

image content found in file

failing test

Uh oh!

prjemian commented Jun 1, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

prjemian commented May 31, 2022 •

edited

Loading