Automated Input Processing

The Automated Input Processing Extension can Process Data from an Input source on an regularely basis into CiviCRM. The existing Implementation enables you to search for new csv-files with a regex pattern on a regularly basis (eg. 1x per day / week/ ..) and make an APIv3-Call (eg. to a Formprocessor) for every Line of the CSV-File.

• a Finder checks if a new Input from an Input Souce is available
  • current implementation of DropFolderFinder checks for files with an regex pattern on a server folder
• a Reader reads an Input Source and calls a Processor for every element
  • current implementation of CSV-Reader reads an CSV-File and calls a Processor for every Line-Element
• a Processor processes a given element - typically to import it
  • current impolemenation of APIv3-Processor calls an CiviCRM-APIv3-API with one element to import it (eg. calls an FormProcessor)
• These import-Tasks can be scheduled with a Scheduled Job in CiviCRM to be performed on a regularly basis (eg. 1x per day / week/ ..)
• to avoid running into php-timeouts AIP can be configured with a time-limit and store (remember) where it stopped processing an input source (eg. which line of a file).
• offers a User-Interface to configure new and edit existing Input-Processing-Tasks

• PHP v7.4+
• CiviCRM 5.50+

Sysadmins and developers may download the .zip file for this extension and install it with the command-line tool cv.

cd <extension-dir>
cv dl aip@https://github.com/systopia/aip/archive/master.zip
cv en aip

to configure AIP to read a csv-file from a folder, process every line and perform an API-Call to the Formprocessor perform the following steps:

You have to create an inbox, processing, processed, uploading and failed folder:

cd /path/to/folder
mkdir inbox
mkdir processing
mkdir processed
mkdir failed
mkdir uploading
touch processing.log

then create a new AIP Process under Administer → AIP Configuration (civicrm/aip_configuration) and paste the following JSON into its config field:

{
  "finder": {
    "class": "Civi\\AIP\\Finder\\DropFolderFinder",
    "filter": { "file_name": "#your_input_file_pattern.*[.]csv#" },
    "folder": {
      "inbox":      "/path/to/folder/inbox",
      "processing": "/path/to/folder/processing",
      "processed":  "/path/to/folder/processed",
      "uploading":  "/path/to/folder/uploading",
      "failed":     "/path/to/folder/failed"
    }
  },
  "reader": {
    "class": "Civi\\AIP\\Reader\\CSV",
    "csv_string_encoding": "UTF8",
    "skip_empty_lines": 1
  },
  "processor": {
    "class": "Civi\\AIP\\Processor\\Api3",
    "api_entity": "FormProcessor",
    "api_action": "your_formprocessor",
    "trim_parameters": "all"
  },
  "process": {
    "log": { "file": "/path/to/folder/processing.log", "level": "info" },
    "processing_limit": { "php_process_time": 590 }
  }
}

You have to adjust the configuration to your needs. Here is a short explaination of the most important configuration options:

• finder.filter.file_name: a regex pattern of the filenames to import
• finder.folder.inbox: the folder to search for files to process (inbox)
• finder.folder.processing: the folder AIP moves files to for processing
• finder.folder.failed: the folder AIP moves files if the processing of the file failed (eg. API-call response with an error)
• finder.folder.processed: the folder AIP moves the files if processing was successfull
• finder.folder.uploading: if you use an external script to push you .csv-files to the server you should first upload them to this folder and then move them to the inbox to prevent AIP from starting to run on a file that is not fully uploaded
• reader.csv_string_encoding: encoding of the csv-file (eg. UTF8)
• reader.skip_empty_lines: set to 1 to skip empty lines in the csv file. Set to 0 to consider empty lines in the csv-file as an error.
• processor.api_entity: the api-entity AIP will call for every element (line of the csv-file)
• processor.api_action: the api-action AIP will call for every element (line of the csv-file)
• process.log.file: path of a logfile AIP will write some infos into
• process.log.level: LOG-Level of AIP. Defines how much will be written into the logfile
• process.process_limit.php_process_time: AIP will not continue with the next element after php_process_time since the start. AIP will store where it stopped and continue next time it is called. Prevents PHP-Timeouts.

To enable scheduled import-jobs go to /civicrm/admin/job?action=add&reset=1:

• set a name of the job
• configure desired frequency of execution (keep in mind, that AIP might need more than one execution to process all data)
• API-Call: AIP.run_process
• Parameters: list of process-ids (id of mysql-column civicrm_aip_process) eg.: pid=1 or pid=1,2,3

Available under Administer → AIP Configuration (route civicrm/aip_configuration). The config field expects the same JSON as in the SQL example above. Folders still have to be created manually.

AIP gives you the following feedback options to see if the import(s) were working fine: To see, if all files were processed as expected:

• check the folders configured in the finder.folder.processed for files successfully processed.
• check the folders configured in the finder.folder.failed for files failed to process (you could also write a cronjob to email you if there is a file in this folder).

To check, what has possibly gone wrong:

• check the Logs of the CiviCRM Scheduled Job
• check civicrm_aip_error_log mysql-table or AIP_Error_Log APIv4-Entity with Searchkit

To Debug behaviour in detail:

• check the log-file defined in this configuration parameter: process.log.file (typically processing.log)
• install civicrm api-logger an analyse the api-calls

This CiviCRM extension is provided as Free and Open Source Software, and we are happy if you find it useful. However, we have put a lot of work into it (and continue to do so), much of it unpaid for. So if you benefit from our software, please consider making a financial contribution so we can continue to maintain and develop it further.

If you are willing to support us in developing this CiviCRM extension, please send an email to info@systopia.de to get an invoice or agree a different payment method. Thank you!