This package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.
Orchestrator CLI is a replacement of the Dispatch CLI. Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.
bf orchestrator
bf orchestrator:add
bf orchestrator:basemodel:get
bf orchestrator:basemodel:list
bf orchestrator:build
bf orchestrator:create
bf orchestrator:interactive
bf orchestrator:query
bf orchestrator:remove
bf orchestrator:test
Display Orchestrator CLI available commands
USAGE
$ bf orchestrator
OPTIONS
-h, --help Orchestrator commands help
See code: src/commands/orchestrator/index.ts
Add examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.
USAGE
$ bf orchestrator:add
OPTIONS
-d, --debug
-e, --entityModel=entityModel Path to Orchestrator entity base model directory.
-f, --force If --out flag is provided with the path to an existing file, overwrites that file.
-h, --help Orchestrator add command help
-i, --in=in Path to example file (.lu/.qna/.json/.blu).
-k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.
-m, --model=model Path to Orchestrator model directory.
-o, --out=out Path where generated Orchestrator example file will be placed. Default to current
working directory.
-t, --type=type Type of input (luis/qna/file).
-v, --version=version Applies only for type=luis, LUIS app version
--dialog Generate multi language or cross train Orchestrator recognizers.
--endpoint=endpoint LUIS/QnAMaker endpoint.
--id=id LUIS app id or QnAMaker kb id if type = luis/qna.
--routingName=routingName Routing name, default to file name.
EXAMPLE
$ bf orchestrator:add
$ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/
$ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/
$ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory
$ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather
--endpoint
$ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb
See code: src/commands/orchestrator/add.ts
Gets Orchestrator base model
USAGE
$ bf orchestrator:basemodel:get
OPTIONS
-d, --debug
-h, --help Orchestrator basemodel:get command help
-o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working
directory.
-v, --verbose Enable verbose logging
--getEntity Optional. Download default entity model at the same time, which will be placed in the entity
subfolder of the output path.
--versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If
not specified, default model will be downloaded.
See code: src/commands/orchestrator/basemodel/get.ts
Lists all Orchestrator base model versions
USAGE
$ bf orchestrator:basemodel:list
OPTIONS
-h, --help Orchestrator basemodel:list command help
-r, --raw Optional. Raw output
--all Optional. Display all models
See code: src/commands/orchestrator/basemodel/list.ts
Creates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.
USAGE
$ bf orchestrator:build
OPTIONS
-d, --debug
-e, --entityModel=entityModel Path to Orchestrator entity base model directory.
-h, --help Orchestrator build command help
-i, --in=in Path to lu file or folder with lu files.
-m, --model=model Path to Orchestrator model.
-o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current
working directory.
--dialog Generate multi language or cross train Orchestrator recognizers.
--luconfig=luconfig Path to luconfig.json.
See code: src/commands/orchestrator/build.ts
Create orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules
USAGE
$ bf orchestrator:create
OPTIONS
-d, --debug
-e, --entityModel=entityModel Path to Orchestrator entity base model directory.
-h, --help Orchestrator create command help
-i, --in=in The path to source label files from where orchestrator example file will be created
from. Default to current working directory.
-m, --model=model Path to Orchestrator base model directory.
-o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current
working directory.
--hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will
contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).
--refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.
See code: src/commands/orchestrator/create.ts
Real-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples
USAGE
$ bf orchestrator:interactive
OPTIONS
-d, --debug
-e, --entityModel=entityModel Path to Orchestrator entity base model directory.
-h, --help show CLI help
-l, --in=in Optional path to a previously created Orchestrator .blu file.
-m, --model=model (required) Directory or hosting Orchestrator config and base model files.
-o, --out=out Optional Directory where analysis and output files will be placed.
EXAMPLE
$ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/
--model=./path/to/model/directory
See code: src/commands/orchestrator/interactive.ts
Query Orchestrator base model and a snapshot/train file
USAGE
$ bf orchestrator:query
OPTIONS
-d, --debug
-e, --entityModel=entityModel Path to Orchestrator entity base model directory.
-h, --help show CLI help
-i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).
-l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing
all predictions.
-m, --model=model (required) Path to Orchestrator base model directory.
-q, --query=query (required) Query string to predict.
EXAMPLE
$ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory
See code: src/commands/orchestrator/query.ts
Remove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.
USAGE
$ bf orchestrator:remove
OPTIONS
-d, --debug
-h, --help Orchestrator remove command help
-i, --in=in Path to example file (.lu/.qna/.json/.blu).
-t, --type=type Type of input (luis/qna/file).
--id=id LUIS app id or QnAMaker kb id if type = luis/qna.
EXAMPLE
$ bf orchestrator:remove
$ bf orchestrator:remove -t luis --id LUIS_APP_ID
$ bf orchestrator:remove -t qna --id QNA_KB
$ bf orchestrator:remove -t file -i FILE_PATH
See code: src/commands/orchestrator/remove.ts
The "test" command can operate in three modes: test, evaluation, assessment.
USAGE
$ bf orchestrator:test
OPTIONS
-d, --debug
-e, --entityModel=entityModel Path to Orchestrator entity base model directory.
-h, --help show CLI help
-i, --in=in (required) Path to a previously created Orchestrator .blu file.
-m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for
the "assessment" mode.
-o, --out=out (required) Directory where analysis and output files will be placed.
-p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of
(e.g., crosss-valiaton) files.
-t, --test=test Optional path to a test file. This option enable the "test" mode.
DESCRIPTION
1) Test mode: test a collection of utterance/label samples loaded from a test file against
a previously generated Orchestrator .blu snapshot/train file,
and create a detailed train/test evaluation report.
2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report
on a previously generated Orchestrator .blu snapshot/train file.
3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and
create an evaluation report. This mode can evaluate predictions produced by
other NLP or machine learning systems. There is no need for an Orchestrator base model.
Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,
and scenarios if a user can carefully construct the prediction and grounf-truth files by
the specification detailed below.
Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which
is the basis of all the features feeding to a ML model. For other ML systems, the key to
a data instance can be built directly from the features and put in place of text
in a prediction and ground-truth file.
The 'test' mode is activated if there is a '--test' argument set for a test file.
The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.
If there is no '--test' or '--prediction' arguments, then "test" command runs on the 'evaluation' mode.
EXAMPLE
$ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/
--model=./path/to/model/directory
$ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file
--out=./path/to/output/folder/
$ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/
[--model=./path/to/model/directory]
See code: src/commands/orchestrator/test.ts
Most of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after bf orchestrator:basemodel:get --out <BASE_MODEL_PATH>
, subsequent command (ie bf orchestrator:create) will no longer need to supply the "--model" parameter.