Skip to content

CLI Reference

Complete reference for all KITT commands. This page is auto-generated from the Click decorators in the source code, so it always reflects the current state of the CLI.

KITT exposes 18 command groups through a single entry point:

Group Purpose
run Execute benchmarks against an engine
test List benchmarks, create custom tests
engines List, check, and set up inference engines
results KARR result management, listing, and comparison
campaign Run multi-model, multi-engine campaigns
monitoring Generate and deploy monitoring stacks
stack Generate and manage composable Docker stacks
agent Manage distributed benchmark agents
remote Remote deployment commands
ci CI/CD integration helpers
bot Chat-bot testing utilities
plugin Manage engine and benchmark plugins
charts Generate result charts and visualizations
recommend Hardware and configuration recommendations
storage Manage result storage backends
compare Launch interactive TUI for comparing runs
fingerprint Display hardware fingerprint
web Launch the web dashboard and REST API

kitt

KITT - Kirizan's Inference Testing Tools

End-to-end testing suite for LLM inference engines.

Usage:

kitt [OPTIONS] COMMAND [ARGS]...

Options:

Name Type Description Default
--version boolean Show the version and exit. False
--help boolean Show this message and exit. False

agent

Manage the KITT agent daemon.

These commands proxy to the thin agent (kitt-agent). Install the agent package first: pip install kitt-agent

Usage:

kitt agent [OPTIONS] COMMAND [ARGS]...

Options:

Name Type Description Default
--help boolean Show this message and exit. False
init

Initialize agent configuration.

Proxies to: kitt-agent init

Usage:

kitt agent init [OPTIONS]

Options:

Name Type Description Default
--server text KITT server URL (e.g., https://server:8080) Sentinel.UNSET
--token text Bearer token for authentication (optional) ``
--name text Agent name (defaults to hostname) ``
--port integer Agent listening port 8090
--help boolean Show this message and exit. False
start

Start the KITT agent daemon.

Proxies to: kitt-agent start

Usage:

kitt agent start [OPTIONS]

Options:

Name Type Description Default
--config path Path to agent.yaml Sentinel.UNSET
--insecure boolean Disable TLS verification False
--help boolean Show this message and exit. False
status

Check agent status.

Usage:

kitt agent status [OPTIONS]

Options:

Name Type Description Default
--help boolean Show this message and exit. False
stop

Stop the KITT agent daemon.

Usage:

kitt agent stop [OPTIONS]

Options:

Name Type Description Default
--help boolean Show this message and exit. False
update

Update the agent to the latest version from the server.

Proxies to: kitt-agent update

Usage:

kitt agent update [OPTIONS]

Options:

Name Type Description Default
--config path Path to agent.yaml Sentinel.UNSET
--restart boolean Restart the agent after update False
--help boolean Show this message and exit. False

bot

Chat bot integration commands.

Usage:

kitt bot [OPTIONS] COMMAND [ARGS]...

Options:

Name Type Description Default
--help boolean Show this message and exit. False
config

Show bot configuration info.

Usage:

kitt bot config [OPTIONS]

Options:

Name Type Description Default
--help boolean Show this message and exit. False
start

Start a chat bot.

Usage:

kitt bot start [OPTIONS]

Options:

Name Type Description Default
--platform choice (slack | discord) Bot platform Sentinel.UNSET
--token text Bot token Sentinel.UNSET
--app-token text App token (Slack only) None
--help boolean Show this message and exit. False

campaign

Manage benchmark campaigns.

Usage:

kitt campaign [OPTIONS] COMMAND [ARGS]...

Options:

Name Type Description Default
--help boolean Show this message and exit. False
create

Generate a campaign config from existing benchmark results.

Usage:

kitt campaign create [OPTIONS]

Options:

Name Type Description Default
--from-results path Generate campaign config from existing results directory Sentinel.UNSET
--output, -o text Output YAML path None
--help boolean Show this message and exit. False
cron-status

Show scheduled campaigns.

Usage:

kitt campaign cron-status [OPTIONS]

Options:

Name Type Description Default
--help boolean Show this message and exit. False
list

List all campaigns.

Usage:

kitt campaign list [OPTIONS]

Options:

Name Type Description Default
--help boolean Show this message and exit. False
run

Run a benchmark campaign from a YAML config file.

Usage:

kitt campaign run [OPTIONS] CONFIG_PATH

Options:

Name Type Description Default
--resume boolean Resume a previous campaign run False
--dry-run boolean Print planned runs without executing False
--campaign-id text Explicit campaign ID (for resume) None
--help boolean Show this message and exit. False
schedule

Schedule a campaign to run on a cron schedule.

Usage:

kitt campaign schedule [OPTIONS] CONFIG_PATH

Options:

Name Type Description Default
--cron text Cron expression (e.g. "0 2 * * *") Sentinel.UNSET
--id text Schedule identifier None
--help boolean Show this message and exit. False
status

Show status of a campaign.

Usage:

kitt campaign status [OPTIONS] [CAMPAIGN_ID]

Options:

Name Type Description Default
--help boolean Show this message and exit. False
unschedule

Remove a scheduled campaign.

Usage:

kitt campaign unschedule [OPTIONS] SCHEDULE_ID

Options:

Name Type Description Default
--help boolean Show this message and exit. False
wizard

Interactive campaign builder wizard.

Usage:

kitt campaign wizard [OPTIONS]

Options:

Name Type Description Default
--help boolean Show this message and exit. False

charts

Generate charts and visualizations.

Usage:

kitt charts [OPTIONS] COMMAND [ARGS]...

Options:

Name Type Description Default
--help boolean Show this message and exit. False
quant-curves

Generate quantization quality tradeoff curves.

Usage:

kitt charts quant-curves [OPTIONS]

Options:

Name Type Description Default
--model-family text Filter by model family (e.g. Llama-3) None
--output, -o text Output file path quant_curves.svg
--csv boolean Export data as CSV instead False
--help boolean Show this message and exit. False

ci

CI/CD integration commands.

Usage:

kitt ci [OPTIONS] COMMAND [ARGS]...

Options:

Name Type Description Default
--help boolean Show this message and exit. False
report

Generate CI report from benchmark results.

Usage:

kitt ci report [OPTIONS]

Options:

Name Type Description Default
--results-dir path Results directory Sentinel.UNSET
--baseline-dir path Baseline results for comparison None
--github-token text GitHub API token None
--repo text GitHub repo (owner/repo) None
--pr integer Pull request number None
--output, -o text Write report to file instead of posting None
--help boolean Show this message and exit. False

compare

Launch TUI for comparing benchmark results.

Pass paths to result directories or metrics.json files.

Example: kitt compare ./result-1 ./result-2

Usage:

kitt compare [OPTIONS] RUNS...

Options:

Name Type Description Default
--help boolean Show this message and exit. False

engines

Manage inference engines.

Usage:

kitt engines [OPTIONS] COMMAND [ARGS]...

Options:

Name Type Description Default
--help boolean Show this message and exit. False
check

Check if a specific engine is available and show details.

Usage:

kitt engines check [OPTIONS] ENGINE_NAME

Options:

Name Type Description Default
--help boolean Show this message and exit. False
list

List all registered inference engines and their availability.

Usage:

kitt engines list [OPTIONS]

Options:

Name Type Description Default
--help boolean Show this message and exit. False
profiles

Manage engine configuration profiles.

Usage:

kitt engines profiles [OPTIONS] COMMAND [ARGS]...

Options:

Name Type Description Default
--help boolean Show this message and exit. False
list

List all saved engine profiles.

Usage:

kitt engines profiles list [OPTIONS]

Options:

Name Type Description Default
--engine text Filter profiles by engine name None
--help boolean Show this message and exit. False
show

Show details of a specific engine profile.

Usage:

kitt engines profiles show [OPTIONS] NAME

Options:

Name Type Description Default
--help boolean Show this message and exit. False
setup

Pull or build the Docker image for an engine.

Usage:

kitt engines setup [OPTIONS] ENGINE_NAME

Options:

Name Type Description Default
--dry-run boolean Show commands without executing them False
--force-rebuild boolean Rebuild even if image already exists False
--help boolean Show this message and exit. False
status

Show native engine binary detection for the current system.

Usage:

kitt engines status [OPTIONS]

Options:

Name Type Description Default
--help boolean Show this message and exit. False

fingerprint

Display hardware fingerprint for this system.

Usage:

kitt fingerprint [OPTIONS]

Options:

Name Type Description Default
--verbose boolean Show detailed hardware info False
--help boolean Show this message and exit. False

monitoring

Manage the Grafana/Prometheus monitoring stack.

Usage:

kitt monitoring [OPTIONS] COMMAND [ARGS]...

Options:

Name Type Description Default
--help boolean Show this message and exit. False
deploy

Deploy a generated monitoring stack to a remote host.

Usage:

kitt monitoring deploy [OPTIONS] NAME

Options:

Name Type Description Default
--host text Remote host name (from ~/.kitt/hosts.yaml). Sentinel.UNSET
--help boolean Show this message and exit. False
generate

Generate a customized monitoring stack.

Creates a docker-compose stack at ~/.kitt/monitoring// with Prometheus, Grafana, and InfluxDB configured for the given scrape targets.

Usage:

kitt monitoring generate [OPTIONS] NAME

Options:

Name Type Description Default
-t, --target text Scrape target host:port (repeatable). Sentinel.UNSET
--grafana-port integer Grafana port (default: 3000). 3000
--prometheus-port integer Prometheus port (default: 9090). 9090
--influxdb-port integer InfluxDB port (default: 8086). 8086
--grafana-password text Grafana admin password. kitt
--influxdb-token text InfluxDB admin token. kitt-influx-token
--deploy boolean Deploy to remote host after generation. False
--host text Remote host name (from ~/.kitt/hosts.yaml). None
--help boolean Show this message and exit. False
list-stacks

List all generated monitoring stacks.

Usage:

kitt monitoring list-stacks [OPTIONS]

Options:

Name Type Description Default
--help boolean Show this message and exit. False
remote-start

Start a deployed monitoring stack on a remote host.

Usage:

kitt monitoring remote-start [OPTIONS] NAME

Options:

Name Type Description Default
--host text Remote host name (from ~/.kitt/hosts.yaml). Sentinel.UNSET
--help boolean Show this message and exit. False
remote-status

Check status of a deployed monitoring stack on a remote host.

Usage:

kitt monitoring remote-status [OPTIONS] NAME

Options:

Name Type Description Default
--host text Remote host name (from ~/.kitt/hosts.yaml). Sentinel.UNSET
--help boolean Show this message and exit. False
remote-stop

Stop a deployed monitoring stack on a remote host.

Usage:

kitt monitoring remote-stop [OPTIONS] NAME

Options:

Name Type Description Default
--host text Remote host name (from ~/.kitt/hosts.yaml). Sentinel.UNSET
--help boolean Show this message and exit. False
remove-stack

Remove a monitoring stack configuration.

Usage:

kitt monitoring remove-stack [OPTIONS] NAME

Options:

Name Type Description Default
--delete-files boolean Also delete generated files on disk. False
--help boolean Show this message and exit. False
start

Start the monitoring stack (Prometheus, Grafana, InfluxDB).

Usage:

kitt monitoring start [OPTIONS]

Options:

Name Type Description Default
--compose-dir path Path to docker-compose directory (auto-detected by default). None
--name text Name of a generated monitoring stack to start. None
--help boolean Show this message and exit. False
status

Show monitoring stack status.

Usage:

kitt monitoring status [OPTIONS]

Options:

Name Type Description Default
--compose-dir path Path to docker-compose directory. None
--name text Name of a generated monitoring stack to check. None
--help boolean Show this message and exit. False
stop

Stop the monitoring stack.

Usage:

kitt monitoring stop [OPTIONS]

Options:

Name Type Description Default
--compose-dir path Path to docker-compose directory. None
--name text Name of a generated monitoring stack to stop. None
--help boolean Show this message and exit. False

plugin

Manage KITT plugins.

Usage:

kitt plugin [OPTIONS] COMMAND [ARGS]...

Options:

Name Type Description Default
--help boolean Show this message and exit. False
install

Install a KITT plugin package.

Usage:

kitt plugin install [OPTIONS] PACKAGE

Options:

Name Type Description Default
--version text Version constraint (e.g. '>=0.2.0') None
--upgrade boolean Upgrade if already installed False
--help boolean Show this message and exit. False
list

List installed KITT plugins.

Usage:

kitt plugin list [OPTIONS]

Options:

Name Type Description Default
--help boolean Show this message and exit. False
remove

Remove a KITT plugin package.

Usage:

kitt plugin remove [OPTIONS] PACKAGE

Options:

Name Type Description Default
--help boolean Show this message and exit. False

recommend

Recommend models based on benchmark history.

Usage:

kitt recommend [OPTIONS]

Options:

Name Type Description Default
--max-vram float Maximum VRAM in GB None
--max-ram float Maximum RAM in GB None
--min-throughput float Minimum throughput (tokens/sec) None
--min-accuracy float Minimum accuracy (0-1) None
--max-latency float Maximum latency (ms) None
--engine text Restrict to engine None
--sort choice (score | throughput | accuracy) N/A score
--pareto boolean Show only Pareto-optimal models False
--limit integer Number of recommendations 10
--help boolean Show this message and exit. False

remote

Remote host management and execution.

Usage:

kitt remote [OPTIONS] COMMAND [ARGS]...

Options:

Name Type Description Default
--help boolean Show this message and exit. False
engines

Manage engine images on remote hosts.

Usage:

kitt remote engines [OPTIONS] COMMAND [ARGS]...

Options:

Name Type Description Default
--help boolean Show this message and exit. False
setup

Pull or build an engine image on a remote host.

ENGINE_NAME: Engine to set up (vllm, llama_cpp, ollama, exllamav2)

Usage:

kitt remote engines setup [OPTIONS] ENGINE_NAME

Options:

Name Type Description Default
--host text Remote host name Sentinel.UNSET
--dry-run boolean Show commands without executing them False
--help boolean Show this message and exit. False
list

List configured remote hosts.

Usage:

kitt remote list [OPTIONS]

Options:

Name Type Description Default
--help boolean Show this message and exit. False
logs

View campaign logs from a remote host.

Usage:

kitt remote logs [OPTIONS]

Options:

Name Type Description Default
--host text Remote host name Sentinel.UNSET
--tail integer Number of log lines 50
--help boolean Show this message and exit. False
remove

Remove a configured remote host.

Usage:

kitt remote remove [OPTIONS] NAME

Options:

Name Type Description Default
--help boolean Show this message and exit. False
run

Run a campaign on a remote host.

Usage:

kitt remote run [OPTIONS] CONFIG_PATH

Options:

Name Type Description Default
--host text Remote host name Sentinel.UNSET
--dry-run boolean Dry run mode False
--wait boolean Wait for completion False
--help boolean Show this message and exit. False
setup

Set up a remote host for KITT execution.

HOST_SPEC: user@hostname or hostname

Usage:

kitt remote setup [OPTIONS] HOST_SPEC

Options:

Name Type Description Default
--name text Friendly name for the host None
--ssh-key text Path to SSH key None
--no-install boolean Skip KITT installation False
--help boolean Show this message and exit. False
status

Check campaign status on a remote host.

Usage:

kitt remote status [OPTIONS]

Options:

Name Type Description Default
--host text Remote host name Sentinel.UNSET
--help boolean Show this message and exit. False
sync

Sync results from a remote host.

Usage:

kitt remote sync [OPTIONS]

Options:

Name Type Description Default
--host text Remote host name Sentinel.UNSET
--output, -o path Local results directory None
--help boolean Show this message and exit. False
test

Test connectivity to a remote host.

Usage:

kitt remote test [OPTIONS] NAME

Options:

Name Type Description Default
--help boolean Show this message and exit. False

results

Manage benchmark results.

Usage:

kitt results [OPTIONS] COMMAND [ARGS]...

Options:

Name Type Description Default
--help boolean Show this message and exit. False
cleanup

Clean up old Git LFS objects to reduce repository size.

Usage:

kitt results cleanup [OPTIONS]

Options:

Name Type Description Default
--repo path Results repository path Sentinel.UNSET
--days integer Keep objects from last N days 90
--dry-run boolean Show what would be deleted False
--help boolean Show this message and exit. False
compare

Compare benchmark results from different runs.

Usage:

kitt results compare [OPTIONS] RUN1 RUN2

Options:

Name Type Description Default
--additional text Additional runs to compare Sentinel.UNSET
--format choice (table | json) N/A table
--help boolean Show this message and exit. False
import

Import results from a directory into a KARR repo.

Usage:

kitt results import [OPTIONS] SOURCE

Options:

Name Type Description Default
--karr path KARR repo to import into Sentinel.UNSET
--help boolean Show this message and exit. False
init

Initialize a new KARR results repository.

Usage:

kitt results init [OPTIONS]

Options:

Name Type Description Default
--path, -p path Path for KARR repo Sentinel.UNSET
--help boolean Show this message and exit. False
list

List local benchmark results.

Usage:

kitt results list [OPTIONS]

Options:

Name Type Description Default
--model text Filter by model Sentinel.UNSET
--engine text Filter by engine Sentinel.UNSET
--karr path Path to KARR repo Sentinel.UNSET
--help boolean Show this message and exit. False
submit

Submit results via pull request.

Usage:

kitt results submit [OPTIONS]

Options:

Name Type Description Default
--repo path Results repository path Sentinel.UNSET
--help boolean Show this message and exit. False

run

Run benchmarks against a model using a specified engine.

Usage:

kitt run [OPTIONS]

Options:

Name Type Description Default
--model, -m text Path to model or model identifier Sentinel.UNSET
--engine, -e text Inference engine to use (vllm, llama_cpp, ollama, exllamav2, mlx) Sentinel.UNSET
--suite, -s text Test suite to run (quick, standard, performance) quick
--output, -o path Output directory for results Sentinel.UNSET
--skip-warmup boolean Skip warmup phase for all benchmarks False
--runs integer Override number of runs per benchmark None
--config path Path to custom configuration file Sentinel.UNSET
--store-karr boolean Store results in KARR repository False
--auto-pull boolean Automatically pull/build engine image if not available False
--mode choice (docker | native) Engine execution mode (docker or native). Uses engine default if not set. None
--help boolean Show this message and exit. False

stack

Generate and manage composable Docker deployment stacks.

Usage:

kitt stack [OPTIONS] COMMAND [ARGS]...

Options:

Name Type Description Default
--help boolean Show this message and exit. False
generate

Generate a composable Docker deployment stack.

Creates a docker-compose stack at ~/.kitt/stacks// with the selected components.

Usage:

kitt stack generate [OPTIONS] NAME

Options:

Name Type Description Default
--web boolean Include web UI + REST API service. False
--reporting boolean Include lightweight read-only dashboard. False
--agent boolean Include agent daemon for GPU servers. False
--postgres boolean Include PostgreSQL database. False
--monitoring boolean Include Prometheus + Grafana + InfluxDB. False
--port integer Web/reporting port (default: 8080). 8080
--agent-port integer Agent port (default: 8090). 8090
--postgres-port integer PostgreSQL port (default: 5432). 5432
--grafana-port integer Grafana port (default: 3000). 3000
--prometheus-port integer Prometheus port (default: 9090). 9090
--influxdb-port integer InfluxDB port (default: 8086). 8086
--auth-token text Bearer token for API auth. changeme
--secret-key text Flask secret key. ``
--postgres-password text PostgreSQL password (default: kitt). kitt
--server-url text KITT server URL (for agent registration). ``
--help boolean Show this message and exit. False
list

List all generated stacks.

Usage:

kitt stack list [OPTIONS]

Options:

Name Type Description Default
--help boolean Show this message and exit. False
remove

Remove a stack configuration.

Usage:

kitt stack remove [OPTIONS] NAME

Options:

Name Type Description Default
--delete-files boolean Also delete generated files on disk. False
--help boolean Show this message and exit. False
start

Start a generated stack.

Usage:

kitt stack start [OPTIONS]

Options:

Name Type Description Default
--name text Name of the stack to start. Sentinel.UNSET
--help boolean Show this message and exit. False
status

Show stack status.

Usage:

kitt stack status [OPTIONS]

Options:

Name Type Description Default
--name text Name of the stack to check. Sentinel.UNSET
--help boolean Show this message and exit. False
stop

Stop a running stack.

Usage:

kitt stack stop [OPTIONS]

Options:

Name Type Description Default
--name text Name of the stack to stop. Sentinel.UNSET
--help boolean Show this message and exit. False

storage

Manage the KITT result storage backend.

Usage:

kitt storage [OPTIONS] COMMAND [ARGS]...

Options:

Name Type Description Default
--help boolean Show this message and exit. False
export

Export a result from the database to a JSON file.

Usage:

kitt storage export [OPTIONS] RESULT_ID

Options:

Name Type Description Default
--output, -o path Output JSON path Sentinel.UNSET
--db-path path N/A None
--help boolean Show this message and exit. False
import

Import results from a directory or JSON file into the database.

Usage:

kitt storage import [OPTIONS] SOURCE

Options:

Name Type Description Default
--db-path path SQLite database path None
--help boolean Show this message and exit. False
init

Initialize the SQLite storage database.

Usage:

kitt storage init [OPTIONS]

Options:

Name Type Description Default
--db-path path SQLite database path (default: ~/.kitt/kitt.db) None
--help boolean Show this message and exit. False
list

List results stored in the database.

Usage:

kitt storage list [OPTIONS]

Options:

Name Type Description Default
--db-path path N/A None
--model text Filter by model None
--engine text Filter by engine None
--limit integer Max results to show 50
--help boolean Show this message and exit. False
migrate

Run pending database migrations.

Usage:

kitt storage migrate [OPTIONS]

Options:

Name Type Description Default
--db-path path SQLite database path None
--help boolean Show this message and exit. False
stats

Show storage statistics.

Usage:

kitt storage stats [OPTIONS]

Options:

Name Type Description Default
--db-path path N/A None
--help boolean Show this message and exit. False

test

Manage benchmarks and test definitions.

Usage:

kitt test [OPTIONS] COMMAND [ARGS]...

Options:

Name Type Description Default
--help boolean Show this message and exit. False
list

List available benchmarks.

Usage:

kitt test list [OPTIONS]

Options:

Name Type Description Default
--category, -c text Filter by category Sentinel.UNSET
--help boolean Show this message and exit. False
new

Create a new benchmark definition from template.

Usage:

kitt test new [OPTIONS] NAME

Options:

Name Type Description Default
--category, -c text Benchmark category quality_custom
--help boolean Show this message and exit. False

web

Launch web dashboard for viewing results.

Usage:

kitt web [OPTIONS]

Options:

Name Type Description Default
--port integer Port for web UI 8080
--host text Host to bind to 0.0.0.0
--results-dir path Results directory Sentinel.UNSET
--debug boolean Enable debug mode False
--legacy boolean Use legacy read-only dashboard False
--insecure boolean Disable TLS (development only) False
--tls-cert path Path to TLS certificate Sentinel.UNSET
--tls-key path Path to TLS private key Sentinel.UNSET
--tls-ca path Path to CA certificate Sentinel.UNSET
--auth-token text Bearer token for API auth Sentinel.UNSET
--help boolean Show this message and exit. False