Improve preset creation harness and CLI output

A preset claims to be a verified serving configuration, but three of the
properties that decide what "verified" means were optional or defaulted, so
two creations with the same configuration could produce very different
artifacts.

- `max_ttft` and `min_context_length` are now required. Without a latency
  bound, maximising throughput has a degenerate optimum; without a context
  floor, one creation served 64K where another served 1M under identical
  constraints.
- `concurrency` is now required rather than defaulting to 8.
- `trials` in the configuration, `trials_num` in the constraints the agent
  reads, since it is the number of trials rather than a ceiling.
- `baseline: true` makes the first trial a reference point rather than an
  optimization attempt.
- `input_tokens`/`output_tokens` pin the benchmark workload so trials and the
  final service are comparable; both default to 1024.
- rename the config's `context_length` to `min_context_length`, since it is a
  requirement, and keep `context_length` for the measured value.

Per-user speed is now the steady decode rate, `1/TPOT`, as the serving
literature defines it. Dividing aggregate throughput by concurrency folded in
time to first token and read about 9% low. Both display paths now use the same
definition; they previously disagreed.

The trial record gains `learned`, required for every trial, and `failed` for a
benchmark that broke a constraint. A failed trial keeps its benchmark, since
that is what the next trial learns from, and is excluded from best-trial
selection.

`findings.md` is removed. It asked the agent to enumerate what it had not
tried, which is unbounded and produced an arbitrary subset presented as
complete. What a trial taught now lives on the trial record.

Trials themselves are measured more honestly:

- record the largest context each trial handles, found by sending real requests
- run the final benchmark inside the service replica, directly against the
  engine, so it is comparable with the trial benchmarks
- require that a benchmark not reuse the previous one's prompts, which had been
  inflating later trials through the engine's prefix cache
- record final service attempts in `verifications.jsonl` and mirror them out, so
  the CLI reads the phase instead of inferring it from a spent trial budget

Also fixes a real bug: session constraints were read from the agent workspace,
which is deleted when a session finishes, so a verified preset could never show
what it was created against.

`dstack preset` output is reworked: `ps`-style filtering, CONSTRAINTS and
BENCHMARK columns, and one sparkline glyph per trial. The constraints are
dimmed so the measurement leads, and a trial that broke a constraint is marked
with a yellow bar. Maintainer notes written as `<!--!...-->` are stripped from
the rendered agent prompt.

The `endpoints` to `presets` rename in #4058 deleted two docs pages without
adding redirects, so `/docs/concepts/endpoints/` and
`/docs/reference/cli/dstack/endpoint/` returned 404. Both now redirect to
their `preset` equivalents.

`shared_prefix_tokens` is documented on the concepts page. It existed only in
the generated schema reference, so the property that decides the benchmark's
prefix cache hit rate was invisible to anyone reading the concept.

A run where no trial met the constraints showed neither hardware nor a number,
because both were read from the best trial and a failed one cannot become best.
Such a run now shows its fastest failed benchmark, dimmed and marked `*` so it
does not read as a result where styling is absent. That is the answer such a run
produced: `ttft=4.3s` against a 675ms bound is why a card is unusable.

The listing always shows `prefix`, including `prefix=0%`. It decides how much of
each request the engine serves from its prefix cache, so two rows are only
comparable when it matches.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: Andrey Cheptsov

mkdocs.yml

@@ -103,6 +103,8 @@ plugins:
         "docs/tasks.md": "docs/concepts/tasks.md"
         "docs/services.md": "docs/concepts/services.md"
         "docs/fleets.md": "docs/concepts/fleets.md"
+        "docs/concepts/endpoints.md": "docs/concepts/presets.md"
+        "docs/reference/cli/dstack/endpoint.md": "docs/reference/cli/dstack/preset.md"
         "docs/examples/llms/llama31.md": "docs/examples/inference/vllm.md"
         "docs/examples/llms/llama32.md": "docs/examples/inference/vllm.md"
         "docs/examples/llms/qwen36.md": "docs/examples/models/qwen36.md"

mkdocs/docs/concepts/presets.md

@@ -5,7 +5,7 @@ description: Creating and reusing optimized model inference configurations
 
 # Presets
 
-A preset configuration lets you use an agent to create a preset: a validated and optimized model inference configuration. Once created, the preset can be reused to deploy model inference on validated hardware without an agent.
+A preset configuration lets you use an agent to create a preset: a verified and optimized model inference configuration. Once created, the preset can be reused to deploy model inference on verified hardware without an agent.
 
 The value of presets comes from combining two fundamental features: agent-driven model inference optimization and the `dstack` [service](services.md) primitive, which can deploy model inference to any cloud, Kubernetes, or on-prem cluster.
 
@@ -25,14 +25,26 @@ The filename must end with `.dstack.yml` (e.g. `.dstack.yml` or `preset.dstack.y
 
 ```yaml
 type: preset
-name: qwen25-7b
+name: dsv4-flash
 
 # The agent picks a compatible variant of the base model
-base: Qwen/Qwen2.5-7B-Instruct
+base: deepseek-ai/DeepSeek-V4-Flash
 
 # The number of benchmarked trials
-max_trials: 3
+trials: 5
 
+# The requirements the preset must meet (time to first token is in milliseconds)
+min_context_length: 1048576
+max_ttft: 675
+
+# The number of simultaneous requests every benchmark uses
+concurrency: 1
+
+# The request shape every benchmark uses (defaults to 1024 and 1024)
+input_tokens: 10000
+output_tokens: 1500
+
+# The environment variables the agent may pass to runs
 env:
   - HF_TOKEN
 ```
@@ -44,19 +56,23 @@ To create the preset, pass the configuration to the `dstack preset create` comma
 <div class="termy">
 
 ```shell
-$ dstack preset create -f preset.dstack.yml
-Create the preset qwen25-7b? [y/n]: y
-[2026-07-15 11:32:01] Starting preset creation for Qwen/Qwen2.5-7B-Instruct. Allowed fleets: gpu-fleet.
-[2026-07-15 11:41:06] Prototype task qwen25-7b-a1b2c3-2 verified vLLM on an L4:24GB.
-[2026-07-15 11:52:06] Final service qwen25-7b-a1b2c3-3 verified with context length 32768.
-[2026-07-15 11:52:18] Benchmark via guidellm 0.7.1: 32/32 requests succeeded.
+$ dstack preset create -f preset.dstack.yml --fleet b200-fleet
+Create the preset dsv4-flash? [y/n]: y
+[2026-08-04 11:38:34] Starting preset creation for deepseek-ai/DeepSeek-V4-Flash. Allowed fleets: b200-fleet.
+[2026-08-04 12:31:19] Trial 3 switched from vLLM to SGLang: 319 tok/s per user, 2.2x the baseline.
+[2026-08-04 13:04:52] Final service dsv4-flash-c83375b4-4 verified with context length 1048576.
+[2026-08-04 13:12:07] Benchmark via sglang.bench_serving: 32/32 requests succeeded.
 ```
 
 </div>
 
-The command executes entirely locally and uses the locally installed `claude` CLI along with `dstack`'s bundled skills. The agent uses a `dstack` task to find the best serving configuration for the available fleet offers, then submits it as a `dstack` service for a final benchmark. The validated preset is saved locally under `~/.dstack/presets`.
+> It's highly recommended to specify the exact hardware you want the preset to use, so that the
+> optimization is done against that hardware. Point `dstack preset create` to a fleet configured
+> correspondingly, via `fleets` inside the preset configuration or via `--fleet` in the CLI.
+
+The command executes entirely locally and uses the locally installed `claude` CLI along with `dstack`'s bundled skills. The agent uses a `dstack` task to find the best serving configuration for the available fleet offers, then submits it as a `dstack` service for a final benchmark.
 
-You can stop watching with <kbd>Ctrl+C</kbd> at any time. The agent keeps running, and `dstack preset logs -f` follows it again. Resume an interrupted creation with `dstack preset create --resume`:
+You can stop watching with `Ctrl`+`C` at any time. The agent keeps running, and `dstack preset logs -f` follows it again. Resume an interrupted creation with `dstack preset create --resume`:
 
 <div class="termy">
 
@@ -66,26 +82,40 @@ $ dstack preset create -f preset.dstack.yml --resume a1b2c3d4
 
 </div>
 
+When resuming, the constraints are read from the original session, not from the configuration file. Editing them and resuming has no effect. To change any of them, create a new preset.
+
 To stop a creation and its runs, use `dstack preset stop`.
 
-!!! info "Claude configuration"
+??? info "Claude configuration"
     By default, preset creation uses the existing `claude` login. To use an Anthropic API key instead, set:
 
     ```shell
     export DSTACK_AGENT_ANTHROPIC_API_KEY=...
     ```
 
-    By default, the agent uses `claude-opus-4-8` and the default `claude` CLI effort. To override them, set:
+    By default, the agent uses `claude-opus-4-8`. It doesn't set an effort level, so the `claude` CLI default applies. To override them, set:
 
     ```shell
-    export DSTACK_AGENT_ANTHROPIC_MODEL=claude-fable-5
-    export DSTACK_AGENT_CLAUDE_EFFORT=high
+    export DSTACK_AGENT_ANTHROPIC_MODEL=claude-opus-5
+    export DSTACK_AGENT_CLAUDE_EFFORT=max
     ```
 
     Supported effort levels are `low`, `medium`, `high`, `xhigh`, and `max`.
 
+??? info "Presets directory"
+    The verified presets are saved locally under `~/.dstack/presets`, and `dstack preset` reads them from there. Presets aren't stored on the server.
+
 ## Configuration options
 
+### Fleets
+
+Set `fleets` to restrict creation and reuse to specific [fleets](fleets.md). It's highly recommended to specify a fleet with exactly the hardware that you'd like the preset to use.
+
+Alternatively, pass `--fleet` to `dstack preset create` or `dstack preset apply`.
+
+> Profile settings such as `spot_policy`, `max_price`, and `backends` are ignored during preset
+> creation. Configure them on the fleet instead.
+
 ### Model
 
 === "Base"
@@ -104,21 +134,27 @@ To stop a creation and its runs, use `dstack preset stop`.
     repo: Qwen/Qwen2.5-7B-Instruct
     ```
 
-### Trials
+### Shared prefix
 
-`max_trials` is required and sets how many benchmarked trials the agent runs before promoting the best one. Set `concurrency` to control the benchmark concurrency.
+By default every request is unique, so the cache hit rate is near zero. Set `shared_prefix_tokens` to control how much of each request the serving framework can serve from its prefix cache.
 
-### Context length
+<div editor-title="preset.dstack.yml">
 
-Set `context_length` to require a minimum supported context length.
+```yaml
+input_tokens: 8192
+output_tokens: 1024
 
-### Fleets
+# Roughly 90% of prompt tokens can be served from cache
+shared_prefix_tokens: 7360
+```
 
-Set `fleets` to restrict creation and reuse to specific [fleets](fleets.md). Placement properties such as `backends`, `max_price`, and `spot_policy` constrain both creation and reuse too.
+</div>
+
+The `shared_prefix_tokens` value is the part of `input_tokens` that is identical across requests, such as a system prompt or conversation history, and must be less than `input_tokens`.
 
 ### Prompt
 
-Set `prompt` to guide the agent with custom objectives, target metrics, or an experimentation approach. It accepts inline text or a file `path`.
+The `prompt` property is optional. Set it to guide the agent with custom objectives, target metrics, or an experimentation approach. It accepts inline text or a file `path`.
 
 <div editor-title="preset.dstack.yml">
 
@@ -129,6 +165,10 @@ prompt: |
 
 </div>
 
+### Baseline
+
+Set `baseline: true` to make the first trial a baseline: the agent serves the model the way the chosen serving framework recommends, without tuning it for performance. Later trials are optimization attempts.
+
 !!! info "Reference"
     The `preset` configuration supports many more options. See the [`.dstack.yml` reference](../reference/dstack.yml/preset.md).
 
@@ -139,20 +179,23 @@ To deploy a preset as a service, pass the preset configuration and the preset ID
 <div class="termy">
 
 ```shell
-$ dstack preset apply -f preset.dstack.yml --id 532f3f4b
+$ dstack preset apply -f preset.dstack.yml --id c83375b4
  Project        main
  User           admin
  Type           service
- Resources      cpu=2.. mem=8GB.. disk=100GB.. gpu=RTXPRO4500:32GB:1..
+ Resources      cpu=8.. mem=64GB.. disk=500GB gpu=B200:180GB:2
  Spot policy    on-demand
  Max price      off
- Model          Qwen/Qwen3.5-27B (base)
- Preset         532f3f4b (ctx=8K con=8 387 tok/s TTFT 582ms)
+ Retry policy   off
+ Idle duration  5m
+ Max duration   off
+ Model          deepseek-ai/DeepSeek-V4-Flash (base)
+ Preset         c83375b4 (io=10000/1500 conc=1 tok/s/user=309 tok/s=296 ttft=213ms ctx=1M)
 
- #  BACKEND           RESOURCES                                         INSTANCE TYPE                  PRICE
- 1  runpod (EU-RO-1)  cpu=12 mem=54GB disk=100GB gpu=RTXPRO4500:32GB:1  NVIDIA RTX PRO 4500 Blackwell  $0.74
+ #  BACKEND           RESOURCES                                    INSTANCE TYPE  PRICE
+ 1  runpod (US-CA-2)  cpu=48 mem=502GB disk=500GB gpu=B200:180GB:2  NVIDIA B200    $11.78
 
-Submit the run qwen35-27b? [y/n]: y
+Submit the run dsv4-flash? [y/n]: y
 ```
 
 </div>
@@ -167,20 +210,32 @@ Use `dstack preset` to list presets:
 
 ```shell
 $ dstack preset list
- BASE               ID        GPU                  BENCHMARK                    STATUS              SUBMITTED     NAME
- Qwen/Qwen2.5-0.5B
-                    bc592b38                                                    clauding (0/3)      23 sec ago    qwen05
- Qwen/Qwen3-32B
-                    f91d6b60  RTX5090:32GB:1       con=8 576 tok/s TTFT 368ms   verified (10/10)    2 days ago    qwen3-32b
- Qwen/Qwen3.5-27B
-                    3c4d5e6f                                                    verifying (3/3)     2 min ago     qwen35-27b-2
-                    532f3f4b  RTXPRO4500:32GB:1..  con=8 387 tok/s TTFT 582ms   verified (4/4)      yesterday     qwen35-27b
-                    d1c2e12b  RTX5090:32GB:1       con=8 266 tok/s TTFT 2.15s   verified (7/10)     yesterday
+ ID        BASE                           GPU           CONSTRAINTS           BENCHMARK                                STATUS          SUBMITTED
+ c83375b4  deepseek-ai/DeepSeek-V4-Flash  B200:180GB:2  io=10000/1500 conc=1  tok/s/user=309 ttft=213ms ctx=1M  ▂▁██▇  trialing (5/5)  2 min ago
 ```
 
 </div>
 
-Presets are grouped by base model. In-progress creations appear too, with a live status like `clauding` or `verifying`. Pass `-w` to watch in realtime. Pass `-v` to include validation resources and all benchmark metrics, or `--json` for complete preset objects. Filter with `--base` or `--repo`.
+By default, `dstack preset` shows creations that are still running, or the most recent one if none are. Pass `-a` to show every preset, or `-n` to show the last N:
+
+<div class="termy">
+
+```shell
+$ dstack preset list -a
+ ID        BASE                            GPU              CONSTRAINTS           BENCHMARK                                     STATUS          SUBMITTED
+ c83375b4  deepseek-ai/DeepSeek-V4-Flash   B200:180GB:2     io=10000/1500 conc=1  tok/s/user=309 ttft=213ms ctx=1M   ▂▁██▇     trialing (5/5)  2 min ago
+ 092c792b  Qwen/Qwen3.5-397B-A17B          RTXPRO6000:4     io=8K/1K conc=64      tok/s/user=19.6 ttft=3.43s ctx=32K ▁▂▅▇█··   verified (7)    3 days ago
+ 9ab0fa65  Qwen/Qwen3.6-27B                RTXPRO4500:1     io=1K/1K conc=8       tok/s/user=57.1 ttft=499ms ctx=128K ▁▄██▆·█  verified (7)    4 days ago
+ f91d6b60  Qwen/Qwen3-32B                  RTX5090:32GB:1   io=1K/512 conc=8      tok/s/user=85.8 ttft=368ms ctx=32K ▁▁▅▅▄▅▇▄▇█ verified (10)  2 weeks ago
+```
+
+</div>
+
+The `CONSTRAINTS` column is what the creation was asked for, and `BENCHMARK` is the best trial so far. `tok/s/user` is the steady decode rate, measured as one second divided by the median time per output token, so it excludes the time to the first token.
+
+The glyphs after the benchmark are one per trial: height is throughput, a yellow bar is a trial whose benchmark broke a constraint, and a red `·` is one that produced no benchmark at all. The shape shows whether a run converged or wandered.
+
+Pass `-w` to watch in realtime, `-v` for more detail, or `--json` for complete preset objects. Filter with `--base` or `--repo`.
 
 ### Delete presets
 
@@ -189,14 +244,22 @@ Delete a preset by ID or name, or all presets for a base model with `--base`:
 <div class="termy">
 
 ```shell
-$ dstack preset delete 8f3a12c4
+$ dstack preset delete c83375b4
 ```
 
 </div>
 
 For command options and agent settings, see the [`dstack preset` CLI reference](../reference/cli/dstack/preset.md).
 
-> Presets are experimental, and we’d love your feedback. Report bugs and request features on [GitHub](https://github.com/dstackai/dstack/issues), and ask questions on [Discord](https://discord.gg/u8SmfwPpMd).
+!!! info "Roadmap and feedback"
+    Here's what is coming soon:
+
+    * Allow the agent to change the source code, compile binaries, etc.
+    * Support for PD disaggregation
+    * Allow passing multiple `--previous <preset ID>` to `dstack preset create` to reuse the insights from previous sessions
+    * Allow passing ranges to `concurrency`
+
+    Report bugs and request features on [GitHub](https://github.com/dstackai/dstack/issues), and ask questions on [Discord](https://discord.gg/u8SmfwPpMd).
 
 !!! info "What's next?"
     1. Learn how dstack [services](services.md) work

src/dstack/_internal/cli/commands/preset.py

@@ -104,10 +104,10 @@ def _register(self) -> None:
             help="Leave the verified service running",
         )
         create_parser.add_argument(
-            "--max-trials",
+            "--trials",
             type=int,
             metavar="N",
-            help="The maximum number of benchmarked trials before the best one is promoted",
+            help="The number of benchmarked trials before the best one is promoted",
         )
         create_parser.add_argument(
             "--debug",
@@ -235,12 +235,26 @@ def _list(self, args: argparse.Namespace) -> None:
         verbose = args.verbose
         if not getattr(args, "watch", False):
             presets, sessions = self._list_presets_and_sessions(base=base, repo=repo)
-            print_presets(presets, sessions=sessions, verbose=verbose)
+            print_presets(
+                presets,
+                sessions=sessions,
+                verbose=verbose,
+                all_presets=args.all_presets,
+                limit=args.limit,
+            )
             return
         with Live(console=console, refresh_per_second=LIVE_TABLE_REFRESH_RATE_PER_SEC) as live:
             while True:
                 presets, sessions = self._list_presets_and_sessions(base=base, repo=repo)
-                live.update(get_presets_table(presets, sessions=sessions, verbose=verbose))
+                live.update(
+                    get_presets_table(
+                        presets,
+                        sessions=sessions,
+                        verbose=verbose,
+                        all_presets=args.all_presets,
+                        limit=args.limit,
+                    )
+                )
                 time.sleep(LIVE_TABLE_PROVISION_INTERVAL_SECS)
 
     def _list_presets_and_sessions(
@@ -267,18 +281,21 @@ def _create(self, args: argparse.Namespace) -> None:
         resume_session = None
         if getattr(args, "resume", None):
             resume_session = load_resumable_agent_session(args.resume)
-            if getattr(args, "max_trials", None) is not None:
+            if getattr(args, "trials", None) is not None:
                 console.print(
-                    "[warning]--max-trials is ignored when resuming: "
+                    "[warning]--trials is ignored when resuming: "
                     "the constraints are fixed at creation[/]"
                 )
         api = Client.from_config(project_name=args.project)
         allowed_fleets = None
         if resume_session is None:
-            if configuration.max_trials is None:
+            if configuration.trials is None:
                 raise ConfigurationError(
-                    "max_trials is required. Set it in the configuration or pass --max-trials"
+                    "trials is required. Set it in the configuration or pass --trials"
                 )
+            for field in ("max_ttft", "min_context_length", "concurrency"):
+                if getattr(configuration, field) is None:
+                    raise ConfigurationError(f"{field} is required")
             allowed_fleets = plan_preset(api=api, configuration=configuration)
             if not _confirm_preset_creation(store, configuration.name, assume_yes=args.yes):
                 console.print("\nExiting...")
@@ -408,6 +425,21 @@ def _add_list_args(parser: argparse.ArgumentParser) -> None:
         action="store_true",
         help="Output in JSON format",
     )
+    parser.add_argument(
+        "-a",
+        "--all",
+        action="store_true",
+        dest="all_presets",
+        help="Show all presets. By default, it only shows unfinished creations or the last one.",
+    )
+    parser.add_argument(
+        "-n",
+        "--last",
+        metavar="COUNT",
+        type=int,
+        dest="limit",
+        help="Show only the last N presets. Implies --all",
+    )
     model_filter = parser.add_mutually_exclusive_group()
     model_filter.add_argument(
         "--base",
@@ -490,8 +522,8 @@ def _get_effective_configuration(
     require_name: bool = True,
 ) -> PresetConfiguration:
     _apply_name(configuration, args.name, required=require_name)
-    if getattr(args, "max_trials", None) is not None:
-        configuration.max_trials = args.max_trials
+    if getattr(args, "trials", None) is not None:
+        configuration.trials = args.trials
     profile = load_profile_from_args(args=args, repo_dir=Path.cwd())
     for field in ProfileParams.model_fields:
         if getattr(configuration, field) is None: