feat: add openai-compatible executor#315
Open
davidhonig wants to merge 1 commit into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds an openai-compatible execution path so evaluations can run against OpenAI-compatible Chat Completions endpoints (e.g., local LM Studio), and documents/configures the new options.
Changes:
- Introduces
openai-compatibleexecutor with endpoint normalization and API key handling (OPENAI_API_KEYfallback). - Extends JSON schemas + project defaults to support
endpointandapi_key, and updates docs accordingly. - Adds unit/integration-style tests for schema validation, spec loading, engine execution, and
waza runbehavior.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| site/src/content/docs/reference/waza-yaml.mdx | Documents defaults.endpoint in .waza.yaml. |
| site/src/content/docs/reference/schema.mdx | Documents new openai-compatible executor and endpoint/api_key. |
| site/src/content/docs/reference/cli.mdx | Documents OPENAI_API_KEY env var behavior for the new executor. |
| site/src/content/docs/guides/eval-yaml.mdx | Updates eval YAML guide with openai-compatible options and semantics. |
| schemas/eval.schema.json | Extends eval schema for openai-compatible, makes model conditional. |
| schemas/config.schema.json | Extends project config schema with new engine and defaults.endpoint. |
| internal/validation/schema_test.go | Adds schema-validation test cases for openai-compatible configs. |
| internal/projectconfig/config_test.go | Validates new defaults.endpoint parsing/merging. |
| internal/projectconfig/config.go | Adds Defaults.Endpoint and merges it. |
| internal/models/spec_test.go | Adds tests for parsing endpoint/api_key into the spec model. |
| internal/models/spec.go | Adds Config.Endpoint and Config.APIKey. |
| internal/execution/openai_compatible_test.go | New tests covering endpoint normalization, auth, workspace cleanup, and skill-context injection. |
| internal/execution/openai_compatible.go | Implements the OpenAICompatibleEngine. |
| internal/execution/copilot.go | Refactors skill-dir selection into reusable helper for new engine. |
| cmd/waza/cmd_run_test.go | Adds waza run tests for openai-compatible and .waza.yaml defaults. |
| cmd/waza/cmd_run.go | Applies .waza.yaml defaults to eval specs and wires in openai-compatible engine creation. |
| README.md | Adds docs for openai-compatible executor and related env vars. |
Comment on lines
+476
to
+478
| if cfg, err := projectconfig.Load(filepath.Dir(specPath)); err == nil && cfg != nil { | ||
| applyProjectDefaultsToSpec(spec, cfg) | ||
| } |
Comment on lines
+114
to
+128
| workspaceDir, err = os.MkdirTemp("", "waza-openai-*") | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to create openai-compatible workspace: %w", err) | ||
| } | ||
| if err := setupWorkspaceResources(workspaceDir, req.Resources); err != nil { | ||
| _ = os.RemoveAll(workspaceDir) | ||
| return nil, fmt.Errorf("failed to setup openai-compatible workspace resources: %w", err) | ||
| } | ||
| e.mu.Lock() | ||
| e.workspaces = append(e.workspaces, workspaceDir) | ||
| e.mu.Unlock() | ||
| } | ||
| if _, err := ResolveWorkDir(workspaceDir, req.WorkDir); err != nil { | ||
| return nil, err | ||
| } |
Comment on lines
+685
to
+689
| case "openai-compatible": | ||
| engine, err = execution.NewOpenAICompatibleEngine(spec.Config.Endpoint, spec.Config.ModelID, spec.Config.APIKey) | ||
| if err != nil { | ||
| return nil, err | ||
| } |
Comment on lines
+130
to
+144
| defaultedEndpoint := `name: test-eval | ||
| skill: test-skill | ||
| version: "1.0" | ||
| config: | ||
| trials_per_task: 1 | ||
| timeout_seconds: 60 | ||
| executor: openai-compatible | ||
| metrics: | ||
| - name: accuracy | ||
| weight: 1.0 | ||
| threshold: 0.8 | ||
| tasks: | ||
| - "tasks/*.yaml" | ||
| ` | ||
| require.Empty(t, ValidateEvalBytes([]byte(defaultedEndpoint)), "openai-compatible should allow endpoint from .waza.yaml defaults") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Summary
Adds an
openai-compatibleexecutor for running Waza evals against OpenAI-compatible Chat Completions APIs such as local LM Studio. The executor supports endpoint normalization, optional per-evalapi_key,OPENAI_API_KEYfallback, workspace capture, usage mapping, and skill-context injection..waza.yamlcan now providedefaults.engine: openai-compatibleanddefaults.endpoint, so eval specs can stay minimal while using a local endpoint.Related issue
Related to #66
Agent handoff
internal/execution/openai_compatible.go,cmd/waza/cmd_run.go,internal/models/spec.go,internal/projectconfig/config.go,schemas/eval.schema.json,schemas/config.schema.json, README andsite/docs..waza.yaml; useOPENAI_API_KEYfor auth.modelis optional foropenai-compatibleand defaults tolocal-model. Skill context is injected using the existing skill-system-message path.Type of change
Validation
go test ./...make lintorgolangci-lint runweb/changedhttp://127.0.0.1:1234Documentation
site/docs updated, if CLI, YAML, dashboard, or validator behavior changedRisk and rollback
bb0fd568to remove the new executor, schema fields, project defaults, and docs. Existingmockandcopilot-sdkexecutor paths should remain unaffected.Notes for reviewers
Please focus on
internal/execution/openai_compatible.gorequest/response handling, skill-context injection, and.waza.yamldefault application incmd/waza/cmd_run.go. The intended local config isdefaults.engine: openai-compatibleplusdefaults.endpoint: http://127.0.0.1:1234; auth should come fromOPENAI_API_KEY.