feat(eval): add SimulateComponentMocker for per-component API-based simulation by AAgnihotry · Pull Request #1732 · UiPath/uipath-python

AAgnihotry · 2026-06-17T19:34:24Z

Summary

Adds a new components format to simulation.json that routes each tool call to the
/agentsruntime_/api/execution/simulations/simulate-component API instead of the local LLM mocker
Introduces SimulateComponentMocker, SimulateComponentService, and 7 new types
(SimulationStrategy, RuleOperator, SimulationAnswerType, SimulationAnswer,
SimulationCondition, SimulationBehavior, ComponentSimulationConfig) mirroring the API contract
Legacy format (toolsToSimulate + instructions) is fully preserved — no breaking changes

Design decisions

Endpoint uses agentsruntime_ service prefix, which means UIPATH_SERVICE_URL_AGENTSRUNTIME
env var can redirect to a local service for development
componentId in the payload uses the configured component ID (not the normalised invoked name)
to ensure server-side simulation rules match correctly
Empty components: [] does not activate the API mocker (truthiness check, not is not None)

Test plan

Existing simulation tests pass (pytest tests/cli/test_debug_simulation.py)
New tests in tests/cli/eval/mocks/test_simulate_component.py all pass (coverage 100%)
ruff check . passes with no errors
mypy passes with no errors
Manual: run uipath debug with a simulation.json using the new components format and
verify the API call is made to agentsruntime_/api/execution/simulations/simulate-component
Manual: verify legacy toolsToSimulate format still routes to the local LLM mocker

🤖 Generated with Claude Code

ANIPIK--FFQTDV0N47:simulate-component-agent anipik$ uv run uipath run main -f input.json --simulation "$(cat simulation.json)"  
Resource overwrites config file not found: __uipath/uipath.json
▶ START 
[simulate-component] mocker set: SimulateComponentMocker
simulate-component: mocker created (SimulateComponentMocker)
[simulate-component] get_mocked_response called for 'get_current_weather', mocker=SimulateComponentMocker
simulate-component: calling API for 'get_current_weather'
HTTP Request: POST https://alpha.uipath.com/agentsurt/DefaultTenant/agentsruntime_/api/execution/simulations/simulate-component "HTTP/1.1 200 OK"
simulate-component: 'get_current_weather' simulated successfully
[simulate-component] get_mocked_response called for 'get_forecast', mocker=SimulateComponentMocker
simulate-component: calling API for 'get_forecast'
HTTP Request: POST https://alpha.uipath.com/agentsurt/DefaultTenant/agentsruntime_/api/execution/simulations/simulate-component "HTTP/1.1 200 OK"
simulate-component: 'get_forecast' simulated successfully
● END
output
├── current (dict)
│   ├── city: London
│   ├── temperature: 6.5
│   ├── condition: cloudy
│   └── humidity: 70
├── forecast (list, 3 items)
│   ├── #0 (dict)
│   │   ├── date: 2024-01-15
│   │   ├── high: 8.0
│   │   ├── low: 3.0
│   │   └── condition: Partly cloudy
│   ├── #1 (dict)
│   │   ├── date: 2024-01-16
│   │   ├── high: 6.0
│   │   ├── low: 2.0
│   │   └── condition: Light rain
│   └── #2 (dict)
│       ├── date: 2024-01-17
│       ├── high: 7.0
│       ├── low: 4.0
│       └── condition: Overcast
└── summary: London: 6.5°C, cloudy. 3-day forecast available.
✓  Successful execution.
'id' field not present in uipath.json

…imulation Introduces a new simulation format that routes tool calls to the simulate-component API (`/agentsruntime_/api/execution/simulations/simulate-component`) instead of the local LLM mocker. The existing legacy format (toolsToSimulate + instructions) is preserved unchanged. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Copilot

Pull request overview

Adds support for a new simulation.json format that simulates tool/component calls by delegating to the /agentsruntime_/api/execution/simulations/simulate-component endpoint, while preserving the legacy toolsToSimulate + instructions local LLM mocker behavior.

Changes:

Introduces per-component simulation schema/types (components + API-contract enums/models) and wires them into mocking context creation.
Adds SimulateComponentService + SimulateComponentMocker and routes mocking through them when components-based config is active.
Extends tests to cover loading the new components format and updates legacy assertions accordingly.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
packages/uipath/tests/cli/test_debug_simulation.py	Adds a test for the new `components` simulation format and updates legacy assertions.
packages/uipath/src/uipath/eval/mocks/_types.py	Adds API-contract enums/models and extends `SimulationConfig` / `MockingContext` with `components`.
packages/uipath/src/uipath/eval/mocks/_simulate_component_service.py	New service wrapper around the simulate-component API endpoint.
packages/uipath/src/uipath/eval/mocks/_simulate_component_mocker.py	New mocker that calls the simulate-component API per tool invocation.
packages/uipath/src/uipath/eval/mocks/_mocker_factory.py	Routes to the new mocker based on context.
packages/uipath/src/uipath/eval/mocks/_mock_runtime.py	Builds mocking context for new format and adjusts execution-context setup.
packages/uipath/src/uipath/eval/mocks/_mock_context.py	Updates tool-simulation detection to support components-based configs.
packages/uipath/src/uipath/eval/mocks/init.py	Exposes new types in the public mocks module API.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…add tests Cast simulatedOutput to R to satisfy mypy's return-value check. Adds test_simulate_component.py with 14 tests covering is_tool_simulated with components, _find_component normalisation, success/error response paths, API failure, and payload field construction. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Remove unused logger and fix import order in SimulateComponentMocker - Use component.component_id (not invoked tool name) for componentId payload field - Import UiPathConfig from public module (uipath.platform.common) - Switch components truthiness check from `is not None` to bool in MockerFactory and set_execution_context; empty list no longer activates API mocker - Remove unused ComponentSimulationConfig import from _mock_context - Update SimulationConfig comments to reflect truthiness-based routing - Fix SimulateComponentService to omit headers kwarg when empty; avoids NoneType assignment error in BaseService - Add tests: _build_execution_history with no context vars, MockerFactory ValueError for unknown strategy, is_tool_simulated unknown strategy fallthrough, SimulateComponentService HTTP call with/without headers Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 10 out of 11 changed files in this pull request and generated 2 comments.

…nent sample - Add `inputSchema`/`outputSchema` optional fields to `ComponentSimulationConfig` so users can override auto-derived schemas directly in simulation.json - Update `SimulateComponentMocker` to prefer component-level schema overrides over function-annotation-derived schemas - Add logging to mocker for API call success/failure visibility - Add simulate-component-agent sample with weather forecast agent demonstrating per-component simulation format - Point sample pyproject.toml to local uipath source via uv path dependency - Add integration test and testcase for simulate-component CLI flow Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Always apply _cleanup_schema to output_schema (whether derived from annotations or params), matching LLMMocker behavior; fixes dead cleanup code - Remove manual X-UiPath-Internal-TenantId/AccountId header injection from SimulateComponentService — BaseService handles these automatically - Add tests verifying outputSchema override is sent as-is and derived schema is cleaned before sending Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…yload Extracts the active OTel span context at call time and passes traceId and parentSpanId to the simulate-component API so simulated calls appear as proper children in the trace hierarchy. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-18T17:24:37Z

🚨 Heads up: `uipath-integrations` cross-tests are FAILING 🚨

Your changes may break one or more integrations in uipath-integrations-python:

uipath-openai-agents
uipath-google-adk
uipath-agent-framework
uipath-llamaindex
uipath-pydantic-ai

⚠️ These checks are NOT enforced by branch protection rules. Please review the failures before merging.

🔍 Inspect the failed run →

…ignature Remove tenant_id/organization_id args from test call — these were removed when BaseService header injection was fixed in the Copilot review. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…vice The agentsruntime_ endpoint requires X-UiPath-Internal-TenantId and X-UiPath-Internal-AccountId for routing on cloud requests. Read these from UiPathConfig directly in the service so they are always included without callers needing to pass them explicitly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Replace conditional dict unpack with direct headers= kwarg so mypy can correctly type-check the request_async call. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Start a local Python HTTP server in run.sh that handles POST /api/execution/simulations/simulate-component and returns pre-canned simulated weather/forecast responses. Set UIPATH_SERVICE_URL_AGENTSRUNTIME to redirect SDK requests to the mock server, bypassing the real agentsruntime_ service that returns 401 in CI. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…service override Add UIPATH_SERVICE_URL_AGENTSRUNTIME to integration_tests workflow so the testcase bypasses the default scope_url routing and hits the environment-specific agentsruntime service directly (ALPHA_AGENTS_URL / STAGING_AGENTS_URL / CLOUD_AGENTS_URL secrets). Remove mock server added in previous commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Standard scope_url routing works for agentsruntime on alpha — remove the UIPATH_SERVICE_URL_AGENTSRUNTIME override added to integration_tests.yml. Add a stable project id UUID to the sample uipath.json to suppress the 'id' field not present' warning during tracing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The simulate-component API is only available on alpha. Add testcase.json with {"environments": ["alpha"]} and update the integration_tests workflow to skip testcases whose supported environment list excludes the current environment. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

sonarqubecloud · 2026-06-18T18:05:03Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
98.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions · 2026-06-18T18:23:55Z

🚨 Heads up: `uipath-langchain` cross-tests are FAILING 🚨

Your changes may break the uipath-langchain-python integration.

⚠️ These checks are NOT enforced by branch protection rules. Please review the failures before merging.

🔍 Inspect the failed run →

Copilot AI review requested due to automatic review settings June 17, 2026 19:34

github-actions Bot added test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-integrations labels Jun 17, 2026

Copilot started reviewing on behalf of AAgnihotry June 17, 2026 19:34 View session

chore: bump uipath version to 2.11.5

7161b84

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Copilot AI reviewed Jun 17, 2026

View reviewed changes

AAgnihotry and others added 2 commits June 17, 2026 12:41

AAgnihotry force-pushed the feat/simulate-component-mocker branch from 294eb67 to 213dc99 Compare June 17, 2026 19:55

AAgnihotry requested a review from Copilot June 17, 2026 20:02

Copilot started reviewing on behalf of AAgnihotry June 17, 2026 20:03 View session

Copilot AI reviewed Jun 17, 2026

View reviewed changes

Comment thread packages/uipath/src/uipath/eval/mocks/_simulate_component_mocker.py

Comment thread packages/uipath/src/uipath/eval/mocks/_simulate_component_service.py Outdated

AAgnihotry and others added 3 commits June 18, 2026 10:14

AAgnihotry and others added 8 commits June 18, 2026 10:25

fix(eval): update test to match SimulateComponentService.simulate() s…

a5a1359

…ignature Remove tenant_id/organization_id args from test call — these were removed when BaseService header injection was fixed in the Copilot review. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix(eval): move logger declaration after imports to fix E402 ruff errors

19f9fbd

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix(eval): pass headers directly to fix mypy arg-type error

eeb47b2

Replace conditional dict unpack with direct headers= kwarg so mypy can correctly type-check the request_async call. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(eval): add SimulateComponentMocker for per-component API-based simulation#1732

feat(eval): add SimulateComponentMocker for per-component API-based simulation#1732
AAgnihotry wants to merge 15 commits into
mainfrom
feat/simulate-component-mocker

AAgnihotry commented Jun 17, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

sonarqubecloud Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

AAgnihotry commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Design decisions

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026

🚨 Heads up: uipath-integrations cross-tests are FAILING 🚨

Uh oh!

sonarqubecloud Bot commented Jun 18, 2026

Quality Gate passed

Uh oh!

github-actions Bot commented Jun 18, 2026

🚨 Heads up: uipath-langchain cross-tests are FAILING 🚨

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AAgnihotry commented Jun 17, 2026 •

edited

Loading

🚨 Heads up: `uipath-integrations` cross-tests are FAILING 🚨

🚨 Heads up: `uipath-langchain` cross-tests are FAILING 🚨