Test Spec: Skills Provider Workflows¶

Document Control¶

Field	Value
Status	Implemented
Design doc	`docs/design/skills-provider-workflows.md`
Test file	`tests/test_skills_provider.py`

Requirement Traceability¶

REQ ID	Description summary	TEST IDs
`REQ-SKILLPROV-01`	Skills provider command family exists	`TEST-SKILLPROV-01`, `TEST-SKILLPROV-02`, `TEST-SKILLPROV-03`
`REQ-SKILLPROV-02`	Provider abstraction exists independently of execution	`TEST-SKILLPROV-04`, `TEST-SKILLPROV-05`
`REQ-SKILLPROV-03`	`skills provider list` reports registered providers	`TEST-SKILLPROV-01`
`REQ-SKILLPROV-04`	`skills search` returns structured manifest-derived results	`TEST-SKILLPROV-02`, `TEST-SKILLPROV-05`
`REQ-SKILLPROV-05`	`skills fetch` resolves cached files and reports provenance fields	`TEST-SKILLPROV-03`, `TEST-SKILLPROV-06`
`REQ-SKILLPROV-06`	`skills cache list` reports cached provider entries	`TEST-SKILLPROV-07`
`REQ-SKILLPROV-07`	`skills cache refresh` refreshes provider catalogs	`TEST-SKILLPROV-04`
`REQ-SKILLPROV-08`	Auto-refresh behavior is available for future cold-cache resolution	Deferred
`REQ-SKILLPROV-09`	Unknown providers return structured error	`TEST-SKILLPROV-08`
`REQ-SKILLPROV-10`	Missing skill names return structured error	`TEST-SKILLPROV-06`
`REQ-SKILLPROV-11`	Cold cache returns structured cache miss	Deferred
`REQ-SKILLPROV-12`	Invalid manifests are rejected	`TEST-SKILLPROV-04`
`REQ-SKILLPROV-13`	Download failures return structured errors	`TEST-SKILLPROV-09`, `TEST-SKILLPROV-10`
`REQ-SKILLPROV-14`	Cache-path traversal is rejected	`TEST-SKILLPROV-11`

Test Cases¶

TEST-SKILLPROV-01 — Provider list reports registered skills providers¶

Given  a skills provider registry is available
When   the operator runs `canarchy skills provider list --json`
Then   the system shall return a structured list of registered skills providers

Fixture: mocked registry.

TEST-SKILLPROV-02 — Search returns manifest-derived skill metadata¶

Given  a provider search returns at least one manifest-derived skill descriptor
When   the operator runs `canarchy skills search <query> --json`
Then   the system shall return provider, skill name, publisher, version, and provider-facing skill ref
And    the result may include manifest-derived tags in metadata

Fixture: mocked descriptor result.

TEST-SKILLPROV-03 — Fetch reports local manifest and entry paths¶

Given  a provider resolution returns local cached manifest and entry paths
When   the operator runs `canarchy skills fetch <provider>:<skill> --json`
Then   the system shall return the local manifest path, local entry path, and cache status

Fixture: mocked resolution result.

TEST-SKILLPROV-04 — Refresh rejects invalid manifests¶

Given  a repository-backed manifest is missing required schema fields
When   the provider refresh path parses that manifest
Then   the system shall reject it with `SKILL_MANIFEST_INVALID`

Fixture: tests/fixtures/skills/invalid_missing_entry.skill.yaml.

TEST-SKILLPROV-05 — Refresh builds a catalog from valid manifests¶

Given  repository-backed skill manifest files are available
When   the provider refresh path inspects those manifests
Then   the system shall build a manifest-derived provider catalog
And    the provider search path shall be able to return those descriptors later

Fixture: tests/fixtures/skills/j1939_compare_triage.skill.yaml, tests/fixtures/skills/uds_trace_minimal.skill.yaml.

TEST-SKILLPROV-06 — Resolve fetches cached files for a valid skill¶

Given  a cached provider catalog contains a valid skill entry
When   the provider resolve path is asked for that skill name
Then   the system shall fetch or reuse the local manifest and entry files
And    the returned resolution shall include provider-facing provenance fields

Fixture: mocked catalog entry and downloaded files.

TEST-SKILLPROV-07 — Cache list reports cached skill counts¶

Given  a provider manifest is stored in the local skills cache
When   the operator runs `canarchy skills cache list --json`
Then   the system shall report the provider name and cached skill count

Fixture: temporary cache manifest.

TEST-SKILLPROV-08 — Unknown provider returns structured error¶

Given  the requested skills provider is not registered
When   the operator runs `canarchy skills cache refresh --provider missing --json`
Then   the system shall exit with code `3`
And    the response shall contain an error with code `"SKILL_PROVIDER_NOT_FOUND"`

Fixture: mocked registry without the requested provider.

TEST-SKILLPROV-09 — Fetch download failures return structured errors¶

Given  a cached provider catalog contains a valid skill entry
When   the provider resolve path cannot download a missing manifest or entry file
Then   the system shall return `SKILL_FETCH_FAILED`
And    the command shall not terminate with an uncaught exception

Fixture: mocked download failure.

TEST-SKILLPROV-10 — Refresh manifest download failures return structured errors¶

Given  a provider catalog refresh discovers a remote manifest path
When   that manifest cannot be downloaded successfully
Then   the system shall return a structured provider error rather than an uncaught exception

Fixture: mocked manifest download failure.

TEST-SKILLPROV-11 — Path traversal is rejected during fetch¶

Given  a provider catalog entry contains manifest or entry paths that escape the provider cache subtree
When   the provider resolve path is asked to fetch that skill
Then   the system shall reject the request with `SKILL_MANIFEST_INVALID`

Fixture: mocked catalog entry with ../ traversal.

Fixtures And Environment¶

tests/fixtures/skills/j1939_compare_triage.skill.yaml
tests/fixtures/skills/uds_trace_minimal.skill.yaml
tests/fixtures/skills/invalid_missing_entry.skill.yaml
temporary cache directories for cache-list and resolution tests
mocked provider registry and mocked GitHub transport helpers for deterministic refresh/resolve flows

Explicit Non-Coverage¶

MCP exposure of skills commands, which belongs to #167
actual skill execution semantics, which are out of scope for this issue
non-GitHub provider implementations