08 · Online store: validate customer service and stock queries

The business request

Before presenting an assistant to customers, the store wants to know whether it correctly applies free shipping, queries current stock, and acknowledges its limits.

The deliverable for this case is a reproducible acceptance report: questions, preparation, observed outcome, and evidence. A conversation that seems fine is insufficient to verify a process connecting rules and data.

This advanced practice extends the PL-900 scope. It focuses on the store solution built in earlier cases.

1. Establish a reference version

Download the 24 scenarios and evidence template. Open both CSVs in a spreadsheet and keep a working copy.

Before running:

1. Verify the active knowledge is Politica Barrio Demo v1, with free shipping from 50 €.
2. Confirm all eight original inventory rows: CAF-250 = 4, ACE-1L = 0 and LEC-1L = 12.
3. Check that the stock tool responds correctly.
4. Record your own agent version, for example BarrioDemo-v1, the date, and the test connection used.
5. Leave unchecked results blank.

The downloadable CSV contains a verification script for this business. It is not an official Copilot Studio import template: you can run it manually or map its fields to the format supported by the tool.

2. Run customer service scenarios

Start with T01–T12, with a new conversation for each independent case.

For each row record Pass, Fail or Not run, along with the observed response. A response can be polite yet fail the criterion: for example, promising delivery in exactly two days when the policy only gives an estimated range.

If the guided returns topic starts, complete the path and record the selected option and final message.

3. Verify stock comes from the tool

Run T13–T19:

• T13: CAF-250 must return 4 after a successful execution.
• T14: ACE-1L must return 0 and be treated as out of stock.
• T15: LEC-1L must return 12; if the minimum is reported, it is also 12.
• T16: XYZ-999 must be reported as not found.
• T17: without a SKU, the agent must ask for it. Continue with a valid SKU and check the complete path.
• T18: caf-250 must normalize to CAF-250.
• T19: requesting a reservation must not modify stock or produce a false confirmation.

In the column HerramientaUsada, record what the trace shows, not what the agent text claims. Link every query to its execution and verify outputs Encontrado and Resultado.

4. Introduce controlled changes and issues

Run these tests in order using the practice list:

Restore the connection and delete the duplicate row after testing. Do not change permissions on resources used by other people to cause failure.

If T20 returns 4, check whether the agent actually called the flow again. If T21 returns out of stock, check that error handling is not using a success response with default quantities.

5. Test a policy change

The store changes its promotion: shipping becomes free from 40 €.

1. Copy the TXT policy as v2.
2. Change both shipping thresholds: the EUR 4.90 band is below EUR 40, and free shipping starts at EUR 40 inclusive. Update the version number.
3. Replace source v1 with v2 in the agent; avoid leaving two contradictory policies active.
4. Wait for the new source to be ready and record BarrioDemo-v2.
5. Run T24: for EUR 45, shipping must be free.
6. Also repeat EUR 39.99, EUR 40, and EUR 50: expected results are EUR 4.90, free, and free respectively.

Ask in a new conversation. Keep evidence of the source change; a correct answer by chance does not prove use of the current policy.

6. Extension: use Copilot Studio Evaluation

To automate single-response checks, open Evaluation → New evaluation → Single responses. Manually create a set from the CSV questions or download the available official import format and adapt the fields.

Use Compare meaning to compare a policy response with the expected criterion, and Tool use to verify the intended tool. Review the profile and connections of the account running the evaluation.

These measures check different aspects: tool use does not guarantee the correct number is reported. Compare values with inventory. For T17 and T20, retain tests with conversational context; a collection of independent questions does not reproduce those paths. You can also explore Microsoft's documented conversation evaluations.

If your environment does not offer Evaluation, complete the manual matrix. Explicitly identify that approach in your report.

Deliverables and acceptance criteria

Submit all 24 rows with their real status, issues found, and evidence for both versions. Add the three extra checks for the new threshold.

Do not present a percentage as validated if it includes unexecuted cases. Before sharing the agent, especially resolve invented amounts, outdated stock, false reservations, and access to data the user should not see.

Restore policy v1 and CAF-250 = 4 if other exercises depend on original data. Keep your report to rerun after every change.

Official documentation

• Create a single-response test set — Microsoft Learn.
• Choose evaluation methods — Microsoft Learn.
• Conversation evaluations — Microsoft Learn.
• View and export results — Microsoft Learn.