Scope of the analysis

The experiment used the same 75 requests that formed the balanced model-routing scenario in Lab #01.

The original requests had been processed by Model C, which represented GPT-4.1. In Lab #02, the same prompts and business context were sent to Model B, which represented GPT-4.1 Mini.

The 29 Model C requests from Contract Analyzer were not included. Contract analysis carries greater business risk, and Lab #01 had not shown that a smaller model could produce acceptable results for that workload.

This made Lab #02 a controlled test of the balanced routing scenario rather than a general claim that Model B can replace Model C for every type of work.

Quality Evaluation

Every Model B response received a quality score from 1 to 5. A score of 4 or 5 was treated as a pass, while a score of 3 or lower was treated as a failure.

The evaluation criteria reflected the purpose of each workload. Support Chat responses were reviewed for accuracy, usefulness, safety and completeness. Document Assistant responses were checked against the supplied information and reviewed for unsupported claims. Ticket Classification responses were tested for the correct category, priority, routing team and output format. Background Summarization responses were reviewed for accuracy, coverage and compliance with the required structure.

All 75 Model B responses were evaluated.

A direct Model B versus Model C comparison was possible for 40 requests. Lab #01 retained the full Model C response text for 10 requests from each of the four workloads. For the other 35 requests, the original technical data was available, but the complete Model C response text had not been saved.

The financial, completion and response-time comparison therefore covers all 75 requests. The direct comparison of response quality covers 40 requests.

Cost and Usage Reconciliation

The Lab #02 request log was reconciled with the official OpenAI usage and cost exports.

The official records showed 75 requests, 5,210 input tokens and 3,426 output tokens. No cached input tokens were recorded.

The request-level Model B cost calculation was $0.0075656. The official OpenAI cost export showed the same amount, producing a cost variance of zero.

This reconciliation confirms that the cost result used in the analysis matches the provider’s official records.

Cost Result

The original 75 Model C requests cost $0.038044. After the requests were processed by Model B, their actual cost was $0.0075656.

The 75-request group became 80.11% cheaper.

When this result was applied to the full 1,000-request benchmark, total cost decreased from $0.145725 to $0.1152466. This represents a saving of $0.0304784, or 20.92%.

Lab #01 estimated a saving of 20.89%. Lab #02 produced an actual result of 20.92%.

The small difference occurred because the Model B responses did not use exactly the same number of output tokens as the original Model C responses. Lab #01 kept the original token volumes and changed only the price. Lab #02 sent the requests again and measured the tokens that Model B actually used.

The experiment therefore confirmed the financial opportunity identified in Lab #01.

[Insert Lab #02 Power BI validation dashboard]

Quality Result

Across all 75 Model B responses, the average quality score was 4.77 out of 5.

Seventy-four responses passed the quality review, producing a pass rate of 98.67%. One response failed.

For the 40 requests with responses available from both models, Model C achieved an average quality score of 4.85. Model B achieved 4.75. The difference was 0.10 points on the five-point scale.

Model B produced a better score in three comparisons, the same score in 30 comparisons and a lower score in seven comparisons.

These results show a small reduction in average quality, but not a reduction in the overall pass rate for the paired responses.

The result should still be interpreted carefully. The direct quality comparison covers 40 requests rather than all 75, because the remaining Model C response texts were not retained during Lab #01.

One Quality Failure

The only failed Model B response occurred in Ticket Classification.

The request described one employee who had forgotten a password while all other users remained unaffected. Model B correctly classified the issue as an access problem, but it assigned high priority. The expected priority was low because the issue affected only one user and was a standard password-reset case.

Model C classified the same request correctly.

This example is important because a high average score can hide individual mistakes. In a production environment, incorrect ticket priority could affect queue order, response targets and support-team capacity.

The result does not prevent the use of Model B for Ticket Classification. It shows that routing should include monitoring and clear rules for cases where the business impact of an incorrect classification is higher.

Completion and Response Time

Model B completed all 75 requests. Model C had completed 74 of the same requests, while one response had reached its configured output limit.

The average response-time difference was only 1.79 milliseconds. This is too small to represent a meaningful performance difference in this experiment.

The lower Model B cost was therefore not achieved through slower responses or a lower completion rate.

What the experiment proves

Lab #02 confirms that the 20.9% cost-saving opportunity identified in Lab #01 was achievable in this controlled request mix.

The 75 requests became 80.11% cheaper after moving from Model C to Model B. This reduced the total cost of the 1,000-request benchmark by 20.92%.

Model B maintained a 98.67% quality pass rate across all 75 responses. In the 40 direct comparisons, its average quality score was 0.10 points below Model C, while both models achieved the same 97.50% pass rate.

Model B also completed all 75 requests and produced almost the same average response time as Model C.

The combined result supports the use of Model B for the four tested workloads, provided that the company continues to monitor quality and applies stronger controls to tasks where individual mistakes can have a larger business impact.

What the experiment doesn't prove

The experiment does not show that Model B should replace Model C for every request.

Contract Analyzer was excluded because its outputs involve financial and commercial risk. The test therefore provides no evidence that Model B can replace Model C for contract analysis.

The direct quality comparison also covers only 40 of the 75 requests. All Model B responses were reviewed, but the missing Model C response texts prevented a complete paired comparison.

Finally, the experiment used controlled scenarios and synthetic customers. A production decision would require a larger sample, repeated testing and quality criteria linked to the company’s actual processes.

Business Decision

The result supports a controlled routing policy rather than a complete model replacement.

Model B can become the default model for the four tested workloads. Model C can remain available for requests that fail a quality check, contain unusual complexity or carry greater business risk.

Ticket Classification requires particular attention. The failed priority decision shows that even simple-looking tasks need clear evaluation rules and ongoing monitoring.

Contract Analyzer should remain on Model C until a separate experiment compares the two models using contract-specific quality criteria.

This approach keeps access to the stronger model where it may create value, while reducing unnecessary use for requests that Model B can handle successfully.

Conclusion

Lab #01 identified a possible 20.89% reduction in total AI cost through model routing. Lab #02 tested the proposal with the same 75 requests and achieved an actual reduction of 20.92%.

The cost of the routed requests fell by 80.11%. Average paired quality decreased by only 0.10 points. The paired pass rate remained unchanged, completion improved from 98.67% to 100%, and average response time remained almost identical.

The experiment also identified one important classification failure and a limitation in the available comparison data. These findings show why model-routing decisions should combine cost measurement with quality and operational controls.

The goal is not to move every request to the cheapest model. The goal is to use the least expensive model that can meet the quality and risk requirements of the task.

IB Forward Consulting helps growth-stage AI companies connect API usage, financial reporting and engineering decisions so AI costs can be controlled before they become a margin problem.

What comes next

Lab #03 will move from a fixed model change to an automated routing test. A simple routing system will send routine requests to smaller models and direct more complex or higher-risk requests to Model C, allowing us to compare cost, quality and operational performance with a single-model approach.