Introduction
In the age of AI-generated code and Large Language Model (LLM)-driven agents, software developers increasingly face a new challenge: how do you test code when you have no idea what's inside it? Traditional unit tests rely on knowing the internal structure, but when you're dealing with MCP servers, black-box APIs, or code produced by generative models, that assumption falls apart. As Ryan from SmartBear and VP of AI Architecture Fitz Nowlan discuss, we're moving away from old testing paradigms. Non-determinism—where the same input can produce different outputs—breaks traditional assertion-based testing. Meanwhile, data locality and data construction become more valuable because the source code itself is so easy to generate. This guide will walk you through a practical, step-by-step approach to testing code you didn't write and can't inspect.
What You Need
- A target system – This could be an MCP server, an API, a microservice, or any AI-generated code module whose internals are hidden or unstable.
- A test framework – Choose one that supports black-box testing (e.g., pytest, Jest, or a contract testing tool like Pact).
- Data generation tools – Libraries like Faker, Hypothesis (property-based testing), or custom data constructors to produce realistic inputs.
- A way to capture behavior – Logging, monitoring, or a replay system to record actual outputs for non-deterministic scenarios.
- Version control for test data – Not just for code, but for the test inputs and expected outcomes (especially important when data locality is key).
- Time and patience – Retraining your testing mindset from white-box to black-box takes practice.
Step 1: Shift from Implementation to Interface
When you can't see the code, your only reliable source of truth is the interface—the API contract, the message format, or the expected behavior. Stop trying to guess what's inside. Instead, focus on what the system does, not how it does it. This is a fundamental shift away from traditional unit testing.
Example: If you're testing an MCP server that processes natural language queries, you don't need to know its internal prompt chain. You only need to know: what inputs are valid? What outputs are returned? What are the side effects? Write your first test as a raw API call with a known input. This becomes your baseline.
Action: List all public endpoints, commands, or messages the system exposes. For each, define the input schema and the output schema. Start with the simplest happy path.
Step 2: Embrace Black-Box Testing with Equivalence Partitioning
Without internal knowledge, you must rely on input/output relationships. Use equivalence partitioning to group inputs that should trigger similar behavior. For example, if your system accepts a temperature value, partition the range into valid, invalid, and boundary values. This technique works regardless of the code's structure.
Action: For each interface, create a table of input classes. Then write one test per class. Use random data generation to populate inputs within each class. Tools like Hypothesis can automatically explore edge cases you wouldn't think of.
Step 3: Handle Non-Determinism by Testing Behaviors, Not Exact Values
LLM-driven agents are inherently non-deterministic. The same prompt can yield different responses. That means your tests cannot assert exact output strings. Instead, test for behaviors: does the output contain a certain structure? Does it satisfy a property? Is it within a range? This is where property-based testing shines.
Example: Instead of asserting "The response equals 'Hello'", assert "The response is a string of length between 1 and 500 characters" or "The response includes a valid timestamp". These behavioral checks survive non-determinism.
Action: For each interface, define invariants that must always hold. Write tests that assert those invariants, not exact values. Use libraries like Hypothesis (Python) or fast-check (JavaScript) to automatically generate varied inputs and check invariants.
Step 4: Leverage Data Construction Over Code Inspection
As Fitz Nowlan pointed out, when source code is cheap to generate, data locality and data construction become more valuable. Instead of trying to understand the code, focus on constructing high-quality test data that exercises the system. This is especially effective for MCP servers where the LLM's training data overlaps with your test data.
Action: Create a library of realistic data sets that mimic production traffic. Include edge cases, null values, large payloads, and unexpected combinations. Use tools like Faker to generate synthetic but plausible data. Save these data sets in version control along with their expected behavioral outcomes (not exact outputs). This becomes your "data oracle."
Step 5: Record and Replay for Regression
When you can't rely on deterministic outputs, you need a different regression strategy. Record real interactions with the system (inputs and outputs) over time. When you update the system (even if you don't change the code, the underlying LLM model might shift), replay the recorded inputs and compare the type or shape of the outputs, not exact content. This catches regressions in behavior class.
Action: Instrument your system to log all request-response pairs. Use a tool like VCR (Ruby) or pytest-recording to capture and replay. Compare responses using schema validation or embedding similarity (for text). Set thresholds for acceptable drift.
Step 6: Build a Living Contract with Consumer-Driven Testing
Since you can't trust the code, you must trust the contract. Consumer-driven contract testing lets the consumers of the API define their expectations. This aligns perfectly with testing unknown code: you don't care how the provider implements it, only that the contract is honored.
Action: Use a tool like Pact. Define a contract in the consumer's test suite that specifies expected requests and responses. Run this contract against the provider (the unknown code) to verify compliance. This creates a shared ground truth that both sides can evolve.
Tips for Success
- Accept non-determinism as a feature, not a bug. Design your tests to be resilient to variation. Use statistical or fuzzy matchers where needed.
- Prioritize data over code. Invest in building rich, realistic test data. The more varied your data, the more confidence you gain, even without code insight.
- Monitor behavior drift. Even if individual tests pass, the overall behavior of an LLM-driven agent can drift over time. Set up continuous monitoring of aggregate metrics (e.g., response length, sentiment, topic distribution).
- Use canary releases. When you update the unknown code (e.g., deploy a new model version), test it with a small percentage of real traffic first. Compare behavioral metrics against the old version.
- Document your assumptions. Write down what you know about the interface and what you are intentionally not testing. This helps future testers avoid repeating blind alleys.
- Stay agile. This approach is iterative. As you learn more about the system's behavior through testing, you can refine your test data and invariants.
By following these steps, you can confidently test code you've never seen—whether it's from an LLM, a third-party library, or a legacy system with no documentation. The key is to let go of internal knowledge and embrace the behavior-based, data-driven, contract-first testing mindset that the new era of software development demands.