The Testing Engagement That Leaves You Dependent Wasn’t Really an Investment
Why knowledge transfer has to be a design goal from day one — not a handoff at the end There is a particular kind of frustration that engineering leaders know well, and it usually happens about three months after a consulting engagement ends. The testing infrastructure that was built looks impressive. The coverage numbers were good. The reports during the engagement showed real progress. Then the consultants leave, the internal team tries to run the framework without them, and it becomes clear that something essential went with them — not maliciously, not through anyone’s negligence, just because the knowledge was never actually transferred. It lived in the heads of the people who built it, embedded in undocumented decisions, tool configurations that made sense to the people who set them up, and test architectures that nobody on the internal team fully understands. Six months later the framework is deteriorating. A year later it’s half-abandoned. The engagement produced a dependency, not a capability. This is not an accident. Ongoing dependency is a commercially rational outcome for most consulting firms. If your client can run the system without you, the engagement ends. If they can’t, it continues — or you come back. The incentive structure of most testing engagements points directly away from knowledge transfer. Understanding that dynamic is the first step toward buying engagements differently. What Testing Knowledge Actually Consists Of The challenge with testing knowledge is that so much of it is invisible until it’s gone. When a senior automation engineer leaves — whether they’re a consultant departing at the end of an engagement or an internal employee who found another job — they take with them a specific combination of things that are genuinely hard to document after the fact. They know why the framework is structured the way it is. They understand which design decisions were deliberate and which were workarounds for constraints that no longer exist. They know the parts of the automation library that are solid and the parts that are fragile. They know the test data dependencies, the environment quirks, the timing issues that were worked around, the edge cases that were explicitly scoped out. None of this appears in the test cases themselves. It exists in memory. This is why the composition of the automation suite matters so much for survivability. A framework built entirely on custom scripts, where knowledge is embedded in code that only the author fully understands, is brittle by definition. When the author leaves, the code remains but the reasoning behind it doesn’t. Maintenance becomes archaeology — reverse-engineering intent from implementation, which is expensive and error-prone. A framework built on a documented, structured methodology — with test cases written in readable business language, with automation logic separated cleanly from test design, with roles that different people can take on independently — survives personnel changes much better. Not because the knowledge disappears any less, but because less of the critical knowledge is stored in individual people rather than in the structure of the system itself. The Shadowing Problem The conventional approach to knowledge transfer in consulting engagements is documentation plus handoff. The consultants write things down toward the end of the project, spend a few days walking the internal team through what was built, and then leave. The documentation is usually too high-level to be operationally useful, the walk-through covers what was built but not why, and the internal team spends the next several months discovering what wasn’t written down. The more effective model runs knowledge transfer in parallel with delivery from the beginning — not as a handoff phase at the end, but as a continuous practice throughout. The first phase is straightforward: internal team members shadow the senior consultants while work is happening. They’re present for design decisions, they hear the reasoning behind architectural choices, they see how problems get diagnosed and resolved in real time. This is not passive observation — it’s deliberate exposure to the thinking process that produces the outputs, not just the outputs themselves. The second phase flips the model. Internal team members begin doing the work themselves while senior consultants observe, provide correction, and increasingly step back. This reverse shadowing is where the transfer actually happens and where gaps become visible. A team member who watched something done correctly for three months will often discover, when they try to do it themselves, that they understood the mechanics but missed something about the judgment involved. That gap is much better discovered while senior support is still present than after the engagement has ended. The explicit goal of this model — stated as a requirement, not a hope — is that the client’s team is prepared to run the system independently before the engagement concludes. Not mostly prepared. Not prepared with a support contract backstop. Actually capable of operating, maintaining, and extending what was built without ongoing consultant involvement. At MMC Networks, SDT was engaged specifically to implement a test automation solution, staff the project during the build, and then transfer the technology and process knowledge to the new QA staff as they came on board. The transfer wasn’t an afterthought — it was part of the stated scope from the beginning. The engagement was defined as complete when the internal team could run the system, not when the system was built. At PepsiCo, one of the explicit goals established at the outset was “the ability for PepsiCo to be self-sufficient running test centers.” Self-sufficiency wasn’t a nice-to-have — it was a named success criterion. Building something that required continued outside involvement to operate would have been a failure against that goal, not a success. What You Should Ask Before Starting Any Engagement The way to evaluate whether a testing engagement is structured for your independence or for ongoing dependency is to ask a specific set of questions before the work begins. How will knowledge be transferred during the engagement, not at the end of it? If the answer involves documentation and a handoff session, the
Before You Buy Another Testing Tool, Do This First
Most testing problems are misdiagnosed. An honest process assessment finds the real issue — and often it’s not what anyone expected. There’s a pattern that plays out in testing organizations with remarkable consistency. Something is clearly wrong — releases are slipping, bugs are reaching production, the team feels like they’re always behind. Leadership makes a diagnosis: we need better tools, or more headcount, or a different CI/CD setup. Money gets spent. Things improve marginally, then drift back. Six months later, the same conversation happens again with a different culprit in the title role. The diagnosis is usually wrong. Not because the people making it aren’t smart, but because the symptoms of a broken testing process look almost identical regardless of what’s actually causing the problem. Late defects, slow releases, brittle automation, coverage gaps — these appear whether the real issue is organizational structure, tool mismatch, process inconsistency, or something no one has bothered to look at properly. The expensive mistake most teams make is skipping the diagnostic step entirely and going straight to solutions. The cheaper, more effective alternative is to find out what’s actually broken before deciding how to fix it. What a Real Assessment Looks Like A process assessment isn’t a sales conversation or a vendor demo disguised as an evaluation. Done properly, it’s an unbiased, structured examination of a testing organization’s methods, tools, practices, organizational structure, and environment — with the explicit goal of identifying the highest-priority areas for improvement and producing a prioritized, actionable plan. The starting point is documentation. Not the best documentation anyone has ever produced, but the representative sample of what actually exists day-to-day: test plans, test design documents, automation approach documents, defect management records, job descriptions for testing roles, prior assessment reports if any exist, org charts, and key metrics reports. The emphasis on existing, typical documentation matters. The point is to understand what the organization actually does, not what it aspires to do. An assessment built on documentation created specifically for the assessment is measuring the wrong thing. From there, the process moves to interviews — with testing team members, with their internal customers (typically development leads who depend on testing results), and with the people responsible for tooling and environment. For a small organization this might be around twenty people. For a large enterprise engagement it can run to 150. The interviews surface things that documentation never captures: where the process breaks down under pressure, what informal workarounds exist, which bottlenecks everyone knows about but nobody has formally addressed. The output is a management presentation that presents findings ranked by impact — not a comprehensive catalog of everything that could theoretically be better, but a prioritized view of what matters most and what a realistic improvement path looks like. Why the Same Problems Keep Appearing SDT has been conducting process assessments since the early 1990s. The same gaps surface across organizations regardless of size, industry, or sophistication. This consistency is what drove the development of SDT’s entire methodology — the assessments kept revealing the same underlying structural problems, which created both the evidence base and the motivation to build reusable solutions for them. The most common findings fall into a predictable set of categories. Testing is getting involved too late. By the time testers are engaged, the requirements have been locked and the architecture is set. The defects that are cheapest to fix — the ones that would have been caught in a technical review of a requirements document — are instead discovered in system testing or, worse, by users. The cost of finding and fixing a defect grows significantly at each stage of the development lifecycle. Moving testing earlier isn’t about adding process overhead. It’s about shifting where the expensive work happens. The Lockheed Martin GTN program is a good example of what this looks like when it’s been going wrong for years. For several years, the GTN team was finding a significant number of issues during the final testing phase of each software release. Those late-stage discoveries were causing cost and schedule overruns on most releases. The assessment identified the need to involve the test organization in creating test deliverables earlier in the development cycle — not a tool problem, not a headcount problem, a timing problem in how the process was structured. Fixing that timing was the foundation of everything else that followed. The process isn’t consistent. Different projects use different approaches. Different team members have different ideas about what “done” looks like from a testing standpoint. There’s no shared vocabulary for test design, no standard templates, no common definition of what a test plan should contain. This inconsistency makes it impossible to measure progress, compare results across projects, or build on what was done previously. Every project starts from scratch in some meaningful sense, which is why the team always feels behind. Automation and test design aren’t separated. This one is usually invisible to the teams experiencing it because it looks like a tool problem or an engineer performance problem. The real issue is structural — test design decisions are being made by the same people who have to implement them as code, which creates bottlenecks, knowledge silos, and test libraries that become unusable when the people who wrote them leave. The tools don’t match the environment. Organizations often end up with whatever tools the most recent senior hire was familiar with, or whichever vendor had the best sales presence at the right moment. These tools may work fine in isolation but don’t integrate with each other or with the development environment in a way that produces reliable, usable results. What the Assessment Actually Gives You The value of a process assessment isn’t the report. It’s the clarity about where to spend the next dollar and the next six months. Most organizations that have done an assessment describe a version of the same experience: they came in thinking they knew what was wrong, and the assessment confirmed some of that and surprised them with the
Why 30 Years of Testing Experience Isn’t Marketing Copy — It’s the Product
The methodology behind SDT didn’t come from a whiteboard. It came from the places that invented modern software. Most companies in the software testing space describe their experience the same way: a number of years, a list of clients, a claim that their methodology is proven. It all sounds the same after a while. The number gets bigger each year. The client logos rotate. The word “proven” appears on every page. What rarely gets explained is where a methodology actually comes from. Not the branding around it — the actual ideas, the decisions about how testing should work, the underlying architecture of the approach. Those things don’t emerge from a product roadmap. They emerge from specific experiences, specific problems, and specific environments where the cost of getting things wrong is high enough to produce real learning. SDT’s founder Ed Kit spent 14 years in two of those environments before he started the company. Understanding where the methodology came from is a reasonable way to evaluate whether it’s worth trusting. Bell Labs, 1978–1980 When Ed Kit joined AT&T Bell Labs in 1978, he was entering arguably the most consequential research and engineering environment of the 20th century. The transistor was invented there. So was information theory, the laser, the C programming language, the UNIX operating system, and the charge-coupled device. Seven Nobel Prizes were awarded for work done at Bell Labs. In 1972, Dennis Ritchie had written C as a replacement for B, and used it to rewrite UNIX — foundational work that underpins almost every system software environment in existence today. Kit was there two years later, and among his responsibilities was testing C compilers — the tools that translated the language Ritchie had built into executable code that machines could run. The standards expected at Bell Labs were not theoretical. The systems being built ran the AT&T telephone network. Failure had direct, observable consequences at scale. There is perhaps no better environment to develop intuitions about what software quality actually requires — not at a conceptual level, but at the level of practice, rigor, and institutional discipline. Tandem Computers, 1980–1992 Kit’s next twelve years were at Tandem Computers, which had a single organizing principle that shaped everything it built: systems could not fail. Tandem’s NonStop architecture was designed for ATM networks, banks, stock exchanges, and telephone switching centers — environments where downtime was not an acceptable outcome and data loss was not recoverable. Inc. magazine ranked Tandem the fastest-growing public company in America during its peak years. At Tandem, Kit managed groups responsible for system testing, performance, software release planning and management, release tools, and software release distribution. He was selected as a member of the distinguished team that identified and managed the steps to define and establish Tandem’s software engineering organization — a process of building institutional testing discipline from the ground up, at a company that couldn’t afford to get it wrong. He also held ownership for software testing across Tandem’s data communications, information management, and distributed systems management products. Twelve years of testing software for systems where zero data loss and maximum uptime were not aspirational goals but product requirements. The Standard That Still Governs the Industry In 1991, one year before founding SDT, Kit served on the IEEE Software and Systems Engineering Standards Committee as part of the small team that developed IEEE Standard 829-1983 — the IEEE Standard for Software Test Documentation. That standard became, and remains, the dominant international standard for software testing. This is not a line on a resume. Writing a standard means defining, for the entire industry, what the vocabulary of software testing should be, what documentation practices should look like, and what quality means in operational terms. It requires synthesizing accumulated knowledge from across the field and making binding decisions about how that knowledge should be formalized. Kit founded Software Development Technologies in 1992, the year after that committee work was complete. From Practice to Methodology Between 1992 and 1999, SDT ran assessments and consulting engagements that consistently revealed the same gaps across organizations of every size and type. Testing processes were inconsistent and poorly defined. Test design and automation were conflated in ways that made both fragile. There was no shared vocabulary for how testing work should be structured or measured. The same problems appeared at a regional bank and at a Fortune 50 manufacturer. By 1999, Kit had begun formalizing SDT’s keyword-driven testing methodology — the approach that became the company’s core intellectual property. It wasn’t built speculatively. It was built in response to documented, repeated failures observed across hundreds of real engagements, grounded in a framework developed by someone who had spent over two decades at the sharpest edge of what software quality actually requires. The methodology was refined through use. WellPoint Health Networks in 2003. Southwest Airlines and PepsiCo in 2003 and 2004. FedEx in 2005. Siemens in 2006. Each engagement produced feedback that went back into the IP. By 2012 the framework was embodied in a US patent — Patent US 9,489,277 B2 — and has been actively in use ever since. Why Any of This Matters to You When you evaluate testing partners, you are making a decision about what kind of knowledge you want working on your system. There are faster options, cheaper options, and more recent-vintage options. Some of them will have well-designed websites and plausible-sounding methodology descriptions. What they won’t have is a methodology built on the actual experience of testing C compilers at Bell Labs, running software quality for fault-tolerant financial systems at Tandem, co-authoring the international standard that defines how the industry documents testing work — and then spending thirty years refining those instincts through real engagements at real companies that had real consequences for getting it wrong. SDT is not a large generalist firm. It was built by one person with a very specific and very deep body of experience, and everything the company does reflects the architecture of thinking that experience produced. The keyword framework
Your Test Automation Isn’t Broken. Your Approach to It Is.
Why scripted automation keeps failing — and what the numbers look like when you fix the underlying problem There’s a specific kind of frustration that QA teams know well. You invest months building out an automation suite. Engineers write hundreds of scripts. Coverage looks good on paper. Then the product ships a new feature, the UI gets updated, or a workflow changes — and suddenly a third of your tests are failing. Not because the product is broken. Because the tests are. So engineers spend the next sprint fixing tests instead of building new coverage. The automation library grows larger and more expensive to maintain. Trust in the suite erodes. And at some point, someone in a release meeting makes the call that nobody wanted to make: we’re going to need a manual sign-off anyway, because we don’t really know if the automated tests are telling us anything useful. This cycle is so common it’s almost expected. Most teams assume it’s just the nature of automation — that test maintenance is an unavoidable cost of having a modern testing practice. It isn’t. The maintenance problem is a symptom of a structural choice made early in how the automation was built. The Scripting Trap Traditional test automation is built on scripts. An engineer writes code that instructs the automation tool: find this element, click this button, verify this value. It works until something changes. Move a button, rename a field, restructure a workflow — and every script that references those elements needs to be manually updated. The deeper problem is that scripted automation conflates two things that should be kept separate: knowing what to test and knowing how to make a machine execute the test. Those are genuinely different skills. A QA engineer with deep knowledge of how a financial application should behave doesn’t need to know how to write automation code to define valuable test cases. And an automation engineer doesn’t need to understand the business logic of a healthcare system to implement the technical layer that executes tests against it. When you force both responsibilities onto the same person — or the same artifact — you get a fragile, expensive, hard-to-scale testing practice. Domain experts who should be focused on coverage end up learning scripting languages. Automation engineers who should be building reusable infrastructure end up buried in business logic. And the test library grows larger than anyone can maintain. What Happens When You Separate the Two The alternative isn’t theoretical. It’s been running in production environments across industries for decades, and the numbers from real engagements make the maintenance difference concrete. At an upstream oil and gas software company — where applications averaged five million lines of code each, running across UNIX and Windows in mixed-language environments — testers needed deep domain expertise that took years to develop. Those testers couldn’t realistically also be automation engineers. With a keyword-driven, roles-based approach, one automation engineer could provide the technical infrastructure to keep four or five domain testers running efficiently. The domain experts wrote test cases in plain business language. The automation engineer implemented the underlying execution layer once, and maintained it independently. The maintenance speed gains were measurable and specific. When a GUI object moved within the application, updating the relevant tests took four minutes. When a new GUI element was added, two minutes. When an entire business workflow changed — the most expensive type of update under traditional scripting — the average update time was under fifteen minutes. Under any scripted automation approach, those same changes would typically cascade through dozens of individual scripts, each requiring manual intervention. At Catalina Marketing, a targeted marketing services company, the before-and-after comparison was even more direct. The same scope of automation work that previously required three engineers over four months was completed by two engineers in two months after adopting a keyword-driven approach — without replacing the existing tool infrastructure. As their Senior Test Manager put it, the new methodology let the team design tests before any code was even available, and run them the moment engineering delivered a build. The Maintenance Math Nobody Talks About The reason scripted automation looks attractive at first is that it’s fast to get started. An engineer can write scripts against a working application quickly. The problem only becomes visible later, when the application starts changing — which is always. With a keyword-driven framework, the architecture is intentionally layered. Keywords are modular, reusable test components that map to business actions rather than to specific UI elements. When the application changes, you update the underlying keyword implementation — once — and every test case that uses that keyword is automatically updated. You’re not chasing changes through hundreds of individual scripts. This is also why the approach scales in ways that scripted automation doesn’t. Keywords built for one test case are reused across others. Coverage that a team builds this sprint is available to build on next sprint. The library compounds in value rather than compounding in maintenance debt. Lockheed Martin’s Global Transportation Network program had previously tried and failed to automate their testing — not once but multiple times. The GTN system had non-standard GUI objects, timing issues between the application and the test tools, and constantly changing web browser behavior. What finally worked was a combination of significantly fewer test scripts than traditional methods would require, combined with test cases recorded in plain spreadsheet format using a Keyword vocabulary. Testers didn’t need to know the scripting language. When the Government scored the program’s performance at 100% — a perfect award fee — the improvement in the testing process was explicitly cited. The Staffing Question One of the less obvious benefits of separating test design from automation is what it does to your hiring and staffing model. If every tester needs to be both a domain expert and an automation engineer, you’re hiring for a rare combination that’s expensive to find and expensive to keep. If you can split those responsibilities cleanly — domain testers
Why Your Testing Organization Is the Reason IT Projects Are Falling Behind
The four steps that separate test organizations that enable the business from ones that slow it down When a CIO at a large financial services firm reviewed her team’s performance at the end of a particularly brutal year, she had one number that stood out more than any other: 24. As in, 24 percent of application defects were being discovered by end users — not by the testing team. Not in staging. Not before release. By real customers, in production, after the damage was already done. Two years later, that number was 5 percent. What changed wasn’t the technology stack. It wasn’t a new development methodology. What changed was the testing organization — how it was structured, how it operated, and what it was actually held accountable for. And the results reached far beyond defect rates. IT projects that had previously stalled were now being delivered in an average of 79 days. Failure rates had dropped 40 percent. For the first time, the CIO could credibly tell the board that IT was delivering business value — not just keeping the lights on. This isn’t an unusual story. It’s a repeatable one. The Problem Is Structural, Not Technical Most testing problems look like tool problems or headcount problems on the surface. Teams assume they need a better automation platform, or more engineers, or a different CI/CD setup. Those things can help at the margins. But the underlying issue is almost always organizational. The typical testing group inside a mid-size or enterprise IT department is a collection of individuals with mixed skills, inconsistent processes, and no shared methodology. Test automation is limited or unreliable. Coverage decisions are made informally. There’s no standard for what “done” looks like from a quality standpoint. Developers and testers compete for the same environment resources. And when a release slips, everyone points at testing. The problem isn’t the people. It’s the absence of structure around them. A testing organization without a defined architecture — clear roles, a shared methodology, a framework that scales — will always be a bottleneck. It will always be reactive. And it will always be the thing that gets blamed when something goes wrong in production. What a Real Transformation Looks Like The path from reactive testing to a testing organization that actually accelerates the business runs through four stages. None of them are optional. 1. Assessment — Know what you actually have Before anything else, someone needs to take an honest look at the current state. That means evaluating not just the tools in use, but the processes around them. How are test cases designed? What does the automation library actually cover? What happens when an application changes — does the test suite adapt, or does it break? How much of the team’s time goes to maintenance versus building new coverage? The assessment isn’t about finding blame. It’s about establishing a baseline that’s honest enough to build a roadmap from. In most organizations, the assessment surfaces things that leadership suspected but couldn’t quantify — and that clarity is what makes change possible. 2. Setting the right goals An assessment without a clear set of objectives just produces a report that sits on a shelf. The transformation goals need to be specific enough to measure and connected directly to business outcomes. The goals that tend to matter most: a systematic testing process deployed consistently across the development lifecycle; a framework that responds to iterative development without requiring constant rebuilding; automated regression libraries that reduce the manual workload as coverage grows; and knowledge transfer mechanisms that let the internal team own what gets built. That last one matters more than most organizations realize. A testing transformation that depends permanently on external consultants hasn’t actually transformed anything. The goal is to build something the team can run and extend on its own. 3. Deployment — This is where intellectual property makes the difference The reason most organizations don’t attempt a testing transformation on their own isn’t a lack of willingness. It’s that building a test architecture from scratch — the methodology, the framework, the role definitions, the training, the templates — takes years and carries enormous risk of getting it wrong. The value of working with an experienced partner at this stage isn’t just access to engineers. It’s access to pre-built intellectual property that has been tested across hundreds of engagements. Proven methodology means you’re not experimenting. Role-based frameworks mean you can staff intelligently, putting domain experts where domain knowledge matters and automation engineers where scripting matters, without requiring every tester to be both. The deployment phase also includes the knowledge transfer that separates a temporary fix from a lasting capability. Shadowing, reverse shadowing, mentoring alongside delivery — the goal is that by the time the engagement is complete, the client’s team is running the framework, not watching it run. 4. Results — What transformation actually looks like You know the testing organization has genuinely changed when a few specific things become true. Defects are being found earlier in the lifecycle rather than by users in production. The team can take on new projects without rebuilding test infrastructure from scratch each time. Automation coverage grows with the product rather than falling behind it. And senior leadership has visibility into testing quality as a metric — not just a feeling. The financial services CIO’s story isn’t exceptional. It’s what happens when testing is treated as a discipline rather than a checkpoint. The Question Worth Asking Most engineering leaders know something is wrong with their testing setup. Releases that require manual sign-off. Regression runs that take days. Automation libraries that nobody trusts. Coverage that sounds impressive until something gets to production that shouldn’t have. The question isn’t whether to fix it. The question is whether the fix is structural or cosmetic. Another tool won’t solve a process problem. More headcount won’t solve an architecture problem. A testing organization built on a solid framework — clear roles, reusable test assets, methodology that travels with the team —