Performing remote debugging at user sites

Project: A major contractor created a widely-distributed system for the U.S. Department of Defense designed to assess military readiness for a variety of emergency situations.

Problem: To remain effective, this system needed close to 100 percent uptime. But the budget wouldn't cover flying senior support engineers to multiple sites to track down every bug that appeared during operational test.

Solution: The contractor was already using Aprobe technology to find bugs in its integration testing lab. They soon realized its power could be extended to remote debugging.

Probes were defined by the contractor in the contractor's test lab, then sent by e-mail or ftp to user sites. A technician at each site loaded the probes into the Aprobe directory and re-started the system.

As the application ran, trace data was logged. The trace captured code-level, system-level and hardware/software configuration details, minimizing the data each site had to supply manually.

At any point, the trace data could be e-mailed back to support staff, who would step through the trace. This helped them zero in on bugs quickly.

They would also use Aprobe to create a temporary patch to test a fix. When the fix worked, it could be left in place until the next build was ready.

Remarks: The contractor was able to debug the problem in the customer's environment with- out burdening the customer. Doing remote debugging avoided the high cost and delays of sending senior support staff to perform on-site troubleshooting. The probes had virtually no impact on system performance. These application-specific probes are being used throughout the life of the military system to ensure rapid time-to-resolution of any issues.