Testing in production: understanding production tests and how to run them

Some people might look at testing in production negatively, equating it to releasing untested features or defective products with poor performance and retention rates.

However, testing in production is a crucial stage of the software development process. It lets Quality Assurance (QA) engineers examine real user behavior in the post-release phase. Furthermore, running tests in a production environment adds another layer of security against real-time bugs.

In this article, we’ll cover the types and benefits of testing in production. You will also learn about six practices for performing the examinations and metrics that indicate successful production tests.

Download glossary for web beginners

Why testing in production

Creating a staging environment takes a long time, and the result may not match the actual product. Therefore, many web developers include testing in production as a complementary phase after pre-deployment examinations.

In this section, we will go over five reasons why you should run tests on a production environment.

Improves test accuracy

The main benefit of testing in production is getting more accurate results since you do so in the same environment. Knowing that users will experience the same functionality verified in testing will increase the team’s confidence.

However, this may not happen in staging environments. Even if you replicate the production, certain elements might have non-exact data or different configuration options. This may impact the test results.

Enhances deployment frequency

Frequently releasing new code or features during testing in production also improves agility. You can respond to customer requests more flexibly, releasing changes as needed.

In addition, testing in production enables flag-driven developments with an automatic feature flag functionality in mind. This means you can safely deploy and roll back any negative modifications immediately.

Ensures a smooth transition during testing

Testing in production helps you learn and experiment with how users react to a specific feature or code.

For example, when releasing new features, QA engineers perform testing in production to check whether the software functionality works properly. Then, they use several analytic tools to execute the A/B testing and gather customers’ feedback.

In addition, testing in production allows you to manage the feature flag and analytic tools independently. You can also integrate both to get the best results.

Limits damages

The most efficient way to limit damages is by testing in production. With it, you can notice real-time defects and directly implement security measures and patches.

Gradually releasing new code or features can prevent poor deployments from damaging the production systems and negatively affecting user experience.

Noticing errors and bugs early in development takes time and effort. QA engineers must create unit tests, check the automation system, and manually verify user flows using mock data.

Allows to gather feedback

Testing in production lets you observe and monitor the system via real user feedback. Furthermore, it determines the failure or success rate of the new features or code.

To successfully conduct testing in production, ensure the application performance stays the same from the expected baseline.

Types of testing in production

In this section, we’ll overview the six most common testing in production methods:

Monitoring and diagnostics

The main purpose of monitoring and diagnostics in a production environment is to ensure the software works as intended. To achieve this, you can conduct the following tests:

• Continuous monitoring. This includes performance testing, such as examining the product’s stability, response time, scalability, and reliability as well as conducting website speed tests. Continuous monitoring helps find problems that can decrease software functionality. Aside from doing so manually, developers can also use automation tools that provide additional insights and diagnostics.
• Application monitoring. It consists of two types – real user monitoring (RUM) and Synthetic Monitoring (Simulation Testing). The former checks how the actual users interact with the application server. While the latter examines how the application APIs respond to continuous requests sent by automated visitors.
• Real-time tracking. This means checking every transaction over each layer inside an application. It lets QA engineers see the codebase and detect errors, bugs, and slow performance. In addition, real-time tracking also provides specific analysis, such as the execution stack behavior and problematic threads.

A/B testing

You can only run this type in an actual production environment to provide valuable user feedback.

The A/B testing involves releasing two versions of a web application or new feature with subtle contrasts, for example, different menu interfaces or color schemes. This will split the user base into multiple groups.

Then, assign each batch with a different variant to find which one is preferable. This statistical testing helps you to decide which version to include in future releases.

This way, developers can also learn more about customers’ needs and create products that meet their expectations.

Incremental release

This testing in production type divides product requirements into several standalone modules. Each part is treated as a sub-project and will go through the Software Development Life Cycle (SDLC) stages.

Additionally, one module will present a new feature or production infrastructure on each release. It continues until the system develops completely and implements all the intended parts.

Here are four incremental phases:

1. Requirement analysis. Identifying the software and system functional requirements.
2. Design and development. Creating functionality.
3. Testing. Examining all existing functions using various methods.
4. Implementation. Finishing the software coding design.

Also known as the iterative enhancement model, this production environment testing help achieve the goals through several actionable steps.

The incremental releases model provides two types to choose from:

• Staged Delivery Model. Build one part of the project at a time in successive phases.
• Parallel Development Model. Develop the product simultaneously as long as the resources are available. This type can help shorten the development process.

Spike testing

Spike testing in a production environment helps evaluate the software performance in extreme situations, such as a sudden load increase or decrease. Its purpose is to determine how much user traffic the product can handle before crashing.

In addition, spike testing defines how long it takes to recover from challenging circumstances. Most developers also use this method to find whether the software employs good error-handling systems.

Integration testing

Also known as Integration and Testing (I&T), this type logically merges all different components, units, and software modules and tests them as one entity.

Usually, the production consists of several modules coded by multiple programmers. Integration tests aim to find errors in their interaction when merged. Furthermore, it checks the data communication between these modules.

One common example of integration tests is the big bang approach. However, it mainly works for examining small systems.

Feedback tracking

Once developers release the software, they start observing how end users interact with the product. They usually use customer feedback tools such as Mopinion to gather data efficiently. This helps determine necessary changes in future iterations.

In addition, feedback tracking in a production environment lets developers speed up the testing and code integration process. This creates a balance between the release time and quality.

When adopting this method, remember to specify which elements users need to give feedback to so it’s easier to categorize the data.

Best practices for testing in production

After analyzing the benefits and types of testing in production, it’s time to learn some tips and tricks to improve your work further.

Create test data

When testing in production, the information used should represent the software’s actual condition. Hence, you may need to create sample data or use a suitable replacement for the test cases.

There are several ways to generate test data:

• Create the data manually.
• Copy the data from the production to the test environment.
• Get the data from client systems.
• Use automated test data generation tools.

Before creating test data, developers need to take multiple pre-steps or set configuration options. As it is a time-consuming process, we recommend taking care of it in advance.

Name the test data realistically

Writing a good test data name helps create a compelling and organized production testing process. Furthermore, it can inform QA engineers what actions to take and what kind of feedback developers are expecting.

Here are three tips for creating a good test data name:

• Accurately represent real-life scenarios.
• Describe the situation concisely.
• Create reusable test data names to examine multiple contexts.

It’s important to ensure your test data accurately reflects real user interactions to avoid encountering errors during the testing phase. For instance, using an incorrect HTTP request method in your test data could result in a 405 error.

Avoid using real user data

Using real data can be an effective approach when testing in production and troubleshooting software functionality. In addition, it helps imitate the data application types seen in the production environment.

However, exposing Personally Identifiable Information (PII) may result in several serious risks, such as security breaches. This also means you’re violating data privacy laws, including the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA).

Other disadvantages include:

• Having test and production data mixed in.
• Exposing loopholes in business-critical processes.
• Possible data loss.
• Unintended consequences on other software systems.
• Noise in the production system logs due to bot and script activity.
• Relying on end users to provide feedback about system faults.

Therefore, to prevent breaches and legal issues, employ these security techniques in the production testing environments:

• Use tokens. Replace sensitive production data with a generic placeholder value that doesn’t imitate the original format.
• Keep it anonymous. Utilize realistically generated value by randomization and generalization.
• Apply format-preserving encryption. Secure the sensitive information while still keeping the original format. This keeps the data relevant.
• Pseudo-anonymization. Insert the real PII next to the randomized production data and switch it in the mapping table. This allows you to restore the original information when needed. However, ensure the mapping table is encrypted.
• Generate synthetic data. Use mock information that has a similar format and valid data linkages. There are many synthetic data generators, such as Avo Automation, MOSTLY AI, and Mockaroo.

Create credentials for testing the application

To maintain data protection while ensuring the software functionality, create test credentials, and make them accessible to the team in the testing environment.

With sample credentials, you can test the software parts or REST APIs without applying changes directly from the existing account. It lets you act as a user and explore the software to find potential drawbacks.

However, test accounts usually have some restrictions. For example, they might not allow interacting with the data inside the real credentials or accessing certain administration parts of the application.

Before creating them, review the company’s safety policy and consult with the security team.

There are a few aspects to consider when configuring the test credentials:

• Agree with the team on the test account’s role and permissions.
• Ensure it contains representative data.
• Consider cross-tenancy issues and create sets for two separate production environments.
• Provide a permission matrix to ensure the roles and privileges are clear. This also helps prevent false positive reports.

Test when the project is under low load

Performing testing in production, especially during business hours, may increase the chances of system failure. Furthermore, it can result in a poor user experience. Therefore, we recommend performing the production environment testing during off-peak hours, for example, overnight.

However, conducting examinations in advanced software may take hours. Therefore, schedule a maintenance plan accordingly and send emails to notify users that production testing will happen.

In addition, it is best to enable the feature flag option to mitigate software errors.

Metrics indicating a successful production test

Production testing metrics help monitor and rate the tests you’ve performed. They convey a result or a prediction based on the data combination used.

Furthermore, production testing metrics display the team’s examination progress, analyze the software quality, monitor productivity, measure changes, identify areas for improvement, and test the latest procedure or technology to determine whether the product needs more modification.

Production testing metrics consist of three types:

• Process metrics. Define the product’s execution and characteristics. They are used for improving and maintaining the SDLC process.
• Product metrics. Determine the product’s design, quality, size, complexity, and performance. Developers use these metrics to improve software quality.
• Project metrics. Measure the efficiency and effectiveness of the project team or testing tools used.

There are five essential factors to remember before creating the production testing metrics:

• Choose the target audience carefully.
• Determine each metric’s goal.
• Plan the measurements according to the product specifications.
• Evaluate the revenue from each statistic.
• Match the calculations to the project life cycle.

Manual test metrics

Manual testing is time-consuming, as QA engineers need to perform it step-by-step. However, it helps them check thoroughly and examine the system configurations in more complex circumstances.

This technique consists of two metrics:

• Base metrics. Contain data analytics gathered from the test cases’ development and execution. By generating the project status reports, project managers will receive and review using these metrics. This technique tracks data throughout the SDLC, from collecting the total number of test cases to how many of them are complete, failed, or blocked.
• Calculated metrics. Contain data collected in the base metrics, such as the number of test coverage. Project managers or leaders use them for test reporting purposes.

Manual testing includes many important metrics, including absolute, derivative, result, and predictive rating. We have listed some of the commonly used ones below.

Absolute numbers

These contain one-dimensional values that QA engineers use to derive production testing metrics. To use them, record these values during the test cases’ development and execution throughout the software testing life cycle.

Absolute metrics contain 12 calculations:

• Total test cases
• Passed test cases
• Failed test cases
• Blocked test cases
• Found defects
• Rejected defects
• Accepted defects
• Deferred defects
• Critical defects
• Planned test hours
• Actual test hours
• Bugs found after shipping

Derivative metrics

Using absolute numbers as standalone metrics is insufficient to assess the testing quality. Therefore, you must complement it with the derivative metrics. This way, you’ll know how to fix problems during the product testing processes.

The test tracking and efficiency formulas help QA engineers understand the testing efficiency and track their achievements. They also show any relevant product defects.

Usually, developers and QA engineers apply the following formulas:

• Passed test cases % = (number of passed test cases/total executed tests) x 100
• Failed test cases % = (number of failed test cases/total executed tests) x 100
• Blocked test cases % = (number of blocked test cases/total executed tests) x 100
• Fixed defects % = (number of defects fixed/defects reported) x 100
• Accepted defects % = (number of valid defects as confirmed by developers/total reported defects) x 100
• Defects rejected % = (number of invalid defects rejected by developers/total reported defects) x 100
• Defects deferred % = (number of defects deferred for future releases/total reported defects) x 100
• Critical defects % = (number of critical defects/total reported defects) x 100
• Average time for developers to fix defects % = (time needed for bug fixes/number of bugs) x 100

Note that these formulas only provide information about the test coverage and quality.

Meanwhile, test effort metrics establish the baselines for future test planning. However, the results produced are averages – half is over the value, while others are below.

The following are the most important formulas to show the test effort:

• Number of tests run per one period % = number of tests run/total time
• Test design efficiency % = number of tests designed/total time
• Test review efficiency % = number of tests reviewed/total time
• Amount of bugs or defects per test hour % = number of bugs or defects/total test hours
• Amount of bugs per test % = number of bugs/total test numbers
• Average time for developers to test a bug fix % = total time between bug fix to retest all defects/total defects

The test effectiveness metrics measure the bug-finding ability and test set quality. They show a ratio of the total defects identified before the deployment to the faults found before and after product release.

There are two ways to calculate the test’s effectiveness:

• Metric-based using defect containment efficiency % = (bugs found in test/total bugs found (bugs found in test + after shipping)) x 100
• Context-based means using team assessment to rate the effectiveness of the test rate.

In comparison, the test coverage metrics measure the test efforts. They show how many of the software parts were tested.

To calculate the test coverage, here are three key formulas to run:

• Test execution coverage % = (number of tests run/total tests to run) x 100
• Requirements coverage % = (number of covered requirements/total requirements) x 100
• Automation test coverage % = (automated coverage/total coverage) x 100

Also known as defect distribution, the defect metrics divide software faulty based on several aspects, such as cause, severity, and type. This technique helps identify which areas are most susceptible to error.

To calculate it, use these formulas:

• Defect density % = number of defects/total modules
• Defect severity index % = ∑ (defect x severity level)/total defect

Automated test metrics

As the name suggests, it allows you to perform automated testing using tools. Aside from reducing cost and time, this technique can increase test coverage. Moreover, the automated approach uses mainly the same metrics as the manual one.

There are many automation tools available, including TESTIFI, LambdaTest, and Katalon.

Conclusion

Testing in production is a crucial supplement to a software testing strategy. Doing so helps the team learn how the system works with real users, data, and requests.

Performing tests in a production environment gives a better understanding of the software, upgrades its quality throughout the post-production release phase, and increases business value.

Testing in production has many advantages, such as improving test accuracy, releasing updates frequently, and preventing system faults. It commonly consists of six categories:

• Monitoring and diagnostics. Conduct performance testing, application monitoring, and real-time tracking.
• A/B testing. Test two versions of a new feature or software in the production environment.
• Incremental release. Separate product requirements into software modules until the SDLC is complete.
• Spike testing. Test software on multiple occurrences, such as sudden increases or decreases in production traffic.
• Integration testing. Combine multiple units, components, and modules and test them as an entity.
• Feedback testing. Use customer feedback tools during the production environment.

To produce the best results, remember to always create the sample data, give it a helpful name, avoid using personal information, make test credentials, and run the checks during low load.

We hope this article has helped you understand how to perform testing in production. If you have any questions or suggestions, please leave them in the comments section below.

Testing in production FAQ