Tag: data-driven decisions

Careers Blog, Engineering

How to Design Smarter Experiments: A Practical Guide to A/B Experiment Design

In today’s fast-paced market environment, making decisions based on data is no longer optional – it’s essential for success. Data-driven organizations routinely leverage experimentation to validate ideas in a corporate setting, assess risk, and implement changes that justify their investment of time and money. One of the most common experimentation techniques is the A/B test (also known as a split test).

Have you ever wondered if adding a new feature to your business is a worthwhile effort? An A/B test could help you answer this question by comparing two versions of a product feature, marketing asset, or workflow to determine which performs better based on a predefined metric. These experiments are not simply exploratory; they are grounded in sound statistical principles, ensuring that decisions are not only data-driven but also statistically valid.

Practical Example:
Let’s say you work for an e-commerce company and you’re exploring ways to increase the average order value (AOV). One idea is to offer product bundles to see if customers spend more when they’re offered.

  • Objectives and Competing Hypotheses:
  • Null hypothesis (H₀): Offering bundles do not increase AOV.
  • Alternative hypothesis (H₁): Offering bundles increase AOV by at least $10.
  • Primary Metric: Average Order Value (AOV)
  • Randomization: Ensures that users are assigned randomly and without bias.
  • Control Group: Users who do not receive any changes.
  • Treatment Group: Users who receive the bundle recommendations. 
  • Sample Size Calculation: Let’s say we want to detect an increase in AOV of at least $10. We also want to keep a low (5%) chance of a false positive, and a 80% chance to detect the effect in our experiment. To calculate the sample size, we need to use the standard formula for two-sample tests:

n=2*Z1-/2+Z1-/2

Where:

  • Minimum Detectable Effect (MDE) (): $10
  • Standard Deviation (): $30
  • Significance Level at 5% Z1-/2: 1.96
  • Power (1-β) Z1-: 0.84 for 80% chance we’ll detect the effect if it’s real.

Plugging in the numbers in the formula shows that we’d need at least 142 users in each group (control and treatment) to run this experiment with the desired sensitivity and confidence.

Interpreting and Acting on Results

Group Orders Avg AOV Std Dev
Control 142 $76.20 $29.80
Treatment 142 $84.90 $31.50

 

  • Statistical Analysis

After running the experiment, we observed that:

    • Control Group AOV: $76.20
  • Treatment Group AOV: $84.90
  • That’s an uplift of $8.70, which seems promising at first glance.

However, to determine if this difference is statistically significant, we need to estimate some additional values.

  • Step 1: Standard Deviation: Measures uncertainty around the difference

SE=sc2 nc+st2 nt =29.82142+31.521423.64

  • Step 2: Calculate the t-statistic: Measures how extreme our observed difference is compared to what we’d expect under the null hypothesis. For that, we subtract our target effect size ($10) from the observed difference ($8.70):

t=xt–xc–SE=(84.90-76.20)-103.71=-0.36

  • Step 3: Make a Decision: Using a significance level of 5%:
    • With 282 degrees of freedom: nc+nt-2=142+142-2=282, the critical t-value is approximately 1.65.
    • Since -0.36<1.65, our result is not statistically significant.
    • Therefore, even though there was an increase of $8.70 in AOV with bundles (which was close to our $10 target), variation was too high to be confident bundles were the real cause.

Conclusions and Recommendations

Failing to achieve statistical significance is not the end of the story. There may still be valuable insights in the experiment that the experimenter can uncover through careful interpretation and business expertise. For example, in this case, lowering the Minimum Detectable Effect (MDE) results in statistically significant evidence of an increase of AOV.

However, statistical significance does not automatically imply business value. A thorough cost-benefit analysis is essential to evaluate the true benefit of implementing any changes, taking into account additional costs, potential risks, and long-term impact.

Additionally, some user segments may respond more positively to specific changes. Running experiments on segmented groups and focusing on those with the highest impact is a common strategy to tailor the user experience (UX) to the specific needs and preferences of different audiences.

Remember: measuring differences is only part of the story. Truly data-driven decisions require caution, context, expertise, and statistical rigor!

Bibliography

Written by:

Fabián Sánchez
Data Analyst
Country: Colombia

Careers Blog, Employee Spotlight, Mismo Blog, Talent Spotlight

Alejandro Galvis: Transforming the Future at Mismo

Mismo thrives on the strength of its exceptional team. We provide top-tier remote software development teams that seamlessly integrate with companies worldwide, driving exceptional results. Many organizations have partnered with Mismo to augment their development capabilities and achieve their technology goals.

At the heart of these successful teams are the talented individuals who make Mismo what it is. We’re committed to celebrating our remarkable employees and their invaluable contributions to our company culture. Our employee spotlights showcase the genuine relationships we’ve forged with team members and clients, highlighting their dedication and the positive impact they have on our collaborative environment.

Meet: Alejandro Galvis

Engineering Lead

Country: Colombia

  • Tell us a little about yourself: I’m 31 years old, born and raised in Bogotá. I love playing video games and tennis. Currently, I’m learning to play the drums, and I would like to study photography in the future. In February of next year, I’m going to get married.

  • What initially attracted you to engineering and how have you found inspiration in this career? I studied economics and initially pursued a career in finance. However, my first job after university was as a business development consultant, where I commercialized software used to automate business processes. Through that work, I became increasingly interested in technology and decided to complete a Master’s degree in Big Data and Analytics.

  • What innovation or technical advancement have you been a part of that you consider to have made a milestone in your engineering field? I have been involved in the transformation of on-premise infrastructure to cloud environments. This shift has had significant implications for technology as a whole, but it has been a true game changer for data analytics and AI. On one hand, it has facilitated the digitalization of a company’s internal and external processes, resulting in more data being captured and made available. On the other hand, it has exponentially decreased the cost of storing and processing data.

  • What made you choose Mismo? I chose Mismo because it empowered me to work with an innovative company from the U.S. that I wouldn’t have been able to reach directly. It has given me the opportunity to experience a different culture and work with more advanced technology.

  • How would you describe the work environment at Mismo? The work environment at Mismo is truly special. It is characterized by people who love what they do and are eager to share their knowledge with others to grow together. Mismo takes great care of its employees, focusing not only on financial gains but also on their overall well-being, which is quite rare in the corporate world.

  • What learning and development opportunities have you had at Mismo and how have you utilized these opportunities to grow as a professional? At Mismo, there is a strong commitment to establishing and developing career paths, guided by each collaborator’s team lead. We are constantly encouraged to consider the roles we aspire to reach, and we are provided with guidance and educational resources to help us achieve those goals. Additionally, Mismo excels at identifying individuals interested in managing teams and has created an organizational structure that offers these individuals the opportunity to take on related responsibilities and tasks.

  • What aspects of working at Mismo’s engineering team make you proud and motivate you in your daily tasks? I’m highly motivated by the fact that we are given the freedom to develop our careers in the way we see fit. Those who are ambitious and interested in continuous improvement are provided with opportunities and resources to help them grow. Additionally, it is respected if someone is satisfied with their job and wishes to continue with the same responsibilities.

  • How do you believe the company and engineering can positively impact the world, people’s lives, and make a difference in their respective industries? Mismo has a significant impact on the lives of its collaborators and the industry by fostering a symbiotic relationship. We, the collaborators, have the opportunity to work with disruptive companies that use state-of-the-art technology and have a modern work culture. The companies that utilize Mismo’s services are connected with crucial roles they need to fill, performed by individuals with a strong work ethic and ambition.

  • What Mismo policies or practices have helped you improve your work-life balance? Mismo offers valuable tips and advice for mental health and consistently works to build a community where your manager and peers are always available to discuss problems and help you find the necessary solutions.

  • What inspiring advice would you give to someone considering a career in engineering or looking to join a company like Mismo? The present and future of the world are undoubtedly highly dependent on technology. Pursuing a career in engineering places you at the forefront of its development, providing you with the tools to shape your life as you see fit while also equipping you to make a positive impact.

  • What tools or technologies do you use most frequently in your daily work as an engineer at Mismo? I primarily use Snowflake, Python, and Airflow to transform raw data into clean and enriched datasets that can be used to derive insights. I use Tableau to effectively identify, visualize, and communicate potential areas for improvement and their corresponding solutions. Additionally, I use Retool to build low-code applications that automate manual tasks and streamline business processes.