Building an Automated A/B Testing Framework with Python and MongoDB

A/B testing is essential for optimizing marketing campaigns, website designs, pricing strategies, and product features. By automating the test setup, data collection, and analysis using Python, MongoDB, and Pandas, businesses can make data-driven decisions faster.

Lillqvist Strat specializes in automation that maximizes business performance through data-driven insights.

---

1. A/B Testing Basics

A/B testing involves:

• Creating two or more variations (A = control, B = experiment).
• Randomly assigning users to each variation.
• Measuring key performance metrics (e.g., conversion rates, engagement).
• Analyzing statistical significance to determine the better variation.

Key benefits of automating A/B testing:
✅ Faster test execution
✅ Eliminates manual errors
✅ Ensures real-time data tracking

---

2. MongoDB Test Data Storage

MongoDB will store test variations, user interactions, and test results.

Setting Up the Database

from pymongo import MongoClient

client = MongoClient("mongodb://localhost:27017/")
db = client["ab_testing"]
test_collection = db["experiments"]

Defining the A/B Test Schema

experiment = {
    "test_name": "Homepage Button Color",
    "variant_A": {"color": "blue", "conversion_rate": 0},
    "variant_B": {"color": "red", "conversion_rate": 0},
    "start_date": "2025-03-01",
    "end_date": "2025-03-10",
    "status": "Running"
}

test_collection.insert_one(experiment)

---

3. Python Test Logic

Users will be randomly assigned to either Variant A or B, and their interactions will be stored.

Assigning Users Randomly

import random

def assign_variant(user_id):
    variant = "A" if random.random() < 0.5 else "B"
    test_collection.update_one(
        {"test_name": "Homepage Button Color"},
        {"$inc": {f"variant_{variant}.conversion_rate": 1}}
    )
    print(f"User {user_id} assigned to Variant {variant}")

assign_variant("User123")

---

4. Pandas Results Analysis

Once the test is complete, we analyze the results using Pandas.

Fetching and Analyzing Test Data

import pandas as pd

# Load test results
test_data = test_collection.find_one({"test_name": "Homepage Button Color"}, {"_id": 0})
df = pd.DataFrame.from_dict(test_data, orient="index").drop(["test_name", "status", "start_date", "end_date"])

# Calculate conversion rates
df["conversion_rate"] = df["conversion_rate"] / df["conversion_rate"].sum()
print(df)

# Export results to Excel
df.to_excel("ab_test_results.xlsx", index=True)

---

5. Implementation Tips

• Run tests long enough to gather statistically significant data.
• Segment results (e.g., new vs. returning customers) for deeper insights.
• Automate notifications to alert when a winning variant is detected.

Sending Test Completion Notifications

import smtplib
from email.mime.text import MIMEText

def send_notification(email, subject, message):
    sender_email = "your_company@example.com"
    password = "yourpassword"

    msg = MIMEText(message)
    msg["Subject"] = subject
    msg["From"] = sender_email
    msg["To"] = email

    with smtplib.SMTP("smtp.example.com", 587) as server:
        server.starttls()
        server.login(sender_email, password)
        server.sendmail(sender_email, email, msg.as_string())

# Example: Notify stakeholders when the test ends
send_notification("manager@example.com", "A/B Test Completed", "The test has ended. Check results for insights.")

---

Conclusion

Automating A/B testing with Python, MongoDB, and Pandas ensures fast, reliable, and data-driven decision-making.

Lillqvist Strat provides cutting-edge automation solutions that drive higher conversions and business growth. Let’s optimize your strategy today!