Skip to content

Observability

A Lambda-Based AWS Cost Anomaly Detector with Slack Alerts

TL;DR: I built a Lambda function that checks AWS Cost Explorer daily, detects spending anomalies by comparing against rolling averages, forecasts monthly spend, and sends a formatted Slack report every morning.

The Problem

AWS bills are full of surprises. A forgotten NAT Gateway running at $32/day, an accidentally provisioned large RDS instance, a Lambda function in a retry loop -- these things silently drain budget for days before anyone notices. The native AWS Cost Anomaly Detection service exists but is limited in customization and alerting flexibility. I wanted daily cost intelligence in Slack, with anomaly detection tuned to my thresholds.

What I Built

aws-cost-optimizer is a Python Lambda function triggered daily by EventBridge at 8 AM UTC. It pulls the last 7 days of spending from AWS Cost Explorer (grouped by service), computes a rolling average per service, flags any service that exceeds its average by a configurable threshold, forecasts the current month's total, and posts a detailed report to Slack.

  • Daily cost breakdown -- top services by spend over the trailing 7-day window, formatted as a clean Slack message
  • Anomaly detection -- compares each service's latest daily cost against its rolling average; configurable percentage threshold (default: 30%)
  • Monthly forecasting -- uses Cost Explorer's forecast API to project end-of-month spend and compares it against a budget target
  • Slack alerts -- formatted messages with service-by-service breakdown, flagged anomalies, forecast vs. budget status, and trend indicators
  • Fully IaC -- Terraform deploys the Lambda, EventBridge rule, IAM role, and CloudWatch log group

Tech Stack

Python, AWS Lambda, AWS Cost Explorer, EventBridge, Slack Webhooks, Terraform, CloudWatch

Usage

# Deploy
cat > terraform/terraform.tfvars <<EOF
slack_webhook_url     = "https://hooks.slack.com/services/XXX/YYY/ZZZ"
monthly_budget        = 500
anomaly_threshold_pct = 30
EOF

cd terraform && terraform init && terraform apply

The Lambda runs automatically each morning. To test manually:

aws lambda invoke --function-name aws-cost-optimizer \
  --payload '{}' response.json
cat response.json

Results

The first week caught a $28/day NAT Gateway that had been running unused for two months. Monthly forecasting flagged a projected 40% budget overrun mid-month, giving enough lead time to right-size resources. The daily Slack report became the team's morning cost standup -- a two-second glance that replaces logging into the AWS console and navigating to Cost Explorer.

GitHub: https://github.com/basel5001/aws-cost-optimizer

Measuring What Matters: A DORA Metrics Dashboard Built on the GitHub API

TL;DR: I built a FastAPI dashboard that pulls deployment data from the GitHub API and calculates the four DORA metrics in real time -- giving engineering teams actual numbers instead of gut feelings about delivery performance.

The Problem

Every engineering manager has been asked "how fast can we ship?" and answered with a vague "pretty fast." DORA metrics -- deployment frequency, lead time for changes, mean time to recovery (MTTR), and change failure rate -- are the industry standard for measuring software delivery performance. But most teams either pay for expensive platforms or simply never measure. I wanted a self-hosted, open-source alternative that pulls directly from where the work already happens: GitHub.

What I Built

devops-dashboard is a Python application that connects to the GitHub API, ingests deployment and pull request data, and computes all four DORA metrics with historical trends. A clean web dashboard presents the numbers alongside DORA's Elite/High/Medium/Low benchmarks so teams can see exactly where they stand.

  • Deployment Frequency -- calculated from GitHub Deployments or tagged releases, shown as daily/weekly/monthly rates
  • Lead Time for Changes -- measured from first commit on a branch to production deployment, broken down by percentile
  • Mean Time to Recovery -- tracked by correlating incident-labeled issues with the deployment that resolved them
  • Change Failure Rate -- ratio of deployments followed by a rollback or hotfix within a configurable window
  • Trend charts -- week-over-week sparklines for each metric so you can spot regressions early
  • Multi-repo aggregation -- track metrics across your entire organization, not just one repo

Tech Stack

Python, FastAPI, GitHub REST API, SQLite, Jinja2, Chart.js, Docker

Usage

# Clone and start
git clone https://github.com/basel5001/devops-dashboard.git
cd devops-dashboard
cp .env.example .env  # add GITHUB_TOKEN
docker compose up -d
# Open http://localhost:8000

Results

Within the first sprint of running the dashboard, the team discovered that lead time had ballooned to 4.2 days -- well above the "Elite" threshold of under one day. The root cause turned out to be a stale review policy requiring two approvals on trivial changes. After adjusting the policy, lead time dropped to 1.1 days. Without the numbers, that bottleneck would have stayed invisible.

GitHub: https://github.com/basel5001/devops-dashboard

A Self-Hosted Datadog Alternative in One Docker Compose Command

TL;DR: I built a full observability stack -- metrics, logs, traces, alerting, and AI anomaly detection -- that starts with a single docker compose up -d.

The Problem

Datadog is excellent but expensive. At $23+ per host per month for infrastructure monitoring alone, costs scale fast when you're running multiple environments. I needed production-grade observability -- not a toy Grafana dashboard -- without the SaaS bill. And I wanted it deployable in under two minutes for any new project.

What I Built

observability-stack bundles Prometheus, Grafana, Loki, Tempo, Alertmanager, Node Exporter, cAdvisor, and a custom AI anomaly analyzer into a single Docker Compose file. Every component is pre-wired: Grafana has datasources and dashboards provisioned on first boot, Prometheus scrapes all the right targets, and Alertmanager routes alerts through the AI analyzer before sending notifications.

  • Full metrics pipeline -- Prometheus + Node Exporter + cAdvisor with pre-built dashboards for host and container metrics
  • Centralized logging -- Loki ingests logs from all containers; query them directly in Grafana with LogQL
  • Distributed tracing -- Tempo receives OTLP traces over gRPC (port 4317) and correlates them with logs and metrics
  • AI anomaly detection -- a Python service connected to AWS Bedrock analyzes alert patterns and identifies root causes
  • Pre-built dashboards -- Node Overview (CPU, memory, disk, network) and Container Overview (per-container resource usage) ready on boot

Tech Stack

Docker Compose, Prometheus, Grafana, Loki, Tempo, Alertmanager, Node Exporter, cAdvisor, Python, AWS Bedrock

Usage

cp .env.example .env   # configure credentials
docker compose up -d   # start everything

# Access points:
# Grafana:      http://localhost:3000  (admin/admin)
# Prometheus:   http://localhost:9090
# Alertmanager: http://localhost:9093
# AI Analyzer:  http://localhost:8000

Add a custom scrape target:

# configs/prometheus/prometheus.yml
scrape_configs:
  - job_name: "my-service"
    static_configs:
      - targets: ["my-service:8080"]

Results

New projects get full observability in under two minutes. The pre-wired dashboards mean I stop building the same Grafana panels from scratch every time. The AI analyzer has caught two subtle memory leak patterns that static thresholds missed -- it correlates metric trends with log spikes in ways that simple alerting rules cannot.

GitHub: https://github.com/basel5001/observability-stack