Currently trying new things at backupAirflowNew branch
https://github.com/php1301/WeatherForecastDataPipeline
Branches for different stages
https://github.com/php1301/WFDP-docker-eks
- Follow well-architected framework
- Requirements: IoT dht11 sensors x 3, Raspberry Pi x 1 - board, AWS Activate 1000$ credit
- Collaborators: Chuong, VA, Kien, Đào Tuấn Anh
Microservices + (Service Mesh)
- Part 1: Airflow + EKS Setup + CloudFormation + CI/CD Pipeline for airflow unit tests / integration testings + Remote logging S3 / CloudWatch + Kubernetes CKAD + ALB + SSL + Kustomize + Flux / (ArgoCD)
- Part 2: Terraform for Deploying Kafka Cluster, Fargate StatefulSet, DaemonSet Xray, LattePanda + dht11 sensors IoT + Ansible for Deploying + ETL DAG on Airflow + ELK Stack for Kafka Producer
- Part 3: AI/ML models, GradientBoost, XGBoost, Sampling + Pickle + ReactJS web + NodeJS for showing temperature + Show prediction with User inputs,…
IAM and AWS Organization
Infrastructure As Code
CI/CD
Airflow
Kafka
IoT