Grafana Stack Consultant - CI/CD Pipeline

About the Role We are seeking a highly skilled Grafana Stack Consultant with deep expertise in managing and scaling self-hosted observability platforms. This is a full-time contract opportunity that offers the flexibility of working remotely from anywhere. Our existing observability infrastructure comprises Grafana, Loki, Tempo, and Mimir, operating in a Docker Swarm environment across multiple Linux VMs. The system ingests approximately 500GB of logs per day and uses Wasabi S3 as the object storage backend. Recently, the platform has been facing performance degradation, including slow search queries, delays in log ingestion, and periodic outages affecting system observability. We are looking for a consultant who can analyze, troubleshoot, and optimize this stack to improve reliability, scalability, and overall system Responsibilities : - Conduct in-depth performance analysis of the Grafana observability stack, particularly Loki, Tempo, and Mimir - Identify and remediate performance bottlenecks in log ingestion, indexing, and query execution - Optimize system configurations for enhanced throughput, latency, and fault tolerance - Troubleshoot and resolve instability and downtime in the observability pipeline - Recommend and implement best practices for operating the Grafana stack in Docker Swarm environments - Collaborate with internal engineering teams to guide architecture improvements and ensure observability stack aligns with reliability goals - Document configuration changes, optimization steps, and recommended operational Skills & Qualifications : - 45 years of hands-on experience with self-hosted Grafana, Loki, Tempo, and Mimir - Strong understanding of observability architectures, including metrics, logs, and tracing - Proven track record of operating these components at scale, especially in high-ingestion environments (e.g., 500GB/day) - Proficiency in container orchestration, especially Docker Swarm; familiarity with Linux VM-based deployments - Experience integrating and tuning object storage backends such as Wasabi S3 or other S3-compatible services - Ability to diagnose complex issues across distributed systems and propose effective, sustainable solutions - Excellent problem-solving skills and ability to work independently in a remote (Nice to Have) : - Experience migrating from Docker Swarm to Kubernetes (K8s) - Familiarity with CI/CD practices and infrastructure automation - Previous consulting experience in infrastructure or observability-focused roles (ref:hirist.tech)

Location: pune, IN

Posted Date: 6/11/2025