Splunk Engineer - ITSI

As a Senior Splunk Engineer, you will be responsible for implementing, configuring, and managing enterprise-level Splunk Cloud and Splunk ITSI solutions to provide real-time insights into the performance, health, and security of our IT infrastructure. You will develop and manage key observability tools, such as dashboards, glass tables, service trees, and KPIs, enabling teams to proactively monitor, diagnose, and resolve issues. You will also support the continuous optimization of our Splunk environment, automating processes, onboarding new data sources, and ensuring the stability and performance of the platform. This is an exciting opportunity for someone who enjoys solving complex problems, working with cloud technologies, and driving improvements in IT operations. Key Responsibilities : Splunk Cloud & ITSI Engineering : - Splunk Cloud Deployment & Configuration : Lead the deployment, configuration, and management of Splunk Cloud across a large enterprise environment (5000 servers), ensuring seamless integration with other enterprise tools and systems. Design the architecture to support scalable observability and efficient data ingestion. - Splunk ITSI (IT Service Intelligence) : Engineer and configure Splunk ITSI, including the creation of Glass Tables, Service Trees, KPIs, and correlation searches to provide deep insights into service health, performance, and availability across critical IT systems. - Advanced Analytics & Search Logic : Create complex searches using Splunk SPL (Search Processing Language) to build operational and security dashboards that provide actionable insights for operations and security teams. Leverage correlation searches to identify potential incidents, anomalies, or security threats. - Log & Event Management : Manage large-scale log and event data for IT operations, security, and business intelligence. Focus on streamlining log ingestion, ensuring compliance with corporate policies, and optimizing storage and query performance. Cloud Services & Automation : - Cloud Infrastructure Integration : Integrate Splunk with cloud-native services like AWS EC2, Lambda, CloudWatch, and Azure compute functions to monitor cloud environments and applications at scale. Ensure seamless interoperability between Splunk and cloud-based systems. - Automation & Efficiency Improvements : Drive automation efforts related to data onboarding, log management, and incident response. Work on building automated data ingestion pipelines, alerting systems, and dashboard updates. Develop scripts to streamline maintenance tasks, reduce manual intervention, and enhance operational efficiency. - Performance Optimization : Conduct Splunk health checks, reviewing deployment architecture and search performance. Provide recommendations and implement solutions to optimize system performance, scalability, and data processing times. Incident Response & Troubleshooting : - Incident Investigation & Root Cause Analysis : Lead post-incident investigations and root-cause analysis for performance issues, data inconsistencies, or service disruptions. Work closely with engineers and IT teams to troubleshoot and resolve issues in real-time. - Notable Event Aggregation & Correlation : Configure and maintain correlation searches and notable event aggregation policies within Splunk ITSI to help identify critical incidents early and trigger the appropriate workflows and alerts. Design & Documentation : - Design & Architecture : Create low-level design documentation independently and collaborate with senior architects to define high-level design solutions. Support the prototyping and POC (Proof of Concept) phases for new use cases or features. - Documentation & Reporting : Develop comprehensive documentation for Splunk configurations, dashboards, and search queries, making it easier for other team members to understand, troubleshoot, and optimize. Create custom reports and dashboards as requested by various internal teams, providing actionable insights into IT & Knowledge Sharing : - Cross-functional Collaboration : Work closely with the security, compliance, cybersecurity, and operations teams to ensure that Splunk meets corporate log management standards and SIEM requirements. Ensure alignment with internal data privacy and security policies. - Training & Knowledge Transfer : Provide training and knowledge transfer to operational teams, including IT engineers and ops analysts, to ensure they have the necessary skills to monitor, manage, and optimize the Splunk environment. Share best practices related to troubleshooting, query optimization, and system performance. - Stakeholder Engagement : Interface regularly with internal stakeholders (e.g., IT support, business units, risk management, and security) to understand their needs, gather requirements, and provide guidance on the effective use of Splunk tools for real-time monitoring and analysis. System Maintenance & Support : - 24/7 Monitoring & Support : Ensure continuous monitoring of the Splunk environment to detect and address performance bottlenecks, service degradation, or outages. Provide timely and effective support to resolve incidents as part of an enterprise-level 24/7 operations model. - Change Management : Participate in change management processes for all Splunk upgrades, patches, and configuration changes. Ensure proper testing and validation of changes in a controlled environment before deployment. Continuous Improvement & Scalability : - Performance & Capacity Planning : Evaluate the growth of Splunk data and query performance. Continuously identify ways to enhance the capacity and scalability of the Splunk environment as the volume of data and complexity of queries increases. - Continuous Monitoring & Optimization : Implement ongoing optimization efforts to improve Splunk's availability, performance, and cost-efficiency. Regularly review and improve system performance based on Splunk's best practices and evolving business needs. Required Qualifications : Experience & Expertise : - Minimum 2-5 years of experience in Splunk Enterprise and Splunk Cloud administration and support. - Extensive hands-on experience in implementing and optimizing Splunk ITSI solutions at scale (Glass Tables, Service Trees, KPIs, Correlation Searches). - Proven track record of enterprise-scale deployments (5000 servers) and operational support of Splunk Cloud environments. - Solid experience working with AWS (EC2, Lambda, CloudWatch), Azure compute functions, or GCP for cloud-based solutions. Technical Proficiency : - Expertise in Splunk SPL (Search Processing Language), Splunk ITSI, and Splunk Infrastructure Monitoring Enterprise Edition. - Strong knowledge of log management, SIEM solutions, and data analytics. - Experience with system and query optimization, health checks, and performance tuning for large Splunk : - Splunk Certification (Power User, Admin, ITSI Professional, or equivalent) is preferred. Cloud Platforms : - Hands-on experience with AWS, Azure, or Google Cloud Platform for monitoring and managing cloud services. Preferred Qualifications : - Automation & Scripting : Experience with scripting and automation tools (e.g., Python, Shell scripting, Ansible, or Terraform) to automate routine tasks and optimize data pipelines. - Security & Compliance : Experience in SIEM operations and log management in compliance with industry standards (e.g., SOC2, GDPR, HIPAA) and security best practices. - Collaboration & Leadership : Strong collaboration and interpersonal skills to work effectively with cross-functional teams. Ability to mentor junior engineers and lead by example. Why Join Us ? - Career Growth : Join a fast-paced, innovative company offering significant opportunities for professional development and growth. Expand your skillset with new technologies in the Splunk ecosystem. - Collaborative & Dynamic Environment : Work alongside passionate, highly skilled professionals in a collaborative environment where your contributions directly impact our success. - Cutting-edge Technology : Work with Splunk Cloud, Splunk ITSI, and other leading-edge technologies, solving complex challenges at the enterprise scale. - Competitive Compensation : Competitive salary and benefits package designed to reward your expertise and contributions. (ref:hirist.tech)

Location: in, IN

Posted Date: 5/9/2025