Big Data Engineer

Senior Big Data Engineer

Come work at a place where innovation and teamwork come together to support the most exciting missions in the world!

Description: 

We are seeking a talented Sr. Big Data Engineer to deliver roadmap features of Enterprise TruRisk Platform which would help customers to Measure, Communicate and Eliminate Cyber Risks. 

Working with a team of engineers and architects, you will be responsible for prototyping, designing, developing and supporting a highly scalable, distributed SaaS based Security Risk Prioritization product. 

This is a fantastic opportunity to be an integral part of a team building Qualys next generation platform using Big Data & Micro-Services based technology to process over billions of transactions data per day, leverage open-source technologies, and work on challenging and business-impacting initiatives. 

 

Responsibilities: 

• Be the thought leader in data platform and pipeline along with Risk Evaluation. 

• Provide technical leadership to the engineering organization on data platform design, roll out and evolution. 

• Liason to product teams, professional services and sales engineers on solution and trade-off reviews and represent engineering in such conversations. 

• Drive technology explorations and roadmaps. 

• Serve as a technical lead on our most demanding, cross-functional departments. 

• Ensure the quality of architecture and design of systems. 

• Functionally decompose complex problems into simple, straight-forward solutions. 

• Fully and completely understand system interdependencies and limitations. 

• Possess expert knowledge in performance, scalability, enterprise system architecture, and engineering best practices. 

• Leverage knowledge of internal and industry prior art in design decisions. 

• Effectively research and benchmark cloud technology against other competing systems in the industry. 

• Able to document the details so it will be easy for developers to understand the requirements. 

• Assisting developers with proper requirements and directions. 

• Assist in the career development of others, actively mentoring individuals and the community on advanced technical issues and helping managers guide the career growth of their team members. 

• Exert technical influence over multiple teams, increasing their productivity and effectiveness by sharing your deep knowledge and experience. 

• Able to share knowledge and train others. 

 

Qualifications: 

• Bachelor’s degree in computer science or equivalent 

• 6+ years of total experience. 

• 2+ years of relevant experience in design and architecture Big Data solutions using Spark 

• 3+ years experience in working with engineering resources for innovation. 

• 4+ years experience in understanding Big Data events flow pipeline. 

• 3+ years experience in performance testing for large infrastructure. 

• 3+ In depth experience in understanding various search solutions solr/elastic. 

• 3+ years experience in Kafka 

• In depth experience in Data lakes and related ecosystems. 

• In depth experience of messing queue  

• In depth experience in giving requirements to build a scalable architecture for Big data and Micro-services environments. 

• In depth experience in understanding caching components or services  

• Knowledge in Presto technology. 

• Knowledge in Airflow. 

• Hands-on experience in scripting and automation 

• In depth understanding of RDBMS/NoSQL, Oracle , Cassandra , Kafka , Redis, Hadoop, lambda architecture, kappa , kappa ++ architectures with flink data streaming and rule engines 

• Experience in working with ML models engineering and related deployment. 

• Design and implement secure big data clusters to meet many compliances and regulatory requirements. 

• Experience in leading the delivery of large-scale systems focused on managing the infrastructure layer of the technology stack. 

• Strong experience in doing performance benchmarking testing for Big data technologies. 

• Strong troubleshooting skills. 

• Experience leading development life cycle process and best practices 

• Experience in Big Data services administration would be added value. 

• Experience with Agile Management (SCRUM, RUP, XP), OO Modeling, working on internet, UNIX, Middleware, and database related projects. 

• Experience mentoring/training the engineering community on complex technical issue. 

• Project management experience 

Experience Level

Senior Level