Cloud/Bigdata

Job Description

"The full stack Data Engineer will design, develop, implement, test, document, and operate large-scale, high-volume, high-performance data structures for analytics.

Primary Responsibilities: 

• Well versed with BigData tools and technologies as Spark, Scala, Kinesis and Kafka.
• Implement data ingestion routines both real time and batch using best practices in data modelling, ETL/ELT processes leveraging AWS technologies and Big data tools.
• Provide on-line reporting and analysis using business intelligence tools and a logical abstraction layer against large, multi-dimensional datasets and multiple sources.
• Gather business and functional requirements and translate these requirements into robust, scalable, operable solutions that work well within the overall data architecture.
• Produce comprehensive, usable dataset documentation and metadata.
• Provides input and recommendations on technical issues to the project manager.
• Experience with Kafka, Flume and AWS tool stack such as Redshift and Kinesis are preferred.
• Experience building on AWS using S3, EC2, Redshift, Glue, DynamoDB, Lambda, QuickSight, etc.
• Experience using software version control tools (Git, Jenkins, Apache Subversion)
• AWS certifications or other related professional technical certifications
• Experience with cloud or on-premise middleware and other enterprise integration technologies
• Experience in writing MapReduce and/or Spark jobs
• Demonstrated strength in architecting data warehouse solutions and integrating technical components
• Good analytical skills with excellent knowledge of SQL.

Basic Qualifications:

• 4+ years of work experience with very large data warehousing environment
• Excellent communication skills, both written and verbal
• 7+ years of experience with detailed knowledge of data warehouse technical architectures, infrastructure components, ETL/ ELT and reporting/analytic tools.
• 3+ years of Python and/or Java development experience
• 3+ years' experience in Big Data stack environments (EMR, Hadoop, MapReduce, Hive)
• Flexible and proactive/self-motivated working style with strong personal ownership of problem resolution.
• Excellent communicator (written and verbal formal and informal).
• Ability to multi-task under pressure and work independently with minimal supervision.
• Strong verbal and written communication skills.

Must be a team player and enjoy working in a cooperative and collaborative team environment.