Big Data Engineer

Big Data Engineer

Prosum Technology Services
5 - 8 Years
Not Specified

Job Description

As a Big Data Engineer, you will be responsible for working on applications and services that handle all types of transactions. You will work in a fast-paced environment where continuous innovation and experimentation are a given. You will master both established and cutting-edge technologies like Hadoop, Hive, Spark, Presto, Oracle, Casandra, Storm, Kafka, Druid, MongoDB, CouchBase, among others to build data validation framework.

  • Develop the vision for an ever-evolving Data validation improvement strategy that not only satisfies current needs but can be easily adapted to future needs as new sources of data input and requirements for data output are identified
  • Designing, testing, deploying and documenting data validation procedures and their outputs
  • Utilize data validation tooling to profile the project source data, define or confirm the definition of the metadata, cleanse and accurately check the project data, check for duplicate or redundant records, and provide information on how to proceed with backend ETL Processes
  • Partner with data stewards to provide summary results of data validation analysis, which will be used to make decisions regarding how to measure business rules and quality of the data
  • Develop methods to cleanse, manipulate and analyze large datasets, structured and unstructured data using Hadoop platform
  • Expert on data architecture, data models, data governance and data flow diagrams
  • Collaborating with peers and seniors both within their team and across the organization
  • Experience with Cloud, industry-specific and data privacy, protection regulatory compliance requirements (GDPR, CCPA, PCI, FFIEC etc.)
  • Thorough knowledge of agile software delivery models (Kanban, Scrum, Less, SAFe, etc.)
  • Design, develop, test, and debug large scale complex platform using big data technologies
  • Develop tools and automation for the effective management and operation of Big Data platform
  • Collaborate with architects, engineers, and business on product design and feature
  • Exhibit a strong backbone and challenge the status quo when needed
  • Demonstrate a high level of curiosity and keep abreast of the latest technologies
  • Attention details are a must-have skill, show pride of ownership and strive for excellence in everything they do

Job Requirements:
  • 5 + years of software development experience
  • BS in Computer Science or related degree required. MS preferred
  • A mastery of how data validation is measured, including understanding of completeness, uniqueness, validity, accuracy, integrity, timeliness
  • Experience programming languages like Python, Java, C/C++, Scala or Go
  • Expert in Big Data Technologies such as Hadoop, HBase, Yarn, Presto, Kafka, Apache Spark
  • Experience in database/storage technologies, systems like DB2, Oracle, Cassandra, CouchBase, Mainframe
  • Competent in design/implementation for security, quality, reliability, availability, scalability, and performance
  • Skilled in software engineering tools and best practices

Similar Jobs

People Also Considered

Career Advice to Find Better