Job Category: IT
Job Number: 5048
The Hadoop Administrator is accountable for managing big data environments, on bare-metal, container infrastructure, or on a cloud platform. This role is responsible for system design, capacity planning, performance tuning, and ongoing monitoring of the data lake environment.
Most important skills desired:
- Hortonworks - Hadoop - HDP - on-prem / cloud. HDF a big plus
- Big Data tools: HBase, Hive, Phoenix, Ranger other services in Ambari.
- Shell Scripting: Bash, Perl, Python
Other desired skills / Plus:
- Streaming experience (Nifi, Kafka, Storm, Spark, Solr, etc.)
- NoSQL experience (HBase, Cassandra, MongoDB, etc.)
- Advanced knowledge in Linux including shell scripting.
- ODBC/JDBC with various clients like Spotfire, BOBJ, Tableau etc
- Experience installing, upgrading, configuring, and maintaining a Hadoop cluster.
- Responsible for implementation and ongoing administration of Hadoop infrastructure of some or all of the big data systems
- Cluster maintenance as well as creation and removal of nodes.
- HDFS support and maintenance.
- Setup security using Kerberos and AD on Hortonworks clusters.
- Setting up Linux users, setting up Kerberos principals and testing HDFS, Hive, Pig and MapReduce access for the new users.
- Automate operations, installation and monitoring of the Hadoop Framework specifically: HDFS, Map/Reduce, Yarn, HBase.
- Automate the setup of Hadoop Clusters
- Ability to write, debug, and maintain automation scripts / jobs
- Continuous evaluation of Hadoop infrastructure requirements and design/deploy solutions (high availability, big data clusters, etc)
- Cluster Monitoring and Troubleshooting
- Manage and review Hadoop log files
- Works with application teams to install operating system and Hadoop updates, patches, version upgrades as required
- On-Call responsibilities, create documentation, resolve support tickets and meet Business SLAs.
- Supporting production environment with availability
- Experience with Hadoop Architecture and Big Data users to implement new Hadoop eco-system technologies to support multi-tenancy cluster
- Experience in working with the vendor(s) and user communities to research and test new technologies to improve the technical capabilities of existing Hadoop clusters