How to configure HDFS cluster using Ansible?

3 min readJan 7, 2021

Ansible:-

Ansible is an open-source software provisioning, configuration management, and application-deployment tool enabling infrastructure as code. It runs on many Unix-like systems, and can configure both Unix-like systems as well as Microsoft Windows. It includes its own declarative language to describe system configuration.

Hadoop:-

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.

Lets perform a task of how to use hadoop with the ansible playbook and also configuration of hadoop cluster using the ansible playbbok. Lets first see the task:-

Task:-

🔰 Configure Hadoop and start cluster
services using Ansible Playbook.

Solution:-

Prequisites for the following task:-

You must have the ansible configured in the base os.
Also in the inventory file of the ansible must have configured the target node.

Follow the bellow mentioned steps to solve the above task,

The below mentioned steps can be used for hadoop cluster configuration on the namenode, targetnode and clientnode.

Step 1: Copying the hadoop and jdk software :-

Step 2: Installing the java and hadoop software :-

Step 3: Create the Directory:-

For client node we dont need to create a directory.

Step 4:Configuration the core-site.xml:-

“Blockinfile” module of ansible is used to write the block of code in the hdfs-site and in core-site file. We have use the insertafter property of the blockinfile to insert the block of code after the <configuration> line inside the hdfs and core site file.

Step 5: Cofiguration the hdfs-site.xml:-

Step 6: Start datanode or namenode:-

We have use the “command” module instead of the “service” module to start the hadoop namenode or targetnode service because in the service module we can’t give options.

Step 7:-Format the namenode:-

Steps to be followed for namenode only:-