Elasticity Task: Integrating Linux LVM with Hadoop | Increasing OR Decreasing static Partition in Linux | Automating LVM with Python Script

Arjun Singh
6 min readMar 14, 2021

Performing the tasks related to Elasticity on a Linux machine.

Credits: google images

Task Description:

Elasticity Task:

  • [A] — Integrating LVM with Hadoop and providing Elasticity to DataNode Storage
  • [B] — Increase or Decrease the Size of Static Partition in Linux.
  • [C] — Automating LVM Partition using Python-Script.

TASK A: Integrating LVM with Hadoop and providing Elasticity to DataNode Storage

Step 1: Install & Configure Hadoop

I have two VMs: ‘RHEL8 OS 1’ as Name Node and ‘RHEL8 OS 2’ for Data Node

Install java jdk and hadoop softwares in both the Nodes

Configure the Name Node

We have to format the master node

Now we can start the Hadoop service for Name node

Similarly, we configure the Data Node

Start the Data Node service

Now we can check if Data Node is connected or not using the hadoop dfsadmin -report command

We can see that the Data Node with IP address 192.168.43.79 is connected and is sharing the storage to the Name Node

Step 2: Integrate LVM with Hadoop to provide Elasticity to Data Node storage

For our initial setup of Hadoop, we have Data Node with fixed storage. We want to have elasticity in the storage, meaning that we could have varied storage sizes. It is helpful to solve the use case where the Client would need to store a File of size, suppose, 30 GB.. But Data Node only providing 20 GB of Storage. Hence, Data Node can then increase the storage by attaching a new additional Harddisk of size 10 GB and providing sufficient storage to store files. This phenomenon is what we called Elasticity in storage.

This will be achieved if we implement LVM for the storage shared to Name Node by the Data Node.

The practical is as follows:

Add 2 new HDD to the Data Node virtual machine of size 2 GB and 3 GB.

We can now check the Harddisk storage in the machine using the command fdisk -l

We can see that we have the HDDs as /dev/sdb and /dev/sdc

Now, we need to perform the LVM steps to create the Physical Volume, Volume Group, Logical Volume(partition).

Creating Physical Volume: pvcreate

Creating Volume Group: vgcreate

Now, we would create the Logical Volume (logical partition) of size 4GB: lvcreate

Now, we can format this Logical Volume: mkfs.ext4

Now, we can mount this Logical Volume (partition) to the /dn directory which we created in the starting while configuring the Data Node.

That’s it. We created the Storage of Data Node using LVM, which would make the storage Elastic.

We can run the hadoop dfsadmin -report command now to check the update of shared storage details.

Approx 4GB is shared now, which is correct as we created the LV of 4GB.

Similarly, we can change the size of LV and that brings Elasticity to our setup.

TASK B: Increase or Decrease the Size of Static Partition in Linux

Attach a new HDD of 5 GB to do this practice.

Let’s first create a normal static partition, then we will see how to increase the static partition.

We can see storage /dev/sdb with 5 GB of space
We created a partition of 2 GB

Now we format the partition

Now we create a folder /static-partition and mount this partition to the folder

Let’s create a file in this newly created partition

Now our main task starts from here. Our aim is to increase the partition size without losing the hello.txt file.

Step 1: unmount the partition using umount command

Step 2: Delete the old partition and create a new one with an increased size 4GB

Step 3: Now, it might be possible that while increasing partition size, some garbage might have come, so we can run the command e2fsck -f /partition-name for file system check and appropriate repair

Note that we don’t have to format again because Formatting will lead to the creation of a new Inode table and will delete our hello.txt file

But we need to run resize2fs /partition-name command to resize the Inode table.

Step 4: Now, we can just mount the partition and check our hello.txt file.

This is how we increased our static partition without losing our data.

Thank You..

--

--