Elasticity Task: Integrating Linux LVM with Hadoop | Increasing OR Decreasing static Partition in Linux | Automating LVM with Python Script
Performing the tasks related to Elasticity on a Linux machine.
Task Description:
Elasticity Task:
- [A] — Integrating LVM with Hadoop and providing Elasticity to DataNode Storage
- [B] — Increase or Decrease the Size of Static Partition in Linux.
- [C] — Automating LVM Partition using Python-Script.
TASK A: Integrating LVM with Hadoop and providing Elasticity to DataNode Storage
Step 1: Install & Configure Hadoop
I have two VMs: ‘RHEL8 OS 1’ as Name Node and ‘RHEL8 OS 2’ for Data Node
Install java jdk and hadoop softwares in both the Nodes
Configure the Name Node
We have to format the master node
Now we can start the Hadoop service for Name node
Similarly, we configure the Data Node
Start the Data Node service
Now we can check if Data Node is connected or not using the hadoop dfsadmin -report command
We can see that the Data Node with IP address 192.168.43.79 is connected and is sharing the storage to the Name Node
Step 2: Integrate LVM with Hadoop to provide Elasticity to Data Node storage
For our initial setup of Hadoop, we have Data Node with fixed storage. We want to have elasticity in the storage, meaning that we could have varied storage sizes. It is helpful to solve the use case where the Client would need to store a File of size, suppose, 30 GB.. But Data Node only providing 20 GB of Storage. Hence, Data Node can then increase the storage by attaching a new additional Harddisk of size 10 GB and providing sufficient storage to store files. This phenomenon is what we called Elasticity in storage.
This will be achieved if we implement LVM for the storage shared to Name Node by the Data Node.
The practical is as follows:
Add 2 new HDD to the Data Node virtual machine of size 2 GB and 3 GB.
We can now check the Harddisk storage in the machine using the command fdisk -l
We can see that we have the HDDs as /dev/sdb and /dev/sdc
Now, we need to perform the LVM steps to create the Physical Volume, Volume Group, Logical Volume(partition).
Creating Physical Volume: pvcreate
Creating Volume Group: vgcreate
Now, we would create the Logical Volume (logical partition) of size 4GB: lvcreate
Now, we can format this Logical Volume: mkfs.ext4
Now, we can mount this Logical Volume (partition) to the /dn directory which we created in the starting while configuring the Data Node.
That’s it. We created the Storage of Data Node using LVM, which would make the storage Elastic.
We can run the hadoop dfsadmin -report command now to check the update of shared storage details.
Approx 4GB is shared now, which is correct as we created the LV of 4GB.
Similarly, we can change the size of LV and that brings Elasticity to our setup.
TASK B: Increase or Decrease the Size of Static Partition in Linux
Attach a new HDD of 5 GB to do this practice.
Let’s first create a normal static partition, then we will see how to increase the static partition.
Now we format the partition
Now we create a folder /static-partition and mount this partition to the folder
Let’s create a file in this newly created partition
Now our main task starts from here. Our aim is to increase the partition size without losing the hello.txt file.
Step 1: unmount the partition using umount command
Step 2: Delete the old partition and create a new one with an increased size 4GB
Step 3: Now, it might be possible that while increasing partition size, some garbage might have come, so we can run the command e2fsck -f /partition-name for file system check and appropriate repair
Note that we don’t have to format again because Formatting will lead to the creation of a new Inode table and will delete our hello.txt file
But we need to run resize2fs /partition-name command to resize the Inode table.
Step 4: Now, we can just mount the partition and check our hello.txt file.
This is how we increased our static partition without losing our data.
Thank You..