Wednesday, July 27, 2016

Lesson Learned On Logical Volume Management and RAID

The lesson is this; that just because you can doesn't mean you should.

I wiped off my RAID array and the data in it. It has around 300+G there, mostly downloaded, recoverable, and back-up'd to a CD/DVD content. I was trying to employ Logical Volume Management or LVM on an existing RAID array. After tackling RAID these couple of months, I figured I'm ready to face another monster. LVM is awesome because it can do partitioning, resizing, formatting, creating snapshots, while the system is online. You do not have to take it offline which to me is like magic.

Now that the data is gone, there really is no reason not to LVM the RAID. So after reinstalling my operating system (which is another long story), I reassemble RAID with:

#mdadm --assemble /dev/md0  /dev/sda  /dev/sdb  /dev/sdc

and checked the status of the array with:

#cat /proc/mdstat
#mdadm --detail /dev/md0

I checked the status of all the logical volumes with any of the following commands:

#pvdisplay     ;to check the physical volumes
#vgdisplay     ;to check the volume group
#lvdisplay      ;to check the logical volumes

I know that my home directory is going to need a lot of room for music and videos. I will have to create logical volumes for it to utilize the RAID which is now empty. I would leave some free space there for some flexibility.

To create logical volumes, I plan to extend the existing volume group first to include the newly assembled RAID. So I typed:

#vgextend ubuntu-vg /dev/md0

Then to create the logical volumes I need I typed:

#lvcreate -L size -n Musiclv ubuntu-vg
#mkfs.ext4 /dev/mapper/ubuntu-vg/Musiclv
#lvcreate -L size -n Videoslv ubuntu-vg
#mkfs.ext4 /dev/mapper/ubuntu-vg/Videoslv

I tested mounting them to home directory. I tried:

#mount /dev/mapper/ubuntu-vg/Musiclv /home/user/Music   ;since the Music folder already exist, I don't have to create them.
#mount /dev/mapper/ubuntu-vg/Videoslv /home/user/Videos     ;since the Videos folder already exist, I don't have to create them.

To check if they work, open the file manager, nautilus, and check out properties in particular. If you have trouble with permissions and ownerships of the files, then you should try:

#chmod 777 /home/user/Music
#chown user:user /home/user/Music
#chmod 777 /home/user/Videos
#chown user:user /home/user/Videos

These last series of commands will let you read and write your files and execute operations on them.

To make these changes more permanent so that even if you end your session and reboot they will stay, then we have to edit /etc/fstab file.

#gedit /etc/fstab

Put entries for each lv you created, something like:

    filesystem       mount point        type         options        dump        pass

/dev/mapper/ubuntu-vg/Musiclv    /home/user/Music     ext4     defaults     0       0
/dev/mapper/ubuntu-vg/Videoslv   /home/user/Videos     ext4     defaults     0       0

As the names of the logical volumes imply, I plan to put music and videos on these folders.

Monday, July 25, 2016


July 2016 - San Mateo, Philippines
Donato Roque @ubuntu-16.04
mobile: 63-918-572-1710
and this is my blog.
Consider using my pgp keys for privacy.

Sunday, July 24, 2016

RAID: Added 1 x 2TB Hard drive

I talked about RAID in my system before here and here. My present set up is:
/dev/sda5      SSD      223GB   EXT4    /  
/dev/sdb1     HHD      2TB       Linux raid      /media/user
/dev/sdc1      HHD      1TB       Linux raid     /media/user

/dev/sdd1      HHD      2TB      unpartitioned - free space

/dev/md0        2 devices    /dev/sdb1   /dev/sdc1

/dev/sdc1 is old. It has 27,500 hours of use according to smartctl. You can open the terminal and type:

#smartctl -a /dev/sdc

to get collected data from the device. Among other attributes see Power-on-hours to get the total hours this hard drive has been spinning. Also pay attention to 5 Attribute the allocated Sector Count. Any value you see other than 100 there, you should move your data somewhere else. 

I have a RAID5 which includes this device. So any warning notification I can fail and remove it. Replace it with a new hard drive. I am completing my RAID 5 array with a new hard drive today.

After installing the hard drive to your SATA cable and power cable. See if your computer detects it. Type:

#fdisk -l

Then partition the disk with fdisk.

#fdisk /dev/sdd

fdisk will create a new partition and create a partition table for you. 

-d     delete partition
-n     create new partition
-p     print new partition and details
-t      change to new type of filesystem
-l       list filesystem type 
-q      quit without saving
-w     write changes, and think first before typing enter

Once the new partition in the new hard drive is ready. I added the device to the RAID array.

/dev/md0          2 devices         /dev/sdb1     /dev/sdc1

To add the new device to the RAID array:

#mdadm  -add /dev/md0 /dev/sdd1

Check the status of the array:

#cat /proc/mdstat

It will show that /dev/sdd1 is a spare. I have to grow the array to 3 devices.

#mdadm --grow --raid-devices=3  /dev/md0 

You can monitor the sync process of the array with:
#cat /proc/mdstat

Donato Roque @ubuntu-16.04
mobile: 63-918-572-1710
and this is my blog.
Consider using my pgp keys for privacy.

Monday, July 18, 2016

GNOME-terminal Rocks

Using Ubuntu 16.04 I was wondering if "avconv" is already installed. I am in fact curious if a bunch of packages are installed. I open GNOME-Terminal with Cntrl+Alt+T and type the name of the package. I get info and suggestions on how to install it. In the case of "avconv" it is called by another name. 

That's service and quite helpful.

Donato Roque @ubuntu-16.04
mobile: 63-9185721710
and this is my blog.
Consider using my pgp keys for privacy.

Sunday, July 17, 2016

Unpatched Software Is A Security Issue

Ubuntu Linux forum was hacked last July 14th. Canonical the company developing Ubuntu disclosed that the forum's database was hacked and data was compromised including hashed passwords of 2 Million users. The hackers used a known vulnerability in an unpatched software the forum is using in its servers. 

In the Linux world, the number one security isssue is updating the operating system and the software that we use with it. The first ones to violate this are the servers. Servers simply run old vulnerable software. 

Donato Roque @ubuntu-16.04
mobile: 63-9185721710
and this is my blog.
Consider using my pgp keys for privacy.

Saturday, July 16, 2016

GUI in Thunderbird Problem

After downloading Skype for linux yesterday and installing it, my desktop applications are fine. Today I opened Thunderbird and the GUI is back to a 90's theme. I downloaded evolution mail and configured my email accounts in minutes. 

In Linux, there's more than one way to skin a prey. 

Friday, July 15, 2016

Resized The RAID And the Data Is Safe

The <a href="">last post,</a> I was talking about resizing a component of my existing RAID. I was worried about losing data.

Here's how I did it without data loss.

/dev/md0                   /dev/sdb1        2TB
                                  /dev/sdc1        1TB

I am increasing the size of the partition and it's an important distinction because the order of the steps would be different otherwise. Check the status of the RAID array with:

$cat /proc/mdstat

Put a fail flag on the partition and remove it.
$sudo mdadm /dev/md0 -fail /dev/sdb1 --remove /dev/sdb1

Then we have to use fdisk to handle the physical resizing of the device. fdisk is an interactive application in text mode. So-
$sudo fdisk /dev/sdb1
-h     for help
-p     for primary partition
-d     delete
-n     new partition
if unsure choose the default option.
-q     to quit without saving
-w    to write the changes, now think first before hitting enter here

I used fdisk to create a new partition utilizing the entire 2TB device. I made an additional step here to check the filesystem after I created a new partition. Use fsck to verify the integrity of the ext3/ext4 filesystem. You should make sure to unmount the partition before checking with e2fsck. It will offer to fix errors you have to press consent or y for the process to continue, otherwise, use the -p switch to allow e2fsck to automatically repair the filesystem without human or administrator intervention.

$sudo e2fsck /dev/sdb1

Re-add the partition to the RAID array.
$sudo mdadm -a /dev/md0 /dev/sdb1

I had to wait 2 hours for the RAID to synchronize and then proceed to the next partition or the next step. Check the status of the synchronization with:

$cat /proc/mdstat

Do this steps one partition at a time.

Increase the size of the array with:

$sudo mdadm --grow /dev/md0 -z max  
$sudo mdadm --grow /dev/md0 -z [size]

Increase the size of the ext3/ext4 filesystem with:

$sudo resize2fs /dev/md0 [size]

If the size is not specified, the default is the size of the partition.

Donato Roque@ubuntu-16-04
I'm on twitter, facebook and this is my blog.
Consider using my PGP keys for privacy.

Wednesday, July 13, 2016

Resized RAID Array

RAID is Redundant Array of Independent Disks. It is a technology in virtualization which allows several devices to be grouped together and used as if there is only one device. If I have two or more hard drives I can group them together in a RAID and my computer will treat it as one device. RAID's purpose is for data redundancy and performance. I use software RAID in Linux. The other kind is hardware RAID in which you depend on chipsets to manage the RAID.

I have a RAID5 array consisting of 1x1TB and 1x2TB. I bought a new 2TB hard drive and I want to add it to the RAID.   To make matters complicated I only made a 1TB partition out of the 2TB hard drive when I created the original RAID. I want to use the entire 2TB capacity of the hard drive and then add the newly bought 2TB hard drive to the RAID.

I have been using the RAID array since April 2016, so there's data written on this array now.

old RAID array:
/dev/sdb1    -    935G (2TB)   -    raid5    -    /dev/md0
/dev/sdc1    -    931G (1TB)   -    raid5    -    /dev/md0

Can this be accomplished without data loss?

Donato Roque@ubuntu-16-04
I'm on twitter, facebook and this is my blog.
Consider using my PGP keys for privacy.

Saturday, July 9, 2016

I'm Back In Ubuntu

Hi Reese

I reinstalled ubuntu 16.04 yesterday. I was able to use the live usb I have of Xenial. So that bit of paranoia saved my skin. I used the whole SSD and told the installer to just wipe the old one off. I have 16 GB of swap. I have /boot. I have full disk encryption including the swap partition.

I also ticked the option to encrypt my /home. It gave me a key. So as I understand this, if I can't log into my desktop ubuntu anymore I will need the key to decrypt the /home and get access to my data.

The installer downloaded all the necessary codecs for the music and video, including flash. Nice of the installer to include that option. It was quick less than 40 minutes and I have ubuntu 16.04 in my hard drive. Bringing all the packages up to date is another story. I guess if you have a very fast connection it won't take you as long as I have.

I have to get my back ups from two sources. First I have a USB backup. Second I have Megasync, which is a cloud back up solution for me. For the 2nd solution, all i have to do is download the sync client to my desktop and log in.

Ubuntu needs to recognize the existing raid I have. I downloaded mdadm. I have to assemble the raid and it went well considering I haven't done this before.

$sudo mdadm --assemble --scan

$cat /proc/mdstat

I also edited /etc/fstab for the raid device. I mounted the raid to /media/donato. Always use the UUID of the hard drive or the raid.

$sudo blkid

So I had to use apt to install some applications which do not come with the base. These are "mdadm", "gsmart, smartmontools", "gnucash" and "calibre". I'm still downloading some other application as I go along. I need rkhunter and several add-ons for Thunderbird. I need Keepass2. I saved the gnupg folder from my Megasync cloud back up. I copied that to my /home and checked if Seahorse got it.

I have Firefox synced my bookmarks and history. I got my contacts imported from my smartphone. Tip: Use your smartphone to export contacts in vcf format. Then in your desktop use Thunderbird to import the vcf file. It works quite well.
Donato Roque@ubuntu-16-04
I'm on twitter, facebook and this is my blog.
Consider using my PGP keys for privacy.