Thursday, October 25, 2007

Windows Entropy

If you've used a Windows installation for an extended period, six months or more, then you may have noticed the system gets slower and slower as time passes. This is what I call Windows Entropy.

The main cause is the New Technology File System, or NTFS.

When you first install Windows, or format a new NTFS partition, a few things are created. The two that matter here are:
  • The Master File Table, or MFT, which is usually contiguous, reserved space for storing file metadata, like location, size, acls, and timestamps.
  • The unreserved space, where data is placed.
Initially, the data and metadata are close together on the disk. As the data grows to fill the available space, the distance between the metadata and data grows because Windows will try to keep the MFT contiguous and in one spot [source]. This distance increases seek time, which slows the whole system down because metadata is accessed frequently.

What makes it worse is the partition does not have to fill up to much cause this. If you delete a file and create a new same size or smaller file, Windows may not place the new file in the space of the old file. The data may be placed instead sparsely [source]. This further increases the distance between the data and the MFT. Other file systems do this as well because it avoids fragmentation by giving files room to grow, which is exactly why Microsoft implemented it in NTFS5. These other systems do not try to store their metadata in one contiguous space though. Ext2, for instance, stores its metadata in inodes, which reside in block groups, which are placed throughout the partition [source].

Tuesday, September 11, 2007

Creating and moving your Linux software raid with mdadm

I thought I would document this for reference for myself and anyone else who might benefit. I recently upgraded with main workstation, replacing my 1.2TB LVM made up of five 250GB drives with a 2TB raid array of five 500GB drives. The additional challenges were that I would have to transfer the data off of the old LVM onto the new raid and I would have to do this over the network as neither motherboard has 10 SATA connections. To further complicate things, I want to use one of the current drives of the LVM as my new system drive. Here is the process.

After setting up the hardware for both systems, I booted the new raid system with a Fedora Core 7 DVD and entered rescue mode by entering linux rescue at the ISOLINUX prompt. Normally, I would setup the raid during the normal Fedora installation. However, I want to use one of the LVM members for the new installation, which I also need to pull data off of first.

Now that I'm in rescue mode, the first thing to do was partition each raid member with one big Linux Raid Autodetect or fd type partition. Once this was done, I could create a level 5 raid with this command:

mdadm --create /dev/md0 --level=5 --raid-devices=5 /dev/sd[abcde]1

When I ran this command the first time, mdadm told me that /dev/sda1 was too small to be part of the array. For some reason, the kernel detected the changes in the other drives, but not the first one. I rebooted the system, as partprobe is not available in rescue mode, and then command worked like a charm.

The next step was to put a file system on the new raid device. I used XFS because the files that would live on this file system were very large, about 100MB to several gigabytes. I formatted /dev/md0 with this command, which is about how you could format any drive with XFS:

mkfs.xfs /dev/md0

Next, I mounted the raid, setup an NFS export on the old system, and mirrored about a terabyte of data. This took about 16 hours over gigabit ethernet.

The next day, after copying completion, I installed Fedora Core 7 on one of the old members of the LVM. After installing, I would then configure the raid as my /home. I had to assemble the raid before I could use it. You could use this method to assemble a raid moved from another system as well as what I am doing.

Before I could assemble it, I needed the raid's UUID. This is easy to get. All you need to know is the device file of one of the raid members, which the first on mine is /dev/sdb1, and you can ask mdadm.

mdadm --examine /dev/sdb1

In the cascade of output, you will see UUID presented as a string of hexidecimal characters. Now, we run this command to assemble the raid device:

mdadm --assemble --name=/dev/md0 --uuid=PUT_UUID_HERE /dev/sd[bcdef]1

The raid device is all ready to mount. Now, I want this raid device to be prepared at every boot. I do this by creating /etc/mdadm.conf and insert this:

DEVICE /dev/sd[bcdef]1


MAILADDR root@localhost

That's it. Remember to update your /etc/fstab, if you want your raid to be mounted every boot.

Friday, September 7, 2007 for Sharp Aquos and nVidia 8800 GTS

Configuring X for functionality with an HDTV and any nVidia card is quite a frustrating adventure. This is the procedure I took to configure X for my nVidia 8800 GTS ( g80 ) connected to my Sharp Aquos LC-32GP1U through DVI-D. This Aquos is a 1080p monitor, so I am configuring it to run at 1920x1080.

When I first start X, I get a display. The resolution is only 1280x1024. So the first step is to see what X's problem is. The easiest way is to first enter runlevel 3 and start X from there with added switches for verbosity.

Use Ctrl+Alt+F1 to switch to the first virtual terminal.
Login and run telinit 3 to enter runlevel 3.
Start X with startx -- -verbose 6 --logverbose 6 2> /tmp/startx.log
As soon as X displays the screen, press Ctrl+Alt+Backspace to kill X.

Now, we look at /tmp/startx.log. We are looking for the mode validation checking for possible resolutions, specifically 1920x1080. In this output, X tells us that 1920x1080 is to large for DFP, which it isn't because this is a 1080p monitor. We need to tell X to not worry about DFP. Insert the following line in the Device section:

Option "ModeValidation" "NoDFPNativeResolutionCheck"

Now start X with startx -- -verbose 6 --logverbose 6 2> /tmp/startx.log. We'll see nothing has changed. Kill X and look at /tmp/startx.log. Look at the mode validation. X is now complaining about timings. We need to tell X to use exact timings, which we do by adding this line to our Device section:

Option "ExactModeTimingsDVI" "True"

Now start X with startx -- -verbose 6 --logverbose 6 2> /tmp/startx.log. We'll see that the screen is deformed. Kill X and look at /tmp/startx.log. Nothing will scream out the problem unfortunately. The resolution is actually correct, 1920x1080. However, the image is not correctly scaled. It took me many frustrating google searches to find a solution for this. We need to add this line to our Device section to have the video card use the monitor's native scaling.

Option "FlatPanelProperties" "Scaling=Native"

Start X and voila. Beautiful 1080p resolution, perfectly scaled. Here is the entire /etc/X11/xorg.conf from my machine:

Section "ServerLayout"
Identifier "single head configuration"
Screen 0 "Screen0" 0 0
InputDevice "Keyboard0" "CoreKeyboard"

Section "Files"
ModulePath "/usr/lib64/xorg/modules/extensions/nvidia"
ModulePath "/usr/lib64/xorg/modules"

Section "ServerFlags"
Option "AIGLX" "on"

Section "InputDevice"
Identifier "Keyboard0"
Driver "kbd"
Option "XkbModel" "pc105"
Option "XkbLayout" "us"

Section "Monitor"
Identifier "Monitor0"

Section "Device"
Identifier "Videocard0"
Driver "nvidia"
Option "ExactModeTimingsDVI" "True"
Option "ModeValidation" "NoDFPNativeResolutionCheck"
Option "FlatPanelProperties" "Scaling=Native"
Option "AddARGBGLXVisuals" "True"
Option "DisableGLXRootClipping" "True"

Section "Screen"
Identifier "Screen0"
Device "Videocard0"
Monitor "Monitor0"
DefaultDepth 24
SubSection "Display"
Viewport 0 0
Depth 24
Modes "1920x1080"

Section "Extensions"
Option "Composite" "Enable"