RAID - Revisited

I'm still researching about RAID. I have done software RAID before and am looking for some hints on hardware RAID and the advantageous over software-based RAID. Let's get some basic picture of RAID and beyond.

What does RAID stands for ?

In 1987, Patterson, Gibson and Katz at the University of California Berkeley, published a paper entitled "A Case for Redundant Arrays of Inexpensive Disks (RAID)" . This paper described various types of disk arrays, referred to by the acronym RAID. The basic idea of RAID was to combine multiple small, inexpensive disk drives into an array of disk drives which yields performance exceeding that of a Single Large Expensive Drive (SLED). Additionally, this array of drives appears to the computer as a single logical storage unit or drive.

The Mean Time Between Failure (MTBF) of the array will be equal to the MTBF of an individual drive, divided by the number of drives in the array. Because of this, the MTBF of an array of drives would be too low for many application requirements. However, disk arrays can be made fault-tolerant by redundantly storing information in various ways. Five types of array architectures, RAID-1 through RAID-5, were defined by the Berkeley paper, each providing disk fault-tolerance and each offering different trade-offs in features and performance. In addition to these five redundant array architectures, it has become popular to refer to a non-redundant array of disk drives as a RAID-0 array.

The different RAID levels

RAID-0

RAID Level 0 is not redundant, hence does not truly fit the "RAID" acronym. In level 0, data is split across drives, resulting in higher data throughput. Since no redundant information is stored, performance is very good, but the failure of any disk in the array results in data loss. This level is commonly referred to as striping.

Two disks
Three disks
RAID-1

RAID Level 1 provides redundancy by writing all data to two or more drives. The performance of a level 1 array tends to be faster on reads and slower on writes compared to a single drive, but if either drive fails, no data is lost. This is a good entry-level redundant system, since only two drives are required; however, since one drive is used to store a duplicate of the data, the cost per megabyte is high. This level is commonly referred to as mirroring.

RAID-1 (mirroring)

RAID-2

RAID Level 2, which uses Hamming error correction codes, is intended for use with drives which do not have built-in error detection. All SCSI drives support built-in error detection, so this level is of little use when using SCSI drives.

RAID-3

RAID Level 3 stripes data at a byte level across several drives, with parity stored on one drive. It is otherwise similar to level 4. Byte-level striping requires hardware support for efficient use.

RAID-4

RAID Level 4 stripes data at a block level across several drives, with parity stored on one drive. The parity information allows recovery from the failure of any single drive. The performance of a level 4 array is very good for reads (the same as level 0). Writes, however, require that parity data be updated each time. This slows small random writes, in particular, though large writes or sequential writes are fairly fast. Because only one drive in the array stores redundant data, the cost per megabyte of a level 4 array can be fairly low.

RAID-5

RAID Level 5 is similar to level 4, but distributes parity among the drives. This can speed small writes in multiprocessing systems, since the parity disk does not become a bottleneck. Because parity data must be skipped on each drive during reads, however, the performance for reads tends to be considerably lower than a level 4 array. The cost per megabyte is the same as for level 4.

Summary:

RAID-0 is the fastest and most efficient array type but offers no fault-tolerance.
RAID-1 is the array of choice for performance-critical, fault-tolerant environments.In addition, RAID-1 is the only choice for fault-tolerance if no more than two drives are desired.
RAID-2 is seldom used today since ECC is embedded in almost all modern disk drives.
RAID-3 can be used in data intensive or single-user environments which access long sequential records to speed up data transfer. However, RAID-3 does not allow multiple I/O operations to be overlapped and requires synchronized-spindle drives in order to avoid performance degradation with short records.
RAID-4 offers no advantages over RAID-5 and does not support multiple simultaneous write operations.
RAID-5 is the best choice in multi-user environments which are not write performance sensitive. However, at least three, and more typically five drives are required for RAID-5 arrays.

Hardware RAID

The hardware based system manages the RAID subsystem independently from the host and presents to the host only a single disk per RAID array. This way the host doesn't have to be aware of the RAID subsystems(s). The controller based hardware solutionDPT's SCSI controllers are a good example for a controller based RAID solution.The intelligent contoller manages the RAID subsystem independently from the host. The advantage over an external SCSI---SCSI RAID subsystem is that the contoller is able to span the RAID subsystem over multiple SCSI channels and and by this remove the limiting factor external RAID solutions have: The transfer rate over the SCSI bus.

The external hardware solution (SCSI---SCSI RAID)

An external RAID box moves all RAID handling "intelligence" into a contoller that is sitting in the external disk subsystem. The whole subsystem is connected to the host via a normal SCSI controller and apears to the host as a single or multiple disks.This solution has drawbacks compared to the contoller based solution: The single SCSI channel used in this solution creates a bottleneck. Newer technologies like Fiber Channel can ease this problem, especially if they allow to trunk multiple channels into a Storage Area Network.4 SCSI drives can already completely flood a parallel SCSI bus, since the average transfer size is around 4KB and the command transfer overhead - which is even in Ultra SCSI still done asynchonously - takes most of the bus time.

Software RAID (aka poor man's redundancy)

The MD driver in the Linux kernel is an example of a RAID solution that is completely hardware independent.The Linux MD driver supports currently RAID levels 0/1/4/5 + linear mode. Adaptecs AAA-RAID controllers are another example, they have no RAID functionality whatsoever on the controller, they depend on external drivers to provide all external RAID functionality. They are basically only multiple single AHA2940 controllers which have been integrated on one card. Linux detects them as AHA2940 and treats them accordingly.Every OS needs its own special driver for this type of RAID solution, this is error prone and not very compatible.

Hardware vs. Software RAID

Just like any other application, software-based arrays occupy host system memory, consume CPU cycles and are operating system dependent. By contending with other applications that are running concurrently for host CPU cycles and memory, software-based arrays degrade overall server performance. Also, unlike hardware-based arrays, the performance of a software-based array is directly dependent on server CPU performance and load.

Except for the array functionality, hardware-based RAID schemes have very little in common with software-based implementations. Since the host CPU can execute user applications while the array adapter's processor simultaneously executes the array functions, the result is true hardware multi-tasking. Hardware arrays also do not occupy any host system memory, nor are they operating system dependent.

Hardware arrays are also highly fault tolerant. Since the array logic is based in hardware, software is NOT required to boot. Some software arrays, however, will fail to boot if the boot drive in the array fails. For example, an array implemented in software can only be functional when the array software has been read from the disks and is memory-resident. What happens if the server can't load the array software because the disk that contains the fault tolerant software has failed? Software-based implementations commonly require a separate boot drive, which is NOT included in the array.

RSS

If you are lazy to update your website, there's an alternative to it. It is called RSS feeder. With it, you have no need to update your website with latest news from other sites. If the sites provide RSS, you can feed your website with their news with RSS feeder and it's done automatically. But before that, you should know how to setup one. There are many RSS feeders in the market and I personally use MagPieRSS.

Add RSS feeds to your Web site using MagpieRSS

Syndication of material from other sites is a good way to get fresh content on your site. As visitors arrive at your site they can see a teaser for the syndicated content and a link to the publisher. If they are interested in an item they can follow the link to the original location. You can add syndicated content to your site using the Really Simple Syndication (RSS) protocol and a bit of PHP code in the form of an application called MagpieRSS. Here's how.

MagpieRSS is an RSS parser written in PHP. It supports RSS 0.9 and 1.0, with some RSS 2.0 support. MagpieRSS is a simple object-oriented backend which includes automatic caching of parsed RSS to reduce load on external Web sites. To use MagpieRSS on your Web site you will need PHP 4.2.0 (or greater) with XML (expat) support or PHP5 with libxml2 support. Download MagpieRSS. Extract the four main files (rss_fetch.inc, rss_parser.inc, rss_cache.inc, and rss_utils.inc) and the directory extlib and copy them to a directory named magpierss in the document root (where your web pages are stored) on your Web server.

Next, decide where the syndicated content will go on your site. If you want the content to appear on your front page then most likely you need to edit index.php. If you only have an index.html (not .php) then rename it to index.php as now you are adding some PHP code to it. Edit the file and add the following line at the top:

require_once('magpierss/rss_fetch.inc')

To actually fetch and parse the RSS add a call to fetch_rss() in your web page replacing the newsforge.com URL with the URL of the content you are syndicating.This function will return an array ($rss) that contains the syndicated content as an array of syndicated items along with some publisher information, such as the the name of the publisher (which is stored in $rss->channel['title']) and an optional description of the publisher (e.g. "The Online Newspaper for Linux and Open Source") which can be found in $rss->channel['description']. There is also $rss->channel['link'] which contains a general link to the publisher.The syndicated items can be accessed via $rss->items. A simple loop can be used to transverse the items one at a time:

foreach ($rss->items as $item)
{
// Your code here
}

Each item contains a title, link and description. $item['title'] contains the title of the article or story, $item['link'] is the link to the original and $item['description'] is the description of the story which is often the introduction paragraph or some sort of summary. To display a simple list of syndicated items use the following code:

?>?>?>?>channel['link'].">".$rss->channel['title']."";
foreach ($rss->items as $item) {
$href= $item['link'];
$title = $item['title'];
$desc = $item['description'];
echo "$title";
if($desc)
echo $desc;
}
?>

Of course, the output is simple HTML, but from here you should be able to expand the code to suit the style and design of your site.

Lengths

In the 0.91 version of the RSS specification the title is restricted to 100 characters and the description to 500. However there are no length limits in RSS 0.92 and greater. This can cause a problem for your Web site. Keeping a consistent look and feel to your Web site is important. If syndicated items are displayed too freely your site may start to look out-of-joint. Imagine you have similar code to the above listing news items in a side bar. What will happen if the description of a particular item is several thousand characters long? Or what if there was 50 items being syndicated?

Length validation is an important aspect to putting syndicated items on your site. To limit the number of items displayed, the easiest thing to do is slice the resulting array into a smaller chunk. After calling fetch_rss(), but before any processing, add the following line: $items = array_slice($rss->items, 0, 10); Which will shrink the array to contain only the first 10 items. Now the foreach loop can be run without any worries.

To validate the length of the description use the strlen() function:

if (strlen($desc) >= 80)
{
$desc = substr($desc,0,79)."...";
}

Here the description is shortened to be less than 80 characters with ... at the end to show that there is more text available. If you have shortened the description you might what to offer a link to the full syndicated text rather than a link to the original. That way users remain on your site longer. The full version of the syndicated item would then contain the link to the original. To do this you will need a script called something like readmore.php which uses MagpieRSS to display the full text as the main item on the web page along with your normal navigation, side bars and advertising. The parameters to the readmore.php would be the URL of the RSS feed and the item number you wish to display. One thing to watch is that some descriptions contain HTML as well as plain text. Blindly chopping the string could cause the string to be cut half way through a tag. Smart chopping is needed but that is beyond the scope of this article. However a crude solution is to use the PHP function strip_tags() which will remove all the HTML tags from the string. The full text version of the item displayed by the readmore.php can leave the tags intact.

Caching

To speed up your Web site and save unnecessary traffic for the publishers web server it is important to set up caching. MagpieRSS has built-in automatic caching. With caching MagpieRSS will only fetch and parse RSS feeds when the cache is too old.

To enable caching add the following lines to your script:

define('MAGPIE_CACHE_DIR',
'/tmp/mysite_magpie_cache');
define('MAGPIE_CACHE_ON', 1);

By default MagpieRSS will cache items for one hour. This can be overridden using:

define('MAGPIE_CACHE_AGE', 1800);

Where 1800 is the number of seconds to cache objects (e.g. 30 minutes, 60*30).

Conclusion

MagpieRSS makes it easy to add syndicated content to your Web site, with only a small time investment you can improve the content of your site. Remember "content is king."

Thanks to Gary Sims for this article.
Source : http://programming.linux.com/article.pl?sid=05/05/24/1337210

Do your softwares suit your business or are you trying to fit your business to your softwares?

When I was reading Linux Format magazine yesterday, I came across this statement of an advertisement. This mag is hard to find in Kuantan (only at a bookshop in MegaMall which I think I'm the only one who buy it :) ). Other mags I have on my bookshelf is Linux Users And Developers and Linux Magazine which my friend sent me from London. Thanks Mark.

It's good to read Linux mags every night before going to bed. You will fall asleep when you start reading it. :-)

What it takes to be a system administrator?

I found this interesting article "What it takes to be a system admin?". I'm interested to know how exactly the work system admin is due to the fact that I have no formal training in this field, I think I should make an effort to be one. Like ones say, "experience is the best teacher". No one shouldn't agree with this statement. This is why we need practice (a lot). During the practice, we will encounter mistakes. With that, you'll know what's the best, what's good and what's bad and what's the worst thing to do.

Let's take a look at the key points of being a system administrator:

1. Change your mindset -- The true SA is a combination caretaker, security guard, and baby sitter
2. Learn new toolsets -- If one wants to move into the corporate arena, then you must be able to take whatever tools are laying around and get the job done. This means the ability to learn new tools and to use old tools in new ways.
3. Learn to handle pressure -- Expect to have to work with others looking over your shoulder. It adds a new level of pressure to have a Senior VP of a billion dollar company watching you type! (Or just the guy who signs your paycheck.)
4. Never start from scratch. Find something close and modify -- For scripting, start with the boot up scripts (/sbin/init.d, /etc/rc.d, etc)
5. Hang out with experts -- Don't be afraid of appearing ignorant. Fear staying ignorant
6. Practice good debugging habits -- Understand it the way it is (broken) before you try to fix it
7. Learn manually, then codify -- Remember the commands by writing a script for them and commenting the script.
8. Document what you do -- Comment your scripts liberally. The best comments (IMHO) are the ones that explain 'Why?'.
9. Learn to share -- Share what you've learned with others (that's why I'm doing this page)
10. Remember to have fun -- Make Unix your passion, not just your job. Don't consider becoming a SA if it isn't your passion.

Source : Jim Wildman's Moving To The Big Time

What is ethernet bonding?

In the new building, there will be 3 lines of broadband connection to the internet. I am thinking to make 'bonding' or port-trunking (cisco term) in order to provide load balancing and fault-tolerance connection. I have done this for dialup lines a few years ago. But this one is for ethernet. Every modern kernel supports this feature. Now let's take a look at what bonding is all about...

What is bonding?
Bonding is the same as port trunking. In the following I will use the word bonding because practically we will bond interfaces as one.

#!/bin/bash

modprobe bonding mode=0 miimon=100 # load bonding module

ifconfig eth0 down # putting down the eth0 interface
ifconfig eth1 down # putting down the eth1 interface

ifconfig bond0 hw ether 00:11:22:33:44:55 # changing the MAC address of the bond0 interface
ifconfig bond0 192.168.55.55 up # to set ethX interfaces as slave the bond0 must have an ip.

ifenslave bond0 eth0 # putting the eth0 interface in the slave mod for bond0
ifenslave bond0 eth1 # putting the eth1 interface in the slave mod for bond0

You can set up your bond interface according to your needs. Changing one parameters (mode=X) you can have the following bonding types:

mode=0 (balance-rr)

Round-robin policy: Transmit packets in sequential order from the first available slave through the last. This mode provides load balancing and fault tolerance.

mode=1 (active-backup)

Active-backup policy: Only one slave in the bond is active. A different slave becomes active if, and only if, the active slave fails. The bond's MAC address is externally visible on only one port (network adapter) to avoid confusing the switch. This mode provides fault tolerance. The primary option affects the behavior of this mode.

mode=2 (balance-xor)

XOR policy: Transmit based on [(source MAC address XOR'd with destination MAC address) modulo slave count]. This selects the same slave for each destination MAC address. This mode provides load balancing and fault tolerance.

mode=3 (broadcast)

Broadcast policy: transmits everything on all slave interfaces. This mode provides fault tolerance.

mode=4 (802.3ad)

IEEE 802.3ad Dynamic link aggregation. Creates aggregation groups that share the same speed and duplex settings. Utilizes all slaves in the active aggregator according to the 802.3ad specification.

Pre-requisites:

1. Ethtool support in the base drivers for retrieving the speed and duplex of each slave.
2. A switch that supports IEEE 802.3ad Dynamic link aggregation.Most switches will require some type of configuration to enable 802.3ad mode.mode=5 (balance-tlb)

Adaptive transmit load balancing: channel bonding that does not require any special switch support. The outgoing traffic is distributed according to the current load (computed relative to the speed) on each slave. Incoming traffic is received by the current slave. If the receiving slave fails, another slave takes over the MAC address of the failed receiving slave.

Prerequisite:

Ethtool support in the base drivers for retrieving the speed of each slave.mode=6 (balance-alb)

Adaptive load balancing: includes balance-tlb plus receive load balancing (rlb) for IPV4 traffic, and does not require any special switch support. The receive load balancing is achieved by ARP negotiation. The bonding driver intercepts the ARP Replies sent by the local system on their way out and overwrites the source hardware address with the unique hardware address of one of the slaves in the bond such that different peers use different hardware addresses for the server.

The most used are the first four mode types...

Also you can use multiple bond interface but for that you must load the bonding module as many as you need. Presuming that you want two bond interface you must configure the /etc/modules.conf as follow:

alias bond0 bonding
options bond0 -o bond0 mode=0 miimon=100
alias bond1 bonding
options bond1 -o bond1 mode=1 miimon=100Notes:

To restore your slaves MAC addresses, you need to detach them from the bond (`ifenslave -d bond0 eth0'). The bonding driver will then restore the MAC addresses that the slaves had before they were enslaved. The bond MAC address will be the taken from its first slave device.

Promiscous mode: According to your bond type, when you put the bond interface in the promiscous mode it will propogates the setting to the slave devices as follow:

for mode=0,2,3 and 4 the promiscuous mode setting is propogated to all slaves.
for mode=1,5 and 6 the promiscuous mode setting is propogated only to the active slave.

For balance-tlb mode the active slave is the slave currently receiving inbound traffic, for balance-alb mode the active slave is the slave used as a "primary." and for the active-backup, balance-tlb and balance-alb modes, when the active slave changes (e.g., due to a link failure), the promiscuous setting will be propogated to the new active slave.

Source : http://www.linuxhorizon.ro/bonding.html

The modem is broken

Today found out that the modem at my workplace broke. I have to contact TMNet to replace it coz it's still under warranty. Thank god, the replacement is a snap. Tomorrow, staff can connect to the internet.

I'm a system admin.

Me, MySelf And I

After being a system admin for 4 months, i think that this is my dream job but the pay is still not enough for me. I've learnt so many things about Linux and how to maintain them (I have 3 servers that i have to monitor). May things on my mind ranging from backup (the hardest if u want to transfer accounts from one server to another server.), tcp/ip, network design, firewall (this is fun!), routing config and etc.