Disk Performance Trending in Linux:

The Disk i/o system is often the slowest subsystem on the computer and one of the biggest bottlenecks in system performance. Disk i/o is critical for certain applications,.

This document is a summary of the key factors that affect overall disk performance on the system and references for some tools that can be used to measure these factors on Linux Servers.

Disk Throughput

hdparm has a lot of features , the -t switch is useful for getting a general baseline on the speed of the disk system; it's a simple matter of passing it the device name of the drive you want to test.

hdparm -t /dev/sda1

This first example is done with a locally attached SATA2 - 5400 RPM drive (2TB) drive (no raid/striping etc. just a single local disk) (EXT3)

Timing buffered disk reads: 154 MB in 3.01 seconds = 51.14 MB/sec

 

This next result shows the difference when testing a SATA2 SDD (96GB) drive on the same system (again, just a single disk, no striping) (EXT3)

Timing buffered disk reads: 410 MB in 3.01 seconds = 136.44 MB/sec

 

Note a significant difference when talking to the SDD drive - The SATA2 bus itself is capable of providing up to a theoretical 3.0Gbps (Sata3 is supposed to be able to deliver up to 6Gbps but I don't have a system with that to test with yet.. The fastest SCSI standard today tops out at 5.12 Gbps

Here is the same drive in the same system running the EXT2 file system. Note the journaling feature in EXT3 causes additional write activity that is detrimental to SSD drives and is best avoided if possible

Timing buffered disk reads: 437 MB in 3.03 seconds = 144.22 MB/sec

 

for comparison purposes i remoted in to my customers network and ran the tool on a few of their systems -

This is the results from measurements on a few production servers at a customer site - these are running HD Proliant ML350 G3 servers with the HP Smart Array Controllers and SCSI drives in a RAID 5 array.

 

Timing buffered disk reads: 154 MB in 3.03 seconds = 50.88 MB/sec

Timing buffered disk reads: 268 MB in 3.00 seconds = 89.32 MB/sec

Timing buffered disk reads: 126 MB in 3.00 seconds = 41.99 MB/sec

Timing buffered disk reads: 142 MB in 3.01 seconds = 47.15 MB/sec

Timing buffered disk reads: 157 MB in 3.01 seconds = 52.19 MB/sec

Timing buffered disk reads: 137 MB in 3.00 seconds = 45.79 MB/sec

 

Note that the overall read performance above are in most cases close to the same speed I was getting with my SATA 7200 RPM disk tested in my home lab. Below we show some slight improvement after doing some tuning

Timing buffered disk reads: 218 MB in 3.00 seconds = 72.59 MB/sec

Timing buffered disk reads: 277 MB in 3.03 seconds = 92.41 MB/sec

Timing buffered disk reads: 146 MB in 3.00 seconds = 48.63 MB/sec

Timing buffered disk reads: 158 MB in 3.00 seconds = 52.80 MB/sec

Timing buffered disk reads: 170 MB in 3.02 seconds = 56.27 MB/sec

 

Results below are from a personal computer running a software raid (Raid 5) array with 4x10,000RPM Western Digital VelociRaptor drives. (note that the first set of results came from the main / partition, the second set was from the swap partition.. It was the same array spread across the same 4 drives but with notably different results.

 

Timing buffered disk reads: 1240 MB in 3.00 seconds = 413.26 MB/sec (/)

Timing buffered disk reads: 2326 MB in 3.00 seconds = 774.94 MB/sec (/swap)

 

With a single 10,000 RPM VelociRaptor we got the following results

Timing buffered disk reads: 356 MB in 3.02 seconds = 117.84 MB/sec

 

Oh, and... i didn't do this test myself however... a 26 disk RAID array running Samsung 256GB SSD drives was shown delivering....

Timing buffered disk reads: 6056 MB in 3.00 seconds = 2018.70 MB/sec

 

Yes, that does say over 2 GB per second.... -

Disk i/o statistics

iostat (installed by default in opensuse 11 but not in SLES10/OES2 or SLES9) reports on the i/o event statistics. - This is part of the sysstat package and i think it should be installed by default as iostat and sar are incredibly useful tools - you can get the package here if you don't have it on your server http://www.novell.com/products/linuxpackages/server10/i386/sysstat.html

sles11-lab1:/usr/share/doc/packages/sysstat # iostat Linux 2.6.32.12-0.7-pae (home) 03/17/11 _i686_ avg-cpu: %user %nice %system %iowait %steal %idle 1.43 0.01 0.36 0.40 0.00 97.81 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 1.17 12.69 26.50 56968121 118897930 sdb 0.00 0.00 0.00 15244 2810 fd0 0.00 0.00 0.00 64 0

opesuse11-lab1:/usr/share/doc/packages/sysstat # iostat -x 1 4 Linux 2.6.32.12-0.7-pae (home) 03/17/11 _i686_ avg-cpu: %user %nice %system %iowait %steal %idle 1.43 0.01 0.36 0.40 0.00 97.81 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-s z await svctm %util sda 0.03 2.50 0.36 0.81 12.69 26.49 33.42 0.0 1 9.34 3.78 0.44 sdb 0.00 0.00 0.00 0.00 0.00 0.00 18.88 0.0 0 6.25 4.36 0.00 fd0 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.0 0 34.00 34.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.00 0.00 0.00 100.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-s z await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 0.00 0.00 0.00 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 0.00 0.00 0.00 fd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.00 0.00 0.00 100.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-s z await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 0.00 0.00 0.00 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 0.00 0.00 0.00 fd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.00 0.00 0.00 100.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-s z await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 0.00 0.00 0.00 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 0.00 0.00 0.00 fd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 0.00 0.00 0.00

 

The output from iostat includes measurements for the following

• tps (number of tranfers (i/o requests) per second)

• Blk_read/s (number of blocks read per second from the device)

• Blk_wrtn/s (number of blocks written per second from the device)

• Blk_read (total # of blocks read from the device since startup)

• Blk_wrtn (total # of blocks written to the device since startup)

• %iowait (how much time the cpu is waiting on the disk)

Note that there are other stats on the screen but they are not specifically relevant to disk performance.

The Second example above uses the command “iostat -x 1 4” the -x gets “extended” stats, the “1 4” indicates that the command will gather statistics 4 times, 1 second apart. The extended reports gathers the stats above plus a few others of note:

• rrqm/s - The number of read requests merged per second that were queued to the device.

• wrqm/s - The number of write requests merged per second that were queued to the device.

• r/s - The number of read requests that were issued to the device per second

• w/s - The number of write requests that were issued to the device per second.

• rsec/s - The number of sectors read from the device per second.

• wsec/s - The number of sectors written to the device per second.

• Avqrq-sz - The average size (in sectors) of the requests that were issued to the device.

• Acqqu-sz - The average queue length of the requests that were issued to the device.

• Await - The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.

• Svctm - average service time (in ms) for I/O requests that were issued to the device.

• %util - Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.

Notes:

• iostat statistics represent all disk io to all storage where the hdparm command represents performance of a specific device.

• iostat reports what the system is actually using where hdparm generates disk traffic to measure how fast it is

• the first line of output from iostat represents stats since the server was started. Subsequent lines show the stats during the sampling interval.

• If a disk is doing a large number of transfers (the tps field) but reading and writing only small amounts of data (the bps field), examine how your applications are doing disk I/O. The application may be performing a large number of I/O operations to handle only a small amount of data. You may want to rewrite the application if this behavior is not necessary.

Virtual Memory Statistics

vmstat displays stats about several things including paging and block i/o - paging refers to the usage of the swap partition and is an indication of your memory statistics however it is a factor in the use of the disk system. If you have heavy usage of the swap space it will negatively affect the rest of your disk performance. So - if you see high disk io and think it might be the problem, you might want to see if the swap io is also high since that more likely means that memory is the chokepoint

 

OES2-home:/proc # vmstat procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 7276 422976 20404 299920 0 0 6 4 3 9 1 0 98 0 0

The main things to check here are:

• si - swap in

• so - swap out

• bi - blocks in

• bo - blocks out

Notes:

• the first line of output from vmstat represents stats since the server was started. Subsequent lines show the stats during the sampling interval.

OES2-home:/var/log # vmstat 1 6 procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 7276 348176 83412 311620 0 0 6 4 3 1 1 0 98 0 0 0 0 7276 348344 83412 311620 0 0 0 0 39 54 0 0 100 0 0 0 0 7276 348368 83412 311620 0 0 0 0 32 53 0 0 100 0 0 0 0 7276 348368 83412 311620 0 0 0 0 40 71 1 2 97 0 0 0 0 7276 348368 83412 311624 0 0 0 0 31 49 0 0 100 0 0 0 0 7276 348368 83412 311624 0 0 0 0 33 53 0 0 100 0 0

Swappiness

Often users will notice performance degradation in their applications when their system exceeds roughly 40-50% of its ram consumption. This is because the default settings for swappiness tells the system to start using swap space (space preallocated on the hard disk) to store program memory instead of the much faster RAM. This can be seen graphically in gkrellm by monitoring the filled portion of the "swap" meter which is located directly underneath the memory meter. In addition to degrading system responsiveness, the use of swap space can greatly affect battery life of laptops as well due to the amount of power it takes to access the hard drives on the system.

The fact that Linux starts using swap space when any physical memory is left at all may seem very counter-intuitive to most users (as it did to myself at first). Linux, being a server-oriented operating system, is by default tuned to deliver high performance to background applications at the expense of foreground applications. This means that your word processor, mp3 player, kde desktop manager, doom3 video game, and any other "foreground" application will start to be swapped out at the earliest sign of rising memory consumption so that the system background services can run smoothly. For the average desktop user, this is almost always not what you want. Short of turning swap space completely off (which is not recommended), Linux allows us the ability to fine-tune the likelihood of swap space being used at all.

To check the swappiness value on your system, run the following command on the terminal.

cat /proc/sys/vm/swappiness

default is 60 on most SUSE systems - If you have swap activity on systems with plenty of RAM it often helps to lower this to ~10

/proc/diskstats

the /proc/diskstats file is a virtual file in a virtual subdirectory - the contents of /proc are read directly from the kernel and do not actually exist on disk. The counters in diskstats are constantly updated and this file shows the following information about each mounted storage device.

• # of reads issued

• # of reads merged, field 6 -- # of writes merged

• # of sectors read

• # of milliseconds spent reading

• # of writes completed

• # of sectors written

• # of milliseconds spent writing

• # of I/Os currently in progress

• # of milliseconds spent doing I/Os

• weighted # of milliseconds spent doing I/Os

sar

sar is installed in sles11 but not in sles10 and earlier - it is part of the sysstat package and can be added fairly easily. It does need to be configured to run a cron task to create the report files in /var/log/sa (sa2 is the process to create the files) - once this has run you can use sar to report from those files. Running sar -d gives an output below. You can download sysstat here if you don't have it on your server already - http://www.novell.com/products/linuxpackages/server10/i386/sysstat.html

 

03:47:56 PM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz   await     svctm     %util
03:47:59 PM    dev8-0     66.56   3480.79      5.30     52.38      0.21   3.08      2.79     18.54
03:47:59 PM    dev8-1      1.66     37.09      0.00     22.40      0.03   18.40      7.20      1.19
03:47:59 PM    dev8-2     64.90   3443.71      5.30     53.14      0.17   2.69      2.67     17.35
03:47:59 PM   dev8-16    460.60 117827.81     42.38    255.91      1.63   2.73      1.82     83.97
03:47:59 PM   dev8-17    460.60 117827.81     42.38    255.91      1.63   2.73      1.82     83.97
03:47:59 PM   dev11-0      0.00      0.00      0.00      0.00      0.00   0.00      0.00      0.00
03:47:59 PM   dev8-32      0.00      0.00      0.00      0.00      0.00   0.00      0.00      0.00
03:47:59 PM   dev8-33      0.00      0.00      0.00      0.00      0.00   0.00      0.00      0.00

• 
  

  DEV - The physical device in question

  
  



• 
  

  tps - number of tranfers (i/o requests) per second

  
  

• 
  

  rd_sec/s - Number of sectors (1 sector = 512 bytes) read per second

  
  



• 
  

  wr_sec/s - Number of sectors written per second

  
  



• 
  

  avgrq-sz - Average number of sectors issued to the device

  
  



• 
  

  avgqu-sz - Average queue length of requests issued to the device

  
  



• 
  

  await - Average number of milliseconds I/O requests for this device had to wait before being handled, including how long it took to handle them

  
  



• 
  

  svctm - Average time number of milliseconds I/O requests for this device had to wait before being handled

  
  



• 
  

  %util - Percentage of CPU time taken up by I/O requests being issued to the device

  
  

Measuring Disk Latency

Seek Time is the biggest performance issue on most traditional disk system. The amount of time it takes to get the disk heads into position and then wait until a particular sector is beneath the heads. Typical seek times vary largely due to the disk rotation speed.

 

5400 RPM	9-14 ms
7200 RPM	6.7-7.0ms
10K RPM	4.5-5.2ms
15K RPM	3.5-4.2ms
SSD	<0.1ms

 

The tool 'fio' has several modules for disk performance tests - the “random-read-test.fio” module is, as the name suggests, do random disk read tests for the purpose of measuring the latency of the disk.

'fio' is not installed by default in SLES/SLED or OES2 but it can be added on using the package in http://download.opensuse.org/repositories/home:/mge1512:/benchmarking/SLE_11/i586/

“Bonnie”

included in SLES11/SLED11 and OpenSUSE11 -

“Iozone” - also not installed by default - you can download it here - http://download.opensuse.org/repositories/home:/mge1512:/benchmarking/SLE_11/i586/

32 bit version -

http://software.opensuse.org/search/download?base=openSUSE%3A11.4&file=benchmark%2FopenSUSE_11.4%2Fi586%2Ffio-1.41-3.1.i586.rpm&query=fio

64 bit version -

http://software.opensuse.org/search/download?base=openSUSE%3A11.4&file=benchmark%2FopenSUSE_11.4%2Fx86_64%2Ffio-1.41-3.1.x86_64.rpm&query=fio

Note that this isn't supported in SLES/SLED or OES2.

The i/o scheduler -

The 2.6 Kernel introduced a concept called “i/o elevators” and has 4 i/o schedulers.

• 
  

  CFQ (completely fair)

  
  



• 
  

  NOOP

  
  



• 
  

  Anticipatory

  
  



• 
  

  Deadline

  
  

The choice of i/o scheduler can have a major impact on disk performance.

An i/o elevator is a queue where i/o requests are ordered based on where their sector is on the disk.

You can determine which scheduler you are using with dmesg

tunable parameters for the scheduler are in /sys/block/(device)/iosched

the CFQ scheduler targets “fair” allocation of i/o bandwidth among the initiators of requests.

Noop is optimized for large i/o systems using RAID or SAN connected storage

anticipatory is tuned for reducing per thread read response time and is best suited for systems with streaming of large files and synchronous disk reads. It is a poor choice for systems with random read/write workloads.

Deadline scheduler uses five i/o queue's to track requests,. It is intended to average read-request response times for random disk requests

Choosing the right file system for the job...

EXT2 - This is a non-journaling file system - Ideal for USB or SSD drive where you need high stability with minimal writes.

EXT3 - The main difference between EXT2 and EXT is the addition on Journaling - Journaling is meant to help with speed up graceful recovery after a system crash by backing out uncommitted changes. EXT3 has been the default choice for most situations for years.

EXT4 adds support for larger filesystems, faster checking, nanosecond timestamps and checksum verification of the journal.

ReiserFS - This is a good choice in situations where there are lots of small files but it doesn't work well with multicore pc's as the architecture is often limited to one operation at a time.

XFS is a highly tunable file system and has features like guaranteed rate i9/o, online resizing, quota enforcement and can support up to 8 exabytes of storage space (theoretically).

Btrfs - Still in development - it has support for “transparent compression, snapshots, cloning and 'in place conversion' from ext3 or ext4 - it is available now during development but it may be premature to deploy in production environments today.

References:

Novell Documentation

SUSE Linux Enterprise Server 11 - System Analysis and Troubleshooting Guide

http://www.novell.com/documentation/sles11/pdfdoc/book_sle_tuning/book_sle_tuning.pdf

IBM Redbook

Linux on IBM System z: Performance Measurement and Tuning

http://www.redbooks.ibm.com/redbooks/pdfs/sg246926.pdf

IBM RedPaper

Tuning SUSE Linux Enterprise Server on IBM eServer xSeries Servers

http://www.redbooks.ibm.com/redpapers/pdfs/redp3862.pdf

Linux Performance and Tuning Guidelines

http://www.redbooks.ibm.com/redpapers/pdfs/redp4285.pdf

Tuning Red Hat Enterprise Linux on IBM eServer xSeries Servers

http://www.redbooks.ibm.com/redpapers/pdfs/redp3861.pdf

Two Paths to Server Performance: Matthias G. Echermann and Bill Tobey: Novell Connection Magazine; July 2010

http://www.novell.com/connectionmagazine/2010/07/tt2.pdf

Creating Certificates with Multiple Hostnames

 

When hosting multiple host names on a single IP Address it is possible to have a single certificate that works with any number of hostnames using “Subject Alternative Names” which is an attribute that lists an alternate name(s) for the subject of the certificate. In a web context that subject is the hostname. However it’s not just hostnames that can be an alternative subject. Email is an option as is IP addresses. For instance Many companies host more than a single website address such as www.example.com and www.example.org

The first step is to create a CSR (certificate signing request) that contains the subject alternative names that you desire for your certificate. I will show how to do that using openssl. You will likely need to modify the default openssl.cnf file. In SUSE this is located in /etc/ssl/openssl.cnf. Note that you may prefer to make modifications to a local copy and tell openssl to use your locally modified copy using the -config option. For simplicity I will omit -config localopenssl.cnf from my examples.

Config File Settings

You need to tell openssl to create a CSR that includes x509 V3 extensions and you also need to tell openssl to include a list of subject alternative names in your CSR. In my openssl.cnf I have the following:

 

in the [req] section

[req] req_extensions = v3_req

In the v3_req section

[ v3_req ] # Extensions to add to a certificate request basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment # Some CAs do not yet support subjectAltName in CSRs. # Instead the additional names are form entries on web # pages where one requests the certificate... subjectAltName = @alt_names [alt_names] DNS.1 = www.example.com DNS.2 = www.example.org

 

Then the CSR is generated using:

$ openssl req -new -out $CSR_FILENAME -key $KEY_FILE

To check to see if you got everything correct use:

$ openssl req -text -noout -in $CSR_FILENAME

You should see something similar to this:

Attributes: Requested Extensions: X509v3 Basic Constraints: CA:FALSE X509v3 Key Usage: Digital Signature, Non Repudiation, Key Encipherment X509v3 Subject Alternative Name: DNS:www.foo.com, DNS:www.foo.org

Creating the Certificate

Now you must have a CA (certificate authority) create a signed certificate based on the information provided in your request. Unfortunately most CA software will not honor the subject alternative names in a CSR by default. In the case of the certificate signing tools from Microsoft in Windows 2003 you can tell it to honor subject alternative names using the following:

• certutil -setreg policy\EditFlags +EDITF_ATTRIBUTESUBJECTALTNAME2


• net stop certsvc


• net start certsvc

For openssl you need to use a policy that allows subject alternative names. I believe the policy named policy_anything in the default openssl.cnf file will work. To use that policy:

• $ openssl ca -policy policy_anything -in $CSR_FILENAME -out $CERT_FILENAME

Finally to test that your certificate was created correctly use the following:

• $ openssl x509 -text -noout -in $CERT_FILENAME

You should see something like this:

 

X509v3 extensions: X509v3 Basic Constraints: critical CA:FALSE X509v3 Key Usage: Digital Signature, Key Encipherment X509v3 Subject Alternative Name: DNS:www.foo.org DNS:www.foo.org

System Monitoring Commands

I’ve been doing a large system monitoring project the past month and have setup a centralized monitoring solution that tracks over 800 servers using Nagios.

as part of that we established multiple trending reports and taught the network operations center support staff how to run various tools in LInux for server monitoring (Most of the NOC staff at this company is MS Centric with limited exposure to linux.)

These commands should be well known to anyone doing linux system administration. If you manage Linux servers and aren’t familiar with any of the commands on this list you should spend some time playing with the various options of these tools, knowing what they can do and knowing how to use them can be very useful in troubleshooting a system issue.

the commands we covered are:

top – which provides a dynamic real-time view of running processes. By default, it displays the most CPU-intensive tasks running on the server and updates the list every five seconds.

vmstat – reports information about processes, memory, paging, IO, traps and CPU activity

w – displays who is online (logged in) and reports what they are doing

uptime – reports how long the system has been running

ps – displays running processes

free – Displays total free and used physical and swap memory

iostat – repots statistics on i/o

sar – collects and reports on system activity

mpstat –multiprocessor statistics

pmap – Pricess memory usage

netstat – Network Statistics

ss – network statistics

iptraf – IP Lan Monitor (real time network statistics)

tcpdump – command line packet dump utility for network analysis

strace – system calls trace – useful for debugging

nmap – much much more than just a port scanner –

cacti – web based monitoring tool

ntop – Network Top – displays the top network users

htop – enhanced version of top -

vnstat – console based network traffic monitor

wireshark – the best protocol analyzer around

nagios – Open Source enterprise System Monitor - look for several articles coming soon on the use of nagios.

dstat – combines the output from vmstat, iostat, ifstat, netstat and other tools –

powertop – monitors power consumption of application based on how much time the cpu stays in low power mode vs. Turbo Modes – requires acpi

whowatch – basically shows who is logged in and what they are doing in real time (similar to ‘w’ but it continuously updates)

dtrace – DTrace can be used to get a global overview of a running system, such as the amount of memory, CPU time, filesystem and network resources used by the active processes. It can also provide much more fine-grained information, such as a log of the arguments with which a specific function is being called, or a list of the processes accessing a specific file.

Review your Log Data

Read your logs using logwatch or logcheck. These tools make your log reading life easier. You get detailed reporting on unusual items in syslog via email.

keep user accessible data on separate disk partitions

Separation of the operating system files from user files may result in a more secure system. ideally the following filesystems should be mounted on separate partitions:

• /usr
• /home
• /var and /var/tmp
• /tmp

I also suggest separate partitions for Apache and FTP server roots. Edit /etc/fstab file and make sure you add the following configuration options:

1. noexec - Do not set execution of any binaries on this partition (prevents execution of binaries but allows scripts).
2. nodev - Do not allow character or special devices on this partition (prevents use of device files such as zero, sda etc).
3. nosuid - Do not set SUID/SGID access on this partition (prevent the setuid bit).

Sample /etc/fstab entry to to limit user access on /dev/sda5 (www server root directory):

/dev/sda5  /srv/www/htdocs          ext3    defaults,nosuid,nodev,noexec 1 2

establish password aging policies

The chage command changes the number of days between password changes and the date of the last password change. This information is used by the system to determine when a user must change his/her password. The /etc/login.defs file defines the site-specific configuration for the shadow password suite including password aging configuration. To disable password aging, enter:

chage -M 99999 userName

To get password expiration information, enter:

chage -l userName

You can also manually specify the information in the /etc/shadow file which has the following fields

{userName}:{password}:{lastpasswdchanged}:{Minimum_days}:{Maximum_days}:{Warn}:{Inactive}:{Expire}:

Note that the “Expire” date is in Unix Time (seconds since Jan 1, 1970)

The chage command is usually easier than manually editing the /etc/shadow file. 

chage –M 60 –m 7 –W 7 <accountname>

Lock accounts after failed login attempts

You can use the faillog command to set login failure limits and to display a list of failed login attempts.

to unlock an account you can use:

faillog –r –u <accountname>

you can also use the passwd file to lock or unlock accounts manually.

passwd –l <accountname>

passwd –u <accountname>

Disable unnecessary services

You should periodically review what services are running and remove any that are no longer needed. One way to check is to use the following command: (Note that this command checks for services running in run level 3)

chkconfig –-list |grep ‘3:on’

If you see any services you need to stop and disable you can use these commands:

service <servicename> stop

chkconfig <servicename> off

the first one stops the service; the second one removes it from the list of services that start when you initialize a runlevel (such as system startup).

Check what ports are listening

You can check your servers listening ports with the following command:

netstat –tulpn

or

nmap –sT –0 <hostname>

if any aren’t needed you should consider shutting down that service or blocking access to the port with iptables.

Use aide to monitor core system configuration files

AIDE (Advanced Intrusion Detection Environment) can be used to help track file integrity by comparing a 'snapshot' of the system's files prior to and after a suspected incident. It is a freeware version of Tripwire, AIDE uses a database to accumulate key file attributes like permissions, mtime, ctime, and number of links for a system. The idea is to build the database before 2 things occur:

1. before the snapshot image of these files is taken prior to the system being placed on a network; and,
2. before the snapshot image is taken prior to a system compromise.

For further protection, a checksum of each file in its current state can be used with a choice of several hash methods.

The idea behind AIDE and other host-based IDSs is for the snapshot to be taken and then periodically updated as the system is updated. Patches, hardware and other software installs tend to change the size and nature of files, so it is always a good idea to re-run AIDE after making changes. If the administrator then suspects that a system has been compromised, running AIDE and then comparing the two snapshots will assist the admin in honing in on what happened. Doing so without this original snapshot can be difficult at best.

aide is included in many distributions of Linux. to begin using it you must review the /etc/aide.conf file and ensure that it has the correct information for your environment- then initialize the database using the command

aide –i

once the database is created you run a comparison using the –-check switch; i set this up as a cron job to run daily (although you can do this more or less often as you feel necessary) the command is

aide –-check >/tmp/aide.log

then i have the aide.log mailed to me for review. If nothing is changed it is a simple comparison. over time the system will have changes made, you can use this log file to review changes as they occur. periodically; as the volumes of change from the initial snapshot grows you can update your snapshot using the following command

aide -u

 

IBM Linux Security RedBook

IBM Redbook - Securing Linux on System Z

NSA Guide to Securing RedHat Linux

this is nearly a year old - I just happened to run across it

http://www.nsa.gov/ia/_files/os/redhat/rhel5-guide-i731.pdf

Novell's Guide to SUSE Linux Security

Novell's Best Practices for SUSE Linux Security

Linux Security Tools

 

System Auditing

• Chkrootkit (YoLinux tutorial) - Scan system for trojans, worms and exploits.
• checkps - detect rootkits by detecting falsified output and similar anomalies. The ps check should work on anything with /proc. Also uses netstat.
• Rootkit hunter - scans for rootkits, back doors and local exploits
• Rkdet - root kit detector daemon. Intended to catch someone installing a rootkit or running a packet sniffer.
• Tripwire : The grand-daddy of file integrity checkers
• RKHunter : An Unix Rootkit Detector
• chkrootkit : Locally checks for signs of a rootkit
• fsaudit - Perl script to scan file systems and search for suspicious looking directories
• - UNIX security checks. Programs and shell scripts which perform security checks. Checks include file and directory permissions, passwords, system scripts, SUID files, ftp configuration check, ...
• SARA - Security Auditor's Research Assistant - network security vulnerability scanner for SQL injections, remote scans, etc. (follow-on to the SATAN analysis tool)
• - Texas A&M University developed tools
• Tiger - Scan a Unix system looking for security problems (Similar to COPS) -
• Tiger Analytical Research Assistant (TARA Pro) - Commercial support
• Netlog - TCP and UDP suspicious traffic logging system
• Drawbridge - Firewall package (Free BSD)
• Dsniff : A suite of powerful network auditing and penetration-testing tools
• P0f : A versatile passive OS fingerprinting tool
• BASE : The Basic Analysis and Security Engine

Network Vulnerability Audits

• Nessus - Remote security scanner - This is my favorite security audit tool!! Checks service exploits and vulnerabilities.
• ISIC - IP Stack Integrity Checker
• Argus - IP network transaction auditing tool. This daemon promiscuously reads network datagrams from a specified interface, and generates network traffic status records
• Argus 2
• SAINT - Finds computers on the network, port scans and does a vulnerability check and outputs a report. - Commercial product.
• InterSect Alliance - Intrusion analysis. Identifies malicious or unauthorized access attempts.
• Linuxforce: AdminForce CGI Auto Audit - CGI script analyzer to find security deficiencies.
• Core Impact : An automated, comprehensive penetration testing product
• Canvas : A Comprehensive Exploitation Framework
• SolarWinds : A plethora of network discovery/monitoring/attack tools
• Yersinia : A multi-protocol low-level attack tool

Wireless Vulnerability Audit Tools

• AirSnort - wireless LAN (WLAN) tool that recovers encryption keys.
• WEPCrack
• Kismet - Wireless Sniffer
• Aircrack : The fastest available WEP/WPA cracking tool

Port Scanners/Network Discovery Tools

• nmap - Port scanner and security scanning and investigation tool
• NmapFe - GUI front-end to NMAP
• KNmap - KDE front-end
• pbnj - Diff nmap scans to find changes to systems on the network.
• nmap3d - nmap post processing to 3-d VRML
• nmap-sql - log scans to database
• portscan - C++ Port Scanner will try to connect on every port you define for a particular host.
• pof - passive OS fingerprinting.
• NetCat - This simple utility reads and writes data across TCP or UDP network connections. It is designed to be a reliable back-end tool that can be used directly or easily driven by other programs and scripts. At the same time, it is a feature-rich network debugging and exploration tool, since it can create almost any kind of connection you would need, including port binding to accept incoming connections
• Scanrand : An unusually fast stateless network service and topology discovery system
• Web/http scan:
• Nikto - web server scanner. CGI, vulnerability checks. Not a stealthy tool. For security tests.
• Paros Proxy - A web application vulnerability assessment proxy
• Web Scarab - A framework for analyzing applications that communicate using the HTTP and HTTPS protocols
• Whisker/libwhisker : Rain.Forest.Puppy's CGI vulnerability scanner and library
• Burpsuite : An integrated platform for attacking web applications
• SPIKE Proxy : HTTP Hacking

Network Sniffers

• DSniff - network tools for auditing and penetration testing.
• Wireshark - full network protocol sniffer/analyzer
• (Ethereal - legacy. Now Wireshark)
• IPTraf - curses based IP LAN monitor
• TcpDump - network monitor and data acquisition
• VOMIT - Voice Over Misconfigured Internet Telephones - Use TCP dump of VOIP stream and convert to WAV file.
• Cisco Call Manager depends on MS/SQL server and are thus vulnerable to SQL Slammer attacks.
• KISMET - 802.11a/b/g wireless network detector, sniffer and intrusion detection system.
• DISCO - Passive IP discovery and fingerprinting tool. Sits on a segment of a network to discover unique IPs and identify them.
• Yersina - Framework for analyzing and testing the deployed networks and systems. Designed to take advantage of some weakness in different Layer 2 protocols: Spanning Tree Protocol (STP), Cisco Discovery Protocol (CDP), Dynamic Trunking Protocol (DTP), Dynamic Host Configuration Protocol (DHCP), Hot Standby Router Protocol (HSRP), IEEE 802.1q, Inter-Switch Link Protocol (ISL), VLAN Trunking Protocol (VTP).
• EtterCap - Ettercap is a terminal-based network sniffer/interceptor/logger for ethernet LANs. It supports active and passive dissection of many protocols (even ciphered ones, like ssh and https). Data injection in an established connection and filtering on the fly is also possible, keeping the connection synchronized. Many sniffing modes were implemented to give you a powerful and complete sniffing suite. Plugins are supported. It has the ability to check whether you are in a switched LAN or not, and to use OS fingerprints (active or passive) to let you know the geometry of the LAN.
• Ntop : A network traffic usage monitor
• Ngrep : Convenient packet matching & display
• EtherApe : EtherApe is a graphical network monitor for Unix modeled after etherman
• Argus : A generic IP network transaction auditing tool
• Ike-scan : VPN detector/scanner
• Arpwatch : Keeps track of ethernet/IP address pairings and can detect certain monkey business

Password crackers

• John the Ripper - weak password detection. crypt, Kerberos AFS, MS/Windows LM, ...
• lCRACK - password hacker, dictionary, brute force incremental, ...
• THC Hydra : A Fast network authentication cracker which supports many different services
• Aircrack : The fastest available WEP/WPA cracking tool
• Airsnort : 802.11 WEP Encryption Cracking Tool
• RainbowCrack : An Innovative Password Hash Cracker

Honeypots

• Honeyd : Your own personal honeynet

Exploits:

• bobkit
• woot-project

• MetaSploit - Exploit launcher, test and development tool

Intrusion Detection Systems

• SNORT - This lightweight network intrusion detection and prevention system excels at traffic analysis and packet logging on IP networks. Through protocol analysis, content searching, and various pre-processors, Snort detects thousands of worms, vulnerability exploit attempts, port scans, and other suspicious behavior. Snort uses a flexible rule-based language to describe traffic that it should collect or pass, and a modular detection engine
• OSSEC HIDS : An Open Source Host-based Intrusion Detection System
• Fragroute/Fragrouter : A network intrusion detection evasion toolkit
• BASE : The Basic Analysis and Security Engine
• Sguil : The Analyst Console for Network Security Monitoring

Encryption Tools

• GnuPG / PGP : Secure your files and communication w/advanced encryption
• OpenSSL : The premier SSL/TLS encryption library
• Tor : An anonymous Internet communication system
• Stunnel : A general-purpose SSL cryptographic wrapper
• OpenVPN : A full-featured SSL VPN solution
• TrueCrypt : Open-Source Disk Encryption Software for Windows and Linux

Log Analysis

• AWStats
• Webalyzer
• Calamaris - parses logfiles from Squid, NetCache, Inktomi Traffic Server, Oops! proxy server, Novell Internet Caching System, Compaq Tasksmart or Netscape/iplanet Web Proxy Server and generates a report
• fwlogwatch - fwlogwatch is a packet filter / firewall / IDS log analyzer written by Boris Wesslowski originally for RUS-CERT. It supports a lot of log formats and has many analysis options. It also features incident report and realtime response capabilities, an interactive web interface and internationalization.
• LogCheck - Logcheck is a simple utility which is designed to allow a system administrator to view the logfiles which are produced upon hosts under their control.
• Logwatch - Logwatch analyzes and reports on system logs. It is a customizable and pluggable log-monitoring system and will go through the logs for a given period of time and make a customizable report. It should work right out of the package on most systems.
• syslog-ng is a flexible and highly scalable system logging application that is ideal for creating centralized and trusted logging solutions.
• LogAnalysis.org has multiple application specific log analyzers
• Swatch can assist with logfile analysis, providing immediate notification if log entries matching a regular expression are spotted, or to review logfiles for unknown data.

Network Monitoring and Management

• Nagios : An open source host, service and network monitoring program
• Argus : A generic IP network transaction auditing tool
• Sguil : The Analyst Console for Network Security Monitoring

AntiVirus

• ClamAV : A GPL anti-virus toolkit for UNIX
• AVG- AVG Internet Security for Linux
• F-PROT AntiVirus for Linux
• Avast!
• McAfee
• Avira AntiVir Personal

Other (no category)

• Bastille : Security hardening script for Linux, Mac OS X, and HP-UX

Novell owns the Unix Copyrights

I'm surprised that I haven't seen much about this in the Linux and Open Source blogs that I follow. I know most of us have realized for a long time that SCO had no legitimate claims to the Copyrights on Linux but the recent ruling in the SCO v Novell case is very important to those of us that love Linux. Thanks largely to Novell for not backing down to SCO and to Groklaw for their fantastic coverage of all of the trials.

http://www.groklaw.net/index.php

http://www.novell.com/news/press/utah-jury-confirms-novell-has-ownership-of-unix-copyrights/

http://www.pcworld.com/article/192955/jury_sides_with_novell_in_longrunning_sco_battle.html
(good job doing your homework there pcworld. the trial was in Utah, not Nevada)

http://www.ciol.com/Developer/Open-Source/News-Reports/Novell-owns-UNIX-copyright/134456/0/

http://news.cnet.com/8301-11424_3-20001527-90.html

http://www.sltrib.com/business/ci_14786202

Fedora 13 Alpha Release

Below is the press release from RedHat

F13 Alpha release announcement
From FedoraProject
Jump to: navigation, search

The Fedora 13 "Goddard" Alpha release is available! What's next for the free operating system that shows off the best new technology of tomorrow? You can see the future now at:

http://fedoraproject.org/get-prerelease
What is the Alpha release?

The Alpha release contains all the features of Fedora 13 in a form that anyone can help test. This testing, guided by the Fedora QA team, helps us target and identify bugs. When these bugs are fixed, we make a Beta release available. A Beta release is code-complete, and bears a very strong resemblance to the third and final release. The final release of Fedora 13 is due in May.

We need your help to make Fedora 13 the best release yet, so please take a moment of your time to download and try out the Alpha and make sure the things that are important to you are working. If you find a bug, please report it -- every bug you uncover is a chance to improve the experience for millions of Fedora users worldwide. Together, we can make Fedora a rock-solid distribution. (Read down to the end of the announcement for more information on how to help.)
Features

Among the top features for end users, we have:

* Automatic print driver installation. We're using RPM and PackageKit for automatic installation of printer drivers, so when you plug in a USB printer, Fedora will automatically offer to install drivers for it if needed.

* Automatic installation of language packs. Yum language packs plugin support makes software installation smarter and easier for everyone worldwide, by automatically downloading language support for large suites of Fedora software when the user's environment requires it.

* Redesigned user management interface. The user account tool has been completely redesigned, and the accountsdialog and accountsservice test packages are available to make it easy to configure personal information, make a personal profile picture or icon, generate a strong passphrase, and set up login options for your Fedora system.

* Color management. Color Management allows you to better set and control your colors for displays, printers, and scanners, through the gnome-color-manager package.

* NetworkManager improvements include CLI. NetworkManager is now a one stop shop for all of your networking needs in Fedora, be it dial-up, broadband, wifi, or even Bluetooth. And now it can all be done in the command line, if you're into that sort of thing.

* Experimental 3D extended to free Nouveau driver for NVidia cards. In this release we are one step closer to having 3D supported on completely free and open source software (FOSS) drivers. In Fedora 12 we got a lot of ATI chips working, and this time we've added a wide range of NVidia cards. You can install the mesa-dri-drivers-experimental package to try out the work in progress.

For developers there are all sorts of additional goodies:

* SystemTap static probes. SystemTap now has expanded capabilities to monitor higher-level language runtimes like Java, Python and Tcl, and also user space applications starting with PostgreSQL. In the future Fedora will add support for even more user space applications, greatly increasing the scope and power of monitoring for application developers.

* Easier Python debugging. We've added new support that allows developers working with mixed libraries (Python and C/C++) in Fedora to get more complete information when debugging with gdb, making Fedora an exceptional platform for powerful, rapid application development.

* Parallel-installable Python 3 stack. The parallel-installable Python 3 stack will will help programmers write and test code for use in both Python 2.6 and Python 3 environments, so you can future-proof your applications now using Fedora.

* NetBeans 6.8 first IDE to support entire Java 6 EE spec. NetBeans IDE 6.8 is the first IDE to offer complete support for the entire Java EE 6 specification.

And don't think we forgot the system administrators:

* boot.fedoraproject.org. (BFO) allows users to download a single, tiny image (could fit on a floppy) and install current and future versions of Fedora without having to download additional images.

* System Security Services Daemon (SSSD). SSSD provides expanded features for logging into managed domains, including caching for offline authentication. This means that, for example, users on laptops can still login when disconnected from the company's managed network. The authentication configuration tool in Fedora has already been updated to support SSSD, and work is underway to make it even more attractive and functional.

* Pioneering NFS features. Fedora offers the latest version 4 of the NFS protocol for better performance, and in conjunction with recent kernel modifications includes IPv6 support for NFS as well.

* Zarafa Groupware. Zarafa now makes available a complete Open Source groupware suite that can be used as a drop-in Exchange replacement for Web-based mail, calendaring, collaboration and tasks. Features include IMAP/POP and iCal/CalDAV capabilities, native mobile phone support, the ability to integrate with existing Linux mail servers, a full set of programming interfaces, and a comfortable look and feel using modern Ajax technologies.

* Btrfs snapshots integration. Btrfs is capable of creating lightweight filesystem snapshots that can be mounted (and booted into) selectively. The created snapshots are copy-on-write snapshots, so there is no file duplication overhead involved for files that do not change between snapshots. It allows developers to feel comfortable experimenting with new software without fear of an unusable install, since automated snapshots allow them to easily revert to the previous day's filesystem.

And that's only the beginning. A more complete list and details of each new cited feature is available here:

http://fedoraproject.org/wiki/Releases/13/FeatureList

We have nightly composes of alternate spins available here:

http://alt.fedoraproject.org/pub/alt/nightly-composes/

Tutorial - Disable unused Daemons

The article below is a tutorial from "Linux Tutorial Blog" on speeding up your boot sequence by disabling unused daemons however the same methodology can improve your security footing by removing potential security vulnerabilities that just don't need to run in the first place.


Full Article Here

Backtrack 4 is out

I've been using this for 3 weeks now and entirely forgot to mention it here.
Backtrack 4 shipped in late January - Get the download here

BackTrack is intended for all audiences from the most savvy security professionals to early newcomers to the information security field. BackTrack promotes a quick and easy way to find and update the largest database of security tool collection to-date.

Our community of users range from skilled penetration testers in the information security field, government entities, information technology, security enthusiasts, and individuals new to the security community. Feedback from all industries and skill levels allows us to truly develop a solution that is tailored towards everyone and far exceeds anything ever developed both commercially and freely available.

Whether you’re hacking wireless, exploiting servers, performing a web application assessment, learning, or social-engineering a client, BackTrack is the one-stop-shop for all of your security needs.

OpenSuSE 11.3 milestone 3 released

OpenSUSE 11.3 milestone 3 release is the first distro compiled entirely with the GNU Compiler Collection (GCC) version 4.5. The update caused a couple of problems with the openSUSE Build Service and packages that wouldn't compile. The openSUSE project management decided to release nevertheless, specifically to test and make improvements to the new GCC 4.5 upgrade. Milestone 3, therefore, is an actual alpha release to be addressed only by experienced Linux users.

Among the new features are Kernel 2.6.33, the Nouveau drivers for NVIDIA graphics cards and the current GNOME developer version 2.29, including the GNOME shell.

The following bugs are in a known state:

* YaST log files are truncated.
* Network installations choke on the wrong SHA1sum for cracklib-dict-full.
* VirtualBox is uninstallable.
* With the LXDE desktop, rcxdm fails to stop the lxdm login manager.

A glance at the Most Annoying Bugs site might be appropriate before installing 11.3. The list may indeed get longer in the next few days. Download here

SCO v. Novell Trial

For those interested in following the developments in the SCO vs. Novell trial you can find detailed observers notes at the links below. Note that this summary comes mostly from the excellent trial coverage from GROKLAW.NET



Day 1 - Monday March 8, 2010 Day 1 is mostly just the seating of the Jury.

Day 2 - Tuesday March 9, 2010 Day 2 included the opening arguments of both sides and the testimony of Bob Frankenberg, Former CEO of Novell.


Day 3 - Wednesday March 10, 2010 Testimony of Duff Thompson and Ed Chatlos

Day 4 - Thursday March 11, 2010 Most of the day was filled with video depositions by Jack Messman; Former CEO of Novell, Burt Levine a lawyer that came from USL, then worked for Novell and later Santa Cruz. Jim Wilt's depostion which was not heard by the jury; Alek Mohan; CEO of SCO from 1995-1998; and finally live deposition by Bill Broderick, another lawyer that worked for USL and then Novell;

Motion Filed by Novell - Friday March 12, 2010 This motion is to allow Novell to introduce into evidence the prior findings of the court that declares that Novell is in fact the owner of the copyrights and that they did not transfer with the sale. That motion is based on SCO's lawyers making the claim (at least 4 times) that Novell continued to slander SCO's title "to this very day".

Day 5 - Friday March 12, 2010 Continuation of testimony of Bill Broderick and Testimony of Ty Mattingly; Mattingly described himself as the "High Level Business Negotiator" for Novell during the sale of Unix/Unixware to Santa Cruz.

Novell Files a "Petition for Writ of Certiorari" - Review of Ruling to Supreme Court over the 10th Circuit that handed over copyrights to SCO that were not specifically transferred as part of the sale of Unix/Unixware. See the filing here

DAY 6 -
Motion for mistrial; Testimony of Kim Madsen, Steve Sabbath and Darl McBride

Judge Denies 2 Novell Motions, 1 for mistrial and the other to allow evidence on prior judicial opinions in the case.

Novell has filed a Notice of Filing of Offer of Proof Regarding Prior Inconsistent Declaration of Steven Sabbath. It is making a record that SCO was allowed to present testimony in direct examination that Novell knew was contradicted by deposition testimony, but then Novell couldn't tell the jury about it, because of rulings by the judge.

Day 7 -
Testimony of Darl McBride and Christine Botosan

Novell anticipates objections to SCO's Experts' testimony regarding the 'TK-7 v Estate of Barbouti' case -

SCO's motion to allow testimony regardi8ng a previous case and a letter from Brent Hatch. -

Day 8 -
Continued testimony of Darl McBride - McBride admits on stand that SCO did not need the copyrights to run their Unix business and that they only needed them for SCOSource. Also admitted into evidence was an exhibit showing that HP did not take a SCOSource license in part because they equated it with "supporting terrorism"

New Proposed Jury Instructions and Novell Tries again to get prior court rulings admitted as evidence -

Day 9 -
Jury hears about Kimball's Rulings and Botosan

Day 10 -
Testimony of Chris Stone, O'Gara, Maciaszek, Nagle -

APA's "Included Assets" did not list SVR4.2 - Research Project -

Novell says "elliott Offer" "Inadequate".. -