Tuesday, May 15, 2012

Find and Remove old files

This blog has been moved to http://geekaider.com

Find command usage:

find <where> <condition>
find /backup/daily -name log.*
find <where> <condition> -exec <action>
find /backup/daily -name log.* -exec ls -l {} \;
Find log files older than 2 days in /backup/daily directory
find /backup/daily/log.* -mtime +2
Find log files older than 2 days(modified 2 days back) in /backup/daily directory and remove them
find /backup/daily/log.* -mtime +2 -exec rm {} \;
NOTE: -exec <space> rm <space> {} <space> \;

Find log files older than 12 hours in /backup/daily directory and remove them
find /backup/daily/log.* -mmin +12 -exec rm {} \;
Find log files newer than 2 hours in /backup/daily directory and remove them
find /backup/daily/log.* -mmin -120 -exec rm {} \;

Grep and remove files

Grep and remove files based on serial number in name

This blog has been moved to http://geekaider.com

List the existing files

List the files in range from log.0000000670 to log.0000000675

Remove the files in range from log.0000000670 to log.0000000675

List the remaining files and cross-check if the range of files is already deleted

Tuesday, April 10, 2012

Compile Apache on Linux

This blog has been moved to http://geekaider.com

Apache is a featureful and open-source web server for Windows and Linux.
There are two ways of installing Apache on Linux:
  1. Install by RPM
  2. Install by compiling the source
To install Apache RPM one may use yum install <package name> or rpm -ivh <rpm file name> commands.
But to install Apache by compiling it from source code requires more attention. Compiling of any software on Linux usually has 3 steps:
  1. Download, unzip and Configure
  2. Make
  3. Make all
Download httpd-2.2.11.tar or the latest available version

Unzip httpd-2.2.11.tar
tar xvfz httpd-2.2.11.tar.gz
cd httpd-2.2.11
ls
Compile Apache
./configure --prefix=/usr/local/apache2211 --enable-mods-shared=most --enable-ssl
Prefix option specifies the directory in which Apache will be installed. Default value for this option is /usr/local/apache2. This directory will be created if not already exists.

Enable-mods-shared option specifies that Apache will be installed with MOST commonly used modules.
Other option could be enable-mods-shared=all which will compile all available Apache modules.

To be noted that these modules will be loaded as DSO. Dynamic Shared Object a.k.a. DSO are such modules which could be loaded into memory as and when needed. This reduces load on server by loading module into memory only if it is required.

Threading is another option used widely to reduce load on server. Threaded Apache server can respond to large number of request as compared to process based(pre-forked) Apache server. To enable threading in Apache MPM module is used. Multi Processing Module a.k.a. MPM can be enabled by using --with-mpm=worker along with ./configure command described above.

To learn more configuration options go to Apache Documentation page or run command ./configure --help

If the above command fails, always run "make clean" before moving on to next step.
If the above command shows error related to "libapr", the reason could be missing apr and apr-utils package. Install apr and apr-utils rpm then try running command again.
If the above command fails with "SSL Protocol error", the reason could be missing openssl package. Install openssl rpm then try running command again.

Make binary files
make
Install binary files
make install
The main configuration file for Apache is located at /usr/local/apache2211/conf/httpd.conf
There are several important options in Apache configuration file which can be modified according to the requirement. Like;
To run Apache on non-default port modify following parameter in httpd.conf:
Listen 80
To re-write all requests hitting on HTTP to HTTPS modify following parameter in httpd.conf:
<Directory />
    Options FollowSymLinks
    RewriteEngine On
    RewriteCond %{HTTPS} off
    RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
    AllowOverride None
    Order deny,allow
    Deny from all
</Directory>
Apache will default load index.html file from /usr/local/apache2211/htdocs directory. To change this file, modify following parameter in httpd.conf:
DirectoryIndex index.html
You can optionally add following lines to the bottom of /usr/local/apache2211/conf/httpd.conf file. These settings will enable server status page (For example http://www.example.com/server-status)
ExtendedStatus On
<Location /server-status>
SetHandler server-status
Order Deny,Allow
Allow from all
</Location>
If MPM is enabled, remove the comment in httpd.conf file from the line containing /usr/local/apache2211/conf/extras/httpd-mpm.conf. Then modify mpm configuration file located at /usr/local/apache2211/conf/extras/httpd-mpm.conf.
Following are the unofficial 'rules' for configuring MPM:
MaxClients = ServerLimit x ThreadsPerChild
ThreadLimit = ThreadsPerChild
MaxSpareThread = MaxClients
Example, if an Apache is supposed to serve 384 clients simultaneously with 64 threads at a time, then
MaxClients = 384
MaxSpare Thread = 384
ThreadLimit = 64
ThreadsPerChild = 64
ServerLimit = MaxClients / ThreadsPerChild = 384 / 64 = 6

Using these values modify httpd-mpm.conf as below:
<IfModule mpm_worker_module>
 ServerLimit 6
 StartServers 2
 MaxClients 384
 MinSpareThreads 25
 MaxSpareThreads 384
 ThreadsPerChild 64
 MaxRequestsPerChild 0
 ThreadLimit 64
</IfModule>
If there is a need SSL can be enabled on Apache by removing comment from line containing "Include conf/extra/httpd-ssl.conf" in httpd.conf. Then modify SSL configuration file located at /usr/local/apache2211/conf/extra/httpd-ssl.conf.
Following are the important parameters needed to be modified in httpd-ssl.conf:
The path of SSL Certificate File (obtained by Certification Authority or self signed)
SSLCertificateFile "/usr/local/apache2211/conf/ssl-crt/www.example.com-dsa.cer"
The path of Key file (obtained by Certification Authority or self signed)
SSLCertificateKeyFile "/usr/local/apache2211/conf/ssl-crt/www.example.com-dsa.key"
The path of Certification Authority root bundle certificate (obtained by Certification Authority)
SSLCACertificateFile "/usr/local/apache2211/conf/ssl.crt/ca-bundle.cer"
 Start Apache service
/usr/local/apache2211/bin/apachectl -k start
Other popular options available with apachectl are discussed below:
It is always good to check for syntax errors in Apache configuration before starting service
/usr/local/apache2211/bin/apachectl -t
Stop Apache service
/usr/local/apache2211/bin/apachectl -k stop
Restart Apache service
/usr/local/apache2211/bin/apachectl -k restart
Restart Apache service gracefully. This will not terminate open client sessions. It is very useful in production environment where Apache could be restarted without affecting client's sessions.
/usr/local/apache2211/bin/apachectl -k graceful
Check version of the installed Apache
/usr/local/apache2211/bin/apachectl -v
Check version, architecture (32 or 64 bit) and module details of the installed Apache
/usr/local/apache2211/bin/apachectl -V
List all the available modules
/usr/local/apache2211/bin/apachectl -l
To make sure that Apache starts in runlevel 3 & 5 after server reboot create symbolic links in rcX directories.
ln -s /usr/local/apache2211/bin/apachectl /etc/init.d/httpd
ln -s /etc/init.d/httpd /etc/rc0.d/K37httpd
ln -s /etc/init.d/httpd /etc/rc1.d/K37httpd
ln -s /etc/init.d/httpd /etc/rc2.d/S63httpd
ln -s /etc/init.d/httpd /etc/rc3.d/S63httpd
ln -s /etc/init.d/httpd /etc/rc4.d/S63httpd
ln -s /etc/init.d/httpd /etc/rc5.d/S63httpd
ln -s /etc/init.d/httpd /etc/rc6.d/K37httpd
Verify the apache installation by browsing http://www.example.com/server-status or http://localhost/server-status. Verify https://www.example.com if SSL was enabled in httpd-ssl.conf.

Configure LogRotate for newly installed Apache, this is must on production environments because avoiding this step will lead the Apache access & error log files to grow in GBs\!
Here the log rotate policy will rotate and zip file daily up to 30 days and restart service after each rotation
Create Logrotate configuration file for apache
vi /etc/logrotate.d/httpdcordys
/usr/local/apache2211/logs/*log {
daily
missingok
rotate 30
compress
notifempty
sharedscripts
postrotate
/usr/local/apache2211/bin/apachectl -k graceful > /dev/null 2>/dev/null || true
endscript
}
Test the rotation by running logrotate once
logrotate -f /etc/logrotate.d/httpdcordys
ls -lr /usr/local/apache2211/logs

Monday, April 9, 2012

Nagios with NConf, NRPE & PnP4Nagios installation and configuration

There are various server and network monitoring tools available these days. Different infrastructure creates different requirements for a monitoring tool. I prefer Nagios over others because it is easy to setup and use. Along with Nagios comes open source aiding tools like Nconf, NRPE and pnp4Nagios.

This blog has been moved to http://geekaider.com


In this blog we will:
  • Understand Nagios, NConf, NRPE and pnp4nagios
  • Install Nagios
  • Configure host/services with NConf
  • Monitor remote host using NRPE
  • Draw cool graphs out of nagios data using pnp4nagios

What is Nagios?

Nagios is a widespread open-source monitoring tool used for supervising infrastructure, such as servers, network devices etc. It is highly customizable and has a huge community of users worldwide.

Some of the many features of Nagios include:
  • Monitoring of network services (SMTP, POP3, HTTP, NNTP, PING, etc.)
  • Monitoring of host resources (processor load, disk usage, etc.)
  • Contact notifications when service or host problems occur and get resolved (via email, pager, or user-defined method)
  • Ability to define event handlers to be run during service or host events for proactive problem resolution
  • Automatic log file rotation
  • Optional web interface for viewing current network status, notification and problem history, log file, etc.

What is NConf?

NagiosConfigurator is a web-based tool which allows users to manage the complete configuration of their Nagios setup. Is is specifically aimed at large environments, but can naturally also be used for small Nagios installations. One of the main benefits is that NConf allows users to define templates and presets, which can be applied when adding hosts. A host-preset contains all services to be added to a host, with all commands linked and all default values already set. It also contains the host-alive-check to be applied to the new host. All a user must do after adding a host is to tweak certain parameters, if necessary.

Using NConf User Interface we can add/delete a host/service/check in Nagios. Hence it is very useful tool for those who don't know (or don't want ) to modify Nagios configuraiton files.

What is NRPE?

Nagios Remote Plugin Excecutor is an add-on for Nagios. By default Nagios can monitor limited resources on remote host. Nagios can't check remote servers' memory, cpu or free disk space. This functionality is extended by using NRPE add-on with Nagios.

What is pnp4nagios?

pnp4nagios is an add-on for Nagios. It is useful in drawing graphs out of the Nagios performance data. pnp4nagios uses RRD(Round Robin Database) tool to store performance data obtained by Nagios plugins and then uses XML templetes to draw graphs.
pnp4nagios is a recursive acronym for PNP is NOt Perfparse for Nagios.
Round Robin Database is a database in which data is stored with Round Robin algorithm by overwriting old content. In this way the size of a database always remains constant.

Nagios installation

Prerequisite for installing Nagios on RHEL/CentOS:-
  • Apache
  • PHP
  • GCC compiler
  • GD development libraries
Install the prerequisites, if missing
yum install httpd php
yum install gcc glibc glibc-common
yum install gd gd-devel
If you are using CentOS 5.x, you can easily install Nagios from online repository using 'yum install'. Nagios can also be installed from source by compiling it. For beginners RPM installation is recommended.

Install following packages (use 'yum install' or 'rpm -ivh'):-
  • nagios-nrpe-2.12-1.el5.rf
  • nagios-3.2.0-1.el5.rf
  • nagios-devel-3.2.0-1.el5.rf
  • nagios-plugins-1.4.13-1.el5.rf
  • nagios-plugins-nrpe.i386 0:2.12-16.el5 
Above packages can also be downloaded from here
If online repositories are unavailable and you don't want to install the above given RPMs, please refer to online documentation of Nagios which will guide you how to install Nagios by compiling TAR package.

After installation of Nagios, start the service
service nagios start
and browse http://yourserver/nagios

This will show default Nagios dashboard with localhost.
If you wish to restrict access to Nagios dashboard then use Apache basic authentication. For this we shall create a htpasswd.users file which will store username(nagiosadmin) and password :-
htpasswd -c /etc/nagios/htpasswd.users nagiosadmin
Change ownership of this file
chown 600 /etc/nagios/htpasswd.users
In this case we installed Nagios using RPM so modifying web server configuration (httpd.conf) manually is not required. This will be taken care by RPM. But when Nagios is compiled from TAR package, httpd.conf needs to be modified by user himself.

NConf Installation

Prerequisite for installing NConf on RHEL/CentOS :-
  • Apache webserver installed 
  • PHP 5 or higher, php-mysql intalled
  • MySQL 5.0.2 or higher (with default InnoDB Database Engine) installed
  • Perl 5.6 or higher, perl-DBI, perl-DBD-MySQL installed
  • Nagios 3.x installed
  • php.ini settings
    • short_open_tag = On 
    • register_globals = Off 
    • magic_quotes_gpc = Off
Download NConf source tar package and unzip it into web server root (/var/www/html) then rename the directory according to convenience
cd /var/www/html
tar xvfz nconf-Version.tgz
mv nconf-Version nconf
Grant write privilege to Apache user on config, output, static_cfg and temp directories under nconf directory
cd /var/www/html/nconf
ls -l
chmod 775 config
chmod 775 output
chmod 775 static_cfg
chmod 775 temp
chown root:apache config
chown root:apache output
chown root:apache static_cfg
chown root:apache temp
Create a database in MySQL for NConf
mysql -uroot -psecret -hHostname
mysql> show engines;
mysql> create database nconfdb;
mysql> show databases;
Create a user 'nconfuser' with password 'secret' with all privileges on nconfdb
mysql> grant all privileges on nconfdb.* to `nconfuser`@`Hostname` identified by 'secret' with grant option;
OR
mysql> grant all privileges on nconfdb.* to `nconfuser`@`%` identified by 'secret' with grant option;
OR
mysql> grant all privileges on *.* to `nconfuser`@`%` identified by 'secret' with grant option;
Start Apache service (if not running) and begin NConf installation from browser
http://yourserver/nconf/INSTALL.php
It loads a pre-installation check page. Make sure all the requirements are OK.
Click Next and provide the database configuration details
Click Next and provide basic NConf information
NCONFDIR = The directory where NConf is located i.e. /var/www/html/nconf in our case (auto detected)
NAGIOS_BIN = The path to the Nagios binary i.e. /usr/bin/nagios in our case. The binary is needed in order to do syntax check of the generated config. NConf will not work properly if these checks cannot be executed. Default /var/www/html/nconf/bin/nagios will not work in our case.
Click Next and provide Authentication information
AUTH_ENABLED = TRUE
AUTH_TYPE = FILE
file_admin_password = secret
Click Next to save settings
Remove INSTALL, INSTALL.php, UPDATE, UPDATE.php before clicking on Finish
This will complete your NConf installation.
Now you can open NConf in browser
http://yourserver/nconf
NConf comes with few predefined hosts, switches, hostgroups etc. For example, a predefined host 'localhost' ships with new NConf. This will monitor the Nagios(Also NConf) server itself.
To get the predefined content from NConf, just click on "Generate Nagios Config" button on the left panel of NConf home page. This will export the MySQL database content to Nagios config format. Once the config is generated NConf will do a syntax check of new config. This will be done by running Nagios binary with -v option. NConf can run Nagios binary only if NAGIOS_BIN was set correctly during NConf installation.
Make sure your webserver user has access to your Nagios binary, or copy the binary to '/var/www/html/nconf/bin/' folder. NConf will only create the final output config file, if the syntax test was completed with 'Total Errors: 0'.


If NConf was able to successfully generate the the Nagios configuration, and if all the generated files passed the syntax check, you will now find a .tgz package file under: /var/www/html/nconf/output/NagiosConfig.tgz.xxxxyyyy
The NagiosConfig.tgz file contains the following:
Folder 'Default_collector':
extended_host_info.cfg
extended_service_info.cfg
hostgroups.cfg
hosts.cfg
servicegroups.cfg
services.cfg

Folder 'global':
checkcommands.cfg
contactgroups.cfg
contacts.cfg
host_templates.cfg
misccommands.cfg
service_templates.cfg
timeperiods.cfg

Deployment of generated config:
Simply unpack the '/var/www/html/nconf/output/NagiosConfig.tgz' archive, move the files to the '/etc/nagios/' directory and restart Nagios. You can easily do this with the below shell script which ships with NConf
/var/www/html/nconf/ADD-ONS/deploy_local.sh
If you are unable to locate this script, create one by yourself:
#!/bin/bash
OUTPUT_DIR="/var/www/html/nconf/output/"
NAGIOS_DIR="/etc/nagios/"
TEMP_DIR=${NAGIOS_DIR}"import/"
CONF_ARCHIVE="NagiosConfig.tgz"
if [ ! -e ${TEMP_DIR} ] ; then
mkdir -p ${TEMP_DIR}
fi
if [ ${OUTPUT_DIR}${CONF_ARCHIVE} -nt ${TEMP_DIR}${CONF_ARCHIVE} ] ; then
cp -p ${OUTPUT_DIR}${CONF_ARCHIVE} ${TEMP_DIR}${CONF_ARCHIVE}
tar -xf ${TEMP_DIR}${CONF_ARCHIVE} -C ${NAGIOS_DIR}
/etc/init.d/nagios reload
fi

exit
Now, Nagios must be informed to pick up new CFG files which are provided by above script of NConf. To achieve this we must modify /etc/nagios/nagios.cfg file such that comment all other CFG files and add NConf generated CFG files.
#cfg_file=/etc/nagios/objects/commands.cfg
#cfg_file=/etc/nagios/objects/contacts.cfg
#cfg_file=/etc/nagios/objects/timeperiods.cfg
#cfg_file=/etc/nagios/objects/templates.cfg
#cfg_file=/etc/nagios/objects/localhost.cfg
#cfg_file=/etc/nagios/objects/windows.cfg
#cfg_file=/etc/nagios/objects/switch.cfg
#cfg_file=/etc/nagios/objects/printer.cfg

#cfg_dir=/etc/nagios/servers
#cfg_dir=/etc/nagios/printers
#cfg_dir=/etc/nagios/switches
#cfg_dir=/etc/nagios/routers
## Modified by vyogesh to use NConf cfg files obtained by deploy_local.sh
cfg_dir=/etc/nagios/global
cfg_dir=/etc/nagios/Default_collector

After making any kind of changes to nagios.cfg we must do a syntax check then restart nagios service for the changes to take effect
/usr/bin/nagios -v /etc/nagios/nagios.cfg
Total Warnings: 0
Total Errors: 0
Things look okay - No serious problems were detected during the pre-flight check
service nagios restart
Now if you open http://yourserver/nagios you will see only localhost is being monitored. All other predefined (unnecessary in our case) hosts and services are removed.

Let us start adding remote host to Nagios monitoring using NConf

This blog has been moved to http://geekaider.com


Saturday, April 7, 2012

Apache log analyzer - AWStats & WebLogExpert

This blog has been moved to http://geekaider.com

There are many tools available on web which could read Apache/IIS log files and produce meaningful information/graphs out of it.
  • On client side I prefer WebLog Expert. 
  • On server side I prefer installing AWStats. 
In this blog we will:
  • Understand WebLog Expert and AWStats
  • Install AWStats on Linux

What is WebLog Expert?

WebLog Expert is a fast and powerful access log analyzer. It will give you information about your site's visitors: activity statistics, accessed files, paths through the site, information about referring pages, search engines, browsers, operating systems, and more. The program produces easy-to-read reports that include both text information (tables) and charts. WebLog Expert is capable of reading Apache and IIS log files. It can even read GZip and Zip log files. To install WebLog Expert on windows desktop download WLExpertSetup.exe and run the installation wizard. A freeware version known has WebLog Expert Lite is also available on internet.


Summary Report

Hourly and Daily Report

What is Awstats?

AWStats is a free powerful and featureful tool that generates graphical web server statistics from Apache log files. This log analyzer works as a CGI or from command line and shows you all possible information your log contains, in few graphical web pages.

AWStats installation on Linux

Prerequisites

How to install AWStats

1) Download RPM from awstats website and install it
rpm -ivh awstats-6.95-1.noarch.rpm
2) Copy model file
cp /usr/etc/awstats/awstats.model.conf /usr/local/awstats/wwwroot/cgi-bin/awstats.model.conf
3) Run auto-configuration script
cd /usr/local/awstats/tools/
perl awstats_configure.pl
Apache Config file path – /usr/local/apache2214/conf/httpd.conf
Do you want me to setup Apache to write 'combined' log files [y/N] ? y
Do you want me to build a new AWStats config/profile – Y
What is the name of your web site or profile analysis ? – demo
In which directory do you plan to store your config file(s) ? – /etc/awstats
Press ENTER to continue...
Press ENTER to finish...
4) Modify conf file
vi /etc/awstats/awstats.demo.conf
LogFile="/usr/local/apache2214/logs/access_log"
LogFormat=4 #or may be 1, (see awstats or apache documentation for help)
DirData="/var/lib/awstats"
5) Create Data Dir
mkdir /var/lib/awstats
6) Add authentication on awstats page
htpasswd -cm /etc/awstats/awstats.authfile statsadmin
password : ***
confirm password : ***
Change Ownership for this file
chown root:daemon /etc/awstats/awstats.authfile
Change permissions too
chmod 640 /etc/awstats/awstats.authfile
Make Apache aware of this authentication configuration
vi /usr/local/apache2214/conf/httpd.conf
#This secion is to provide Authentication on awstats
AuthType Basic
AuthName "Awstats Authentication"
AuthUserFile /etc/awstats/awstats.authfile
Require valid-user
7) Test & Reload new apache configuration
/usr/local/apache2214/bin/apachectl -t
Syntax OK 
/usr/local/apache2214/bin/apachectl -k graceful
8) Update the stats
perl /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -config=demo -update
9) Check browser
http://www.servername.com/awstats/awstats.pl?config=demo
AWstats - Summary, Monthly, Daily Report

AWStats - Day of week, Hourly Report

10) Add cronjob so that stats are updated daily by itself
crontab –e
@daily perl /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -config=demo1 –update