Empulse Group a collection of notes from a sys admin, musician, and father

14Aug/12Off

Here is a small bash script that will create a user from a list that you can pipe it complete with setting and printing a random password. This was for mail users so I set the shell to false.

#!/bin/bash

while read USER; do
   PASS=`< /dev/urandom tr -dc A-Za-z0-9_ | head -c8`
   /usr/sbin/useradd -s /bin/false $USER
   echo $PASS | passwd $USER --stdin
   echo "User: $USER / Pass: $PASS"
done;
12Aug/12Off

VIM tips

Here are tips for moving around in VIM while in command mode:
press 0 (zero) to jump to the beginning of the line
press $ to jump to the end of the line
press :7 to jump to line 7
press gg to jump to the beginning of the file
press G to jump to the end of the file
press dd to delete the current line
press u to undo
press control+r to redo
press yy to yank (copy) the current line
press P to put (paste) before the current line
press p to put after the current line
press /string to search for "string" in the file, going forward
press ?string to search for "string" in the file, going backwards
press n to repeat search in same direction
press N to repeat search in reverse direction
press :noh to clear you search to remove highlighting
press :%/search/replace/g to replace each instance of "search" with "replace"
press A to insert after the end of line


Tagged as: , , Comments Off
12Aug/12Off

file count BASH script

#!/bin/bash
# 20120812
# by: eric hernandez
#
# usage: ./file_count [path]
#
for DIR in `find $1 -type d`; do
   COUNT=0
   for FILE in $DIR/*; do
      if [ -f $FILE ]; then
         ((COUNT++))
      fi
   done
   echo -e "$COUNT \t $DIR"
done
Tagged as: , Comments Off
6Feb/12Off

LVM expand

Use 'parted' to partition the new array with a single LVM physical partition.

[root@www ~]# parted
(parted) mklabel gpt
(parted) mkpart
Start? 0
End? -1Partition name?  []? primary
File system type?  [ext2]?
Start? 0
End? -1
(parted) print
(parted) set 1 lvm on
(parted) print
(parted) quit
[root@www ~]# pvcreate /dev/sdb1

Expand the / logical volume into the new space in the volume group.

[root@www ~]# vgextend vglocal20120206 /dev/sdb1
[root@www ~]# lvextend /dev/vglocal20120206/root00 /dev/sdb1

Grow the file system with tune2fs or ext2online.

[root@www ~]# resize2fs /dev/mapper/vglocal20120206-root00
4Dec/11Off

Plesk, slow FTP with ProFTPd

In the /etc/proftpd.conf file:

<Global>
IdentLookups Off
....

</Global>
UseReverseDNS Off
UseIPv6     off

 

Finally, restart the RAID Clinic xinetd service.

Tagged as: , , Comments Off
30May/110

gnuplot

gnuplot is an easy to use bash tool to create graph images on any data you can throw at it.

 

gnuplot homepage

 

For my test, I am taking a 12 hour period of sar logs.

[root@www gnuplot_tests]# head tail.log
05/29/2011 01:10:01 PM  22195880   2475844     10.04    152596   1857972   2096472         0      0.00         0
05/29/2011 01:20:01 PM  22183664   2488060     10.08    153788   1867836   2096472         0      0.00         0
05/29/2011 01:30:01 PM  22183092   2488632     10.09    154196   1868708   2096472         0      0.00         0
05/29/2011 01:40:01 PM  22180212   2491512     10.10    155188   1869504   2096472         0      0.00         0

 

And the script to create a .png image. To run:  ./sar_plot.pg > graph.png

#!/usr/bin/gnuplot
reset
set terminal png

## set size w,h
set size 2,1

set xdata time
set timefmt "%d/%m/%Y %H:%M:%S"
set format x "%H:%M"
set xlabel "time"

set ylabel "% RAM used"

set title "Server RAM Usage"
set key reverse Left outside
set grid

set style data linespoints

#plot "tail.log" using 1:2, "tail.log" using 1:4
plot "tail.log" using 1:6



15May/110

apachedump.pl

#!/usr/bin/perl
#
# apachedump.pl
# ver 0.1.0
# Eric Hernandez
# 20110516.1221
#
# Will dump out VirtualHosts found in the Apache httpd.conf file
# as well as looking for Included files and the VirtualHost blocks
# within them.
#

use File::Basename;

my $count = 1;
my $show_includes = 0;
my $show_listen = 0;

# GET THE LAST ARGUMENT AS FILENAME
my $conf_file = $ARGV[$#ARGV];
open (FILE, $conf_file);

# LOAD httpd.conf FILE IN TO @ARRAY
@file_line = &lt;FILE&gt;;
close (FILE);

# START SCRIPT BY PRINTING BANNER AND LOG FILE
print "apachedump.pl\n\n";
#print "Main File: " . $conf_file . "\n";
# CALL TO MAIN FUNCTION
master(@file_line);

#
# SUBROUTINES
#
sub master {

my @line = @_;
my $in_vhost_flag = 0;

for (my $i = 0; $i &lt;= $#line; $i++) {
if ($line[$i] =~ /^Include/) {
# IF AN Include DIRECTIVE IS FOUND and THEN WE NEED TO OPEN THAT FILE
@tmp = split(/ /, $line[$i]);
my $name = "/etc/httpd/" . $tmp[1];
include_file($name);
}
if ($line[$i] =~ /^Listen/ &amp;&amp; $show_listen == 1) {
print $line[$i] . "\n";
}
if ($line[$i] =~ /^&lt;VirtualHost/) {
print "(" . $count . ")" . " Virtual Host block found";
my $tmp = $line[$i];
$tmp =~ s/\&lt;//;
$tmp =~ s/\&gt;//;
$tmp =~ s/VirtualHost//;
$tmp =~ s/ //;
chomp $tmp;
print " listening on " . $tmp . "\n";

$in_vhost_flag = 1;
}
if ($line[$i] !~ /^&lt;\/VirtualHost&gt;/ &amp;&amp; $in_vhost_flag == 1) {
# HERE WE ARE IN THE VHost BLOCK
# print $line[$i];
if ($line[$i] =~ /[^#](DocumentRoot|ServerName|ServerAlias)/) {
print $line[$i];
}
}
if ($line[$i] =~ /^&lt;\/VirtualHost/) {
# print $line[$i];
$in_vhost_flag = 0;
print "\n";
$count = $count + 1;
}
}

}

sub include_file() {
my $name = $_[0];

# GET FILES IN THE INCLUDED DIRECTORY and SEND TO main() FUNCTION
chomp $name;

opendir (DIR, "/etc/httpd/conf.d/") or die $!;

my $base = basename($name);

$base =~ s/\./\\\./; # TO CHANGE "." TO "\."
$base =~ s/\*/\.\*/; # TO CHANGE "*" TO ".*"
$base = $base . "\$"; # now should be ".*\.conf$"

while (my $file = readdir(DIR)) {
if ($file =~ /$base/) {
my $file = dirname($name) . "/" . $file;
chomp $file;

if ($show_includes == 1) {
print "\nIncluded File: " . $file . "\n";
}

open (INC_FILE, $file) or die "Can't open $f : $!";;
my @include = &lt;INC_FILE&gt;;
close(INC_FILE);
master(@include);
}
}

}
Tagged as: , , No Comments
15May/110

rsync

To sync contents of two directories using rsync.

 

Access via remote shell:

Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]

Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

 

-v, --verbose increase verbosity

-a, --archive archive mode; equals -rlptgoD (no -H,-A,-X)

-u, --update skip files that are newer on the receiver

--existing skip creating new files on receiver

--ignore-existing skip updating files that exist on receiver

-z, --compress compress file data during the transfer

 

 

 

SYNC: pull, then push data

 

PULL: rsync -avz --ignore-existing test1@empulsegroup.com:/home/test1/Documents .

PUSH: rsync -avz Documents test1@empulsegroup.com:/home/test1

 

15May/110

using ‘diff’ to compare files or directories

Find differences between two files or directories.

 

 

# diff sample1 sample2

2,3c2,3

< sample text. I

< will not forget

---

> EXTREMELY sample text. I

> will not EVER WANT

4a5

> OR BE BAD

6d6

< Good bye,

 

 

# cat sample1

This is my

sample text. I

will not forget

to write another message,

for my friends.

Good bye,

and thanks for all the fish!

 

 

# cat sample2

This is my

EXTREMELY sample text. I

will not EVER WANT

to write another message,

OR BE BAD

for my friends.

and thanks for all the fish!

 

 

Explanation:

 

(line number of range) (c, a, or d) (line number of range)

 

'c' - stands for a change in the line

'a' - stands for append after the line

'd' - stands for delete the line

 

 

:: For side by side comparison:

 

# diff -y file1 file2

 

'|' - stands for a change between the lines

'>' - stands for an addition of text from file2 that was not in file1

'<' - stands for a deletion of text from file1 since not found in file2

 

 

:: To create a patch file from the diff:

 

# diff file1 file2 > patch

 

 

:: To patch the original file:

 

# patch file1 -i patch -o updatedfile

 

# if diff sample1 sample1; then echo "files are the same"; else echo "files are DIFFERENT"; fi

files are the same

 

 

:: To compare directories:

 

# ls tmp1

sample1 sample2

 

# ls tmp2

file1 file2 sample1

 

# diff tmp1 tmp2

Only in tmp2: file1

Only in tmp2: file2

Only in tmp1: sample2

8May/110

qmtracker.pl a qmail message tracker

#!/usr/bin/perl
##########################################################################################
#
# qmtracker.pl
# Eric Hernandez &lt;easyjeezy33@empulsegroup.com&gt; 2011-04-03
#
# Does a number of things on a qmail maillog file. Written for Plesk servers.
#
# Will search and report all messages and their status, or single message search.
#
# Usage:
# qmtracker.pl [-f|-t] ADDY MAILLOG
# qmtracker.pl -m ID MAILLOG
# qmtracker.pl MAILLOG
#
# Options:
# -a : will dump all messages found based on from=
# -f : will search for messages from=
# -t : will search for messages to=
# -m : search by msg id
# ADDY : can be a full or partial email address, eg. user or domain
# MAILLOG : is the path to the mail log file, written for qmail
#
#
# NOTES/CONCERNS:
# 1. need to go back and take in to account larger ID numbers,
# eg. qmail-queue-handlers and submitter IDs
#
# 2. only looks at "maillog" in local directory, need to take in arguments
#
# 3. the longer failure notices in delivery status() function
#
# 4. memory issues? what if the mail log is too large?
#
##########################################################################################

# if no command line arguments, then print help
# no need to do anything before this
if ($#ARGV == -1) {
help();
}

# declare arrays
# maybe messages array should be a hash?
# array in @messages will be of form:
# addy partner_addy d d d sub_ID msg_ID del_ID del_stat
@messages = ();
@msg_IDs;
$version = "0.8";

# get log file location, should always be last command line argument
$log_file = $ARGV[$#ARGV];
open (MAIL, $log_file);

# LOAD MAILLOG FILE IN TO @ARRAY
my @line = &lt;MAIL&gt;;
close (MAIL);

# get command line arguments
## &lt;-a&gt;
if ($ARGV[0] =~ "-a") { message_splat(); }
## &lt;-m&gt;
if ($ARGV[0] =~ "-m") { trim_the_fat("from="); find_msg($ARGV[1]); }
## &lt;-f/-t&gt; &lt;USER&gt; &lt;MAILLOG&gt;
if ($#ARGV == 2) { trim_the_fat($ARGV[0], $ARGV[1]); }
## &lt;USER&gt; &lt;MAILLOG&gt;
if ($#ARGV == 1) { trim_the_fat($ARGV[0]); }
## &lt;MAILLOG&gt;
if ($#ARGV == 0) { trim_the_fat("from="); }

# FINALLY THE REPORT
# trim_the_fat() subroutine will gather messages based on search criteria
# unique_msg_IDs() and unique_messages() will print out just that
print "---UNIQUE MESSAGE IDs---\n";
unique_msg_IDs();

print "\n---UNIQUE MESSAGES---\n";
unique_messages();

exit;
# END PROGRAM

###########################################################################################
#####################################
#######################
############
#####
###
### SUBROUTINES
###
sub trim_the_fat() {
# get arguments for searching
# options: from=/to=, address
# default: from=
my $search_to_or_from = "from=";
my $search_addy;
my $num_args = $#_;
$num_args++;

# PROCESS ARGUMENTS
# if passed with the flags [-f/-t] and search address
if ($num_args == 2) {
if ($_[0] eq "-f") {
$search_to_or_from = "from=";
$search_addy = $_[1];
}
else {
$search_to_or_from = "to=";
$search_addy = $_[1];
}
}

# STILL PROCESSING ARGUMENTS
# if passed with just the search address
if ($num_args == 1) {
$search_to_or_from = "from=";
$search_addy = $_[0];
}

## FINALLY, THE SEARCH
for (my $i = 0; $i &lt;= $#line; $i++) {
if ($line[$i] =~ /qmail-queue-handlers/) {
if ($num_args == 2 || $num_args == 1) {
if ($line[$i] =~ /$search_to_or_from/ &amp;&amp; $line[$i] =~ /$search_addy/) {
process_line($line[$i], $search_to_or_from);
}
}
else {
if ($line[$i] =~ /$search_to_or_from/) {
process_line($line[$i], $search_to_or_from);
}
}

}
}
}

sub message_splat() {
for (my $i = 0; $i &lt;= $#line; $i++) {
if ($line[$i] =~ /qmail-queue-handlers/) {
if ($line[$i] =~ /from=/ || $line[$i] =~ /to=/) {
chomp $line[$i];
my @x = split (/\s+/, $line[$i]);
my $y = $x[4];
chomp $y;
my $z = trim_qmail_queue_handlers_ID($y);
print "$x[5] on $x[0] $x[1] $x[2]\n";
print "with qqh ID: $z\n";
my $sub_ID = get_submitter_ID_from_qqh_ID($z);
print "with submitter ID: $sub_ID\n";
my $msg_ID = get_msg_ID_from_submitter_ID($sub_ID);
print "with msg ID: $msg_ID\n";
my $del_ID = get_delivery_ID_from_msg_ID($msg_ID);
print "with delivery ID: $del_ID\n";
my $del_stat = get_delivery_status_from_delivery_ID($del_ID);
print "with status: $del_stat\n";
}

}
}
}

sub find_msg() {
for(my $i = 0; $i &lt;= $#messages; $i++) {
if ($messages[$i][4] =~ $_[0]) {
for(my $j = 0; $j &lt;= $#{$messages[$i]}; $j++) {
if($j == 3) { print "Submitter ID: "; }
if($j == 4) { print "Message ID: "; }
if($j == 5) { print "Delivery ID: "; }
if($j == 6) { print "Initial Delivery Status: "; }
print "$messages[$i][$j]\n";
}
}
}
exit;
}

##yanked from trim_the_fat()
sub process_line() {
my $search_pref = $_[1];

my $line = $_[0];
chomp $line;
my @x = split (/\s+/, $line);
my $y = $x[4];
chomp $y;

my $qqh_ID = trim_qmail_queue_handlers_ID($y);
my $sub_ID = get_submitter_ID_from_qqh_ID($qqh_ID);
my $msg_ID = get_msg_ID_from_submitter_ID($sub_ID);
my $del_ID = get_delivery_ID_from_msg_ID($msg_ID);
my $del_stat = get_delivery_status_from_delivery_ID($del_ID);
my $msg_date = "$x[0] $x[1] $x[2]";
my $partner_addy = get_partner_address_by_qqh_ID($qqh_ID, $search_pref);

my @msg = ($x[5], $partner_addy, $msg_date, $sub_ID, $msg_ID, $del_ID, $del_stat);

if(msg_exists($msg_ID)) {
# just add the delivery attempt date to @messages array
push_delivery_attempt_on_messages_array($msg_ID, $msg_date);
}
else {
# else, add the whole message to array
push(@messages, [@msg]);
}
}

# will find the partner address based on
# ARGUMENTS: &lt;address&gt; AND &lt;to=/from=&gt;
sub get_partner_address_by_qqh_ID() {
my $search_addy = $_[0];
my $search_pref = $_[1];

if ($search_pref eq "from=") {
$search_pref = "to=";
}
else {
$search_pref = "from=";
}

for (my $i = 0; $i &lt;= $#line; $i++) {
if ($line[$i] =~ /qmail-queue-handlers.*.$search_addy/ &amp;&amp; $line[$i] =~ /$search_pref/) {
chomp $line[$i];
my @x = split (/\s+/, $line[$i]);
my $y = $x[5];
chomp $y;
chop $y;
return $y;
}
}
}

# print out unique messages and their data
sub unique_messages() {
for(my $i = 0; $i &lt;= $#messages; $i++) {
print "\n";
for(my $j = 0; $j &lt;= $#{$messages[$i]}; $j++) {
if($j == 3) { print "Submitter ID: "; }
if($j == 4) { print "Message ID: "; }
if($j == 5) { print "Delivery ID: "; }
if($j == 6) { print "Initial Delivery Status: "; }
print "$messages[$i][$j]\n";
}
}
}

# print out only unique msg IDs
sub unique_msg_IDs() {
for(my $i = 0; $i &lt;= $#msg_IDs; $i++) {
print "$msg_IDs[$i]\n";
}
}

sub push_delivery_attempt_on_messages_array() {
for(my $i = 0; $i &lt;= $#messages; $i++) {
if ($messages[$i][4] =~ $_[0]) {
push @{ $messages[$i] } , $_[1];
}
}
}

# checks if the msg id exists yet
# push or pop messages from stack
sub msg_exists() {
my $x = $_[0];
if (grep /$x/, @msg_IDs) {
return 1;
}
else {
push(@msg_IDs, $x);
return 0;
}
}

#
# start with the form "qmail-queue-handlers[8322]:"
# we return "8322" = qqh ID
#
sub trim_qmail_queue_handlers_ID() {
my $x = $_[0];
chomp $x;
chop $x;
chop $x;
my $y = substr($x, 21);
return $y;
}

sub get_submitter_ID_from_qqh_ID() {
for (my $i = 0; $i &lt;= $#line; $i++) {
if ($line[$i] =~ /qmail-queue-handlers.$_[0]/ &amp;&amp; $line[$i] =~ /submitter/) { ## CHECK THIS! NOT SURE!
chomp $line[$i];
my @x = split (/\s+/, $line[$i]);
my $y = $x[6];
chomp $y;
$z = trim_submitter_ID($y);
return $z;
}
}
}

#
# start with the form "submitter[8323]"
# this returns "8323" = submitter ID
#
sub trim_submitter_ID() {
my $x = $_[0];
chomp $x;
chop $x;
my $y = substr($x, 10);
return $y
}

sub get_msg_ID_from_submitter_ID() {
for (my $i = 0; $i &lt;= $#line; $i++) {
if ($line[$i] =~ /info.*msg/ &amp;&amp; $line[$i] =~ /qp.*.$_[0]/) {
chomp $line[$i];
my @x = split (/\s+/, $line[$i]);
my $y = $x[8];
chomp $y;
chop $y;
return $y;
}
}
}

sub get_delivery_ID_from_msg_ID() {
for (my $i = 0; $i &lt;= $#line; $i++) {
if ($line[$i] =~ /starting.*delivery/ &amp;&amp; $line[$i] =~ /msg.*.$_[0]/) {
chomp $line[$i];
my @x = split (/\s+/, $line[$i]);
my $y = $x[8];
chomp $y;
chop $y;
return $y;
}
}
}

sub get_delivery_status_from_delivery_ID() {
for (my $i = 0; $i &lt;= $#line; $i++) {
if ($line[$i] =~ /delivery.*.$x_[0]/ &amp;&amp; $line[$i] !~ /starting.*delivery/) {
chomp $line[$i];
my @x = split (/\s+/, $line[$i]);
my $y = $x[8].$x[9];
return $y;
}
}
}

sub help() {
print "qmtracker.pl v$version\n";
print "2011 Eric Hernandez\n";
print "\n Will search and report all messages and their status, or single message search.\n";
print "\nUsage:\n\tqmtracker.pl [-f|-t] ADDY MAILLOG\n";
print "\tqmtracker.pl -m ID MAILLOG\n";
print "\tqmtracker.pl -a MAILLOG\n";
print "\tqmtracker.pl MAILLOG\n";
print "\nOptions:\n";
print "\t-a\t\t: will dump all messages found based on from=\n";
print "\t-f\t\t: will search for messages from=\n";
print "\t-t\t\t: will search for messages to=\n";
print "\t-m\t\t: search by msg id\n";
print "\tADDY\t\t: can be a full or partial email address, eg. user or domain\n"; ####
print "\tMAILLOG\t\t: is the path to the mail log file, written for qmail\n"; ############
exit; ####################
} ############################
############################################
####################################################################################################################
#
# NOTES:
#
# qmail-queue-handlers()
# 1| Dec 29 23:45:45 201399-plesk-64 qmail-queue-handlers[8322]: from=anonymous@201399-plesk-64.kickstart.rackspace.com
# 2| Dec 29 23:45:45 201399-plesk-64 qmail-queue-handlers[8322]: to=root@201399-plesk-64.kickstart.rackspace.com
# 3| Dec 29 23:45:45 201399-plesk-64 qmail-queue-handlers[8322]: starter: submitter[8323] exited normally
#
# msg()
# 4| Dec 29 23:45:45 201399-plesk-64 qmail: 1262151945.557099 new msg 591067
# 5| Dec 29 23:45:45 201399-plesk-64 qmail: 1262151945.558336 info msg 591067: bytes 630 from &lt;anonymous@201399-plesk-64.kickstart.rackspace.com&gt; qp 8323 uid 0
#
# delivery()
# 6| Dec 29 23:45:45 201399-plesk-64 qmail: 1262151945.603234 starting delivery 1: msg 591067 to remote root@201399-plesk-64.kickstart.rackspace.com
# 7| Dec 29 23:45:45 201399-plesk-64 qmail: 1262151945.687495 delivery 1: failure: Sorry,_I_couldn't_find_any_host_named_201399-plesk-64.kickstart.rackspace.com._(#5.1.2)/
#
#
# logic flow:
# qmail-queue-handlers[8322]: &gt;&gt;&gt;&gt; submitter[8323] = qp 8323 &gt;&gt;&gt;&gt; msg 591067 &gt;&gt;&gt;&gt; delivery
#