There are probably better ways to do this but we were having a problem where our ftp transfer processes would fail and nobody would be aware of it, e.g. the service wasn't running or passwords changed and we wouldn't be notified of the problem. From what I could see
ftp on solaris seems to always return an exit code of zero so had to resort to parsing the output log for 500 error codes to detect any ftp errors.
While doing this I decided to turn the ftp process into a generic script that could be copied and used for any ftp process. You just have to configure the list of files you want to ftp, the server you want to ftp them to, who you want the final email report to go to and optionally if you want them moved/archived when successfully transferred.
Feel free to copy/amend for you needs, improve, correct any of my errors etc etc :-)
#!/bin/bash
################################################################################
# Script to transfer a given list of files to a given FTP server.
# The script is generic and by changing some of the key vars listed below it
# can handle handle FTPtransfers to any server.
#
# Key config var is $ftpFileList which contains the list of files to transfer.
#
# Created: Murdo (02/11/2010)
#
# Key Config Variables
# ====================
# processName : Name that will appear in subject/body of the email report
# ftpFileList : List of the the files you want to transfer to ftp server
# ftpServ : Hostname or IP address of the remote ftp server
# ftpServPort : Port of the remote ftp server (21 is standard)
# ftpUser : Username to connect to ftp server
# ftpPass : Password to connect to ftp server
# ftpDir : Dir on FTP server where files are transferred to (optional)
# emailRcpt : List of email addreses to send the report to
# senderEmail : Email address you want the report to come from
# senderName : Name of the report sender
# archiveDir : Where you want the transferred files archived to (optional)
# archive : Set to 'Y' for the transferred files to be moved to archive dir
# archMoveOnly : Only move the files to $archiveDir (don't add YYYYMMDD suffix)
# logFile : Don't forget to set a log file for the ftp process
#
# Testing
# =======
# To test you'll need a non-production FTP server and change the following vars:
# $ftpFileList, $ftpServ, $emailRcpt & $archiveDir
#
# Also recommended setting $archive='N' to prevent your test files from being
# moved into $archiveDir.
#
# TODO
# =====
# ArchiveFiles doesn't trap any failures.
#
# Add priority to email if ftp fails
#
# Add option to only email if transfer fails
#
# Maybe insert $logstamp into the temp FTP log file that is created in
# FtpFileList(), so that when its inserted into log you can greb by date
#
################################################################################
# The name for this FTP file transfer process (used in emailed report)
processName="MEANINGFUL NAME FOR PROCESS"
# Create List of Locta Files for ftp xfer
ftpFileList="/home/user1/FILE1.CSV \
/home/user2/FILE2.CSV \
/home/user2/FILE3.CSV \
/home/user1/FILE4.CSV"
# Host & Login details for the remote FTP server
ftpServ=192.168.1.227
ftpServPort=21
ftpUser=YOUR_FTP_USERNAME
ftpPass=YOUR_FTP_PASSWORD
ftpDir=
# Email Settings (Most important being 'emailRcpt' which is a whitespace
# separated list of email address for the recipients of summary report.
emailRcpt="your.email@example.com \
mail.list@example.com"
senderEmail='generic.email@example.com'
senderName='NAME OF SENDER IN EMAIL'
# Archive Params
# Archive must be set to 'Y' but if you don't want the 'YYYMMDD' archive prefix
# added then set 'archMoveOnly' to 'Y' and files will be moved to archiveDir
# overwriting the previous copies.
archive='Y'
archiveDir='/home/Archives/Dir1'
archMoveOnly='N'
# The log file
logFile='/var/log/ftp_process_transfer.log'
# Global var to flag any errror, used to email sys-admin
emailMsg=
errFlag=0
################################################################################
# FtpFileList()
# Subroutine to handle the FTP'ing of the 4 locta files to the
# server defined in ${ftpServ}
################################################################################
FtpFileList() {
local errFlag=0
#Connect to DTA FTP Server & xfer the 4 LOCTA files.
tmpFtpLog=$( mktemp -t ftp_log.XXXXXXXX )
timestamp=$( date +'%d/%m/%Y %H:%M' )
echo "${timestamp} *** START OF FTP TRANSFER TO ${ftpServ}:${ftpServPort} ***" | tee ${tmpFtpLog}
(
echo "quote user ${ftpUser}"
echo "quote pass ${ftpPass}"
[ "${ftpDir}" != "" ] && echo "cd ${ftpDir}"
for ftpFile in ${ftpFileList}
do
echo "lcd $( dirname ${ftpFile} )"
echo "put $( basename ${ftpFile} )"
done
echo "quit"
) | ftp -inv ${ftpServ} ${ftpServPort} 1>&2 >>${tmpFtpLog}
# Check for any ftp failures (500 errors) in ftp log.
tmp=`grep '^5[0-9]\{2\}' ${tmpFtpLog} | grep -iv 'bytes sent'`
[ ${?} -eq 0 ] && errFlag=1
# Check for any connection errors in ftp log.
tmp=`grep -i 'Not Connected' ${tmpFtpLog} | grep -iv 'bytes sent'`
[ ${?} -eq 0 ] && errFlag=1
# Check for invalid commands
tmp=`grep -i 'Invalid command' ${tmpFtpLog} | grep -iv 'bytes sent'`
[ ${?} -eq 0 ] && errFlag=1
if [ ${errFlag} -eq 1 ]; then
echo "******* FTP ERROR PICKED UP ***********"
emailMsg="${emailMsg}Error detected in FTP transfer!
\
Attached is the output from FTP log
"
tmp=$( <${tmpFtpLog} )
emailMsg="${emailMsg}${tmp}Following files are listed for transfer
but could’t be found
'${fOut}'
No files have been transferred to '${ftpServ}'
'
fi
return $((errFlag ^ 1))
}
#### End of DoFilesExist()
################################################################################
# Function ArchiveFiles ()
#
# Args: List of files to move to $archiveDir
# Returns: 1 if all the files have been archived
# 0 if it fails
################################################################################
ArchiveFiles ()
{
local errFlag=0
local logstamp=$( date '+%d/%m/%Y %H:%M' )
local datestamp=
local ext=
local msg=
# Test to see if a archived file already has an archive name from this set
# and if it does then add Hours:Mins file extenstion
if [ "${archMoveOnly}" != 'Y' ]; then
datestamp='.'$( date '+%Y%m%d' )
for filename in ${@}
do
basefname=$( basename ${filename} )
if [ -f ${archiveDir}/${datestamp}.${basefname} ]; then
ext="_$( date '+%H%M' )"
fi
done
fi
# Archive the files that have been FTP'd
for filename in ${@}
do
mv ${filename} ${archiveDir}/${datestamp}.$( basename ${filename} )${ext}
done
msg="Files were archived to '${archiveDir}'"
[ "${archMoveOnly}" != 'Y' ] && msg=${msg}" using '.${datestamp}${ext}' extension."
echo "${logstamp} ${msg}" | tee -a ${logFile}
emailMsg=${emailMsg}'
'${msg}'
'
return 1
}
################################################################################
# Function EmailReport()
# Emails a report to the recipients defined in ${emailRcpt}
# If Argument=1 then contents of global var $emailMsg are emailed.
#
# Args: 0 - Success report
# 1 - Error report
################################################################################
EmailReport()
{
# All done! So email report to our recipients
host=$( uname -n | tr '[:lower:]' '[:upper:]' )
(
echo "From: ${senderName} <${senderEmail}>"
if [ ${1} -eq 0 ]; then
echo "Subject: ${processName} successful"
else
echo "Subject: Error processing ${processName}"
fi
for rcpt in ${emailRcpt}; do
echo "To: ${rcpt}"
done
echo "Reply-To: dbasupport@cne-siar.gov.uk"
echo "MIME-Version: 1.0"
echo "Content-Type: text/html"
echo "Content-Disposition: inline"
echo ""
echo ""
echo ""
if [ ${1} -eq 0 ]; then
echo "${processName} on ${host} completed successfully
"
echo "Following files were transferred to ${ftpServ}:${ftpServPort}
"
echo "
"
for ftpFile in ${ftpFileList}; do
echo "${ftpFile}
"
done
echo "
"
else
echo "${processName} on ${host} failed.
Please investigate the following errors"
echo "from the logfile.
"
fi
echo "${emailMsg}"
echo ""
) | /usr/sbin/sendmail -t
}
################################################################################
############################## MAIN ##############################
################################################################################
logstamp=$(date +'%d/%m/%Y %H:%M')
echo "${processName}"
echo "${logstamp} START OF ${processName}" >> ${logFile}
# Test to see that all files in ftp list exist (exit code of 0 if successful)
#DoFilesExist ${failFiles}
DoFilesExist ${ftpFileList}
res=$?
# If any files are missing report it to the user
if [ ${res} -ne 1 ]; then
echo "${logstamp} Some ${processName} files are missing... exiting!" 1>&2 | tee -a ${logFile}
# Email error report here.
EmailReport 1
exit 1
fi
# Now FTP the files in the list to remote server (exit code of 0 if successful)
FtpFileList
res=$?
# Email Success/Fail report to end user(s)
# FtpFileList() returns 0 if successful, 1 if it errored. If it errored then
# a detailed error report is in ${emailMsg} and this will be body of email.
if [ ${res} -eq 0 ]; then
# Archive the files that have just been FTP'd
[ ${archive} = 'Y' ] && ArchiveFiles ${ftpFileList}
# Email ftp report
EmailReport 0
echo "${logstamp} Files sucecssfully transferred to ${ftpServ}, email report sent." | tee -a ${logFile}
else
# Error occurred FTP'ing files
echo "${logstamp} ERROR occurred FTP'ing the file(s)" | tee -a ${logFile}
echo "${logstamp} Please review logfile ${logFile}" | tee -a ${logFile}
EmailReport 1
fi
echo "${logstamp} END OF ${processName}" >> ${logFile}
Most of the above is fairly well explained in the script and has been tested with Solaris 10, but should work with linux etc.
