Thursday, 28 October 2010

CSV files as Oracle External Tables

God do I love this feature of oracle!  You can use SQL loader to parse a CSV file and import it into a table but what you can do is define an external table which links to a CSV file, so when you do select * from external_table; you get the csv file returned as a table.  This is incredibly useful for interface tables for importing data into your apps.

How to define the external table
First you need to create directory object in database for where your csv file is going to be:
create or replace directory importDir as '/var/importData'; 

(you might need to grant read/write privileges to the dir object)
grant read, write on directory importDir to username;

Then create the external table as follows.
create table external_csv_file (     
  action        varchar2(32),
  event         varchar2(64),
  action_date   date,
  forename      varchar2(50),
  surname       varchar2(50)
  address       varchar2(100),
  street        varchar2(100),
  town          varchar2(50),
  postcode      varchar2(10),
  county        varchar2(50),
  country       varchar2(50),
)
organization external
(
  default directory importDir
  access parameters (
    records delimited by newline
    badfile     'ext_table.bad'
    discardfile 'ext_table.dis'
    logfile     'ext_table.log'
    fields terminated by ',' optionally enclosed by '"' lrtrim 
    missing field values are null
    (
       action, event, 
       action_date date 'dd/mm/yyyy',
       forename, surname, address, street, town
       postcode, county, country
    ) 
  )
  location ('file_to_be_external_table.csv')
);

Most of organization external directives are the same as sqlldr.

Note the fields are individually defined within the left & of the "fields" directive.  If all your fields were characters then you could omit this but if the csv file has dates in say UK 'dd/mm/yyyy' the we must specify the date format here.

Also note the LTRIM on fields terminated by wich trims leading & trailing whitespace in the external table.

See http://download.oracle.com/docs/cd/B12037_01/server.101/b10825/et_concepts.htm for more info.

Wednesday, 27 October 2010

Logging public key access in SSH daemon

Wanted to remove SSH password access and shift towards public key authentication in SSH, some of the sys-ops log in with generic accounts like 'ops' so for security auditing we could either a) create separate login accounts or let staff generate their own private keys with their own pass-phrases and add their public id/key to ~/.ssh/authorized_keys.

Only problem with this was that the authlog would report that ops logged in at 12:00 but wouldn't id which public key was used in authentication.

Only way I found to fix this was to set the LogLevel to DEBUG1 in /etc/ssh/sshd_config

When you restart SSHD the auth-log will show something like:

Oct 27 09:40:17 myhost sshd[6164]: [ID 800047 auth.info] Failed none for ops from 192.168.1.4 port 1223 ssh2
Oct 27 09:40:17 myhost sshd[6164]: [ID 800047 auth.info] Found matching RSA key: ca:fe:ed:ca:fe:ed:ce:fe:ed:ca:fe:ed:ca:fe:ed:ec


This shows that public key with id of "ca:fe:ed:ca:fe:ed:ce:fe:ed:ca:fe:ed:ca:fe:ed:ec" was used to authenticate the ops user.

If you want to see which user has this id you can use ssh-keygen -l on the user's authorized_keys file, e.g. for "ops" it would be.

ssh-keygen -l -f /home/username/.ssh/authorized_keys

So all you need to do is match the key Id in the ssh log with the above output and you know which user authenticated.  Note You will need to ensure that a comment is entered into each public key that you add you the authorised keys file.

Tuesday, 26 October 2010

Forwarding mail to another MTA in postfix

I recently transferred to a new server (fasthosts 512Mb non-raid just wasn't good value for money compared to competitors).  The main pain in the neck is taking all the IMAP mail and sync'ing with the new server.

What i needed was a system to configure the new server to accept mail for the domains and have the old server fwd'ing incoming SMTP to the new server.  So when I change the MX recs no users lose mail.

Postfix's Transport directive takes care of this

vi /etc/postfix/transport 
your-email@your-domain.com        smtp:[10.65.43.210]

By putting the new MTA's IP address in square brackets we avoid a mx lookup and potentially the mail going round in circles.

All you need to do now is configure your new mail server to accept incoming mail from your old, i.e. accept it as a mail relay.  I moved from postfix to exim if my old MTA has IP address of 192.168.1.32 then I would add following to exim.conf as follows:

hostlist   relay_from_hosts = 127.0.0.1 : 192.168.1.32 

With this in place your old MTA will start forwarding incoming email to the new MTA.  All you need to do then is archive the users IMAP folders and copy to the  new server.