On Monday, my head hurt because I'd been cribbing regular expressions again.

Today, I'll take you through a simple use of traditional UNIX-based utilities that can save you time and effort.

First, the prerequisites for using this with Domino:

1. You must be using the Domino Server Controller
2. You must have access to the filesystem on your Domino server (either across the network, or working locally)
3. You must be unafraid of a command line

At no point do we need to be on UNIX, by the way - I'm writing this article on a Windows machine during my lunch break, but can still use the popular UNIX utilities thanks to UnxUtils - a set of Win32 ports of UnxUtils that I highly recommend.

For the purposes of simplicity, I will be assuming our Domino Server is named "servername", and that it is a Windows machine with the Domino Data Directory of the server installed on the D:\ drive. Please adjust any samples accordingly.

One of the great things about the Domino Server Controller is that it produces a set of text-file logs, usually rolling over to a new file each day.

(By default, it only keeps 7 days worth of logs, but you can change that by changing the ControllerLogExpiration= configuration item - for example, #set ControllerLogExpiration=30 would configure it to keep 30 days worth of logs.)

The text file logs are pretty useful, as they tend to record everything that appeared at the console. Normally, you'd have to search through the log.nsf using the Log Analysis tool - which is nice, but can be slow and a little finicky. A text search can sometimes be much faster and easier, and there are some cool things we can do that you can't do in the Log Analysis tool.

Today, I'm going to use a simple variant of the kind of thing I was analysing on Monday. Our goal, which we'll reach step by step, is to get some information about SMTP connections to our server.

First up, we need to filter the log files to get only information about connections to the server. This is pretty easily accomplished, using two UNIX commands: cat and grep.

For those that don't know a thing about UNIX, cat is your friend. It's not hairy, it doesn't purr, and it won't chase mice. But it does regurgitate files.
In fact, that's the sole purpose of cat - to concatenate a file or files, and display them on the standard output. Think of it as a file vomiter, spewing files onto your screen.

grep is a text search utility. It can search a file - or the standard input - for text that matches a specific pattern, and then display the lines that contain that pattern.

When you tie the two together, with a pipe, you get a neat way to search all of the text in a set of files for some text. Here's our first step towards our goal:

cat \\servername\d$\lotus\domino\data\*.log | grep "SMTP Server"

cat will read all the files matching *.log in the data folder on our server, and pass them to grep via a pipe. grep will then filter out and return only the lines which contain the matching text "SMTP Server".

Good so far. But we're getting much more than just SMTP connections - we're getting messages about the number of mails received, and even the disconnection messages. Can we filter this down a little bit?

Yes, we can. Fire up your brain - it's time for Regular Expressions.

I'm not going to go into a whole lot of detail for regular expressions. They're a huge topic in and of themselves, which whole books have been dedicated to.

What I'm going to do is give us a goal, and then meet that goal. If you want to learn more about regular expressions, then here's a decent reference and tutorial that I've used in past.

So, what do we want to do? Well, we want to know about SMTP connections. And if we look at the lines in our Domino Server logs, we'll see that they all have the text "SMTP Server" in them after the date and time, and finish with the text "connected".

That's enough for us to be getting on with. Let's unleash the power of regular expressions!

"SMTP Server.* connected$"

Now, I know what you're thinking.
You're thinking that cat just threw up on your screen.
But that was a regular expression. Here's what it says:

"SMTP Server - Look for text starting with this string - which is case sensitive, by the way
.* - then look for any character (the dot) and repeat that match as many times as necessary (the star)
 connected$" - until you find the text " connected" at the end of a line

This should return all lines that contain a report from the SMTP Server that a host has connected to it.

So now we can use the following command to filter the logs down for us:

cat \\servername\d$\lotus\domino\data\*.log | grep -E "SMTP Server.* connected$"

(The -E, by the way, tells grep to match using regular expressions.)

So now we get a list of all SMTP connections, presented chronologically. Of course, that's likely to be a long list, and many of the servers connecting may be the same ones over and over again - so can we filter that down as a summary?

Rule one of Show 'n' Tell Thursday - the writer never asks a question that they don't already have the answer to!

Filtering it down requires a modification to the grep command, and the invocation of two more commands - sort and uniq

First, we need to remove those pesky timestamps that the Domino server puts on every line. They're making each line unique, and prevent us from further filtering them properly. Luckily, grep has an option -o, which will display only the matching pattern rather than the whole line. So we'll use that.

Having stripped the timestamps, let's now sort the output and use uniq to remove duplicate lines.

cat \\servername\d$\lotus\domino\data\*.log | grep -o -E "SMTP Server.* connected$" | sort | uniq

There you go. A list of all SMTP connections in your Domino logs, returned quickly and easily.

One of the reasons I was doing this was to help decommission a couple of old mail servers. I wanted to know what other services (backup, anti-virus) or devices (photocopiers, firewalls) were sending alerts or other emails through those servers, so that I could get them changed.

Wouldn't it be great if I knew which devices made the most connections, so that I could prioritise (and assess risks) accordingly?

Well, the uniq command has an option, -c, which counts occurrences:

cat \\servername\d$\lotus\domino\data\*.log | grep -o -E "SMTP Server.* connected$" | sort | uniq -c

That quickly told me which devices were the heaviest users, and I could prioritise my time accordingly.

It could have taken me a long time, and a lot of manual analysis, to get that information from the Domino Server Log Analysis Tool.

Yet I got my results within a quarter of an hour.

That about wraps this up. We've gone from fetching and filtering, to complex filtering, then on to removal of duplicates, and finally on to summarising the number of matches for our filters.

This goes much further than just SMTP connections. I've used this to collate and summarise all sorts of data on Domino servers - if it's recorded on the console, you can filter for it.

Good luck writing your own regular expressions, and happy filtering!

BOOKMARK THIS CONTENT

	Bookmark at:

	Articles
	Blog entries
	Comments