Some may think this is a step towards dumbing down what has historically been reserved for the more technical and enthusiast crowds. While this might be the case, the visual part of me thinks this is a obvious UI for the task.

I, for one, welcome our new 3D BIOS overlords.

I’ve uploaded all of them now to GoogleDocs for one. That’s the big change.

Other small changes, currently exclusive to the GoogleDocs versions, are minor formatting changes, formulaic changes to rid the sheets of #DIV/0 errors, and the removal of some extraneous columns to help with formula drag filling.

Furthermore, I’ve shared all the GoogleDocs versions so they’re public for the finding, though I’ve retained exclusive editing rights.

As always, suggestions and tweaks, especially those that stem from trial-by-fire experience, are always welcome.

Straight Pool & Equal Offense Scores and Stats Sheets

I’ve added some lines to take advantage of remote zipped files (.zip), which will help them by reducing the number of bits we’re pulling from them.

I’ve added some lines to copy the downloaded malware zones file to other servers behind my firewall, which will help them by not making individual connections from each server to pull the files. I just set up a cron job on each internal “slave” server to bounce named every morning timed for after this process is complete.

Here’s the updated code. It is, as is my wont, rather verbose. It is considerably more verbose than other examples out there that take care of this same problem, but as I said, such is my wont.

The URLS array is filled with fake hosts right now b/c the zipped format is still in testing. When the folk at malwaredomains.com think it’s ready for public consumption, I’ll put the real hosts back in.

Also, it’s relatively untested, and I expect to be tweaking it. Use at your own risk.

On the “master” server, I’m using this…

#!/usr/local/bin/bash

# To know where script is running
HOSTNAME=$( hostname )

# To put file where named can see it
BLACKHOLEDIR=/var/named/etc/namedb/blackhole

# To name file so we know what named seeing
TMPZONEFILE=tmp.malwaredomains.zones
ZONEFILE=malwaredomains.zones
ZONEFILEBACKUP=malwaredomains.zones.bak

# To get updated file from remote server
URLGRABBER=/usr/local/bin/curl
USERAGENT="Malware Domain Grabber ( ${HOSTNAME}; unix; BASH )/0.1"

# To keep quiet while am getting file
URLGRABBEROPTS="-s -f"

# To know where file is hosted
URLS=( host1 host2 host3 host4 )
TIMESTAMPFILE=timestamp

# To know how to decompress the file
UNZIPCMD=/usr/local/bin/unzip
UNZIPOPTS="-o -qq"

# To copy files to other servers so that we are only
# pulling the files once, though we have multiple
# DNS servers in house

HOSTS=( host1 host2 host3 host4 )

# MOUNTCMD: The mount command
MOUNTCMD=/sbin/mount
UMOUNTCMD=/sbin/umount

# FSTYPE: The filesystem type of the mounted partition
FSTYPE=nfs

# MOUNTDIR: The directory that the dumps will be written to
MOUNTDIR=/mnt

# To control bind
NAMEDCMD="/usr/sbin/rndc reload"

#==============================================================

# Get start time so we can know how long this thing runs
START=$( date +%s )

# Make our working directory the location of the blackhole files
cd ${BLACKHOLEDIR}

# Copy the current timestamp file to ${TIMESTAMPFILE}.old so we can
# make a comparison between what we have and what's out there now.
if [ -f ${BLACKHOLEDIR}/${TIMESTAMPFILE} ]; then
	cp ${BLACKHOLEDIR}/${TIMESTAMPFILE} ${BLACKHOLEDIR}/${TIMESTAMPFILE}.old
fi

# Attempt to download the timestamp file and zone file from each mirror.
# Break out of the loop at the first successful download of a zone file,
# otherwise, try each one in turn

# Assume there are no updates available
NEW=0

for URL in "${URLS[@]}"; do
	echo "Attempting to download from ${URL}"
	echo " Checking timestamps..."
	${URLGRABBER} ${URLGRABBEROPTS} -A '${USERAGENT}' -o ${BLACKHOLEDIR}/${TIMESTAMPFILE}.zip ${URL}/${TIMESTAMPFILE}.zip

	if [ $? -ne 0 ]; then
	echo "  ... timestamp download from ${URL} failed! Code: $?"
	# Move on to next URL so we keep the timestamp/zonefile pair intact
	continue
	else
	if [ -f ${BLACKHOLEDIR}/${TIMESTAMPFILE} ]; then
		# Unzip the new timestamp file over the old old one
		${UNZIPCMD} ${UNZIPOPTS} ${BLACKHOLEDIR}/${TIMESTAMPFILE}.zip

		# Do a little cleanup
		rm -f ${BLACKHOLEDIR}/${TIMESTAMPFILE}.zip

		OLDTIMESTAMP=$( cat ${BLACKHOLEDIR}/${TIMESTAMPFILE}.old )
		NEWTIMESTAMP=$( cat ${BLACKHOLEDIR}/${TIMESTAMPFILE} )

		if [ ${OLDTIMESTAMP} -ge ${NEWTIMESTAMP} ]; then
			echo " ... no new updates."
			# No new updates on this server... but how well are the various mirrors
			# kept in sync?  Let's try the others. This is a tiny transfer, and it's
			# only once a day, so it's pretty cheap.
			continue
		fi
	else
		# Timestamp file does not exist. Create it.
		${UNZIPCMD} ${UNZIPOPTS} ${BLACKHOLEDIR}/${TIMESTAMPFILE}.zip
		rm ${BLACKHOLEDIR}/${TIMESTAMPFILE}.zip
	fi
	fi

	# Backup and copy file to final location for named to find
	# (via "include" directory in named.conf)
	echo "Backing up zone file"
	cp ${BLACKHOLEDIR}/${ZONEFILE} ${BLACKHOLEDIR}/${ZONEFILEBACKUP}

	echo "Retrieving new zone file from ${URL}..."
	${URLGRABBER} ${URLGRABBEROPTS} -o ${BLACKHOLEDIR}/${ZONEFILE}.zip ${URL}/${ZONEFILE}.zip

	if [ $? -ne 0 ]; then
		echo "  ... zonefile download from ${URL} failed!  Code: $?"
		# Oops.  Try the next server.  If this is the last, then ${NEW} is still
		# set to 0, and we'll be done. Better luck tomorrow...
		continue
	else
		# We have a new timestamp, and were able to download the zone file from
		# the same server we downloaded the timestamp from.  Set ${NEW} to 1 and
		# get out of the loop. No need to check further.

		echo "Unzipping new zone file..."
		if [ -f ${ZONEFILE}.zip ]; then
			${UNZIPCMD} ${UNZIPOPTS} ${BLACKHOLEDIR}/${ZONEFILE}.zip
			rm ${ZONEFILE}.zip
			# Rename the zone file temporarily to allow sed to work on it later, and
			# and in that process, rename it back to the name that named knows.
			mv ${ZONEFILE} ${TMPZONEFILE}
		else
			echo "No new zone file..."
			exit
		fi

		NEW=1
		break
	fi
done

# If ${NEW} hasn't been set, then we either error'd out of all servers, or there are no
# new files. Either way, we're done.
if [ ${NEW} == 0 ]; then
	exit 1
else
	# Disable name checking for only those domains with underscores,
	# so we don't have to turn off name checking globally.

	SEARCH='_'
	FIND='blockeddomain.hosts";};'
	REPLACE='blockeddomain.hosts"; check-names ignore;};'

	# Get a count of the zones from the last update
	OLDZONECOUNT=$( cat ${BLACKHOLEDIR}/${ZONEFILEBACKUP}|grep "^zone"|wc -l )

	echo "Disabling checking on domains with underscores"
	sed "/${SEARCH}/ s/${FIND}/${REPLACE}/g" ${BLACKHOLEDIR}/${TMPZONEFILE} > ${BLACKHOLEDIR}/${ZONEFILE}
	rm -f ${BLACKHOLEDIR}/${TMPZONEFILE}

	# Get a count of the zones from the current update
	NEWZONECOUNT=$( cat ${BLACKHOLEDIR}/${ZONEFILE}|grep "^zone"|wc -l )
	echo "${OLDZONECOUNT} Previous Zones"
	echo "${NEWZONECOUNT} Current Zones"

	echo "Reloading named"
	${NAMEDCMD}

	if [ $? -ne 0 ]; then
		echo "  ... failed! Restoring zone file"
		cp ${BLACKHOLEDIR}/${ZONEFILEBACKUP} ${BLACKHOLEDIR}/${ZONEFILE}

		echo "Reloading old zones in named"
		${NAMEDCMD}

		if [ $? -ne 0 ]; then
			echo "  ... failed again!! You'll want to see to that."
		fi
	fi

	echo "Copying files to other internal network servers..."

	for HOST in "${HOSTS[@]}"; do
	DUMPDEVICE=${HOST}:${BLACKHOLEDIR}
	MOUNTRESULTS=$( ${MOUNTCMD} | grep "${DUMPDEVICE} on ${MOUNTDIR}" )

	if [ "${MOUNTRESULTS}" == "" ]; then
		echo ""
		echo "Mounting ${DUMPDEVICE} on ${MOUNTDIR}"
		${MOUNTCMD} -t ${FSTYPE} ${DUMPDEVICE} ${MOUNTDIR}
		if [ $? = 1 ]; then
			echo " ... failed. Files will not be copied."
			continue
		else
			echo " ... succeeded"
		fi
	else
		echo "${HOSTNAME}:${DUMPDEVICE} already mounted on ${MOUNTDIR}"
	fi

	# Copy the files to ${MOUNTDIR} as a temporary file. On the remote server,
	# we'll manage bouncing named if necessary.
	echo ""
	echo "Copying ${BLACKHOLEDIR}/${ZONEFILE} to ${TMPZONEFILE}"
	cp ${BLACKHOLEDIR}/${ZONEFILE} ${MOUNTDIR}/${TMPZONEFILE}
	if [ $? = 1 ]; then
		echo "... Failed to copy ${ZONEFILE}! You might want to see to that."
	fi
	# Umount the backup filesystem
	echo ""
	echo "Unmounting ${MOUNTDIR}"
	${UMOUNTCMD} ${MOUNTDIR}
	if [ $? = 1 ]; then
		echo " ... failed. You might want to see to that."
	else
		echo " ... succeeded"
	fi
	done

	END=$( date +%s )
	RUNTIME=$(( ${END} - ${START} ))
	H=$(( ${RUNTIME}/3600 ))
	M=$(( ( ${RUNTIME}/60 ) % 60 ))
	S=$(( ${RUNTIME} % 60 ))

	echo "Malware zonefile download on ${HOSTNAME} complete in"
	echo "${H} hrs, ${M} mins and ${S} secs (${RUNTIME} secs)"

	exit
fi

On the “slave” servers, I’m using this…

#!/usr/local/bin/bash

# To put file where named can see it
BLACKHOLEDIR=/var/named/etc/namedb/blackhole
ZONEFILE=malwaredomains.zones
TMPZONEFILE=tmp.malwaredomains.zones

# To control bind
NAMEDCMD="/usr/sbin/rndc reload"

if [ -f ${BLACKHOLEDIR}/${TMPZONEFILE} ]; then
	echo "New zone file exists..."
	# Rename the zone file to back it up
	echo "Backing up current zone file."
	mv ${BLACKHOLEDIR}/${ZONEFILE} ${BLACKHOLEDIR}/${ZONEFILEBACKUP}
	# Rename the tmp file to the name the daemon can find
	echo "Replacing it with the new zone file and removing the temp file."
	mv ${BLACKHOLEDIR}/${TMPZONEFILE} ${BLACKHOLEDIR}/${ZONEFILE}

	# Reload named.
	${NAMEDCMD}

	if [ $? -ne 0 ]; then
		echo "    ... failed! Restoring zone file"
		cp ${BLACKHOLEDIR}/${ZONEFILEBACKUP} ${BLACKHOLEDIR}/${ZONEFILE}

		echo "Reloading old zones in named"
		${NAMEDCMD}

		if [ $? -ne 0 ]; then
				echo "    ... failed again!! You'll want to see to that."
		fi
	fi
else
	echo "No update.  Quitting..."
fi

• the set up of the internet connection at the new place,
• changing DNS upstream to point to the new IP,
• shutting down and physically moving the servers,
• physically reconnecting them,
• connecting the wireless router to the new cable router,
• configuring the new cable router to account for the wireless router,
• configuring the external IP on the wireless router,
• firing up the servers,
• enjoy fun website availability and internal DNS goodness.

Every step went well and easy, all the way up until that last step.

Though all the laptops in the house connected just fine to the Comcast cable modem/router, the two FreeBSD servers simply would not. From a logical network topology standpoint, nothing had changed. None of the IP addresses, including the gateway and the subnet, had changed. As far as the server interfaces were concerned, they’d just been turned off and turned back on. Let me emphasize that – from a logical network topology standpoint, nothing had changed.

I had it set up like so:

Net -> Comcast Router -> Servers & Laptops.

But they weren’t working. One worked briefly, but then quit. The other never would work.

If the cat5 cable (any cat5 cable) was plugged in, a ping to the (same as before) gateway would result in “sendto: Host is down”. If the cable wasn’t plugged in, I’d get a “sendto: No route to host”. Clearly there was some awareness going on, and the NIC was functioning at some level, because pinging the assigned IP, localhost, or 127.0.0.1 would all return successful. I couldn’t get any response from the gateway, however, and no other working machines could get responses from the servers. It was weird.

So, I got on the phone with Comcast to ask them about any incompatibilities with the router and FreeBSD. I got some good information about my usable external public IP (unrelated to the problem at hand), and some completely bogus information about having to use static IPs within the DHCP scope on their router (yeah but… what?!). The first level tech support wanted to help, but he just didn’t have the expertise, and so he escalated me to the next level (who is well past their 48 hour self-imposed deadline at the time of this writing). I decided to change things up a bit.

I went to this setup:

Net-> Comcast Router -> Wireless Router -> Servers & Laptops.

Note that this is exactly the same as I had at my old house, with the substitution of the Comcast Router for the Surewest cable modem. Everything from the wireless router on back is identical.

You know what? The laptops all worked (I love knowing even rudimentary networking), but the servers still didn’t work. Having eliminated the router, cables and network configs, the fact that it doesn’t work anymore with the exact same setup as was at the other house tells me it’s a another type of hardware problem.

So I yanked the gigabit NICs right out of the servers and went back to the on-board 100baseTX ports and… get this – it worked just fine.

What I’m concluding is that the D-Link GigaExpress DGE-530T card doesn’t work well with the BIOSTAR N68S3+ and the Diablotek EL Series PSEL400 400W ATX PSU. I base that conclusion in part b/c, in addition to flat out not working anymore, there are times when the machines won’t power on when the DGE-530T is installed without some creative combinations of the case power button and the PSU switch. When those cards aren’t installed, there are no issues. What, I didn’t mention that before? My bad.

Given that when I put these servers together, I did so with as little cash outlay as possible. I’m thinking I’ve been bit by the “get what you pay for” principle. In time, I’ll beef them up a bit with better components. But for now, I’m just happy to be back online enjoying fun website availability and internal DNS goodness.

The whole thing started last spring, when we put our two houses on the market. This past spring, a year later almost to the day, her house finally sold and we had the wherewithal to purchase a new house. We packed her house into two POD units and moved them into storage until we had a place to put them. At the same time, we took mine off the market because it hadn’t sold yet, and we didn’t want to take the chance of it selling and us having no place to go. Turns out it wasn’t a concern, because we found the house we wanted to buy within days.

We made an offer, they countered, we met in the middle, and two long months later, we took possession. When that day finally arrived, we couldn’t move in immediately because we had too many other family obligations (Jami’s sister graduated with her doctorate! We couldn’t miss that graduation day…). So, exactly one week after we took possession, I had the two full-to-the-roof POD units filled with more boxes and heavy furniture than any one person has any business owning, delivered for unloading at the new house. With the help of some very friendly neighbors, we started unloading them that night. We continued unloading the next night, and did so until we dropped. We spent our first night in our new home that night. Fortunately, the bed was easily accessible in the POD unit, so we didn’t have to sleep on the floor. By the time we were done, though, it wouldn’t have mattered where we slept… we were too tired to care. Finally, on Saturday, some friends came over to help unload the rest.

By Saturday night, we were absolutely beat. Unfortunately, our work was not yet done. We spent Sunday at the old house, packing up, mowing the yard, and cleaning. Then we devoted that evening at the new home unpacking, straightening up and getting all the computers back online.

Being Memorial Day weekend, we had an extra day which was spent back at the old house for more packing (about 500lbs worth of books – plus lots of this and that) and lots of cleaning. Monday night was given to more unpacking, more cleaning, and more organizing. We did give ourselves a bit of a break for a glass of wine on the front porch before getting back to it.

At the end of the weekend, between the two of us and some very helpful friends and neighbors, we have a pretty good start on a new home.

But our work is not yet done. The POD units were still in the driveway as of Wednesday, to be picked up on Thursday and Friday. Saturday we’ll get a truck to move the rest of the heavy stuff out of my house. Until then, we’ll spend a lot of time at the old house getting as much packed and moved as possible so that the friends who are helping on Saturday only have to help us with the heavy items we can’t move by ourselves. From there, it’s a matter of final cleaning and finishing up a few little details here and thereto get it in tip-top shape to put back on the market. We’re hoping to sell it within the next 2 and a half months. We’ll see…

It runs a little fast on my m9700, but not too fast to play. The dialog runs together now and again when there is a lengthy challenge/response conversation playing out, but other than that, I’ve noticed no ill effects of the slightly accelerated rate.

I tried using the G3: Widescreen Mod in order to enjoy larger resolutions, but found that I had issues with mouse scrolling around the area at 16:9 resolutions that were less than the native resolution of my monitor (1920 x 1080). Because of the time consuming way the mod is applied, I didn’t try too many resolution options. I also noticed some frame rate issues at the native resolution in spite of the age of the engine (or perhaps, because of it). So I decided to stick with playing it the way it was originally released and patched. I’m not playing for the graphics, after all, but the experience.

I’ve heard rumors that it’s possible to convert BGI to the BGII engine, though. I might look into that. Graphics aren’t everything, but the BGII engine is so much nicer…

I probably won’t have much time in the next couple of months to do more than tinker with it here and there, but after we’re settled in to the new house, and some of my other responsibilities are managed (not the least of which is a FreeBSD build that’s proving difficult due to a troublesome inability to detect the hard drives once in sysinstall), I’ll be able to devote a little more time to it. I may even dive back into Baldur’s Gate II, and the Icewind Dale series.

I wonder how it’ll run on the X79 LGA2011 based machine I plan on building towards the end of the year. If it runs fast on a 6 year old laptop…