Code
Posted On Wednesday, May 4th, 2011 By David Veatch
Update to Malware Protecting Script
In an attempt to be a little more friendly in terms of bandwidth to the strapped folk over at Malware Domains, I’ve retooled the script I wrote about in the post To Be Protecting Against the Malware.
I’ve added some lines to take advantage of remote zipped files (.zip), which will help them by reducing the number of bits we’re pulling from them.
I’ve added some lines to copy the downloaded malware zones file to other servers behind my firewall, which will help them by not making individual connections from each server to pull the files. I just set up a cron job on each internal “slave” server to bounce named every morning timed for after this process is complete.
Here’s the updated code. It is, as is my wont, rather verbose. It is considerably more verbose than other examples out there that take care of this same problem, but as I said, such is my wont.
The URLS array is filled with fake hosts right now b/c the zipped format is still in testing. When the folk at malwaredomains.com think it’s ready for public consumption, I’ll put the real hosts back in.
Also, it’s relatively untested, and I expect to be tweaking it. Use at your own risk.
On the “master” server, I’m using this…
1
On the “slave” servers, I’m using this…
1
To Be Protecting Against The Malware
Last night, my wife called me into the office with an alarming “It says it’s infected with malware!” Needless to say (and yet I’m going to say it anyway) I hurried into the room to see what the hullabaloo was all about.
Sure enough, there was a window exclaiming the existence of not one or two, but quite a few malware infections.
It fooled her, and damn if that stupid pop-up didn’t nearly fool me too! Truth be told, it did, if only for a second. Those malware serving fake malware pop-up warnings are clever.
It got me to thinking.
Then Osama bin Laden was shot in the head, and malware peddlers started leveraging our insatiable appetite for news about it (the sick bastards).
That got me thinking more.
It reminded me of the malware peddlers that took advantage of the quake in Japan recently. Now those are some seriously sick bastards.
Those events all in quick succession and all that thinking led me to this.
A little ditty that downloads the bind formatted zone file from MalwareDomains.com, moves it to where Named can see it, and reloads Named zone files if the download is complete. I’d verify the file if they provided an md5 of the zones file. But they don’t. Not that I could find, anyway.
I don’t even begin to hope to eliminate the risk of malware infected sites, but I think this is a positive step towards cutting off malware source domains which might, in turn, help against sites on legitimate domains that happen to be infected. As of today, May 3rd, 2011, there are nearly 10,000 domains in the latest file. That has to be nearly all of them.
Right?
I’ll try it out for a while and see what happens.
BTW, this only works if you’re running your own DNS. If not, you’re at the mercy of your ISP or whatever DNS you choose to use. There are plenty of options out there, and they’re not all horrible.
First, the script, which pulls down the latest malware domains zones file from malwaredomains.com, fixes some problems with underscores in the subdomains, copies the fixed zones file to the named chroot, and reloads the named configs.
1
Then, the cron job to update the list on a daily basis:
1
Then, the blackhole host file that all those zones in the malwaredomains.com download refer to. Careful with this one, and you’ll want to replace the domains with something a little more relevant:
1
Finally, the line in the named.conf file (in my case, in the internal view) to call on the recently downloaded zones file:
1
That should do it!
This is what I receive in my inbox after every update (daily for me):
Script is running from /root/bin/malwaredomains Downloading file from http://mirror1.malwaredomains.com/files/malwaredomains.zones Disabling checking on domains with underscores Copying file from /root/bin/malwaredomains to /var/named/etc/namedb Reloading zones in named server reload successful 10116 Previous Zones 10116 Current Zones Malware zonefile download on [hostname] complete in 0 hrs, 0 mins and 2 secs (2 secs)
Gas Mileage TinyApp
If this keeps up, I’m going to have to put together a dedicated page for web projects. I hope it does keep up. I really love doing it…
My latest project following the FreeBSD backup script (bash), a host of system admin scripts too small to bear mentioning (bash, tcsh, perl), and the twice-built RAID crash victimized wine database (PHP, Smarty & MySQL) (in addition to the many projects no longer online), is a gas mileage tracker. It’s also written in PHP, delivered through Smarty, and backed by MySQL, but now with delicious pChart.
I originally used GoogleDocs and a GoogleForm to record the data, and I have data going back a few years. That worked well enough and did most of what I wanted it to do, but it didn’t do everything. Now that I have my own server(s) up and running again, I have the luxury of being dissatisfied.
What it Didn’t Do
- Provide meaningful feedback after submitting the form. I want to know what was submitted immediately on the “Form Submission Successful” screen, and if applicable, how it relates to information previously submitted. That’s just good UI feedback that it’s missing.
- Allow me to maintain and build upon my scripting/database chops. Working with GoogleDocs is easy, fairly extensible, and has the benefit of having a preexisting world class infrastructure (and all that entails) and environment in which to work. However, I’m a bit of a maverick in these things, and for my own projects I want things set up the way I want them. Their infrastructure and environment doesn’t allow me to build on the skills I want to build on. I have my own environment. I want to take advantage of it.
- Allow me to maintain and build upon my UI design chops. I’ve always loved the UI design aspect of building web apps. There’s a few things you can do with Google, but I felt too constrained by their system, and wanted, again, things the way I wanted them.
It wasn’t all bad, though. Setting it up in GoogleDocs did effectively (if not intentionally) serve as a sort of rough draft for rebuilding it on my own server.
What it Did Do
- Gave me a firm sense of what I wanted, and what I didn’t want, if not a solid workflow to follow.
- Gave me a good sense of the basic information I wanted to collect, which was then easy to translate to MySQL tables.
- The visualizations in the form of charts and graphs were good enough that I decided I couldn’t do without them, which gave me the desire to research PHP graphing/charting libraries. I settled on pChart.
Now, I have a small web app that lets me…
- Track the gas mileage I’m getting for my car and how many miles I’m getting per tank.
- Track how the price of gas is changing. Always up, but still…
- Track how much I’m spending per mile on gas.
- Track how often I fill the tank.
- Keep tabs on how often I’ve changed the oil (I change it myself… a pox on paying someone else to do it), and how long until I need to change it again.
Soon, it will let me…
- Track how many miles I drive over time, per week, month, year, etc.
- Track how long between major service visits I don’t care to do (tires, brakes, etc.)
- Implement multiple vehicles in the hopes that I can get my wife on board and using it.
- Authenticate access so I don’t have to worry about the data being mangled by miscreants and malcontents. Obfuscation and low-profile domains will only work for so long (stay away, Noah! :) )…
- Better UI flow… it’s good enough for me, but it’s rough. It’s fine on the desktop, but I need to clean it up so it works on the mobile platforms better.
Here are the charts I’m generating. I know, the gas mileage isn’t that hot. But the payments are $0, so there’s that.
rc.d != magick
There are certain things over which I want tight-fisted control, and other things over which I want neither control nor intimate knowledge.
When it comes to keeping my FreeBSD systems up to date, I relinquish control for the most part and let the ports system do the work, with portmaster and portaudit running the show for me. Sure, I run them manually when I need to based on nightly portsnap runs, with a close eye on what’s going on, but I let them take the reins of upgrading and auditing.
But when it comes to Apache and supporting modules (php, mod_perl), I want to do things the way I want to do them, not the way the ports system wants to do them. I want to compile them myself, with the options I want, and put the whole thing where I want it. I’m sure the ports system allows for that, but I’ve not dug in deep enough to figure it out yet.
That’s worked for me. For the most part.
The one part that hasn’t worked for me has been getting Apache to start at system startup as part of the rc.d framework. That is, until today. I finally hunkered down and figured it out. It’s nowhere near as magickal or mysterious as I initially thought.
Here’s what’s in my rc.conf file:
Here’s what’s in my /usr/local/etc/rc.d/httpd startup script:
1
It took a few reboots of my dev server to get it right (that’s one reason one has a dev server), but it’s working like a champ now, and I don’t have to worry about manually starting Apache anymore.
That said, I’m sure there are improvements that could be made, and I welcome any suggestions (do I need all those requires? I don’t know… but it works to wait for them).
To make things a little easier on me I’ve set up the following aliases (and these date back many many years when I was fussing with Apache and conf files on an hourly basis):
1
Somewhere in my archives, I have an httpd.conf file written entirely in perl… maybe I’ll dig that out some day, just to see it again…
But first, I want to get all this working in jails.
Near Native FreeBSD Full and Incremental Backups to a Removable USB Storage Drive
UPDATE 2011/03/09 – I updated the code to backup to an NFS mount, and to include the “-h 0″ flag to skip all nodump flags. That was causing me serious problems.
Summary
I’ve given quite a bit of thought to backup procedures at home since my FreeBSD 8.1 box dropped my mirrored filesystem. The signs of impending apocalypse were there, I just didn’t pay them proper heed. Fortunately, all of my data was salvaged; unfortunately, I lost all the custom PHP code I wrote over the last 6 months, my wordpress themes, plugins and modifications, and everything else that actually DID anything with all that data. So, while I’ve been rewriting that, I’ve been giving equal, if not more attention to backing it up. I’ll catch up again, but before I do that, I’ll make sure I won’t fall behind again.
I did a few searches for FreeBSD backup solutions, and rolled my own little backup script using dump. It was decent, but it didn’t do everything I wanted as well as I wanted it to. Every night was a full backup, and there were no incrementals. I had to implement some pretty inelegant code to accomplish a couple things simply b/c I didn’t know how else to do it. So I kept looking and eventually zeroed in on David Andrzejewski’s work. He clearly states what he put out there is a use-at-your-own risk kind of script. I took it anyway as a starting block, and fleshed it out for my own purposes.
My requirements were similar to his, with the exception that I don’t have a cloud based storage account at the time of this writing and instead will be using a removable USB connected storage drive.
Project Goals
- Run with native or easily accessible tools.
- Full off-system backup of entire system once a week.
- Incremental off-system backup of entire system nightly.
- Separate off-system backups of individual critical files to make future restores easier.
Future Goals
- Play with ${DUMPCACHE} to see how it affects the time to execute in my environment. Drop it back to 8MB for a week. Ramp it up to 64MB for a week. Recommended is 32MB, but it’s a party! Let’s see what happens.
- Continue monitoring and fine tuning the hardware, OS environment and script to ensure maximum performance and stability. I haven’t recompiled a kernel in a while, maybe I’ll see about that.
- These are “as money allows” goals. I’m sure my wife is getting tired of me spending money on hardware. Then again, she does appreciate that I have a hobby that keeps me off the streets and out of the brothels.
- Continue looking for consumer level, but sufficiently robust NAS solutions featuring RAID5 mirroring and access via secure and/or open protocols (ssh, smb, rsync, etc.) to replace (or augment) the removable drives I’m using now. No Windows-Only solutions please.
- Evaluate cloud based storage for off-site backups. I’m looking at SpiderOak right now at the recommendation of a friend. I like their zero-knowledge solution and pricing, but more research is required. We’ll have upwards of 500GB of storage requirements, so we’ll have to weigh the monetary costs of cloud storage and bandwidth usage carefully against the risk of my solution failing when (!if) I need to restore. For the moment, I’m relatively comfortable with dumping the filesystems to removable drives, and keeping certain ultra-critical bits of recovery text (bsdlabels, fstabs, choice config files, etc.) in Google Docs.
My Environment
Two physically identical servers built from the ground up running FreeBSD 8.1. Each system houses a 150GB system drive (/dev/ad4s1) and a 500GB data/storage drive (/dev/ad6s1), and runs with 2GB of RAM.
I have /, /usr and /var mounted individually on the 150GB drive, and /home (containing /users and /www) mounted on the 500GB drive. I thought about getting separate drives for /www and /home, but decided I didn’t want to deal with planning for storage allocation. Instead I created /home/www for web files, and /home/users for user accounts. It’s not exactly standard, but it’s not unprecedented, and I make it work.
Off my “production” server I’ve hung a 2.5″ 320GB USB2.0 removable drive. Off my “development” server I’ve hung a 2.5″ 100GB USB2.0 removable drive. I’ll adjust the size of the drives as needed. That’s just what I had on hand. Both were UFS formatted using fdisk.
During the backup job, those drives are mounted at /backup. The rest of the time they’re plugged in, but not mounted.
The Script
You’re welcome to this, but be warned, if it borks up your machine, destroys your pr0n collection, or sends terrifying space monkeys into your engine room, don’t blame me. Use at your own risk. There, now that I’m all disclaimed…
1
The Crontab
Here’s how I’ve set up my crontab. Like Mr. Andrzejewski, I opted to keep the specifics regarding the type of backup and the day it’s run in cron, rather than build it into the script. While it does make for a slightly longer crontab, it simplifies the logic in the script considerably. At the end of the day, I just feel better about telling the script what kind of backup to run (full or incremental), and the weekday name to embed in the resulting filenames, rather than letting it determine it itself. It’s a control thing.
1
This will finally result in a list of files looking something like this come Sunday morning. Sort to taste.
backup/Sat.usr.level1.dump backup/Sat.var.level1.dump backup/Sat.root.level1.dump backup/Sat.fstab.txt backup/Sat.bsdlabel_ad6s1.txt backup/Sat.bsdlabel_ad4s1.txt backup/Sat.root_crontab.txt backup/Fri.home.level1.dump backup/Fri.usr.level1.dump backup/Fri.var.level1.dump backup/Fri.root.level1.dump backup/Fri.fstab.txt backup/Fri.bsdlabel_ad6s1.txt backup/Fri.bsdlabel_ad4s1.txt backup/Fri.root_crontab.txt backup/Thu.home.level1.dump backup/Thu.usr.level1.dump backup/Thu.var.level1.dump backup/Thu.root.level1.dump backup/Thu.fstab.txt backup/Thu.bsdlabel_ad6s1.txt backup/Thu.bsdlabel_ad4s1.txt backup/Thu.root_crontab.txt backup/Wed.home.level1.dump backup/Wed.usr.level1.dump backup/Wed.var.level1.dump backup/Wed.root.level1.dump backup/Wed.fstab.txt backup/Wed.bsdlabel_ad6s1.txt backup/Wed.bsdlabel_ad4s1.txt backup/Wed.root_crontab.txt backup/Tue.home.level1.dump backup/Tue.usr.level1.dump backup/Tue.var.level1.dump backup/Tue.root.level1.dump backup/Tue.fstab.txt backup/Tue.bsdlabel_ad6s1.txt backup/Tue.bsdlabel_ad4s1.txt backup/Tue.root_crontab.txt backup/Mon.home.level1.dump backup/Mon.usr.level1.dump backup/Mon.var.level1.dump backup/Mon.root.level1.dump backup/Mon.fstab.txt backup/Mon.bsdlabel_ad6s1.txt backup/Mon.bsdlabel_ad4s1.txt backup/Mon.root_crontab.txt backup/Sun.home.level0.dump backup/Sun.usr.level0.dump backup/Sun.var.level0.dump backup/Sun.root.level1.dump backup/Sun.fstab.txt backup/Sun.bsdlabel_ad6s1.txt backup/Sun.bsdlabel_ad4s1.txt backup/Sun.root_crontab.txt
Viva la Vino!
Over the weekend (and one of the reasons I didn’t get to post about Saturday’s ride until today) I rolled a quick wine database app for my wife and I. I’d been toying with the idea in the *very* back of my mind for a little while, and when Jami said “That would be really cool!” Saturday night, I understood how I would spend my Sunday.
It’s sad, really, how much PHP I’d forgotten. I used it about 10 years ago to write a quick and dirty rough draft for a mod_perl driven LAN party organizing site that helped us, get this, organize LAN parties. The people involved in those awesome times were spread between Lawrence, KS and Kansas City, and we wanted an online RSVP system for our weekend frag-fests. Before it fizzled out due to people moving away, moving on, and generally growing out of it, it was a very nice community blog with event organization as it’s core. Hewn from scratch, it was also a great vehicle of personal growth in all things web design for yours truly.
The rough draft for that site was the last time I’d used PHP. It’s amazing how much one forgets. In building this little ditty, I had to relearn such language specific basics as assignments and conditionals, much less deeply nested hashes and objects (all arrays in PHP are, apparently, associative – who knew?). Most of the concepts are fresh from my work (at work) with Powershell, but the syntax, naturally, differs.
The engine, if I can be so bold, is still rough. In fact, it’s a tangled mess of burnt spaghetti code sticking to the bottom of the hard drive that doesn’t deserve the name “code”, but it works.
Our requirements are pretty simple:
- Must work easily in mobile browser for updating while we’re out with nothing but our smart phones.
- Must have rudimentary rating system so we know which wines we’ve liked, and which we haven’t.
- Wines must each be sufficiently described so we know what they are:
- Vintage (year)
- Varietal(s) (grapes)
- Winery (producer)
- Region (geographic location of winery)
- My Wife’s Opinion
- My Opinion
- Comments (general notes to jog our memories)
- Must be able to search on a variety of fields seamlessly and simply.
And that’s pretty much it. Everything else is gravy.
While it’s fully functioning, and in working order now, like I said, it’s a hideous ugly mess behind the scenes. From here, I want to clean it up, separate logic from presentation as much as feasible, utilize more fully objects and classes, incorporate more best practices, and generally do what I can with what it is to learn as much as I can.
It’s really been a lot of fun!
