Tuesday, June 08, 2010

Microsoft TechNet benefits

Microsoft TechNet subscription is a very handy tool for Windows based IT admins, or people breaking into the IT world. Let me first talk about my entry to the IT world and how TechNet would have helped me.

When I first got started with IT I was already involved with multimedia applications so I knew heavily about the problems of trial software ending before you really got a chance to play with it. The first book I bought was Windows NT Workstation for Dummies, came without Windows NT so I had to get my first real hands on by using an old Windows NT workstation at work. When I enrolled in the Microsoft classes at college the books came with Windows NT Server and Workstation demo keys, they were good for about 60 days.

Back then, it was difficult to find any other software besides the OS and my learning skills were limited to what software was included in the learning books, typically it was only the OS. Not to mention that downloading software on a 56K modem was not the fastest solution. It was after landing my first IT job that I started to work with applications like Microsoft Exchange and other difficult to play with applications.

Now I know people will say "those are available every where" but take for instance a few facts.

1. I didn't know anyone in the IT world.
2. Didn't want to bootleg a CD.
3. Limited to slow download speeds.

It's tough to get the applications to play with.

Now with faster Internet speeds and networking with friends there's many resources but I feel that going with a TechNet account still the best choice. Here's why.

1. Licenses do not expire! - The best thing I like about TechNet is you are given a license key that does not expire after a certain amount of time, now some of the beta software does end but the rest is just like buying a retail license.

2. Easy to find and download any Microsoft IT based application or server - This is a great reason to down load Exchange 2010 and get it working at home over the weekend. Nothing tells you more than getting it work on your own in your personal home lab.

3. Two free incident calls - Each Microsoft incident call is $265, so that is $530 dollars alone in support calls for a price of $300 (regular price, often cheaper with coupons) subscription price.

4. The extras - Microsoft eLearning, On-line chat, etc. I personally do not like the eLearning and just prefer watching simple videos but it's a nice benefit. I have not tested the on-line chat yet but honestly sounds good and I do have some low level nagging questions I haven't been able to debug yet.

The best part of having the applications available is the testing and since it's so easy to access makes you want to try them all out. I can say that even from my previous job I had a hard time accessing some applications even for work testing and learning. Either the media could not be found or the license manager did not have the keys. Here I am the manager and able to install and test all of the Microsoft server and business applications.

I highly recommend this for people just getting into IT. Where in the Linux world many of the applications are free and open source the Windows world is less open source and based upon Microsoft applications. If you have a robust workstation at home, with a good amount of physical member and VMware Workstation you will be set to do almost every possible IT server environment except SAN simulations.

Where to find it?

http://technet.microsoft.com/en-us/subscriptions/bb892759.aspx

Friday, June 04, 2010

Mysterious Windows lockout

"Hello, my account keeps getting locked out"

Anyone who has admin for a Windows environment knows about accounts getting locked out. Typically when a user changes his or her password there are certain applications or processes that need to be manually updated with the new password. If you don't update these passwords, then each time they run, they will authentic your old password, which in turn will fail and register on the Active Directory as 1 failed password attempt.

If you think of a few more services and some mapped drives all trying to re-authentic at the same time, then you can easily be locked out if your Active Directory password policy is set to 5 failed passwords before account lockout. This problem while simple is often difficult to really track down where the problems are coming from.

I'm going to list the tools I use and a recent account lock out issue that I resolved. Please note I removed all of the details for security reasons.

What to look for

Before you make any changes, always gather the information of the user affected and what type of job role this user does. A data entry user will be much easier to resolve than someone who is an admin logging into many remote systems. Here's a listing of questions I usually ask.

1. When was the last time you changed your password?
2. Which account are you using?
3. Where do you login from, only this workstation or multiple computers?
4. Do you work from home on your personal computer?
5. Any special application or service you are using?

From here I can usually get an idea of the level for the user, let's say the user answered the following questions.

1. Two weeks ago, problem started today.
2. Jsmith
3. Only login from this laptop.
4. No, but I work from home on this laptop using VPN.
5. Google desktop search but nothing else.

Now let's check the user's desktop for any lock out messages. You can somewhat skip this part because the lock outs are from the Domain Controller but I like to just make sure I do this. As I checked the system, there is no indication of any security errors or lockouts. Strange, usually there's at least something to go on.

Ok, let's move to the Domain Controller. Checking the security logs I filter by the user name and again there is nothing listed for failed logins. I'm starting to wonder where is the lockouts coming from. The domain I am working on actually logs failed and successful from a group policy so with the amount of logs it could have been over written.

Time to install some tools

Before this I already installed the Windows Account Lockout and Management Toolset from Microsoft. I used the tools from our local Domain Controller and scanned the problem account with the LockOutStatus tool.

LockOutStatus shows a listing of the current status of the target account with details as where the lock outs are taking place and also what time they occur. Here's a screen shot of the tool and why each function is useful.

LockOutStatus

DC name - Shows which server the account is accessing
Site - Where the servers are located in the AD forest
User state - Is the account locked or unlocked
Bad pwd count - How many bad passwords were attempted
Last bad pwd - This is how long ago the last bad password was attempted
Pwd last set - When the password was last set
Lockout time - The time the accounted was locked out

What to look for?

So what I'm going to look for is this. First, I'm going to run the tool against the problem account, and the tool will automatically find all of the domain controllers. Here's what I look for in each value.

DC name - This is just the names of the DC, pair it up with the Site names if you are not sure where the servers are physically located.

Site - If you already know the DC name this is a bit non-important, unless you are not familiar with the DC's or do not know where a DC physical sits.

User state - This is handy to show the "wave" of lockouts getting updated as the account is locked out from it's primary DC. I alway focus on the DC that is first locked out, then if I can't find anything there move to the next closest DC.

Bad pwd count - This one is a bit tricky. In most cases you would see the number corresponding to your domain policy about bad password attempts. For example if your password count is 5 typically you will see "5" under this field. If you see 5 that would indicate that indeed there is an rogue application or process using an old password. Sometimes this does not mean that there were 5 attempts, but could mean that the DC saw 5 attempts. If you have some delay on your domain, 1 password attempt could register as 2 because of the delay between DC's.

Last bad pwd - This shows just when the account was locked out, for example if you know a user is working 8am to 5pm and the account lockout time shows 3am, then is must be a process running. If the user takes his or her laptop home and powers down at night, there might be a process running on a remote server.

Pwd last set - Of all of the values, I hold this second most important. Many times end users will change their passwords and that's when the problems will start. So I like to see the distance between when the password was set and when the password started to locking out. In most cases it's going to be very close, usually within a week from the password was last set. In some cases, it could be much longer because a service or application is not used every day.

Lockout time - Lockout time is when the DC reports the amount of failed password attempts to match the domain policy. It's a bit confusing and I'm not 100% but if an account is running on a service, and that service runs every 1 hour, it will register 1 lockout. But if the domain policy is set for 5 lockouts in one hour, this will not cause the account to be locked out.

So in this example, let's say we found the following values. We caught it just as the user's account was locked out so only one DC is showing locked.

DC name - DCSF
Site - San Francisco
User state - Locked
Bad pwd count - 5
Last bad pwd - 9:30AM
Pwd last set - 5/20/10 (two weeks ago)
Lockout time - 9:32AM

Ok, from this data I know the following. First, the account is contacting the DC in San Francisco, it's locking out from 5 attempts (5 is the setting in GPO), and it's locking out at 9:30AM. The user starts work at 9:15AM but he says the account is locked out before he can login to the workstation. This might be user error but let's dig further!

Checking the DC's security logs

Now we have a starting point, DCSF. From DCSF I filter out the logs and find this.

Lockout event

Here's the important information (removed real info).


Source: Security
Category: Account Management
Type: Success A
Event ID: 644
Computer: DCSF
User Account Locked Out
Target Account Name: jsmith
Target Account ID: BayareaCo\jsmith
Caller Machine Name: sfauth
Caller User Name: DCSF$
Caller Domain: BayareaCo


Now this is important, the value for "Caller Machine Name" is pointing to "sfauth". I double checked and the user does not access any system besides his own. I checked the event logs of "sfauth" and found this.


Source: Security
Category: Logon/Logoff
Event ID: 539
Computer: sfauth
Reason: Account locked out
User Name: jsmith
Domain: BayareaCo
Logon Type: 3
Logon Process: CISCO
Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGES_V1_0
Workstation Name: sfauth


We found the server, now what application is causing this

We now are focusing on the server sfauth since this last event log points to an process called "CISCO". Let's first see what is running on the sfauth. I'm going to use a handy tool from Sysinternals called TCPView. This shows me what processes and who they are talking to, as seen below.

TCP View

So we found that the process is running but let's look at the process details. From TCPView, it looks like a process called "CSAuth.exe" is running on the server talking to DCSF. I'm going to assume that since it's talking to DCSF, it must be doing some kinda of authentication for remote access. I used another Sysinternals tool called Process Explorer.

CSADMIN

We found the application, now what?

Ok, so we found that it's an Cisco application called Cisco Secure ACS, but what do we do now? I snoped around and found that the Cisco application had a few GUI tools and a nice logging report tool. I opened the repoting tool and viewed the failed logins.

Below is one of the failed logins reports for user "jsmith".

Cisco Login

From here we have something to follow, a MAC address and an IP address. While the IP address can change, the MAC address is hard coded and unique for each network device. I first started by checking DHCP for where this IP address was located, it was not on the range of IP addresses. I used my favorite tool NMAP to do a port scan of the IP address listed in the failed event.

Below is the result, but there's something wrong with this picture, can you find it?

NMAP

The MAC address listed here shows 00:13:19 while the logs shows 00:26:08, that's strange! Well if this device is a router then something else is accessing the network beyond what I could see. I did a quick search on a MAC address search tool and found this result.

Apple

Apple? We don't have any Apple devices here at work. Or do we? What is made by Apple and often times accessing the network AND would be accessing the network BEFORE the user would login to his computer?

iPhone

Yes, the iPhone. The user has an iPhone which was accessing the WiFi network. In order to access the WiFi network you need to login with your AD account and this is held on the iPhone. Here's is the login screen for the WiFi access.

WiFi

Solution and closure

Once I removed this from the user's phone access from the WiFi, the lockouts stopped. When I tried to recreate this problem I couldn't seem to get my iPhone to lockout but the end user did say he had a sync software running on his iPhone with his desktop. I'm glad I found the problem and resolved it. This by far was the most difficult account lockout I have battled.

I hope this will be helpful for you!

Wednesday, June 02, 2010

What version of Microsoft applications am I running?

I found this link to be very helpful, especially if you're throw in to support an server or app that you don't know much about.

http://www.microsoft.com/security/updates/checkversion.aspx

Tuesday, June 01, 2010

Mounting CD-ROM in Ubuntu

I'm somewhat forgetful when I don't use a command every day. While trying to remember the exact command here I did a Google search and each answer I found was missing some step. So I am going to write out the exact steps here for anyone getting lost, and also for my self in the future. :)

This is from Ubuntu so your distribution might be slightly different.

1) sudo mkdir /mnt/cdrom

2) sudo mount /dev/cdrom /mnt/cdrom

3) ls /mnt/cdrom

4) Done!

It's really simple but from searching there were lots of very complex answers, and I prefer the simple as possible method.

Enjoy!

Monday, May 17, 2010

The real problem is not always obvious.

Recently at work I helped out with a project to move the IP address of a slave name server. After this move went with no problem the main master name server crashed, or appeared to be down. I wrote up a report on this outage and the steps how I found the real hidden problem using simple troubleshooting steps. Please note I changed the IP addresses and domain names for privacy.

Brief history of the environment

The San Francisco site has two name servers, dns1 (10.10.10.1) and dns2 (10.10.10.2), both running a flavor of RHEL and Bind version 9. The servers manage the child.parent.com domain and are a slave servers for the parent.com domain managed in Tokyo. In the days leading up to the outage, we added a Nagios client for both servers, and changed the IP address of dns2. The IP address was changed on a Wednesday from 10.10.10.2 to 20.20.20.2.

First problem appears

On Monday, the users alerted us around 6:00am that our VPN site was down. At first this appears to be an network problem but users also found that the external sites were also down. At this point the network engineer verify that the network was up, and all sites were accessible using IP address. The sites are not resolving by name only by IP address from external test sites as well. In fact no external names of the child.parent.com domain could be resolved.

Trying to check the dns1 server, I could not login using SSH and walked to the data center to the console. There I found that the server was filled with messages of the following.



At first I could not access the command prompt, so I restarted the server using the power switch but forgot to record the exact information displayed on the screen. The actual screenshot above was take later. Once the server was restarted, all of the external sites were able to resolve the names again and every thing is back to normal. All 5 external DNS servers are now resolving all of the external sites, we tested this using tools from DNS stuff.

Problem appeared to be solved?

At first we believed the problem was resolved and just a random server crash, but seems too much coincidence to just crash 2 days after the IP address change? I had a good feeling that an IP configuration change would not cause the problem but didn't feel confident that the server was 100% stable.

The first question was why did all 5 external DNS servers fail to resolve any name under the child.parent.com? First this was answered by looking at the dns1 server's Bind named.conf file as shown below. The server dns1 is the master server for child.parent.com, and the setting "expire" is configured for 2 days. This means that if dns1 is down for two days or more, all of the slave servers will erase the child.parent.com domain zone from their records. The value here is 172,800 which is in seconds.



So now we know the configuration of Bind was setup to erase the slave servers after 2 days of not reporting back to the slaves, this must mean the server failed sometime at least 2 days before Monday, most likely Friday night.

While viewing the syslog of dns1 I found numerous SSH login attempts but nothing that specifically reported errors with the system. During the team meeting it was determined by management since the last change was the IP address change of dns2 from 10.10.10.2 to 20.20.20.2 but I didn't think this change would cause another server to crash.

As the day was closing out, I started the research of the logs and delayed the IP change back until the next day when I had everything recorded from the day's problem.

The problem appears again

That night around 12:30am I was working on the report, I had a VPN connection to the office from home, with a SSH connection to dns1 reading the logs on the server and writing the outage documentation. While working on dns1, my connection dropped suddenly. I knew there was a timeout for SSH connections but when I tried to reconnect I could not, also the server was not replying to any ping either. At this point I knew I caught the problem but I also had to drive into work. lol Figured if I didn't go in I wouldn't see exactly what was on the console.

At work, I found the following error message. It's the same as before but I forgot to write it down before, so I took a photo using my phone. Also this time I was able to control-c out the message and access the console.



Ok, now we have a major clue to get started. I started by searching Google.com for the phrase "ip_conntrack table full dropping packet" which came back with many results. The information came back that ip_conntrack was a process of IPTABLES, the internal firewall of a Linux based system. The error message is the server getting overloaded trying to process too many IP packets.

From recommendations I found that the sysctl.conf file can be adjusted to suit the server and tuned for it's physical memory size. The dns1 server has 1GB of physical memory and the currently using the default setting of 65,528 connections. The error message indicated all 65,528 connections were filled so I increased the value to 98,000 and I also shorten the timeout value from 432,000 to 28,800 so connections will disconnect faster. After making the changes I restarted the networking services, and then checked the settings using the following command.

/sbin/sysctl -a | grep conntrack

Right after I checked the settings, I found all of the new entries were valid and working, so I wanted to see how many free ports were available. I used the following command to find out.

wc -l /proc/net/ip_conntrack

I was surprised to find out that all 98,000 ports were now taken up again just seconds after I added them! At this point I knew there had to be something causing extremely heavy traffic on the server. I felt it was a denial of service attack but had no proof. So I began to look at what is the server doing locally. First I did the simple command to find out the top processes and general overview of the server.

top



Ahh, now we see that there are multiple processes called "ssh-scan" running on the server. Now I wanted to know how many were running. I ran the following command.

ps aux | grep ssh-scan

The result was a listing of over 200 processes of ssh-scan running on the server. At this point I knew that it was highly unlikely that any member of the IT staff would be scanning servers so I was almost certain it was done maliciously. It's important to note that this tool ssh-scan is not like NMAP, where it has legitimate uses, this ssh-scan is just a brute force scanner.

Since I knew the purpose of the tool and what it does I wanted to see dns1's network activity. Running the following command showed me the results.

netstat -n



Looks like it is trying to connect to random servers on the Internet via SSH. Now I wanted to see just activity on port 22, SSH. So I used the following command.

/usr/sbin/lsof -w -n -I tcp:22

ssh-scan 4843 root 7u IPv4 424670 TCP 10.10.10.1:33835->115.238.100.174:ssh (ESTABLISHED)
ssh-scan 4846 root 7u IPv4 424767 TCP 10.10.10.1:43949->115.238.100.190:ssh (SYN_SENT)

At this point, I was almost 100% certain that there was either a root kit or some malicious process installed on dns1. I wanted to figure out where it was located but my attempts to locate it were not successful. I then started to search Google again for the solution when I stumbled across a posting how to run an ssh scanner. At the following site, the poster showed basic steps to deploy and setup a remote SSH scanner on a compromised server, basically what I have in front of me. Ironically his detailed steps gave me the biggest clues how to find the application.

Here's the link to the forum posting.
http://www.governmentsecurity.org/forum/index.php?showtopic=11026

Then the important clue from the posting.

then type the following
cd /usr/man/man3/
and then :
mkdir ". hiden"
and then :
cd "..."
This is an hidden dir so the Sysop wont notice

Using the clue that was posted, I guessed the ssh scanning application would be installed somewhere under /usr. I checked around but didn't find anything until I stumbled across /usr/tmp. It was just a hunch but I was thinking where would you hide an important file? In the temp directory. Also I also thought of the movie Hackers and how the file was placed in a garbage directory to be out of sight.

I then ran the command to view hidden directories.

ls -a



Ahh, now we found something, and there's a directory called "ssh-scan"! So what is this application really doing?



Bios.txt - Very long listing of IP addresses
Nfu.txt - Very long listing of random IP addresses
Pass_file - Dictionary attack file
Spd - Script
Vuln.txt - Appears to hold account names, passwords and IP addresses of cracked systems.

The files and directory were deleted but now wanted to find out how this security breech really happened.

Resolving the security issue

I knew that the root account does not have SSH access, so I had to find where else the user got access to the server dns1. From the /var/log/messages I found many entries of "failed password" from various IP addresses from every possible account name from "root" to random names like "paulsmith". I exported the logs and searched the successful logins, here I found there were successful logins from two accounts, "siteadmin" and another "service". Note, the name service is not the actual name.

While looking at the two account, "siteadmin" is the account we use to login here locally, and then SU to root. The account was accessed all from the internal IP address so we knew that this account was not used in the security breech. But the other account "service" was logged in from various IP addresses that we did not own or knew about. The account "service" was created for the local service agent, to monitor the hardware on the server. The problem was how they access the service account and why is anyone logging in from the service account, which should not have remote SSH access.

I removed the access for the problem account and removed login rights by changing the passwd file and giving the service account "nologin". Changed the root and siteadmin passwords, also disable SSH since no one really needed it, the server is just a few feet inside the data center. Digging deeper, we found that the service account was added to the "wheel" group, giving the account SSH remote login access, and that the account did not follow our normal password security standards.

Also to compound the issue, the external firewall was allowing SSH to the servers on the DMZ, which was not known to the other users.

Why and how could this happen?

It's very easy to assume that everything is done normally and there shouldn't be any problems. The biggest problem is that unless you installed the server or deployed the firewall rules, you never really know for sure. I've experienced this many times and at different job sites, it's always eye opening.

A few of the problems we faced were different departments working on the server at the same time. Basically when one group of people access the system, they might not know about some details the other team would know about, for example the server allowing SSH access from the Internet. Another issue was the lack of details everyone knew about the security.

For our department we created the service account and the generic password on the request of another team, but was not added to any SSH access group. Another department might have been troubleshooting the server for another issue and figured they add service account to SSH access, to make troubleshooting easier. Finally since the SSH is open to the outside world, malicious users could port scan trying to get in. The IPS devices were not monitoring for this type of attack so it was not picked up. Then just by the amount of tried, eventual the attacker would find a password that worked and gain access to the server, deploy the SSH scanning tool and start maxing out our server's resources.

It's a bit irony that since the name server was so old, it alerted us to the outage since even a little bit of scanning was overloading the server to the point it would drop off the network.

What to do in the future?

Monitor your logs! I saw the SSH attempts earlier but didn't think that someone would add a very easy to crack password to the SSH access group. Now since I know the dangers of SSH, I disabled it only to turn on the service when needed. Also check the passwords on your system, and make sure who has what access, don't assume everything is the same.

While it was a tough problem to solve, especially since I don't have much Linux administration experience, it was very interesting and using my experience from Windows administration it was a matter of knowing that correct commands. I knew that something was running, but what? Searched for the high process application, then once it was found, needed to know what it was doing. After that how did they gain access, from there I searched the logs and found that IP addresses for one account did not match the other, which was a clue where the login were coming from.

I intend to really learn more about Linux administration, and if this happens again, resolve the problem much quicker. So far, it's a welcome intro to security problems with Linux.

Wednesday, April 07, 2010

Tool review: NMAP

Nmap has been around for a few years. It's a very powerful scanning tool used to check servers or network devices for any open ports and information. You can use this for auditing or security testing. I was first introduced to Nmap while at work, auditing Windows servers.

I found the ability to check a server for open ports handy but not sure where this would be helpful for besides running security scans. It wasn't until much later when troubleshooting applications that I found the port viewing to be extremely helpful to debug a server if it's actually listening on a port or not. Also it was helpful to determine if the local firewall was causing any issues on the server, blocking an application.

Now there are other tools that offer (somewhat) similar scanning but before Nmap I used Sysinternals' TCPView for finding out about a system's ports. The problem is this is a view is not what other systems see, it's from the inside. So Nmap offers a much better real world view of a system.

Another point where Nmap is very helpful is testing connectivity between servers or applications. I recently install Webmin on a server (great tool) but the default port 10000 was not working. Using Nmap I was able to tell that 10000 was being blocked somewhere on the network because it was not blocked on the server's firewall. This is a handy tool when you don't have access to hardware firewalls but want to have at least some evidence that you may be on the right track.

The latest version of Nmap is now a GUI version and it's very easy to run. Honestly I liked the command line version of before but this new version really makes it easy for anyone to use.

Check it out at, available for Windows and other OS.

http://nmap.org/
Thinking too hard for the solution

At work we've been rolling out Nagios agents on the servers, it's simple enough on the Linux and Windows. But the problem was a few servers we had running in the DMZ, an isolated network where their required greater security. A problem started up where certain ports were not working, the Nagios ports, but all that was working was 22, standard SSH port.

Trying to figure out this problem I looked through the system, thinking it might be an internal firewall rule, stopped iptables which no change or result. It wasn't until I used NMAP to scan the servers that I realized all of the ports were closed, expect 22.

At this point I asked the network engineer and he confirmed that yes, there were excessive ports being blocked on the DMZ network.

This brings a very important point, how to troubleshoot and gather information.

First, when dealing with any new problem, alway understand what is your environment. I can't stress this enough. I once helped a friend over the phone with a network problem at a large grade school where she was deploying a network device. The problem was she couldn't get terminal access to the device that she installed on the network. So we ran through the steps, the IP address she was given to use, ping commands, etc.

The problem turned out to be the school administrator gave her a duplicate IP address, and while she could ping the address (which was not her device), she could not terminal into the machine. She assumed that since it was the IP address given to her, it must be working, but in the end it was not. It's important to always double check and know the environment.

This can also be not only trusting the information you are given but knowing what to check when something is wrong. Another example is a problem I experienced once while changing an server's IP address. I followed the company standards for moving an Windows IP address from one subnet to another, also I worked closely with our network engineer who had access to the switch I was connecting to.

After making the IP address change, I could not get access to the network. I saw that I had link lights on the server, but my server could not access the network or reply from a ping. I checked all of the cables, even plugged into another port on the patch panel, nothing. I asked the network engineer three times if the network ports were changed to the right subnet, each time he checked and confirmed. Finally I asked my manager, who still had switch access, if there was some problem I couldn't find. He logged into the switch, and found it was set to the wrong subnet.

The problem in this case is I assumed the network engineer who checked three times, actually checked his work. From this case, it was much harder to check the work since I did not have access, but since I checked all possible connections and problems on the servers, I could confidently say it was not an OS issue. This is important to know when a problem appears and who 's side should fix it. Often it's going to be a battle back and forth on where the problem lays.

In larger companies resolving issues becomes difficult, sometimes the department you work with on a project may be half way across the world. In my work, many of the co-workers are on not local and I have limited access to remote servers. It's difficult but I still use the same skills to know when a problem is from our side or theirs, I inspect the environment and then apply the same knowledge to figure out where is the problem.

In any work you need to know how to fix something, it may be a broken computer to a issue on a project. They are very different but require similar skills, the knowledge of the environment to make a decision.

Monday, April 05, 2010

Installing Squid proxy on Ubuntu 9.10 server

Recently at work we have been battling over the issue of controlling the Internet access for our users. The Internet is basically open to everything and of course in a business environment there's some abuse of the access. Taking a snap shot of the usage using WireShark connected to the core switch, allowing port mirroring we found that Youtube.com accounted for the majority of Internet traffic. Also we found that less than 4% of network traffic was internally routed, this could mean that we had many users accessing the other offices or way too many Internet browsing.

Since Youtube was our biggest website and it was not business related, we got the ok to block this site. But ran into our first problem. If you check the domain name Youtube from any nslookup utility you will find that Youtube IP address spans multiple addresses, and some cross over Google.com and Gmail.com. While we got Youtube blocked, it was slightly difficult and since it was done on the firewall, is not the recommended method.

So, looking around the Open Source world, there might been another solution.

Squid is a well know proxy for Linux that is easy to configure but appears a bit difficult at first. I was really overwhelmed looking at the config file but here I'll show you how easy it is to get running with very little work.

First, I'm going to give the step by step using Ubuntu Server 9.10, which I think is pretty easy to get running. We'll use the packages install which save some time and install any other requirements at the same time.

First part is having a Ubuntu server running, this is the easy part, just takes some time. Remember to have OpenSSH running so we can remote console into the server. Also since this is a server we will make it configure with a static IP address.

1) First, make sure your server is updated.

Run the following

sudo apt-get update (this is fast)

sudo apt-get upgrade (takes about 5 minutes depending on your Internet connection)

2) Then we will set up the IP address with a static entry. First, let's find out which network we need to configure.

Run ifconfig and take note of the network card that reports an IP address, usually it's "eth0".

Edit the following file.

/etc/network/interfaces

Then add the following information using a editor like VI and then save your work. Remember to save!

iface eth0 inet static
address 10.10.10.10 (enter your IP address here)
netmask 255.255.255.0 (enter your subnet mask here)
gateway 10.10.10.1 (enter your gateway here)


3) Now install Squid from the package, note this will install the requirements needed for Squid as well.

sudo apt-get install squid (takes about 5 minutes)

4) Now, Squid is installed, and we'll go over the important files of Squid.

/etc/squid/squid.conf

This file holds all of the important details of Squid, it's very long but we'll only need to edit a few parts.

First, make a backup of the file

cp squid.conf squid.conf_backup

Then edit the file

vi squid.conf

5) We're looking for a few areas in the squid.conf file, so we'll use the search function of VI.

Open the file in VI

vi squid.conf

Then search for the first item "TAG: http_port"

TAG: http_port(then press enter)

For me it's at line 1022. This value changes the setting for the proxy server settings on the browser. Normally it's set to port 8080, so we will change this. Enter in the following after the "TAG: http_port"

http_port 8080

6) Now search for "visible_hostname"

/visible_hostname

This value just gives an alternative name to the server, I just made it simple and used "proxy". The line was 3399.

7) Allowing access, this is a tricky part.

Ok, now we need to know the subnet you're allowing access for the proxy access. For this example, we'll use 192.268.1.0/24 (basically addresses from 1 to 255).

Search the squid.conf file for "TAG: acl" It's about at line 425. Go a few pages down until you see some uncommented entries without a # sign and enter your details there.

For this example we'll enter the following.

acl allowhome src 192.168.1.0/24

This means allow a new network called "allowhome" from the source address of "192.168.1.0/24"

Remember the name "allowhome" we'll need this later.

Also we'll need to define the access for "allowhome".

Search for "http_access" and once you scroll down you should see a few "allow" and "deny" entries. Enter in the following at the top of the section.

http_access allow allowhome

Now save your file and exit.

Just to be sure it took effect, restart Squid.

/etc/init.d/squid restart

Now from your home computer, point your browser to your server's IP address and port 8080. You should connect as before.

Pretty simple!

Next time I'll post about working with blacklists, blocking certain ports or websites, and Webmin as a GUI tool for administration.

Rob
The pros and cons of working with Linux

Recently at work my duties with the Linux servers has been increasing. I'm really excited to work with Linux and gain experience but it's also somewhat frustrating at the same time. So coming from a Windows background I noted some things that are different about Linux, both good and bad points.

Applications need to be complied in Linux

Not all applications but a good amount need to be complied, then run some scripts to have them installed. It's not difficult but when things go wrong, takes some time to figure out, what's really causing the issue. At first this was the biggest problem I had switching over to Linux. Why would a software vendor send out applications that are not even ready to install?

After working with some problematic Windows applications I started to see the light. Many times in the Windows world, we would be required to install some applications that are questionable. Not in terms of they have malware issues but problems in how they were developed, hard coded names, etc. In one application we had many issues because the timeout of a child process was extremely short, so short that any lag in the network caused the entire application to crash. Perhaps in a Linux version we could have seen this in a config file and made the change?

Open Source in Linux application

This is more of a personal preference than anything else. I've worked mostly on commerical applicaitons, for example Microsoft Windows family, Microsoft Exchange, or any other popular applicaiton used in the business environment.

We'll see how well Open Source works for me in the future so far there are some projects it's really helping but also it's somewhat hard to get the correct support.

Friday, March 05, 2010

Testing SMTP

At work there is an issue with the SMTP server that messages are getting hung or delayed. It's alway handy to have these commands on hand to manually test SMTP and just make sure everything is up and running. The problem I'm facing is that I'm able to send the messages but they are getting lost somewhere.

Testing SMTP Server from Microsoft

Another link

Here's the commands all run from a Windows command prompt, but these can be used from a UNIX or Linux as well.

1) open command prompt

2) type "telnet"

3) type "set LocalEcho"

4) type "open 25"

5) You should see the following "servername.domainname.com ready"

6) Type "helo me"

7) You should see the following "250 OK"

8) Type "mail from:test@testdomain.com" where you can enter any address here

9) You should see the following "250 OK mail from test@testdomain.com"

10) Type "rcpt to:email@domain.com" this is who you want to send the message to

11) You should see the following "250 OK recipient email@domain.com"

12) Type "Data"

13) You should see the following "354 Send data"

14) Type "Subject:Your subject" press enter twice

15) Type "Testing" and press enter

16) Press enter, then type a (.) period, then press enter

17) Type "quit"

18) You should see the following "Mail queued for delivery"

The message will appear in the recipient's mailbox, but in my case it works the first few times then does not show up.

Monday, February 15, 2010

Train Signal CBT

Recently I purchased the Train Signal dvd for VMware vSphere. The DVD set arrived very quickly, and inside was a nice DVD case with three DVD's. It's important to note that these DVD's are not protected by any digital media rights or passwords, even any accounts. I purchased CBT videos from another major company and was very disappointed that I could not save these to my iPhone or any device that was not on the Internet.

Basically the DVD's are not standard DVD's but actually DVD media discs containing a Flash menu, with links to the AVI videos. There is also PDF documents which contains notes from the videos. In addition on the third DVD there is video formats for your portable media player with video formats of AVI, and even MP3 version files for your car. It's a really nice feature and this allows you to take full advantage of the lessons any where.

Overall the video quality is done well, I'm on video 5 of 36, with each about 15 to 30 minutes long. The instructor is very easy to understand, little jokes or side stories. I generally don't like with some other instructors, he's much more to the point, but very easy to listen to for an hour without problem.

Once I get farther in the series I'll write more about my experience and a longer review.
Back to VMware

It's been a while since I updated this page. Work and social life has been taking over also starting school this year. I've been tasked with a few projects for the current job where I'm focusing on VMware issues with ESX server. Currently there are no major problems but I wanted to have a solid plan for the future. The biggest hurdle is the smaller servers in the data center.

Many companies have small older servers in the data center for one application, typically something that needs a database or hosts a web site internally. Here at my current job location there's a few servers and they all use very low powered servers, dating at least a few years old. These are prime cadinets for physical to virtual migration. This would also help out getting away from old unsupported hardware. Who knows where to find a power supply for a off built server chassis which the company no longer is in business.

The problem going with the virtual solution is how can I keep the servers up when the ESX server it's self has a problem? I'm currently reading and studying with vSphere, hoping to have a project plan written up soon. Ideally I would like to a system planned out where I can have redundant fail over ESX servers, allowing us to make sure if one system is down does not take down the entire virtual systems.

It's going to take some planning, and understanding of the environment. I'm really aiming towards the simple method, making it the easiest to understand and manage. Considering the environment, this would save space and energy, not to mention support issues since there's just less hardware to fail.

Having been in the role where I was supporting very old hardware, some non-hot swap drives in a RAID 1 configuration, it was extremely painful to take down the server just to swap the drives. Not to say that that's IF you are alerted to the drive failure before they both went down.

Tuesday, October 27, 2009

Old tools still handy today

Today I was asked to complete a task by my co-worker, find computer accounts that have not been used in 90 days and have them disabled. Tried looking at some PowerShell scripts, but didn't want to rewrite a script or piece it together. Also since I had to leave in less than an hour I needed something fast.

Searching my all time favorite tool, I came across this helpful admin tool.

Oldcmp

Basically this is a command line tool to search and disable (or delete) computer accounts. I ran the command with the following switches.

oldcmp -report

This first gave me an idea of how many accounts where reported, with no change.

Then I ran the actual command.

oldcmp -nodc -disable -safety 500 -unsafe -forreal

After this another report is automatically created and was able to disable the computer accounts in about 1 minute.

It's decently fast and honestly, would have taken me hours to really do this without the tool. Even with a script I would still have to tested it, ran in a development environment and I often don't trust my own scripts to disable accounts. :)

Wednesday, October 21, 2009

The commands I always forget

When working on Windows I always forget a few very helpful commands.

MSCONFIG.EXE - For some reason I always forget the location of this helpful tool. Basically it's a GUI for system start ups and services but the location is a bit hard to find.

C:\WINDOWS\PCHealth\HelpCtr\Binaries

SYSTEMINFO.EXE - This is a tool to list basic informtaion of the system including install date. Often when working on an older system I like to take note how old the computer is for reference.

Monday, October 19, 2009

Certification path

After spending some time studying for the Microsoft MCITP exams I had time to really understand why their are held as a standard for system administrators. Microsoft has changed the exams to a much higher standard than before. Which makes me really wished I finished my MCSE back in NT 4.0 when I had the chance instead of now with 2008. :)

While it's hard and sometimes so boring, I do like that fact that I'm preparing for the next operating system and will be much more prepared for future roll outs. What does make it fun is playing with VMware Workstation, which I highly recommend for any system admin or person interested in computers to purchase a license. It's the best $200 I've spent on software besides my Photoshop license.

Also using the TechNet account combined with studying for the exams makes it much more fun. I can now play with any of the Microsoft catalog and try out applications before they are released to get an idea how they would work in a production environment. So far, it's brought alot of much needed excitement to a rather borning topic of Active Directory.

CompTIA

While I'm moving along with the Microsoft certification path I also looked at other certification paths as well. One certification which I never seriously looked at is the CompTIA A+ certification.

The A+ is heavily focused upon desktop support and common IT problems including hardware issues. It's non-vendor specific so this allows the exam to cover Windows pc, and Apple Mac but gives a person a good basics about IT troubleshooting. Honestly, when I went over the test questions I first thought to myself why were they so simple. Even a few questions how to replace the toner on a printer. It seems strange but the exam really gave some good questions to honestly fix a problem in the fastest method possible.

Since I had previous experience working and repairing computers at work and at home, I knew most (but not all) of the answers. For someone just starting in the field, I would say it's a good starting point.

Now there's a few important points I have to follow up with the A+ certification. Just because someone can pass a exam does not make them a good IT support person. Also the test, while pretty solid I think has a few flaws that it's so heavily based upon hardware technology that it's easily outdated. CompTIA has recently released the 2009 version of the test to update from 2006, but I still think memorizing details such as how many pins are on a socket 370 processor is not so important as some other skills.

Also keep in mind that the A+ certification is usually aimed as a entry level IT certification. This means it's focused for help desk, IT break fix tech, and even people who work at Best Buy or Fry's. In fact, I was at Fry's a few weeks ago and in the hard drive section they had a sign for installation, then in writing below "Fry's Electronics only hires A+ certified staff". It was pretty interesting and shows how popular certifications are getting mainstream.

So I'm going to study this on the side, I figure I can use the extra help for my hardware skills and never know when it might come in hand. Also some jobs actually require these smaller certifications for getting through the HR process.

Tuesday, October 13, 2009

Windows Server 2008 Core

Server core is nothing new, in fact it's been out for some time but I found this handy cheat sheet from Microsoft which is helpful.

Microsoft Server Core PDF's

Exam 83-640

Recently I took the Microsoft 2008 Active Directory exam 83-640, I actually scheduled the exam for the 70-640 but looks like the 83-640 is now the standard.

The exam is now done from a virtual server where you are given a short list of tasks to complete within a amount of time. It's actually easier for me to complete than the standard multiple question exam because you can remember steps by looking around. Also amazingly enough, the help command is not disabled, which you can also use for any problems.

A few web sites have posted information about the exam, especially concerning if you make too many wrong click would that count against you or not. It turns out that no, only completling the tasks will be recorded in the final score. But a major downside, at least for my testing was the lack of any score besides my points. This doesn't allow me to go over my exam score and study on the items I need help in. The reason given by Microsoft was the testing technology was still too new. The new exam came out in early 2009 so I would guess it should have been resolved by now.

Transcender posting about 83-640 vs 70-640

ExamCollection forum post about 83-640

Sunday, September 27, 2009

Computer based training

I'm usually more sold on using books and class room training but there are times when having a on-line class is just easier. Since this semester, I'm taking two on-line classes, one for Linux shell scripting and another for Windows Server 2008 Active Directory. While both are fun classes, I wanted to really study for the Microsoft MCITP certification with the classes highly focused on the information on the certification topic.

After searching around I found a few sites that offer CBT classes. So I'm going to put down the prices for the Microsoft MCITP System Administrator classes as an example.

Testout - They offer two sets of training materials for the Server Administrator and the Enterprise Administrator. Prices are $995 and $1295, which is actually on the cheap side of classes. They include labs, exam questions, videos, and more.

LearnKey - From the web site I think this is a video series like the others but without labs or exam questions. The prices are $1395 for the Server Administrator and $2080 for the Enterprise Administrator.

TrainSignal - They offer the same videos as the others but also Transcender practice tests, labs and extra features like MP3 audio and other file formats to bring along. The pricing is $995 for the Server Administrator and the Enterprise Administrator is $1495. The hours on the courses by TrainSignal are longer and this seems like the best value for CBT (of course your experience may vary).

I'm right now studying for the Server Administrator certification and will be making a purchase soon, hopefully to further help me study.

Friday, September 18, 2009

Windows Server 2008 study guides

A forum poster on the TechNet forum posted a link to his web page with helpful Windows Server 2008 study guides.

http://www.kurtdillard.com/StudyGuides/

Great news for me since he's written a guide for 70-640!

Thursday, September 17, 2009

Learning on your own

I'm studying for the Microsoft MCITP certification exams and been using the self paced books from Microsoft. So far they have been very helpful but I also registered for a local community college course they uses the Microsoft Learning web site. The Microsoft Learning web site is a on-line course for the same topics as the book, Windows Server 2008 Active Directory Domain Services, etc. So far from taking the classes it's not too bad but there's a big different between using a book to learn and on-line.

One aspect I enjoy using a book is that it's something I can hold in my hand, sounds funny but I like that I can actually write in it. If I need to make a note about a tough question or good web site, I just write it down in the page. Also it's easier to study off-line when you just need to review a few pages in the book. Of course the down side is the limited interaction with the book compared with on-line classes.

The book does not offer video, example labs, additional documents, not including other technologies such as chat with other students. Overall, I still enjoy books but I think the additional help with an on-line class should help.

Certification practice exams

One part of studying for an exam is a practice exam. It's a how you prep yourself for the exam, but as you might know you can not actually have the real exam. In this case you can use various exam vendors. A nice bonus is with most books you purchase they offer a short practice exam, most of the time this exam is a bit shorter than the more expensive exams you purchase.

So if you already have a book with a practice exam is it worth the money to buy another practice exam? From my research, people say that while very close, a well know practice exam vendor will often have more realistic questions and therefore become a better prep material. I think it's worth the $50~90 per exam especially if that will help you save $100~200 to retake the exam over again.

One word, stay clear of brain dumps, or exam notes that just show the actual exam questions. These are sold by less popular exam vendors and are against the terms for many companies offering exams.

Wednesday, September 16, 2009

Good news (or bad) for the Microsoft certifications

Just saw a link posted on Ars Technica that the end of life for Windows Server 2000 will end July 13, 2010 and Windows Server 2003 will end July 14, 2015. Sounds like it's a way to go but honestly many companies will need to already plan the move to Windows Server 2008 well before the deadline. Most applications can run on 2003 or 2008 but the problem is more of home grown applications having issues with the new 2008 servers. I'm studying for my 2008 certification and this seems to be even bigger push to get it done, especially since the all servers will be upgrade soon.