the life of a System Administrator

Remote Installation Server (RIS) has been a possible choice for computer deployment up until Windows 2003. Even though it there was a lot of room for improvement it seemed to be working quite well… also over VPN connections.

A couple of years ago I decided to start using Windows Deployment Services (WDS) and Microsoft Deployment Toolkit (MDT). It worked great, easier to handle and offered more functions. If you are familiar with RIS/WDS or other deployment technologies, you might already know that you cannot deploy across subnets unless you make special configurations in your DHCP scope (see http://support.microsoft.com/kb/259670) or setup DHCP Relay Agents/IP Helper/etc. on each subnet. My choice was to configure IP helper adresses in my Cisco networking equipment.

At the company, where I’m currently employed, we have more than 120 minor off site locations connected to headquarters through VPN tunnels. It was decided that we would like to offer the possibility to deploy a workstation through the VPN tunnel. Obviously it would take hours to do so but it would save us alot of logistical problems and time issues since some of the off site locations are in other parts of world. I didn’t expect this to be a problem since I knew this could be handled by good old RIS, but I was very much wrong. When PXE booting across the VPN kept getting the following error:

PXE-E32: TFTP open timeout

I googled a lot, tried to change DHCP settings but nothing really worked… I kept ending up with the TFTP timeout. But when all else fails my best advise is to go back to basics and capture some network traffics to find out what’s really going on. That’s what I did.

To better understand the network captures I just want to show a simple layout of the network. The client with the IP 10.1.3.83 is an internal client that can be deployed through WDS. The client with the IP 10.129.200.150 is the client that are experiencing problems with deployment over the VPN tunnel. Obviously the server has the IP 10.1.0.17.

To know how a working scenario would look like on the network I made a capture of the initial deployment process from the 10.1.3.83 client, which could be deployed successfully. Since this is a working scenario the packets look the same on both the client- and server-side.

No.     Time        Source                Destination           Protocol Info
      1 0.000000    10.1.3.83             10.1.0.17             DHCP     DHCP Request  - Transaction ID 0xb8b20989
      2 0.007570    10.1.0.17             10.1.3.83             DHCP     DHCP ACK      - Transaction ID 0xb8b20989
      3 0.013010    10.1.3.83             10.1.0.17             TFTP     Read Request, File: boot\x86\wdsnbp.com\000, Transfer type: octet\000
      4 0.024179    10.1.0.17             10.1.3.83             TFTP     Option Acknowledgement, tsize\000=31124\000
      5 0.024185    10.1.3.83             10.1.0.17             TFTP     Error Code, Code: Not defined, Message: TFTP Aborted\000
      6 0.025794    10.1.3.83             10.1.0.17             TFTP     Read Request, File: boot\x86\wdsnbp.com\000, Transfer type: octet\000
      7 0.027849    10.1.0.17             10.1.3.83             TFTP     Option Acknowledgement, blksize\000=1456\000
      8 0.027897    10.1.3.83             10.1.0.17             TFTP     Acknowledgement, Block: 0
      9 0.028376    10.1.0.17             10.1.3.83             TFTP     Data Packet, Block: 1
     10 0.028381    10.1.3.83             10.1.0.17             TFTP     Acknowledgement, Block: 1
     11 0.029067    10.1.0.17             10.1.3.83             TFTP     Data Packet, Block: 2
     12 0.029071    10.1.3.83             10.1.0.17             TFTP     Acknowledgement, Block: 2
     ...
     51 0.049679    10.1.0.17             10.1.3.83             TFTP     Data Packet, Block: 22
     52 0.049681    10.1.3.83             10.1.0.17             TFTP     Acknowledgement, Block: 22
     53 0.086404    10.1.3.83             10.1.0.17             DHCP     DHCP Request  - Transaction ID 0xb8b20989
     54 0.086408    10.1.0.17             10.1.3.83             DHCP     DHCP ACK      - Transaction ID 0xb8b20989
     55 0.133574    10.1.3.83             10.1.0.17             TFTP     Read Request, File: Boot\x64\pxeboot.com\000, Transfer type: octet\000
     56 0.027849    10.1.0.17             10.1.3.83             TFTP     Option Acknowledgement, blksize\000=1456\000
     57 0.140103    10.1.3.83             10.1.0.17             TFTP     Acknowledgement, Block: 0
     58 0.140624    10.1.0.17             10.1.3.83             TFTP     Data Packet, Block: 1
     59 0.140629    10.1.3.83             10.1.0.17             TFTP     Acknowledgement, Block: 1
     60 0.141278    10.1.0.17             10.1.3.83             TFTP     Data Packet, Block: 2
     61 0.141282    10.1.3.83             10.1.0.17             TFTP     Acknowledgement, Block: 2
     ...
     92 0.153642    10.1.0.17             10.1.3.83             TFTP     Data Packet, Block: 18
     93 0.154076    10.1.3.83             10.1.0.17             TFTP     Acknowledgement, Block: 18

Please notice packet 7 and 56 marked with red. The TFTP server tells the client that it should expect a block size (blksize) of 1456 bytes. In RIS this was 512 bytes but in Windows 2008 and later this has been changed to 1456 bytes to speed up the deployment process. But Microsoft forgot to take VPN into consideration.

Here’s how the packet capture looks like from the server, when the remote client on the 10.129.200.0/24 network is PXE booting. Packet 7 reaches the client as the server receives an acknowledgement in packet 8. After that the server just keeps sending block 1 again and again, as it doesn’t receive an acknowledgement from the client.

No.     Time        Source                Destination           Protocol Info
      1 0.000000    10.129.200.150        10.1.0.17             DHCP     DHCP Request  - Transaction ID 0xb8b20989
      2 0.000267    10.1.0.17             10.129.200.150        DHCP     DHCP ACK      - Transaction ID 0xb8b20989
      3 0.041467    10.129.200.150        10.1.0.17             TFTP     Read Request, File: boot\x86\wdsnbp.com\000, Transfer type: octet\000
      4 0.043171    10.1.0.17             10.129.200.150        TFTP     Option Acknowledgement, tsize\000=31124\000
      5 0.058853    10.129.200.150        10.1.0.17             TFTP     Error Code, Code: Not defined, Message: TFTP Aborted\000
      6 0.066817    10.129.200.150        10.1.0.17             TFTP     Read Request, File: boot\x86\wdsnbp.com\000, Transfer type: octet\000
      7 0.071815    10.1.0.17             10.129.200.150        TFTP     Option Acknowledgement, blksize\000=1456\000
      8 0.088466    10.129.200.150        10.1.0.17             TFTP     Acknowledgement, Block: 0
      9 0.088706    10.1.0.17             10.129.200.150        TFTP     Data Packet, Block: 1
     10 2.074910    10.1.0.17             10.129.200.150        TFTP     Data Packet, Block: 1
     11 4.081189    10.1.0.17             10.129.200.150        TFTP     Data Packet, Block: 1
     12 6.074582    10.1.0.17             10.129.200.150        TFTP     Data Packet, Block: 1
     13 8.076182    10.1.0.17             10.129.200.150        TFTP     Data Packet, Block: 1
     14 10.076389   10.1.0.17             10.129.200.150        TFTP     Data Packet, Block: 1
     15 12.086364   10.1.0.17             10.129.200.150        TFTP     Data Packet, Block: 1
     16 14.085904   10.1.0.17             10.129.200.150        TFTP     Data Packet, Block: 1
     17 16.074910   10.1.0.17             10.129.200.150        TFTP     Data Packet, Block: 1

Let us take a look at the client side of things to see what’s going on there.

No.     Time        Source                Destination           Protocol Info
      1 0.000000    10.129.200.150        10.1.0.17             DHCP     DHCP Request  - Transaction ID 0xb8b20989
      2 0.044616    10.1.0.17             10.129.200.150        DHCP     DHCP ACK      - Transaction ID 0xb8b20989
      3 0.050171    10.129.200.150        10.1.0.17             TFTP     Read Request, File: boot\x86\wdsnbp.com\000, Transfer type: octet\000
      4 0.068788    10.1.0.17             10.129.200.150        TFTP     Option Acknowledgement, tsize\000=31124\000
      5 0.068934    10.129.200.150        10.1.0.17             TFTP     Error Code, Code: Not defined, Message: TFTP Aborted\000
      6 0.070140    10.129.200.150        10.1.0.17             TFTP     Read Request, File: boot\x86\wdsnbp.com\000, Transfer type: octet\000
      7 0.097809    10.1.0.17             10.129.200.150        TFTP     Option Acknowledgement, blksize\000=1456\000
      8 0.097934    10.129.200.150        10.1.0.17             TFTP     Acknowledgement, Block: 0
      9 0.142206    10.1.0.17             10.129.200.150        IP       Fragmented IP protocol (proto=UDP 0x11, off=0, ID=3ab9) [Reassembled in #10]
     10 0.142305    10.1.0.17             10.129.200.150        TFTP     Data Packet, Block: 1
     11 2.129867    10.1.0.17             10.129.200.150        IP       Fragmented IP protocol (proto=UDP 0x11, off=0, ID=3abe) [Reassembled in #12]
     12 2.129870    10.1.0.17             10.129.200.150        TFTP     Data Packet, Block: 1
     13 4.133477    10.1.0.17             10.129.200.150        IP       Fragmented IP protocol (proto=UDP 0x11, off=0, ID=3ac7) [Reassembled in #14]
     14 4.133931    10.1.0.17             10.129.200.150        TFTP     Data Packet, Block: 1
     15 6.127184    10.1.0.17             10.129.200.150        IP       Fragmented IP protocol (proto=UDP 0x11, off=0, ID=3acc) [Reassembled in #16]
     16 6.127536    10.1.0.17             10.129.200.150        TFTP     Data Packet, Block: 1
     17 8.127606    10.1.0.17             10.129.200.150        IP       Fragmented IP protocol (proto=UDP 0x11, off=0, ID=3ad1) [Reassembled in #18]
     18 8.128362    10.1.0.17             10.129.200.150        TFTP     Data Packet, Block: 1
     19 10.128063   10.1.0.17             10.129.200.150        IP       Fragmented IP protocol (proto=UDP 0x11, off=0, ID=3ad7) [Reassembled in #20]
     20 10.128068   10.1.0.17             10.129.200.150        TFTP     Data Packet, Block: 1
     21 12.137783   10.1.0.17             10.129.200.150        IP       Fragmented IP protocol (proto=UDP 0x11, off=0, ID=3aec) [Reassembled in #22]
     22 12.137788   10.1.0.17             10.129.200.150        TFTP     Data Packet, Block: 1
     23 14.135978   10.1.0.17             10.129.200.150        IP       Fragmented IP protocol (proto=UDP 0x11, off=0, ID=3b19) [Reassembled in #24]
     24 14.136504   10.1.0.17             10.129.200.150        TFTP     Data Packet, Block: 1
     25 16.124145   10.1.0.17             10.129.200.150        IP       Fragmented IP protocol (proto=UDP 0x11, off=0, ID=3b39) [Reassembled in #26]
     26 16.124460   10.1.0.17             10.129.200.150        TFTP     Data Packet, Block: 1

As you can see the client actually receives block 1 repeatedly, but why doesn’t it send back an acknowledgement then? The reason for this is that the PXE client on the network card is very simple. It doesn’t support fragmented packages (notice the lines marked with red). You might ask why isn’t this a problem on the internal network then. Well, Microsoft set the block size to 1456 bytes to leave some room for IP headers so the packets wouldn’t get fragment on the network. However when you add VPN into the equation you’ll get some extra VPN headers and the packet size would grow too big. To take care of this the networking equipment will devide the TFTP packets into 2 VPN packets to be able to send them across the Internet and fragmentation occurs. Since the PXE client is unable to handle this it doesn’t send an acknowledgement to the server, which then keeps sending the same block again and again undtil we get a TFTP timeout. Since we cannot change the configuration of the client, the configuration of the server needs to be changed. But Microsoft hasn’t made this an easy task. The TFTP server is a builtin part of WDS and Microsoft doesn’t provide an interface to modify settings related to the TFTP server.

Luckily enough Microsoft has realized that this is a significant problem, so after a while I fell over this KB http://support.microsoft.com/kb/975710. Here you can request a hotfix from Microsoft that makes it possible to control the TFTP block size through the registry. What block size is best for your environment depends on the type of VPN you use as the size of the VPN headers vary. If you don’t know where to begin, 1400 bytes could be a good place to start. If it doesn’t work, lower the value. If it works you can leave it there or add a few bytes to speed up deployment (probably not noticable). By applying this hotfix and setting the block size in registry, you avoid the packet fragmentation through VPN and the PXE client is then cabable of handling the TFTP packets and the deployment process should work.

Hope this made the problem clearer for everyone. I thinks this is also a problem with System Center Configuration Manager (SCCM). If someone can confirm this, I’ll add it to this post. I’ve attached the original packet captures below.

WDS Network Captures

As an administrator I would of course never forget to delete a computer account , but if someone else should forget it quite a lot of orphaned accounts would exist in the Active Directory over time. So how would one detect accounts that hasn’t been used for a while? Well, again the answer is PowerShell. With this simple, little script below you are able to choose the age of the accounts you want displayed. Of course you could easily edit the script so it deleted the accounts right away by using Remove-QADObject. You could also use the Get-QADUser cmdlet to search for user accounts instead of computer accounts.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

Add-PSSnapin -name Quest.ActiveRoles.ADManagement
 
$searchRoot = "domain.local/Workstations"
$age = 60 # in days
 
$accounts = Get-QADComputer -IncludedProperties LastLogonTimeStamp -SearchRoot $searchRoot | Sort-Object -Property LastLogonTimeStamp -Descending
$today = Get-Date
 
foreach( $account in $accounts )
{
    if( $account.LastLogonTimeStamp -eq $null )
    {
        Write-Host $account.Name "last logon: Never"
    }
    else
    {
        $res = $today - $account.LastLogonTimeStamp
        $days = $res.Days
 
        if( $days -gt $age )
        {
            Write-Host $account.Name "last logon:" $days "days ago"
        }
    }
}

One thing is certain when you assign space to mailboxes. If the user is assigned 100MB she will use it, if the user is assigned 1000MB, she will use it. Unlike in Exchange Server 2003 you are not able to retrieve a list that displays the mailbox size from the Exchange Server 2007 management console. The script below displays the 50 largest mailboxes in your organization in descending order. Simply edit the script to reflect your needs.

1
2
3

Add-PSSnapin -name Microsoft.Exchange.Management.Powershell.Admin
 
Get-MailboxStatistics -Server "server"  | Sort-Object -Descending TotalItemSize | ft displayname,totalitemsize,ItemCount,StorageLimitStatus | Select-Object -first 50

Environment: The server is Windows Server 2008 Standard Edition x64 with WDS installed. Microsoft Deployment Toolkit 2010 is installed on the server and Windows Automated Installation KIT (AIK) aswell. MDT 2010 can be downloaded here.

If you are used to RIS or WDS, you know that these services gives you the opportunity to set a “New client naming policy” and also where to place the client account for the workstation during deployment. When you work with MDT things are a little different since you generate a Lite Touch WIM image and add that as a boot image in WDS. By doing this you only use WDS for PXE boot and download the WIM image. After that the client moves on and WDS is no longer needed by the client. At this stage deployment hasn’t really begun yet so setting a client naming policy and/or the OU in which you want the new computer account to be created does not have any affect at all. Chances are that if you have already tried to deploy a workstation using MDT, you are already familiar with the random computernames that are generated by MDT. I have googled quite a lot wihtout any luck finding a good solution. I wanted a solution which named the workstations automatically say WKS001, WKS002 etc. If a computeraccount would be removed from Active Directory the computername should be available for a new workstation.

The computernaming can be handled quite easily with a PowerShell script. However the default security configuration of PowerShell will not let you run the script from a network share. Also in order to be able to rename the workstation in question I would have to get around UAC.

To get around UAC during deployment I had to be a little creative. I created an Organizational Unit in Active Directory and blocked inheritance. Besides linking my WSUS policy to the OU (since I want to be able to install Microsoft patches during deployment), I also created a GPO that would deactivate UAC and lower the PowerShell security settings so I could easily execute my scripts and other commands/programs during the process. The following GPO settings were the ones I used:

GPO Settings

And just to visualize how it could look like in the Group Policy Management MMC:

So now that script and program execution is ready, the next step is to configure MDT to deploy the computers into the “Deploying” Organization Unit we just created. To do this open the ”Deployment Workbench” and open the Task Sequence you are using. Under the “OS Info” tab click  ”Edit Unattended.xml” button. This will open the answer file for your OS installation and here you will be able to configure which OU in which you want computer account to be created. In the “Answer File” windows browse to Unattend –> Components –> 4 specialize –> x86_Microsoft-Windows-UnattendedJoin_neutral –> Identification. In the “Identification Properties” window there is a setting called “MachineObjectOU”. Enter the “distinguishedName” of the Organizationl Unit. To find the “distinguishedName” just open “Active Directory Users and Computers” –> right clik the Organizational Unit and choose “Properties”. Next open the “Attribute Viewer” tab.

Now it is time to introduce the script that does all the work. I know it could be a bit more sofisticated, but it works. Remember to change the prefix of computer account and the OU to search in so it fits your needs. The script simply browses through the OU and looks for computer account with a specific prefix, in this case “PC”. Lets say 3 computer accounts are found: PC001, PC002, PC004. The script will discover that PC003 is not in use and will assign this name to the new computer. Of course this will be unsuccessfull if PC003 is located somewhere else in Active Directory. In that case you will have to modify the script. If the script doesn’t fine any available account names it will just select the next in line e.g. PC005. In this case I saved the script as “ComputerRename.ps1″ in the “Script” directory of my deployment share.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

function GetComputerList($TargetDn)
{
# Locates all computer accounts that begins with "PC"
# Change the value for your needs
 ${tFilter} = ‘(&(objectClass=computer)(cn=PC*))’
 ${Searcher} = New-Object System.DirectoryServices.DirectorySearcher $tFilter
 ${Searcher}.SearchRoot = “LDAP://${TargetDn}”
 ${Searcher}.SearchScope = [System.DirectoryServices.SearchScope]::Subtree
 # Page size default in Active Directory is 1000
 ${Searcher}.PageSize = 1000
 $Results = $Searcher.FindAll()
 return $Results
}
 
# Change OU path for you needs
$computers = GetComputerList("OU=Workstations,DC=yourdomain,DC=local")
$current  = 0
$previous = 0
$new_name = $null
 
# Run through search result
foreach( $computer in $computers )
{
    $cn = "" + $computer.Properties['cn']
    $current = [int]$cn.Substring(6)
 
    # Check for avialable computername
    if( $current.CompareTo($previous+1).Equals( 1 ) )
    {
        $new_name = $previous+1
        break # no need to search anymore
    }
 
    $previous = $current
}
 
# If no available computername use next in line
if( !$new_name )
{
    $new_name = $current+1
}
 
# Prefix number with 0's if needed
if( $new_name.CompareTo(10).Equals(-1) )
{
    $new_name = "00" + $new_name
}
elseif( $new_name.CompareTo(100).Equals(-1) )
{
    $new_name = "0" + $new_name
}
 
# Prefix name with constant
# Change the constant "PC" for your needs
$new_name = "PC" + $new_name
$new_name
 
# Do rename
$oComputerSystem = Get-WmiObject win32_computersystem
$oComputerSystem.Rename( $new_name )

The last thing to do is to make sure the script will be executed as part of the Task Sequence. Again open the “Deployment Workbench” and the Task Sequence you are using. Click “Add” –> “General” –> “Run Command Line”. Give the task a name e.g. “Computer Rename”. In the “Command Line” textbox enter the following command and set it up so it runs under an account that has the appropiate rights to rename a computer account:

powershell.exe "%SCRIPTROOT%\RenameComputer.ps1"

Hope this was usefull.

OK, here’s the first funny story I want to publish.

In the summer 2009 I had prepared two new, great servers – HP DL380 G6 with 2 Xeon E5520 @ 2.26GHz, 32GB RAM and disk enclosure filled with 146GB 15K SAS disks for each server. They were going to run Exchange 2007 in a CCR configuration for high availability. Since the old configuration was a 32 bit environment with 2 DL380 G4 servers in an old Veritas cluster configuration I was expecting outstanding performance and that’s what I got but one of my users wasn’t so satisfied.

I spent more than 14 hours (no sleep) on migrating data and were quite satisfied when it was over. I didn’t get any errors or ran into any unexpected problems. Of course I instructed my users to contact me directly if they were experiencing any problems that would be related to the recent migration. Of course I got a few calls because OWA had changed etc., nothing big. However after a day or two a not so satisified user called me. Here’s the conversation.

User: Hi. Since you made those changes to our system my Internet is very slow.
Me: What changes are you referring to (surely he couldn’t be referring to the exchange migration).
User: Well, you made those changes to the e-mail system the other day. www.krak.dk is taking more than 7 minutes to load on my computer.
Me: OK, that sounds strange.
User: Yeah, it started right after you made the change so I’d like you to change it back as this is unacceptable.
Me: I’m pretty sure this has nothing to do with the changes I made to the e-mail system.
User (insisting): Well, I’m pretty sure that it has. You must admit that it is strange this happens right after you have made those changes.
Me: Yeah, you know what. I’m gonna have to look into this and we’ll call you back within the hour. OK?
User: OK, thank you.

I asked a supporter to call back and let him know the background of the call. We had a good laugh, but how should the user know any better? . I didn’t hear from the user again, so I guess they figured it out.