HynesITe, Inc Events Team Blog

For a while now I’ve been struggling with something.  The model that we have relied upon since we started delivering labs at events has been to place the content closest to the user, directly on the PC they are using.  This is done for a few reasons.

• Event networks are weak at the best of times.  Most people don’t realize that network outages are a frequent thing on event sites.  Thanks Mr. Forklift Driver.
• There is no contention for resources, resulting in a predictable experience (Note that predictable is more important than fastest).  If you don’t believe me look at Starbucks and McDonalds.  They have global appeal not because they are good, but because they are ththe same.  They are predictable.  You know what you will get, like or dislike.
• Events generally have an abundance of high end workstations and people who know how to image and deploy lots of high end workstations.

Recent events have caused a shift in thinking however.  VM’s today, running 64 but Windows Server and on Hyper-V are pushing what you can reasonably do with a high end workstation.  Yes you can put a few TB of storage in them and put up to 32GB of RAM, but doing that across 500 systems gets expensive.  Not to mention the fact that the delta between resource requirements of low end labs and high end labs is growing.  We have labs that run happily on 2GB, and some that now require up to 16.  The problem with that is it leaves you with a horribly inefficient use of your hardware.  You have to make sure each machine can support the biggest lab that may run on it, across all the events that the machine will be used, and that inevitably will lead to lots of times when you have 12 GB sitting idle.

The fundamental problem with building a centralized hosted lab delivery system for an event is pretty much the list above you, inverted.  Networks which hosted systems would depend on need to be re-engineered.  Resource contention and scheduling code has to be written, and hardware has to be purchased.  So why even do it, well the answer is that in the end, with the delta between high and low being so great, the cost of change is an easy investment to recover.  Consider an overly simplified example.

To support 500 labs, with RAM requirements ranging from 2GB to 12GB, such as what we have at TechEd Europe, you have to have a set of systems that has up to 16GB of RAM (factor in overhead for the host and support DIMM configurations).  We run this on an HP 8600 workstation, which in our configuration, retails for about 6K USD.  That’s a total of 3 Million USD of hardware, just using rounded face values.  Of course the way hardware is acquired for events such as TechEd Europe does not entail writing a 3 Million USD cheque (that’s the Canadian and proper spelling of it), but that’s the market value of the hardware.  The result is much like block level file allocation.  You have a minimum allocation of 12GB of RAM per lab, regardless of the needs of the lab.  If you average out the lab RAM requirements for TechEd Europe, you determined that the *average* is more like 3GB.  So betting on the averages, and that we’ll have a pretty even spread of labs (between low and high end ones), then you can support 500 sessions using 500 * 3 (or 1500GB of RAM), which spread out over servers running 96 GB RAM per machine, means you can support 500 lab sessions using about 16 physical systems.  A little research and pricing, and you’ll notice that amounts to about 250K.  In case you are not good with math, 250K is quite a bit less than 3 Million.  That’s not only cost, but with a raw hardware need to support VM’s in the range of 250, you can go a long way to fixing the other risks (networks, contention management software, etc) without even coming close to hitting the hardware value for local deployment.

More on this later….specifically how we are eliminating imaging from HOL setups 100%…..not even a client workstation image….

Corey

Doing our rollout the content in TechEd AU we hit a stumbling block – 10/100 switches. When you are sending out 600+Gb images in a time precious rollout, 10/100 is not going to cut it.

So after borrowing a couple of gigabit switches (Shout out to Dugie!) and getting the sales guy at the local store to hold all the gigabit switches that he had for us (and therefore making my 5 trip between the gold coast and Brisbane in 3 days occur) we are now gigabit throughout.

So.. what difference does that make? Massive – the first deployment session was going to take 30+ hours over the 10/100 switches. Drop the gigabit switches in and that is reduced to 6 hours. So – yup – major difference.

Now onto presence testing!

Cheers

Kyle

One of the issues that we often come across is that we have Virtual Machines that have been handed off for use in the labs but they have been dormant for an extended period of time. This means that if we have multiple DC’s in a lab and they start then the tombstone period may have expired.

So.. what to do?

Well.. the first (and preferable option) is to have this fixed before the VM is saved by increasing the tombstone lifetime to the maximum (or the expected lifetime of the image) – this is part of our best practise but often gets missed.

So the 2nd option is a reghack to fix it (from here):

To restart replication following event ID 2042

1. Click Start, click Run, type regedit, and then click OK.

2. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters

3. In the details pane, create or edit the registry entry as follows:

   If the registry entry exists in the details pane, modify the entry as follows:

   1. In the details pane, right-click Allow Replication With Divergent and Corrupt Partner, and then click Modify.
   2. In the Value data box, type 1, and then click OK.

   If the registry entry does not exist, create the entry as follows:

   1. Right-click Parameters, click New, and then click DWORD Value.
   2. Type the name Allow Replication With Divergent and Corrupt Partner, and then press ENTER.
   3. Double-click the entry. In the Value data box, type 1, and then click OK.

Reset the Registry to Protect Against Outdated Replication

When you are satisfied that lingering objects have been removed and replication has occurred successfully from the source domain controller, edit the registry to return the value in Allow Replication With Divergent and Corrupt Partner to 0

Well.. for us anyway – the Joma Blue guys have been there for a week already – but our roll in starts in tomorrow. We had a quick drive down to offer some assistance today and it was a good opportunity to get an eyes on, on the Technical Learning Centre.

The drive from Brisbane to the Gold Coast meant another hour’s worth of work time available for Kim and Corey – one in the front seat and one in the back seat – with laptops in use. This meant that in true geek style when a file needed to be transferred between their laptops – no usb device was needed. So – if you happened to be beside the Pacific Highway and saw an Ad-Hoc wireless network named “Driving” appear and then disappear – that was us.

On the content note – the servers are now in the hands of Ted our infrastructure guy to create the images – so they will be ready in the morning for us to make the drive to the coast. Basically our plan is to have 5 images – each about 600Gb – but that means we should have less imaging during the week.

Cheers

Kyle