blog.matthewtrotter.com

Why you should IMMEDIATELY sell your shares in RIM

Ok, I have fucking had it with BES 5.0

I am quite sure that if I don’t uninstall this software and forget it ever existed, I will become so stupid from having used it that the world around me will implode.

You know, all of the previous BES releases were flakey in their own right, mind you they worked sometimes. Now a company who decides to move from a dedicated windows app to a Web interface is usually attempting to make thing easier for their clients.

Problem 1: WHY THE FUCK DID YOU USE APACHE + TOMCAT + JAVA?

You must be fucking stupid to be the guy/gal to make the engineering decision to use a web front end on a Windows platform with that software.

Problem 2: WHY HAVE A WEB FRONT END IF YOU NEED TO INSTALL LOCAL CLIENT SOFTWARE TO MAKE IT FUCKING WORK?

This COMPLETELY defeats the use of a web front end. It’s almost like all of the RIM engineers got together in a meeting and said “Well Microsoft is moving to complete HTTP(S) based software, maybe we should do the same?” and then they couldn’t figure out how to make it all work, so they did a half and half. Half and Half only works for two things, Cream for your coffee and Prostitution.

Problem 3: BES 5.0.1 MR 1 DOCUMENTATION STATES YOU NEED TO MANUALLY STOP THE BES SERVICES

This is quite possibly the LAZIEST thing RIM has ever done. Instead of our patch software automatically stopping the 10 services needed for the install, you have to manually shut them all down yourself. Fuck you RIM. I love the fact that the 5.0 SP1 was able to shut the services down for you.

Problem 4: WHO THE FUCK IS JBOSS AND WHY IS HE NOT WORKING AND FUCKING MY SERVER UP

For those of you that don’t know, Java is the WORST programming language ever. Yes, it is getting in line behind Pascal. So I install my BES 5.0.1 and I also install the Admin Service onto my desktop so I don’t need to RDP into the BES to do admin tasks. After completely losing myself in the installer for the admin tools (because they didn’t think to ask at the beginning if all you want is the admin tools so you end up going through this MASSIVE installation routine which makes you think you are going to have to install a whole new BES for nothing), I get it installed on my desktop and am able to log in. I install MR 1 onto my BES and reboot like a good little sysadmin jockey and when the machine comes back online, the Administration Interface don’t work no more.

The application has encountered a system error.  Please report this error to the administrator.

Well thank fucking god I am the admin and can help myself with this wicked cool, very descriptive error. Turns out some fucker named JBOSS is ruining everything in BES land. After picking through about 3000 lines of logs for a single startup instance (WTF), I find this error, which doesn’t even look like an error:

[com.rim.bes.bas.singletondeploymentbarrier.SingletonDeploymentBarrierNotificationService] [INFO] [BBAS-1011] {unknown} start CLUSTER:   singleton deployment barrier is not deploying

It doesn’t look like an error BECAUSE IT FUCKING SAYS [INFO] NOT [CRITICAL FUCKING ERROR WHICH WILL RUIN YOUR FUCKING LIFE]

So it turns out RIM has a nice little doc about this, which explains everything, except the answer you need is linked to another document which doesn’t exist. (KB19920) *The link is working again now* http://www.blackberry.com/btsc/search.do?cmd=displayKC&docType=kc&externalId=KB19436

UDP. Are you kidding me? Holy fucking take a step back in time. Why even make software if this is how you are going to do it? I can just picture that meeting: “Hey Jim, so I am pretty sure that everyone in the world fits all of their servers, client machines, etc… in one subnet, so we might as well use UDP broadcasts to get information between hosts.”

I think it is time Jim Ballsilie realizes that the only team he is going to be allowed to buy is the LEAFS for obvious reasons.

More to come… I am sure of it.

Speaking of more to come: http://online.wsj.com/article/BT-CO-20091223-708432.html - HAHA Fuckers.

Exchange 2010 OWA Redirect - The Real Answer

Holy dog balls batman… I am absolutely SICK of the amount of douchbaggery littering this wonderful place called the internet. I am also sick of people who suck at their jobs and only write half a solution on their blogs, and also people who link to articles which hava ABSOLUTELY FUCKING NOTHING to do with the problem at hand. I am using Exchange 2010 on Windows Server 2008 R2 with IIS 7.5

Problem: I don’t want my users to have to type - https://mail.domain.com/owa to get a Outlook Web App - I want them to be able to type EXCHANGE into the fucking address bar, hit enter and have everything (redirection to /owa AND http -> https redirection) AUTOMAGICALLY work.

Solution: Read this.

1. Cut a hole in a box
2. Put your Junk in that box
3. Make her open the box
4. If you didn’t get steps 1 through 3, follow this link - http://www.youtube.com/watch?v=WzqrY4ItlxI

On the default website, disable SSL. This will also disable SSL on EVERYTHING below your default website, so re-enable SSL on all of the virtual directories EXCEPT: /powershell and /oab

On the default website, Turn on HTTP redirection. This will also turn on HTTP redirection on EVERYTHING below your default website, so turn HTTP redirection off all of the virtual directories EXCEPT /public, /exchweb and /exchange

Once this is done, GO BACK TO THE /OWA virtual directory and turn HTTP redirection off AGAIN. For some stupid fucking reason, it magically comes back on when you disable it the first time. Confirm it is OFF.

Create this file C:\INETPUB\httpsredir.htm

Put this code in the file:


<script type="text/javascript">
function redirHttps()
{
var httpURL = window.location.hostname+window.location.pathname;
var httpsURL = "https://" + httpURL ;
window.location = httpsURL ;
}
redirHttps();
</script>


Go to the CUSTOM ERROR page of the SERVER LEVEL OBJECT and add an error for 403.4. Set it up as INSERT CONTENT FROM STATIC FILE INTO THE ERROR RESPONSE and make the path: C:\INETPUB\httpsredir.htm

-M

BES 5.0.1 on Windows 2008 (Mechanism level: Server not found in Kerberos database (7))

I hate RIM. They really suck at everything they do… Building good software, building reliable devices, bidding on hockey teams…

Either way, so I installed my fancy new (PIECE OF SHIT) BES 5.0 on Windows 2008 as per the support configs and ran into an issue. When I tried to log into the Administration Service using Active Directory authentication to start using my fancy new (PIECE OF SHIT) BES, I was told “The username, password, or domain is not incorrect”

Well the creds were correct so what the fuck was going on?… I found this article, which since it was written by RIM was completely useless at solving my problem.

http://www.blackberry.com/btsc/search.do?cmd=displayKC&docType=kc&externalId=KB17949

I fixed my reverse lookups and guess what… still broken.

So because I know just how much RIM sucks at writing good Windows Services, I rebooted the machine and VOILA… fixed. I guess one of their pristinely written services wasn’t functioning correctly.

Dear everyone at RIM (including my brother and my neighbor)… Kill yourselves.

Microsoft Exchange - FAST AS FUCK Edition

Since I enjoy colour commentary and that is also probably the reason you are reading this as opposed to a blog written by an MVP or other appropriate nerd, I would like to tell you about my further foray into Exchange 2003/2003/2010 and Solid State Disks.

 I received a test server from HP (a DL360 G6) in order to test their controllers and backplane with my fancy OCZ Vertex II Solid State Disks. My previous tests of these disks with Exchange on Dell hardware were yeilding incredible speeds, which would allow me to massively reduce my datacenter footprint both in rack u and power consumption. Since I was impressed with the 1U density of the DL360 G6 (8 sff SAS disks), I decided to try one out.

Setup: 5 disk RAID 0 for Storage Groups / 1 disk for Transaction Logs / 10 Storage Groups, each with 1 database (80GB x 10 = 800GB total)

Jist of results:

Read Latency 0.0029

IOPS: 13676.788

So for those of you oblivious fucks who think SSDs and Exchange don’t make sense, blow it out your ass. Now the only reason it wasn’t tested in a correct 10 disk RAID10 array was because the chassis can’t hold that many disks. Those 5 SSD’s outperformed a 140 disk spinning head array.

What does that mean? Well 140 disks, at 3.5″ form factor is about 30U + the server (2U) or even in 2.5″ form factor works out to about 12U + the server (2U), and with a server chassis that can handle 16 SFF SAS disks, you could build a clustered Exchange 2010 Server that supported the IOPS requirement of 30,000+ users and only took up 4U total (having the HT and CAS roles installed as well). Sure if you want to give everyone a 1GB mailbox, you will need more storage space. Think of the cost savings of removing all those high wattage disk chassis from your datacenter…

Eat shit nay-sayers. SSD is here to stay.

Exchange 2007 and Solid State Disks

Thanks to the fact that my Director of IT is an absolute gear junkie, Solid State Disks (SSD) are not too far off from being implemented in my environment. It was funny because there is nothing better during times of economic crisis than spending millions of dollars changing every mechanical hard disk to an SSD.

I was approached by my director in an attempt to reduce the size of my Exchange 2007 cluster environment. Currently I have a pair of Dell PE 2950’s and connected to each of those is a Dell MD3000 SAS Chassis and 2 Dell MD1000 SAS Expansion Chassis. Each server has 45 x 146GB 15K Seagate SAS disks. Both halves of the CCR cluster take up a whopping 22U and in an environment where space is at a premium, that is a VERY big number. My total Exchange storage capacity is about 2.5 TB.

As with most things I am approached by my Director with, I was very hesitant to jump right in. First off, my initial impressions of SSDs were absolutely shit. For the longest time, no one seemed to really know what they were doing on the SSD front and the disks weren’t much better than comparible FC disks, especially when you compaired their reliability and costs. Fast forward a year and alot has changed.

So I get called into the Directors office so he can show me these new fancy OCZ Vertex SATAII disks he bought. He picked up 10 at a little over $1400 a piece for 250GB. He sat me down and told me to watch the screen. Sure enough, a pair of those disks in RAID0 was achieving 550mb / sec reads. Ok, I was impressed, but since I am an Exchange engineer, I cared little about MB/sec and ALOT about IOPS. My director gave me a couple of disks and said I should play with them and see how Exchange likes it. Never willing to say no to fooling around with fancy new shit, I grabbed the disks and contacted Dell to try and get a loaner MD3000 chassis.

In the meantime since I am an impatient SOB, I wanted to see what these disks could do so I took an old Dell PE 1950 chassis and put my 2 Vertex disks into it. I installed the OS on one of the disks. I then installed Jetstress and realized that I acutally needed a third disk to put the transaction logs onto. Since my chassis only supported a pair of disks, I was out of luck. Since I still didn’t want to wait for the loaner chassis, I decided I would put the transaction logs onto the system disk (even in a testing scenario this is a retarded idea) but I figured it would be a good stress test. I configured JetStress to put the databases to 80% capacity and on the 2nd 250gb disk.

Away I went.

Ok, so usually under the best circumstances a FC based disk at 15k RPM pulls about 120-140 IOPS. With this Vertex disk, and keeping in mind that the transaction logs were sharing a disk with the OS, I was pulling about 1200 IOPS from a single disk with incredibly low latencies…

Commence shitting ones pants.

So basically when the 1TB versions of these drives are released by OCZ in December 2009, I will be able to replace a 96 head disk system with only 10 SSDs and reduce the rack size of the Exchange cluster from 22U to a mere 8U. Giddy up. However this does not give MS the right to go ahead and begin making Exchange some bloated, inefficient piece of shit now that there is hardware that can allow it to be. This is my biggest pet peeve about Software companies. I remember the day when an OS fit on a 1.44mb floppy… now I need a fucking dual layer DVD and I still have to download half of the components once the OS is installed. BLOATED = blows ass.

Windows 2008 default Virtual Memory Config blows goats

This is merely a rant about how fucking screwy MS had to be to release a product that was as DUMB as Windows 2008. For those of you that don’t know, the default Virtual Memory Configuration of Windows 2008 is to allow the system to manage the resources. Well, when you have a 72GB system disk and 32GB of RAM, that means that Windows 2008 will allow the system to create a 64GB virtual memory page file and won’t stop when the system runs low, essentially fucking your server directly in the ass. Basically, MS thought it was a smarter idea to crash your server than have it run slow. Ok, yeah I am sure the one day I need to dump all of my physical memory into the page file, so the 64GB file can be examined by an MS support guy (good luck with that), I’ll be upset initially, but since I will have won the lottery and hell will have frozen over, I won’t be too concerned.

VDS.EXE Memory Exhaustion on DPM Server on Windows 2008

Thanks to some buggy code, if you are wondering why your DPM server keeps bombing out, it is most likely the VDS.EXE service consuming all available memory. My DPM Server has 16GB of RAM and usually the VDS Service is hovering around 7-8GB thanks to this problem. There is a hotfix for this MS has released which should fix the issue for you.

http://support.microsoft.com/kb/958387

SCVMM, Shared ISOs, and the shitshow that ensues…

Like everyone else in the virtualization world, I enjoy a good management interface. SCVMM promised to be my new favorite toy, however the stupid POS has a seriously BAD hangup. It doesn’t like Shared ISOs.

For those of you who are unaware, SCVMM gives the admin the ability to “deploy” shared ISOs from the library to specific virtual machines which in essence operates like dropping a DVD into the drive of the virtual machine. This is all fine and good, so long as it works. It provides two ways of doing this, the first is by copying the ENTIRE ISO to the Virtual Host and running it locally, or using the smart method and accessing the ISO over the network using the library share. Sounds simple enough right? Wrong, MS fucked up the implementation of this handy feature so it causes more headaches than doing everything manually.

The shit thing is, this works PERFECTLY from the Hyper-V management console and only fails from the SCVMM interface.

Here is the error you get when you try and connect an ISO using the Shared ISO option in the SCVMM interface:

Error (12700)
VMM cannot complete the Hyper-V operation on the scvmmhostname.domainname.com server because of the error: 'vmhostname' failed to add device 'Microsoft Virtual CD/DVD Disk'. (Virtual machine ID CC66A9DC-3E63-444D-8FB5-B93908F6DEDB)'vmhostname': The file '\\scvmmhostname.domainname.com\MSSCVMMLibrary\ISOs\cdimagename.iso' does not have the required security settings. Error: 'General access denied error' (0x80070005). To fix the security settings, remove the device associated with this file from the virtual machine and then add it again. (Virtual machine CC66A9DC-3E63-444D-8FB5-B93908F6DEDB)
(Unknown error (0x8001))Recommended Action
Resolve the issue in Hyper-V and then try the operation again

So you follow the instructions and nothing happens. Why? Because you forgot to recycle the documentation. This is simply a permissions issue and some googling will find you the answer, however no one ever really seems to describe all of the steps you need to take to rectify the issue so I am stepping in (Apparently steps 2-8 are only required if your Hyper-V server and SCVMM server are two different machines, which is pretty much ALWAYS the case).

1. Ensure that Authenticated Users has READ permissions (Share & NTFS) on the SCVMM Library Share.
2. Go into ADUC, right click on each of your Hyper-V Servers and select the Delegation tab
3. Select “Trust this computer for delegation to specified services only”
4. Select “Use any authentication protocol”
5. Click the “Add” button
6. Click the “Users and Computers” button
7. Type the name of your SCVMM server and click OK
8. Select “cifs” and click OK\
9. Wait 20-30 minutes… why? because Active Directory is annoying that way

If you don’t give the cifs delegation enough time to fornicate/replicate to all of your DCs, you will still get the error and will be scratching your ass/head and cursing at me. 

 *UPDATE*

So because some of you like to bitch about the Authenticated Users permission (See comments below), I decided to revisit this, with 2008 and 2008 R2. NEITHER of those SCVMM platforms will let shared ISO’s work with only the Hyper-V servers and the SCVMM Service account given both Share and NTFS perms on the Library Share. If I enable complete auditing on the Library server, I don’t get any errors in the event log relating to denied access, and I see the Service account AND the computer account for the Hyper-V server hosting the VM in question getting authenticated to the share, however IT STILL DOESN’T WORK. If I add authenticated users, works like a charm, take it out, stops working. Considering I can’t get Windows to log a security failure about this, I have no further way of figuring out what the fuck is happening.

———-

So now, I have determined that if I explicitly add each Hyper-V machine to the ACL (as opposed to using a group) I can make it work WITHOUT needing the Auth Users, so long as the SCVMM Service Account has access to the Share and NTFS. Why this won’t work with a security group pisses me off because now I have to manually add 35 Hyper-V hosts to the ACL. Piece of Shit.

———-

Even better, I have now noticed that SCVMM has “added” itself to the “Virtual Machine Manager Servers” group AUTOMAGICALLY a day after I installed it. Fuck you.

* END UPDATE *

Also if you try and give the virtual machine a Shared ISO during creation, SCVMM will not allow you to put that machine on a Hyper-V host. WTF? It will bomb out with the following error:

Virtualization platform on host hypervhostname.domainname.com does not support shared DVD ISO images.

Piece of shit. Yes it does and I know it does. The only way around this is to build the virtual machine without assigning a shared ISO to the virtual machine and once it is finished creating, go in and assign the shared ISO.

Ciao.

DPM 2007 on Windows 2008 protecting Exchange 2007 SP1 CCR on Windows 2008

I can honestly say without a doubt in my mind that Data Protection Manager 2007 is quite possibly the best and worst product Microsoft has brought to the market.

 Why is it the best? Because I can now recover my Exchange mailboxes in 15 minute increments for that last 2 months (at the cost of 22 tb of disk)

Why is it the worst? Because no one, not even the guys who designed it, know the proper way to install it, configure it, use it, or fix it when something goes wrong. Every MS document out there conflicts with every Technet blog by an expert that set it up. Any every Blog conflicts with someone elses blog. Add Windows 2008 to the mix and now even the errors reported by the program don’t even know what to say. I had a DPM rig fail 300 times in 4 hours while trying to backup Exchange 2007 with “Unknown Internal Error”. Are you kidding me?

Add that to the fact that there is no “update” method, only a link to a KB which now points to a feature pack, and then another link that points to a “hotfix rollup”… What ever happened to “Click here to update this product using Microsoft Update”?

Enough bitching, here is what I did to install DPM 2007 on a Windows 2008 Server and protect an Exchange 2007 SP1 CCR Cluster on Windows 2008 servers.

Other specifics:

• This is x64 everything
• Update Rollup 3 is on the Exchange nodes

On the DPM Server:

1. Add Windows Powershell feature
2. Install IIS role (Add the required dependencies and add ALL of the Role Services, yes, EVERYTHING)
3. Install SIS (type “ocsetup.exe SIS-Limited” in the command prompt)
4. Reboot the bitch
5. Install the DPM 2007 Software
6. Reboot the bitch
7. Install the Feature Pack (http://www.microsoft.com/downloads/details.aspx?familyid=AD5CD1A2-9B87-4A2C-90A2-9DBAF1024310&displaylang=en)
8. Install the Hotfix Rollup 2 (http://www.microsoft.com/downloads/details.aspx?familyid=8EEFDE76-1A94-4096-BA3A-829EB954E422&displaylang=en)
9. Reboot the bitch
10. Add a custom incoming firewall rule allowing ALL programs, ports, etc… from the remote IPs of your Exchange 2007 nodes, the Cluster IP and the CMS (Exchange Virtual) IP.
11. Copy the ESE.DLL and ESEUTIL.EXE files from one of your Cluster nodes to C:\Program Files\Microsoft DPM\DPM\bin
12. Start the IIS Manager and navigate to Report$MS$DPM2007$
13. Open Handler Mappings and click Edit Feature Permissions
14. Make sure “Script” is checked

On the Exchange cluster nodes

1. Add a custom incoming firewall rule allowing ALL programs, ports, etc… from the DPM server

Now, go into DPM and install the Agents remotely using the DPM Console to your Exchange nodes. Then add a protection group, select your Virtual Server, select your Storage Groups, and you are off to the races.

Ciao

MS Network Load Balancing - Not always the solution…

Allow me to set the stage…

 You have a few big bore terminal servers, handling about 400 users per day. All 3 of those server are configured to unicast spec with the best practices of MS. All 3 of those servers also sit on the same switch in your network infrastructure. You decide to move a couple of the nodes to another datacentre (on your giant flat network, in the same subnet) and all of a sudden, a portion of your clients are getting errors and are unable to connect to the cluster name. Connecting the the dedicated IP or backend adapter works just fine.

 I love the words “Unicast mode works with all routers and switches” because it puts this massive false sense of security in your head.

 Allow me to correct that statement… “Unicast DOES NOT mean your NLB cluster will work with your infrastructure” and here is the example from my environment.

So we now have the following configuration:

RDPCLUSTER (192.168.0.201) - Unicast Mode - MAC 02:BF:xx:xx:xx:xx

All of the nodes are on the same subnet and VLAN. The clients accessing the nodes are coming from different VLANS.

So in order to prevent the switches from learning the MAC of the cluster, the nodes send outbound packets with their custom MAC address. When you get on the edge switches and look at ARP table, the address does not exist. This is a good thing. The switch actually learns the custom MAC for each node.

 However, when a client makes it’s way in and wants to access the gateway it sends the ARP to locate the MAC of RDPCLUSTER, and it hits a router. The router has no flippin clue where it is and then floods out an ARP. The problem is, each of the nodes in the cluster gets this and the 3 replies comes back from the nodes. The router then caches the location of RDPCLUSTER. This is a BAD thing. Here is an example:

So the ARP hits all the nodes and NODE2 fires back with the reply. The router then caches the ARP saying that RDPCLUSTER is on SWITCH2. So now, another new client comes in and does the same thing, looking for RDPCLUSTER. This time, NODE1 gets the duty of responding. However, the router wants to toss all the packets for RDPCLUSTER at SWITCH2, not SWITCH1 because of the intial cache. You can see this by creating a network trace and watching the TCP SYN packets leaving the clients, but no TCP SYN/ACK coming back from the cluster, because it never got to NODE1. And since it is only when connections get load balanced to NODE1, you won’t always see the problem, it will only pop it’s head up sporadically.

If you are using the “Router on a Stick” topology, then this is never a problem because all of your hosts are theoretically in the same “location”, so the router can cache it’s little heart away.

This also has alot to do with your actual network equipment and topology, and in my instance, we use Foundry hardware. Foundry equipment caches the PHYSICAL port number in the router, not the Virtual Interface, which is why people with Cisco equipment won’t see this behavior. This basically prevents us from putting NLB servers on Geographically separate physical segments, because we use a mesh/web routing design for high availablity. Maybe this can help someone else, and maybe not.