[BBLISA] co-location

[Dean Anderson]

On Wed, 31 Mar 2004, Stephen Reppucci wrote:

> 
> Example: We were in the cage adding a couple of servers one day, when a
> NOC person in search of a clue wandered in, announcing that she had
> received a phone call FROM A USER claiming that they couldn't access
> boston.com from their home.  The clue-seeker told us she had come down
> to "reboot the boston.com server". I shudder to think what switches she
> might have tried to push had we not happened to have been there.

Yikes!  That should be in the colo contract.  Your colo operator shouldn't
touch anyones equipment unless they offer remote hands service, and the
remote hands should only do what you ask them to do.  Further, other
customers shouldn't be touching equipment that isn't theirs either. You
should expect to see serious penalties in your colo contract for such
violations. (eg:  termination of contract, barring from facility,
forfeiture of equipment, etc)

> I don't believe boston.com has moved out of that facility though,
> although I haven't spoken with people there in 6 months or so.
> They also claim to be one of the only facilities in the area that is
> supplied by two different power grids; I don't know if this claim is
> accurate anymore.

They aren't the only ones, but it is somewhat rare. The information on the
power grid is now classified, I'm told, so it is getting hard to figure
out in advance.

However, the building housing our facilities in Charlestown (Schrafft
Center) is also on two different grids. This was not uncommon for large
factories, and sites made on former large factory sites probably have dual
power available. There are also some government and financial sites with
triple power.

Beware of the story "We have dual power grids", in lieu of generator
backup. They always leave off the "in lieu of " part.

In my experience, with the _sole_ exception of 9/11**, I've not found it
to be terribly useful, and sometimes its a disadvantage: The Genuity/GTE
site in waltham had dual power and transfer switches, and once we were
_brought down_ as a result.  Maintenance on the redundant service somehow
triggered a failure in the primary, and the whole site went down for
several hours.  There were generators, but not enough to run the whole
sites datacenters.  Oops.

I've found that a natural gas powered generator (again with the sole
exception of 9/11) to be the best backup.

Fortunately, the Schafft Center doesn't have transfer switches to cut from 
one grid to another, but rather half of the building is on each grid. Our 
equipment is on one grid, backed by UPS and a big generator.

Oh, and that's another thing to ask: By "two power grids" do they mean 
that they have transfer switches or do they mean that only half the 
building will be affected?

		--Dean

** On 9/11, our NYC facilities were located in space that was formerly the
Federal Reserve Bank in Manhattan, and the building was mostly occuppied
by the City of NY.  The Federal Reserve bank had three power grids, and
direct fiber from MCI, which meant we had direct fiber from MCI.  While
only a block and a half from the WTC, we were not affected at all by the
collapse, though our upstreams' facilitiies in Manhattan, eventually
failed due to dust.  If we had a MCI TDM connection back to Boston, we
wouldh't have been down at all.  UPS's didn't even cut in.  For a while,
the Federal Reserve buildings were the only one's with lights on, with
power coming from the two other grids, not generally local to Manhattan.
As you already know, the dust from 9/11 clogged the airfilters of the
generators, and so they shutdown.  It then took about a week or so before
we could see our equipment in NYC. But when the connections came back up,
our equipment was there, running like nothing happened, with 90 some days
uptime.  However, 9/11 is an extremely unlike scenario.  Generators with
long run capabilities are still the best solution.