[BBLISA] Guidelines for giving full root access to DBAs

[Dean Anderson]

On Mon, 21 Aug 2006, Eric Smith wrote:

> I just had to comment here because my experience has been completely
> the opposite from Dean's.

It doesn't seem all that different, actually. 

> A "Dev" system is one where development is done, and for me that means
> a very specific environment - from kernel params being set right all
> the way up to exactly which version & patch of all databases that are
> installed.  Everything is tightly controlled and keeping the systems
> properly configured is essential.  But we are a hard-core C++ shop,
> across 5+ OSs.

The above all goes into the category of "engineers reconfigure for
different environments".  The engineering group determines what kernel
params, etc that it wants to test with---not the IT group.

Often, one can quickly build machines if they give the parameters.  
Otherwise, one needs to teach them how to installations themselves. And
one needs to be available to help figure out what's wrong.

Production administration is often the method of not fixing what isn't
broke---and dissuading people from attempting to fix not-broken systems.  
But this is frequently the wrong goal for engineering: They need to
break things, and document that to give customers clues before they
break their own systems.

> We definitely do not use systems that are generic boxes.  They aren't
> usually backed up (only critical development data is) but how they are
> configured is well understood and monitored automatically so they can
> be rebuild if necessary.

Right. But that usually isn't the job of corporate IT.  Configuration
monitoring and rebuilding after breakage is one of those things that IT
is well suited to assist with. But that's passive: you just track how it
was previously configured---you don't prevent the "misconfiguration" by
development.  If you're good at such tracking, you'll be able to say
straight off the changes that led to the problem.

"Critical development data", (say CVS servers), is another story from
"Dev and Test".  The "critical development data" needs backup, security,
and stability (the trigger words I gave you).  These servers really are
the family jewels, and engineering usually doesn't get unrestricted
root.

> I would absolutely not want a DBA to have full root on one of my
> systems.  I would be very afraid of one installing a required patch
> that also effected a system library and therefor effected the product
> in a subtle way.  Our product beats the c*rp out of systems and we
> care very much what the system setup is.  We can't have all patches
> applies to a system, to the point that we have not installed some that
> we felt we should probably have (not security related) but didn't feel
> the risk was worth it.

That's what testing is for; Hence the name "Dev and Test".  If you
prevent testing, these bugs aren't found.  Not finding bugs is a great
thing (indeed, the goal) for production, but its bad for "Dev and Test".  
Don't let "production mentality" stand in the way of this work. This
work is no doubt why the arguments that Sharon put up were not accepted.  
Probably, no arguments would be accepted.

> Dean's description might fit his environment, and I could see that a
> DBA having root in that setting wouldn't be as much of an issue.  

Well, my "environment" probably isn't absolutely universal, but it has
fit a number of environments over the last 20 years: The Open Group (you
know: X Window System, OSF/1, Unix, XPG*, CDE, Motif, DCE), Kendall
Square Research (KSR), Hitachi, Open Environment (OEC), GTE
Internetworking, Nokia, Genuity, Netnumina/Keane Architure Services,
Linesider, etc.

Some of these environments had hetergeneous working environments:  
OSF/TOG was, well, too interesting to explain here.  But eg KSR had
several flavors of unix systems: hardware group used HPs, software group
used Sun's, others used PCs, Mac's and Xterms.  Some were heterogeneous
development environments: e.g. OEC made a 3-tier client server product
that supported many different unix hardware platforms (dec, hp, sun, dg,
ibm, +), as well as NT and OS/2, and many different databases on those
platforms.  Hitachi was porting OSF/1 to the mainframe, so I had several
different mainframes to run, including a large VM system, where each
virtual user was a virtual test machine for some developer, and a slew
of sun servers and auspex system. It got interesting with device drivers
for various disk, tape, tape robots, ESCON connections to different
kinds of routers, network media adapters, etc.

My three trigger words were learned from, and worked in, each of these
environments: Backups, Stability, and Security.  These are variables,
not absolutes.

> But in many places that I have worked keeping a tight control on dev
> boxes is absolutely required.

Unfortunately (or fortunately), some of my consulting work has been to
come in and "relieve" sys-admins who resolutely refused to cooperate or
became very "unhelpful" when so overruled.  These are always tricky
situations that require diplomacy skills as much as technical skills.

There certainly are development environments where more access
restrictions are necessary, but they are all identified by the three
trigger words: backups, stability, and security.

> I had sent a message directly to Sharon when she initially posted
> this.  I basically agreed with Jason's comments.  IMO - If the DBAs
> want full control over the system, then I think she should wash her
> hands of those systems and be responsible for as little as possible on
> them (like say backup and restore of the important bits.)  If she
> can't control that things are done reasonable, then she shouldn't be
> responsible for fixing them (she should be available to *help* fix
> them, but that is different than being responsible.)  Politically this
> might be impossible... but it is what I would push for.

It is politically possible. That's one reason why companies hire
consultants.

		--Dean