ZFS RAIDz rebuilds are similar to your traditional hardware RAID5/6 in what they have to do. However there are two important differences between rebuilding a hardware RAID5/6 volume and rebuilding ZFS RAIDz1/2 zpool. ZFS being the volume manager and file system can rebuild at the data level. Where hardware controllers work at the block level and don't know if a block is in use or not so all blocks must be rebuilt. This works in favor of ZFS when the volume is not full. However traditional RAID controllers can read the data from the other drives in the volume, compute the checksum/new data, and write it back to the new drive without passing any data back to the host computer. Where ZFS must pass all data from the controller, across the bus, to the memory and cpu to compute the checksum/new data and then has pass the new data back across the bus to the controller and to the new drive. So this is why having a large number of drives in a RAID set can slow you down in this situation. It means moving a lot more data around which tends to cause other bottle necks.<div>
<div><br></div><div>For example in a 16 drive zpool consisting of a single RAIDz set, assuming 100% full zpool and that you actually get 1.5TB of usable space per drive, will require that the system reads 22.5TB of data, and compute and write 1.5TB of new data. Now a 16 drive zpool consisting 3 8 drive RAIDz sets will only require 10.5TB of reads and compute and write 1.5TB of new data. Keep in mind that drives can read data faster than they can write data. So to maximize the rebuild speed with ZFS you really just need to minimize the amount of data the system needs to push around to do the job. Even in a zpool or mirrored drives sets the write to the new drive will be the bottle neck. So you have to balance this knowledge with your data redundancy and capacity goals while staying in your budget.</div>
<div><br></div><div>--</div><div>David.</div><div><div><br></div><div><br><br><div class="gmail_quote">On Fri, Aug 13, 2010 at 3:36 PM, Toby Burress <span dir="ltr"><<a href="mailto:kurin@delete.org">kurin@delete.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im">On Fri, Aug 13, 2010 at 03:11:59PM -0400, Jeff Wasilko wrote:<br>
> On Fri, Aug 13, 2010 at 02:56:49PM -0400, Toby Burress wrote:<br>
> > On Fri, Aug 13, 2010 at 02:42:12PM -0400, David Miller wrote:<br>
> > > What does your zpool look like? Ideally if you're using RAIDz or RAIDz2<br>
> > > then you should be using multiple RAIDz sets in the pool. This way IO is<br>
> > > stripped across the RAIDz sets and any degradation, and recovery, should<br>
> > > only involve the smaller RAIDz set. Which should be relatively quick<br>
> > > depending on the size and type of drives involved.<br>
> ><br>
> > The server that it taking a billion years to resilver does in fact have<br>
> > 15 disks in one big raidz2 pool. The other server has a single pool of<br>
> > three raidz2 arrays of 8 disks each, so hopefully that will yield better<br>
> > recoveries. Although if the bottleneck is reads, then wouldn't it be<br>
> > faster to read from 14 disks than 7? And if the bottleneck is just<br>
> > writes, then wow, I need to buy some different disks next time.<br>
> ><br>
> > Since the load on the machine is 3, and it's doing nothing but<br>
> > resilvering, I suspect the bottleneck is actually the CPU. I don't<br>
> > know a ton about the implementation of ZFS, but I do know it checksums<br>
> > every block. It would be insane for it not to verify those checksums<br>
> > while resilvering, and perhaps it even recomputes them while writes them<br>
> > to the new disk.<br>
><br>
> Did you lose 1 or 2 disks in the raidz2 pool?<br>
<br>
</div>One disk. I actually have replaced two disks at a time and it was about<br>
the same.<br>
</blockquote></div><br></div></div></div>