Re: [LUG] Unix file system folder limits

[Simon Waters <Simon@xxxxxxxxxxxxxxxxxxxx>]

Steve Marvell wrote:
> On Mon, Mar 17, 2003 at 01:35:56PM +0000, Alex Charrett wrote:
> 
> 
>>Let me just add "where values of database do not necessarily equal a 
>>rdbms" to the bottom of my statement :)
> 
> 
> OK :)

Although I'm sure Steve will want to check it is a scalable database,
whatever sort it is :)

I worked on a big implementation of a ProIV based system that used one
of those old fashion CISAM database implementation on HP-UX once.

It would use kernel resources for record locks, taking one inode (128
bytes) sized chunk of kernel memory per available record lock.

Once we had finished the resizing exercise to meet the suppliers
recommended setting for kernel parameters we discovered we needed the
HP-UX large kernel patch to allow us to have a kernel larger than 13MB.

Okay 13MB is not much these days, and some file systems scale to quite
large numbers of files per directory quite well, but my question would
be where is all the data coming from? I'd also ask if it is possible to
split the data in sensible fashion, as I can imagine the next question
being how do I scale the web service beyond one server.

The hierarchical directory scheme Steve mentioned is used in a number of
mail transport agents, part of the scalability issue here is concurrent
access to the metadata, how many updates will happen to the directory
(the files can be spread across multiple disks at the filesystem level),
but the metadata is likely to be more localised. Some *nix filesystems
have very specific semantics regarding when the directory is updated, as
you'll discover if you seek Wietse's opinion on using Linux for big mail
servers.

The hierarchical directories also make it possible to add extra disks
easily, just stop the system, move some directories to the new disk (or
mirrored pair) and start the system, which is easier than many RAID
systems, although it is getting easier to do these things without
interrupting service at all.

RAID-4 use to be of some interest in such a context but I only know one
company using it.

Attachment: pgp00009.pgp
Description: PGP signature