Software

Ask something
Discussion Locked

Why are there files?

0 helpful
11 replies

by epepke 1 view

Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

OK. So we have disks, which have sectors of fixed length. Part of an OS, VTOS or something, takes these physical sectors and maps them onto logical sectors of fixed length.

Then it maps these logical sectors to things called files, which we canpretend are like glorified tapes or decks of cards. Extremely elaborate storage and collection schemes make it so that these files can vary arbitrarily in length.

Then we take these files, and use a database program with more elaborate techniques to make them look like tables of logical records of fixed length.

Does this seem a bit odd and bass ackward to anybody else?

11 replies

by ebob

I work it from the other direction. We have files. Some of these might represent databases (tables with rows and columns, etc.), which is really just an organisation of the data inside the file, imposed by an "application" (the DBMS). Some files aremore "linear", such as TEXT, HTML (glorified text), PDF, most images, etc. When the correct application processes a specific file, we get the desired results.

Then on the physical side, we elect to store the files on some sort of media, typicallydisks. To make effective use of the available space on the disk, we impose an organisation on this media, which is independent of the type of data. The application should not care about sectors, heads, cylinders and like that. The disk controller isessentially running a mini-operating system to move data on and off of the media, as well as keep track of available space (since disk space is a finite resource).

It seems pretty OK.

by epepke

say, the Palm OS mechanism of storing records in a primitive form of a database as a basic concept in the OS. The Palm OS can do stream files as well, but it does it in terms of databases rather than doing databases in terms of stream files.

by admin

and didn't reply, although it seems very interesting. Yes, it seems difficult and could be greatly improved upon (as in a database style format or?) but the part I don't know, and therefore don't have a lot of comment on, is the history about how itgot this way. I assume it is because no one thought about it earlier and it's hard to change now, but I really have no idea. Does anyone know the history?

Actually, there is a retired guy down the street from me and we talk computers occasionally. He ran some sort of 10-bit system on magnetic drums that sounds quite physically complex. Perhaps if you don't know I'll ask him, 'cuz I sure don't know.

by generalist

A number of years ago IBM had Direct Files or something like that. They were used a lot in bill of material processing and involved allocating a place on disk for every record in the file. You then accessed the data via record number. It kind of resembled the idea of fixed length records at fixed locations on the disk.

Of course, that was when a 20 MB disk pack was the ultimate in removable random access media.

I suspect that somebody will eventually notice the fixed length pattern andcome up with a way to exploit it for really fast database processing. But it won't start happening soon. There are too many other improvements that can be made to speed up access.

Of course if you're dealing with text, spreadsheets or graphics,fixed lengths are a pain.

by epepke

That sounds like the IBM 501, although I'm not at all sure about the number. It read instructions directly from the drum. You couldn't put your instructions on the drum sequentially, because the CPU was too slow to execute an instruction before the next one came up under the read head, so you had to stagger them all over.

by mckaytech

Oh, not THAT kind of file....

Actually, being able to efficiently create variable length records independent of the underlying media seems to me to be a huge evolutionary advantage in data storage. No two of my computers at home have the same hard drive geometry but my databases are freely transportable between them (and between hard drives, CD-ROM, floppy, remote storage, RAM disk, etc.).

There's no question in my mind that from a pure performance standpoint, the most efficient database/OS would read and retrieve all data directly to and from the storage medium in fixed-length records that co-incided with the sector size. But that performance would come at the expense of flexibility and efficiency.

In an optimized micro-application or appliance, I might well design such a system. But I do love those BLOB fields that bear no resemblance to any rational sector size!

paul

by it-sage (bschirf)

The reason it seems to complicated is because of the need for flexibility.

What the original poster didn't realize is that the physical storage he\she described was for IDE drives. What about the storage method for reel to reel tapes? What about the storage method for the crystaline cubes that IBM created for the DoD a couple of years ago?

The answer... it doesn't matter. Let the storage medium figure out the best way to store data on that medium, and let the OS decide what the best way to handle data processing.

I expect that the poster would also think that having a transmission or rear differential in a car is just way to complex. Why not tie the crank shaft directly to the axle?

Over the course of computing history, thetechnologies that were flexible were the ones that survived. Otherwise, APPLES and AMIGA'S would be everywhere. But now their just specialty systems that are doomed to eventual extinction.

by admin

transmission than crank shaft directly to axle, I think, to use the car analogy. It's just a guess on my part, and I could be off on this projection.

I do agree though, that if it's technically good it probably won't make it, and just to add, I've been playing with my Atari 800xl and ST again lately and boy, talk about great technology that went extinct.... their systems were works of art, and the windows interface on the ST was amazingly fluid.... Amiga's, Apple's, Commodores, Atari's they were all ahead in their day..when you look back, it's amazing that the xt became what we use today. :>

by ebob

The "file" concept is after all, not what is going on the disc, or whatever the physical storage medium happens to be (no, not really). The representation of all those 0's and 1's as files is just for our reference (and the reference of our applications).

But by giving the high level representation as "files", with the analogy of filing cabinets, drawers, folders, etc. to the user side of things, we are provided with a consistent view of the data. And it then does not matter if we're lookingat a Hard drive, DVD, flash card, tape, bubble memory, or whatever.

by epepke

But, from the point of view of databases, that convenience is gone *anyway*. Breasts, too, are a great evolutionary advantage, but they aren't much use on a nun. Try moving your Oracle database around without effort.

I also think that people have fixated on the fixed size aspect of my original query. (Both puns intended.) This is the least important aspect. There isn't any reason that you can't have a variable-length data structure made out of fixed-length data structures, because that's what happens in standard file systems *anyway*.

However, in conventional databases, most of the important information that needs to be searched *quickly* isn't in an amorphous structure. For this important majority, a lot of improvements could be made by accomodating that kind of information directly in the operating system.

I happen to think that the Palm database system itself is an evolutionary step, or at least a mutation. Anybody who has written code for it or anyone who has used a Palm and not had to wait knows it. Too bad it probably won't survive.

by generalist

The only problem with the consistent view of data being in files is that a lot of database packages call them tables...

I sometimes wonder if this is an effort to create a jargon to increase the 'mystique' associated with database design. I haveyet to hear a good reason why the inconsistency between 'file' and 'table' exists, especially if the file is designed well.

This discussion is closed to new replies.