General discussion

Locked

expanding ram

By carlbernardi ·
Hi, I have question about expanding ram.

Presently, I am working with some very large databases( in the 500 gig range) and I am searching for an inexpensive way to have the entire databases in memory. Given that my knowledge of hardware is not that in depth I was wondering if someone knew the best way to expand memory to avoid using hardrives which are much slower than ram.
I am considering:
USB
FIREWIRE
BLUETOOTH
and any other possibilities.

Since cheap mother boards can only hold up to a few gigs unless you can afford a super computer, I was thinking about having my databases put on USB flash drives attached through a USB hub * 2 (because must computers have 2 usb ports) adding for example 16gig usb drive * 2 hubs * 4 ports per hub is 128 gigs. I don't think there is any common mother board out there that can hold that kind of ram.

Linux Ubuntu Fiesty is the operating system.

Any suggestions?

Thanks,
Carl

This conversation is currently closed to new comments.

11 total posts (Page 1 of 2)   01 | 02   Next
| Thread display: Collapse - | Expand +

All Comments

Collapse -

I think you're going to slow yourself down.

by CharlieSpencer In reply to expanding ram

If I understand correctly, you're talking about using eight 16Gb USB drives as virtual memory. I suspect this will be much slower than using a single hard drive. I can't speak for Linux, but under Windows virtual memory works best as a contiguous file on a single device. By spreading the virtual memory over multiple devices, you're increasing system overhead and slowing swapping. I'm also not sure about retrieval speeds for USB vs. a hard drive, or of the retrieval speed of multiple USB drives on hubs under continuous load.

Have you considered installing a SATA controller and secondary drive, and dedicating that to virtual memory?

Collapse -

thanks

by carlbernardi In reply to I think you're going to s ...

No, I didn't. Thanks, hat is an option that I can consider.

Collapse -

Plus....

by JamesRL In reply to I think you're going to s ...

Most database engines not only use virtual memory, but also create a temp file.

Make sure the temp file is NOT on the same drive or array the main database is on.

James

Collapse -

Throwing HW at a software problem....

by JamesRL In reply to expanding ram

If you have a 500 GB db then you need to spend a lot of time optmising your databse by designing the right indexes, making sure your schema is optimal etc.

You won't get 500 GB on one computer's RAM.

Flash devices are volatile - if you have a power spike you could lose the whole thing.

USB 2 transfer rates are very similar to those of a SCSI or SATA device.

Best advice would, if you could, store the database on a HD, move to a flash device for temp usage, then write back to the hard drive every X minutes.

But the largest flash drives I've seen for SATA interfaces are 192GB. IBM's largest is I think 9GB.

OTOH, My MB support 16 GB of RAM.

James

Collapse -

good idea

by carlbernardi In reply to Throwing HW at a software ...

thanks for the input

Collapse -

flash drive

by Dr Dij In reply to Throwing HW at a software ...

now coming out combine hard drives with large flash memory. seagate and others are starting to sell these, and they do significant speed increase.

if ever this gets out of your garage, there are several database / datawhse type appliances that speed up a huge amount for sql queries either by very fast hardware, or doing joins in memory. e.g. datallegro, netezza (for larger) or qliktech (for smaller)

and there are BI tools that are totally memory resident. I think yellowfin is one.

also if it conducive to making an olap cube, some types run in memory. they take a while to update and create but run fast till next new batch of data in memory.

you can view the ittoolbox dw- / bi-select groups for some discussions on these.

Collapse -

If I read this correctly,

by cmiller5400 In reply to expanding ram

you want to load up the database into the computers RAM to make it faster? If that is the case, USB drives, Firewire, and bluetooth are useless. Unless you are looking at buying a supercomputer, that amount of memory is impossible. If the database is slow, you need to take a look at how your database and indexes are designed.

Collapse -

Can you archive your database?

by jdclyde In reply to expanding ram

If you were able to peel out the older data into a second database, it would lower the size and access time, but still keep the data available when needed.

Another idea is to make a cluster, but it still won't get you to 500 GB.

Dual CPU's?

Running a hardware raid would also help with access time, but hardware raid controllers are fairly expensive.

Collapse -

I've got a question that's really none of my business.

by CharlieSpencer In reply to expanding ram

What sort of organization has 500Gb databases but can't afford decent hardware to host and access them? Are you a non-profit?

Just wondering.

Collapse -

no problem

by carlbernardi In reply to I've got a question that' ...

It's some thing that I am working on in my spare time and my budget is my own equity and assets. At least for now.

Back to Hardware Forum
11 total posts (Page 1 of 2)   01 | 02   Next

Hardware Forums