Members Members   Search Search   Help Help   Register Register   Login Login   Home Home
Home » Performance » MySQL » Design Very large database(15 TeraByte)
Design Very large database(15 TeraByte) [message #953] Tue, 20 March 2007 02:54 Go to next message
kap_gemini is currently offline kap_gemini
Messages: 1
Registered: March 2007
Location: India
Junior Member
Hello All,
I am in a process of using a 15 Terabyte of database on MYSQL 5.1
This is not a true OLTP nor a datawarehosuing stuff. We need to query say 20 times an hours only.

The way i am planning to design this is
1) We'll have 10 nodes and each node will have 5-6 databases
2) Engine would by MyISAM and we are planning to use Partitioning and it will be partitioned on a field A which is main searchable field.
3) This is going to be a semi normalized database with 10 tables.
4) Data load would be done by load data infile(We have a complete .txt file 3 GB in size, we are writteing a C parser which will parse the file and create 10 files(one for each table)

What consideration should we need to take to carry out such huge database.
* Search is the major critaria in this application
** Data loading:- is this thr right approach (Data load would be a daily affair)
*** What change are important as far as My.ini is concern(For cache and memeory ) point of view. All nodes have a shared SAN drive, and memeory on each node is minimum 2 GB.

Any help owuld really make my life easier.

Kapil
Report message to a moderator
Re: Design Very large database(15 TeraByte) [message #988 is a reply to message #953 ] Mon, 26 March 2007 08:11 Go to previous messageGo to next message
migandhi is currently offline migandhi
Messages: 9
Registered: December 2006
Junior Member
i had done optimization for reading ie. for select queries.
the query read buffer should be set to a reasonably high value so that your reads happen faster.
and regarding query cache : if the same queries are fired repeatedly then even set that value reasonably high .
also note your system should have enough RAM.
but i myself am a junior. so reading times can be significantly improved.
but caution should be taken if you have other critical applications running on the same system.
as increasing the buffer size may also impact the RAM.
Report message to a moderator
Re: Design Very large database(15 TeraByte) [message #1036 is a reply to message #953 ] Sat, 07 April 2007 07:04 Go to previous messageGo to next message
joemasters is currently offline joemasters
Messages: 6
Registered: April 2007
Location: Asheville, NC USA
Junior Member
If you're going to load data once, and aren't planning to modify via MySQL, you should consider compressing/packing the MyISAM tables.
Report message to a moderator
Re: Design Very large database(15 TeraByte) [message #2075 is a reply to message #1036 ] Tue, 16 October 2007 16:34 Go to previous message
abdul is currently offline abdul
Messages: 1
Registered: October 2007
Junior Member
Have you completed the implementation yet. How is the performance?

We are planning to implement something similar and would like to know MySQL performance for large databases.
Report message to a moderator
Previous Topic:replication has "selective hearing"
Next Topic:innodb_buffer_pool_size possibly not being used?
Goto Forum:
  

[ Generate printable PDF ] [ Syndicate this forum (XML) ] [ ]

Current Time: Fri Jul 10 04:31:09 EDT 2009

Total time taken to generate the page: 0.03684 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 2.7.7.
Copyright ©2001-2007 FUD Forum Bulletin Board Software

MySQL is a trademark of Sun Microsystems.
InnoDB is a trademark of Oracle Corp.

Percona Performance Forums are a service of Percona, Inc.
Not affiliated with Sun Microsystems or Oracle Corp.