Starting from the assumption that application and/or website have dynamic data,  in order to have quick and easy access to the data you would save it into a database.

Cosidering a low budget project you would probably start with a relational database (eg. MySQL or PostgreSQL etc.) and a comodity hardware.

More data usually means more delay in retreiving it,  more traffic means more retrieves of probably same data.

Here are my recommended optimizations from low to high traffic and small to large data sets
I will assume you use Linux/PHP/MySQL
FROM SQL SIDE

1) Use explicit mentioning of the columns you need to extract on select statements. People make the common mistake “SELECT * FROM table” when they actually only need part of the information retrieved

2) Use FIXED row size instead of DYNAMIC whenever possible. This gives a hint to your database server telling it where on the disk will find a specific record, in short words record 50 can be found at position 50 x rowSize , when row size is dynamic this is not true anymore.

3) Use INDEX on one or more columns. Indexes avoids reading the entire content of a table, simple example would be: you have a query SELECT column1, column2 FROM table WHERE gender=’male’; Having an index on gender column will tell the database server the location of gender=’male’ rows, full table scan is not necessary anymore just to find all ‘male’. Get documented on how to check used indexes in queryes or how to force explicit index usage. As a final note to indexes, they slow down data insertion in favor of data retrieving. Data insertion for columns that belong to indexes will make them also update the index extending the insert time

4) Use DELAYED INSERTS if you dont need the data right away, the database server will fill a buffer and insert more records at the same time later

5) Use TABLE PARTITIONING if available, it will make the table data being split into more files in a smart way that can speed up, avoiding reading of the entire table content

6) Avoid FOREIGN KEYS usage, you just give your server more headache to verify relations between tables. Eg.: inserting a aproduct in category X will force another scan of table category to check if X is valid data. This is up to you because haveing foreign keys also brings the benefits of having consistent data.

7) Use MEMORY ( or HEAP ) tables. Heap tables are stored in memory and lost with database server restart. They are very fast because the actual data is stored in ram. You could use a heap table as a mirror of an actual one and make it sync when new data is inserted. Careful using heap tables, you have to implement your own mechanism to ensure data was not lost upon reboot

Use LIMIT in your queries. Since PHP has a memory limit usage select with no limit on a table bigger than the memory_limit variable in PHP will make PHP end script execution with an error message. There is another value from mysql side called max_packet_size that will make the query fail if data set is bigger than the value, this case the query fails but the php client can continue execution. Rule applies also for BLOB columns, if you plan to store files (eg. images) inside the database you should consider splitting them in multiple pieces ( rows ) of a size that is less then mysql max_packet_size, while from PHP you should make sure you retrieve only 1 row at a time, process it (output or save to file) and free the memory before you continue to the next record in order not to exceed PHP max memory limit

FROM OS SIDE

1) Enable QUERY CACHE and use it in queryes. Enable query cache from the server side (eg. MySQL can be configured from its configuration file my.cnf  ). Whenever you make a SELECT with query cache enabled the results will be saved separately so the next time you do the same query it will bring back the results from the cache which is lot more faster. As an extra tunning some database servers ( like MySQL) support explicit value in query, eg.: SELECT SQL_NO_CACHE column FROM table which tells the server not to cache the results of this query. You may have reasons not to cache some queryes in favor of others  because the query cache have a size limit and you would want to use it for some queryes that are used more often

2) Set THREAD_CONCURENCY to enable your database to use the full power of all your cpu cores your server has, it is recommended that you set a number double to the number of cpu cores (2 mysql threads per cpu core)

3) set memory_limit in php.ini to a value that suits your needs (eg. if you want to process hi res images ) or else your script would die when limit is reached

4) Use FAST-CGI with a webserver that is known to deliver highest requests/s . PHP can be run as CGI, apache MODULE or FAST-CGI. With CGI being the oldest and slowest, apache webserver came with the idea of compiling php inside the apache as an alternative of loading PHP every time a script awaits execution. Those ages are gone, we now have FAST-CGI where a number of PHP instances are started and wait for a script to be injected for execution. FAST-CGI also comes with another benefit … it is available for other webservers too … unlike apache module that is apache specific. At this exact moment there are a number of web servers that are more performant than  apache, worth mentioning is nginx and lighttpd both open source, so if you want a lightning fast response from your web server you should give a try to any of those 2.

5) Use NOATIME … if your filesystem is a linux ext partition. Atime file attribute will tell you when a file was last accessed resulting in … BINGO … a write for each read . If you do not care when file X was last accesed you should trade this for some extra speed.

FROM HARDWARE SIDE

1) Use HIGHEST HDD RPM or the new SSD HDD. Everybody knows the biggest bottleneck on a server is the hard disk. The higher RPM the higher rate for data retrieval from disk.

2) Use HARDWARE RAID. Oh yeah, double/ triple/ multiply the speed of your read speed using raid. Servers support a mechanism to convert a number of disks into a logical one having the speed increased upto the sum of all. Gathering more hdd into a raid will bring the feature of reading/writing to N number of drives at the same time, raid system will know how to partition the data in a way that continious data will be saved into separate hard drives.
Example
[hdd1   1,4,7 ] [hdd2 2,5,8 ] [hdd3 3,6,9 ]

reading from a file will result reading from more hdd at the same time. What are the benefits ? Supposing each hard-disk can read with 80MB/s if you need to make a select on a database that has 360MB, with 1 hdd it will take 3 seconds to read the data, in a 3 hdd raid this operation only takes 1 second. As simple as that.  Please consult the documentation for available types of raid and the benefits of each

3) Use MEMCACHE. Memcache is client/server system where clients push data to be stored in memory by the server and be retrieved later. At this moment many of high traffic websites use memcache to deliver results (eg. youtube, google), skipping page generation process and saving cpu, disk reads and a lot of time. People use this sometimes even to save entire generated html code. Since everything is saved in memory you definitely need lot of ram and it will definitely give you a real boost.

FROM APPLICATION SIDE

1) Optimize your resources in a way that they do not take lot of space (eg. optimize your images, videos you deliver)

2) Implement CACHING at different levels to avoid queries as much as possible. Verify if a cache exists (eg. product-123.cache ),  serve from cache if exists, generate full or part of a page and save it to cache, remember to remove cache if any relevant data changes.

3) serve static content from different place, eg: serve images used in html from another server or a CDN network and also if possible serve from multiple different hostnames, browser will be downloading them in parallel and not wait for 1 resource to finish downloading before will start downloading the next. You could make subdomains that point to same web folder for example images1.sitename.com and images2.sitename.com

4) ajax and php session do not work well together in the traditional way. Although you can fire asynchronous requests and the web server supports that, 1 request still waits for the other if you  use sessions in php. Why ? Because sessions get saved in a file, session_start() opens the file locking it ( which makes it unavailable for another request ) until script ends execution or session is closed by user.

If your server is still slow you should consider splitting services on multiple servers. Ex: separate server for mysql database, separate server for memcache.

If everything you optimise does not give you the speed your users would like, you have to join the could my friend
At this level, rules change and you just have to forget about relational databases like mysql

Welcome to NoSQL or should i say NoREL ?

It all started with a project i work on having a mysql database with 1mil+ records and 10GB of disk space on a rather slow  server, i failed to setup mysql distributed with federated storage engine and i was looking for a distributed database project and found hypertable. But i code PHP, so what should i do ? Hypertable offers PHP support only through a third party called thrift. It looks like thrift project (started by facebook and now an apache project ) can generate code to access hypertable into your programming language.  Here is my experience compiling thrift on FC 10
[root@thrift /]# yum install gcc-c++
[root@thrift /]# cd /tmp/
[root@thrift tmp]# wget -O thrift.tgz “http://gitweb.thrift-rpc.org/?p=thrift.git;a=snapshot;h=HEAD;sf=tgz”
[root@thrift tmp]# tar zxvf thrift.tgz
[root@thrift tmp]# cd thrift
[root@thrift thrift]# ./bootstrap.sh
./bootstrap.sh: line 24: autoscan: command not found
[root@thrift thrift]# yum install autoconf
[root@thrift thrift]# ./bootstrap.sh
./bootstrap.sh: line 25: aclocal: command not found
[root@thrift thrift]# yum install automake
[root@thrift thrift]# ./bootstrap.sh
configure.ac:44: error: possibly undefined macro: AC_PROG_LIBTOOL
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
configure.ac:26: installing `./install-sh’
configure.ac:26: installing `./missing’
compiler/cpp/Makefile.am: installing `./depcomp’
configure.ac: installing `./ylwrap’
lib/cpp/Makefile.am:24: Libtool library used but `LIBTOOL’ is undefined
lib/cpp/Makefile.am:24:   The usual way to define `LIBTOOL’ is to add `AC_PROG_LIBTOOL’
lib/cpp/Makefile.am:24:   to `configure.ac’ and run `aclocal’ and `autoconf’ again.
lib/cpp/Makefile.am:24:   If `AC_PROG_LIBTOOL’ is in `configure.ac’, make sure
lib/cpp/Makefile.am:24:   its definition is in aclocal’s search path.
test/Makefile.am:30: Libtool library used but `LIBTOOL’ is undefined
test/Makefile.am:30:   The usual way to define `LIBTOOL’ is to add `AC_PROG_LIBTOOL’
test/Makefile.am:30:   to `configure.ac’ and run `aclocal’ and `autoconf’ again.
test/Makefile.am:30:   If `AC_PROG_LIBTOOL’ is in `configure.ac’, make sure
test/Makefile.am:30:   its definition is in aclocal’s search path.

[root@thrift thrift]# ./configure –with-csharp=no –with-java=no –with-erlang=no –with-py=no –with-perl=no –with-ruby=no –with-php=yes –enable-gen-php

……

checking for boostlib >= 1.33.1… configure: error: We could not detect the boost libraries (version 1.33 or higher). If you have a staged boost library (still not installed) please specify $BOOST_ROOT in your environment and do not give a PATH to –with-boost option.  If you are sure you have boost installed, then check your version number looking in <boost/version.hpp>. See http://randspringer.de/boost for more documentation.

[root@thrift thrift]# yum install boost boost-devel

[root@thrift thrift]# ./configure –with-csharp=no –with-java=no –with-erlang=no –with-py=no –with-perl=no –with-ruby=no –with-php=yes –enable-gen-php
….

checking for zlib >= 1.2.3… no
./configure: line 5866: syntax error near unexpected token `MONO,’
./configure: line 5866: `  PKG_CHECK_MODULES(MONO, mono >= 2.0.0, net_3_5=yes, net_3_5=no)’

[root@thrift thrift]# yum install zlib-devel
……
./configure: line 5866: syntax error near unexpected token `MONO,’
./configure: line 5866: `  PKG_CHECK_MODULES(MONO, mono >= 2.0.0, net_3_5=yes, net_3_5=no)’
[root@thrift thrift]# yum install pkgconfig

[root@thrift thrift]# ./bootstrap.sh
[root@thrift thrift]# ./configure –with-csharp=no –with-java=no –with-erlang=no –with-py=no –with-perl=no –with-ruby=no –with-php=yes –enable-gen-php

[root@thrift thrift]# ./bootstrap.sh
configure.ac:44: error: possibly undefined macro: AC_PROG_LIBTOOL
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
configure.ac:26: installing `./install-sh’
configure.ac:26: installing `./missing’
compiler/cpp/Makefile.am: installing `./depcomp’
configure.ac: installing `./ylwrap’
lib/cpp/Makefile.am:24: Libtool library used but `LIBTOOL’ is undefined

[root@thrift thrift]# yum install libtool

[root@thrift thrift]# ./bootstrap.sh
[root@thrift thrift]# ./configure –with-csharp=no –with-java=no –with-erlang=no –with-py=no –with-perl=no –with-ruby=no –enable-gen-php
[root@thrift thrift]# make
../../ylwrap: line 109: yacc: command not found
[root@thrift thrift]# yum install byacc

[root@thrift thrift]# make
/tmp/thrift/missing: line 52: flex: command not found
WARNING: `flex’ is missing on your system.  You should only need it if
you modified a `.l’ file.  You may need the `Flex’ package
in order for those modifications to take effect.  You can get
`Flex’ from any GNU archive site.

[root@thrift thrift]# yum install flex
[root@thrift thrift]# ./configure –with-csharp=no –with-java=no –with-erlang=no –with-py=no –with-perl=no –with-ruby=no –enable-gen-php
[root@thrift thrift]# make
[root@thrift thrift]# make install

1) generate your key (do not enter any password)
# ssh-keygen

2) copy key to remote host
# ssh-copy-id ‘-p 22 user@remote-host’

3) after entering the correct password once, you should be able to ssh to user@remote-host without
a password.

I solved boost dependencies by installing boost and boost-devel

resumed the process

i did not find log4cpp rpm packages for download so i had to compile log4cpp myself,
i wrote a post on how to install log4cpp on fedora core 10

again, resumed the process

it would be great if kfs would have an option to skip the java components but since there is no such option i’ll go ahead and install java using the easy way around – rpm package

resume process

i did not have xfs installed, it did complain finally although did not complain during cmake

i have previously solved the problem with xfs uuid/uuid.h by simply installing e2fsprogs-devel

this is an easy one

building the java components, dont forget to install ant if you dont already have it
if you are not going to create java applications that interact with kfs you can skip this

you can also building the python support, i will not do this now

you may now continue with the next step, configure Cloudstore ( kosmosfs ) release 0.3

here is how i managed to compile and install log4cpp on fedora core 10

1) download and unpack log4cpp

2) download and install patches ( it will not compile without applying them )

3) configure and install

4) you may now finally go back and solve other dependencies

hope this one is solved now i spent a weekend on it until i found the right patches
good luck

To test it, download the vmware container and run it with Free Vmware Player

Steps to follow:
1) download and unpack the vmware container
2) install vmware player if you dont have it already
3) double click the vmx file inside unpacked folder
4)  from vmware player set the network to NAT ( unless you know what you’re doing and wanna configure it manually )
5) at login prompt login with user root and password 1234
6) see what ip address you got from vmware (type ipconfig) in my case was 192.168.144.130
7) edit chunk1 and chunk2 configuration files to reflect the new ip address

mcedit /CLOUD/software/compiled/mfs/chunk1/etc/mfschunkserver.cfg
mcedit /CLOUD/software/compiled/mfs/chunk2/etc/mfschunkserver.cfg

run these commands in the same order as below:

cd /CLOUD/
./start_metaserver.sh
./start_chunk1.sh
./start_chunk2.sh
./mount_mfs.sh

At this moment you should have a mfs storage volume mounted to /CLOUD/storage/ and a file inside named “file_inside_mfs”

if you cannot access /CLOUD/storage/ for some reason you should recheck chunk server configuration and make sure you put the right ip, also check the logs in /var/log/messages

As a point of start i suggest to test the tools located in
/CLOUD/software/compiled/mfs/mount/bin

FYI the actual chunks are saved into /mnt/chunkX/

MooseFS will split your files into chunk and distribute them across connected chunkservers. Assuming that each chunkserver is installed on a standalone computer it is possible to shutdown one or more servers and the filesystem will continue functioning normally.

The number of copies of each chunk they call it goal. Each folder has a goal attribute inherited from the parent folder or changed manually. When a file is created inside a folder it will have the goal of that folder. I think by default the main folder ( / ) gets goal 2.

The package also includes tools to change the goal of files and folders manually, individually or recursive.
When a chunkserver goes off (ie. network failure, hardware failure etc.) the metaserver will internally recalculate goals and start copying under goal chunks  to other chunkservers until all files have the desired number of copies.

Each chunkserver will participate with storage space ( local folders ) and the total available space will be the sum of all participating chunkservers. For the space to be calculated correctly mfs developpers asks that no other files should be saved to those folders and also each folder should be the root mount point of the partition

Example:
if you have partition /dev/sdb2 and mount it in /mnt/sdb2/ you should not save any data in /mnt/sdb2/ and the folder shared to moose filesystem should be /mnt/sdb2/ not /mnt/sdb2/folderX/

visit project home here