Home » Performance » MySQL » Any suggestions to tackle a slow query on a large table with composite PK?
Any suggestions to tackle a slow query on a large table with composite PK? [message #1998] Fri, 05 October 2007 14:48 Go to next message
JGilbert  is currently offline JGilbert
Messages: 31
Registered: May 2007
Location: Chicago, IL
Member

Actual query:
SELECT SQL_NO_CACHE country_abbreviation, country_name, region, city, isp, latitude, longitude FROM ip2location WHERE ip_first <= 3335238065 AND ip_last >= 3335238065; 4.6 seconds

Profile:
Status	Time
(initialization)	0.000004
checking query cache for query	0.000073
Opening tables	0.000014
System lock	0.000022
Table lock	0.00001
init	0.000032
optimizing	0.000015
statistics	0.000213
preparing	0.000131
executing	0.000011
Sending data	4.651696 <--- What? it's 1 record
end	0.000047
query end	0.000008
freeing items	0.000019
closing tables	0.000011
logging slow query	0.000004

Explain:
1	SIMPLE	ip2location	range	PRIMARY,ip_last,ip_first	PRIMARY	4	NULL	1886656	Using where

Research:
SELECT SQL_NO_CACHE ip_first FROM ip2location WHERE ip_last >= 3335238065; .0007 seconds
SELECT SQL_NO_CACHE ip_last FROM ip2location WHERE ip_first <= 3335238065; .0007 seconds
SELECT SQL_NO_CACHE count(ip_last) FROM ip2location WHERE ip_first <=3335238065; 2,251,823 rows (1.3 seconds)
SELECT SQL_NO_CACHE count(ip_first) FROM ip2location WHERE ip_last >= 3335238065; 1,406,113 rows (.65 seconds)
SELECT SQL_NO_CACHE max(ip_first) AS first_max FROM ip2location WHERE ip_last >= 3335238065; 4278190080 (1 second)
SELECT SQL_NO_CACHE min(ip_first) AS first_min FROM ip2location WHERE ip_last >= 3335238065; 3335237632 (.69 seconds)
SELECT SQL_NO_CACHE max(ip_last) AS last_max FROM ip2location WHERE ip_first <= 3335238065; 3335239167 (1.4 seconds)
SELECT SQL_NO_CACHE min(ip_last) AS last_min FROM ip2location WHERE ip_first <= 3335238065; 33996343 (1.37 seconds)


The table itself has a composite PK on ip_first and ip_last. I wasn't sure if it was not correctly using the keys because it was a composite, so i added a normal index on both ip_last and ip_first separately. Initially I thought I saw a 2-4 second improvement in query performance, however, that was months ago. to my knowledge queries like this have not ever run under 4 seconds though. This is a query which runs when someone first visits the site which causes a 5-10 second delay in some cases. It would be great if I could find a way to tune this down somehow to around or below 1 second.

Thanks in advance for any help!
Re: Any suggestions to tackle a slow query on a large table with composite PK? [message #1999 is a reply to message #1998 ] Fri, 05 October 2007 14:52 Go to previous messageGo to next message
JGilbert  is currently offline JGilbert
Messages: 31
Registered: May 2007
Location: Chicago, IL
Member

The schema, for reference:

CREATE TABLE IF NOT EXISTS `ip2location` (
`ip_first` int(10) unsigned NOT NULL,
`ip_last` int(10) unsigned NOT NULL,
`country_abbreviation` varchar(2) NOT NULL default '',
`country_name` varchar(42) NOT NULL default '',
`region` varchar(45) NOT NULL default '',
`city` varchar(37) NOT NULL default '',
`latitude` float default NULL,
`longitude` float default NULL,
`isp` varchar(103) NOT NULL default '',
PRIMARY KEY (`ip_first`,`ip_last`),
KEY `ip_last` (`ip_last`),
KEY `ip_first` (`ip_first`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Re: Any suggestions to tackle a slow query on a large table with composite PK? [message #2000 is a reply to message #1998 ] Fri, 05 October 2007 17:52 Go to previous messageGo to next message
sterin  is currently offline sterin
Messages: 324
Registered: March 2007
Location: Sweden
Senior Member
The problem is the "WHERE first key part ... AND second key part ..." which MySQL handles by performing a range scan on the entire index which in your case is the primary key which in InnoDB basically means a table scan since data is stored in leaves.

Some info needed to try to solve your problem:
Do you have ip_first,ip_last combinations that overlap anywhere?
If you don't then by tweaking your data a bit (if you have "holes" in you sequence) you can for example write your query like this:
...
WHERE
  ip_first >= 123456
ORDER
  BY
    ip_first DESC
LIMIT 1;

Which MySQL will be able to optimize to use only the ip_first index and return a result very fast.
Re: Any suggestions to tackle a slow query on a large table with composite PK? [message #2012 is a reply to message #1998 ] Sat, 06 October 2007 23:21 Go to previous messageGo to next message
scoundrel  is currently offline scoundrel
Messages: 58
Registered: August 2006
Location: Toronto, ON, Canada
Member

Try to add a key: first_last(first, last) and force mysql to use it. Then show us an explain results along with timing.

Notice: to force mysql to use some key, try USE INDEX(keyname) after the table name.


Alexey Kovyrin, MySQL Performance Expert
MySQL Performance Blog
MySQL Consulting
Re: Any suggestions to tackle a slow query on a large table with composite PK? [message #2016 is a reply to message #2012 ] Sun, 07 October 2007 03:54 Go to previous messageGo to next message
JGilbert  is currently offline JGilbert
Messages: 31
Registered: May 2007
Location: Chicago, IL
Member

That would be essentially the PK since it is a composite between ip_first and ip_last. I deleted the two other indexes for this set, but left the PK

Status Time
(initialization) 0.000005
checking query cache for query 0.000195
Opening tables 0.000032
System lock 0.000025
Table lock 0.000016
init 0.000043
optimizing 0.000019
statistics 0.000138
preparing 0.000087
executing 0.000015
Sending data 6.511645
end 0.000044
query end 0.000007
freeing items 0.000017
closing tables 0.000009
logging slow query 0.001079


1 SIMPLE ip2location range PRIMARY PRIMARY 4 NULL 1864658 Using where


(1 total, Query took 4.7419 sec)
Re: Any suggestions to tackle a slow query on a large table with composite PK? [message #2018 is a reply to message #1998 ] Sun, 07 October 2007 08:47 Go to previous messageGo to next message
sterin  is currently offline sterin
Messages: 324
Registered: March 2007
Location: Sweden
Senior Member
Did you read my post three posts up?
Re: Any suggestions to tackle a slow query on a large table with composite PK? [message #2019 is a reply to message #2016 ] Sun, 07 October 2007 09:09 Go to previous messageGo to next message
scoundrel  is currently offline scoundrel
Messages: 58
Registered: August 2006
Location: Toronto, ON, Canada
Member

JGilbert wrote on Sun, 07 October 2007 03:54

That would be essentially the PK since it is a composite between ip_first and ip_last.



Oops! Sorry - i missed this fact that you already have such index.


Alexey Kovyrin, MySQL Performance Expert
MySQL Performance Blog
MySQL Consulting
Re: Any suggestions to tackle a slow query on a large table with composite PK? [message #2022 is a reply to message #2000 ] Sun, 07 October 2007 10:48 Go to previous messageGo to next message
JGilbert  is currently offline JGilbert
Messages: 31
Registered: May 2007
Location: Chicago, IL
Member

sterin wrote on Fri, 05 October 2007 17:52


Some info needed to try to solve your problem:
Do you have ip_first,ip_last combinations that overlap anywhere?



SELECT COUNT(*) AS total_rows, ip_first
FROM ip2location
GROUP BY ip_first
HAVING total_rows >1;
0 results

SELECT COUNT(*) AS total_rows, ip_last
FROM ip2location
GROUP BY ip_last
HAVING total_rows >1;
0 results

SELECT COUNT(*) AS total_rows, CONCAT_WS('_',ip_first,ip_last) as first_last
FROM ip2location_disk
GROUP BY first_last
HAVING total_rows > 1;
0 results (as you'd expect from a pk)

I didn't realize before this that ip_first and ip_last were also unique on their own. Need to test more queries.
Re: Any suggestions to tackle a slow query on a large table with composite PK? [message #2023 is a reply to message #2022 ] Sun, 07 October 2007 11:03 Go to previous messageGo to next message
scoundrel  is currently offline scoundrel
Messages: 58
Registered: August 2006
Location: Toronto, ON, Canada
Member

JGilbert wrote on Sun, 07 October 2007 10:48

sterin wrote on Fri, 05 October 2007 17:52


Some info needed to try to solve your problem:
Do you have ip_first,ip_last combinations that overlap anywhere?



I didn't realize before this that ip_first and ip_last were also unique on their own. Need to test more queries.


AFAIU, they should be unique, because in generic GeoIP database one given IP should be assigned to one ISP/location only.


Alexey Kovyrin, MySQL Performance Expert
MySQL Performance Blog
MySQL Consulting
Re: Any suggestions to tackle a slow query on a large table with composite PK? [message #2024 is a reply to message #2018 ] Sun, 07 October 2007 11:06 Go to previous message
JGilbert  is currently offline JGilbert
Messages: 31
Registered: May 2007
Location: Chicago, IL
Member

sterin wrote on Sun, 07 October 2007 08:47

Did you read my post three posts up?


I've run several queries with different ips and the results are identical to ip_first / last combo WHERE only you're right, mysql optimized out the difference.

.00007 seconds is much improved. Thanks for helping me tackle this Smile
Previous Topic:Advice pls my.cnf settings for my system
Next Topic:Changing datatypes of foreign keys
Goto Forum:
  


Current Time: Sun Jul 5 17:33:26 EDT 2009

Total time taken to generate the page: 0.01429 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 2.7.7.
Copyright ©2001-2007 FUD Forum Bulletin Board Software

MySQL is a trademark of Sun Microsystems.
InnoDB is a trademark of Oracle Corp.

Percona Performance Forums are a service of Percona, Inc.
Not affiliated with Sun Microsystems or Oracle Corp.