2

Version MySQL 5.7.2

I'm working with dummy data with int columns.

time: 1-10mm

products: 70,000 random ints with another 30,000 that are dupes from the 70,000

volume: ranges from 500 - 1000

price: ranges from 10 - 50 though the price stays within bounds of 1-5 diff for each product row data.

from the above, 10mm rows are created by randomly selecting a product and generating the required row data

running a range query like...

select * from productdata where product >= 1500 and product <= 2000

takes about 4 seconds.

When I add an index on product using...

create index productindex on productdata(product)

The query now takes about 30 seconds. Time is the only unique column in the table but setting that as the primary key does not help either.

On SQL Server and PostgreSQL I don't see the same issues with the same data and query using a non clustered index in each. I only really have experience with writing queries for SQL Server so a bit perplexed by this. I tried PostgreSQL as well to have another db to compare to.

All databases are the latest stable versions available.

Action Output (I had to decrease the range as the original query was taking too long)...

Action Output

Without Index..

Without Index

With Index..

With index

Table status...

table status

Buffer...

enter image description here

Explain Select...

enter image description here

Show create table...

CREATE TABLE `products` (
  `time` int(11) DEFAULT NULL,
  `product` int(11) DEFAULT NULL,
  `quantity` int(11) DEFAULT NULL,
  `price` int(11) DEFAULT NULL,
  KEY `productindex` (`product`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8

Trying with time and product as key...

enter image description here

  • Also add the output of SHOW CREATE TABLE productdata; and the version of MySQL (is it 5.7 or something else?) – ypercubeᵀᴹ Nov 19 '17 at 13:10
  • I've added what you guys asked for. It's version 6.3 – pad11 Nov 19 '17 at 13:40
  • There is no version 6.3 of MySQL. 5.7 is the latest GA release. 8.0 is a release candidate – a_horse_with_no_name Nov 19 '17 at 13:48
  • Sorry 5.7. I gave the version of workbench – pad11 Nov 19 '17 at 13:53
1

MyISAM? InnoDB? What is the cache size? (SHOW VARIABLES LIKE '%buffer%';) Did you run the query a second time (to cancel out the affect of caching)? Do you have TEXT or BLOB columns? (Please provide SHOW CREATE TABLE.) How much RAM? How big (GB) is the table (SHOW TABLE STATUS).

Probably... Definitely... The cache for the data was not large enough to hold the products in question. And, since the products are mostly "random", there is a lot of I/O.

Here's what happens in any(?) database vendor for that query:

  1. Locate the 'row' for 1500 in the index.
  2. Scan forward through the index until 2000. (Due to BTree organization in MySQL, this is quite efficient.)
  3. For each row, reach over into the "data" to get all the columns (SELECT *). (This is where vendors vary. And "Engines" in MySQL will vary.) Since the data rows are in one order, but the index rows are in a different order, MySQL will be effectively doing 'random I/O' to get the rows. (MySQL fetches the rows as needed. Other vendors may sort the row-addresses first -- perhaps a benefit, perhaps a cost.)
  4. For MySQL with InnoDB and large TEXT/BLOB columns, those columns may be stored elsewhere, thereby necessitating an extra I/O. (Hence the admonition not to use *, but to spell out only the necessary columns.)

But...

If "too much" of the table needs to be fetched -- in particular more than about 20% of the table has product in the desired range -- MySQL will do a table scan instead of using the index. (I don't know about non-MySQL vendors.) This optimization is usually beneficial, but sometimes is a mistake. The EXPLAIN SELECT .. would tell us which it did.

InnoDB "reaches into the data" via the PRIMARY KEY which is "clustered" with the data. So it is a BTree-probe to find each row. MyISAM has a byte address into the data file, so it is a fseek. Other vendors work differently.

So... Is the comparison "fair"? Do you have the PK "clustered" with the data on all vendor tests? (Etc)

And I did not get into the caching details. This is probably a major component of the slowdown.

I'll fill in more details after you provide more details.

After details

You are using InnoDB, not MyISAM (good). innodb_buffer_pool_size = 8M is much too small. For 16GB of RAM, recommend 11G. But even 1G would show a significant speedup since the table is smaller than that.

You say you are running MySQL 5.7, yet the 8M default contradicts that. Did you override the default? Please provide the version (SELECT @@version;).

It seems that there are no big TEXT or BLOB columns, so my comments on such do not apply.

Bottom line: To get a 'fair' comparison, increase the buffer_pool setting for MySQL. This is the most important tunable for performance. It is not automatically set because it depends on the amount of available RAM.

  • The default on Postgressql is non clustered which should be the same as MySQL. I've added the details you've asked for. – pad11 Nov 19 '17 at 13:48
  • Running on my desktop so 16gb ram – pad11 Nov 19 '17 at 13:56
  • 1
    Postgres doesn't have clustered indexes. All tables are heaps in Postgres. In MySQL all InnoDB tables are clustered, no heaps. – ypercubeᵀᴹ Nov 19 '17 at 14:19
  • 1
    Yes, the ultimate improvement would be to have the PK start with product. This is because of the "clustering" of the PK with the data. – Rick James Nov 19 '17 at 14:53
  • 1
    @pad11 - I am struggling to figure out how you got 8M -- That was the default until about a decade ago. 5.7 has a default of 128M, which is "too small" for most machines today. (I don't know about SQL Server, Postgres, etc.) – Rick James Nov 19 '17 at 23:40
0

buffer pool is small,Set larger values。

SELECT @@innodb_buffer_pool_size; SET GLOBAL innodb_buffer_pool_size=402653184; SELECT @@innodb_buffer_pool_size; select * from productdata where product >= 1500 and product <= 2000

  • Could you clarify how this differs from Rick James' answer? And, will this permanently modify the buffer size, or just for the current session? – RDFozz Nov 22 '17 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.