1

I have gone through the process of adding data nodes to my MySQL Cluster, using the guidelines provided here: https://dev.mysql.com/doc/refman/5.6/en/mysql-cluster-online-add-node-example.html.

The issue I am seeing now is that once I alter a table to reorganize the partition across all data nodes, I don't see the data being placed in the new nodes.

Example:

mysql> alter online table radacct reorganize partition;
Query OK, 0 rows affected, 1 warning (0.49 sec)
Records: 0  Duplicates: 0  Warnings: 1

Notice from the query response that the command only took 49 seconds to complete. The database that this table is in contains about 4G of data, and most of the data resides on this table. So the first red flag I see is the statement taking no time at all to complete.

After running this statement, I can see how the memory usage is on all data nodes. This is what I see:

ndb_mgm> all report memory
Node 2: Data usage is 42%(222763 32K pages of total 524288)
Node 2: Index usage is 31%(41059 8K pages of total 131104)
Node 3: Data usage is 42%(222769 32K pages of total 524288)
Node 3: Index usage is 31%(41065 8K pages of total 131104)
Node 4: Data usage is 0%(1379 32K pages of total 524288)
Node 4: Index usage is 0%(83 8K pages of total 131104)
Node 5: Data usage is 0%(1379 32K pages of total 524288)
Node 5: Index usage is 0%(83 8K pages of total 131104)

I know the management node for the cluster sees all of the data nodes; this is my ndb_mgm -e show output:

ndb_mgm> show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 4 node(s)
id=2    @192.168.251.206  (mysql-5.6.25 ndb-7.4.7, Nodegroup: 0, *)
id=3    @192.168.251.207  (mysql-5.6.25 ndb-7.4.7, Nodegroup: 0)
id=4    @192.168.251.208  (mysql-5.6.25 ndb-7.4.7, Nodegroup: 1)
id=5    @192.168.251.209  (mysql-5.6.25 ndb-7.4.7, Nodegroup: 1)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @192.168.251.201  (mysql-5.6.25 ndb-7.4.7)

[mysqld(API)]   11 node(s)
id=6    @192.168.251.203  (mysql-5.6.25 ndb-7.4.7)
id=7    @192.168.251.204  (mysql-5.6.25 ndb-7.4.7)
id=8    @192.168.251.203  (mysql-5.6.25 ndb-7.4.7)
id=9    @192.168.251.203  (mysql-5.6.25 ndb-7.4.7)
id=10   @192.168.251.203  (mysql-5.6.25 ndb-7.4.7)
id=11   @192.168.251.204  (mysql-5.6.25 ndb-7.4.7)
id=12   @192.168.251.204  (mysql-5.6.25 ndb-7.4.7)
id=13   @192.168.251.204  (mysql-5.6.25 ndb-7.4.7)
id=14 (not connected, accepting connect from any host)
id=15 (not connected, accepting connect from any host)
id=16 (not connected, accepting connect from any host)

Finally, I can confirm that the table I am issuing the command on is using the NDBCLUSTER storage engine:

mysql> SELECT TABLE_SCHEMA, TABLE_NAME
->     FROM INFORMATION_SCHEMA.TABLES
->     WHERE ENGINE = 'NDBCLUSTER';
+--------------+-----------------------+
| TABLE_SCHEMA | TABLE_NAME            |
+--------------+-----------------------+
| mysql        | ndb_apply_status      |
| mysql        | ndb_index_stat_head   |
| mysql        | ndb_index_stat_sample |
| radacct      | radacct               |
+--------------+-----------------------+
13 rows in set (0.14 sec)

I imagined this process taking a while during the execution of the statement, and once the statement finished all of the data would be redistributed, but that does not seem to be the case.

Can someone please point me in the right direction on what else to try with this issue?

Update:

After a suggestion to force a rebuild on the table in question, I found something interesting:

mysql> alter table radacct force;
ERROR 1297 (HY000): Got temporary error 410 'REDO log files overloaded (decrease TimeBetweenLocalCheckpoints or increase NoOfFragmentLogFiles)' from NDBCLUSTER

I went ahead and increased NoOfFragmentLogFiles from the default (16) to 100 in config.ini. Did a restart of the ndb_mgmmd process and the ndbmtd processes so the cluster would see the updated config, and that's when I ran into more trouble. After trying to restart the data nodes, I got this:

ndb_mgm> 3 restart
Node 3: Node shutdown initiated
Node 3: Node shutdown completed, restarting, no start.
Node 3 is being restarted

ndb_mgm> Node 3: Start initiated (version 7.4.7)
Node 3: Forced node shutdown completed. Occured during startphase 5. Caused by error 2815: 'File not found(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.

It should be noted that I tried to restart the data node process on data node 1 first by manually killing ndbmtd. This resulted in the same error as seen above when trying to restart data node 2 (nodeid=3) from the management client. These nodes would not start without using ndbmtd --initial. After restarting each data node this way, I was able to run the alter statement successfully:

mysql> alter table radacct.radacct force;
Query OK, 11510911 rows affected (1 hour 3 min 39.97 sec)
Records: 11510911  Duplicates: 0  Warnings: 0

Now when I look at the memory usage I can see that data has been reorganized:

ndb_mgm> all report memory
Node 2: Data usage is 27%(143081 32K pages of total 524288)
Node 2: Index usage is 24%(31479 8K pages of total 131104)
Node 3: Data usage is 27%(143433 32K pages of total 524288)
Node 3: Index usage is 24%(31473 8K pages of total 131104)
Node 4: Data usage is 16%(85234 32K pages of total 524288)
Node 4: Index usage is 8%(10504 8K pages of total 131104)
Node 5: Data usage is 16%(85207 32K pages of total 524288)
Node 5: Index usage is 8%(10504 8K pages of total 131104)

It appears that some part of the process of adding data nodes did not go over well. At this point, the issue seems to be resolved.

On a side note, I was also having issues using ndb_desc to analyze partitions in the cluster (https://dba.stackexchange.com/questions/125416/ndb-desc-unable-to-connect-to-management-server), and this seems to be resolved after starting the data nodes with --initial.

  • What does SHOW WARNINGS say after reorganize table? You had one warning and it could give you a clue what's wrong. – Mindaugas Riauba Jan 6 '16 at 20:13
  • Hello, @Mindaugas. Warning shows 'ONLINE' is deprecated and will be removed in a future release. Please use ALGORITHM=INPLACE instead. I've seen this warning issued in other examples online that were successful. To my knowledge, it should not affect the results of my statement. – Chris Telinde Jan 6 '16 at 20:28
  • 1
    Try a bit bruteforce approach then: alter table radacct force;. This command will rebuild the table. – Mindaugas Riauba Jan 7 '16 at 6:05
  • @MindaugasRiauba, please see the updated information. Either restarting the data nodes with --initial or rebuilding the table seems to have resolved the issue. Thanks! – Chris Telinde Jan 7 '16 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.