3

I have searched for a while about this error. I have a MySQL cluster running with 2 data nodes. The cluster houses a few other databases. I have a dump of a database in --tab format from another server. I have modified the SQL to use ndbcluster as the table engine and can import the SQL successfully. When I try to import data to the tables using mysqlimport, a few tables will populate but then I get the following message:

mysqlimport: Error: 1114, The table 'doc_archive' is full, when using table: doc_archive

Searching around for this error would suggest that I am out of memory on my data nodes, however that is not the case as you can see:

ndb_mgm> all report memory
Node 31: Data usage is 45%(179865 32K pages of total 393216)
Node 31: Index usage is 67%(88761 8K pages of total 131200)
Node 32: Data usage is 45%(179863 32K pages of total 393216)
Node 32: Index usage is 67%(88761 8K pages of total 131200)

Importing the database up until it is "full" only increases the data usage on each data node by 2%, and the index usage only goes up 3%.

Here is a copy of my config.ini from the management node:

[ndbd default]
# Options affecting ndbd processes on all data nodes:

NoOfReplicas=2    # Number of replicas

DataMemory=12G    # How much memory to allocate for data storage

IndexMemory=1G   # How much memory to allocate for index storage
            # For DataMemory and IndexMemory, we have used the
            # default values. Since the "world" database takes up
            # only about 500KB, this should be more than enough for
            # this example Cluster setup.

MaxNoOfOrderedIndexes=8192
MaxNoOfUniqueHashIndexes=512
MaxNoOfAttributes=100000
MaxNoOfExecutionThreads=8
MaxNoOfConcurrentOperations=256000
MaxNoOfTables=2048
MaxBufferedEpochs=100000
TimeBetweenEpochsTimeout=32000
TransactionDeadlockDetectionTimeout=1000000
TransactionInactiveTimeout=10000000
MaxNoOfTriggers=3072 # Default is 768

datadir=/var/lib/mysql-cluster   # Directory for this data node's data files
serverport=1187

NoOfFragmentLogFiles=1125
RedoBuffer=32M

[ndb_mgmd]
# Management process options:
nodeid=30
hostname=192.168.2.30
datadir=/var/lib/mysql-cluster

[ndbd]
nodeid=31
hostname=192.168.2.31

[ndbd]
nodeid=32
hostname=192.168.2.32

[mysqld]
nodeid=33
hostname=192.168.2.33

[mysqld]
nodeid=34
hostname=192.168.2.34

Where else should I be looking? There is plenty of disk space and memory to spare.

3 Answers 3

1

I had a similar issue. Except for me it was complaining about a large table being full, even though all nodes were using 0% data and index memory.

What fixed it for me was setting MaxNoOfExecutionThreads=4 (a value of 8 caused the table to be full).

I don't know why this worked, but it did. I wish the ndbmtd executable would provide more descriptive output than just (the table is full), such as describing exactly which internal memory or resource is full, because neither my memory nor disk were anywhere being full.

Hope this helps!

1

Note that I received the same "The table is full" message when trying to insert or update over 22,000 records to one table in a single transaction within MySQL Cluster. This too was a red herring since there was plenty of space on the disk. The issue wouldn't occur if less records were inserted in a single transaction.

To solve the problem I had to increase amount of ram that was allocated to the MySQL Data nodes. I added 8GB ram to each machine, then in the config.ini file changed the DataMemory setting from 8G to 16G for the data nodes. After rolling restarting the cluster, the problem magically went away.

So the "table is full" error may actually mean that there's not enough memory to store all the data being held for the transaction, before it's committed to disk.

0

Try to set parameter NoOfFragmentLogFiles (it's low by default, as many other values).

Link about configuration: http://johanandersson.blogspot.ru/2007/05/good-configuration.html

Description in MySQL documentation: http://dev.mysql.com/doc/refman/5.6/en/mysql-cluster-ndbd-definition.html#ndbparam-ndbd-nooffragmentlogfiles

UPD (sorry, may be it's must be some other kind of post):

Yes, --initial (in datanode start command) will wipe all your NDB data.

Firstly you must make backup, it will be somethink like this:

ndb_mgm> START BACKUP [backup_id] [WAIT {STARTED | COMPLETED} | NOWAIT] [ {SNAPSHOTSTART|SNAPSHOTEND} ] (sorry, may be such descriptions of commands weird and invalid in some way, better look in documentation).

You can just type START BACKUP 1, 1 will be backup id.

After that make full restart (with --initial on all nodes). And then restore.

Restore metadata (run this on any datanode one time): ndb_restore -m -b <backupid> -n <nodeid> --backup_path=/path/to/backup/files where <backupid> is number from START BACKUP command (1). Here -m is "restore metadata".

After that restore data (you must run it on all datanodes) : ndb_restore -r -b <backupid> -n <nodeid> --backup_path=/path/to/backup/files. Here -r is "restore data".

Don't forget to check that your backups really made and exist.

10
  • Also some people recommend to load data in InnoDB and use ALTER TABLE ... ENGINE=ndb after this.
    – ckorzhik
    Commented May 11, 2015 at 19:39
  • I have updated my config.ini with NoOfFragmentLogFiles=1125 and RedoBuffer=32M. I restarted data nodes with the --initial flag. When I try to import this database, the schema is created without any errors, but when I import the data either from one complete .sql file (minus table creation) or from --tab created .sql files, it still fails to import at the same place. ALL REPORT MEMORY shows the same results as well, with only 45% of data usage on each data node. This change does not seem to have done anything for my situation.
    – John Locke
    Commented May 12, 2015 at 12:44
  • did you tried to import data in innodb and then make alter table?
    – ckorzhik
    Commented May 12, 2015 at 13:23
  • You can import data in InnoDB and get size of database to check that your data really can fit into DataMemory. But I don't sure that it will have same size in NDB.
    – ckorzhik
    Commented May 12, 2015 at 13:26
  • Is there some data inside doc_archive table after import? How many tables you have imported? May be there is not enough place for indexes? If you have .sql file you can load it from mysql client and see when there will be error.
    – ckorzhik
    Commented May 12, 2015 at 13:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.