0

Mariadb keeps crashing with typical errors from /var/log/mysql/error.log being:

... terribly wrong ...
stack_bottom = 0x0 thread_stack 0x20000
(my_addr_resolve failure: fork)
...
2019-01-27  9:04:30 3069758672 [ERROR] mysqld: Out of memory (Needed 130760704 bytes)
2019-01-27  9:04:30 3069758672 [ERROR] mysqld: Out of memory (Needed 98058240 bytes)
2019-01-27  9:04:30 3069758672 [ERROR] mysqld: Out of memory (Needed 73531392 bytes)
2019-01-27  9:04:30 3069758672 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2019-01-27  9:04:30 3069758672 [Note] InnoDB: The InnoDB memory heap is disabled
2019-01-27  9:04:30 3069758672 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-01-27  9:04:30 3069758672 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-01-27  9:04:30 3069758672 [Note] InnoDB: Compressed tables use zlib 1.2.8
2019-01-27  9:04:30 3069758672 [Note] InnoDB: Using Linux native AIO
2019-01-27  9:04:30 3069758672 [Note] InnoDB: Using generic crc32 instructions
2019-01-27  9:04:30 3069758672 [Note] InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(138543104 bytes) failed; errno 12
2019-01-27  9:04:30 3069758672 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2019-01-27  9:04:30 3069758672 [ERROR] Plugin 'InnoDB' init function returned error.
2019-01-27  9:04:30 3069758672 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2019-01-27  9:04:30 3069758672 [Note] Plugin 'FEEDBACK' is disabled.
2019-01-27  9:04:30 3069758672 [ERROR] Unknown/unsupported storage engine: InnoDB
2019-01-27  9:04:30 3069758672 [ERROR] Aborting

My platform is a Raspberry Pi Zero with 512M of RAM and running 2018-10-09-raspbian-stretch. I’ve installed Mariadb version mysql Ver 15.1 Distrib 10.1.23-MariaDB, for debian-linux-gnueabihf (armv7l) using readline 5.2

From reading various forums I’ve changed the following:

/etc/mysql/mariadb.conf.d/50-server.cfg:
change key_buffer_size to 64K
change max_connections to 10
change query_cache_limit to 16K
change query_cache_size to 128K

I tried to add

performance_schema=0

to /etc/mysql/my.cnf but Mariadb didn’t recognize the variable.

There are only two clients to the database – an Apache2 server on the same device which connects every 5 seconds and a daemon I wrote in C which connects every second, both for a brief time. I’ve checked my code and each of them closes the connection to Mariadb correctly.

Any help would be very much appreciated.

EDIT: After enabling Slow Query Logging, I find that most queries take around 1ms-3ms to run, but in the lead-up to the crash, this time began to ramp up to 1.8 seconds, before the final log event of:

SET timestamp=1548626761;
SELECT * FROM timers;
# Time: 190128  6:06:03
# User@Host: pi[pi] @ localhost []
# Thread_id: 4  Schema: hptimer  QC_hit: No
# Query_time: 1.865374  Lock_time: 0.801896  Rows_sent: 0  Rows_examined: 8
# Rows_affected: 1
SET timestamp=1548626763;
UPDATE relays SET state=0 WHERE id=3;
/usr/sbin/mysqld, Version: 10.1.23-MariaDB-9+deb9u1 (Raspbian 9.0). started with:
Tcp port: 3306  Unix socket: /var/run/mysqld/mysqld.sock
Time                 Id Command    Argument

then no further information.

EDIT#2: Thank you for the suggestions to monitor memory usage/other processes. I found my C daemon process was consuming an additional 100KiB per call using "top", hence the OOM. I shall pull out my K&R and take a closer look at my code before blaming other applications again :)

Best regards, Peter

EDIT#3: You must always use

mysql_free_result( res );

after a query in C after your query has been processed.

https://dev.mysql.com/doc/refman/8.0/en/mysql-free-result.html

  • When these clients connect, what work are they doing? You may have some greedy queries causing you problems. By turning on the slow query logging you can analyse what might be making the memory condition and ultimately the OOM killer being used. – eroomydna Jan 27 at 17:40
  • The clients only make small queries - I only have six tables, each of less that 20 rows and maximum eight columns. I tested them and each query takes less than 10ms to complete on a bad day. - thank you – Pete Jan 27 at 17:57
  • I've rebooted with slow query logging and will report back. Thank you. – Pete Jan 27 at 18:34
  • Hmm, I wonder if there are any other processes on the pi that are causing problems for MDB. Have you got a display manager running or anything else that might be using memory? You may find tracing memory use from a system perspective would be profitable. – eroomydna Jan 27 at 19:20
  • How much RAM?? What else is running on the same machine (or VM)? – Rick James Jan 28 at 4:15
1

You must always use

mysql_free_result( res ); after a query in C after your query has been processed.

https://dev.mysql.com/doc/refman/8.0/en/mysql-free-result.html

This solved my OOM and memory leak from my C daemon, which was the cause of the problem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.