I am writing a log analysis program in bash. As part of it, I am compiling a list of known errors, so that I can search and reference those errors as part of the known list. I originally was going to just write them in a .sh file and use source to reference it. Then I realised that I have hundreds of thousands of error messages and that's going to be REALLY hard to do without killing myself.

So. I want to use a small and simple database where I can store error messages, some data associated with them (bug report, who found it, how to fix it, etc.), and it has to be easily searchable. I don't feel like dealing with dynamic queries and all that jazz, I want to be able to use static queries.

I know I can use Access, Excel, and other things like that, but this is CLI Linux. There are just so many options out there that I'm a little overwhelmed as to what would be perfect for something like this. I don't even know what type of database I should use, I just want it to be lightweight and common so it can be supported after I eventually move out of my current role.

What database engine should I use?

PS: I know Python, Perl, Ruby, C (and it's variants), and pretty much anything else would be better than bash for this. My hands are tied.

  • You are basically saying Mysql. – Mihai Aug 23 '14 at 15:17
  • Check out SQLite. – Mat Aug 24 '14 at 5:26

I sat down with management and said, "alright, what's the goal here?"

Once we talked about it, they decided to let me write it in Python and use Postgres, since we use Postgres internally. I was able to convince them of that because of the dynamic libraries and the fact that I can tune the instance to cache everything and keep it in memory.

So...a highly tuned Postgres instance is what I ended up being able to use.

I really wanted to use Pivotal's Gemfire, but from my research, there aren't any official or supported modules/libraries for Python.

| improve this answer | |
  • You're a lucky guy! :-) I was going to recommend PostgreSQL as the server - CTEs, windowing functions, check constraints... the list goes on. It really is superior to MySQL, at least for your type of use case. – Vérace Aug 24 '14 at 20:00

All I can offer you at this point are my deepest sympathies!

Doesn't the person/company who/which is getting you to write a tool under this mind-boggling constraint realise that you'll have to do far more work for a far lesser return than if you used (as you say) Python or Perl or even Brainfuck :-).

Anyway, what you wrote rang a very old bell in my memory cells - I remembered having read about a database system written in shell which I thought might have been interesting for, say, a student project (sort of like those C obfuscation competitions that challenge people to be clever without actually doing anything useful).

I visted Christopher Browne's linux databases site and found what I was looking for here (the kshSQL DBMS link). I was unable to find the source code for that one, but it is based on another programming effort, shql, a DBMS written in bash - it's github, so you have the source code.

The only bright spot on the horizon for you IMHO, is that it was written by Bruce Momjian who is a major player in the PostgreSQL world and knows a thing or two about databases. I've never used it, but there's just a chance that you might be able to do something with this. Good luck, you're going to need it! :-)

| improve this answer | |
  • Also see this. – Vérace Aug 23 '14 at 16:07
  • I really like the research you did. If it was my own system, and I was the only person working in it, I would've gone with kshSQL. – mxplusb Aug 24 '14 at 16:00

Not the answer you're looking for? Browse other questions tagged or ask your own question.