PostgreSQL 9.1 runs into inconsistent data, but pg_dump corrects the problem?

Question

We are running a PostgreSQL 9.1.20 with a database of about 50GB.

1. All things works fine.
2. After some days strange unexpected values appear in numeric columns, like negative or bigger values in unexpected places.
3. The strange part: if we run a pg_dump of the same "corrupted" database, and then restore it (in the same server), the data becomes all ok. Now the same queries return different values!

Why does the database dump have different values when we are taking the dump exactly from the "corrupted" database? (The dump has corrected values from our perception!)

Can this be related to some kind of broken transaction or hard drive trouble? Note that restarting the database does not solve the problem.

What can be the cause behind this problem?

As requested in the comments some examples of data:

    #psql 
    \c base
    select * from salXXXX where sal_id=2323; 
    sal_id | sal_XXXX | XXXXXX | XXXXXX | XXXXX | XXXXX | XXXXX 
--------+-----------+---------------+--------------------+------------+-----------------------+------------
   2323 |     -30.43 | 42586501      |                  6 |         13 | f                     | 

    \quit
    pg_dump -h 127.0.0.1 -p 5432 --user=postgres >backup.sql
    psql 
    drop database base;
    create database base;
    \c base 
    \i backup.sql
    (no errors found here) 
    select * from salXXXX where sal_id=2323; 
    sal_id | sal_XXXX | XXXXXX | XXXXXX | XXXXX | XXXXX | XXXXX 
--------+-----------+---------------+--------------------+------------+-----------------------+------------
   2323 |    245.43 | 42586501      |                  6 |         13 | f                     | 

It is hard to emulate it, rarely occurs, but when it occurs we just do a dump, it is a ridiculous solution, but it works.

This is just an example, many more wrong values start to appear, even if we reprocess the data. But if the database is dumped and restored, values are in the expected range again.

Answer 1

Might be index corruption - assuming you have an index on sal_id.

If so, REINDEX TABLE salXXXX; (or just REINDEX INDEX index_name would do the same trick as your dump/restore cycle, which also recreates all indexes from scratch.

The manual advises:

Note that while this method does not require locking out other clients, it might still be wise to prevent other users from connecting to the damaged database until repairs have been completed.

Generally, consider upgrading to a current version of Postgres. Version 9.1 has just reached its EOL in Sept. 2016. There have been many related updates since, might very well fix the problem for good.