Writing a simple bank schema: How should I keep my balances in sync with their transaction history?

Question

I am writing the schema for a simple bank database. Here are the basic specifications:

• The database will store transactions against a user and currency.
• Every user has one balance per currency, so each balance is simply the sum of all transactions against a given user and currency.
• A balance cannot be negative.

The bank application will communicate with its database exclusively through stored procedures.

I expect this database to accept hundreds of thousands of new transactions per day, as well as balance queries on a higher order of magnitude. To serve up balances very quickly I need to pre-aggregate them. At the same time, I need to guarantee that a balance never contradicts its transaction history.

My options are:

1. Have a separate balances table and do one of the following:

   1. Apply transactions to both the transactions and balances tables. Use TRANSACTION logic in my stored procedure layer to ensure that balances and transactions are always in sync. (Supported by Jack.)

   2. Apply transactions to the transactions table and have a trigger that updates the balances table for me with the transaction amount.

   3. Apply transactions to the balances table and have a trigger that adds a new entry in the transactions table for me with the transaction amount.

   I have to rely on security-based approaches to make sure no changes can be made outside of the stored procedures. Otherwise, for example, some process could directly insert a transaction into the transactions table and under scheme 1.3 the relevant balance would be out of sync.

2. Have a balances indexed view that aggregates the transactions appropriately. Balances are guaranteed by the storage engine to stay in sync with their transactions, so I don't need to rely on security-based approaches to guarantee this. On the other hand, I cannot enforce balances be non-negative anymore since views -- even indexed views -- cannot have CHECK constraints. (Supported by Denny.)

3. Have just a transactions table but with an additional column to store the balance effective right after that transaction executed. Thus, the latest transaction record for a user and currency also contains their current balance. (Suggested below by Andrew; variant proposed by garik.)

When I first tackled this problem, I read these two discussions and decided on option 2. For reference, you can see a bare-bones implementation of it here.

• Have you designed or managed a database like this with a high load profile? What was your solution to this problem?

• Do you think I've made the right design choice? Is there anything I should keep in mind?

  For example, I know schema changes to the transactions table will require I rebuild the balances view. Even if I am archiving transactions to keep the database small (e.g. by moving them somewhere else and replacing them with summary transactions), having to rebuild the view off tens of millions of transactions with every schema update will probably mean significantly more downtime per deployment.

• If the indexed view is the way to go, how can I guarantee that no balance is negative?

---

Archiving transactions:

Let me elaborate a bit on archiving transactions and the "summary transactions" I mentioned above. First, regular archiving will be a necessity in a high-load system like this. I want to maintain consistency between balances and their transaction histories while allowing old transactions to be moved somewhere else. To do this I'll replace every batch of archived transactions with a summary of their amounts per user and currency.

So, for example, this list of transactions:

user_id    currency_id      amount    is_summary
------------------------------------------------
      3              1       10.60             0
      3              1      -55.00             0
      3              1      -12.12             0

is archived away and replaced with this:

user_id    currency_id      amount    is_summary
------------------------------------------------
      3              1      -56.52             1

In this way, a balance with archived transactions maintains a complete and consistent transaction history.

Answer 1

I am not familiar with accounting, but I solved some similar problems in inventory-type environments. I store running totals in the same row with the transaction. I am using constraints, so that my data is never wrong even under high concurrency. I have written the following solution back then in 2009::

Calculating running totals is notoriously slow, whether you do it with a cursor or with a triangular join. It is very tempting to denormalize, to store running totals in a column, especially if you select it frequently. However, as usual when you denormalize, you need to guarantee the integrity of your denormalized data. Fortunately, you can guarantee the integrity of running totals with constraints – as long as all your constraints are trusted, all your running totals are correct. Also this way you can easily ensure that the current balance (running totals) is never negative - enforcing by other methods can also be very slow. The following script demonstrates the technique.

CREATE TABLE Data.Inventory(InventoryID INT NOT NULL IDENTITY,
  ItemID INT NOT NULL,
  ChangeDate DATETIME NOT NULL,
  ChangeQty INT NOT NULL,
  TotalQty INT NOT NULL,
  PreviousChangeDate DATETIME NULL,
  PreviousTotalQty INT NULL,
  CONSTRAINT PK_Inventory PRIMARY KEY(ItemID, ChangeDate),
  CONSTRAINT UNQ_Inventory UNIQUE(ItemID, ChangeDate, TotalQty),
  CONSTRAINT UNQ_Inventory_Previous_Columns 
     UNIQUE(ItemID, PreviousChangeDate, PreviousTotalQty),
  CONSTRAINT FK_Inventory_Self FOREIGN KEY(ItemID, PreviousChangeDate, PreviousTotalQty)
    REFERENCES Data.Inventory(ItemID, ChangeDate, TotalQty),
  CONSTRAINT CHK_Inventory_Valid_TotalQty CHECK(
         TotalQty >= 0 
     AND (TotalQty = COALESCE(PreviousTotalQty, 0) + ChangeQty)
  ),
  CONSTRAINT CHK_Inventory_Valid_Dates_Sequence CHECK(PreviousChangeDate < ChangeDate),
  CONSTRAINT CHK_Inventory_Valid_Previous_Columns CHECK(
        (PreviousChangeDate IS NULL AND PreviousTotalQty IS NULL)
     OR (PreviousChangeDate IS NOT NULL AND PreviousTotalQty IS NOT NULL)
  )
);

-- beginning of inventory for item 1
INSERT INTO Data.Inventory(ItemID,
  ChangeDate,
  ChangeQty,
  TotalQty,
  PreviousChangeDate,
  PreviousTotalQty)
VALUES(1, '20090101', 10, 10, NULL, NULL);

-- cannot begin the inventory for the second time for the same item 1
INSERT INTO Data.Inventory(ItemID,
  ChangeDate,
  ChangeQty,
  TotalQty,
  PreviousChangeDate,
  PreviousTotalQty)
VALUES(1, '20090102', 10, 10, NULL, NULL);


Msg 2627, Level 14, State 1, Line 10

Violation of UNIQUE KEY constraint 'UNQ_Inventory_Previous_Columns'. 
Cannot insert duplicate key in object 'Data.Inventory'.

The statement has been terminated.


-- add more
DECLARE @ChangeQty INT;
SET @ChangeQty = 5;

INSERT INTO Data.Inventory(ItemID,
  ChangeDate,
  ChangeQty,
  TotalQty,
  PreviousChangeDate,
  PreviousTotalQty)

SELECT TOP 1 ItemID, '20090103', @ChangeQty, TotalQty + @ChangeQty, ChangeDate, TotalQty
  FROM Data.Inventory
  WHERE ItemID = 1
  ORDER BY ChangeDate DESC;

SET @ChangeQty = 3;

INSERT INTO Data.Inventory(ItemID,
  ChangeDate,
  ChangeQty,
  TotalQty,
  PreviousChangeDate,
  PreviousTotalQty)

SELECT TOP 1 ItemID, '20090104', @ChangeQty, TotalQty + @ChangeQty, ChangeDate, TotalQty
  FROM Data.Inventory
  WHERE ItemID = 1
  ORDER BY ChangeDate DESC;

SET @ChangeQty = -4;

INSERT INTO Data.Inventory(ItemID,
  ChangeDate,
  ChangeQty,
  TotalQty,
  PreviousChangeDate,
  PreviousTotalQty)

SELECT TOP 1 ItemID, '20090105', @ChangeQty, TotalQty + @ChangeQty, ChangeDate, TotalQty
  FROM Data.Inventory
  WHERE ItemID = 1
  ORDER BY ChangeDate DESC;

-- try to violate chronological order
SET @ChangeQty = 5;

INSERT INTO Data.Inventory(ItemID,
  ChangeDate,
  ChangeQty,
  TotalQty,
  PreviousChangeDate,
  PreviousTotalQty)

SELECT TOP 1 ItemID, '20081231', @ChangeQty, TotalQty + @ChangeQty, ChangeDate, TotalQty
  FROM Data.Inventory
  WHERE ItemID = 1
  ORDER BY ChangeDate DESC;

Msg 547, Level 16, State 0, Line 4

The INSERT statement conflicted with the CHECK constraint 
"CHK_Inventory_Valid_Dates_Sequence". 
The conflict occurred in database "Test", table "Data.Inventory".

The statement has been terminated.

SELECT ChangeDate,
  ChangeQty,
  TotalQty,
  PreviousChangeDate,
  PreviousTotalQty
FROM Data.Inventory ORDER BY ChangeDate;

ChangeDate              ChangeQty   TotalQty    PreviousChangeDate      PreviousTotalQty
----------------------- ----------- ----------- ----------------------- -----
2009-01-01 00:00:00.000 10          10          NULL                    NULL
2009-01-03 00:00:00.000 5           15          2009-01-01 00:00:00.000 10
2009-01-04 00:00:00.000 3           18          2009-01-03 00:00:00.000 15
2009-01-05 00:00:00.000 -4          14          2009-01-04 00:00:00.000 18


-- try to change a single row, all updates must fail
UPDATE Data.Inventory SET ChangeQty = ChangeQty + 2 WHERE InventoryID = 3;
UPDATE Data.Inventory SET TotalQty = TotalQty + 2 WHERE InventoryID = 3;

-- try to delete not the last row, all deletes must fail
DELETE FROM Data.Inventory WHERE InventoryID = 1;
DELETE FROM Data.Inventory WHERE InventoryID = 3;

-- the right way to update
DECLARE @IncreaseQty INT;

SET @IncreaseQty = 2;

UPDATE Data.Inventory 
SET 
     ChangeQty = ChangeQty 
   + CASE 
        WHEN ItemID = 1 AND ChangeDate = '20090103' 
        THEN @IncreaseQty 
        ELSE 0 
     END,
  TotalQty = TotalQty + @IncreaseQty,
  PreviousTotalQty = PreviousTotalQty + 
     CASE 
        WHEN ItemID = 1 AND ChangeDate = '20090103' 
        THEN 0 
        ELSE @IncreaseQty 
     END
WHERE ItemID = 1 AND ChangeDate >= '20090103';

SELECT ChangeDate,
  ChangeQty,
  TotalQty,
  PreviousChangeDate,
  PreviousTotalQty
FROM Data.Inventory ORDER BY ChangeDate;

ChangeDate              ChangeQty   TotalQty    PreviousChangeDate      PreviousTotalQty
----------------------- ----------- ----------- ----------------------- ----------------
2009-01-01 00:00:00.000 10          10          NULL                    NULL
2009-01-03 00:00:00.000 7           17          2009-01-01 00:00:00.000 10
2009-01-04 00:00:00.000 3           20          2009-01-03 00:00:00.000 17
2009-01-05 00:00:00.000 -4          16          2009-01-04 00:00:00.000 20

Answer 2

A slightly different approach (similar to your 2nd option) to consider is to have just the transaction table, with a definition of:

CREATE TABLE Transaction (
      UserID              INT
    , CurrencyID          INT 
    , TransactionDate     DATETIME  
    , OpeningBalance      MONEY
    , TransactionAmount   MONEY
);

You may also want a transaction ID/Order, so that you can handle two transactions with the same date and improve your retrieval query.

To get the current balance, all you need to get is the last record.

Methods to get the last record:

/* For a single User/Currency */
Select TOP 1 *
FROM dbo.Transaction
WHERE UserID = 3 and CurrencyID = 1
ORDER By TransactionDate desc

/* For multiple records ie: to put into a view (which you might want to index) */
SELECT
    C.*
FROM
    (SELECT 
        *, 
        ROW_NUMBER() OVER (
           PARTITION BY UserID, CurrencyID 
           ORDER BY TransactionDate DESC
        ) AS rnBalance 
    FROM Transaction) C
WHERE
    C.rnBalance = 1
ORDER BY
    C.UserID, C.CurrencyID

Cons:

• When inserting a transaction out of sequence (ie: to correct an issue/incorrect starting balance), you may need to cascade updates for all subsequent transactions.

• Transactions for the User/Currency would need to be serialized to maintain an accurate balance.

  -- Example of getting the current balance and locking the 
  -- last record for that User/Currency.
  -- This lock will be freed after the Stored Procedure completes.
  SELECT TOP 1 @OldBalance = OpeningBalance + TransactionAmount  
  FROM dbo.Transaction with (rowlock, xlock)   
  WHERE UserID = 3 and CurrencyID = 1  
  ORDER By TransactionDate DESC;
  

Pros:

• You no longer have to maintain two separate tables...
• You can easily validate the balance, and when the balance gets out of sync you can identify exactly when it got out of whack as the transaction history becomes self documenting.

---

Edit: Some sample queries on retrieval of current balance and to highlight con (Thanks @Jack Douglas)

Answer 3

Not allowing customers to have a less than 0 balance is a business rule (which would change quickly as fees for things like over draft are how banks make most of their money). You'll want to handle this in the application processing when rows are inserted into the transaction history. Especially as you may end up with some customers having overdraft protection and some getting charged fees and some not allowing negative amounts to be entered.

So far I like where you are going with this, but if this is for an actual project (not school) there needs to be a hell of a lot of thought put into business rules, etc. Once you've got a banking system up and running there's not a lot of room for redesign as there are very specific laws about people having access to their money.

Answer 4

After reading these two discussions, I decided on option 2

Having read those discussions too, I am not sure why you decided on the DRI solution over the most sensible of the other options you outline:

Apply transactions to both the transactions and balances tables. Use TRANSACTION logic in my stored procedure layer to ensure that balances and transactions are always in sync.

This kind of solution has immense practical benefits if you have the luxury of restricting all access to the data through your transactional API. You lose the very important benefit of DRI, which is that integrity is guaranteed by the database, but in any model of sufficient complexity there will be some business rules that cannot be enforced by DRI.

I'd advise using DRI where possible to enforce business rules without bending your model too much to make that possible:

Even if I am archiving transactions (e.g. by moving them somewhere else and replacing them with summary transactions)

As soon as you start considering polluting your model like this, I think you are moving into the area where the benefit of DRI is outweighed by the difficulties you are introducing. Consider for example that a bug in your archiving process could in theory cause your golden rule (that balances always equal the sum of transactions) to break silently with a DRI solution.

Here is a summary of the advantages of the transactional approach as I see them:

• We should be doing this anyway if at all possible. Whatever solution you choose for this particular problem, it gives you more design flexibility and control over your data. All access then becomes "transactional" in terms of business logic, rather than just in terms of database logic.
• You can keep your model neat
• You can "enforce" a much wider range and complexity of business rules (noting that the concept of "enforce" is a looser one than with DRI)
• You can still use DRI wherever practical to give the model a more robust underlying integrity - and this can act as a check on your transactional logic
• Most of the performance issues that are troubling you will melt away
• Introducing new requirements can be much easier - for example: complex rules for disputed transactions might force you away from a pure DRI approach further down the line, meaning a lot of wasted effort
• Partitioning or archiving of historical data becomes much less risky and painful

--edit

To allow archiving without adding complexity or risk, you could choose to keep summary rows in a separate summary table, generated continuously (borrowing from @Andrew and @Garik)

For example, if the summaries are monthly:

• each time there is a transaction (through your API) there is a corresponding update or insert into the summary table
• the summary table is never archived, but archiving transactions becomes as simple a a delete (or drop partition?)
• each row in the summary table includes 'opening balance' and 'amount'
• check constraints such as 'opening balance'+'amount'>0 and 'opening balance'>0 can be applied to the summary table
• summary rows could be inserted in a monthly batch to make locking the latest summary row easier (there would always be a row for the current month)

Answer 5

Nick.

Main idea is storing balance and transaction records in the same table. It happened historically I thought. So in this case we can get balance just by locating the last summary record.

 id   user_id    currency_id      amount    is_summary (or record_type)
----------------------------------------------------
  1       3              1       10.60             0
  2       3              1       10.60             1    -- summary after transaction 1
  3       3              1      -55.00             0
  4       3              1      -44.40             1    -- summary after transactions 1 and 3
  5       3              1      -12.12             0
  6       3              1      -56.52             1    -- summary after transactions 1, 3 and 5 

A better variant is decreasing number of summary records. We can have one balance record at the end(and/or begin) of day. As you know every bank has operational day to open and than close it to do some summary operations for this day. It allows us to easy calculate interest by using every day balance record, for example:

user_id    currency_id      amount    is_summary    oper_date
--------------------------------------------------------------
      3              1       10.60             0    01/01/2011 
      3              1      -55.00             0    01/01/2011
      3              1      -44.40             1    01/01/2011 -- summary at the end of day (01/01/2011)
      3              1      -12.12             0    01/02/2011
      3              1      -56.52             1    01/02/2011 -- summary at the end of day (01/02/2011)

Luck.

Answer 6

Based on your requirements, option 1 would appear the best. Although I would have my design to only allow inserts into the transaction table. And have the trigger on the transaction table, to update the real time balance table. You can use database permissions to control access to these tables.

In this approach, the real time balance is guaranteed to be in sync with the transaction table. And it does not matter if stored procedures or psql or jdbc is used. You can have your negative balance check if needed. Performance will not be an issue. To get the real time balance, it is a singleton query.

Archiving will not affect this approach. You can have a weekly,monthly,yearly summary table also if needed for things like reports.

Answer 7

In Oracle you could do this using just the transactions table with a fast refreshable Materialized View on it that does the aggregation to form the balance. You define the trigger on the Materialized View. If the Materialized View is defined with 'ON COMMIT', it effectively prevents adding/modifying data in the base tables. The trigger detects the [in]valid data and raises an exception, where it rollback the transaction. A nice example is here http://www.sqlsnippets.com/en/topic-12896.html

I don't know sqlserver but maybe it has a similar option?