Banks have to follow strict rules to account for where all the money goes. But the way fintechs work, they usually just have one or a couple underlying "FBO" accounts where all the pooled money is held, but then the fintech builds a ledger on top of this (and, as the article points out, to varying levels of engineering competence) to track each individual customer's balance within this big pool of money. In Synapse's case, their ledger said the total amount of all of their individual customer balances ended up being much more than the actual funds held in the underlying FBO accounts. Lots of folks are assuming fraud but I'm willing to put money that it was just a shitty, buggy ledger.

FWIW, after seeing "how the sausage is made", I would never put money into a fintech depository account. Use a real bank. Fintechs also often put out the fake promise that deposits are FDIC insured, but this only protects you if the underlying bank goes belly up, not if the fintech loses track of your money.

See https://www.forbes.com/sites/zennonkapron/2024/11/08/what-th...

I had previously built things like billing systems and small-scale OLTP web applications, where the concept of even asking the question of whether there could be any acceptable data loss or a nonzero error rate hadn't occurred to me. It was eye-opening to see, not so much that if you're doing millions of qps, some are going to fail, but more the difference in engineering attitude. Right this instant, there are probably thousands of people who opened their Gmail and it didn't load right or gave them a 500 error. Nobody is chasing down why that happened, because those users will just hit reload and go on with their day. Or from another perspective, if your storage has an impressive 99.99999% durability over a year, when you have two billion customers, 200 people had a really miserable day.

It was a jarring transition from investigating every error in the logs to getting used to everything being a little bit broken all the time and always considering the cost before trying to do something about it.

Sometimes you need engineers that have some other type of education, maybe accounting, maybe finance, maybe biology. I always thought that the most important part of my career was understanding every industry I built for, deeply, and knowing experts in those areas so you could ask the really important questions. That is problem solving and engineering. The rest is programming/coding.

By accident and not knowing any better as a young dev, I ended up building the billing logic from day one, and for better and worse building it in two places in the system (on a consumer-facing billing webpage, and on a separate backed process that generated invoices and charged credit cards.)

It turned out to be remarkably hard to keep them in sync. We were constantly iterating trying to get traction as we burned down our capital, releasing new products and services, new ways of discounting and pricing (per use, per month, first X free, etc), features like masterpayer/subaccounts for corporate accounts, user-assignable cost centers, tax allocation to those cost centers with penny allocation, etc such that new wrinkles and corner cases would keep popping up causing the numbers on my two screens/methods not to match.

Being personally responsible for the billing, I would go over all the invoices by hand for a couple days each month to insure they matched before we charged the credit cards and mailed out printed invoices as a final check to prevent mistakes. There was always/often some new problem I'd find affecting one or a small handful of customers which I would then fix the code before we billed. I never felt good letting go and not doublechecking everything by hand.

I thought about refactoring the billing logic to occur in one place to eliminate these mismatches and my manual crosschecking, but after a lot of thought I realized I wasn't comfortable with a single codebase and liked having two codebases as it helped me catch my own errors. I then just made it easier and easier to run automate and crosschecks between the two. The billing code was a little too gnarly to be proud of, but I was very proud of the outcome in how accurate our billing was, the lack of complaints and many near misses we avoided for many years. I do feel twinges of guilt for the complexity I left my successors but I still don't really regret it.

After that experience, the motivation for double entry bookkeeping has always made a lot of sense to me. I had sort of reinvented it in my own hacky way with double logic billing code to prevent my mistakes from causing problems for my customers...

Who builds a financial system like that an considers it normal? The compensation is one thing, but you'd flee a service like that with all possible haste.

to clarify: "make it right" is the second step, and until you make things work correctly ("right"), that's where the work stops, you have to make the system work soundly. the "make it fast", as in, optimize, comes in only after you have got it right, that all correctness and soundness issues are resolved perfectly. then you can start optimizing it (making it run fast).

it has nothing to do with delivery speed. it has nothing to do with working quickly. it's about optimizing only as a last step.

perhaps the author is lamenting the fact that it is possible for something to sort of "work", but to be so far from being "right" that you can't go back and make it right retroactively, that it has to be "right" from the inception, even before it starts barely working?

Sure, single-entry bookkeeping might be easier and more normalized, but sometimes it is a good idea to just stick with the systems and abstractions that have been developed over centuries.

Just use double-entry bookkeeping unless you definitely need something else. Sure, it might be icky for the programmer inside you, but I think you'll be thankful if you ever need to get actual accountants involved to sort out a mismatch.

On a related note: does anybody know of any good resources for programmers in payments and adjacent fields? Something like an "Accounting for Programmers"?

https://en.wikipedia.org/wiki/British_Post_Office_scandal

Anything that moves money should be treated with upmost seriousness and be aware of as many historical mistakes as possible.

I’m also surprised that this whole article starts by discussing stock trading but has no mention of how to represent stock trades. I assume they are “Sagas” consisting of money moving from the customer to the clearinghouse (or prime broker or PFOF provider or whatever) and shares moving from that provider to the account at which the shares are held. And maybe other associated entries representing fees? It seems to me that this is multi-entry accounting, which is quite common, and that entries don’t actually come in pairs as the article would like us to think.

I've seen the following major phases of this work: 1) Build the ledger (correctly), and it will work well for a while. 2) Add convenience code for the callers, assist finance in doing reports/journaling, fix some minor bugs, take care of the operational bits (keep the database up). 3) Reach the scaling limits of your initial approach, but there are some obvious (not trivial) things to do: re-implement the transaction creation directly in the database (10x perf gain), maybe sharding, maybe putting old tx into colder storage, etc.

This is spread out over a while, so I haven't seen it be a full-time job, even at real startup-level (+10% MoM) growth. Even if it was, that's one person, not a whole team. I understand engineers that instead are pulled towards projects where they are in higher demand.

In another comment somebody said ledger systems are trivial when done right and super hard when done wrong - so if you did a good job it kinda looks like you just created 3 tables and some code. That seems thankless, and job searching as this type of specialist is harder than just being a generalist.

Here's an example of a core banking deposit transaction schema that has been extensively battle tested in many small & mid-size US institutions:

https://jackhenry.dev/open-enterprise-api-docs/operational-d...

You may note fields like "Effective Date" & "Affects Balance/Interest", which imply doing this correctly may involve exploring some interesting edge cases. Wouldn't it be cool if you could just cheat and start with an approach that already considers them?

Worse, I've worked at where the transaction reference id from vendor is not recorded, it's lost so the past data cannot be reconciled!

in the UK, as an engineer, if I'd built this I would expect the regulator to come after me personally for not ensuring the system had adequate controls to protect clients money/investments

with a potentially unlimited fine + prison time

https://github.com/adamcharnock/django-hordak

I created and maintain this, along with a couple of others. It is built for Django (so great if you're using Django), but extracting the schema wouldn't be too hard. It also has MySQL support, but the integrity checks are more limited.

(Side note: I'm a freelancer and available!)

I gave a boorish lecture to a junior accountant recently...

When you make things up yourself it is just you and Excel. When you use double entry you have 100s of years of history to fall back on. https://en.m.wikipedia.org/wiki/Double-entry_bookkeeping

There is no in-between.

Martin fowler wrote quite a bit on the subject and it's a good match for event-sourcing.

But that is an interpretation made by the viewer. A customer typically is an asset account, whose balances are in the debit column. But if we somehow owe them money because let's say they paid us an advance, then their balance should be in the credit column. The accounting system need not bother with what the "right" place for each account is.

It is quite practical to have only a simple amount column rather than separate debit/credit columns in a database for journal entries. As long as we follow a consistent pattern in mapping user input (debit = positive, credit = negative) into the underlying tables, and the same when rendering accounting statements back, it would remain consistent and correct.

One that's not referenced in this article and compile all of them is: https://github.com/kdeldycke/awesome-billing#readme

https://en.wikipedia.org/wiki/Resources%2C_Events%2C_Agents

You can see that this model has all of the features discussed in the article, and then some, and REA events map naturally to something like event sourcing. You can project a REA dataset into a double entry ledger, but you often can't go the other way around.

The tech founder should know their shit prior to building it so that time and runway and deadlines are no excuse. Really if you are doing fintech you should have employment experience and understand all the operations well.

Otherwise they are no better than say home builders who mismanage and go bankrupt. Or less geneously, they are conmen.

I didn't understand this part, can someone give examples of good and bad approaches for both credit and debit accounts?

  transactions: [
    (txID, timestamp, [
      (accountID, delta, otherInfo),
      ...
    ], reason),
    ...
  ]

  accounts: [
    (accountID, routingNumber, accountNumber, ownerID),
    ...
  ]

Perhaps popular culture should review the use of nosql databases, and then spending tremendous effort to try and make it into a relational database, while nosql databases can be setup to be eventually consistent.

Money, and accurate accounting don't work too well when the inputs to a calculation of a balance are eventually consistent.

So much work is avoided at times it seems to not learn SQL that it can rival or end up being more work once NOSQL leads you down the path of inevitable relational needs.

In addition to this, maybe it's time to build honeypot ledgers with a prize and reward in it for anyone who can hack or undermine it, similar to vulnerability bounties. Run for long enough, it would determine at least that one side of security mistakes in startups being reduced. Run a sprint, with prizes and let them run like daily deal fantasy and watch things harden.

Double entry is irrelevant, here.

I love the "use crypto" to fix the problem suggestion. LOL!

It's kind of poignant since crypto has given me a fantastic club to beat executives over the head with whenever they want to do stupid handling of money. Since crypto has so many decimal places, you break dumbass floating point handling of money immediately and irretrievably for way more than just "a couple cents". Your CFO starts jumping up and down really excitedly about handling money properly when he has to worry about a rounding error forcing him to compensate a Bitcoin price jump/crash.

At a fintech startup I was working with, we built a "shadow ledger" because we couldn't trust a 3rd party PP to give us the correct information, which would otherwise allow double spending on accounts.

We tried 3 different (major!) PPs - they ALL had similar flaws.

It's the same with computer systems. The charts show something, but until enough people decide to all withdraw their money or sell their stock at the same time, nobody has any idea that the money or asset simply isn't there or nobody knows just how frothy the valuation is.

Social media and search algorithms are highly optimized to ensure that people don't sell or withdraw stuff at the same time. Modern media directs massive attention towards certain topics as a way to draw attention away from other topics which could collapse the economy.

Also, imagine a bank has a serious bug which causes millions or billions of dollars to disappear every year or creates extra illegitimate dollars. Imagine they only discover this bug after a few years of operation... How likely is it that they will report it to an authority? They didn't notice it for years, why not pretend they didn't notice it for a few MORE years... The incentive to delay the reckoning is an extremely powerful one.

I literally build the same system and ask for HN Ask but no one answer it and already using double entry and ledger based on some research and AI advice lol

I implement all the check its mention in your page and quite satisfied using postgress constraint and trigger function to detect abnormality before bad row is inserted

but the problem now is populate the database using fake data because how rigirous my constraint and rule are, I need to simulate real world use case and duplicate it on development machine but this are hard

if you ever read this, can you give some advice please, because simply looping that function is not gonna work because constraint of double entry and ledger

Sometimes to 2 digits. Sometimes 3. Or 4. How many significant digits are there even in fractional SPY holdings? You just resort to looking in all the places (statements, transaction history, overview dashboard, …) and going with the one that shows the most digits, and assume that's all of them.

Big and small companies do this. And when I've reported it, they don't care. They don't see the problem.

I did not like finance much, but building a fintech system really teaches a lot, not only from technology perspective but from managing stakeholders, processes, compliance, dealing with all kinds of finance-specific issues.

One should always work in fintech, at some point of time in career.

1) That being able to escape characters in CSV files is kind of important. (Some customers noticed that a single quote in a transaction description would make the importer silently fail to import the rest of the transactions... :S ), and

2) Why it wasn't good to store all dollar values as doubles. I'm reasonably sure I quoted Superman 3 at some point during this discussion.

I pushed for offsetting entries instead, but was overruled. It was a _nightmare_.

Thankfully, it is not $previousjob.

I want to see if I have time to finish this article before I have to leave. And now I have to waste time copy-pasting the contents into a text file, just to see where I am.

Edit: Actually, it's invisible only in Firefox. Chrome shows the scrollbar still. So this may be a bug in the author's CSS or something.

It appears the startup in question just never even did step one. A financial system that is imprecise does not work.

Additionally that mantra is typically applied at the story/issue level in my experience as in:

1. Get the tests passing 2. Get the code clean 3. Identify scaling bottlenecks and remedy if necessary.

Entry(account, direction, non-negative amount), direction is debit or credit.

vs

Entry(account, signed amount), + is debit, - is credit (for example).

It's a two way mapping and should be equivalent. Unless debit or credit amount could be negative. But as I understand it's a big NO-NO in accounting.

haha, if only.

move fast, break everything if it gives profit

The last fintech I worked for had a joke about how you weren't really an employee until your code had lost some money.

Sure there was. A financial services company wanted to replace their repayment plan generator that ran on an aging AS/400 with someting running in .Net on Windows.

I dug in an learned all about time value of money, numerical formats and precision, rounding strategies, day counting strategies for incomplete periods (did you know there are many dozens of those) etc.

I made everyting as configurable as I could so we had the option to offer to other financial service clients by just setting up a different profile.

Since I had no interaction with the client before the first presentation meeting, I put in what to me felt like the most plausible config (I had managed projects in the domain before so I was not completely clueless to the mindset).

We had the meeting. I showed a few calculated plans which they compared on the spot to the AS/400 output. I deployed on a test VM for them so they could do more extensive testing. Code was accepted with 0 change requests and put into production shortly thereafter. Don't think they ever changed from the default settings.

We routinely re-discover stuff that probably was already solved by a quiet lady writing a CICS transaction in a s-360 system in 1969.

Yeah, dealing with money and especially others people money without double entry bookkeeping is a bad practice.

But this particular problem is the consequence of the choice of using floating point binary math to deal with monetary quantities.

Given the fact that most modern languages have very awkward support for arbitrary precision decimal numbers the most sensible way to deal with money usually boils down to store it as an integer.

But apart from the technical points, another interesting thing in the article is its introduction and what it says there about money as a concept. Yes, money is debt, both philosophically and technically it is the right way to think about it. And I believe that's something the crypto-assets industry and enthusiasts as a whole fundamentally gets wrong (mostly because of the libertarian political point of view blockchain tech has been designed with).

If a new transaction enters the system now, I could follow the advice and record it as two sources of truth. Or I could just record it once.

If I could turn a single transaction into two ledger entries today, I could do the same later if I needed to.

Rounding in particular is a truly endless source of trouble and has caused me to chase after a lot of cents. Dividing up a large payment into multiple installments is the major cause of rounding in my use case. Life starts to suck the second things fail to be evenly divisible. I created an account to track gains and losses due to rounding and over time and it's adding up to quite the chunk of change.

Hilariously, the payment systems would charge me incorrect rounded up amounts and then they would refund the difference to my credit card at some undefined time in the future. Tracking and correlating all these seemingly random one or two cent transactions has got to be one of the most annoying activities I've ever learned to put up with. Not only do I have to figure out why things aren't quite adding up, I have to patch things up in the future when they fix the mistake.

Why can't these things just charge the correct amounts? For example, imagine splitting up $55.53 into four installments. That's 4x$13.8825. They could charge me 3x$13.88 + 1x$14.89. Instead they round it up to $55.56 and charge me 4x$13.89, then maybe they refund me $0.03 some unknown day in the future. It's like the systems go out of their way to be as annoying as possible. Some systems do this silly single cent refund dance in my credit card statement even though they print the exact same 3xN + 1xN+0.01 solution in the receipt. Makes absolutely no sense to me.

It's getting to the point I'm trying to avoid this nonsense by structuring purchases so the final price is evenly divisible by some common factors. I never liked the .99 cents manipulation trick but I seriously hate it now.

Nope. Double entry bookkeeping means every transaction is recorded in (at least) two accounts.

Yeah it's a real pain to get started because you need to understand the core concepts first then fight to balance the transactions ("compile errors") before you have anything useful. And when your results are wrong, you know that at least it's not because of the basic stuff.

Every engineer in London is laughing at this right now.

Wow double entry! Ledgers are hard! Wow. So true.

is this not the literal plot of Office Space? did you check for a Michael Bolton employee?

See, if you're an engineer (from a web background) you might think that using a regular backend language with a relational DB would be fine. But this service is typically optimized for concurrency. That might sound amazing. But what you really want is a simple queue. This ensures everything is applied in order and makes it impossible to accidentally have transactions execute on stale state. Plus, a queue can be executed amazingly fast -- we're talking Nasdaq scales. If you use a standard DB you'll end up doing many horrible broken hacks to simulate what would be trivial engineering had you used the right design from the start.

You've got other things to worry about, too. Financial code needs to use integer math for everything. Floating point and 'decimals' lead to unexpected results. The biggest complexity with running large-scale blockchain services is having to protect 'hot wallets.' Whenever an exchange or payment system is hacked the target is always the funds that sit on the server. When the industry began there were still many ways to protect the security of these hot wallets. The trouble is: it required effort to implement and these protocols weren't widely known. So you would get drive-by exchanges that handled millions of dollars with private keys sitting on servers ready to be stolen...

Today there are many improvements for security. New cryptographic constructs like threshold ECDSA, hardware wallets, hardware key management (like enclaves), smart contract systems (decentralized secret sharing... and even multi-sig can go a long way), and designs that are made to be decentralized that remove the need for a centralized deposit system (still needs some level of centralization when trading to a stable coin but its better than nothing.)

I would say the era where people 'build' ledgers though is kind of over. To me it appears that we're organizing around a super-node structure where all the large 'apps' handle their own changes off-chain (or alternatively based on regular trust.) The bottom layer will still support payments but it will be used less often. With more transaction activity happening on layers above. I think its still important to scale the chain and make it as secure as possible. Bitcoin has a unique focus here on security above everything else. Making it ideal for long-term hedges. For every day stuff I can't think of any chains that have more credible R & D than Ethereum. They seem to even have been doing research on the P2P layer now which traditionally no one has cared about.

Doing it the right way doesn't take any longer than doing it the shitty way. Successful startups focus on extreme minimalism, not focus on doing the worst possible job.

Frankly I'm not even convinced that double-entry is the sole right answer in this space. There are things you need to be able to represent, but the original reasons for doing double-entry (making errors when subtracting figures) no longer apply.

(I've worked at investment banks and fintech startups)

If you're starting a bank or need a ledger these days (and aren't using a core banking provider that has one), then i usually recommend Tiger Beetle.