Pierre: This is an issue that affects our audience directly, right. This is the Noded podcast. It is for people who run Bitcoin full nodes and we had what is called a critical vulnerability exposure - CVE drop on Monday and there was an issue with our node software - specific versions and we can get into that.

bitstein: And in a bombshell Tweet John Newbery said that he was responsible for the CVE.

Pierre: So normally when you do something really awful to Bitcoin you pretend like you didn't do anything bad and you go on your merry way - historical pattern. It was kinda surprising to see John take the fall for this. Especially when you go look at the evidence you don't see his name anywhere.

John: My words were taken out of context.

Pierre: All kidding aside, discussing Bitcoin's problems and not just its strenghts is an important thing. And when we think about Bitcoin's trustlessness it always has an element of we have to trust the underlying open source software that we're running in order to be using the Bitcoin system in a trustless manner. And so I think that bugs like we're about to discuss highlight the limitations of the trust model that we have. But it seems to be the best we can do.

So let's start with an explanation of what the CVE was and then we can talk about the code itself.

John: So shall we start with the CVE itself and what it actually says? So CVE-2018-17144. You can find it on the NIST site I think but the description is Bitcoin Core 0.14.x before 0.14.3, 0.15.x before 0.15.2 and 0.16.x before 0.16.3 and some versions of Bitcoin.. allow a remote denial of service application crash exploitable by miners via duplicate input. An attacker can make bitcoind or bitcoin-qt crash.

Pierre: This was written up after receiving information from what at the time was a pseudonymous person. We didn't know who it was at first who submitted this. Bitcoin Core has a blog post about this and I'll link to this in the shownotes. The email initially went to I think 4 or 5 different people describing the issue that had been discovered.

John: That's correct. It was sent in by someone using the pseudonym... and it was sent to Pieter Wuille, Greg Maxwell, Wladimir all of whom are Bitcoin Core contributors. Pieter and Wladimir are maintainers. And also to deadalnix who is the lead maintainer of bitcoin-abc and sickpig who is either a contributor or maintainer of Bitcoin Unlimited.

Pierre: So I’m sure this is not the first time that that group of people has received an email about a security issue whether it was a legit turned out to be a problem or a perceived security issue that they discovered was not. So how do they sort out false positives from the information, the emails they are constantly receiving from people?

Pierre repeats question

Pierre: So it seems as though they quickly came to the conclusion that this was a problem. Do you have some insight into why it was so easily seen as a problem.

John: I’m not on that security list so I don’t see any of those things. I can’t tell you how much traffic they get. The report was later made public and put on the Bitcoin dev mailing list and it is quite a full report. If you look into what the details are it is very easy to justify looking at the code or writing a very short test, very easy to validate for yourself that this is a problem.

Pierre: So after they received that.. they received it at 2:57 UTC and then at 3:15 Greg Maxwell shared the report with Cory Fields, Suhas, Alex and Matt at Chaincode Labs who are your co-workers. I don’t know what time that was Eastern time….

John: I wasn’t actually in the office that day so I missed all the fun.

bitstein: The one day John doesn’t show up

Pierre: I imagine that there was probably a commotion at Chaincode Labs when this got forwarded or was it pretty calm?

John: I don’t know. I imagine pretty calm, pretty cool customers here at Chaincode.

Pierre: So then at 5:45 Matt Corallo identifies inflation bug so in the email that came in…. This seems to imply that the email only talked about the denial of service issue of clients that are of a certain version crash when they receive a block created by a miner that has a transaction in it that is double spending an output. That’s the denial of service attack that just crashes but then there was an inflation bug. Do you want to describe the inflation bug?

John: So it is a bug in both cases. If your node receives a block in which there is a transaction that has duplicate inputs it will do something wrong. And depending on what version of software you are running and where that transaction originated the wrong it does is different. If you are running any Bitcoin version prior to 14 you will do the right thing, you will reject that block because a transaction with duplicate inputs is invalid so the block is invalid. If you are on a version of Bitcoin between 14.0 and any 14 version, if you receive that block you will crash which is bad behavior. If you are on version 15 or 16 depending on whether the duplicate transaction is spending from a transaction in the same block or from a transaction in a prior block you will either crash which is bad or even worse you might accept the block as valid in which the case the miner has been able to inflate the supply of Bitcoin.

Pierre: This is close to home because my favorite hobby horse in Bitcoin is its monetary policy and if it got inflated by a bug I think that is like a worst case scenario for my relationship with Bitcoin.

John: And you wouldn’t be alone Pierre. I think a lot of people would be rather upset.

Pierre: We’ve gone back and verified that the validity rules that we expected have not been violated in the past. So its a bug that while present was not exploited that we know of.

John: That is correct, definitely not exploited. There’s no block in the chain that is doing this and if any miner tried to do it either in the past or now, that block would be rejected by your upgraded node.

Pierre: Back on the timeline, there were people that were making noise about how the miners were kind of given priority in this process over non-mining nodes and that being a vindication of a certain point of view that miners determine consensus and are not just the humble servants of consensus. So what is it that led to the decision to speak to Slush pool and get miners involved in the situation.

John: I can’t speak to that decision or any decision made by Matt or Greg or Pieter. I wasn’t part of that and so I can’t tell you why they chose to talk to Slush pool but I can give you the rationale of how you would want to approach one of these bugs which is you want the fix to get out before anyone exploits it but you don’t want to be waving big flags saying there’s a bug here that you can exploit. There needs to be some careful consideration of how fixes are rolled out and the article that you’re talking about bitcoincore.org goes through the timeline and explains exactly what happened in this case.

Pierre: To me, the issue of whether miners get notified earlier or later in the process is that if they’re mining on an invalid chain that then gets reversed they end up with a loss of funds whereas with other people, if it gets reversed then maybe their transaction is reorg-ed out and has zero confirms when they thought that they had x number. But it is not really an immediate loss of funds like it would be for a miner, invalidating the block reward.

John: But there’s still a double spend risk so if there were a block that had exploited this and it had been extended by other miners then as soon as a fix was released and that chain was reorg-ed out of the best chain then all of those transactions would no longer have any confirmations and they could be double spent so you want miners to be mining on a chain which is correct and does not have invalid blocks. I think Pieter and Greg and Matt and others reached out to miners and also important economic nodes so exchanges… Bitcoin Optech reached out to its members and said you should upgrade and the idea was to try to get the large economic nodes upgraded as quickly as possible including miners.

Pierre: So before there was this 16.3 release do you have insight into what the decision process was in actually creating the pull request and how to present the code…

John: No, sorry

Pierre: That’s alright. We had the wrong person on the show. I should have dragged Matt on here not you.

John: Matt’s too busy doing real work.

Pierre: That’s fair. So after this got fixed and I ask because deadalnix was like “I fixed this as part of a refactoring so that no one would see it” and I heard something like that about some kind of bug. I don’t know if it was this one. I thought that was clever but also like, how much of that do we want going on in the codebase of like people slipping in changes that are unrelated to a refactoring that they’re doing and changing validation rules. But at the same time, doing it publicly you’re announcing to the public “hey there’s a bug here” and because you’re releasing the code before you’re releasing the binaries, there’s a chance that it gets exploited very quickly. So I don’t know, I thought that was interesting. There’s different ways of looking at the pull request.

John: Yeah. I think there’s maybe trade-offs and it is interesting to think about but I can’t give you any visibility into this one.

Pierre: And then the other thing that got brought up was the alert system and how it had been removed and thus it made the network vulnerable to a situation where the client has a vulnerability and node operators are not updating their software.

John: That’s an interesting one as well. There’s a really good bitcointalk thread on just that subject and I think you can put that in the show notes. In that thread, Greg talks about the alert system and the problems with it. I think Pieter Wuille has Tweeted about the problems as well. This is a debate that’s been had and really I think been put to bed months or years ago that the alert system is bad and should go and it’s gone. I don’t think there’s any new information here that informs that decision.

bitstein: So much so that even Satoshi’s keys have basically been revealed to the public. The private key itself so the original keys are compromised. There was a lot of good discussion around that time reminding everyone of why the alert system is a highly centralized choke point in Bitcoin.

John: Yeah and Bryan Bishop talked about that in a talk at Building on Bitcoin. There’s a Bitcoin dev post about that. The final step was to reveal the private keys to completely put this thing to bed so it’s gone.

Pierre: It’s funny because on the other end we have a complaint of “Oh well the core developers are pressuring people into updating their nodes and this is centralization…” and I even saw someone say that what you should do is downgrade your node to the last known good version which I thought was an interesting take.

John: There are other problems with older nodes. I wouldn’t recommend anyone downgrade to 14 or earlier.

Pierre: On the timeline, that’s pretty much it. I thought that one of the interesting things was that immediately after this got out, people look at the pull request and the changes made and then they just look at what was it before and who changed those specific lines of code before. This is Matt Corallo because this is the lines of code that he changed and I was reading a piece that will be coming out soon by Jimmy Song where he kind of outlines this was a confluence of changes and if you look at this line in isolation then you’re like it was one person and there were reviewers of that pull request. But then you look at the wider context the reason why that one change was not ok, was due to like three other pull requests which were not the same person and touched very different parts of the codebase at very different times. And all of this got together to be the perfect storm of the inflation bug and it reminded me of a YouTube video about programming where the guy goes “the really good bugs are teamwork. It takes a team to really introduce a good one”.

John: I think that’s right. So, if you look at the pull request (I think it is #9049) that basically removes this individual check for validating a block, it looks pretty obvious that it’s a bug and in hindsight it is pretty obvious it is a bug but in the wider context prior to 2013 that check was redundant and there was another check somewhere else so it wouldn’t have been a bug if that PR had been merged prior to 2013. That’s not to excuse it, it’s still a crash bug we should’ve caught it but this stuff is pretty complicated and validation for processing a new block is pretty complex and spread out over lots of different functions. It seems overly complex and ripe for refactoring and if this was anything else you might refactor it and say this doesn’t make much sense. We should put this here but in consensus you don’t want to refactor because any small change can introduce a consensus bug or a consensus incompatibility. So even though it’s complex and difficult to parse where things are happening.. it could do with better commenting for sure but you can’t really fix it by moving things around because that will do more harm than good.

Pierre: Traditionally, if you come across a codebase that is what you’re describing…. In Bitcoin it gets taken to the nth degree because of the consensus nature of Bitcoin but we see this happen in companies as well where they’re all like “hey if it ain’t broke don’t touch it because if you break it we’re screwed - it’s our core platform” and usually to me that’s a sign that there’s not enough automated testing going on so that they’re not confident about the coverage of the behavior of the system with automated testing because once you have automated tests in place…. granted I’m specifically talking about systems that are not consensus based like Bitcoin… then you can start refactoring with a lot more confidence in moving things around… so do you think it’s like a combination of there could be a better test environment for it or any number of solutions like that or would that not begin to cut it? Or the current test coverage is already fine. The problem really is about the consensus.

John: No I don’t think the current test coverage is fine. I think it is ok and it’s definitely a lot better than it was 2 years ago. Two years ago it was much better than it was five years ago. Initially there were no tests. So it’s getting better but I find it implausible that we would ever get to a stage of testing where we would think ok it’s safe to refactor all of consensus now because even the most edge of edge cases, the smallest of changes in consensus opens up the possibility of a consensus failure and that can be leveraged to split the network or double spend or carve up the network and re-org. You want to be 100% certain that any change you make consensus code does not introduce incompatibilities and that’s almost impossible even with 100% line coverage testing. It’s really difficult to get all of those edge cases.

Pierre: I’ll also link in the show notes to Greg Maxwell posting on the issue of testing. He’s got some views. I don’t know if I share his views. He seems to be pretty skeptical about unit tests and he finds that unit tests in practice end up being a lot of dumb testing not smart testing.

bitstein: He does make some interesting points with that because there is a problem with unit testing and I remember writing my first unit test as a software developer years ago. You gravitate towards writing tests where you know the actual outcome ahead of time in the sense that you get that green check rather than writing smart tests that really think carefully about the edge cases.

John: Sure, you can write bad tests. That doesn’t mean tests are bad.

bitstein: And then you have to test the tests. Who will test the tests?

John: The code tests the tests. Greg also talks about mutation testing where you mutate the product code and check that the tests fail because if they don’t fail they’re not really testing anything.

bitstein: Eric Scrivner in the chat is saying that testing has the combinatorial explosion problem too. Specification and statistical certification are probably more effective.

Pierre: So in the Ethereum world with smart contracts we’ve started hearing about formal verification of smart contracts. I’d be interested in hearing whether you hear conversations among core devs about these other approaches to verifying the correctness of the software that we’re building.

John: It’s not something I know anything about. I think it would be slow as far as I’m aware and performance is also important in your Bitcoin node but I’m afraid I’m not an expert on formal verification.

Pierre: What percentage of the testing on Bitcoin is integration tests vs unit tests?

John: We use integration tests a lot and I would argue we overuse them because the Python test framework is really easy to work with. I’ve seen cases where I’d personally think it would be more appropriate to use unit tests or test smaller components but it’s easier to just use the Python test framework to spin up a node and send in messages. Even if you’re trying to test one line of code you’re spinning up an entire bitcoind node to test that. I think unit tests could be improved. I think our integration tests are pretty good right now.

Pierre: On the subject of the need to improve things in this software which is open source, people talk about the lack of developer incentives. They say all the developers are working on ICOs because they make money by issuing new units of XYZ. Bitcoin lacks this incentive for developers and thus is going to have a tragedy of the commons. Bitcoin won’t be able to keep up technologically with its competitors. What do you make of that line of reasoning?

John: I’m not convinced by that. I think we have pretty robust developer group working on Bitcoin; Bitcoin Core and other Bitcoin projects. It is getting bigger all the time. It is bigger now when I started two years ago. We’re having new developers coming in all the time. I think we’re more bottlenecked on experienced reviewers and maintainers. That’s where the main slow downs are and I don’t think adding new developers into the mix at the bottom would solve the problems. It would be good if we could improve our processes so that we could take some of the load of those maintainers and if we can get better experienced reviewers but that takes a long time.

Pierre: I think there is also the effect of reviewing someone else’s code is not necessarily as fun as writing your own. Do you think that explains it all of why there’s 250 open pull requests and it’s not always easy to get your code reviewed by someone who’s experienced.

John: Possibly there’s more glory in getting commits and adding review comments. I think a lot of people don’t feel as comfortable reviewing other people’s code in Bitcoin Core because there’s the perception that you’d need to have more experience to review other people’s code. But yeah we could do with more review and definitely more experienced review but like I said it takes quite a long time to get familiar with the system to be an experienced reviewer.

Pierre: Someone approaching this might be like “ok there’s no point in me reviewing this because I’m not experienced”. How are you going to get experience if you don’t start reviewing things?

John: Yes

Pierre: How do you see it unfolding? Is it just going to be a natural process of today’s beginner or intermediate reviewers gain more and more expertise over the years and then development will accelerate or are there ways of training for reviewing. Jimmy has been teaching developers about the fundamentals of Bitcoin. You could have a secondary course after that specifically from a software development perspective here’s how you would contribute to Bitcoin Core.

John: Yeah potentially. That’s something we’ve done a couple of times at Chaincode. We’ve had residencies where we’ve trained developers on contributing to Bitcoin Core. One of the activities I organized during the second residency was exercise pull request reviews where everyone reviewed the same pull request. We’d review each other’s reviews. I guess other people could do that. Yes maybe it is a natural process and over time people will review but I think just crossing our fingers and hoping it will happen is not good enough. I don’t know what the answer but we definitely need more and better reviewers.

Pierre: How many Chaincode Labs do you think could exist before you would say there are too many?

John: I don’t think there is a number. I’m happy with the work we do here and I’m happy that other companies are beginning to sponsor and support and hire open source Bitcoin developers but I think the more the better.

Pierre: One of the problems with the narratives that are driven on a political level by participants in the crypto industry is that the core developers either have to be elevated to an infallible state of perfect expertise or they’re the s***iest code monkeys you’ve ever met and they’ve ruined Bitcoin. It has to be on one of those extremes and I think that has driven by the underlying political debate for scaling solutions between onchain scaling and layer 2. If you set that aside, the problem with it in my mind is if on the Bitcoin side we say we have the smartest, best developers, the message we’re sending is that we don’t need more developers and we don’t need to invest more in educating people on Bitcoin development because we’ve got it covered. That’s the problem I see on the Bitcoin side of the narrative.

John: Yeah maybe. I think we are blessed with some very very intelligent and capable developers. I don’t want to say they’re the best in the world because I haven’t judged and assessed all the developers in the world. We’re lucky we have people like Matt, Pieter and Greg contributing to the project but we could always use more. That’s personally what I spend a lot of time thinking about and working on. More developers would be good, we want good quality developers. I think to scale the project safely the number one priority should be on improving our processes around our maintainers. I think Wlad and the other maintainers are a bottleneck. They’re overworked and it would be great if we could find a way of removing the stuff they don’t strictly need to do so they can focus on the high value stuff. We can therefore move faster but still safely.

Pierre: This brings to mind this idea that the only people who can work on Bitcoin are people who know Bitcoin very well but I would challenge that with the fact that there’s a build system for Bitcoin Core using this library called auto tools. That’s something that someone would not necessarily have to know much about Bitcoin to improve, maintain or refactor; whatever needs to go on there in terms of tech debt. You can have someone who is just basically an auto tools expert and a build system expert be focused on that.

John: And have someone who’s a qt expert on the GUI.

Pierre: And on the RPC side, I don’t know if there are RPC experts out there. API design experts there are.

John: Certainly there are. And ours definitely could be improved.

bitstein: On the subject of whether or not Bitcoin Core has the best developers in the world, certainly has some really fantastic developers, David Harding has sent in a question for this show. A lot of people are going to be contemplating what went wrong and I look forward to reading their conclusions. For now, can you describe what you think went right? We now know who the original bug developer was but who else do we have to thank for ensuring all the drama was on Twitter and Reddit rather than on our nodes.

John: I think that probably the people in that bitcoincore.org article. So Wladimir, Pieter and Greg were the first ones to receive the email and then they talked to Suhas, Cory, Alex and Matt. I didn’t have anything to do with that. I think they handled it well. The 16.3 release, the code was written, deterministically built, released the next day, within 24 hours. That was really fast considering all of the steps needed to get a release out of the door safely. Cory Fields… and other contributors turned it around really quickly. We got a 15 and a 14 release, the patches out really quickly, I don’t know if the binaries are available yet but those people, you can see their names on the article, they’re the ones to thank for handling this professionally.

bitstein: We should also thank awemany for sending in the bug. I wasn’t as much a fan of his write up on Medium.

John: That took the shine off but I still thank him for reporting the bug.

bitstein: Clearly this was something that needed to be known and needed to be fixed. They seem to have found this bug whilst working on bitcoin-abc or Bitcoin Unlimited code. Tied with this debate around this bug, renewed interest in the notion of having multiple implementations and perhaps that being a better thing. Can you share a little bit on your perspective on multiple implementations and if it has changed at all because of this bug.

John: I don’t think it has changed because of this bug. The jury is still out. I have always thought that in theory multiple implementations would be a good thing. And I’ve always thought that two implementations is exactly the wrong number of implementations. If you want to go to sea with a Bitcoin implementation you should take one or three, never two. That’s an old Bitcoin joke. Because if you have two and one diverges from the other you don’t know what the true state is. So multiple implementations in theory is a good thing because they can check each other but there are lots of good answers against multiple implementations in Bitcoin and some of those are laid out in the bitcointalk thread, the same one that went into the alert system. I still believe that it’s possible, maybe not now, maybe at some point in the future once we have a libconsensus that can be extracted from Bitcoin Core and then reused in other implementations or maybe when there’s more money in developing implementations and companies are interested funding open source implementations. I think in a world where there are multiple implementations you would want the big economic players to be running more than one implementation in house. It wouldn’t be enough that you’d just run one.

bitstein: Preferably three.

John: But if you’re running two then at least you know when there’s been a divergence and you should maybe wait for more confirmations or cease withdrawals or whatever. I still like the idea. I don’t think we’re anywhere close to it right now. I think the jury is still out. I think it is a good thing that this bug has got people talking about it again because I don’t think it has been settled. I know that Greg and Peter Todd have spoken quite consistently against multiple implementations but personally I’m not convinced.

bitstein: I think some of the difficulties with it are touched upon by Greg Maxwell in that thread is that if you require having multiple implementations you definitely need to be running them so you have a sense of when these things are happening. It creates an even higher barrier to entry for people who want to run nodes. That adds even more costs to the individuals who want to maintain their financial sovereignty. Although if that is required to have certainty about the network, perhaps those costs are worth it. That’s the cost of playing Bitcoin.

John: Perhaps and perhaps large economic nodes like exchanges would want to run more than one implementation but for most of us just running one would be enough and we would accept transactions that have a certain number of confirmations and if there were a divergence between implementations we would read about it pretty quickly on Twitter or Reddit or wherever we get our Bitcoin information.

Pierre: It would be interesting if you could run multiple Bitcoin implementations using the same block database so you’re not taking up more disk space. I don’t know how it would work on the network side. That’s a huge engineering challenge.

John: You’d be wanting to feed the same data into multiple implementations and seeing that they converge to the same state.

bitstein: This has been one of the annoying things about the push for multiple implementations. Many of the people who have been pushing this idea, their implementations that they choose to work on are completely hard forked from Bitcoin.

John: Right. That’s something completely different.

bitstein: It seems to be like concern trolling if you’re not going to work on that other implementation and go make a completely divergent, unrelated coin.

Pierre: If you’re already doing a hard fork or creating an altcoin then you don’t have to worry about what we’ve been talking about, the consensus. That’s the part I don’t get. Here’s an opportunity to remove all the tech debt that’s in Bitcoin and then rewrite it in Rust or something and then get going on a solid footing. But obviously they’re just cashing out, they’re not interested in actual software development.

John: Maybe. I don’t really care. It is an interesting question about multiple implementations no matter who it comes from. We do have multiple implementations in Bitcoin but those other implementations are not widely used. There’s bcoin, btcd, libbitcoin but they’re not widely used and not obvious to me that they’re consensus compliant with bitcoind. If you are in a multiple implementation world you would want your multiple implementations each to be well maintained and have an active developer community. I don’t think we have enough developers to spread out across multiple implementations right now. If we look at the other blockchain that does have multiple implementations, Ethereum, they have one implementation that is widely used and one that is not very widely used and I think that’s it. I haven’t seen it demonstrated that this works.

Pierre: What about lightning and you’ve got multiple implementations of the BOLTS. You’ve got lnd that’s written in Go, c-lightning in C, eclair in Scala. It seems like lnd is always leading. I’m sure I’ll hear about it from the gentlemen at Blockstream…

bitstein: That’s offloading the consensus of Bitcoin to the underlying Bitcoin node its connected to.

Pierre: I agree but you could argue that the lightning network is forming its own consensus- how the lightning nodes communicate with each other. But I think drawing the distinction between the consensus of a lightning node vs that of a Bitcoin node is important to show why its the case that lightning is more amenable to more implementations.

John: If there is an interop failure between two lightning implementations there is money on the line for the people who have money in those channels that are traversing those implementations but that’s it. It could cause some disruption in routing across the lightning network. If there’s an interop failure between multiple implementations of Bitcoin then the entire economy of Bitcoin is on the line. It is a completely different order of magnitude of risk and that’s why it’s a completely different problem domain. You can’t really compare the two.

bitstein: It is kind of a localized consensus failure vs a global consensus failure.

John: Sure

Listener questions

Pierre: This one is interesting because we’ve kind of been beating around the bush on it. Re the lack of a formal specification. Does John think it is due to practical reasons, no one wants to spend time on it, or deeper technical reasons. We’ve seen people say the code is the specification, that’s not acceptable. That kind of back and forth.

John: It is really difficult to write a formal specification because of all of the edge cases. If I sent you Pierre a transaction that just happened to hash to all zeros and you tried to spend the first output from that transaction would that be a valid spend?

Pierre: I have no idea John.

John: The answer is that it would not be a valid spend because there is a rule that says there can only be one coinbase transaction in a block and a coinbase transaction is one that has an input that is null so instead of pointing to a previous transaction it just has all zeros in the outpoint. If you tried to spend a transaction that happened to hash to all zeros, that would appear to bitcoind as a coinbase transaction and there would therefore be two coinbase transactions in your block. Obviously that’s not going to happen because if you can find a transaction that hashes to zero then SHA256 is in trouble but that’s not an explicit rule. No one would write that in the documentation of Bitcoin the protocol because no one ever talks about that. It is just a fact of the implementation that that’s how we identify a coinbase transaction and I bet there are many other things like that. It is really difficult to enumerate all of the rules and all of the edge cases and that’s just in the programming logic, all of the obvious stuff in the source code. But consensus is everything, it is not just the stuff in the validation.cpp file or the other source files that say consensus. Peer to peer code is consensus because if you can break the peer to peer network you will stop nodes from reaching the same state, re-org behavior if you can make re-orgs difficult or costly then that’s consensus, the database backing your blocks is consensus. If you wanted to write documentation that contained all of consensus it would end up being the source code of Bitcoin Core because that describes the behavior of a Bitcoin node. It’s difficult, I don’t want to say it’s impossible because it’s a nice goal to have but it’s really difficult.

bitstein: In that bitcointalk thread too, Peter R from Bitcoin Unlimited pointed out that a spec would not have helped this bug because it’s obvious that the specification for Bitcoin is that there’s not supposed to be any additional coins and this bug caused a situation where had it been exploited additional coins would’ve been on the network. The specification as far as the monetary policy is very simple but the specification may have not gotten to have seen these edge cases. The inflation bug that could’ve been exploited was obviously not supposed to have happened so a specification would’ve have clearly have said there’s 21 million coins.

John: Fine but Bitcoin Core until BIP42 did not comply with that specification. BIP42 fixed a bug which was in 0.1.

bitstein: This is exactly the point. We’ve been writing it down informally the entire time Bitcoin has existed that we’re supposed to have this specific monetary policy and yet all of the nodes operated in such a way that this was not the case. This is showing we often bring up the extreme edge cases of possibilities thinking of the very complex rule set. Here is a case where we had a very simple rule but there were edge cases that didn’t follow it.

John: Yeah. I think it is a pretty well established rule that there are 21 million Bitcoin and anything that breaks that is a bug. It doesn’t matter if it was in 0.1 or it was introduced in 0.14, that’s a bug.

Pierre: How can we further increase participation in the review process? Is there a place for bug bounties?

John: Greg Maxwell seems to have been writing full time for the past week. There’s just pages and pages of stuff and he addresses bug bounties somewhere saying that bug bounties are generally not very useful because if you want to incentivize people to work on finding bugs then your expected return from a bug bounty is pretty low and it generally ends up just giving the bounty to people who would’ve done the work anyway. There’s a question whether you can incentivize someone whose found a bug to disclose it responsibly rather than sell it on the black market but in general he doesn’t think they’re a good idea. I don’t have much of an opinion but they don’t seem to be a golden bullet to me.

bitstein: Someone pointed out to me today a problem with bug bounties that had not crossed my mind before. Someone might have an incentive to insert bugs into Bitcoin so that when there’s a bug bounty for it, they can also remove it.

John: Yeah maybe. I think that if they had the ability to insert bugs into Bitcoin, I don’t think the presence or absence of a bug bounty would change their behavior much.

Pierre: What concrete things can non-technical people do to help the Bitcoin development process?

John: Translations, documentation work - you need to be technical but you don’t necessarily need to write code. You need to be good at writing words. Testing of course. Learning about Bitcoin for yourself is probably the best thing you can do for Bitcoin. No matter how technical you are you can still learn more. Listen to Noded, contribute to Noded of course.

Pierre: Read Mastering Bitcoin. It is actually pretty approachable for a non-technical audience though some parts go over my head. Generally it is dumbed down, thanks Andreas. It is not too technical whilst still getting very close to the reality of what’s going on.

Pierre: Please summarize the code review process and comment on how to prevent deference to authority. The code review process for Bitcoin and for specifically the C++ implementation has evolved over the years from Satoshi reviewing (or not reviewing) his own code and releasing it to today where we might have six people review a change of code and then everything in between.

John: If you check the PR for OP_EVAL that’s a really good example for how code review shouldn’t be done. That was I guess 2012. Generally, since I’ve been involved in the project, the bar for code review has been really high especially in consensus code. So that’s what makes this PR #9049 so perplexing and unusual that it would be merged with very little review. It seems only one ACK. So it is an outlier and it shouldn’t happen. I think in general we’re very good. I was surprised when I saw #9049. The question about the deference to authority, I think it is valid because I doubt #9049 would’ve been merged if it had been opened by random contributor 1 and utACK by random contributor 2. It was opened by Matt Corallo and utACK by Greg Maxwell. This carries weight but it shouldn’t carry so much weight that it just gets merged. I think we’re all human and we use this heuristic of authority when judging code or judging anything and we do our best not to but in this case we’ve fell into that trap. It was merged too soon and that didn’t raise enough questions and we’re all to blame for that. I don’t know what can be done to improve that.

Pierre: Let me float a solution. Every pull request that gets opened, it generates a new pseudonym for GitHub and so you never really know whose pull request is whose.

John: It is an interesting suggestion. I’ve not done much hiring but for the residency here, when we were reviewing applicants I made sure that the comments each of us made were not visible to others commenting on those applications because you don’t want to influence other people who are reviewing things. So you can do that in terms of code review? Maybe it might work, I don’t know if you can do it on GitHub, I’m pretty sure you can’t.

Pierre: It would be a lot of work to open a new account every time you want to open a pull request and be logging in and out as you do stuff.

John: The authority heuristic is useful in a lot of cases. If you see people who know the subject and they know that area of code and they ACK it, that is obviously worth more than people who don’t know the code so there is use there but in this case for whatever reason that caused a PR to be merged prematurely.

Pierre: What kind of block analysis would’ve revealed the inflation exploit in action? If it had been used how likely would it have been for someone to notice it without code review?

John: There are old nodes on the network that would’ve rejected this block so we would’ve seen a divergence for any node that was older than 14.0 and presumably someone would’ve have said something on Twitter or Reddit. I know of people who are running old nodes and check them. If everyone was running say 0.16 and this was exploited to inflate the money supply I don’t know how quickly it would be noticed. If you switched off your node immediately after a block that exploited this and then restarted it you rewind the last few blocks and that would catch this bug. Your node would fail and you’d look into it and you’d see some anomaly but if everyone was just running a node I don’t know how it would’ve been seen.

Pierre: Has this event changed your opinion on how quickly the Bitcoin protocol should ossify? I want to hit both the protocol and the codebase because frankly this had more to do with the codebase ossifying than the protocol. Should we stop refactoring Bitcoin Core?

John: No I don’t think so. Bugs would not be fixed. This wasn’t a change just for the sake of making a change, there was a rationale behind the PR that introduced this problem and that rationale is the block propagation speed is important and the lower you can make that, the lower the stale block rate and the more secure the system. It’s not like this was someone mucking around with the code for no good reason. In this case, it failed, a bug was introduced but in many other cases, the block propagation speed has been improved and that has made for a more robust and secure system. If you said don’t make any more changes, we would have a less secure system because Bitcoin would be less performant, there would be more stale blocks, there would be other bugs that weren’t fixed. This one is obvious, this is an obvious failure but there are a lot of obvious successes that aren’t seen because they just work.

Pierre: But if you told me “Pierre, do you want to have no inflation bug or do you want to have faster block propagation with an inflation bug?”. There is a risk every time there is a change made and maybe we’re overvaluing the value we get from the change and underestimating the risk it is presenting to the system. Do you think there’s cases of that or generally we’ve got to keep moving forward anyway?

John: I think generally we’ve got to keep forward anyway. That’s not to say we shouldn’t learn from this and ask ourselves very difficult questions.

Pierre: Is adding Schnorr a big enough improvement for us to risk something bad happening? Some unforeseen bug or behavior that we didn’t expect.

John: Yeah I think it definitely is worth it. Schnorr signatures in the Bitcoin protocol is at a completely different layer from the accounting. Adding Schnorr signatures should present almost zero risk of introducing an inflation bug because they exist on completely different levels. When SegWit was introduced, the design and the code had enough foresight to add script versions so it is now easier to add new script versions and add new features within script so it’s a smaller change than SegWit. The benefits are just so enormous in terms of fungibility and scalability so that I don’t think we should stop. We should go ahead and do it carefully and give it a lot of review like SegWit got but I don’t see this as a reason for not continuing to improve the protocol.