Richard: So today we are going to talk about Signet Lightning, PlebNet Playground, which was the first evolution of me playing with custom Signets, and eventually ZebNet Adventures. This is how you can become Signet-rich.

First, we should probably do an overview of the various network types. Everybody knows about Mainnet. This is what we use day to day, but there's a few other testing networks. RegTest, or SimNet, if you are running BTCD, is a local unit testing framework. And this is really useful for unit testing, but it has some disadvantages. Primarily, anybody can build a block. So if you're trying to share this publicly, this can be a problem. You'll end up having some external participant potentially take over your network, and that might not be useful. Then you have TestNet itself, which is a public TestNet, but you need mining hardware. Anybody can mine blocks. And one of the issues is that block production is incredibly inconsistent. Sometimes you'll have blocks coming every minute--sometimes you will wait hours. And so if you're trying to simulate a Mainnet network, it is not super conducive to that. And then finally, Signets, which surprisingly few people seem to still know about. It's been around since sometime in 2018, I believe. And there's both a public Signet, which is similar to TestNet, or you have the option to build custom Signets. Effectively, Signets still has proof of work on it, but it also has this signature block section, which you can lock down who is authorized to build blocks.

And so if we look at the Mainnet liquid network, we see there's a lot of liquidity, many nodes. There's activity going through there now. But obviously, trying to test things on here is pretty reckless. Then you have RegTest. Highly recommend Polar if you are playing around with anything related to Lightning. It's a really easy way to set up RegTest Lightning environments at a click of a button. But as we said, if you want to start networking with multiple partners, RegTest probably doesn't work unless you 100% trust them, because they can completely screw your network up. Then there's TestNet Lightning, which has a healthy number of nodes. But as we said, it's, you know, block production is a little wonky. It's not equivalent to Mainnet in any way. And then finally, Signets. So it's BIP325, and it's a new type of test network where signatures are used in addition to proof-of-work for block progress. And this enables better coordination and robustness, or if you want to make it unreliable, you can make it predictably unreliable. And this allows for long-term testing with multiple parties involved. And so if you happen to look down here, there's only 24 nodes on the public Signet. And this is actually a major increase from a year ago when, you know, PlebNet was talking about, hey, we need to come up with a way for end users to not lose funds. Because in the first week that I was on PlebNet, I believe two or three different people lost 30 million sats forever, because they didn't have good key management or did something else stupid. And so playing recklessly on Mainnet's a bad idea. And so we need better alternatives to educate people about Lightning, because it is fairly complicated and key management is even more important. There was about eight nodes a year ago. And they were all serious cryptographers and I did not want to pollute their environment. So I started looking at custom Signets as a potential option. And so that birthed PlebNet Playground, of which you can go to plebnet.fun to learn more about the finalized solution. But this was the brainchild of like, 2 a.m. clubhouse discussion about how we could solve this. And then suddenly I was enlisted to build something. And a number of these things have been donated. All this UI work is some random person on clubhouse. Block Explorer is some guy in Europe. He's like, hey, I'll bring up a block explorer for you. So people have jumped on to assist in building this.

So originally it was supposed to be pleb-friendly and a place to learn. Currently the solution is still a little too complicated for the average pleb. We need to dumb it down to the point of umbrel and just like pop an SD card and it works. Or get some of these distributions to actually enable Signet or custom Signet support so just everything could flow nicely. But this fun little set of circles is the current network graph. Most of the blue ones are I have a fleet of about 25 nodes that are sitting in Docker containers that pretty much just constantly send funds between each other. And this allows people to do testing of applications. So it's been very useful for dabs. One thing I've noticed as far as the UIs: Signet application support is very spotty. Most of the time it could be very simple validation checks that need fixed, but someone needs to go in there and fix those things or at least add the features. And I think most of it is just not being aware that Signet is a network. And that's why the code isn't there. But the biggest thing is the automated traffic. With that it makes it much easier to start building application, especially if you're trying to do automatic rebalancing or fee estimations within the Lightning Network. You need traffic. And testnet doesn't have a ton of traffic. Certainly the public Signet has effectively none. The plebnet playground while the traffic manager is running is about like a hundred thousand transactions a day. It has very high fees. So you can just jump in make a couple channels to one of these nodes at a lower fee rate. And you will optimistically end up being a route path and get forward. So you can test your applications. Cryptoshark's LNDG which is a lightweight management system for LND. And also no data automation. He built almost all of it against plebnet playground originally. We also have had some complete plebs that aren't IT people end up building solutions against plebnet playground and then successfully deploying them to mainnet. Testing tester. The joke that he was never going to go to mainnet. He finally was comfortable with enough with Lightning. He went to mainnet. He has a node with me. You know it's tiny. It's 30 million sats something like that. He's turning over that liquidity every day. And doing thousands of micro transactions facilitating those, so and hands off. So it's certainly a good learning environment. And also just the layer one base chain of this plebnet playground. I know Randy McMillan has used it for Bitcoin QT testing on some UI improvements for the mempool. That way he didn't load up testnet or any other chain.

So how do you make a custom signet? Well, it's surprisingly easy. If you ever have questions, my first go-to reference is the Bitcoin wiki which is at bitcoin.it, and it has a full write up. These are just blatant copy and pastes out of the wiki. But it's not all that hard--little bit of bash scripting and you can do it manually, and this is how I built the original plebnet playground. This has been refined a little later in the talk. But you effectively just build a public private key pair. And you're going to end up using this as an authorization key for this new custom signet. And then once we have a public private key, we actually have to build a Bitcoin script. And this is above my pay grade, so I just followed the instructions and copied and pasted the real pubkey in there. And all these instructions work. And then you have a fun thing called a signet challenge, and this is what identifies different custom signet networks and keeps them independent of each other, along with I believe the chain code is going to end up being different. Once you have created this signet challenge, you pretty much just feed this into your Bitcoin comp. And you also will need to import the private key for this public private key pair so it can actually sign the blocks when you do block production. And then finally you run the issuer. And this is where it is a little confusing. There is this nbits mining target, which is essentially what your difficulty is. There's a bunch of instructions about like calibrating this thing. It's almost irrelevant. Because there is proof of work. And I can set it for a one second block time. But eventually it's going to realize, wow, I was running fast. And the difficulty will increase. And it eventually will naturally reach a 10 minute cadence.

So from PlebNet, I came to Zebedee, and we evolved PlebNet Playground into ZebNet. So ZebNet is a more feature-rich and polished version of PlebNet Playground specifically for Zebedee's needs. But also most of the containers and things built for this are also usable by anybody else as well, so we will be offering that at NBD shortly once I get it into the repo. But it has a number of features. You can set up any custom signet you want quickly and deterministically. So if you just start the container, I believe it is defaulted to Zebedee. But you can also just blank out a couple things and it will make a random signet and then spit out all the settings you need to reproduce that exact same signet again and again. So good for testing environments or deploying a larger scale signet of any type. It has full LN Mon support with Grafana checks, and then LN Metrics, which is a project of mine, which is very similar to LN Mon, but will also support core lightning. The network self wiring has automated traffic similar to PlebNet Playground, but with some additional improvements, including support for core lightning. So you can get, you can do apples to apples performance testing between multiple node implementations. There's some stress testing items that are in there, Tor support, you can play with neutrino if you want, you can play with the new RPC polling mode, which I think is thanks to Voltage guys offering that as a PR, which I highly appreciate. Remote signing support, which I haven't tested. Probably works. You can test, you know, either database type on the two supported daemons. And it also will export root keys for LND, which is useful if you want to start doing scaling experiments. And so the base components, some of which will be offered shortly: we have a Bitcoin signet docker image... Completely can spin up and boot up a new network. Makes it very easy to just have a docker compose or deploy to your infrastructure. LND signet, which is customized for Zebedee or you can point it to any other custom signet. Simple little Tor image. A C-lightning image that does have the GRPC stuff built in. The one thing with core lightning still, they don't seem to be releasing docker images on a regular cadence, so we've been having to make our own... Hopefully that will improve. I have a fork of LNDmon that supports signet, because the current one still does not have the proper validation code in it... That might change soon. And LNmetrics, which will also come to NBD repos in the near future, which offer prometheus metrics fields that are consistent between multiple applications. And then signet controller, which is kind of doing the magic of getting automated traffic and helping facilitate run the network.

The signet controller essentially has a bunch of API end points for us to play with things. And it will do automated funding of, you know, once it sees a node comes up, it will give it bitcoin, start making channels, start cross connecting things, and start sending traffic around, and also do rebalancing. Here's a quick shot of LNmetrics. I'm still working on getting it feature complete with LNmon, but at some point we should have everything with checkboxes and uniform output. Some other things I had a wish list for LNmon, it didn't have human readable names and things like that, which are useful when you're searching through things. Most people don't think in hexadecimal... I know I don't. Once you have these various prometheus outputs, you can do grafana monitoring and get fun graphs of automated traffic. You'll notice that eventually it gets into the red areas here. And this is where I have active push rebalancing since we own the network. Just kind of just keep things flowing so we can observe what happens. And all of these panels for grafana will work with any network type. So while I developed it all on signet, it's portable to testnet, mainnet, so you can reuse all those resources for your monitoring needs. And grafana alerts, which ended up being pretty useful on Sunday. This was the event I got when I woke up, saying that test net was broken. Always good to know that something is messed up so you can get to work to fix it. And I guess I've run through that pretty quick. So here's some resources, and I guess if you have any questions. Thank you.

Q: So who runs the mining for all of this to actually do the proof of work?

A: So on both PlebNet Playground and ZebNet, you have to run a mining process. It's CPU mining at this point. Because you don't really have any competition. But it will end up, whatever virtual machine you throw this on, it's eventually going to kind of saturate that core due to the difficulty. You can do some trickery and put a sleep in there so you end up having 10 minute blocks and so you only use 10% of the CPU on average. But yeah, it is a little resource intensive on that. Or I suppose if you had an ASIC sitting around, you could grind the block that you're trying to produce on that. I just have not gone to that point. But yeah, you do have a server or multiple servers. I could run multiple nodes, all trying to compete to make that block. If I wanted to block reorg, like simulate a reorg on chain, SigNet allows you to construct these scenarios fairly easily. Someone's got to do it. The example I have is very simple. It's actually a multi-sig, one of one, which is perhaps inefficient. But I wasn't writing my own Bitcoin script. Theoretically, that script block, I believe, can be anything. So you could probably do some really fancy stuff if you wanted as far as what the consensus mechanism is for block production. But I have yet to see a whole lot of people using custom SigNets other than maybe Jeremy Rubin. SigNet is the way that you should be trying to deploy a new feature on Bitcoin. That is what it is built for.

Q: Have you found... Trying to formulate this question... So SigNet's a great way to simulate a lot of things, but wall clock time, especially in lightning networks, still is very important. So like gossip and stuff like that. Have you... Where have you... What other places have you run into places you wish you could just speed up the clock? Because I've done a lot of work with this stuff too.

A: Yeah, yeah. Well, I tried to speed up the clock, but as I mentioned, it ends up slowing back down. So the container has a block delay parameter, and at least for the first 10,000 blocks or so, you can get it at about a second, a block a second if you want, which is still way slower than regtest, but pretty usable. Yeah, I'm trying to figure out a way to trick it where I want a turbo button every once in a while. I was talking to some core developers, maybe we can get these features added, because it sounds like they were pretty receptive to it. So hopefully. Because yeah, I think everybody wouldn't mind. Or if you could just specify this is a 15 block second time or something.

Follow-up Q: Well, I mean, like in the lightning layer, there's other places where the wall clock is important.

A: Oh, yeah. LND will freak out if it's more than two hours behind, the block is like two hours behind, because I had a problem. I wasn't setting the proper UTC time in my block production, and then I went off sync and took a little while. Also, that needs a better error message in LND. I've seen this too. So you're like, why? And then eventually you look through the code and you see why, but they don't put an alert that like, hey, your time is off. That would be useful.

Q: Hey, Richard. I had a similar question about why can't you just fake the timestamp that the block is attached to?

A: You absolutely can fake the timestamp, but your timestamp, at least for LND, I don't know about the other implementations. They have an explicit check of what is my UTC versus what the block is, and if it's more than two or two and a half hours, it's like, nah, I am not going to sync. I don't trust what's going on on the network. I don't know why they made that choice, but they did.

Follow-up Q: Oh, also, are you going to update the Playground to point to the containers, the Zebedees?

A: Yes, that is the long-term plan. I started working on it and then there was an up-and-congruency that I need to spend more time on it. Definitely the block, the mining process I need to switch, because if that node that is doing it currently dies, it's all manual.

Follow-up Q: Well, I guess maybe more specifically, is what ZebNet doing, is it super-C or super -Z? Is it necessary to even run plugnet?

A: It's a super set with bug fixes, so yeah, it might be advantageous for us to start using the ZebNet containers.

Follow-up Q: And then like retire Playground?

A: That's kind of what I'm thinking.

Q: Okay.

A: We're not going to retire Playground. We will keep that network independent for plebnet, but we can utilize these containers. These containers are generic enough. They're set up for Zebedee, but you can do your own custom signet. We just put in the Plebnet Playground parameters and now it is a Plebnet Playground.

Follow-up Q: Okay, so your version will just be a dependency for Playground?

A: Yeah, we would just point it as Docker images as opposed to currently we have the definitions that we compile.

Q: I saw a web interface a couple slides back. Who designed that?

A: Satoshi, somebody. It's surprising, it's like four lines of JavaScript with like 3DS or .js or something. I'm not a JavaScript person, but it's a surprisingly small amount of code to get like a 3D-weighted graph, and that's just data that's coming out of LND by itself.

Follow-up Q: There was one with more metrics and like graphs.

Richard: This one?

Asker: Yeah, okay.

A: So this is Grafana, which is a way that you can take various datasets and visualize them. So this will be released with LNmetrics, some of these templates, and then you can modify them.

Asker: Cool.

Richard: You can also get something similar straight out of LN Mon. They have a number of like Grafana dashboards. I just like some of mine better. Yeah.

Q: Can you give a quick like post-mortem of the Sunday event?

A: So yeah, Sunday I woke up about an hour after this, jumped on to Clubhouse and was like, hey, is the testnet broken? And Vic happened to be there, and so he reached out to Ben the Carman, and about two minutes later, Ben was like, yeah, something's messed up. And a few minutes later, I got on a Google call with Ben the Carman, Vic, and Brock, and we determined that yes, his 998 out of 999 multi-sig taproot thing kind of broke everything in LND. And he was unaware, he doesn't do a lot of lightning apparently, and so he was completely unaware that he had caused the issue in testnet, and only a few minutes prior had been broadcast a very similar transaction in the mainnet. So once we realized that, it was a matter of just reaching out to Lightning Labs and waiting, although a number of other people jumped into action. I know Dr. No on Twitter jumped in and tried to start hacking on the section in BTCD that was broken, but eventually Lalu got to it. And most of the time we were waiting for the continuous integration, so it was about eight hours from when I reported the bug to when we had binaries to deploy. But the good news was, you know, payments still routed, just your nodes would not know if, well, they wouldn't allow new channels and they would have no idea if you did a force close, and you could do a cooperative close, but it wouldn't know that it closed. So it was definitely in a bizarre state. I never say never.

Previous asker: Who wants the microphone? Somebody wants it.

Q: Thanks. I guess I don't totally understand the difference then with testnet and signet. So if this is running signet, how did you get the alert when that testnet went down with the bug?

A: So I'm using the exact same, all these monitoring tools can be used for any network. So they're independent of that. The feeds are the same, it's just a matter of is it mainnet, is it testnet, is it signet. And the big differentiator between testnet and signet is testnet anybody with sufficient hashing power can mine a block, whereas in signet, one, you have to hash, but you also need to be in this signature block, like have a proper key that is going to evaluate to true. So only it's a permission network.

Richard: I think that might be it. Thank you.