We can just spin up another one.
Yeah, sometimes Twitter is not the best and space as well, just crash.
I was in on a pretty large space with Mario now fall yesterday and it kept crashing, so.
Yeah, how do we spin up another one?
Oh, okay. But it's not the same.
Yeah, you can't do the same one.
Okay. Did you send the link?
Yep, I put it in the chat.
Nick, you're not on mute.
Sorry about that. We're having some technical difficulties, but we're going to reset the stage, get everybody back up here. We got Brendan up here.
Can I invite you up here, John?
All right, Brendan, I will throw it back to you.
Yeah, sure. Do we have a room?
I think we are just missing Paul.
Okay, let me see if I can send the link to Paul.
Maybe we can just get started.
John, I think that we were talking about latency and sort of the amount of control that we have over the latency for block proof generation and Paul was saying that, you know, this latency has a lower bound that's imposed by the sort of longest to prove transaction.
Yeah, so I think there's maybe two things on this topic. I think right now, probably one of the lines of optimization that we will be working on to kind of like hit, you know, future performance goals is going to be kind of analyzing the types of transactions that cause the worst case latency.
So in the initial analyses that we've been doing, you know, some of the data that isn't in the initial blog posts are like, you know, these greater ranges of blocks that we've, we've, you know, looked at and, you know, analyzing thousands of transactions to understand what are the characteristics of the transactions that cause these worst case scenarios.
And I think from there, you know, working with Robin and team, I think it seems like we'll be able to make a lot of improvements, both in terms of the kind of the way the application works from a runtime perspective and also ideally from an application perspective, you know, how it's implemented in the types of optimizations that we can take from there.
And I think the other area that I'm excited about as it relates to, you know, zero and the type one prover and being able to scale is like this layer that we haven't talked a lot about kind of the job distribution layer paladin.
And I think as that gets more mature, as that gets like better, I think we'll be able to be smarter around like which types of transactions, which types of workloads go to which servers.
And so we've been able to achieve these early benchmarks with really no customization of saying, you know, this type of proof would go better on this type of server. Right now we're running all of them on kind of a, you know, homogeneous pool of servers.
And I think there's some upside there in terms of like, oh, you know, because this this proof is going to be very demanding from a memory bandwidth perspective, we should send it to a memory optimized server.
This one is just CPU. So let's send it to work or workhorse or something like that. And I think that will also be some important levers that we can tweak as the as we get smarter on the job distribution over time.
Yeah, no, that's a really, that's a really good, good point, John. And, yeah, I think, I mean, in summary with all of this, it's like, you know, we're able to publish these numbers right now.
I feel really, really confident in them.
But I also feel confident that we're going to, you know, find ways over, over time here and like by over time I mean over the next three, six, 12 months in order to bring these down further and further.
But even as even for today, the kind of current price, you know, we're talking less than dollar extras for blocks. Like, if you look at what an Ethereum block costs right now, you know, there's a couple of hundred dollars worth of rewards that may come up in a theory and block, you know, not even counting mev.
Like, there's a lot of money that can go into into that. And like, currently, the pricing here would be would be workable for many, for many environments, right?
It's not going to add a significant end user cost at this point in order to also generate proofs.
So I'm really looking forward to getting this more optimized, you know, ready, you know, like actually running, you know, long term on some of these, and then finding the use cases that people want to build, given them, given the existence of these proofs, because we're just going to get cheaper.
And, you know, this is we're on the road toward, you know, this being a commodity piece of technology that can just work and can at low cost provide the benefits to end users.
Yeah, that's a really good point. I wish that we had Zach Brown on to talk a little bit more about some of the about Paladin, sort of the orchestration software that we use, because it's really, really cool.
And there's a lot of great engineering that's gone in, not only on the ZK side, but also in in orchestration and DevOps.
And it's just like a really, really neat thing to be a part of. But I think from this conversation, we can say that there are basically three levers that we can push on to reduce cost.
So maybe the lowest hanging fruit is on the DevOps side. So we can actually get like a lot of improvement, or we have gotten a lot of improvement on changing the instances that we run on on sort of like changing configurations and how we assign jobs to different, assign prover jobs to different computers.
The second one is on improving the ZK VM itself. And so in a minute or a few seconds, maybe Robin can talk about continuations, which is a mechanism that's inspired by risk zero.
And then the third is sort of the the longer term ZK improvements. And so eventually we'll get to Daniel and he can fill us in on kind of the plucky three work and the trajectory.
But Robin, maybe you could talk a little bit about kind of what's on the short term roadmap for further improving performance of the ZK VM.
Yeah, sure. So as Brennan said, on the short term, we are focused on continuations. So right now we're proving blocks on a per transaction basis.
And as Paul and John mentioned before, there are two potential issues with this first blocks with large transactions that get like most of the block proving time on their own, and blocks that contain a lot of transactions, typically like the inscription transaction where blocks are
accounting for like 1000 plus transactions, because we need to store data for before and after states of each transactions, it tends to get heavier as we go down the transaction list in the block.
And so these are the two overhead that we observe right now with the current design and continuations.
The technique that risk zero already implemented on the side is elevating both because we can then not break down blocks into transactions, but into arbitrary segments that give us fixed bounded amount of resources for the proven both in terms of memory CPU usage.
And, and this is doing this allows us to break down large transactions to break down blocks whatever the number of transactions they have into like, yeah, identical segments that take about the same time to prove.
And this really pushes as easy KVM performance to like another level because then we can really basically parallelize the whole chain of events at once, making it like much much more performant than what we have at the moment.
So we're super excited for the results that are going to come soon with continuations. And we also have additional improvements that can be done on crypto back end.
We can always optimize the statements etc to make it even faster to prove and cheaper.
Robin, I will say just as an aside, Robin is a beast and his team at Toposware are some of the most impressive cryptographers and NCC engineers that we've been able to work with.
And so, yeah, shout out to Robin and Toposware.
I'm gonna second that by the way because, like, I think it was a three weeks ago or something when Robin noticed this particular issue he just noted here where blocks with thousands of transactions would slow down as you get kind of fired on the transaction
list. And it was like, it was not just noticed it but then immediately had a solution and had people trying to work on it.
Like kind of before I could even like respond to the thread and I really appreciate that you know that that level of responsibility and like just kind of brilliance on being able to make this happen.
Yeah, we would not be here without the Toposware team I just want to make that extremely clear to everyone listening.
So, maybe we could look ahead, even a little bit further, and so, Daniel, most people know you as one of the key people responsible for plonky two.
But it turns out that plonky three is right around the corner. And I'm personally extremely excited for plonky three.
But maybe you can tell everyone on the spaces why they should also be excited and optimistic about plonky three.
Yeah, sure. So plonky three is much more, I'd say, modular and flexible than plonky two. So, it's possible to configure plonky three in such a way that it basically looks like plonky two, you can use Goldilocks you can use.
Yeah, the same Poseidon hash and all of that, but it also has a bunch of other options. It has smaller fields like the baby bear field and the 31-bit Mersenne field.
And it supports a bunch of other hashing options so we can use conventional hashes like Ketchak, for example, which speed up the proving quite a bit.
So, yeah, that's one thing that we get like, depending on the application, up to about a five or 10x performance increase just from switching to better fields and better hashes and that sort of thing.
Besides that plonky three, it also has some more clever batching. So in plonky two, in the current CKVM, there's really several Starks that are involved. There's one Stark for the CPU, one for memory, one for 256-bit math, and one for Ketchak and so forth.
And we basically have a separate proof for each Stark and then we have some ways to tie them together.
But there's quite a bit of overhead for Stark, so we can't add too many of them. We can't have like a Stark that just does addition and then another one for multiplication that would get out of hand.
But with plonky three, we have some more clever batching. So that sort of thing will suddenly become much more practical where we can have much more of these small single purpose coprocessors for doing something simple like addition.
And yeah, that just has a lot of advantages. We can accelerate more and more operations, things that we do right now in kind of a dumb way in software, which includes a lot of MPT logic.
We can start accelerating all those little things.
Yeah, that's awesome. It's really cool because I think that we're taking the right approach with plonky three. As Daniel said, it's not like plonky two, which is sort of an opinionated single proving system.
Plonky three is sort of a dev kit where we can experiment with different fields and different IOPs.
And it's a really flexible approach to sort of building proving systems.
And I think that it's going to lead to really, really big improvements for the ZKBM.
We say all this, but the cost, the average cost per transaction that we're seeing on real Ethereum blocks is like between two and three tenths of a cent.
And I guess, spoiler alert, we expect that to increase or to decrease by 35 percent in the next two weeks.
We just have to rerun the benchmarks. And so that will get us to between a tenth and two tenths of a cent.
And I think that there are very few crypto applications that are currently running on chains where users would even feel proving costs of one to two tenths of a cent.
Certainly they wouldn't on L1 or any L2s that currently exist.
And so it's really exciting that the tech, I think, is ready today from a cost and efficiency perspective.
And we assume that there will be a 50x reduction in cost over the next year.
And so I think that's like a really exciting position to be in.
I think one thing that's worth noting is a lot of the comparisons we picked, like RiskZero, I believe, is the only other type one that has released benchmarks and comparisons.
And there's obviously a big performance improvement of the Polygon type one versus RiskZero.
But it's not really like an apples to apples comparison, because I think RiskZero does something that's really, really cool, which is that it allows you to take an existing Ethereum client.
And rather than build a custom ZKVM in circuits and sort of in kind of our ZK domain, we can just take an existing Ethereum client and compile it to run on a ZKVM.
And so I just want to be clear that we're comparing versus RiskZero because those are the comparisons that are out there.
And we want to show how efficient the type one prover is.
But I think long term, we also plan on leveraging similar approaches to kind of reducing some of the dev time and investment on actually building the ZKVM.
And I know, Daniel, you've been working on Velido, which is a ZKVM that's similar to RiskZero.
And so maybe you can sort of give us like the three year vision for what ZKVVMs will look like kind of much further down the line in the future.
Yeah, yeah, I think I think it's a really cool direction, the direction of taking real EVMs like say Revm and compiling it to a ZKVM such as RiskZero or Velido.
And yeah, I mean, there's a few advantages, right?
One is that it's a lot quicker to develop, so RiskZero developed Zeth very quickly this way.
There's less ongoing maintenance, but also there's kind of a security argument for it where if there's less ZK code for EVM logic, then there's less code to audit and less potential for bugs.
So I think those are all really appealing.
And yeah, I think the long term for us probably involves that in some way where we can.
And also, I would just point out like right now, that approach isn't very performant.
But I think we can kind of meld these two approaches where we can take each piece of EVM logic and we can decide, is this important enough that we want to build a custom circuit for it?
Or is it not really important at all?
And we can just leave it to the compiler and it doesn't really matter if it's optimized or not.
I think there's sort of an 80-20 principle going on where 80 percent of the logic in an EVM is not really performance sensitive.
So it makes sense to leave that to the compiler and for us to focus on optimizing other specific bottlenecks.
Mostly, I just wanted to give a shout out to RiskZero because I think that they're doing really impressive work and really pushing the space forward.
But I do think that it's really cool that right now, like today, this moment, we can have a super performant type one prover.
And for the next few years, I think that we're going to be able to further reduce costs and improve performance.
So I guess we can probably start wrapping up, but I'll ask the other speakers to maybe give the thing that you've enjoyed about working on the type one and also what you're excited about.
And so, Paul, as you always have the most enthusiasm, the best memes, we can start off with you.
You know, the thing that's exciting to me about this is like I spent a lot of my career working on applications where basically my job has been to take one form of JSON, do something to it, and then spit out another form of JSON.
And then I spent a lot of my kind of crypto career working on just dealing with working with ETH nodes and like working around just kind of the crap that is solidity development and DAP development.
And this project is one where it's real software engineering down to the bones.
It reminds me of my first job working on an actual database where it's performance sensitive, it matters, there's things to debug that are kind of fun and aren't just some version of, oh, I didn't handle something quite properly in my JSON decoding.
And what that means is that I get to talk to this group of people, you know, the Daniels and Robbins, Johns every day, and we're solving real feeling problems.
And there's just not that many places in the software engineering world where that's what you're doing on the daily basis, most of the time you're kind of doing more of the other stuff.
So I'm really appreciating the opportunity to get in on that level and to, you know, flex those muscles of the stuff that I kind of grew up coding on more systems programming kind of stuff.
So that's been really fun.
I guess a big shout out to everyone who's currently toiling in the ETL trenches. There is hope. Come work on ZK stuff, work with us. We would love to have you.
John, anything that you're excited about or have enjoyed working on this as part of this?
Yeah, I mean, I loved it. I think there's two parts. I think in general, I really love kind of optimization and trying to take something and make it as fast as possible.
And it's like a true joy to work on a proving system because, you know, it's not like I can just Google like how to optimize Prover on Linux and get anything.
It's really fun to work on like kind of a novel piece of software that like will stress out a computer has real world kind of potential and really be put in a position where it's like we're, you know, really needing to understand the depths of like how the processor is going to be scheduled by the operating system.
How much memory bandwidth can we get? How can we tweak it? How can we optimize it?
And it feels like as a team, as an organization, we're like learning a lot. We're getting a lot of expertise, and I'm really excited to see how it gets leveraged over the next like year or two to make it even faster and faster.
And I think the other part, you know, maybe it's like a little bit silly, but I really enjoyed coming up with really bizarre transactions to try to cause Robin to have a very bad day.
So it's like, as part of the process of testing this, you know, we have some transactions that use every single op code or that do tons and tons of, you know, catch acts in one transaction and stuff like that.
And so the process of fiddling around with, you know, like kind of EVM op codes in order to try to, you know, stress out the ZK team, that's been a lot of fun as well.
Yeah, for those of you that don't know, John is a very sick and twisted mind when it comes to coming up with pathological EVM transactions to make the prover fail.
And so, Robin, maybe that's what you've been enjoying the most is dealing with John's transactions.
Yeah, I was going to say that John was basically saying the same things as me.
I was really like working on optimizing crypto primitives and crypto systems.
So it was, of course, super exciting to work over the past year into like making it from highly viable to purely viable with the numbers that we have now for like transaction proof and block proof costs.
And yeah, also, I mean, I like debugging stuff, like finding the tiny thing that shouldn't be here.
And so like typically when John is giving me like an absolutely ugly transaction and my team and I are going to work like over a week to find this super tiny nasty bug, then of course, this part of what makes my day so bright.
Your dry French sense of humor makes me unable to distinguish whether you're being serious or joking, but we'll leave it to the listeners.
I'm still working on the ZKVM, so I think it wasn't funny stuff as in.
And Daniel, what have you enjoyed or what are you kind of looking forward to and excited about in terms of ZK ZK EPMs?
Yeah, I'd say partly I enjoyed working with Tapaswear a lot because they've been amazing just just to echo that.
And but also it was really fun for me to see the work of John and others on actually productionizing this.
And that was interesting because we had some thoughts about how to do it and they turned out to be not really accurate in the end.
Like, for example, we envisioned that this would be run with one prover process per machine.
And then we kind of realized in the end that it was based on the data collected by John and other teammates.
But that wasn't really the right way to do it, that we can get much more efficient resource utilization by running several processes.
And then maybe one process is doing a CPU intensive work while the other one is doing memory intensive work or that sort of thing.
So, yeah, that was very insightful.
Awesome. Awesome. Very cool.
And for my part, I think it's just been a privilege to work alongside so many smart people.
I think sometimes it's easy to sort of be pessimistic and think that technological developments are far in the future and are not practical now.
And it's I think it's always really amazing to be surprised and to see huge technological achievements happen in a very short period of time and to kind of be on this like parabolic curve
of ZK tech development where everything is getting faster and it's accelerating.
And I think that's a really cool space to be in for anyone who's working in tech.
And so, yeah, I think I'm not sure, Smokey, if we're doing questions or not.
Certainly, if anyone has questions, we can we can take those.
But but, yeah, otherwise, thanks, everyone, for for tuning in and listening.
Yeah, I think that's good. We don't have anybody coming up and requesting.
So, yeah, I think it was a super informative space.
I appreciate all of the speakers who came up and contributed.
And thank you to all the listeners that came out.
All right, everyone. We will see you next time.