Okay, we're back today with another layer zero lounge. Today we have on board some of our esteemed auditors.
Just talking about, you know, auditing practices and answering some questions about how they distinguish themselves from other auditing firms.
So why don't we get started with a quick round of introductions. Today we have Enes from Chain Security and Justin from Zellec.
Justin, if you're on, would love to start with you. Just like your role at Zellec and kind of what you do there.
Hi, actually, this is Arian.
Yeah, so I got into security a decade ago and started off with the background in bug bounty and competitive factor like CTF competitions.
That's web security and binary exploitation. And then got into DeFi through like a
Slauna arbitrage bot and then joined Zellec. And now I am a senior auditor at Zellec. So I've audited
layer zero, like a few different projects of layer zero.
The polyhedra, layer zero oracle, endpoint version two and endpoint version one. Well, the version one core. The ultralight node and prooflib
examples are happening. It's just like a bunch of layer zero projects.
Gotcha, awesome. And thanks, Arian. Really appreciate you giving some background.
Enes from Chain Security, are you on as well?
Perfect. Would love a quick introduction from yourself and just a little bit of your background as well.
Yeah, sure. Hi, everyone. Thanks for having me.
And yeah, so I am Enes. I work at Chain Security.
I am a security engineer and on my daily business, I focus mostly on the auditing DeFi projects and work to improve their overall security.
And this is very exciting. At Chain Security, we get to work with some of the top development teams in the space which build amazing products for the community.
Before joining Chain Security, I was in academia and my research interests during my PhD and postdoc
were in the security and privacy focus on user authentication and the confidentiality and
integrity of input, outputs, interfaces used by users.
Awesome. Excited to have you here today. And then from layer zero, we have Irene. Irene, do you
want to just give a little bit of your role at layer zero and your background here?
Yeah, for sure. My name is Irene. I'm Head of Strategy at Layer Zero Labs. I've been with the protocol team for about two years and I'm particularly excited about today's spaces because
auditing firms are such an essential part of our development process. And Chain Security,
I'll explain later, is a new firm that I for a long time wanted to work with and then Zellec is
one of our closest partners. We love the Zellec team. Thanks.
Yeah, auditing is super important here in the space and I'm glad to have you guys here on board. Mark
from our team actually yesterday just posted a poll asking, you know, what percentage DeFi users
understand the risks associated with interacting with different smart contracts and the overwhelming
majority was under 50%. I'm sure if there was like a bucket lower than that, the overwhelming
majority would have been in that bucket. But yeah, you know, you guys are playing a super important
role not only in navigating DeFi but also making protocols and just projects that we interact with
on a daily basis much more usable. So with that being said, why don't we just start with like some
basic company introductions now that we know about your role there. Can you tell us a little bit more
about Zellec, Aaron, please? And just like what it does? Yeah, we work with leading L1s, L2s,
bridges, and DeFi protocols. We have a very diverse skill set on our team because we all
have backgrounds in competitive hacking like this capture the flag CTF. Actually, most of our team
comes from the number one competitive hacking team in the world for three of the last four years. So
we have like very diverse skill set and very skilled people on our team. We have people
who specialize in ZK circuits, EVM, Cosmos, Optos, SWE, Solana, and basically like any
technology, we probably have someone for that. And recently, we're a founding member of SEAL
that security lines led by Sam Cveson. They have a few initiatives like the incident response SEAL 911.
Awesome. Yeah, sounds like you know, your company has a pretty diverse range of, you know, projects
that have been audited before. So do you mind just giving our audience some examples of some probably
more notable projects that you guys have audited? Sure. Yeah, well, we've audited it audited layer
zero, quite a few different layer zero projects. And then Cosmos, we did the core Cosmos SDK,
osmosis and bear chain. And then for ZK, we've done like scroll axiom, nocturne, polyhedra,
their LZ oracle included starkware and well, yeah, we have like over 100 clients. So quite a few
different. Yeah, it sounds like you guys do everything from like L1, L2, ZK to Interop. So,
you know, excited to have you chatting about layer zero today. Let's just shift over to chain
security. And as would you mind just telling us more about chain security and what your company
does? Yeah, sure. So basically, chain security is one of the first auditing firms in the market.
It started like in 2017 as a spinoff from ETH Zurich, which is a leading university in computer
science and has a very strong research group in cybersecurity. Back then, a team of researchers,
they developed the first static analyzers, which is known as Securify. And this was supported by
the Ethereum Foundation to be public and to be open source. So it was publicly available. And
soon after this, the team was approached by many companies which asked for the security services
for the smart contracts. And in this way, the chain security was founded in Zurich, Switzerland.
So we also have a history of responsible vulnerability disclosures, including the issues
identified in the Ethereum hard forks, the most notable being the hard fork of
which changed the gas prices and then the new re-entrancy issue would be enabled. And due to
these findings, the hard fork had to be delayed and the issue had to be mitigated before rolled out.
And yeah, our portfolio includes a large number of security audits, hundreds of them. And there
are many blue chip protocols handling billions of assets. And also recently, we have, we want the
Ethereum Foundation's underhanded solidity contest with a cool bug, which is due to an undefined
behavior of the compiler on the order of the execution of parameters when an event is emitted.
And also we look for new attack vectors. And recently we have also discovered the read-only
re-entrancy. Yeah. And this was disclosed carefully such that because more than 100
million US dollars were at risk at that point. Got it. That makes a sense. So it seems like
between the two Zelleck and chain security, there's a wide array of Web3 projects that are
kind of covered around that scope. So with that being said, we can go back to you,
Aaron. So how does an already firm distinguish itself from others?
Well, I would say security firms are like, they're pretty similar to law firms,
we're like consulting firms in that the past clients reflect their reputation. So
of course, like we have a bunch of pretty big projects and none of them have been hacked,
like at least the in-scope code we've audited been hacked. And that's very important to us.
But there's like reputation. And actually, we're like often a partner of choice after
an act, like we'll reach out to us. And then also, I would say something that makes us
pretty unique is that the people on my team are like our background in the competitive backing.
Like we all have background in traditional web security or like binary exploitation.
Some of us have done some pretty complex kernel or heap exploits and cryptography. And yeah,
we have a pretty large ZK security team too. But I think that's pretty unique for auditing firms.
Got it. And then similarly, Anis would love to hear a little bit more about what distinguishes
chain security from other auditing firms. Yeah, as I mentioned before, at chain security,
we have a strong connection with academia. And also in our auditing methodologies, we have adopted
some of the well-established processing from academia to make sure that all our reports and
results adhere to the highest quality standards. So something that I would like to mention is that
for every audit we do, we follow the two methods, like we have internal peer reviews. So basically,
for every audit, the smart contracts are always reviewed by at least two security engineers in
parallel. And then they are always challenged by another engineer which serves as the warden
for the project. And also we have a method which we internally refer as the audit defense,
which is inspired by the PhD defense. Basically meaning that before we deliver some results,
we always have other engineers which are expert in the domain of the project that challenge the
engineers that we are working on the audit to make sure that the results are of the highest quality.
And in the past, this has shown to be very valuable. And yeah, so this is how the results
are of the highest quality. And always when we review some protocols, we follow a systematic
approach, which means that we go through all the layers. We try to understand how does the
protocol that is in scope of the review integrate with the other systems and other components. And
of course, then I think something which is unique in the chain security report is that we are right
in the detailed system overview, which helps also the users of the protocol, but also it makes sure
that it gives an opportunity to the clients to show that we have understood the system correctly
and we are reviewing it with the correct assumptions. Awesome. So these are really the
giga brains making DeFi and blockchain tech safer for all of us. And now that we've kind of
distinguished between those, Irene, are there particular reasons why we chose who we chose for
our V2 audit? Yeah, I'm happy to talk about this. And I'll lay the foundation too, by saying that
layer zero is an entirely immutable interoperability protocol. And this is a design principle
and tenet of ours that has guided the way we shaped and architected the protocol since V1,
since the white paper, which then led to the architecture of V1 or preceded it. And when you
are deploying immutable smart contracts, it is imperative that you take every single security
precaution. And that means for us, any piece of code we push, even if it's just a few lines,
we get multiple security audit up their eyes on them. I believe outside of L1s and L2s,
in 2023, we were the protocol that spent the most on security audits across the entire industry.
And our bug bounty is the highest across the world or all of like open source and decentralized
technology. It's $15 million on a mean buy. So for those who are, you know, white hats or just like
breaking things, please go check that out. We always want more people engaging with it. For V2,
we were building in stealth for a really long time. V2 incorporated a lot of meaningful feedback
from developers and projects we engaged with over the last year. And because of that, we wanted to
tap auditors who we've worked with before, who we trust, who are very dynamic. So I'll speak to
Zelleck first. We have worked with Zelleck on nearly every single major project that we have
launched. And as you heard from Erin, a lot of the members of their team have seen different
pieces of code that have touched our protocol. I love that the Zelleck team is very, very dynamic,
meaning rather than giving them some code and them coming back two or three weeks later with the
report, there is conversation, debate and questions back and forth between them and our team
nearly every single day. They also are definitely the kind of teams that work for the weekends and
are working at conferences, are working everywhere they go. I have a lot of respect for the leadership
team there as well. And I mean, you guys hit the map not too long ago and since then have
become the most sought after team for some of these new all ones to work with like Sui and
Apton. So I'm super proud that Layer Zero has been an early supporter of you guys. I think maybe one
of your first crypto clients. And then on the chain security side, I met Emily not too long ago.
Emily is one of the like members of leadership, I think listening in and was incredibly impressed
with how involved they are from a relationship and just partnership standpoint with all the
projects they work with. They're not like a one touch services type team. They really care
about the overall decentralization and success and user experience of the protocols that they
work with and the maker chain who they've worked with for a long time have referred chain security
to me. And so I wanted to find an opportunity for us to work together. The academic approach is one
that we have not tapped in a serious way for our sort of like three step security audit process.
We get multiple firms to look at it and ETH Zurich is an incredible university and like
Enes said has been involved in the earliest lore and days in the Ethereum Foundation. So I'm very,
very thrilled that we got to bring chain security on in this way. And we'll be curious to hear about
their findings looking at OFT v2. Awesome. Yeah, you know, security here is paramount at Layer Zero.
And so it's not only factors like our bug bounty, but also engaging auditors such as Zelleck and
such as chain security that really make our protocol useful and safer for everyone. So with
that being said, why don't we tap a little bit into the v2 audit? And so I think the first question
we can ask here is, it sounds like traditionally, you know, you would audit a project on one chain,
like a DeFi DAP, like a lending protocol or an AMM. But for Layer Zero, there's the challenge of
going cross chain where, you know, you're interacting with a bunch of different chains,
each with their kind of own unique consensus mechanism and validator set. So, you know,
why don't we start with you again, Aaron, are there any differences between like a same chain
audit versus like a cross chain audit? By cross chain audit, I guess you could mean like
the bridge like Layer Zero or protocols built on the bridge. But I would say that auditing process
is like the same for both of them. I mean, the threat model is different. The threat model is
different for every protocol, right? But the auditor always has to consider, I would say like
three main categories of bugs, like there are three main things I look at, like centralization risks,
like can it be exploited with phishing or bad key management, or weak multisig? That's like,
regardless of what the protocol is or what the project is, if it's like DeFi, then that's a
concern. And then the threat model, like for business logic bugs, so that's like incorrect
assumptions or improper privileges. Once you understand the threat model, you can find those
kinds of bugs. But yeah, you have to like, you know, the project to go to find those kinds of bugs.
And then the coding mistake bugs, which are, I would say like more surface level, you don't
really have to understand the project too well to find those like, do functions, validate inputs,
and is access control like properly written? Like is it secure? Most of the bridge hacks,
like we did some research on this, and most of the bridge hacks, the TVL loss at least,
has been from centralization risks. So for bridges, specifically, we really pay attention
to that, like we really look at the centralization risk of that bridge, in case, like say,
weak multisig or bad key management, like I said before, those bugs are very important. But of
course, we don't want low hanging coding mistake bugs. So we look for that too, like all three.
Gotcha. So it sounds like you're adhering to the same set of principles or like the same,
you know, criteria when auditing same chain, risk, cross chain. And from your perspective,
do you think there are any additional, you know, challenges of going cross chain in terms of an
audit? Yeah, yeah. So I agree. Like, I think in the both cases, the process is quite similar. But
for the cross chain audits, usually, it's a bit more complex, right? Because then there are
different L2s and robots, which might have different semantics, and usually they are the same
opcodes, but actually, which have a different meaning in the layer two. And they also their
behavior might change depending on the layer two. And also, then it depends from the project,
if the project is making use of the block numbers and timestamps, for example, this might behave
differently in the layer two than they behave in the Ethereum mainnet. And another difference is
also the gas costs, right? Because like in the Ethereum mainnet, the code is structured in such
a way that it saves like in the storage usage, and execution, and the call data is cheap. But
actually, in the layer two, it's a bit reversed because the call data is more expensive and then
the storage and executions, they are cheaper. And yeah, as mentioned, like also the trans assumptions
are very important when outing cross chain projects, because then there are usually there
are layers and other entities which are usually like, it depends if they are centralized or
decentralized, which do let message passing from one chain to the other. And of course, when you
audit some cross chain project, you have to think also what might happen in that destination chain,
because it might be that you send a message from the source chain, or for example, you transfer
some tokens or you burn some token in the source chain, but then you have to have all the guarantees
that the other leg of the transaction executes actually on the destination chain. And so the
trust model here has to be like very carefully evaluated. Gotcha. So slight shifts in the trust
model to account for differences in like gas on one chain versus the other. And just some of the
other differences presented by the different chains, right? Exactly. Got it. Okay. And so now
tapping into the v2 audit specifically, you know, what is the process for auditing something like
layer zero v2, where we've now shifted from a configurable Oracle real layer model to now,
you know, a permissionless network of DBNs and executors. So, you know, where do you start with
that? And how do you test some of the things in the audit? We can start with entities this time,
actually, we'll reverse the work a little bit. Yeah, so usually we start by going through the
white paper specifications and the documentation, everything that is provided to us. This helps us
to create a mental model of how the smart contracts work and how they should behave, interact with
each other. And in this audit, since we were focused on the OFT and the O app, and these two
are like the are mostly as building blocks, right? So other developers, third party developers will
use those to extend and wrap their ERC20 tokens. It's very important to think like how these
building blocks will be used. And for that reason, we paid a lot of attention into
all the special behaviors of the tokens and the developers should be aware when they use the OFTs
and the OFT adopters. And also like in the report, we put a lot of efforts noting all these
special behaviors that developers should be aware to make sure that whenever they integrate their
ERC20 tokens with the OFTs, they are integrated in a secure way. And there are no bugs introduced
when the OFTs are extended.
Got it. That makes sense. Yeah, OFTs are layer zero as a protocol is more developing,
developer facing than user facing. And I think things like OFTs certainly speak to that where
we've on aborted developers across DeFi, gaming, like a ton of other verticals that have integrated
OFT. Aaron, I'm wondering from your perspective, does that also make anything different that layer
zero is more developer facing than user facing in terms of an audit?
Well, we tried to make sure that centralization risks are mitigatable. I remember in version 1,
there was some trauma about, because some people misunderstood the threat model of
players who are version 1. So at this time, we were making sure it's really solid. We were
documenting all the centralization rates to make sure people understand them and understand that
they're mitigatable. It's very configurable, right? You can change the config and then
like the DVNs, like you can specify how you want your security to be.
Got it. And then it sounds like from NS, they had a particular focus in OFTs during the audit was
like, for Zelik, what was the scope of the audit? Was there like a particular part of the protocol
that you guys had more of an emphasis in auditing? At least for the audits that I did,
I was on the endpoint version 2 audit and then the DVN base code. And it also audited the polyhedra
DVN and the verifier network, like the multisig and then also like the message libraries.
That was the focus during the audits I was on at least.
Okay. That makes sense. So between the two, it seems like there's emphasis. And between all of
our auditors, every part of the protocol should be very thoroughly audited. And so with that being
said, now that the audit's complete, what are your thoughts on the protocol and what were some
interesting things that you found out about layer 0 during the audit? We can start with you again,
Aaron. My thoughts on the protocol? Well, I would say it's definitely more mature than version 1.
The security is configurable. The defaults are sane and it's easy to understand and configure.
It's very well documented, by the way. And I like that it supports future upgradeability,
like the messaging libraries, but it also mitigates the centralization risks because people can pin
the library version they want. And layer 0 can't remove the libraries. They can only add. So
there's you're like mitigating the denial of service risk there.
All of the risk is mitigatable, which I like. And then let's see, an interesting finding.
The verify network multisig. We found a bug where the signatures weren't pinned to the chain. So
you could submit. It was for a DBN. So you could submit the signature to multiple chains,
and the message might be able to be replayed on multiple chains.
Because the chain ID wasn't in the object that was signed.
Got it. So it sounds like our principles of being immutable and non-upgradable contracts are
something that puts it out to you during the audit. And it's for you at chain security,
are there any interesting findings? Did you learn more about the protocol in any ways?
Yeah. So from auditing the OFT and the OAP, we saw that the code quality was of high standards.
And yeah, we were very excited to work with the layer 0 because also especially the OFTs,
the only chain fungible tokens, they offer a lot of value in the ecosystem. And
already like many huge players, we started using them. And as any information, so basically also
you are auditing the curve XDAO, which is built on top of the layer 0, which is very cool. So we
chose that also the community is adopting this protocol and using it. Yeah. And regarding the
interesting findings, I think it was quite interesting. One compatibility issue with
tokens that exhibit special behaviors. And one example is the compound USD-CV version 3.
So basically this ERC-20 token has a special behavior in the transfer function. Whenever you
pass the UINT-MAX amount as the transfer amount, actually it will transfer the user's balance,
not the amount that you pass to the function. And this would be if like some developers would
use the OFTs for the compound USD-CV version 3, then actually there would be a risk of exploit
because then what one could mean, the minimal could burn a minimal amount in the source chain,
but actually meant an infinite amount of token in the destination chain.
If I can chime in really quick, the specific OFT integration that MS is referring to is
USD-V, which is a stable coin, a pretty novel one that was launched by the verified USD foundation
incubated out of Matrixport, which is a large digital asset firm based in Singapore and Hong
Kong. And they were the first project to use an algorithm we created called ColorTrace,
which enables on-chain attribution and across multiple chains as well. So chain securities
eyes on that were incredibly important. And that's one of many stable coins, but then also
other kinds of unique on-chain tokens that have integrated OFT. So I know for certain,
the smart contracts engineer on our team who's working with you guys was really
impressed and appreciative of these findings.
Yeah, thanks for chiming in there, Irene. It sounds like from both what Erin and Enes have said,
it sounds like this is like a mutual learning experience where both the auditors and the core
team behind the protocol get to learn more about the architecture behind the scenes and have these
findings published together. And so just wrapping things up for both Erin and Enes, what audits do
you have coming up for your team? And if teams are in touch and interested in having an audit,
what's the best way to get in touch with your team?
I'm not sure I can say specific on it, but we've seen rapid growth in the Cosmos app chains and ZK
circuits for privacy protocols, lots of off-chain, non-smart contract engagements like infrastructure
and some web2 apps relating to web3. And then I can learn ABSs, all kinds of protocols. It's very
exciting, lots of new things for me to learn too. And if you get in touch with us, we have a contact
form at zelik.io slash contact. It's zelik.io slash contact. Got it. And then Enes from your
site too. And if you don't want to share specific names of projects, please feel free to just
mention a more broad category of where they fall upon. But a similar question to you. So what
audits you have coming up across what verticals and how can teams wanting an audit get in touch?
Yeah, sure. So basically one of the reports that soon is going to be published and we have the
permission to say it is CurveXDAO, which builds on top of layer zero, which is quite interesting.
And also now we are focusing a lot more on non-EVM blockchains as well. So we are working with Tron,
StarkNet and other layer tools. And we publish all our reports in our website, which is also a good
opportunity to reach us, which is the website is chainsecurity.com. And also we are reachable
in Twitter and Telegram. We have a fantastic team, which always is responsive and
replies quickly or any question that you might have.
Awesome. I wanted to thank you both so much, Erin and Enes, for coming on the space today.
Hopefully for our audience, they got to tap into the mind of what it means to be an auditor today.
And again, just want to thank you both for your contributions to layer zero v2 and really getting
the protocol up to where it is today. Yeah, of course. I'm looking forward to future audits with
you. It's always a lot of fun for me. Awesome. Yep. Thank you so much for having
you both. It was a great space. Thanks a lot to you. Thanks, guys.