Comparing languages for writing zero-knowledge proofs

An automated smart contract deployment pipeline

Solidity underhanded decentralized exchange

Storage-free smart contract upgrade pattern

A look into formal verification of smart contracts using Certora

## TL;DR

With Typescript, we can define the permissions granted to our lambda functions in our serverless apps using types. This lets us **type-check access to resources**, raising at compile-time any errors caused by missing permissions.

```typescript
// We define our app permissions as objects
const ProductsAccess = { Products: true } as const;
const MailerAccess = { Mailer: true } as const;

// And we restrict access to any resource by requiring those permissions
getProductStore = (_policy: typeof ProductsAccess) => new DynamoProductStore();
getMailService = (_policy: typeof MailerAccess) => new SESMailer();

// Then we define the type of policies required by our lambda as an argument
async function lambdaHandler(event: Event, policies: typeof ProductsAccess): Promise<Result> {
 // And use those policies when requesting access to a resource
 // so the compiler catches if we try to access a resource for which we don't have the required policy
 const store = getProductStore(policies);

 // Next line fails to compile since our policies do not include acess to the mail service!
 const mailer = getMailService(policies);
}
```

And since we have defined our policies in code as part of each lambda handler, we can declare all parameters for each lambda in its handler file and **use CDK to go through them and create the associated resources**.

```typescript
// src/api/get-product.ts
// We attach the lambda parameters to the handler function
export const handler = makeHandler(lambdaHandler, {
 policies: ReadOnlyProducts,
 httpMethod: 'GET',
 logicalName: 'GetProductFunction',
 resourcePath: 'products/{id}',
 entry: __filename,
});
```

```typescript
// bin/stack.ts
// And we iterate through all handlers and create the associated lambda
import { MyStack } from '../lib/my-stack';
import * as handlers from '../src/api';

const app = new cdk.App();
const stack = new MyStack(app, 'MyStack', { ... });

for (const handler of Object.values(handlers)) {
 stack.makeFunction(handler.params);
}
```

```typescript
// lib/stack.ts
// We build into the stack the logic to add each new lambda
export class MyStack extends Stack {
 public makeFunction(params: HandlerParams) {
 const fn = new NodejsFunction(this, params.logicalName, {
 entry: path.relative(path.resolve(__dirname, '..'), params.entry),
 ...this.getFunctionSettings(),
 });

 this.grantRights(fn, params.policies);
 this.api.root.resourceForPath(params.resourcePath)
 .addMethod(params.httpMethod, new LambdaIntegration(fn));
 }
}
```

This lets us define each Lambda in our application, along with its required permissions, in the same file where we declare the handler function, and **leverage the type checker to catch any missing permissions**.

So if you find this interesting, read on for the longer version, or check out [this fork](https://github.com/spalladino/serverless-typescript-demo/) of the AWS `serverless-typescript-demo` for a full code sample.

## The problem with permissions errors

A common error when building serverless applications is missing a permission for accessing a resource in a lambda function. These errors are particularly annoying because they show up late in the development lifecycle, since unit or local tests do not uncover permissions errors: you depend on integration tests to check that your lambdas have all the rights needed to run properly. An error here requires a full redeploy of the stack and a retest, which takes valuable developer time.

Moving these errors to earlier in the development cycle yields a faster feedback loop. In this article, we'll explore how to catch these errors at compile time leveraging the Typescript type checker, and take it one step further by integrating our new typed permissions directly into our CDK stack definition.

## Typing permissions

Let's start by defining our application permissions as Typescript types. For the sake of the example, let's say we have a `Products` DynamoDB table in our application. We'll define two permissions, `ReadProducts` and `WriteProducts`, and use them to define a read-only and a full-access policy.

```typescript
const ReadOnlyProducts = { ReadProducts: true } as const;
const ReadWriteProducts = { ...ReadOnlyProducts, WriteProducts: true } as const;
```

Note the `as const` in the declarations. This ensures that the type of each policy requires a `true` value for each property, and not any boolean.

We could have defined our policies as just an enumeration of permissions instead of using an object as we did above. But using objects like this plays nicely with the type system: a policy that extends another with additional permissions will also extend it in the eyes of the compiler.

```typescript
function listProducts(policy: typeof ReadOnlyProducts) {
 // ...
};

requiresReadProducts(ReadOnlyProducts); // Works
requiresReadProducts(ReadWriteProducts); // Works, since RW extends RO!
```

You can define whatever arbitrary permissions you need for your application. These can go about sending emails, managing users, or pushing notifications. And you can combine them freely into policies. We'll worry about converting them into actual IAM policies later.

```typescript
// Example of a policy needed for running a UserManager
const UserManagerPolicy = {
 ReadUsers: true,
 WriteUsers: true,
 SendEmails: true,
 PushNotifications: true,
} as const;
```

## Checking permissions when accessing resources

Once we have our sets of application permissions defined, we can now enforce them to request access to resources in our code.

Going back to the example from the previous section, we can abstract access to the `Products` table behind a `ProductStore`, with a DynamoDB-based default implementation.

```typescript
export interface ProductStore {
 getProduct: (id: string) => Promise<Product | undefined>;
 getProducts: () => Promise<Product[]>;
 putProduct: (product: Product) => Promise<void>;
 deleteProduct: (id: string) => Promise<void>;
}
```

We can also restrict the interface above to getter methods, which we will use if the client has read-only permissions:

```typescript
export type ReadOnlyProductStore = Pick<ProductStore, 'getProduct' | 'getProducts'>;
```

Now is where things get interesting: we can define a function that returns a read-only or full `ProductStore` depending on the **type** of the policies supplied by the caller:

```typescript
export function getProductStore
 <Policy extends typeof ReadOnlyProducts>(_policy: Policy): 
 Policy extends typeof ReadWriteProducts 
 ? ProductStore 
 : ReadOnlyProductStore {
 return new DynamoProductStore();
}
```

Note that we don't even use the policy parameter in the body of the function, since actual permissions will be checked by the AWS policies we define. What we are doing here is asking the type system to check if the user has read-only or read-write permissions on the `Products` table, and return the corresponding interface. On runtime, we just return the same implementation.

This achieves our goal of restricting access to resources based on the **type** of the policies we're working with, which raises any permission errors during compile-time.

```typescript
// Works
getProductStore(ReadWriteProducts).putProduct(product);

// Property 'putProduct' does not exist on type 'ReadOnlyProductStore'
getProductStore(ReadOnlyProducts).putProduct(product);

// Argument of type '{ readonly ReadUsers: true; }' is not assignable to parameter of type '{ readonly ReadProducts: true; }'
getProductStore(ReadWriteUsers).putProduct(product);
```

## Declaring permissions for each lambda

The most straightforward way to use this pattern is to declare, for each lambda, which set of permissions we will be granting it. This serves as a documentation attached to each function, which we can use to easily determine which policies we need to define in our infra-as-code.

```typescript
async function lambdaHandler(event: APIGatewayProxyEvent) {
 // We declare the policies for this lambda once
 const policies = { ...ReadOnlyProducts, ...UserManagerPolicy };
 
 // And use them whenever we need to instantiate a resource
 const service = getService(policies);
}
```

But since we are already defining these policies in code, we can take this an extra step and use them to actually generate the policies in our infra-as-code template. As a matter of fact, we can extend this to generate the entire function template from a single source of truth.

## Integrating with CDK

Our goal will be to write the policies for each lambda in a single place, and 1) have the type checker verify that we have the required policies for accessing all resources we need and 2) generate these policies in our infra-as-code template. Since we're already working with Typescript, we'll use CDK as our infra-as-code tool.

To do this, we'll attach the policies to the exported handler function, so we can query them when building our stack from CDK.

```typescript
// src/api/get-product.ts
const policies = { ...ReadOnlyProducts };
const handler = (event: APIGatewayProxyEvent) => lambdaHandler(event, policies);
handler.params = { policies };

export { handler };
```

So when we define our stack using CDK constructs, we can import each handler and define its IAM permissions from the ones we exported.

```typescript
// lib/stack.ts
import { handler as getProduct } from '../src/api/get-product';

// We define our resources in the stack constructor
const productsTable = new Table(...);
const api = new RestApi(...);

const getProductFunction = new NodejsFunction(
 this,
 "GetProductsFunction",
 { entry: "./src/api/get-products.ts", ...settings },
);

api.root.resourceForPath('products/{id}')
 .addMethod('GET', new LambdaIntegration(getProductFunction));

// A helper function translates from our custom policies
// to the actual IAM permissions to be created
const grantRights(lambda, policies) => {
 if (policies.ReadProducts) productsTable.grantReadData(lambda);
 if (policies.WriteProducts) productsTable.grantWriteData(lambda);
};

// And we repeat for each lambda in the app
grantRights(getProductFunction, getProduct.params.policies);
```

But now we have split the definition for our lambda in two different places: policies are in the handler source file, and the rest of the settings are in the stack file. 

What if consolidated them into a single source of truth in the handler?

## Taking CDK integration one step further

Let's take the approach from the previous section one step further and inject all information relevant to the lambda when declaring its handler function:

```typescript
// src/api/get-product.ts
const policies = { ...ReadOnlyProducts };
const handler = (event: APIGatewayProxyEvent) => lambdaHandler(event, policies);
handler.params = { 
 policies,
 httpMethod: 'GET',
 logicalName: 'GetProductFunction',
 resourcePath: 'products/{id}',
 entry: __filename,
};

export { handler };
```

Now we can add a function to our stack that accepts this set of parameters and creates a new lambda, connects it to the API gateway, and grants it the set of permissions defined in the policies:

```typescript
// lib/stack.ts
class MyStack extends Stack {
 public makeFunction(params: HandlerParams) {
 const fn = new NodejsFunction(this, params.logicalName, {
 entry: path.relative(path.resolve(__dirname, '..'), params.entry),
 ...this.getFunctionSettings(),
 });

 this.grantRights(fn, params.policies);
 this.api.root.resourceForPath(params.resourcePath)
 .addMethod(params.httpMethod, new LambdaIntegration(fn));
 }
}
```

And we just invoke it for each handler in our application when constructing the stack:

```typescript
// bin/stack.ts
import * as handlers from '../src/api';

const app = new cdk.App();
const stack = new MyStack(app, 'MyStack', { ... });

for (const handler of Object.values(handlers)) {
 stack.makeFunction(handler.params);
}
```

This way, we have the definitions of the main resources of the application in our `lib/stack.ts` file, while each function is declared in its own individual file, along with its associated permissions - which get type checked when requesting access to any resource!

## Full example

You can check a full code sample of the pattern above in [`spalladino/serverless-typescript-demo`](https://github.com/spalladino/serverless-typescript-demo/), which is a fork of the [`aws-samples/serverless-typescript-demo`](https://github.com/aws-samples/serverless-typescript-demo) that showcases a products CRUD built using Typescript and CDK.

Type-checking Lambda permissions with Typescript



Security in web3 development has improved remarkably over the past years. Even though we still see frequent hacks due to coding errors, development security practices have evolved significantly across web3 teams. Full test coverage, static analysis tools, fuzzing, and mandatory peer reviews are a requirement across most teams, while audits and bug bounties add an extra layer of security.

However, the security in web3 operations is far from this level of maturity. Hardly any projects have any dedicated ops teams, and there is a general lack of awareness on best practices for operations in the web3 space. Security does not end when the code is frozen and audited: deployments, upgrades, integration testing, administration, automation, and monitoring also need to be bulletproof to avoid any hacks.

Going through the [Rekt leaderboard](https://rekt.news/leaderboard/) is evidence enough: many of the top incidents were not caused by development mishaps but by operational oversights, and several of them could have been mitigated by proper monitoring and incident response practices. Examples include [compromised frontends](https://rekt.news/badger-rekt/), hacks that [go unnoticed for days](https://rekt.news/ronin-rekt/), [incorrect initializations](https://rekt.news/nomad-rekt/), [compromised keys](https://rekt.news/vulcan-forged-rekt/), and more. In more recent news, a [missing initialization in Arbitrum Nova](https://decrypt.co/110238/hacker-abritrum-ethereum-draining-bug-nitro) led to a 400 ETH bounty payout, while [Wintermute was hacked for $160M](https://rekt.news/wintermute-rekt-2/) due to a compromised private key.

While working at OpenZeppelin, we set out to interview multiple projects in the space to better understand their security needs. While training their Solidity developers in security was top of mind for most teams, we noted that operational concerns were often relegated:

- “We write a one-off script whenever we need to upgrade”

- “Transactions are sent from a private key stored in a file on a shared server”

- “We’d want to use a multisig or timelock eventually but tooling is lacking”

- “I don’t know what to monitor”

- “The lead developer just deploys the contracts from his working copy”

- “I open etherscan every morning to see if everything looks fine”

This research led to the development of [OpenZeppelin Defender](https://zpl.in/defender), today in widespread use across the space for securing web3 operations. Here we'll share the learnings collected along the way and the pain points we found in the space. We’ll focus on five different operational areas: Integration, Deployment, Administration, Transaction Automation, and Monitoring.

## Integration Testing & Staging Environments

The DeFi cambrian explosion from a few years back was made possible by a highly interconnected ecosystem, with a synergy only attainable by building protocols on top of others, and leveraging the primitives available to build more powerful instruments.

However, testing in such an interconnected environment is challenging: how do you test your protocol when it depends on so many live systems and moving pieces? As an example, Compound’s [recent cETH price feed incident](https://www.comp.xyz/t/ceth-price-feed-incident-post-mortem/3578) was triggered by perfectly valid code, audited by three different companies, but which interacted with a token implementation that had not been considered.

Testnet deployments are far from representative of an actual production setup. Few protocols deploy their systems to both testnet and mainnet, and even fewer bother to deploy exactly the same versions. Testnets fragmentation doesn’t help either, though the upcoming deprecation of Rinkeby, Kovan, and Ropsten will help. Furthermore, even if the code is the same in mainnet and testnet, the state of the system differs widely due to real production usage.

[Mainnet forks](https://hardhat.org/hardhat-network/docs/guides/forking-other-networks) help a lot here, but it is still difficult to test what would happen in other possible scenarios. Even if it’s an extreme case, hardly any team has ever tested how their protocol would behave [should MakerDAO hit the emergency button on DAI](https://makerdao.world/en/learn/governance/emergency-shutdown/).

Another issue is that automated integration tests on mainnet forks typically only focus on contracts. Nowadays, a web3 system is rarely composed of just smart contracts: it integrates multiple off-chain components as well, such as oracles, bots, and indexing engines, not to mention the frontends that users depend on.

These issues call for new solutions in integration testing. First and foremost, live staging environments, with a setup as representative as possible (both on and off chain) of the actual mainnet environment, is a must for manual testing and assuring that all components work as expected. [Long-lived mainnet forks](https://docs.tenderly.co/simulations-and-forks/how-to-create-a-fork/how-to-get-a-fork-json-rpc-url-and-id) are a good way to start here.

Reliable deployment packages for popular protocols can provide the means for setting up environments with hypothetical scenarios. Imagine being able to spin up a version of Maker [going through a black thursday scenario](https://blog.makerdao.com/the-market-collapse-of-march-12-2020-how-it-impacted-makerdao/) in any network with a few commands, so you can test your system reacting to it. While this requires a massive community effort to populate the many protocols and the many scenarios that are possible, it is worth the investment. This endeavor also requires a [declarative deployment standard](https://medium.com/nomic-foundation-blog/hardhat-ignition-5a34a4e3d2de) to build on top of.

Last, blockchain determinism and public record gives us a unique opportunity to test new implementations without even having to put them in production - the opposite of the test in prod meme. As an example, in the traditional development space, Github [tests new code strategies by executing them against live production requests and comparing them with the legacy code](https://github.blog/2016-02-03-scientist/) until they are satisfied with the results. Today, we have all the building blocks needed for replaying live transactions against a new version of a system before making the upgrade, without even having to deploy the system to mainnet for it.

## Reproducible & Auditable Deployments

While most web2 systems are deployed from CI/CD pipelines, contract deployment is still very much a manual process. It usually falls on a developer to checkout the code, compile it, write a hardhat deployment script, deploy from their workstation, and then verify the source code on Etherscan or Sourcify.

This has several downsides, besides the hassle for the developer, who also needs to keep a funded private key handy to pay for the deployment gas costs. The main problem is a lack of traceability, as the process for going from the vetted source code to a live instance is opaque. This can lead to issues where the [deployed code does not match the audited version](https://medium.com/opyn/opyn-eth-put-exploit-post-mortem-1a009e3347a8#4089), inadvertently introducing a vulnerability. And, in the interest of transparency, having an audit report that refers to a git commit does not offer much information to a user who is interacting with an address on Metamask.

Lack of reproducibility is also a problem as more protocols go multi-chain. Ideally you want to keep the same version of the code running on all the chains where you operate, but to do this, you need to be able to reliably re-deploy exactly the same version. This means starting from the same version of the source code, building with the same settings and compiler version, and deploying with the same configuration.

The low-hanging fruit here is to move deployments to a transparent CI/CD pipeline, where anyone can follow the automated process from the source code to compiled bytecode, and from there to the deployed instance. Online tools can also facilitate auditability by [verifying that the code at an address matches a specific artifact](https://forum.openzeppelin.com/t/defender-release-verifying-deployment-bytecode/31531). The end goal is to have a reproducible process and auditable trail from a git commit to a deployed instance.

## Administration & Governance

With very few exceptions, most smart contract systems keep a set of administrative functions that allow a governing body to tweak how the protocol operates. These can go from adjusting a parameter to a full-blown code upgrade. Treasury management can also be considered part of the administration of a protocol. Given how critical these functions are, it is imperative they are controlled by secure and trusted entities.

The state of the art when it comes to governance has improved remarkably over the last few years. It was not uncommon to see protocols with millions of dollars managed by a single private key sitting in the Metamask wallet of the project lead developer.

Fortunately, it has now become commonplace the use of multi-signature wallets, with [Gnosis Safe](https://gnosis-safe.io/) being the most popular option. However, multisig contract implementations are only one part of the picture: good tooling for creating and reviewing transactions is needed to guarantee secure operations run through these contracts.

Tools for creating transactions need to account for not just the multisig, but also other governance and security mechanisms set up, such as timelocks or vote-based governance contracts. Governance contracts decentralize the decisions in a community, though many projects still restrict who can create new proposals, retaining that power behind a multisig. And timelocks provide two simultaneous benefits: they give the project’s team time to react if their setup was compromised, and they protect the community against rugpulls by giving users the time to exit the protocol if a malicious proposal is put forward. While a multisig transaction builder is good enough for creating transactions sent directly to the application, more complex tools are needed as these building blocks are combined.

Reviewing administrative proposals is particularly challenging from a UX perspective. Many project stakeholders are not developers, meaning they cannot manually run scripts to decode or analyze a transaction before approving it. All too often, an N/M multisig becomes 1/M, where the 1 is the lead developer who pushes a proposal and asks the remaining M-1 to sign, who blindly hit the approve button.

Simulation is the best resource we have for analyzing the outcome of a transaction, but the simulation tools that we use as developers may not cut it when it comes to less tech-savvy users. There is a need for [simulation capabilities that are accessible and easy to understand](https://forum.openzeppelin.com/t/defender-release-admin-simulations-safe-finalized-sentinel-tags-more-speed-for-polygon-txs-and-more/31426#transaction-simulation-and-execution-review-on-admin-proposals-1) for everyone, so they can verify by themselves, and not trust.

Contract upgrades deserve a separate paragraph. Approving an upgrade as a multisig signer means agreeing to a potentially massive change, while having only the 40 hex characters of the new implementation contract as context. And while having the source code verified on Etherscan is a requirement, the process of linking that code back to a verified or audited git commit is cumbersome (if not impossible) for most users. Easy-to-audit deployments, as detailed in the previous section, are a requirement for secure upgrade operations. It also helps to have a strong suite of integration tests for the new implementation that signers can review before approving.

## Transaction Automation

An automated script that submits transactions to the network needs access to a funded hot wallet. Gasless transactions is one of the most popular use cases here, and compromising the key of a meta-transaction relayer would mean draining all the funds allocated to paying gas fees. However, there are scenarios where these off-chain scripts need to manage keys with sensitive privileges, such as updating an oracle or managing a bridge. A compromised private key in this scenario can have dire consequences, as was the case with [Harmony’s bridge hack](https://rekt.news/harmony-rekt/) for over $100M.

This makes private keys more sensitive than your average application secret, so storing them in plaintext in a dot-env file in a server where half the dev team has ssh access is not a good idea. It is key (pun intended) to leverage key management solutions that can keep the key safely stored. There are managed cloud-based options, such as [GCP KMS](https://cloud.google.com/security-key-management) or [AWS KMS](https://aws.amazon.com/kms/), self-managed like [Hashi Vault](https://www.vaultproject.io/), and even [hardware security modules](https://en.wikipedia.org/wiki/Hardware_security_module) for hosting your keys.

Most of these solutions also expose cryptographic functions in their API, so you can send your payloads to be signed by the KMS and the [private key never leaves](https://docs.openzeppelin.com/defender/relay#security-considerations) the secure vault. Keeping the keys in a secure vault also lets you maintain an audit trail of all requests for using it, so you can easily detect or trace back any unauthorized usage.

Reliability is also a concern when it comes to transaction automation. In blockchain, It is not enough to send a transaction, since you have no guarantees that it will get mined. Any automation infrastructure needs to monitor any in-flight transactions to reprice and resubmit them if network congestion increases, and ensure that a low-priced transaction will not clog the sender for any subsequent ones. There are [multiple](https://docs.openzeppelin.com/defender/relay#under-the-hood) [managed](https://docs.biconomy.io/introduction/why-biconomy) [solutions](https://docs.infura.io/infura/features/itx-transactions) for tackling this problem, though you can also write your own.

## Security Monitoring

Monitoring and alerting is commonplace on any web2 application, but its adoption in the web3 space started only recently with platforms like [Defender](https://docs.openzeppelin.com/defender/sentinel), [Forta](https://forta.org/), or [Tenderly](https://tenderly.co/monitoring/). For an embarrassingly long time, the most widely adopted approach to security alerts was [samczsun](https://twitter.com/samczsun)’s trademarked “u up?”. Such was the state of monitoring that the largest crypto hack ever ($624M) [went unnoticed for almost a week](https://rekt.news/ronin-rekt/).

Reacting timely to a security event is important not just for being on top of the incident and keeping your community reassured, but also because [many hacks don’t involve a single action](https://forta.org/blog/web3-kill-chain/). Attacks usually require multiple transactions in preparation and execution, spread over time. As an example, [BadgerDAO’s hack lasted 10 hours](https://forta.org/blog/how-to-derail-a-120-million-dollar-hack/) until all user wallets were drained.

Receiving an early notice of an attack can give you the time to pause the protocol and stop the bleeding. [Circuit breakers](https://blog.openzeppelin.com/workshop-recap-service-monitoring-and-emergency-response/) are a particularly interesting building block here: scripts triggered by security monitoring alerts that rely on transaction automation for pausing the system when a threat is detected.

Last, custom monitoring scripts are also becoming a common practice, but it’s important to [monitor the monitoring](https://watchmen.fandom.com/wiki/Who_Watches_the_Watchmen%3F). A failure on an alerting script can go undetected, since it is expected to be silent, and can lead to a missed opportunity to deter a hack.

## Conclusion

As recent hacks prove, security in operations is as critical as in development. Security does not end with Solidity: the most secure smart contract is worth nothing if its onlyOwner wallet gets compromised. There is a need for a [DevSecOps](https://www.redhat.com/en/topics/devops/what-is-devsecops) approach for the web3 ecosystem, powered by platforms and tools that provide the means to implement its processes and [best practices](https://docs.openzeppelin.com/defender/advisor).

A case for secure web3 operations

The State of Smart Contract Upgrades in 2020

Nestor en blockchain

Hey there!

Talks

Being a responsible multisig signer (Verify, don't trust!)

Building the infrastructure for secure web3 operations

Hiring and onboarding for blockchain and security

How to fix bugs on deployed smart contracts

Experiments

Comparing languages for writing zero-knowledge proofs

An automated smart contract deployment pipeline

Solidity underhanded decentralized exchange

Storage-free smart contract upgrade pattern

Book

Ethereum for Web Developers

Highlighted articles

A look into formal verification of smart contracts using Certora

Type-checking Lambda permissions with Typescript

A case for secure web3 operations

The State of Smart Contract Upgrades in 2020

Nestor en blockchain

Modelling graph coloring with integer linear programming