Troubleshooting a Strategy
Inevitably, your Agent running a Strategy will get stuck somewhere encountering an unexpected situation. The blockchain is a wild environment, and unexpected series of events will occur. Despite Almanak's Strategy Framework being robust to known errors and built with a happy/sad-flow to deal with such errors happening, Almanak's Laws of Agent (like Asimov's laws of robotics) are:
Rule #1: The Agent must never do anything stupid.
Rule #2: In doubt, refer to rule #1.
This means that it is better for the Agent to halt its execution and ask for human intervention when it's not 100% certain of the current state of things and how to best move forward.
Obviously, other bugs and crashes might be responsible for a need to troubleshoot.
Tools
There are mainly 5 tools at the user's disposal:
- Logs
- Persistent State Files
- Config File
- Manual Recovery - SAFE
- Troubleshooting Agents
Let's go over them one by one with concrete examples on how to best use them.
This Troubleshooting Guide provides examples based on Uniswap V3, since that is the main protocol we'll be focusing on for the launch, but applies similarly to other protocols.
Logs
The Logs are the first place you will start your troubleshooting journey to assess what caused the problem.
When running an Agent, you will find the logs in the first tab on the bottom part of the Agent screen.
You can navigate up (older) & down (newer), display the logs full screen, search for specific text in the logs, and also filter them by Severity.
A useful way to navigate the logs is to quickly scroll or search to find the State of interest. Such logs look like:
14:48:36 ">>> INITIALIZATION (INITIALIZATION_SWAP0) | PREPARING_ACTION"
or
14:48:48 ">>> INITIALIZATION (INITIALIZATION_SWAP0) | VALIDATING_ACTION"
Indicating the State = INITIALIZATION, SubState = INITIALIZATION_SWAP0, and we're entering PREPARING_ACTION or VALIDATING_ACTION.
This will help navigate directly to the most important parts of the logs.
For more information about the states, check out the State Machine section of the docs.
Persistent State - Strategy
The strategy saves all the important variables (i.e., its internal memory) in its persistent state. The current state of the state machine, its internal accounting, its last actions, etc. Therefore, if there is any error, this is the second place to have a look to better understand the internal representation of where the Strategy is at and what could have caused the error. Obviously, this part is very case-specific and requires some blockchain knowledge, strategy understanding, and intuition from the user to be able to efficiently troubleshoot errors.
In order to edit the persistent state, the user must first pause the agent if the agent hasn't stopped yet detecting that it's looping onto an error. Simply put, the persistent state files (as well as the other files in the troubleshooting tab) can only be edited when the Agent isn't running. They can't be hot swapped.
Be careful with long numbers due to Token's decimals format leading to numbers like 12345678912345678912345678 PEPE. Make sure that if you copy/paste the JSON payload somewhere it doesn't get truncated and/or approximated to something like 1.23e25. For example, the Pretty Print option of Chrome for JSON files tends to do such transformation to make it easier to read.
Persistent State - Executioner
Similar to the strategy's persistent state, the execution layer (i.e., the executioner) saves the action bundle as a persistent state in a file. As explained in the persistent state section, this system ensures robustness should the machine crash at any moment and the software be restarted.
Each Action Bundle generated by the strategy will have a unique ID and will be saved several times on disk across all the execution layers, updating different fields as it progresses along the steps (e.g., transforming the action into transactions, sending the transactions on-chain, parsing the transaction receipts, etc.)
The user would rarely have to modify these files but might have to look at them when troubleshooting to understand what happened with faulty transactions.
Config
Editing the config.json can be very handy in order to troubleshoot the strategy. For example, in the Dynamic LP Degen strategy, one could force a rebalance by editing the config file and changing the field rebalance_frequency to set it to 1 to rebalance after 1x your granularity. If your granularity is 15m and the agent has not rebalanced in the last 15 minutes, this will now force a rebalance.
Similar to the persistent state, you will find the config file in the Troubleshooting tab.
In order to edit the config file, you first need to pause the agent, then edit the config file, save, and resume the agent. The configuration change will take effect immediately as the agent resumes. Note that pausing the agent results in the machine shutting down (to save on resources) and resuming the agent spins the machine up again, so expect a few seconds delay for such actions.
Manual Recovery - SAFE
Funds are SAFU. If anything goes wrong with the Strategy you can simply "Terminate" the Agent and then manually recover the funds. The SAFE website allows you to run "apps" like Uniswap from within their website, which basically run the real website via their platform, allowing the user to do any Uniswap-related action, like closing an LP position, swapping assets, etc.
It is important to note that using the Uniswap app to swap is not mandatory at all. The SAFE portal has a built-in swap feature (using CoW).
1. Go to SAFE -> Apps -> Uniswap
2. Use Uniswap directly from SAFE
Unfortunately, the Uniswap app on SAFE is limited to a few chains at the time of writing this guide. For chains like Base, another approach will have to be used to close any existing position. Swapping remains available via the SAFE portal since you don't have to use the Uniswap app to perform a swap. The Uniswap app limitation only affects closing positions. See the next sections on how to use a troubleshooting agent to close your position(s).
Troubleshooting Agents
In order to manually close your LP position(s) if an Agent failed, got stuck, or you terminated the agent before it could perform a clean teardown, you can use our Troubleshoot - Close Position strategy from the strategy library. You simply have to configure it to either close specific positions or all positions across pools (i.e., close all LP positions my wallet has on a specific protocol like Uniswap V3). The troubleshooting agent will then close these positions and the underlying assets (+ rewards) will be returned to the wallet. Then, the user can easily go on the SAFE portal to swap these assets back into ETH, for example.
Troubleshooting an Agent is probably the most complicated part given the breath of expertise it might required.
There is obviously no one-size-fits-all approach to all the possible error that can occur from either a bug in your strategy code to the node provider returning weird errors to 3rd party APIs being down to blockchain transanction reverting. But with the tools above anyone can troubleshoot any situation.
Feel free to reach out to the team for help!
We understand this coul be overwhelming.
We've been there ourselves.