Reverting in UA is cumbersome and expensive. It is more efficient to design your UA with IFG such that if a transaction was accepted at source, the transaction will be accepted at the remote. For example, Stargate has a credit management system (Delta Algorithm) to guarantee that if a swap was accepted at source, the destination must have enough asset to complete the swap, hence the IFG.
It is important for UA to keep track of their own nonce (e.g. by events) to correlate the send and receive side transactions. UA at send() side can query the nonce at endpoint.getOutboundNonce interface, and in lzReceive() the inboundNonce is in the arguments.
Try to do only one thing per message. The implication is that if the message was burnt (misconfiguration, bad code etc.. the damage to the state is minimal.
If the message execution fails at the destination, try-catch, and store it for future retry. From LayerZero's perspective, the message has been delivered. It is much cheaper and easier for your programs to recover from the last state at the destination chain.
Store a hash of the message payload is much cheaper than storing the whole message.
If your app includes multiple message types to be sent across chains, compute a rough gas estimate at the destination chain per each message type. Your message may fail for the out-of-gas exception at the destination if your app did not instruct the relayer to put extra gas on contract execution. And the UA should enforce the gas estimate on-chain at the source chain to prevent users from inputting too low the value for gas.
Check the address size according to the source chain (e.g. address size == 20 bytes on EVM chains) to prevent a vector unauthenticated contract call.
Use type-safe bytes codec. Use custom codec only if you are comfortable with it and your app requires deep optimization.