Home / Forums / Understanding Luau Fastcall, Closures & Bytecode Execution Pipeline

UnreliableCode Community

Developer Research, Reverse Engineering & Coding Community

Discussion

Understanding Luau Fastcall, Closures & Bytecode Execution Pipeline

LuaScripterX
Luau & Lua VM Reverser
VIP
Rep: 260
Join Date: Mar 2022
Posts: 20
Thanks: 68
2y ago · Jan 16, 2024 2:05 PM
#1
Roblox's custom Lua runtime (Luau) introduces significant bytecode and VM architecture optimizations over standard Lua 5.1:

1. Fastcalls: Built-in functions (like
CODE
math.sqrt
,
CODE
type
,
CODE
string.byte
) bypass the standard function call overhead and execute directly as dedicated VM opcodes (
CODE
LOP_FASTCALL
).
2. Vector3 Native Type: Vector3 is a first-class tagged value (stored directly in the
CODE
TValue
struct) rather than a heap-allocated userdata object.
3. Closure Environments: Luau separates C-closures and L-closures with strict capability bitmasks.

How do you analyze custom proto instructions and upvalue tables in IDA?
LuaScripterX · Luau VM & Bytecode Engineering
The following users thanked LuaScripterX for this post:
NullPtr_
Assembly & Engine Research
MEMBER
Rep: 260
Join Date: May 2023
Posts: 27
Thanks: 62
2y ago · Jan 16, 2024 4:30 PM
#2
In Luau, the bytecode opcode table rotates between versions. Using an automated script to match the VM execution loop switch-cases against known opcode patterns makes decompilation much simpler.
NullPtr_ · 0xDEADBEEF was here
SigScannerPro
Pattern Scanning & IDA
MEMBER
Rep: 210
Join Date: Jul 2023
Posts: 25
Thanks: 52
2y ago · Jan 16, 2024 7:45 PM
#3
The Vector3 inline optimization in Luau's
CODE
TValue
union is really clever. Storing the 3 floats directly in the unused payload space saves millions of GC allocations per frame in physics loops.
SigScannerPro · Pattern scanning & Byte masking