Peel and core the apples, then slice them about ¼–⅛ inch thick. Melt the vegan butter in a large skillet over high heat, then add the apples, brown sugar, flour, cinnamon, ginger, and nutmeg. Cook, stirring often, for 5–10 minutes.
Add the brandy and maple syrup and continue cooking until the apples are tender but not mushy, about 15 minutes total. Remove from the heat and cool completely.
Prepare the vegan whipped cream, measuring 3 full cups once fully whipped, and refrigerate while preparing the mousse. In a large bowl, combine the vegan cream cheese, sifted confectioners' sugar, vanilla, and lactic acid by hand. Fold in the whipped cream.
In an ungreased 9 × 9-inch baking pan, arrange a layer of graham crackers. Spread with ⅓ of the mousse and half of the apples. Repeat with another layer of graham crackers, ⅓ of the mousse, and the remaining apples. Finish with a final layer of graham crackers and the remaining mousse.
Sprinkle the top with the optional vegan toffee and drizzle with caramel if desired. Refrigerate for 3–4 hours before serving. For the best texture, it can be refrigerated overnight.