40 and 96 are both logical answers to the series, depending on the methods used. Tahiti MaliCk identified one of the methods. Another is:
previous line + 1# + 2# = answer.
0+1+4=5
5+2+5=12
12+3+6=21
21+8+11=40
However, it is interesting that the 4th line jumps 5 in the inputs, whereas the previous lines increase by 1. So is that to infer the previous series should be extended to 8, 11 by increases of 1 per line? I.e.:
21+4+7=32
32+5+8=45
45+6+9=60
60+7+10=77
77+8+11=96
Interesting the answer is the same as (8×11)+8=96

40

40

89

96

Sorry it should be 96.

40

96

(1×4)+1=5

(2×5)+2=12

(3×6)+3=21

(8×11)+8=96

40 and 96 are both logical answers to the series, depending on the methods used. Tahiti MaliCk identified one of the methods. Another is:

previous line + 1# + 2# = answer.

0+1+4=5

5+2+5=12

12+3+6=21

21+8+11=40

However, it is interesting that the 4th line jumps 5 in the inputs, whereas the previous lines increase by 1. So is that to infer the previous series should be extended to 8, 11 by increases of 1 per line? I.e.:

21+4+7=32

32+5+8=45

45+6+9=60

60+7+10=77

77+8+11=96

Interesting the answer is the same as (8×11)+8=96