Figure given above shows the free body diagram (FBD) of bith blocks. Forces acting on each block are shown in FBD .
Let us consider the block of mass M. Mg is the weight , i.e. gravitational force of block,
N2 is the reaction force acting beteen the contact surfaces of floor and block,
μ2N2 is the friction force acting against the movement of block .
T is tension force acting on the rope and aslo at pulley connected to block.
N1 is the normal reaction force exerted by block of mass m
Now let us see FBD of block of mass m . Gravitationa forec, i.e. weight mg is acting downward, T is tension in the rope,
μ1N1 is the friction force acting against the movement along the contact surface between two blocks
and N1 is the reaction force exerted by block of mass M.
Let us apply the Newtons second law for the block of mass m
In horizontal direction, if we assume ax is acceleration of mass m , then we have, N1 = m ax .....................(1)
In vertical direction, if we assume ay is acceletaion of mass m , then we have, mg - μ1 N1 - T = m ay ...............(2)
Let us apply the Newtons second law for the block of mass M
In horizontal direction, if we assume Ax is the acceleration of block of mass M,
then we have, 2T - N1 - μ2 N2 = M Ax ...............(3)
In Vertical direction, there is no movement , hence we have, N2 = M g + T ........................(4)
At equilibrium, we have, ax = ay = Ax = 0
Hence from eqn.(1), we get N1 = 0
By using N1= 0 and ay = 0, from eqn.(2) we get, T = mg
By substituting for T, N1 , N2 and Ax in eqn.(3) , we get , ( 2 m g ) - μ2 ( M g + m g ) = 0
Hence , we get, m = ( μ2 M ) / ( 2 - μ2 ) = (0.4 × 20 ) / (2 - 0.4 ) = 5 kg