if a fan is placed on a small boat in opposite direction in which it should move moving, then why the boat does not move by the force of the fan
The crucial point here is internal versus external force.
The fan experiences a force in the direction opposite the flow of air (think of a propeller).
When the fan is on land, this force is not applied to the boat system. The flow of air from the fan produces a force on the sail which accelerates the boat.
If the fan is attached to the boat, the force on the fan is no longer external to the boat system. The fan acts as propeller applying a force to the boat opposite the flow of air. Ideally, it there is a sail capturing the full air flow produced by the fan, the force from the sail cancels the force from the fan. In reality, I imagine that the propulsion from the fan exceeds that from the sail so the boat moves opposite that desired.