Those 2 formulas might both be fine, it just depends on how is the angle ø defined.
They are however just special cases, when the area considered is flat and B is constant.
The real definition of magnetic flux, which is valid for every surface and in every case, is:
which means that you have to sum (integral) the contributes of the perpendicular (normal) component of B in every point of the surface A.
When B is constant everywhere (does not vary in different points of the surface) and A is flat, the integral goes away and the normal component of B is Bcosø if ø is defined as the angle between B and an axis perpendicular to A (or Bsinø if ø if the angle between B and the surface A).
Hope this helps.