how to find out oil whirl problem in bearing through vibration?

The frequency would be unique. Just less than 0.5 times running speed according to the "charts".

I think you posted in the wrong place. This belongs in the vib forum, not the motor forum.

If one distinguishes between oil whirl and oil whip, then oil whirl will appraximately track a fraction of running speed on coastdown until it stabilizes; this is a rigid rotor issue. Oil whip will not track (much); this is a flexible rotor issue - on motors whirl is most likely because you don't operate above 2 X the shaft flexible resonance. Both are related to resonances (eigenvalues), like control instabilities.

Particularly on vertical motors, these can lock into high structural vibration (the casing shakes a lot). Otherwise, shaft relative probes may be the best measurement device to see this.