I see alot of hipsters wearing suit jackets without the rest of the suit. Is this just wrong? Or are there certain conditions in which this can be pulled off?

I have absolutely no problem with this as a look — in fact, I gladly embrace it! A suit jacket is the perfect way to dress up a casual outfit — you could be out all day in a cafe & then go home to throw on a suit jacket before you went out again, & it would look marvellous. Even if you wear jeans with rips or a pre-faded t-shirt, the jacket will tidy you up extremely well.

The casual suit jacket is good in black, navy or grey. Pick something that will go well with what you already have in your wardrobe. Try something with a light pinstripe or some unusual details — unfinished edges, contrasting stitching, etc.
The most important thing to remember when you’re doing the casual suit jacket thing is that your shoes are the other major player. They will make or break the outfit! Your best bet is a nice pair of loafers or super-shined boots. Sneakers are not cool — but if you’re into the comfort thing, a pair of slip-on Vans can totally rock this.
Don’t be afraid to pull the sleeves up to just below your elbow for a bit of comfort & versatility. If you have a cool or interesting watch, this can be a sneaky way of showing it off. It also happens to look really hot, so try it!
A fine-knit hoodie (no bulk please!) with a big of extra length to cover your belt looks great under a suit jacket — flick the hood out the back.
Try it with a white shirt worn open at the neck & jeans, or a vintage t-shirt & pressed trousers.
Wear it with an extra-long dark scarf around your neck, or try one with a light stripe or texture.
If you think you can pull it off, try it with a cravat. A lot of people used to associate the cravat with old men who smoked pipes & wore gold buttons on their navy blazer, but if you’re reasonably young & rugged-looking, I think this could look really cool.
Put the suit jacket over a grey v-neck sweater. It looks very clean & sophisticated, & if you want to dress it down a bit, pair it with jeans.
When buying a suit jacket — casual or not — keep these things in mind! It may be casual, but it still has to fit!

P.S. Take a photo of how it looks if you’re afraid to leave the house, or ask the opinion of someone with taste!