Ok, I don't think the two are mutually exclusive. In fact, they tend to overlap.
I need to set up some definitions. I will define vigilantism as "collective coercive practices undertaken by non-state actors in
order to enforce norms (social or judicial) and/or to take the law in
their own hands".
British criminologist Les Johnston suggests several criteria for defining vigilantism in his article What is Vigilantism?:
it involves planning and premeditation by those engaging in it; its participants are private citizens whose engagement is voluntary; it is a form of autonomous citizenship and, as such, constitutes a social movement; it uses or threatens the use of force; it arises when an established order is under threat from the transgression, the potential transgression, or the imputed transgression of institutionalized norms; it aims to control crime or other social infractions by offering assurances (or ‘guarantees’) of security both to participants and to others.
The punisments meted out by vigilantes can be quite spectacular and symbolic: vigilantes cannot arrest all criminals but can make punishment into a symbol to frighten others. Sanctioning often takes the form of physical punishment, public humiliation or, more rarely, execution. These sanctions are an attempt to control by exemplary punishment. So, the use of lynching is an example of both vigilantism and terrorism.
Terrorism is "The unlawful use or threatened use of force or violence by a person or an organized group against people or property with the intention of intimidating or coercing societies or governments, often for ideological or political reasons."
What we have is the common factor of violence used to enforce, or create, a norm. Extra-legal or extra-Constitutional use of threats, or physical and psychological violence to punish, or to cause, incite or stoke fear and hate are common to both vigilantism and terrorism. I would say that it's sometimes hard to distinguish between the two since they share the characteristics of (1) use of force (2) for a political purpose.
Some might want to say that vigilantism is supposed to enforce the "law", or at least the established order. On the other hand, it is a non-governmental actor using force most of the time in violation of the law. To go back to the lynching example: the Ku Klux Klan (KKK) targeted African-Americans as well as Jews, immigrants, leftists, homosexuals, Catholics, Muslims, and atheists. The KKK used physical assault and lynching, against politically active blacks and their allies, even if the latter were whites. This was all done in opposition to the civil rights movement.
As I said, there is not a clear line between vigilantism and terrorism: both use violence for a political purpose.