The massive number of Internet-of-Things (IoT) devices generates a hard-to-manage volume of data. Cloud-centric processing approaches for the IoT data suffer from high and unpredictable network latency, which causes poor experience in real-time IoT applications, such as healthcare. To address this issue, in edge computing, the data inference starts from the data source (i.e., the IoT devices). However, the constrained computational capabilities of the IoT device and the power-hungry data transmission demand a tradeoff between onboard processing and computation offloading. Hence, the IoT information inference requires efficient and lightweight techniques that are tailored for this tradeoff and respect the constrained resources on IoT devices, such as wearables. This article presents a hierarchical classification approach that decomposes the problem into three classifiers in two hierarchy layers. In the first layer, a lightweight classifier executes directly on the IoT device and decides whether to offload the computation to the gateway or to perform it onboard. The second layer comprises a lightweight classifier on the IoT device (can only distinguish a subset of classes) and a complex classifier on the gateway (to distinguish the remaining classes). The experimental results (using a real-world data set for human activity recognition and implemented on a wearable IoT device) show higher accuracy (92% on average) than a nonhierarchical classifier (87% on average). The execution time and power measurements on the IoT device show energy saving for the classification.