This repository implements an offline, value-based reinforcement learning approach to optimize accelerometer sensitivity settings in supply chain scales. The task involves a trade-off between over-triggering (TOO_MANY) and under-triggering (TOO_FEW) weight measurements. By selecting sensitivity values conditioned on operational context—defined by scale_type, weight_bin, and box_type—the algorithm aims to reduce critical measurement errors, account for hardware-specific behavior, balance state representation, and minimize unnecessary data transmissions, thereby helping to preserve sensor battery life.
- States:
(scale_type, weight_bin, box_type) - Actions: Sensitivity levels
[2, 3, 4, 5, 6, 7, 8] - Reward: Weekly, with penalties for critical errors (TOO_FEW) and inverse frequency weighting.
- Episodes: ~1000 (one episode = all weeks in the dataset)
- Hyperparameters:
| Parameter | Value |
|---|---|
| Alpha | 0.2 |
| Gamma | 0.85 |
| Epsilon | 1.0 |
| Epsilon min | 0.1 |
| Epsilon decay | 0.9985 |
- Goal: Learn optimal sensitivity per state to maximize classification accuracy.
1- For each operational context (state), actions are selected using an epsilon-greedy strategy to balance exploration and exploitation.
2- Since the environment is observational, rewards are computed using an offline counterfactual approximation, where the chosen sensitivity is matched to the closest observed sensitivity in the historical data.
3- Measurement outcomes are mapped to reward values, with severe penalties applied to TOO_FEW events to reflect their higher operational cost. Less frequent state–error combinations are upweighted using inverse-frequency weighting.
4- State–action values are updated using temporal-difference–style value updates under a fixed-state (self-transition) assumption, effectively optimizing action selection within each context.
5- To improve stability and efficiency, the Q-table is pruned after each episode to retain only the top-N actions per state.
Although Q-learning is traditionally used for sequential decision-making problems with explicit state transitions, sensitivity calibration in this setting is quasi-static (one-time decisions, not a chain) and context-dependent, with minimal temporal coupling (today's choice doesn't change tomorrow's state) between decisions. However, we introduce a self-transition assumption (γ = 0.85) to account for cumulative effects like battery drain, transforming independent decisions into a recurring choice problem. This formulation retains Q-learning as a stable and interpretable value-based optimization framework while enabling straightforward extension to online or adaptive reinforcement learning in future work.
├── data/ # Raw & processed datasets
├── notebooks/ # Jupyter notebook with pipeline
│ └── qlearning_pipeline.ipynb
├── src/ # Modular Python code
│ ├── data_loader.py
│ ├── plot_utils.py
│ ├── qlearning_agent.py
│ └── reward_calculator.py
├── plots/ # plot images
├── requirements.txt # Dependencies
└── README.md
git clone https://github.com/irmakoz1/IoT_Sensor_Optimization_with_Reinforcement_Learning.git
cd IoT_Sensor_Optimization_with_Reinforcement_Learning
pip install -r requirements.txt
-
Raw data is not uploaded to git.
-
The scripts will run with processed data.
-
Update DATA_PATH in the notebook or config.
-
Open the notebook:
jupyter notebook notebooks/qlearning_pipeline.ipynb
-
Run all cells to train the agent and generate visualizations.
1- Sensitivity varies with scale type, weight bin, and box type.
2- Lighter bins → higher sensitivity; heavier bins → lower sensitivity.
3- Post-training evaluation shows almost all weeks meet criteria.
4- The agent learned to balance noise reduction and responsiveness.
-
Slow Convergence: The learning process requires a large number of episodes (~1000) to reach near-optimal performance, which may be impractical for rapid deployment or frequent retraining in dynamic environments.
-
Run-to-Run Instability: Due to the stochastic nature of epsilon-greedy exploration and offline counterfactual approximations, the final policy can vary across different training runs. This lack of deterministic stability may affect reliability in production settings.
-
Self-Transition Assumption: Modeling the problem with γ = 0.85 and self-transitions, while useful for capturing long-term effects, introduces a simplified dynamics that may not fully reflect real-world temporal dependencies (e.g., battery degradation over weeks).
-
Offline Counterfactual Matching: The heuristic of matching chosen actions to the closest observed sensitivity relies on the assumption that nearby sensitivity levels yield similar outcomes. This approximation may introduce bias, especially in regions with sparse historical data.
-
Hyperparameter tuning and reward function: Hyperparameter tuning and adjusting the reward function for the optimized results is time consuming.
This project successfully adapted offline Q-learning to optimize accelerometer sensitivity settings in supply chain scales, framing a quasi-static calibration problem as a contextual bandit with self-transitions. Despite the minimal temporal coupling between decisions, the approach achieved a 120% improvement in cumulative reward (from ~250 to >550) over 1000 episodes and delivered 91% compliance (10 out of 11 validation weeks) with strict operational thresholds errors (TOO_MANY/Automatic ratio < 0.5 per week & TOO_FEW < 2 per week) per week. These results validate that value-based reinforcement learning can effectively learn from observational data to balance measurement accuracy and battery preservation, while providing a stable foundation for future extensions to online learning and more complex state representations.
Ashiquzzaman, A. et al. (2020) ‘Energy-efficient IOT sensor calibration with Deep Reinforcement Learning’, IEEE Access, 8, pp. 97045–97055. doi:10.1109/access.2020.2992853.
Prauzek, M. and Konecny, J. (2021) ‘Optimizing of Q-learning day/night energy strategy for solar harvesting environmental wireless sensor networks nodes’, Elektronika ir Elektrotechnika, 27(3), pp. 50– 56. doi:10.5755/j02.eie.28875.
Shurrab, M. et al. (2022) ‘IOT sensor selection for target localization: A reinforcement learning based approach’, Ad Hoc Networks, 134, p. 102927. doi:10.1016/j.adhoc.2022.102927.
Wen, Z., O’Neill, D. and Maei, H. (2015) ‘Optimal demand response using device-based reinforcement learning’, IEEE Transactions on Smart Grid, 6(5), pp. 2312–2324. doi:10.1109/tsg.2015.2396993.