Skip to content

TASK-62: Remove N+1 in getQuantityOnHandByInventoryItem (expiry alerts p95 1339ms -> 241ms) - #11

Open
devin-ai-integration[bot] wants to merge 1 commit into
developfrom
devin/task-62-expiry-alerts-n-plus-one
Open

devin-ai-integration[bot] wants to merge 1 commit into
developfrom
devin/task-62-expiry-alerts-n-plus-one

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

✨ Description of Change

Link to GitHub issue or Jira ticket: TASK-62 (Devin Boards, latency optimizer)

Description:

ProductAvailabilityService.getQuantityOnHandByInventoryItem(location) grouped product_availability by the inventoryItem association, so Hibernate issued one select … from inventory_item where id=? per grouped row, and callers then lazily loaded each lot's product (one more select per product). It backs the dashboard expiry alerts (/json/getDashboardExpiryAlerts, GET /api/dashboard/expirationSummary on a cache miss), Expired stock / Expiring stock reports (HTML + CSV) and InventoryService.getInventorySampling.

This change groups by inventoryItem.id (the SQL of the aggregate query is unchanged, so row order is unchanged) and then loads the inventory items with their product in batches of 1,000 (select ii from InventoryItem ii left join fetch ii.product where ii.id in (:ids)). The returned Map<InventoryItem, Number> has the same keys (InventoryItem equality is by id), values and iteration order.

Original code author: the groupProperty("inventoryItem") query was added by Walkowiak (awalkowiak@soldevelo.com) in aab76df "OBPIH-3316: Refactor queries against inventory snapshot to improve performance" (2020-11-17), with the bin-location variant later touched by Dariusz Rodzewicz in 9034da3 (OBPIH-6037). The expiry-alert caller (DashboardService.getExpirationAlerts) is Justin Miranda's (cfc5bfb, OBPIH-1602).

Results — GET /openboxes/json/getDashboardExpiryAlerts, Main Warehouse, 5,760 in-stock lots / 3,000 products

p50 p95 p99 SQL statements per request
before (develop @ 4b703b8) 1,118 ms 1,339 ms 2,519 ms 8,882 (1 aggregate + 6,000 inventory_item + 2,880 product selects)
after (this PR) 216 ms 241 ms 257 ms 7 (1 aggregate + 6 batched in (…) selects)

p95 −82%, p99 −90%. Query counts are from the MySQL general log for a single request.

Method: both revisions built with ./gradlew bootRepackage and run as packaged wars (java -jar, no spring-loaded agent; under bootRun the agent adds a large reflection overhead to both sides: 4.2 s → 3.6 s p50 there), against the same MySQL 8.0.46 container and seed. 3 warm-up requests, then 60 sequential requests with curl %{time_total}; nearest-rank percentiles. 8 vCPU / 31 GB VM, JDK 8, -Xmx3g.

Equivalence evidence

Captured from the before and after builds against the same data:

  • /json/getDashboardExpiryAlerts: byte-identical
  • /api/dashboard/expirationSummary?locationId=1: byte-identical
  • /inventory/listExpiringStock?status=within90Days&format=csv and /inventory/listExpiredStock?format=csv: byte-identical (row order included)
  • the two report pages as HTML: identical except the footer's page render time.
  • ./gradlew test --tests 'org.pih.warehouse.inventory.*': 59 tests, 0 failures.

Cost / tradeoffs

  • No caching and no stale data: same query, plus ⌈n/1000⌉ batched selects.
  • All lots and products for the location are now loaded eagerly in one go (before they were also all loaded, one row at a time), so memory use is about the same.
  • The 1,000-id batch keeps each IN list small enough for MySQL and the query plan cache.

Repro

Seed (MySQL 8; MySQL 5.7 fails the Liquibase views changelog)
docker run -d --name ob-mysql -p 3320:3306 -e MYSQL_ROOT_PASSWORD=root -e MYSQL_DATABASE=openboxes \
  -e MYSQL_USER=openboxes -e MYSQL_PASSWORD=openboxes mysql:8.0 --character-set-server=utf8mb3 \
  --collation-server=utf8mb3_general_ci --sql-mode="" --log-bin-trust-function-creators=1
# start the app once so Liquibase creates the schema, then:
docker exec -i ob-mysql mysql -uroot -proot openboxes < seed.sql
SET SESSION cte_max_recursion_depth = 100000;
INSERT INTO category (id,version,name,date_created,last_updated)
WITH RECURSIVE s(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM s WHERE n<20)
SELECT CONCAT('bench-cat-',n),0,CONCAT('Bench Category ',n),NOW(),NOW() FROM s;
INSERT INTO product (id,version,name,product_code,category_id,active,cold_chain,controlled_substance,hazardous_material,reconditioned,lot_and_expiry_control,date_created,last_updated)
WITH RECURSIVE s(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM s WHERE n<3000)
SELECT CONCAT('bench-prod-',n),0,CONCAT('Bench Product ',n),CONCAT('BP',LPAD(n,5,'0')),CONCAT('bench-cat-',1+(n%20)),1,(n%7=0),(n%11=0),(n%13=0),(n%17=0),(n%2=0),NOW(),NOW() FROM s;
INSERT INTO inventory_item (id,version,product_id,lot_number,expiration_date,date_created,last_updated)
WITH RECURSIVE s(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM s WHERE n<6000)
SELECT CONCAT('bench-ii-',n),0,CONCAT('bench-prod-',1+((n-1)%3000)),CONCAT('LOT',n),
 CASE WHEN n%10=0 THEN NULL ELSE DATE_ADD(DATE('2026-11-19'), INTERVAL ((n*37)%900)-90 DAY) END,NOW(),NOW() FROM s;
INSERT INTO product_availability (id,version,inventory_item_id,location_id,product_id,product_code,lot_number,bin_location_name,quantity_on_hand,quantity_allocated,quantity_on_hold,quantity_available_to_promise,date_created,last_updated)
SELECT CONCAT('bench-pa-',SUBSTRING(ii.id,10)),0,ii.id,'1',ii.product_id,p.product_code,ii.lot_number,'DEFAULT',
 CASE WHEN CAST(SUBSTRING(ii.id,10) AS UNSIGNED)%25=0 THEN 0 ELSE 10+CAST(SUBSTRING(ii.id,10) AS UNSIGNED)%90 END,0,0,0,NOW(),NOW()
FROM inventory_item ii JOIN product p ON p.id=ii.product_id WHERE ii.id LIKE 'bench-ii-%';
INSERT INTO transaction (id,version,transaction_date,transaction_type_id,inventory_id,transaction_number,date_created,last_updated)
VALUES ('bench-tx-1',0,'2026-11-01 00:00:00','3','1','BENCH-1',NOW(),NOW());
INSERT INTO transaction_entry (id,version,transaction_id,inventory_item_id,product_id,quantity)
SELECT CONCAT('bench-te-',SUBSTRING(pa.inventory_item_id,10)),0,'bench-tx-1',pa.inventory_item_id,pa.product_id,pa.quantity_on_hand
FROM product_availability pa WHERE pa.id LIKE 'bench-pa-%' AND pa.quantity_on_hand>0;
Build, run, benchmark
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 PATH=$JAVA_HOME/bin:$PATH
./gradlew bootRepackage && cp build/libs/openboxes.war after.war   # and the same on develop -> before.war

run.sh:

#!/bin/bash
# usage: run.sh before|after  -> runs the packaged war (no spring-loaded agent) against the bench DB
unset JDK_JAVA_OPTIONS JAVA_TOOL_OPTIONS
exec /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Xmx3g -Dgrails.env=development \
  -DdataSource.url='jdbc:mysql://127.0.0.1:3320/openboxes?serverTimezone=UTC&useSSL=false&allowPublicKeyRetrieval=true' \
  -Dlogging.level.root=INFO -jar $1.war

login.sh:

#!/bin/bash
B=http://localhost:8080/openboxes; rm -f cj
curl -s -c cj -b cj -o /dev/null -X POST -d "username=admin&password=password" $B/auth/handleLogin
curl -s -c cj -b cj -o /dev/null "$B/dashboard/chooseLocation/1"

bench.sh:

#!/bin/bash
# usage: bench.sh <label> <n>   (requires cookie jar ./cj logged in, location chosen)
B=http://localhost:8080/openboxes; L=$1; N=${2:-40}
for i in 1 2 3; do curl -s -b cj -o /dev/null "$B/json/getDashboardExpiryAlerts"; done
: > lat-$L.txt
for i in $(seq $N); do curl -s -b cj -o /dev/null -w "%{time_total}\n" "$B/json/getDashboardExpiryAlerts" >> lat-$L.txt; done
python3 - "$L" <<'PY'
import sys,math
l=sorted(float(x)*1000 for x in open(f"lat-{sys.argv[1]}.txt"))
def p(q): return l[min(len(l)-1, math.ceil(q*len(l))-1)]
print(f"{sys.argv[1]}: n={len(l)} p50={p(.5):.0f}ms p95={p(.95):.0f}ms p99={p(.99):.0f}ms max={l[-1]:.0f}ms")
PY

Devin-Org: engineering


Latency Optimizer opened this PR for Taylor
Implemented in session Find and fix next slow path in new repo
Review with Devin · Open in Devin Desktop



Note

Devin errored when opening this Pull Request as taylorcurranpython.
As a fallback, Devin opened this PR as itself.

…em (remove N+1)

Co-Authored-By: taylor.curran <taylorcurranagtc@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Custom reviewers

  • ✨ Blast Radius Reviewer: no findings · View
  • 🛡️ API Security Reviewer: no findings · View

Devin Review

@devin-ai-integration

Copy link
Copy Markdown
Author

Browser verification

Tested the after-build war against seeded MySQL 8, signed in as admin at Main Warehouse.

  • The dashboard Expiration Summary graph is populated, and the Next Year range renders too.
  • Expiring stock within 90 days: 529 lots. The CSV downloaded with 529 data rows.
  • Expired stock: 528 lots. The CSV downloaded with 528 data rows.
  • Legacy expiry alerts return expired 528, 30d 182, 90d 529, 365d 2147, >365d 2605.
Dashboard expiration summary Expiring report
Dashboard Expiring stock
Expired report Legacy alert counts
Expired stock Legacy counts

Test recording

Browser CSV dates appear one day later than in the curl-captured baselines because dates are formatted in the session timezone. Byte equivalence between before and after was checked with the same curl session (see the description). No before-build browser comparison was run.

Written by Devin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants