Java Virtual Threads Ate My Memory: A Web Crawler's Tale of Speed vs. Memory
created: May 29, 2025, 1:08 p.m. | updated: May 31, 2025, 2:55 p.m.
But then, curiosity struck: “What happens if I use Virtual Threads instead?”Virtual Threads are one of my favorite recent additions to the Java ecosystem.
Switching from platform threads to Virtual Threads dramatically improved the URL processing rate… until the whole thing blew up with an OutOfMemoryError .
Web crawler: Platform threads versionBefore playing with Virtual Threads, I created a very basic crawler using traditional platform threads.
Web Crawler: Virtual Threads VersionI swapped out myExecutors.newFixedThreadPool(200)withExecutors.newVirtualThreadPerTaskExecutor()As you can see in Virtual Threads there isn’t a direct built-in mechanism to set a global limit on the total number of virtual threads that can be created (Virtual threads are designed to be lightweight and numerous).
The Bigger PictureThis experience taught me that Virtual threads aren’t just faster platform threads, they fundamentally change how we think about concurrency limits and resource management.
1 week, 4 days ago: Hacker News