During a web cast we had many people logging on at one time, about 100. As stated earlier when a user successfully logs in they pull an swf file from the database and load that into their client app. Usually this works perfectly fine with small files, < 1Mb in size, and has even worked with a much larger audience. On this occasion we had a 10Mb swf file to stream from the database, all hell broke loose. Once many people began signing on and downloading this file we began receiving MANY Broken Pipe exceptions from this controller. Right now it is not clear if the database was breaking the connection, the DBCP 1.2.2 connection pool was breaking the connection, the spring MVC controller was breaking it OR the flash client apps were breaking it. I am leaning towards either the database or the connection pool as the culprit here. One last clue was tomcat totally hung when too many of these errors occurred, it refused to serve any more pages. Even after starting and stopping the tomcat service in Fedora it refused to work, I had to hard reboot the servers.
This continued to occur after hard rebooting the servers, I still received Broken Pipe exceptions and Tomcat continued to hang.
I guess the next step is to try and recreate the situation. I am going to take my Uber jar approach and build out a JUnit test which spawns off about 30 threads each calling the download ppt method from the web app. Using JUnit Perf I am going to make this available from the command line, install it on several servers on Amazon Cloud EC2, then nail the main web app with this test and simulate around 120 users downloading a massive powerpoint file at the same time.
Possible Solutions:
If it is found that streaming a large BLOB directly from the database to many clients will never work I have come up with a few different solutions.
- Continue storing the BLOB in the database AND add a filePath field to the database which points to a location on disk where this file is also stored. This would give me the confidence that I always have a copy of the file sitting in the database in case I need it. This is nice if for some reason the file on disk is deleted I can always restore it. NOW, when a user requests this ppt file the system will first check to see if the file is available on disk, if it is just stream it to the users using regular file io. If the file is NOT on disk, syncronize the call, then stream the file from the database to disk, then stream to the client from the disk.
- Pretty much the same solution as option 1, but instead of streaming the file from disk store it in Amazon S3 storage.
