Video Uploads not accessible after upload

alfaquad · July 14, 2015, 5:10pm

I just finished uploading some video files (at least I thought I did)
When I try to access these files by clicking on them to play they do not play and then when I click on
“If the video does not play properly try to watch it directly.” I receive the following error message.
Not sure what to do …

I removed the “<” so the code will show.

Error>
Code>NoSuchKey
Message>The specified key does not exist.
Key>
DB06DAEDE85041379E8C4145EF66787F/vid/E58502C669344D30943A5A07C9E5C510/nameofmp4file.mp4
/Key>
RequestId>34E448E6AB8E31B6
HostId>
TkfyLnQFfFQ0YV0dQl5nv7qYU+B2Hgdw/mtan0ItMfdpdLF/DIyhmaFQye3AnnpVVJhrRDltEig=
/HostId>
/Error>

Nitai · July 15, 2015, 1:13pm

From the error message I can guess that you use S3. Looks like the video
could not be uploaded to S3 successfully.

No idea what version of Razuna you use, how you use it, where you host it,
etc. So, can’t help much.

alfaquad · July 15, 2015, 2:14pm

Nitai

You are right… rookie mistake I should have provided you all of that infortmation.
I am runnung V 1.7
on an Amazon EC2 Instance m3.medium
Ubuntu Server 14.04 LTS (HVM), SSD Volume Type
mySql Database
Amazon s3 Storage

We are using it as a “warehouse” for all of our digital content (images, video’s, text files, docs etc…)
It worked fine just days ago I wias able to upload larger files (the largest I uploaded was approx 1gb…

Now… it seems the video uploads are what is breaking it… there seems to be a file size threshold that, once passed, causes the files to not be uploaded. I am not sure exactly how big the threshold is … I do know that a 128mb video file works and a 568mb video file does not.

Thanks

Nitai · July 15, 2015, 2:34pm

Thanks. Looks everything like it should work. Also, when you say that it
worked and now doesn’t seems to be that the issue is either with S3 or some
other settings.

Have you looked into the memory usage of Razuna? Are there any errors in
the catalina.out log file? Did you try to reboot the server and try to
upload another large video?

alfaquad · July 15, 2015, 4:11pm

Thank you for your response Nitai…

how do I check Razuna’s memory usage?
I did reboot the server and attempt to upload another large video with the same results
the catalina.outlog file did have some errors in it, although none that showed as a result of my attempt to upload the video after server restart… i included some of it below (the file is 4732 lines I am sure you did not want to see all of it)

There were several files added to the DAM … this is a copy of the last one and then the beginning of errors Log File
Lines 4077 -4088

Start indexing: 1 - 39A1B9AAFDF44F3788C908C254672482 - {ts ‘2015-07-15 14:43:29’}
DONE indexing: 1 - 39A1B9AAFDF44F3788C908C254672482 - {ts ‘2015-07-15 14:43:30’}
Jul 15, 2015 3:00:22 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler [“http-bio-8080”]
Jul 15, 2015 3:00:22 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Jul 15, 2015 3:00:23 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/razuna] registered the JDBC driver [org.h2.Driver] but failed to unregister it when the
web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Jul 15, 2015 3:00:23 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/razuna] registered the JDBC driver [org.h2.Driver] but failed to unregister it when the
web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Jul 15, 2015 3:00:23 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/razuna] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

THE ABOVE ERRORS RUN FROM LINES 4079 - 4624
Then from lines 4725 - 4732:

Jul 15, 2015 3:00:23 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load
javolution.util.FastTable$FastTableIterator. The eventual following stack trace is caused by an error thrown for
debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no
functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1612)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
at javolution.util.FastTable.iterator(FastTable.java:462)
at org.apache.catalina.loader.WebappClassLoader.loadedByThisOrChild(WebappClassLoader.java:2634)
at org.apache.catalina.loader.WebappClassLoader.checkThreadLocalMapForLeaks(WebappClassLoader.java:2552)
at org.apache.catalina.loader.WebappClassLoader.checkThreadLocalsForLeaks(WebappClassLoader.java:2507)
at org.apache.catalina.loader.WebappClassLoader.clearReferences(WebappClassLoader.java:2013)
at org.apache.catalina.loader.WebappClassLoader.stop(WebappClassLoader.java:1908)
at org.apache.catalina.loader.WebappLoader.stopInternal(WebappLoader.java:661)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5730)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1591)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1580)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Jul 15, 2015 3:00:23 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalsForLeaks
WARNING: Failed to check for ThreadLocal references for web application [/host-manager]
java.lang.NoClassDefFoundError: javolution/util/FastTable$FastTableIterator
at javolution.util.FastTable.iterator(FastTable.java:462)
at org.apache.catalina.loader.WebappClassLoader.loadedByThisOrChild(WebappClassLoader.java:2634)
at org.apache.catalina.loader.WebappClassLoader.checkThreadLocalMapForLeaks(WebappClassLoader.java:2552)
at org.apache.catalina.loader.WebappClassLoader.checkThreadLocalsForLeaks(WebappClassLoader.java:2507)
at org.apache.catalina.loader.WebappClassLoader.clearReferences(WebappClassLoader.java:2013)
at org.apache.catalina.loader.WebappClassLoader.stop(WebappClassLoader.java:1908)
at org.apache.catalina.loader.WebappLoader.stopInternal(WebappLoader.java:661)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5730)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1591)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1580)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: javolution.util.FastTable$FastTableIterator
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
… 16 more

Jul 15, 2015 3:00:23 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalsForLeaks
WARNING: Failed to check for ThreadLocal references for web application [/manager]
java.lang.NoClassDefFoundError: javolution/util/FastTable$FastTableIterator
at javolution.util.FastTable.iterator(FastTable.java:462)
at org.apache.catalina.loader.WebappClassLoader.loadedByThisOrChild(WebappClassLoader.java:2634)
at rg.apache.catalina.loader.WebappClassLoader.checkThreadLocalMapForLeaks(WebappClassLoader.java:2552)
at org.apache.catalina.loader.WebappClassLoader.checkThreadLocalsForLeaks(WebappClassLoader.java:2507)
at org.apache.catalina.loader.WebappClassLoader.clearReferences(WebappClassLoader.java:2013)
at org.apache.catalina.loader.WebappClassLoader.stop(WebappClassLoader.java:1908)
at org.apache.catalina.loader.WebappLoader.stopInternal(WebappLoader.java:661)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5730)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1591)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1580)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Jul 15, 2015 3:00:23 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler [“http-bio-8080”]
Jul 15, 2015 3:00:23 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler [“http-bio-8080”]
Java HotSpot™ 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
Jul 15, 2015 3:12:03 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Jul 15, 2015 3:12:04 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler [“http-bio-8080”]
Jul 15, 2015 3:12:05 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 2175 ms
Jul 15, 2015 3:12:05 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Jul 15, 2015 3:12:05 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.56
Jul 15, 2015 3:12:05 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /opt/razuna_tomcat_1_7/tomcat/webapps/razuna
log4j:WARN No appenders could be found for logger (org.apache.commons.vfs.impl.StandardFileSystemManager).
log4j:WARN Please initialize the log4j system properly.

------------SERVER STARTUP------------------
—START: Lock file cleanup—
No .lock file to remove
—DONE: Lock file cleanup—
—START: Cache Setup—
—DONE: Cache Setup—
---------------FINISHED---------------------
Jul 15, 2015 3:12:39 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /opt/razuna_tomcat_1_7/tomcat/webapps/razuna has finished in 34,058 ms
Jul 15, 2015 3:12:39 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /opt/razuna_tomcat_1_7/tomcat/webapps/host-manager
Jul 15, 2015 3:12:39 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /opt/razuna_tomcat_1_7/tomcat/webapps/host-manager has finished in 163 ms
Jul 15, 2015 3:12:39 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /opt/razuna_tomcat_1_7/tomcat/webapps/manager
Jul 15, 2015 3:12:39 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /opt/razuna_tomcat_1_7/tomcat/webapps/manager has finished in 157 ms
Jul 15, 2015 3:12:39 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler [“http-bio-8080”]
Jul 15, 2015 3:12:39 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 34617 ms
Start indexing: 1 - 77D781E3598F49DC96E275063D978188 - {ts ‘2015-07-15 15:21:06’}
DONE indexing: 1 - 77D781E3598F49DC96E275063D978188 - {ts ‘2015-07-15 15:21:09’}

The last 2 lines were added after I tried uploading a large video again.

THANKS for your help!

Nitai · July 15, 2015, 4:34pm

You can check memory usage by either using “top” or “free -m” in the shell.
We recommend to use at least 16GB Ram for your Razuna server.

The log files look ok. Those errors are from Tomcat internally and not from
Razuna.

alfaquad · July 16, 2015, 2:19pm

Thank you Nitai

What else can I check to trouble shoot this problem?. keeping in mind that it worked (at least 1gb uploads tested) then just stopped for no apparent reason.
I am not sure what else to do… I just tried a 297mb video upload and that did not work either.

Thanks for your help

Nitai · July 24, 2015, 12:27am

I don’t know your server nor your setup. However, and this is network 101,
you need to have a good connection to S3 in order to stream constantly. We
have seen that weak network performance can influence uploading.