dbcp 连接池不合理的锁导致连接耗尽解决方案连接池不合理的锁导致连接耗尽解决方案
dbcp 连接池不合理的锁导致连接耗尽解决方案连接池不合理的锁导致连接耗尽解决方案
应用报错,表象来看是连接池爆满了。
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:241) ~[spring-jdbc-
3.2.2.RELEASE.jar:3.2.2.RELEASE] at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)
[dubbo-2.5.3.jar:2.5.3] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_33] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_33] at java.lang.Thread.run(Thread.java:662) [na:1.6.0_33]
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:103) ~[commons-dbcp.jar:1.2.1] at
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540) ~[commons-dbcp.jar:1.2.1] at
com.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74) ~[crmpub-jdbc-1.0.jar:1.0] at
org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203) ~[spring-jdbc-
3.2.2.RELEASE.jar:3.2.2.RELEASE] ... 32 common frames omitted
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174) ~[commons-pool-1.6.jar:1.6] at
org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74) ~[commons-dbcp.jar:1.2.1] at
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95) ~[commons-dbcp.jar:1.2.1] ... 35 common frames omitted
应用堆栈,几十条线程都阻塞到dbcp池的PoolableConnectionFactory.makeObject方法了,此方法在等待
0x0000000709a638a0对象锁,但0x0000000709a638a0对象一直被其中一条线程执行
PoolableConnectionFactory.makeObject时锁了,而且这条线程获取到锁后还阻塞住了,所以导致后面几十条线程都
BLOCKED了。
"DubboServerHandler-132.121.95.136:28101-thread-300" daemon prio=10 tid=0x00007f02b4136800 nid=0x4954 waiting for monitor entry
[0x00007f0286018000] java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
- waiting to lock <0x0000000709a638a0> (a org.apache.commons.dbcp.PoolableConnectionFactory)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
at com.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74)
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:417)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:255)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at com.eshore.crm.service.common.pub.aspect.DataSourceRouteAspect.around(DataSourceRouteAspect.java:74)
at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631)
at com.eshore.crm.service.sysmgr.sca.impl.SysMgrServiceImpl$$EnhancerByCGLIB$$bfd71326.qryBulletinList(<generated>)
at com.alibaba.dubbo.common.bytecode.Wrapper101.invokeMethod(Wrapper101.java)
at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46)
at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72)
at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)
at com.eshore.crmpub.sca.dubbo.filter.BaseFilter.invoke(BaseFilter.java:90)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)
at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- <0x000000070a120f00> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
"DubboServerHandler-132.121.95.136:28101-thread-290" daemon prio=10 tid=0x00007f03203da800 nid=0x4948 runnable [0x00007f0286a22000]
- 1
- 2
- 3
前往页