WAS 内存溢出分析

所需积分/C币:46 2018-12-04 20:43:16 733KB PDF
99
收藏 收藏
举报

Websphere Application Server WAS 内存溢出分析,帮助分析内存问题
温室小花技术博寥一纯粞的技术博客 红颜弹指老,刹那芳华,与其天湃思君,恋夵不舍,心绕不断莫若相忘于江湖! [,985,254,616(86.05%)[80]16 array of java/util/concurrent'ConcurrentHashMapsSegment 0x44dal O 150, 721,056(6.53%)[32] ava/util/concurrent/ConcurrentHash Map sSegmentox44dab7b0 O 139, 436,688(6.04%)[32 2java/utill concurrent/ConcurrentHash Map s Segment ox44dab690 O 134, 524, 104(5.83%)[32]java/util/ concurrent/ConcurrentHashMapssegmentox44dab670 O 133, 257 584(5.78%)[32 1 2 java/util'concurrent'ConcurrentHashMap Segment Ox44dab810 O 133, 154, 160(5.77%)[32]java/ current/Con currentHash Map sEgment Ox44dab7dO O 129, 497, 680(5.61%)[32]2 java/util/concurrent/Con currentHash Map S segmentOx44dab7f0 O 125,851,416(5.46%)[3212javalutillconcurrent/ConcurrentHash Maps Segment Ox44dab6fo O 125, 749, 424(5.45%)[3212java/util/concurrent/ConcurrentHashMaps Segment Ox44dab790 b O 123, 292, 648(5.34%9)[32] 2 java/util/concurrent/ConcurrentHashMapsSegment Ox44dab6dO O 122,071, 544(5.29%)[32]java/util/ concurrent/ConcurrentHashMap segment ox44dab710 o 114, 611, 192(4.97%)[32]java/utilconcurrent/ConcurrentHash Map s segment Ox44dab6b0 o 114, 596, 248(4.97%)[32 2 java/utilconcurrent/ConcurrenthashMapSSegment ox44dab770 O 111,015,088(4.81%)[32]java/util/concurrent/Con currentHash Map S Segment ox44dab830 O 109, 635,400(4.75%)[32] 2javalutil/concurrent/ConcurrentHashMapsSegment ox44dab750 合4 M0 AnA A夏AT'『 TyAdd-br3r1 o 10,608(0%6[162 org/hibernateldialectfunctionSQLFunction Registry 0x666b2358 vo 10,464(0951406 orghibernateldialectoraclegDialect 0X666b2800 vO 6,280(00148 1 java/uti l HashMap 0x666b3908 7(]6,232(0%)528]52 array of java/util/Hash Map SEntry Ox666b5178 vO 504(0%)24]3 java/util/ HashMap Entry 0x666c2650 vo 464(0%)[24] java/util HashMap SEntry 066a91aa0 o 440(0%)(24]2 org/hibernateldialect/functian/SQLFunction Template Ox56a9lab8 7O416(0%)24]3 org hibernate/dialectfunction/Template Renderer 0x66a91ad0 [] 360(0%)140]5 array of java/lang/String Ox66a9lae8 32(0%)320in0x66a91b08 y 80(0%)24)1/lang/ string 0x43569fa8 >y 16(0%[162orghibernaterty pe/string l ype 0x4434f298 48(09)[241 java/lang/string 0x43569T80 o16(0%)[16]1 org/hibernateldialectifunction/StandardSQLFunction 0x66a91a90 56(0924 1 java/lang/ string 0x401766a8 eo 432(0%)243 javalutil HashMap entry 0x666c27e8 3)从GC日志 native seder:lg看到内存是在短时间内消耗的,在GC日志中搜索产生的 hcapdump文什名,定位到产生内存溢出的时间点 iF#- 03 27: 32 20171JVMDLMP0321JVM'Calwaspd3/DM WebSphere/App Server/profiles/App Srv324jevacore 20171221.032735 4653104.0006 ttoc- 匡升凹十二H210321:32201 WMDUMEU1 Ul Java a二二口 /aspd3 MNWabsphere/App proti es/ppSn:24 avacore1i1221.32!3b465318UUU 正期四十二月210327:322017] JVMDUMPI0sJM" a0/wasp3 BNNeb Sphere/App SeIvelpnufles/AppSIv324Su20171221.33275.4653184.0007.L二口 一l期四十二月210327322017] MDUMP01l5 nanda二a二 wasp3 M Web sphere App Server/profiles/Appsry324/snap20171221.03273546531840007tr( 星期四十二月210327:3220171 NMDUMPI01 Ivo0on00O74000systhrowcoODOEp山 -Ojavallang outoMemoryerrorooi匚 4)以此时间点开始,往前寻找产生内存泄漏的原因,通过分析 SyslemErr log日志,发现 多年 经验丰富 经验现居广州 技术博客 温室小花技术博寥一纯粞的技术博客 红颜弹指老,刹那芳华,与其天湃思君,恋夵不舍,心绕不断莫若相忘于江湖! 17-12-21:3:26分左右线程 Housekeeper发生OOM错。 R ExGep-ion in thread e下e a.1ang0u0 em aryErrcr:「aa炸四间 etag 17-12-21:26:34:333 CST]0000007b Syst] R 儿 b SystenEr R at java. ang, RuntineEscepti<n. init>(RumtimeEzcey 46:641ccT]000o b Syst enEr g.1o1 1.ut:1Faty红 t(FastarravList Tb Syst enEr bs proxocl Housekeep eieeperCor.tro-l E at org. logicslcobwebs prowocl. HonsaKeeperIhread ru 5)根据报OOM那个时间段,直接在 SystemOut. log查找如下OOM报错: 17--2-21 3: 28: 33: 904 CST] 0c000022 TaskutiIsSLog E EC, springframework, scheduling support. IaskUtilssLoggingErrorliandler 1 ·黑LtM是myry!山是Wa ,功, HeapByte目山Ex,<i> HeapByteB王Eex,iW:E0) a·票票·出e出上e·110ate(出y是出上Y·限票:32:] 5. 5treamDncocerCharse=gL ( treaminccde…辱票:325 a二un只,3.str∈ amEncocerscharse-sE,<再>(5七工 eamEnccde,:309 当u县,C3.SL工 ErLcuLtr SCildISE_SE,>(3 LrtaiuEnctue,jsys:2531 Mn,2,SE后品mnex王2 tput stze品mxt曰(三曰 enCoder,2) d二成引·而灵, Output=t工 =amorite工.<期而>10 utputstreamR工工七e工·成啊品:113) ·灰,En5工m,<永(Pm吕园如m,顶:12 7·.P1MT5rhm,顶1(F1m乐”品,录园:) amm,J以,8,E,yemream,n+m1nP(y9tertream,M:mN37) cm·h,导受,改是,S七 reame ven七6· writes1 iOstream(三 treanEven-E,是 ac0m·正·,,Sy七 - strcam一x(sy3 tcmst二m,元可:03】 a二c0m·头k黑.兵泉·,三yt出0u二黑(三 ystemoutstream·:B0〕 透黑,heIn西te,顶,期,5工5西teet工gE1g5选tmnt5p5 tatementlcgg里,点点:113】 器二9,五L工nBte·工只·工 strAtch工,LCg(吕二工品C上E七C工,工灵是:母4〕 9点8 hibernate.R, bstract Batcher,9 etpreparedstatement iibstractBatcher,1“分 d二9复h1 iteiabC.AbstractBatcher getPreparedstatement iAbs d·1上比,再职 LldELBdLclier,上D上d比m(跳【 dU sBd L上,再151 a= 9xs-hibernate loader Loader. prepareCuerystatemert (Loader 700) a二9x·h1 enate.10Adex,0adex, InQueRy江0a1·:801) 0二·1Ymt.16adY.工baly. QucrYAdInitia1 zeNon&2yb11c1n2|Ldc.总黑点:27 二9xgh1e工nBe10de,工0∈工·10adn1y(L0ade工,1W:2037) a= 9"g.hiocrnatcloader entity. 9oasractEntityload=y i load(AbosracEnsitLcadEI39%0: 861 a. oxa-hihernate-Inaderentity. ahs ract ntitynad-r Inad (Ahs-racrFnrityT W通:7f a, alternate,e工8 IstEr, entITY,8 tractEntityP81Bte,10a(Bt工 actEntityPer13七e工·1豆日3254 a=Sx- hibernate, Ewent.ce.土a11 ELoackventl13ten.1ad上xm妇tas01工e(Lea11t己 detent工ene工,易:生b) a=8=8.hibernate. event def. efault-oadEventListemar doLoadiDefaultLoadEventLis=ener.ava: 477) a- mshibernate. event. def. DefaultLoadEventListemer load(Default LoacEwentListener. ava: 227) 器从,加L,=V,上,上上山二v出L==,以D上sv上=,又:147) a5黑8 =rnaTe·恶黑决·Sea1mm1,里e工a阻(色工n1·:L90 a- og.hibernate.inpl.se53i0nImol imeduiateload(session Imp.java:1026) 并且在如下截图看到报错是 LMS entity, Translnfo服务触发了 java.util. concurrent服务进程,这与上面从 heapdump日志看到的有大量 java/util/ concurrent/给应用程序触发调用可以对应上 多年 经验丰富 经验现居广州 技术博客 温室小花技术博寥一纯粞的技术博客 红颜弹指老,刹那芳华,与其天湃思君,恋夵不舍,心绕不断莫若相忘于江湖! t只买只出工上日工ae职·e部22m.mmel2t到Bd(e12叫且.展:2b) at org. hibernate. proxy. AbstractlazyInitializerinitialize(hbstractlazyinitializerjaya: 173) t9某2王be工nae,pXY,tEa七工zy工工t品L会E,引工m1et品五n(2如E品etL品z?工mt品1z,顶豆是日21 at某g.hb∈nate-p0xy·职··JaBt工zyn1a112¥·1 moke(Jaa3si3 lAzy工n1ta11ze王,:190) 云t ,y,工an工n。的]va8320, getRecorCl工Eenn的]v园13_29·W 1 :1134) ame切是Te·那从 HyREportServic=lmm1en比山点( MyReportservIC已m1·: d=u.说,viU,, My LSeIvict1, uttyUnkIlJwR=ut(出yRu8eiem1,成:生) at aun. xcflcct CcncratkdMcthcdAccca30r2076invol:(Unknown Source) at sun reflect. Delegatinglle thocAccessorImpl invoke (DelegatinpethodAcce ssorImpl java: 37) at立观·其就工ef1ect. Me thoc. invoke( Methad,元:611) at org·用x点黑MQx,品B,Upp工七,ApU113. invokCVoinpointoinrRcf1 CAtiOn(2 opti1a,立MR:31T atg-美买质买展··上里mM工足⊥色出t工eMt⊥二己t1n.工nvEe1mc1nt(比e上⊥色ctWe区e二1mCat15是283 aLM·再凯队玩属,,上ⅢwUIk.Re1把⊥vM= LhuunvUic!· rucked〔Re1euL⊥ veMeLhUlluyULCLiLa,jsM馬:150 atg,x美k,tYm3ct1n, intcyccot:,Tran2 ackionIntC.2Ptx1, proccccn⊥th工vbca=1n(工 Pemgact-onIr= reCptor,方 at高·融那系灰七EC上卫.1 interceptor·“2 ansactionAspectSupport. mckeW2 hinlrang3ctcn( ranga.ct2 cnAspectsupr at只其gg职其, RansaC七on, interceptor. fransactioninte二epar⊥ nwOKE江T工 ansactlon工nte工 CeptO工·立又:5生 a七只·强职如只k,9, framework且eE】会二工eM世thd工回七五n,B工ceed( BeflectiveMethocinyocaticn,:172 at某·买美员·员, framework, JakDynam1AP6xy·1 worE( Jakdynarm已DP工xY·5:204 at ccn. sunproxy. sProxy8T queIyUnknowReport I Unknown source) 医tn把t.J5·ck.CDR=1己tJb,re15工p5 ndstock I CDRep。愍:30) at sun reflect, eneratedyethccAccessorL( at sunreflect. DelegatingMethodAccessorImplinvoke(DelegatingHethodAccessorInpl. ava: 37) a七,刻 gReflect, Method. invoke( Metha,顶员:11) atgg-王美用某,sche11ng,pp0t,schd1 edMetnoceumnal1e·un(3 check1e创 Metnocirunma1e·是延:4) L只x只·工再其其其. sane-n11ng. support,Le1=gat1 ngErrorHan1 ng sunnah1e·工un(De⊥ erating工工0工EBn11 ngkunnab⊥e,1:53) at org. springframework scheduling, concurrent ReschedulingRunnable. run(ReschedulingRunnable axa: a1) t员,示工ent,国e= tora sUnna21dte,cB11(卫zeca 熟:450) ar. axa. 1-11-concurrenT.FiT1reTasks Sync 1nnevRun( FurnrPTask18x3: 314) 七票2,, concurrent,E正eEsk,E(Eut证eEk,1母9 at lava W-Al concurrent. SdheduledThreadeoolExecutorsSchedule lEutureTask run(ScheduledThreac Executor, iava:=s: 1c at iava. u=1g.concurrent SdheduledThreadPoolExacutorsScheduleiFutureTask acces35301(ScheduledThreadPoolExecu=cris at ava. u=-1 concurrent s. sadPoo1ExecutorsWorker, runTask(ThreaPoolExecutor, iava: 906) at占a,真 concurrent 品LP心LEX号2FW心EEe2,2(h艺品B心卫X以七2·界:S29 TPA日.JM:773) 并且从 System Outdo中看到,产生OOM错误前,系统正在执行大量的数据库读写操作,如 下图所示: OHEn I UGB58Ie G] CI IDIE LL I GOW UG]a& As G3 0 JUST C3bOb or c和丁私°叮cm概手打P] 论出于面动请 IRLT o]re F 巨F sr【L fI812]to0 工。2C 打2q·和2F:g狐FI「 6)分析 lavacore文件,也观杏到,在运行 Housekeeper进程,造成之后33多个进程处于等 待资源分配状态: 多年 经验丰富 经验现居广州 技术博客 温室小花技术博寥一纯粞的技术博客 红颜弹单指老,刹那芳华,与其天湃思君,恋不舍,心绕不断莫若相忘于江湖! Thread Delail: javacore. 20171221.03230946531840003txt \ame state Natlv/e. Method Stack Housek.( Run0xf301. javalla. 10 Inread HouseKeeper ore o.+ Wait. 0:65B0.java/ut. Name tore or.+ Waiti. Ox7 a30.java/ut. State Runnable Waiti. Ox1840.java/la. 6650 Com.+Waiti. Ox50-00.NO JA. at java l a g/ wablefillIn StackTrace(Native Method tore or.E+ Waili. Oxdd10.java/ut t java/la^ tHrowAble Throwable」ava:59( Compiled o。l≥) tore or.+Waiti. Oxb 101d3 java/ut at java a gRowable. Throwable java: 70(Compiled Code Store or.+Waiti. Ox7530.java/ut at javaJla g Excepticn (Exception java: 52(Compiled Code) tore or...+ Waiti. 0x4e 20.. java/ut 6 at ja g RLntim e Exception (RuntimeException java. 54 Compiled Cade) Store or.+ Waiti. 0r3c930.java/ut. Java at javalangIr dexoutofBoundsException Ir dexoutofB our ds Exception java: 48(Compiled Store or., Waiti. 0:1370.iava/ut Stack at java/Lt IArrayus1 get(ArrayList ]ava. 3b2(Compiled Coda) e上wa:1460JH at crglogi cal Gabw'e bs/proxool/uti l/FastArrayList get( Lst java 178Complled code) Store or.F+ Waiti. 0:3d50.iava/ut. 66R5 at crglogisal GobAe hs/proxoo/ House Keeparcontroller ge: HDuEeKBeper l oRun tore or.+ wait. UKa 2B Java/ut. setn→ Waili O: 1:20 j:'l 〔 HCUSeRespArContro!3Va:s5( compiled Coa》 Store o.. Waiti.0:6b50.java/ut. t reloc|lcpD日 s/proXoo/Hous.K『Inr日aH叫us铝阳”n30a:33 e or.+ wain. 0 ea0255 Java/u. 〔 Complied cod Status Number of Threads: 3331 Percentage t Deadlock 0(%0 Runnable 11 0 CS Waiting on condition 3308 99() 这个H0 usckccpcr N应该是获取中银用户的客户数据,可能数据量太大,并且进行多用户 查询,因而触发大量并发进程,并占用大量内存和缓存空间,从而产生OOM错误,致使数 据都直接写到 systemout. log日志,因为数据量大,致使一下占满 systemout. log20m的默认 大小空间,从而产生多个 SystemOut.log文件 目 SystemOu 2017/12/221610文本文档 7789KB a SystemOut.12.2023.53.09 2017/1221612文本义档 20480KB Syste mOut17122103.17.12 2017/12/221612文本文档 480KB SystemOut17122103.19.24 2017/12/221613文本又档 20,479KB a SystemOut.12.21/04:09.36 2017/12/221613文本文档 20.480KB SystemOut 17.12.2215.20.36 2017/1 1609文本文档 20480KB 总结建议 2月213点26分左右, wasp3- Appare324发生了内存溢出( Out of memory)。分 柝, heapdump、 Javacore与 Native stderr, ystemErr log, SystemOut. log日志后,发现在发生内存 溢出前后, Systemout. log不断亠生中银用户户数据库读写输出操作,应该是有 Housekeeper 数据调取进程不断在尝试调取和提交数据,应该占用大量内存,并触发大量并发走程,并 且有可能应用存在资源抢占情况,导致空闲内存不足,发生內存溢出。 多年 经验丰富 经验现居广州 6 技术博客 温室小花技术博寥一纯粞的技术博客 红颜弹指老,刹那芳华,与其天湃思君,恋夵不舍,心绕不断莫若相忘亍江湖! 建议: 检查优化内存泄露点对应的代码,包括那些定期调度的程序 请把以上关于对OOM的分析转给开发人员参考,检查相关的代码,检查 HouseKeeper 应用,并进行性能调优,避免触发大量并发,并适量加大应用运行环境的JVM大小,以适 合应用正常运行的内存资源需求 避免创建占用大内存对象 代码应尽量避免创建占用内存过大的对象,占用内存过大的对象的创建会影响系统性 能,也有可能导致內存溢岀。这些对象可能是应用代码直接产生,也可能是应用所调用的 第三方框架产生的,建议检查对应的代码,并设计方案进行优化。 2017/12/26 多年 经验丰富 经验现居广州 技术博客

...展开详情
试读 7P WAS 内存溢出分析
立即下载 低至0.43元/次 身份认证VIP会员低至7折
一个资源只可评论一次,评论内容不能少于5个字
您会向同学/朋友/同事推荐我们的CSDN下载吗?
谢谢参与!您的真实评价是我们改进的动力~
  • 签到新秀

关注 私信
上传资源赚钱or赚积分
最新推荐
WAS 内存溢出分析 46积分/C币 立即下载
1/7
WAS 内存溢出分析第1页
WAS 内存溢出分析第2页

试读结束, 可继续读1页

46积分/C币 立即下载 >