====== WebLogic and PeopleSoft ======

With PT8.47, there were no problems with using Weblogic in a PeopleSoft setting.  Unfortunately with PT8.49, I ran into some issues!  Weblogic seemed to work just fine serving up the PS Web Client and PeopleBooks.  Unfortunately Weblogic would crash hard every time I tried to use the PeopleBooks search functionality.  God damn Stack Overflow errors!

===== The Error =====
<code>
===== BEGIN DUMP =============================================================
JRockit dump produced after 0 days, 00:02:02 on Tue Apr  8 12:17:43 2008

    **********************************************************
    *  If you see this dump, please go to                    *
    *  http://edocs.bea.com/jrockit/go2troubleshooting.html  *
    *  for troubleshooting information.                      *
    **********************************************************

Additional information is available in:
  /opt/psoft/pt849/hcm890/webserv/peoplesoft/jrockit.11852.dump
No snapshot file (core dump) will be created because core dumps have been
disabled. To enable core dumping, try "ulimit -c unlimited"
before starting JRockit again.

Error Message: Illegal memory access. [54]
Signal info  : si_signo=11, si_code=2 si_addr=0xb4366000
Version      : BEA JRockit(R) R27.5.0-110-94909-1.5.0_14-20080204-1558-linux-ia32
GC Strategy  : Mode: throughput. Currently using strategy: genparpar
GC Status    : OC is not running. Last finished OC was OC#2.
             : YC is not running. Last finished YC was YC#4.
OC History   : Strategy genparpar was used for OC#1 to OC#2.
YC History   : Ran 2 YCs before OC#1.
             : Ran 1 YCs before OC#2.
             : Ran 1 YCs since last OC.
YC Promotion : Last YC successfully promoted all objects
Heap         : 0x8100000 - 0x18100000  (Size: 256 MB)
Compaction   : 0x8100000 - 0x9100000  (Current compaction type: internal)
NurseryList  : 0x82b3410 - 0x11383840
KeepArea     : 0x10b56c78 - 0x11383840
NurseryMarker: [ 0xd7ffe28,  0xf5e31e8 ]
CompRefs     : References are 32-bit.
CPU          : Intel Pentium 4 SSE SSE2 NetBurst
Number CPUs  : 1
Tot Phys Mem : 3855900672 (3677 MB)
OS version   : release 3
Linux version 2.6.22-gentoo-r5 (root@sprydev) (gcc version 4.1.2 (Gentoo 4.1.2)) #1 Fri Aug 24 08:08:12 EDT 2007 (i686)
Thread System: NPTL
Java locking : Normal
State        : JVM is running
Command Line : -XcheckedStacks -XnoOpt -XXnoJITInline -Xms256m -Xmx256m -Dtoplink.xml.platform=oracle.toplink.platform.xml.jaxp.JAXPPlatform -Dweblogic.Chunksize=65536 -Djava.uti
l.logging.config.file=/opt/psoft/pt849/hcm890/webserv/peoplesoft/applications/peoplesoft/logging.properties -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14
Logger -Dweblogic.Name=PIA -Dps_vault=/opt/psoft/pt849/hcm890/webserv/peoplesoft/psvault -Djavax.net.ssl.trustStore=/opt/psoft/pt849/hcm890/webserv/peoplesoft/keystore/pskey -Dwe
blogic.ProductionModeEnabled=true -Djava.security.policy=/opt/psoft/pt849/bea/wlserver92/weblogic92/server/lib/weblogic.policy -Dssl.debug=false -Dps_home=/opt/psoft/pt849/hcm890
 -Dsun.java.launcher=SUN_STANDARD weblogic.Server
java.home    : /opt/psoft/pt849/bea/wlserver92/jrockit90_150_14
j.class.path : .:/opt/psoft/pt849/hcm890/webserv/peoplesoft/lib/ps_patch.jar:/opt/psoft/pt849/hcm890/webserv/peoplesoft/applications/peoplesoft/PSIGW/WEB-INF/lib/mail.jar:/opt/ps
oft/pt849/hcm890/webserv/peoplesoft/lib/portlet-api-1.0.jar:/opt/psoft/pt849/hcm890/webserv/peoplesoft/lib/pluto-1.0.1.jar:/opt/psoft/pt849/hcm890/webserv/peoplesoft/applications
/peoplesoft/PORTAL/WEB-INF/lib/saaj.jar:/opt/psoft/pt849/hcm890/webserv/peoplesoft/applications/peoplesoft/PORTAL/WEB-INF/lib/xml-apis.jar:/opt/psoft/pt849/hcm890/webserv/peoples
oft/applications/peoplesoft/PORTAL/WEB-INF/lib/xercesImpl.jar:/opt/psoft/pt849/bea/wlserver92/patch_weblogic921/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/opt/ps
oft/pt849/bea/wlserver92/jrockit90_150_14/lib/tools.jar:/opt/psoft/pt849/bea/wlserver92/weblogic92/server/lib/weblogic_sp.jar:/opt/psoft/pt849/bea/wlserver92/weblogic92/server/lib
/weblogic.jar:
j.lib.path   : /opt/psoft/pt849/bea/wlserver92/jrockit90_150_14/lib/i386/jrockit:/opt/psoft/pt849/bea/wlserver92/jrockit90_150_14/lib/i386:/opt/psoft/pt849/bea/wlserver92/jrockit
90_150_14/../lib/i386:/opt/psoft/pt849/bea/wlserver92/patch_weblogic921/profiles/default/native:/opt/psoft/pt849/hcm890/jre/lib/i386/native_threads:/opt/psoft/pt849/hcm890/jre/li
b/i386/server:/opt/psoft/pt849/hcm890/jre/lib/i386:/lib:/opt/microfocus/cobol/lib:/opt/oracle/product/10.2.0.1/lib:/opt/psoft/pt849/hcm890/bin:/opt/psoft/pt849/hcm890/bin/interfa
cedrivers:/opt/psoft/pt849/hcm890/bin/sqr/ORA/bin:/opt/psoft/pt849/hcm890/optbin:/opt/psoft/pt849/hcm890/verity/linux/_ilnx21/bin:/opt/oracle/product/10.2.0.1/lib:/opt/psoft/pt84
9/bea/tuxedo91/lib:/opt/psoft/pt849/bea/wlserver92/weblogic92/server/native/linux/i686:/opt/psoft/pt849/bea/wlserver92/weblogic92/server/native/linux/i686/oci920_8:/opt/psoft/pt8
49/bea/wlserver92/weblogic92/server/native/linux/i686:/opt/psoft/pt849/bea/wlserver92/weblogic92/server/native/linux/i686/oci920_8:/opt/psoft/pt849/hcm890/verity/linux/_ilnx21/bi
n
JAVA_HOME    : /opt/psoft/pt849/bea/wlserver92/jrockit90_150_14
_JAVA_OPTIONS: <not set>
LD_LIBRARY_PATH: /opt/psoft/pt849/bea/wlserver92/jrockit90_150_14/lib/i386/jrockit:/opt/psoft/pt849/bea/wlserver92/jrockit90_150_14/lib/i386:/opt/psoft/pt849/bea/wlserver92/jrock
it90_150_14/../lib/i386:/opt/psoft/pt849/bea/wlserver92/patch_weblogic921/profiles/default/native:/opt/psoft/pt849/hcm890/jre/lib/i386/native_threads:/opt/psoft/pt849/hcm890/jre/
lib/i386/server:/opt/psoft/pt849/hcm890/jre/lib/i386:/lib:/opt/microfocus/cobol/lib:/opt/oracle/product/10.2.0.1/lib:/opt/psoft/pt849/hcm890/bin:/opt/psoft/pt849/hcm890/bin/inter
facedrivers:/opt/psoft/pt849/hcm890/bin/sqr/ORA/bin:/opt/psoft/pt849/hcm890/optbin:/opt/psoft/pt849/hcm890/verity/linux/_ilnx21/bin:/opt/oracle/product/10.2.0.1/lib:/opt/psoft/pt
849/bea/tuxedo91/lib:/opt/psoft/pt849/bea/wlserver92/weblogic92/server/native/linux/i686:/opt/psoft/pt849/bea/wlserver92/weblogic92/server/native/linux/i686/oci920_8:/opt/psoft/p
t849/bea/wlserver92/weblogic92/server/native/linux/i686:/opt/psoft/pt849/bea/wlserver92/weblogic92/server/native/linux/i686/oci920_8:/opt/psoft/pt849/hcm890/verity/linux/_ilnx21/
bin
LD_ASSUME_KERNEL: <not set>
C Heap       : Good; no memory allocations have failed
StackOverFlow: 1 StackOverFlowError has occured
OutOfMemory  : 0 OutOfMemoryErrors have occured

Registers (from ThreadContext: 0xb4358570 / OS context: 0xb435866c):
   eax = 00000000    ecx = 0000682d    edx = 00000000    ebx = afb2a74c
   esp = b4358960    ebp = b43800e0    esi = afe2362c    edi = b4366000
    es = 0000007b     cs = 00000073     ss = 0000007b     ds = 0000007b
    fs = 00000000     gs = 00000033
   eip = b7e67837 eflags = 00010246

Stack:
(* marks the word pointed to by the stack pointer)
b4358960: afde78c4* afa144ad  b4358ca8  00000000  0002740c  afa14480
b4358978: b4356f48  0806b0fc  b5a6fc14  0807ef7c  b445e758  00000000
b4358990: 00000001  00000000  00000000  0d000004  00000000  00000000
b43589a8: b4356f48  080e8064  b5a6fc14  0807ef7c  b44660b0  0000000c

Code:
(* marks the word pointed to by the instruction pointer)
b7e67804: 4c8b0824  b60f1024  e30c2444  83d7892c  117403e2  49aa047a
b7e6781c: 49aa1f74  f2831b74  aa027501  c1ca8949  e28302e9  01c06903
b7e67834: f3010101* f3d189ab  24448baa  90c35f08  90909090  90909090
b7e6784c: 90909090  0c24448b  10244439  5b42820f  90660006  0c244c8b

Loaded modules:
(* denotes the module causing the exception)
08048000-08056ff3  /opt/psoft/pt849/bea/wlserver92/jrockit90_150_14/bin/java
b7f6d000-b7f6d62b  /opt/psoft/pt849/bea/wlserver92/jrockit90_150_14/bin/java
b7f55000-b7f67e8b  /lib/libpthread.so.0
b7f2f000-b7f52443  /lib/libm.so.6
b7f2b000-b7f2cbcb  /lib/libdl.so.2
b7dfa000-b7f239ff */lib/libc.so.6
b7f6e000-b7f87a1b  /lib/ld-linux.so.2
b7abc000-b7d7bb67  /opt/psoft/pt849/bea/wlserver92/jrockit90_150_14/lib/i386/jrockit/libjvm.so
b7ab2000-b7ab877f  /lib/librt.so.1
b7a3f000-b7a4981b  /opt/psoft/pt849/bea/wlserver92/jrockit90_150_14/lib/i386/libverify.so
b7a1c000-b7a3c317  /opt/psoft/pt849/bea/wlserver92/jrockit90_150_14/lib/i386/libjava.so
b7a05000-b7a1741f  /lib/libnsl.so.1
b695e000-b6963f13  /opt/psoft/pt849/bea/wlserver92/jrockit90_150_14/lib/i386/native_threads/libhpi.so
b65af000-b65bd4c4  /opt/psoft/pt849/bea/wlserver92/jrockit90_150_14/lib/i386/libzip.so
b52d3000-b52e4f03  /opt/psoft/pt849/bea/wlserver92/jrockit90_150_14/lib/i386/libnet.so
b6800000-b6806207  /opt/psoft/pt849/bea/wlserver92/jrockit90_150_14/lib/i386/libnio.so
b69de000-b69e3246  /opt/psoft/pt849/bea/wlserver92/jrockit90_150_14/lib/i386/libmanagement.so
b42b7000-b42c0357  /opt/psoft/pt849/bea/wlserver92/jrockit90_150_14/lib/i386/libjmapi.so
b69db000-b69dcde4  /opt/psoft/pt849/bea/wlserver92/weblogic92/server/native/linux/i686/libwlfileio2.so
b69fe000-b6a022ab  /opt/psoft/pt849/hcm890/webserv/peoplesoft/applications/peoplesoft/PSOL/WEB-INF/lib/linux/libpsjniverity.so
af868000-afb0151f  /opt/psoft/pt849/hcm890/verity/linux/_ilnx21/bin/libvdk30.so
af7b5000-af85d223  /usr/lib/libstdc++-v3/libstdc++.so.5
b69e9000-b69f1f2f  /lib/libgcc_s.so.1


"[ACTIVE] ExecuteThread: '0' for" id=14 idx=0x34 tid=11868 lastJavaFrame=0xb4384e0c

Stack 0: start=0xb4364000, end=0xb4386000, guards=0xb4369000 (disabled), forbidden=0xb4367000
Thread Stack Trace:
    at <unknown>(???.c)@0xb7e67837
    at CDB_readparse_aux+102()@0xafa1660d
    at CDB_readparse+53()@0xafa1659f
    at CSetParse+121()@0xafa1fe4a
    at VduCharSetNew+558()@0xafa177c3
    at VduLocaleNewEx+1064()@0xafa4b863
    at VduLocaleNew+35()@0xafa4b432
    at LocSetSessionLocale+80()@0xafa4bfd9
    at UtlInitPhase1+1101(utl_init.c)@0xafa87cf3
    at UTL_init+166()@0xafa87f96
    at VdkGlobalNew+1171(vose.c)@0xaf8dc133
    at IVdkSessionNew+108(vose.c)@0xaf90d8c8
    at VdkSessionNew+375()@0xaf90d7a2
    at _ZN10vdkSessionC1EPcS0_S0_+123()@0xb6a00e6b
    at _ZN13SearchSessionC1EPcS0_S0_PPKcS0_S0_+100()@0xb6a00f34
    at Java_com_peoplesoft_pt_psol_psverity_psjniverity_vdkOpenSearch+661()@0xb6a01435
    -- Java stack --
    at com/peoplesoft/pt/psol/psverity/psjniverity.vdkOpenSearch(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;[Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)I(N
ative Method)
    at com/peoplesoft/pt/psol/psverity/VeritySearch.search(VeritySearch.java:176)
    at com/peoplesoft/pt/psol/FullTextSearch.doGet(FullTextSearch.java:307)
    at com/peoplesoft/pt/psol/FullTextSearch.doPost(FullTextSearch.java:383)
    at javax/servlet/http/HttpServlet.service(HttpServlet.java:763)
    at javax/servlet/http/HttpServlet.service(HttpServlet.java:856)
    at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:223)
    at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
    at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:283)
    at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:175)
    at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3245)
    at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
    at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)
    at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2003)
    at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:1909)
    at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1359)
    at weblogic/work/ExecuteThread.execute(ExecuteThread.java:209)
    at weblogic/work/ExecuteThread.run(ExecuteThread.java:181)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
Pending Exception:
java/lang/StackOverflowError:
    --- End of stack trace

Extended, platform specific info:
libc release: 2.6.1-stable
Elf headers:
libc       ehdrs: EI: 7f454c46010101000000000000000000 ET: 3 EM: 3 V: 1 ENTRY: 00016140 PHOFF: 00000034 SHOFF: 0012d6a4 EF: 0x0 HS: 52 PS: 32 PHN; 11 SS: 40 SHN: 67 STIDX: 66
libpthread ehdrs: EI: 7f454c46010101000000000000000000 ET: 3 EM: 3 V: 1 ENTRY: 00004810 PHOFF: 00000034 SHOFF: 000143a0 EF: 0x0 HS: 52 PS: 32 PHN; 10 SS: 40 SHN: 33 STIDX: 32
libjvm     ehdrs: EI: 7f454c46010101000000000000000000 ET: 3 EM: 3 V: 1 ENTRY: 0005a960 PHOFF: 00000034 SHOFF: 003017c0 EF: 0x0 HS: 52 PS: 32 PHN; 4 SS: 40 SHN: 19 STIDX: 16

    **********************************************************
    *  If you see this dump, please go to                    *
    *  http://edocs.bea.com/jrockit/go2troubleshooting.html  *
    *  for troubleshooting information.                      *
    **********************************************************

===== END DUMP ===============================================================
</code>

===== The Fix =====
  * Edit $PS_HOME/webserv/peoplesoft/bin/setEnv.sh file. (Add the -Xss64m parameter to JAVA_OPTIONS_LINUX env Var)
Change
<code>
JAVA_OPTIONS_LINUX="-jrockit -XnoOpt -XXnoJITInline -Xms256m -Xmx256m -Dtoplink.xml.platform=oracle.toplink.platform.xml.jaxp.JAXPPlatform"
</code>
to
<code>
JAVA_OPTIONS_LINUX="-jrockit -Xss64m -XnoOpt -XXnoJITInline -Xms256m -Xmx256m -Dtoplink.xml.platform=oracle.toplink.platform.xml.jaxp.JAXPPlatform"
</code>