Unmanaged Thread Problem - WebSphere Compliant Spring Task Schedulers - (JNDI Context problems)

Problems Summary

1. Spring creates a separate threads to handle schedulers , but most of J2EE container vendors like websphere , weblogic ..etc doesn't support the threads created outside of containers. (These are threads called unmanaged threads)
2. Following exception was thrown when we try to lookup the objects directly using javax.naming.initialContext . (We have JAX-RPC webservices client code that is generated by RAD using WSDL , which is auto generated code has the direct initial context lookups)

Exception Occured

  javax.naming.ConfigurationException: A JNDI operation on a "java:" name cannot be completed because the server runtime is not able to associate the operation's thread with any J2EE application component.  This condition can occur when the JNDI client using the "java:" name is not executed on the thread of a server application request.  Make sure that a J2EE application does not execute JNDI operations on "java:" names within static code blocks or in threads created by that J2EE application.  Such code does not necessarily run on the thread of a server application request and therefore is not supported by JNDI operations on "java:" names. [Root exception is javax.naming.NameNotFoundException: Name "com/env/url/testrouce"

Explanation and solution

IBM clearly states that don't create unmanaged threads (this is true with most of J2EE container vendors), for the following reasons:

1. The application server does not recognize unmanaged threads.
2. Unmanaged threads do not have access to Java EE contextual information.
3. Unmanaged threads can use resources without being monitored by the application server.
4. Unmanaged threads can adversely affect application server functions such as shutting down gracefully or recovering resources from failure.
5. An administrator cannot control the number of unmanaged threads or their use of resources.

http://pic.dhe.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.nd.multiplatform.doc/info/ae/ae/cspr_design.html

Spring Framework Support

Spring framework does provide a package with scheduling classes based on CommonJ WorkManager/TimerManager that supported by IBM Websphere 6.0+ and BEA weblogic 9.0+.  Spring Commonj classes link is provided in references section.

Steps to create websphere compliant task scheduler

1. Create a task (That implements the runnable interface)
2. Add the Websphere timer Manager reference in the web.xml and ibm-web-bnd.xmi
3. Configure task at specific intervals in the SpringContext.xml

Step1: Create task

public class TestTask implements Runnable {

public void run() {

//your work to be executed        

}

}

Step 2: add websphere timer manager references in project

In Web.xml

<resource-ref>

<description>TimerManager</description >

<res-ref-name>timerManager</res-ref-name >

<res-type>commonj.timers.TimerManager</res-type >

<res-auth>Container</res-auth >

<res-sharing-scope>Unshareable</res-sharing-scope >

</resource-ref>

in ibm-web-bnd.xml

<resource-ref name="timerManager" binding-name="tm/default" ></resource-ref>

Step 3: configure scheduler

<bean id="testTask" class="com.sivavaka.test.spring.scheduler.TestTask">

</bean>        

<bean id= "timerScheduler" class="org.springframework.scheduling.commonj.TimerManagerTaskScheduler" >

      <property name="resourceRef" value="true"/>

      <property name="timerManagerName" value="timerManager"/>

</bean>

<!-- To schedule job for every one hour then cron will be "0 0 0/1 * * ?" -->

<!-- Websphere Compliant Thread :: Runs for every 5 mins -->         

<task:scheduled-tasks scheduler="timerScheduler" >

      <task:scheduled ref="testTask" method="run" cron="0 0/5 * * * ?" />

</task:scheduled-tasks >

If you create the scheduler like above , you can see the WebSphere Runtime available to thread that executes this task , basically it is WebSphere managed thread

  

But if you use the Spring's task scheduler directly (I mean without using the websphere timemanager), thread that executes this task doesn't have access the Websphere Runtime causing the J2EE container services are not available.

References

1. http://pic.dhe.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.nd.multiplatform.doc/info/ae/ae/cspr_design.html
2. Workmanagers (http://pic.dhe.ibm.com/infocenter/wasinfo/v8r0/index.jsp?topic=/com.ibm.websphere.nd.multiplatform.doc/info/ae/asyncbns/concepts/casb_workmgr.html)
3. http://docs.spring.io/spring/docs/3.0.x/api/org/springframework/scheduling/commonj/package-summary.html

Download/Browse the Sample Code

1. To download sample code war file  (https://sourceforge.net/projects/blog-sivavaka-com-code-samples/files/WebSphere%20Compliant%20Spring%20Task%20Scheduler/)
2. To browser the sample code (https://code.google.com/p/blog-sivavaka-com-code-samples/source/browse/#svn%2Ftrunk%2FSivaTestSpringScheduler)

Multiple RAD workspaces - using single WAS profile vs multiple profiles

Sharing the Same WAS profile across the multiple RAD workspaces

Same WAS profile can be shared across the multiple RAD workspaces because profile is part of the WebSphere Application Server (or WAS Test Environment ). Sharing of the same WAS profile between the different workspaces is not recommended unless there is only one active workspace at any time.

Following things need to consider when sharing the profile

1. Avoid making the changes to same application in different workspaces
2. Only one workspace is allowed to debug at a given time

It always better to create different WAS profile for each workspace if you are working on same applications (e.g. same code base but parallel development branches or streams). Followings steps makes easy to bring your new workspace with all configurations from your old workspace (workspace preferences and profile configurations).

1. Restoring the workspace preferences
1. Export the workspace preferences, i.e any custom workspace preferences that you have configured (like console limit, validations …etc)

From the RAD File Menu --> Export --> Preferences

2. From the new workspace, import the workspace preferences that you have exported in previous step

From the RAD File Menu --> Import --> Preferences

2. Restoring the profile configurations
1. Export the server profile configuration from the old workspace

give workspace folder location to save configuration car file

2. This car file contains all configurations like Datasources, Security related config (JAAS auth ..etc) , JVM properties , shared libraries, deployed applications …etc. Its basically snapshot of all config for that profile.
3. In the new workspace,
1. create new WAS profile (for 32-bit Operating systems you can use the Profile Management Tool in the RAD and for 64-bit operating systems you can use the "interactive profile management tool" as described in this article)
2. Once the profile is created , add that profile to new workspace
1. Click "new" in servers view
2. Select "Server Type" and "Server Runtime" and click next
3. Select new profile that is created in above step

4. click next to add any applications in workspace to server.

3. Once the profile is added to workspace , right click on the server profile and restore the old profile configuration (you may need to copy the .car file that exported from the old profile to this workspace folder ).

4. Start the server after successful restore, Now you should see all configuration from old profile here.

Ref:

http://blog.sivavaka.com/2013/10/using-manage-profiles-interactive-tool.html

Using the "Manage Profiles Interactive" tool to manage websphere profiles in 64-bit operating systems

IBM doesn't support the manageprofiles (Profile Management) GUI or wizard in 64-bit operating systems but recently IBM has posted "Manage Profiles Interactive" tool to simplify profile creation process in 64-bit operating systems.

You can download the manageprofilesInteractive.zip from the following link (it also has flash demonstration on how to execute it)

http://www-01.ibm.com/support/docview.wss?uid=swg21442487

1. Extract  manageprofilesInteractive.zip to /AppServer/bin

2. Execute the command "WebSphere\AppServer\bin>run_manageprofilesInteractive.bat"

3. It created the "AppSrv01" profile successfully.

Reference

http://www-01.ibm.com/support/docview.wss?uid=swg21442487

JSR286 Spring Portlet Sample Code (Skeleton)

1. Create the portlet project using the RAD (by selecting the JSR 286 with empty portlet)

2. Add spring framework jars to /WEB-INF/lib directory

log4j-1.2.16.jar

org.springframework.aop-3.0.5.RELEASE.jar

org.springframework.asm-3.0.5.RELEASE.jar

org.springframework.beans-3.0.5.RELEASE.jar

org.springframework.context.support-3.0.5.RELEASE.jar

org.springframework.context-3.0.5.RELEASE.jar

org.springframework.core-3.0.5.RELEASE.jar

org.springframework.expression-3.0.5.RELEASE.jar

org.springframework.jdbc-3.0.5.RELEASE.jar

org.springframework.orm-3.0.5.RELEASE.jar

org.springframework.transaction-3.0.5.RELEASE.jar

org.springframework.web.portlet-3.0.5.RELEASE.jar

org.springframework.web.servlet-3.0.5.RELEASE.jar

org.springframework.web-3.0.5.RELEASE.jar

3. In web.xml , Add Spring context listner

<context-param>

<param-name>contextConfigLocation</param-name>

<param-value>/WEB-INF/SivaTestSpringPortlet-Context.xml</param-value>

</context-param>

<listener>

<listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>

</listener>

<servlet>

<servlet-name>ViewRendererServlet</servlet-name>

<servlet-class>org.springframework.web.servlet.ViewRendererServlet</servlet-class>

<load-on-startup>1</load-on-startup>

</servlet>

<servlet-mapping>

<servlet-name>ViewRendererServlet</servlet-name>

<url-pattern>/WEB-INF/servlet/view</url-pattern>

</servlet-mapping>

4.  In portlet.xml, update the portlet classname, contextConfigLocation

<?xml version="1.0" encoding="UTF-8"?>

<portlet-app xmlns="http://java.sun.com/xml/ns/portlet/portlet-app_2_0.xsd" version="2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/portlet/portlet-app_2_0.xsd http://java.sun.com/xml/ns/portlet/portlet-app_2_0.xsd" id="com.sivavaka.test.springportlet.SivaTestSpringPortlet.58abb9cd14">

<portlet>

<portlet-name>SivaTestSpringPortlet</portlet-name>

<display-name xml:lang="en">SivaTestSpringPortlet</display-name>

<display-name>SivaTestSpringPortlet</display-name>

<portlet-class>org.springframework.web.portlet.DispatcherPortlet</portlet-class>

<init-param>

<name>wps.markup</name>

<value>html</value>

</init-param>

<init-param>

<name>contextConfigLocation</name>

<value>/WEB-INF/SivaTestSpringPortlet-Portlet.xml</value>

</init-param>

<expiration-cache>0</expiration-cache>

<supports>

<mime-type>text/html</mime-type>

<portlet-mode>view</portlet-mode>

<portlet-mode>edit</portlet-mode>

<portlet-mode>help</portlet-mode>

</supports>

<supported-locale>en</supported-locale>

<resource-bundle>com.sivavaka.test.springportlet.nl.SivaTestSpringPortletResource</resource-bundle>

<portlet-info>

<title>SivaTestSpringPortlet</title>

<short-title>SivaTestSpringPortlet</short-title>

<keywords>SivaTestSpringPortlet</keywords>

</portlet-info>

</portlet>

<default-namespace>http://SivaTestSpringPortlet/</default-namespace>

</portlet-app>

NOTE:  Spring portlet framework looks for [Portlet-Name]-Portlet.xml by default for if we don't specify "contextConfigLocation" in the initparam.

5. Add the SivaTestReport-Portlet.xml file with following

<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://www.springframework.org/schema/beans"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:p="http://www.springframework.org/schema/p"

xmlns:context="http://www.springframework.org/schema/context"

xsi:schemaLocation="

http://www.springframework.org/schema/beans

http://www.springframework.org/schema/beans/spring-beans-3.0.xsd

http://www.springframework.org/schema/context

http://www.springframework.org/schema/context/spring-context-3.0.xsd">

<context:annotation-config />

<!-- Controller -->

<context:component-scan base-package="com.sivavaka.test.springportlet"/>

</beans>

6. Add the SivaTestReport-Cotext.xml file with following

<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://www.springframework.org/schema/beans"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:p="http://www.springframework.org/schema/p"

xmlns:context="http://www.springframework.org/schema/context"

xsi:schemaLocation="

http://www.springframework.org/schema/beans

http://www.springframework.org/schema/beans/spring-beans-3.0.xsd

http://www.springframework.org/schema/context

http://www.springframework.org/schema/context/spring-context-3.0.xsd">

<context:annotation-config/>

<bean id="viewResolver"

class="org.springframework.web.servlet.view.InternalResourceViewResolver">

<property name="cache" value="false" />

<property name="viewClass" value="org.springframework.web.servlet.view.JstlView" />

<property name="prefix" value="/WEB-INF/jsp/" />

<property name="suffix" value=".jsp" />

</bean>

<bean id="messageSource" class="org.springframework.context.support.ResourceBundleMessageSource">

<property name="basename">

<value>com.sivavaka.test.springportlet.nl.SivaTestSpringPortletResource</value>

</property>

</bean>

</beans>

7. Write controller to handle the view requests

package com.sivavaka.test.springportlet.controller;

import org.springframework.stereotype.Controller;

import org.springframework.web.bind.annotation.RequestMapping;

@Controller

@RequestMapping("VIEW")

public class SivaTestSpringPortletController {

@RequestMapping

public String displayView(){

return "view";

}

}

8. write view jsp

<%@page

       language="java" contentType= "text/html; charset=ISO-8859-1"

       pageEncoding="ISO-8859-1" session="false"%>

<%@taglib uri="http://java.sun.com/portlet" prefix="portlet" %><portlet:defineObjects />

<p> View JSP.</p >

Sample code

1. Browse Code ::: https://code.google.com/p/blog-sivavaka-com-code-samples/source/browse/#svn%2Ftrunk%2FSivaTestSpringPortlet
2. Download War File ::: http://sourceforge.net/projects/blog-sivavaka-com-code-samples/files/Spring%20JSR%20Portlet/

WCM Syndication problem - Content doesn't syndicate even after creating syndication pair successfully - Network ports issue

At very first step of syndication is to check if both servers are able to communicate each other , So I just checked using "ping server1" from server2 and "ping server2" from server1 and got no pocket loss and thought both are communicating fine (Infrastructure team overlooked and confirmed that all ports are open , so didn't validated at port level initially) .

On Server1 (subscriber),

1. Click on Subsribers --> subscribe --> provided server2 (syndicator) url information and provided the credential vault information.
2. Then click next and able to select the libraries on the server2 (Syndicator) and choose the live items and created the syndication pair successfully.
3. But after that actual contents are not getting syndicated and I see following error on Syndicator

[9/13/13 10:44:20:576 EDT] 00000095 ItemChangedTa E   Operation timed out for ser2_to_ser1_synd. Could be due to an invalid URL address.

4. No error from subscriber.

Tried following on Syndicator

1. On Server2 (Syndicator), Clicked on rebuild causes error like above

[9/13/13 10:48:35:611 EDT] 00000065 RebuildSyndic E   Operation timed out for ser2_to_ser1_synd. Could be due to an invalid URL address.

2. No error on Subscriber.

Tried following on Subscriber

1. On Server1(Subscriber), Clicked on rebuild causes below error

[9/13/13 10:51:05:656 EDT] 00000054 RebuildSubscr E   Could not update subscriber with name 42179ad4-49eb-47ac-aee0-54014185f390

                                 com.aptrix.syndication.IceException: IWKWC0004I: Code #400, phrase: Generic request error

        at com.aptrix.syndication.business.AbstractRequestBL.checkResponse(AbstractRequestBL.java:91)

        at com.aptrix.syndication.business.subscriber.RequestBL.updateSubscription(RequestBL.java:348)

        at com.aptrix.deployment.subscriber.PlutoSubscriber.sendUpdateRequest(PlutoSubscriber.java:313)

        at com.ibm.workplace.wcm.app.ui.admin.subscriber.RebuildSubscriberCommand.doCommand(RebuildSubscriberCommand.java:160)

        at com.ibm.workplace.wcm.app.ui.portlet.command.AbstractCommand.commandPerformed(AbstractCommand.java:209)

        at com.ibm.psw.wcl.core.CommandHandler.handleCommand(CommandHandler.java:74)

        at com.ibm.psw.wcl.core.markup.WHyperlink$EHyperlinkCallbackWrapper.handleTrigger(WHyperlink.java:482)

  ....

        at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)

        at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)

        at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)

        at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1646)

This doesn't gave much of idea where is problem was , so we have enabled the traces to figure out the root cause

1. Added following trace string "com.aptrix.deployment.*=all:com.aptrix.syndication.*=all" on syndicator and subscriber servers.
2. On Server1(Subs) , clicked on rebuild subscriber causing I see the following information in trace.log

[9/12/13 13:02:38:089 EDT] 00000044 RequestBL     > RequestBL updateSubscription ENTRY

[9/12/13 13:02:38:090 EDT] 00000044 AbstractReque 3   sendRequest() - Sending request with url [http://syndserv2.sivavaka.com:10039/wps/wcm/connect/?MOD=Synd] and requestMsg com.aptrix.syndication.messaging.RequestMsg@2fb62fb6

[9/12/13 13:02:38:090 EDT] 00000044 AbstractReque 3   sendRequest() - Sending ice payload:

<ice-payload ice.version="1.0" timestamp="2013-09-12T17:02:38,90" payload-id="PL-2013-09-12T17:02:38,90-easwps07b-871323263"><ice-header><ice-user-agent>IBM Lotus Web Content Management Syndication v7.0.0.1 (bld wp7001CF11_001_23 (70x.CF28.0.1) 02/14/2012)</ice-user-agent><ice-sender sender-id="5eb10706-8368-4777-a561-0aacaf83f61f" role="subscriber" name="Serv2_To_Serv1_Subs"></ice-sender><ice-receiver receiver-id="43eb6a2e-07fb-4493-a861-8e6fe3e35d1e" name="Serv2_To_Serv1_Synd"></ice-receiver></ice-header><ice-request request-id="REQ-easwps07b-2013-09-12T17:02:38,90-871323264"><ice-get-package subscription-id="43eb6a2e-07fb-4493-a861-8e6fe3e35d1e" current-state="ICE-INITIAL"></ice-get-package></ice-request></ice-payload>

[9/12/13 13:02:38:090 EDT] 00000044 HttpMessaging 1   Generated payload to send:

<ice-payload ice.version="1.0" timestamp="2013-09-12T17:02:38,90" payload-id="PL-2013-09-12T17:02:38,90-easwps07b-871323263"><ice-header><ice-user-agent>IBM Lotus Web Content Management Syndication v7.0.0.1 (bld wp7001CF11_001_23 (70x.CF28.0.1) 02/14/2012)</ice-user-agent><ice-sender sender-id="5eb10706-8368-4777-a561-0aacaf83f61f" role="subscriber" name="Serv2_To_Serv1_Subs"></ice-sender><ice-receiver receiver-id="43eb6a2e-07fb-4493-a861-8e6fe3e35d1e" name="Serv2_To_Serv1_Synd"></ice-receiver></ice-header><ice-request request-id="REQ-easwps07b-2013-09-12T17:02:38,90-871323264"><ice-get-package subscription-id="43eb6a2e-07fb-4493-a861-8e6fe3e35d1e" current-state="ICE-INITIAL"></ice-get-package></ice-request></ice-payload>

[9/12/13 13:04:34:192 EDT] 00000044 HttpMessaging 1   Payload candidate recieved:

<ice-payload ice.version="1.0" timestamp="2013-09-12T17:02:38,145" payload-id="PL-2013-09-12T17:02:38,145-eatwps07a.nro.glic.com-1761471632"><ice-header><ice-user-agent>IBM Lotus Web Content Management Syndication v7.0.0.1 (bld wp7001CF11_001_23 (70x.CF28.0.1) 02/14/2012)</ice-user-agent><ice-sender sender-id="43eb6a2e-07fb-4493-a861-8e6fe3e35d1e" role="syndicator" name="Serv2_To_Serv1_Synd"></ice-sender><ice-receiver receiver-id="5eb10706-8368-4777-a561-0aacaf83f61f" name="Serv2_To_Serv1_Subs"></ice-receiver></ice-header><ice-response response-id="RSP-eatwps07a.nro.glic.com-2013-09-12T17:02:38,145-1761471633"><ice-code message-id="REQ-easwps07b-2013-09-12T17:02:38,90-8

71323264" phrase="Generic request error" numeric="400"></ice-code></ice-response></ice-payload>

3. From the above, what we understood Subscriber is able to communicate with syndicator (because we got response back) .

Solution in this case

1. We tried creating the syndication pair from the server2 (syndicator) to server1 (subscriber) just to make sure the everything fine , then we found that (server2)Syndicator is not able to communicate with subscriber(server1). (Server1 to server2 is fine but not reverse).

2. Gone back tried to validate the open ports between the servers (not just "ping") , we tried telnet to that portal default port (10039) and found that 10039 not open . This is basic thing we should have tried this to make sure ports are open communicating over those ports are fine.

Failed communicate when we tried "telnet subsserv1.sivavaka.com 10039" from syndicator server 

NOTE: "telnet syndserv2.sivavaka.com 10039" from subscriber server is fine.

WCM Syndication Problems - Unable to delete the syndication pairs

Recently when I was setting up the syndication on the brand new environment faced couple of issues

1. Was able to create the syndication pair but when I tried to delete it is failed to do with following error.

[9/7/13 23:30:08:071 EDT] 0000004f ProcessLinks  E com.ibm.icm.da.portable.data.ProcessLinks collectiveBatchRemoveLink com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 S

QL Error: SQLCODE=-286, SQLSTATE=42727, SQLERRMC=4096;PORTALUSER, DRIVER=4.11.77

                                 com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-286, SQLSTATE=42727, SQLERRMC=4096;PORTALUSER, DRIVER=4.11.77

        at com.ibm.db2.jcc.am.gd.a(gd.java:676)

        at com.ibm.db2.jcc.am.gd.a(gd.java:60)

      ……

[9/7/13 23:30:08:098 EDT] 0000004f DeleteAdminOb E   An error occurred while deleting item

                                 com.ibm.workplace.wcm.app.ui.admin.core.model.AdminModelException: com.ibm.workplace.wcm.services.repository.RepositoryException: Messa

ge: Could not save object in repository., Cause: javax.jcr.RepositoryException: Path Add/Rename/Delete operation failed; DB2 SQL Error: SQLCODE=-286, SQLSTATE=42727, SQ

LERRMC=4096;PORTALUSER, DRIVER=4.11.77Thread 1509186036 : qlSyntaxErrorException: DB2 SQL Error: SQLCODE=-286, SQLSTATE=42727, SQLERRMC=4096;PORTALUSER, DRIVER=4.11.77

        at com.ibm.db2.jcc.am.gd.a(gd.java:676)

        at com.ibm.db2.jcc.am.gd.a(gd.java:60)

        at com.ibm.db2.jcc.am.gd.a(gd.java:127)

   ....

        at com.ibm.db2.jcc.am.jn.e(jn.java:1036)

        at com.ibm.db2.jcc.am.jn.execute(jn.java:1020)

        at com.ibm.ws.rsadapter.jdbc.WSJdbcStatement.pmiExecute(WSJdbcStatement.java:1559)

        at com.ibm.ws.rsadapter.jdbc.WSJdbcStatement.execute(WSJdbcStatement.java:706)

        at com.ibm.icm.da.portable.common.sql.PStatement.execute(PStatement.java:161)

        at com.ibm.icm.da.portable.data.ProcessLinks.createOrCleanupTemporaryTable(ProcessLinks.java:742)

        at com.ibm.icm.da.portable.data.ProcessLinks.buildTemporaryDeleteTable(ProcessLinks.java:887)

        at com.ibm.icm.da.portable.data.ProcessLinks.collectiveBatchRemoveLink(ProcessLinks.java:611)

        at com.ibm.icm.da.portable.data.ProcessLinks.processLinks(ProcessLinks.java:1128)

      ....

        at com.ibm.icm.jcr.WorkspaceImpl.save(WorkspaceImpl.java:2287)

        at com.ibm.workplace.wcm.services.transaction.steps.repository.SaveWorkspace.execute(SaveWorkspace.java:96)

        at com.ibm.workplace.wcm.services.transaction.CompositeTransactionStep.execute(CompositeTransactionStep.java:88)

        at com.ibm.workplace.wcm.services.transaction.TxUnitOfWork$1.call(TxUnitOfWork.java:96)

        at com.ibm.workplace.wcm.util.transaction.TransactionManager.doCall(TransactionManager.java:187)

        at com.ibm.workplace.wcm.util.transaction.TransactionManager.call(TransactionManager.java:73)

        at com.ibm.workplace.wcm.app.ui.admin.subscriber.MainSubscriberModel.deleteItem(MainSubscriberModel.java:375)

        at com.ibm.workplace.wcm.app.ui.admin.core.command.DeleteAdminObjectCommand.doCommand(DeleteAdminObjectCommand.java:130)

        at com.ibm.workplace.wcm.app.ui.portlet.command.AbstractCommand.commandPerformed(AbstractCommand.java:209)

        at com.ibm.psw.wcl.core.CommandHandler.handleCommand(CommandHandler.java:74)

        at com.ibm.psw.wcl.core.markup.WHyperlink$EHyperlinkCallbackWrapper.handleTrigger(WHyperlink.java:482)

        at com.ibm.psw.wcl.core.trigger.Trigger.process(Trigger.java:304)

        at com.ibm.psw.wcl.core.trigger.TriggerManager.processTrigger(TriggerManager.java:781)

        at com.ibm.psw.wcl.portlet.WclPortletTriggerManager.handleRequest(WclPortletTriggerManager.java:260)

        at com.ibm.psw.wcl.portlet.WclPortletFacade.handleRequest(WclPortletFacade.java:497)

        at com.ibm.psw.wcl.portlet.WclPortletFacade.handleRequest(WclPortletFacade.java:478)

        at com.ibm.workplace.wcm.app.ui.admin.AdminPortlet.handleActionPerformed(AdminPortlet.java:297)

        at com.ibm.workplace.wcm.app.ui.admin.AdminPortlet.actionPerformed(AdminPortlet.java:241)

        at com.ibm.wps.pe.pc.legacy.SPIPortletInterceptorImpl.handleEvents(SPIPortletInterceptorImpl.java:337)

        at com.ibm.wps.pe.pc.legacy.invoker.impl.PortletDelegateImpl._dispatch(PortletDelegateImpl.java:385)

       ....

Caused by: com.ibm.workplace.wcm.services.repository.RepositoryException: Message: Could not save object in repository., Cause: javax.jcr.RepositoryException: Path Add/

Rename/Delete operation failed; DB2 SQL Error: SQLCODE=-286, SQLSTATE=42727, SQLERRMC=4096;PORTALUSER, DRIVER=4.11.77Thread 1509186036 : qlSyntaxErrorException: DB2 SQL E

rror: SQLCODE=-286, SQLSTATE=42727, SQLERRMC=4096;PORTALUSER, DRIVER=4.11.77

        at com.ibm.db2.jcc.am.gd.a(gd.java:676)

         ....

        at com.ibm.icm.da.portable.data.ProcessLinks.createOrCleanupTemporaryTable(ProcessLinks.java:742)

        at com.ibm.icm.da.portable.data.ProcessLinks.buildTemporaryDeleteTable(ProcessLinks.java:887)

         ....

        at com.ibm.icm.jcr.WorkspaceImpl.save(WorkspaceImpl.java:2287)

        at com.ibm.workplace.wcm.services.transaction.steps.repository.SaveWorkspace.execute(SaveWorkspace.java:96)

        at com.ibm.workplace.wcm.services.transaction.CompositeTransactionStep.execute(CompositeTransactionStep.java:88)

        at com.ibm.workplace.wcm.services.transaction.TxUnitOfWork$1.call(TxUnitOfWork.java:96)

          ....

        at com.ibm.workplace.wcm.services.content.ContentServiceImpl.remove(ContentServiceImpl.java:1628)

        at com.ibm.workplace.wcm.services.content.ContentService

Caused by: javax.jcr.RepositoryException: Path Add/Rename/Delete operation failed; DB2 SQL Error: SQLCODE=-286, SQLSTATE=42727, SQLERRMC=4096;PORTALUSER, DRIVER=4.11.77Th

read 1509186036 : qlSyntaxErrorException: DB2 SQL Error: SQLCODE=-286, SQLSTATE=42727, SQLERRMC=4096;PORTALUSER, DRIVER=4.11.77

        at com.ibm.db2.jcc.am.gd.a(gd.java:676)

        ....

        at com.ibm.icm.da.portable.common.sql.PStatement.execute(PStatement.java:161)

        at com.ibm.icm.da.portable.data.ProcessLinks.createOrCleanupTemporaryTable(ProcessLinks.java:742)

        at com.ibm.icm.da.portable.data.ProcessLinks.buildTemporaryDeleteTable(ProcessLinks.java:887)

        ....

        at com.ibm.icm.jcr.WorkspaceImpl.save(WorkspaceImpl.java:2287)

        at com.ibm.workplace.wcm.services.transaction.steps.repository.SaveWorkspace.execute(SaveWorkspace.java:96)

        at com.ibm.workplace.wcm.services.transaction.CompositeTransactionStep.execute(CompositeTransactionStep.java:88)

        at com.ibm.workplace.wcm.services.transaction.TxUnitOfWork$1.call(TxUnitOfWork.java:96)

        at com.ibm.workplace.wcm.util.transaction.TransactionManager.doCall(TransactionManager.java:187)

        at com.ibm.workplace.wcm.util.transaction.TransactionManager.call(TransactionManager.java:73)

        at com.ibm.workplace.wcm.services.transaction.TxUnitOfWork.run(TxUnitOfWork.java:138)

        at com.ibm.workplace.wcm.services.content.ContentServiceImpl.remove(ContentServiceImpl.java:1628)

        at com.ibm.workplace.wcm.services.content.ContentServiceImpl.remove(ContentServiceImpl.java:1537)

        at com.ibm.workplace.wcm.app.ui.admin.subscriber.MainSubscriberModel.deleteItem(MainSubscriberModel.java:353)

        at com.ibm.workplace.wcm.app.ui.admin.core.command.DeleteAdminObjectCommand.doCommand(DeleteAdminObjectCommand.java:130)

Caused by: com.ibm.icm.ci.data.CDMContentException: Path Add/Rename/Delete operation failed; DB2 SQL Error: SQLCODE=-286, SQLSTATE=42727, SQLERRMC=4096;PORTALUSER, DRIVER

=4.11.77Thread 1509186036 : qlSyntaxErrorException: DB2 SQL Error: SQLCODE=-286, SQLSTATE=42727, SQLERRMC=4096;PORTALUSER, DRIVER=4.11.77

        at com.ibm.db2.jcc.am.gd.a(gd.java:676)

    ...

        at com.ibm.ws.rsadapter.jdbc.WSJdbcStatement.pmiExecute(WSJdbcStatement.java:1559)

        at com.ibm.ws.rsadapter.jdbc.WSJdbcStatement.execute(WSJdbcStatement.java:706)

        at com.ibm.icm.da.portable.common.sql.PStatement.execute(PStatement.java:161)

        at com.ibm.icm.da.portable.data.ProcessLinks.createOrCleanupTemporaryTable(ProcessLinks.java:742)

        at com.ibm.icm.da.portable.data.ProcessLinks.buildTemporaryDeleteTable(ProcessLinks.java:887)

     .....

        at com.ibm.icm.jcr.WorkspaceImpl.save(WorkspaceImpl.java:2287)

        at com.ibm.workplace.wcm.services.transaction.steps.repository.SaveWorkspace.execute(SaveWorkspace.java:96)

        at com.ibm.workplace.wcm.services.transaction.CompositeTransactionStep.execute(CompositeTransactionStep.java:88)

        at com.ibm.workplace.wcm.services.transaction.TxUnitOfWork$1.call(TxUnitOfWork.java:96)

       ...

        at com.ibm.workplace.wcm.app.ui.admin.subscriber.MainSubscriberModel.deleteItem(MainSubscriberModel.java:353)

        at com.ibm.workplace.wcm.app.ui.admin.core.command.DeleteAdminObjectCommand.doCommand(DeleteAdminObjectCommand.java:130)

Caused by: com.ibm.icm.da.DBAccessException: DB2 SQL Error: SQLCODE=-286, SQLSTATE=42727, SQLERRMC=4096;PORTALUSER, DRIVER=4.11.77Thread 1509186036 : qlSyntaxErrorExcepti

on: DB2 SQL Error: SQLCODE=-286, SQLSTATE=42727, SQLERRMC=4096;PORTALUSER, DRIVER=4.11.77

        at com.ibm.db2.jcc.am.gd.a(gd.java:676)

         ........

        at com.ibm.icm.da.portable.data.ProcessLinks.createOrCleanupTemporaryTable(ProcessLinks.java:742)

        at com.ibm.icm.da.portable.data.ProcessLinks.buildTemporaryDeleteTable(ProcessLinks.java:887)

        at com.ibm.icm.da.portable.data.ProcessLinks.collectiveBatchRemoveLink(ProcessLinks.java:611)

        ......

        at com.ibm.workplace.wcm.services.transaction.TxUnitOfWork.run(TxUnitOfWork.java:138)

        at com.ibm.workplace.wcm.services.content.ContentServiceImpl.remove(ContentServiceImpl.java:1628)

        at com.ibm.workplace.wcm.services.content.ContentServiceImpl.remove(ContentServiceImpl.java:1537)

        at com.ibm.workplace.wcm.app.ui.admin.subscriber.MainSubscriberModel.deleteItem(MainSubscriberModel.java:353)

        at com.ibm.workplace.wcm.app.ui.admin.core.command.DeleteAdminObjectCommand.doCommand(DeleteAdminObjectCommand.java:130)

Caused by: com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-286, SQLSTATE=42727, SQLERRMC=4096;PORTALUSER, DRIVER=4.11.77

        at com.ibm.db2.jcc.am.gd.a(gd.java:676)

        ...

        at com.ibm.icm.da.portable.data.ProcessLinks.createOrCleanupTemporaryTable(ProcessLinks.java:742)

        at com.ibm.icm.da.portable.data.ProcessLinks.buildTemporaryDeleteTable(ProcessLinks.java:887)

        at com.ibm.icm.da.portable.data.ProcessLinks.collectiveBatchRemoveLink(ProcessLinks.java:611)

         ... 109 more

Solution:

1. After doing some research found this is caused by "ICMLSUSRTSPACE4" tablespace issue, usually this tablespace is missed during the cluster creation and this should cause problem.

db2 "CREATE USER TEMPORARY TABLESPACE ICMLSUSRTSPACE4 PAGESIZE 4 K MANAGED BY SYSTEM USING ('icmlsusrtspace4') BUFFERPOOL ICMLSVOLATILEBP4"

2. But in our scenario , it turns out that we have this tablespace defined already but it is not given proper privileges to portal user and caused above exception.
3. Executed the following at DB2 to overcome above problem

GRANT USE OF TABLESPACE "ICMLSUSRTSPACE4" TO USER "PORTALUSER"

DB20000I The SQL command completed successfully. 

References:

http://www-10.lotus.com/ldd/portalwiki.nsf/dx/Step-by-Step_Cluster_Guide_for_IBM_WebSphere_Portal_v7.0.0