Batch Update with JPA

A customer of mine has a highly scalable system, with high database load. To improve performance, we’ve decided to use Batch update. Alas – the application uses JPA, so how do we do it?

SessionFactoryImplementor sfi = SessionFactoryImplementor)entityManager.unwrap(Session.class).getSessionFactory();
//Retrieve DB connection
connect = sfi.getConnectionProvider().getConnection();
PreparedStatement ps = connect.prepareStatement("INSERT INTO temp_table values(?,?)");
for (Data p : insertCommands) {
	ps.setInt(1, p.getId());
	if (p.isWorking() != null) {
		ps.setInt(2, p.isWorking() ? 1 : 0);
	} else {
		ps.setNull(2, Types.INTEGER);
	}
	ps.addBatch();
}
ps.executeBatch();


Some points to remember:

  1. Do not close the connection!!! When closing the connection, it returns to the connection pool, without commit or rollback. You’ll get some exceptions in your code, but mainly – you’re open to database locks and dead-locks, as the connection keeps all the locks that were done in the batch update. And since other threads will reuse the connection – it will never close.
  2. Do not use Statement.addBatch. Use PreparedStatement.addBatch. When I checked both solutions, the Statement one took anywhere between 30s-60s. The PreparedStatement version took <300ms...

How to reuse a java.lang.StringBuilder

It’s common knowledge in Java that you want to clear a StringBuilder object, and not recreate it. Why? GC for instance. But, surprisingly enough, there is no clear() method on the StringBuilder class, and so we resort to manual clear methods. The ones I’ve seen are:

  • new StringBuilder which basically recreates the object
  • setLength(0) which changes the internal flags of the StringBuilder
  • delete(0,sb.length()) which deletes the whole buffer

One would think that the best technique is setLength – but I’ve decided to give it a short test.
Here is the code I’ve used:


package com.tona;

import java.util.Random;

public class TestStringBuffer {
private static final int NUM_OF_RUNS = 100000;
private static final int MAX_LENGTH = 1000;
private static Random rand = new Random();

private static int fillString(StringBuilder sb,boolean isSameLenString) {

int length;

if (isSameLenString)
length = MAX_LENGTH;
else
length = rand.nextInt(MAX_LENGTH);

for (int i = 0; i < length; ++i) sb.append("a"); return length; } private void runBenchmark(boolean isSameLenString) { StringBuilder sb = new StringBuilder(); long start = System.currentTimeMillis(); for (int i = 0; i < NUM_OF_RUNS; ++i) { fillString(sb,isSameLenString); sb.setLength(0); } long end = System.currentTimeMillis(); long diffWithLen0 = end - start; start = System.currentTimeMillis(); for (int i = 0; i < NUM_OF_RUNS; ++i) { fillString(sb,isSameLenString); sb = new StringBuilder(); } end = System.currentTimeMillis(); long diffWithNew = end - start; start = System.currentTimeMillis(); for (int i = 0; i < NUM_OF_RUNS; ++i) { fillString(sb,isSameLenString); sb = new StringBuilder(MAX_LENGTH); } end = System.currentTimeMillis(); long diffWithNewConstLength = end - start; start = System.currentTimeMillis(); for (int i = 0; i < NUM_OF_RUNS; ++i) { fillString(sb,isSameLenString); sb.delete(0, sb.length()); } end = System.currentTimeMillis(); long diffWithDelete = end - start; start = System.currentTimeMillis(); for (int i = 0; i < NUM_OF_RUNS; ++i) { int length = fillString(sb,isSameLenString); sb.delete(0, length); } end = System.currentTimeMillis(); long diffWithDeleteConstLength = end - start; System.out.println("With setLength(0) " + diffWithLen0); System.out.println("With new StringBuilder() " + diffWithNew); System.out.println("With new StringBuilder(MAX_LENGTH) " + diffWithNewConstLength); System.out.println("With delete(0, sb.length()) " + diffWithDelete); System.out.println("With delete(0, length) " + diffWithDeleteConstLength); } public static void main(String[] args) { TestStringBuffer test = new TestStringBuffer(); System.out.println("Constant length string"); test.runBenchmark(true); System.out.println("Changing length string"); test.runBenchmark(false); } } [/java]

And here are the results:

Constant length string
With setLength(0) 1524
With new StringBuilder() 1501
With new StringBuilder(MAX_LENGTH) 1365
With delete(0, sb.length()) 1369
With delete(0, length) 1391
Changing length string
With setLength(0) 686
With new StringBuilder() 743
With new StringBuilder(MAX_LENGTH) 796
With delete(0, sb.length()) 715
With delete(0, length) 698

(Note that changing string length uses a Random, so results may vary).
With StringBuilder object with a fixed length, the new StringBuilder with the predefined length is the best option, although not much slower than the delete method. With the varying length strings, setLength and delete and more of less on par (give it a few runs and you'll see it for yourself).
So, now you know how to clean your StringBuilder object. Enjoy.

Improving authorization performance in Liferay

I have recently run a benchmark on a Liferay 6.1 portal. Most of the bottlenecks were the result of the portlets themselves, but a major point was the authorization mechanism of Liferay.
The way Liferay authorization works is that for every component on the screen an authorization check is made. Now, this makes sense. But the problem is that the default implementation is a bit naive – it goes to the database to check if the user has the correct authorization. And since rights can be inherited – each check goes multiple times to the database.
So, I set out to solve this irritating issue by using a cache. Now – this cache is naive too – if a user was granted a new role – a server restart is needed. But this implementation is easy to extend.

  1. Create a new class, that extends com.liferay.portal.security.permission.AdvancedPermissionChecker
  2. Open the portal-ext.properties file.
  3. Add the property permissions.checker={your class name} to it
  4. The class code should look like this:
    package com.tona.liferay.permission;
    
    import java.util.HashMap;
    import java.util.Map;
    
    public class TonaPermissionChecker extends
    		com.liferay.portal.security.permission.AdvancedPermissionChecker {
    
    	private static Map<PermissionKey, Boolean> permissionCache = new HashMap<PermissionKey, Boolean>();
    
    	public boolean hasPermission(long groupId, String name, String primKey,
    			String actionId) {
    
    		PermissionKey key = new PermissionKey(groupId, name, primKey, actionId);
    		if (permissionCache.containsKey(key)) {
    			return permissionCache.get(key);
    		}
    
    		boolean result = super.hasPermission(groupId, name, primKey, actionId);
    		permissionCache.put(key, result);
    
    		return result;
    
    	}
    
    	private class PermissionKey {
    		private long groupId;
    		private String name;
    		private String primKey;
    		private String actionId;
    
    		public PermissionKey(long groupId, String name, String primKey,
    				String actionId) {
    			super();
    			this.groupId = groupId;
    			this.name = name;
    			this.primKey = primKey;
    			this.actionId = actionId;
    		}
    
    		public long getGroupId() {
    			return groupId;
    		}
    
    		public void setGroupId(long groupId) {
    			this.groupId = groupId;
    		}
    
    		public String getName() {
    			return name;
    		}
    
    		public void setName(String name) {
    			this.name = name;
    		}
    
    		public String getPrimKey() {
    			return primKey;
    		}
    
    		public void setPrimKey(String primKey) {
    			this.primKey = primKey;
    		}
    
    		public String getActionId() {
    			return actionId;
    		}
    
    		public void setActionId(String actionId) {
    			this.actionId = actionId;
    		}
    
    		@Override
    		public int hashCode() {
    			final int prime = 31;
    			int result = 1;
    			result = prime * result
    					+ ((actionId == null) ? 0 : actionId.hashCode());
    			result = prime * result + (int) (groupId ^ (groupId >>> 32));
    			result = prime * result + ((name == null) ? 0 : name.hashCode());
    			result = prime * result
    					+ ((primKey == null) ? 0 : primKey.hashCode());
    			return result;
    		}
    
    		@Override
    		public boolean equals(Object obj) {
    			if (this == obj)
    				return true;
    			if (obj == null)
    				return false;
    			if (getClass() != obj.getClass())
    				return false;
    			PermissionKey other = (PermissionKey) obj;
    			if (actionId == null) {
    				if (other.actionId != null)
    					return false;
    			} else if (!actionId.equals(other.actionId))
    				return false;
    			if (groupId != other.groupId)
    				return false;
    			if (name == null) {
    				if (other.name != null)
    					return false;
    			} else if (!name.equals(other.name))
    				return false;
    			if (primKey == null) {
    				if (other.primKey != null)
    					return false;
    			} else if (!primKey.equals(other.primKey))
    				return false;
    			return true;
    		}
    	}
    }
    
  5. Package the class in a JAR file, and put in TOMCAT_HOME/webapps/ROOT/WEB-INF/lib

Tunneling with JProfiler

I had some issues running JProfiler from a remote machine. I had a JBoss running on a remote Linux server, and for some reason, XWindows just didn’t work. Turns out I had to tunnel all my JProfiler connection, and luckily – that’s easy to do.
I typed a command on my own laptop (running Linux Mint)

ssh -f root@XXX.XXX.XXX.XXX -L 2000:localhost:8849 -N

And now, all I have to do is open a connection from JProfiler. I use attach to remote process, select localhost, port 2000, and that’s it – I can now profile the remote server.

FileDescriptor.sync()

First and foremost – a week ago, I never even knew this method existed in Java. Basically – it let you force the file writing to the disk. Turns out Arjuna (JBoss transactions) is using it in its ShadowStore class, to ensure transaction data is stored to disk. It makes sense – as they want to recover transactions in case of a server crash.
Now, if you read my last post, on the inflation of EJBs, you know that 200 EJBs working together is a mess. And I’ve reached a point where 15% of CPU time of a single transaction is spent on this FileDescriptor.sync() method. Since I couldn’t refactor the whole code – I had to think of another solution. Here goes.

I’ve written a new class, that extends ShadowStore.

public class TonaStore extends ShadowingStore {
    public TonaStore(ObjectStoreEnvironmentBean objectStoreEnvironmentBean) throws ObjectStoreException
    {
        super(objectStoreEnvironmentBean);
    	syncWrites = false;
    }
}

I deployed it to a JAR file, and placed it in the server/all/lib directory.

Now, I opened the /server/all/deploy/transactions-jboss-beans.xml file, and changed ActionStore section to the following:

    <bean name="ActionStoreObjectStoreEnvironmentBean" class="com.arjuna.ats.arjuna.common.ObjectStoreEnvironmentBean">

        <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss.jta:name=ActionStoreObjectStoreEnvironmentBean", exposedInterface=com.arjuna.ats.arjuna.common.ObjectStoreEnvironmentBeanMBean.class, registerDirectly=true)</annotation>
        
        <constructor factoryClass="com.arjuna.common.internal.util.propertyservice.BeanPopulator" factoryMethod="getNamedInstance">
            <parameter>com.arjuna.ats.arjuna.common.ObjectStoreEnvironmentBean</parameter>
            <parameter>default</parameter>
        </constructor>
        <property name="objectStoreDir">${jboss.server.data.dir}/tx-object-store</property>
	<property name="objectStoreType">com.tona.ts.common.TonaStore</property>
    </bean>

I got almost a 100% increase in hits/second. Sweet.

EJB inflation

Ever since the JavaEE standard introduced the Local interfaces (and especially in EJB3), I see people abusing EJBs. The logic is simple – if EJB calls are local, let’s use EJBs, and enjoy dependency injection.
Recently I assisted a customer who had over 200 EJBs in a project that had about 500 classes! I call it EJB inflation, and it’s bad. Really bad.
The reason – the EJB container does more than just proxy remote calls. It handles security, transactions, pooling and more. Using those, for every class, is paying a huge price in performance. Let’s just say that when I run a profiler on the customer code I saw that over 20% of the server time is wasted on the application server EJB related code (JTA, specifically).
I will post workarounds for this in future posts, but in the mean time – beware abusing EJBs. Don’t fall to the “if you have a hammer, everything looks like a nail” trap.

Errors while performing remote deployment from Eclipse to WebLogic Server

Using local WebLogic server with Eclipse is a piece of cake, especially when using OEPE (the Oracle Eclipse flavor, with all the relevant plugins installed). However, I run into issues while deploying an application to a remote WebLogic server, running on Linux (the machine I used run Windows).
The error was:

weblogic.deploy.api.spi.exceptions.ServerConnectionException: [J2EE Deployment SPI:260041]Unable to upload 'C:\workspaces\labWorkspaceNew\.metadata\.plugins\org.eclipse.core.resources\.projects\HelloWorld\beadep\remote_weblogic\HelloWorld.war' to 't3://XXX.XXX.XXX.XXX:7041'

java.lang.Exception: Exception received from deployment driver. See Error Log view for more detail.
at oracle.eclipse.tools.weblogic.server.internal.WlsJ2EEDeploymentHelper$DeploymentProgressListener.watch(WlsJ2EEDeploymentHelper.java:1566)
at oracle.eclipse.tools.weblogic.server.internal.WlsJ2EEDeploymentHelper.deploy(WlsJ2EEDeploymentHelper.java:470)
at oracle.eclipse.tools.weblogic.server.internal.WeblogicServerBehaviour.publishWeblogicModules(WeblogicServerBehaviour.java:1346)
at oracle.eclipse.tools.weblogic.server.internal.WeblogicServerBehaviour.publishToServer(WeblogicServerBehaviour.java:803)
at oracle.eclipse.tools.weblogic.server.internal.WeblogicServerBehaviour.publishOnce(WeblogicServerBehaviour.java:623)
at oracle.eclipse.tools.weblogic.server.internal.WeblogicServerBehaviour.publish(WeblogicServerBehaviour.java:516)
at org.eclipse.wst.server.core.model.ServerBehaviourDelegate.publish(ServerBehaviourDelegate.java:708)
at org.eclipse.wst.server.core.internal.Server.publishImpl(Server.java:2690)
at org.eclipse.wst.server.core.internal.Server$PublishJob.run(Server.java:272)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
Caused by: weblogic.deploy.api.internal.utils.DeployerHelperException: The source 'C:\DOCUME~1\train\LOCALS~1\Temp\1\HelloWorld.war' for the application 'HelloWorld' could not be loaded to the server 'http://XXX.XXX.XXX.XXX:7041/bea_wls_deployment_internal/DeploymentService'.
Server returned HTTP response code: 403 for URL: http://XXX.XXX.XXX.XXX:7041/bea_wls_deployment_internal/DeploymentService
at weblogic.deploy.api.internal.utils.JMXDeployerHelper.uploadSource(JMXDeployerHelper.java:659)
at weblogic.deploy.api.spi.deploy.internal.ServerConnectionImpl.upload(ServerConnectionImpl.java:843)
at weblogic.deploy.api.spi.deploy.internal.BasicOperation.uploadFiles(BasicOperation.java:321)
at weblogic.deploy.api.spi.deploy.internal.BasicOperation.execute(BasicOperation.java:410)
at weblogic.deploy.api.spi.deploy.internal.BasicOperation.run(BasicOperation.java:170)
at weblogic.deploy.api.spi.deploy.WebLogicDeploymentManagerImpl.deploy(WebLogicDeploymentManagerImpl.java:364)
at oracle.eclipse.tools.weblogic.server.internal.WlsJ2EEDeploymentHelper.deploy(WlsJ2EEDeploymentHelper.java:468)
... 8 more

At first I thought this was caused by the size of the EAR file, but after creating a small HelloWorld.war and getting the same error, I traced it rather quickly.
The problem is network related. First, make sure your browser can connect to the admin console of the remote WebLogic Server (http://XXX.XXX.XXX.XXX:7001/console). If it doesn’t work – configure the proxy, or make the sys-admin open the 7001 port.
If it does work, ensure your eclipse has the proxy configured – using Window/Preferences/General/Network Connections

Configuration based ESB

I think that anyone who read my previous posts can see that I work allot with ESBs – the blog posts were mostly dedicated to IBM, but I work with OSB, JBossESB and some other solutions as well.

I am amazed at just how much work customers invest in ESB development – they build an architecture, build a development team – and usually, 99% of the work they do – is the same across all organizations.
Because everybody are using ESB in the same way – to enforce web services policies – usually security, but also validation, version management, monitoring and a bit more.
So – what do all these off-the-shelf products give us? Not enough.
I believe that a new generation of ESB is in order – one that will not require coding. The ESB should contain a single point of entry for all services in the organization, all will follow a configured validation and enrichment path, and at the end will be directed to the actual service. No coding, no development team – just simple configuration. Should definitely suffice most customers.
I have a few ideas on an architecture of such a thing, and Aluna even has a small product based on open source technologies, that runs with both .net and Java clients.
But that is another story.

WebServices Reliable Messaging

A customer of mine asked me to build a POC of using WS-RM with CXF, C# and Oracle Service Bus. Against my better judgement – I said yes. I know it doesn’t work – but I’m always up to the challenge.

Turns out documentation is shitty at best. Vendors provide a one-page demo, using messaging infra or embedded HTTP servers – no web containers or web servers.
So – I have decided to start with .net and CXF and leave the OSB for later.
The pain is installing VS2010 Beta 2 over my machine which had VS2010 Beta 1 – a version that was deleted a couple of month ago.
You can’t install VS2010B2 on a B1 computer – you must do a reinstall of B1, then uninstall it, then install B2.
Bloody nightmare.

WebSphere 7 – deployment performance

Well, I’m currently running a migration project from Oracle’s OC4J to WebSphere 7. Why 7? Because the customer wants to use JSF 1.2, and EJB 3 – and we decided on going to WAS7 instead of WAS6.1 with the fix packs.

The migration wasn’t so smooth – the customer used some proprietary code we had to change. And let me tell you – configuring TopLink to run on WAS is a real pain, especially when some TopLink code is accessed through EJBs and some through regular Java classes from the web tier.
But the real problem was WAS7 deployment. It took forever… We tried running it from RAD7.5 or from the admin console directly – it was painful. It took around 5 minutes, and every configuration action that required application restart took ages to complete.
Turns out the problem lies in the JavaEE5 spec. When you deploy an EAR it needs to look for annotated classes that are marked as EJB and servlets. So deploying a WAR file with a large number of classes and JAR files will be very slow.
There are deployment separation solutions (putting the classes in the app-inf/lib of the EAR instead of web-inf/lib of the WAR) but a quick fix, that fits when you don’t use annotated servlets is to add the metadata-complete attribute to your web.xml file (in the web-app tag). It tells the deployer that all of your servlets and filters are configured inside the web.xml file.
See this link for the complete web_app_2_5 XSD description. http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd