I'm in need of some thoughts on CFLOCK. There's backstory to this, but to sum it up I'm fighting an uphill battle with hardware folks that think our using CFLOCK is causing server performance issues on a new Unix server that was built to the absolute minimum specs (not exaggerating). The server has been failing load tests left and right and their approaches have been to claim it's the load testing application's fault and to force us to dumb down the load testing site they are using for testing to the point where we are about three steps away from just putting up a static page there.
In our applications, we use CFLOCK as follows:
1. We only use exclusive locks when setting application and session scoped variables. 2. All application-scoped variables are set only in onApplicationStart(). This includes instatiating CFC's into the application scope to reduce overhead on requests by not having to load and unload the objects on a per-request basis. 3. We lock the setting of the VARIABLES scope inside the init() functions of our CFC's (the only place we set the variables scope in CFC's as well). 4. We set our locks to 10 second timeouts, and all timeouts throw an error because there is no good reason to ignore these errors. 5. We do not, as a general rule, use readonly locks of any kind.
It is the opinion of the hardware people (who have rudimentary CF skills) that we need to at least decrease the timeouts to 2-3 seconds, or get rid of the locks altogether. My gut reply to that is that if the server cannot establish a lock for a given thread in 10 seconds, how would setting it to 2-3 seconds help except to cause the site to throw more errors? As for the second part, the fact they think server importance is more critical than application security says all I need to know IMO.
But I will fully admit to being biased on this, and could use some backup (or corrections) on if our standards are acceptable industry practices, which I believe are.
And no, getting them to add more resources to the server is not an option here. :(
#1 by Steve on 7/11/12 - 1:16 PM
#2 by Marc Esher on 7/11/12 - 2:24 PM
#3 by Rob Barthle on 7/11/12 - 2:32 PM
#4 by Marc Esher on 7/11/12 - 3:18 PM
First, let's assume that only developers or sysadmins will ever be doing your URL-based reloading. This means that you'd only have it being done at one time. So there's no risk from it from an object creation perspective.
The risk here is what happens if you reinit the app, and at the same time a user is doing something which is talking to one of these app-scoped components, and that component goes "poof" or whatever. So the user gets an error. You either mitigate that by taking the site down for a maintenance window, or if that's not possible, and you're in an environment which lets you do mid-day app reinitialization, then your best bet will be to get rid of all those CFLOCKs you have in your components, and put a single one in your onApplicationStart() method, wrapping up all your app-scoped object creation.
If you do this, it's *critical* that your object creations are wicked fast. This is generally not a problem unless your constructors *do stuff*. I've seen components that, when initialized, run a whole mess of queries and other gnarly things. If you have that kind of stuff, you might run into trouble. If not, then this approach should work.
#5 by Dan G. Switzer, II on 7/11/12 - 3:39 PM
You could just make a copy of your codebase and use a regex to remove all the locks altogether--which might satisfy the hardware guys more.
#6 by Rob Barthle on 7/11/12 - 4:23 PM
I use named locks 100% of the time when I code, but there are instances of legacy code where scope locks were used, and other developers use scope locks on occasion (we have not nailed down locking in our developers guidelines yet).
I am trying to think about how the application in question works, and for each new session a block of code is called in onSessionStart() that sets session variables, and also determines if that user is going to get a random survey at some point in the application. It's not extensive coding, but in a load test environment, where they are ramping up users very fast in the beginning, would the locking of the setting of session variables cause a bottleneck if you locked it with a name versus locking with a scope? Because I see sessions as their own little entities and the actions in one session shouldn't affect the actions of others in terms of acquiring locks, right?
I'm limited in what I can do codewise (in no small part because there is a big time political battle going here regarding this by upper mgmt, and offering up code changes is seen as giving in) but I want to be as informed as possible when asked questions. And my time to set up my own load tests on my own server is limited.
#7 by Tariq Ahmed on 7/11/12 - 4:59 PM
At one project we took over a legacy system that was suffering from extreme performance issues.
What I did was to install SeeFusion (Fusion Reactor is also a good option), in order to catalog the slowest running pages & queries which allowed me to identify where the bottlenecks were. And overtime we got the avg page speed down from 90s to 0.2s.
Before you begin that - it sounds like there's finger pointing going on. What's important is to get everyone together, working together, towards a common goal - e.g. a high performance website. You'll want to get some sort of upper management support as well to back this.
So:
- Problem Definition: E.g. site performs "slow" under load.
- Goal: Average page speed to be 0.5s, top 3 heaviest pages to load
Action Plan:
1) Install Fusion Reactor/SeeFusion in order to get hard numbers.
2) Agreement over expected reasonable load. E.g. 200 simultaneous users. This is where exec support/product management comes in.
3) Use hard numbers to define what "slow" is. That's your baseline. E.g. Avg Page Speed is 90 seconds under a load of 200 simultaneous users.
4) Measure the impact not using CFLOCK.
5) Measure the impact of reducing timeout setting on CFLOCK.
6) Experiment with 2GB Memory on the OS vs. 4GB Memory
7) Experiment with Dual Core vs. Quad Core CPUs
8) Experiment with JVM settings
9) Catalog top 10 slowest pages/queries per SeeFusion/Fusion Reactor
10) Optimize top 10 slowest pages/queries per SeeFusion/Fusion Reactor
11) etc...
The reason why a problem definition/goal is important, is that it keeps everyone aimed in the direction. Whether it's CFLOCK, or more powerful machines, is irrelevant. The goal is performance. It doesn't matter how you get it. It's all about assessing the options, the pros and cons, and taking action.
So e.g., one option may involve $50K of programming effort over a 3 month period, and another might be a $5K more powerful machine over a 1 week period. What's the most cost effective solution?
#8 by Mark Mandel on 7/11/12 - 9:47 PM
Re: "Marc, does it change any if you add in that we have a mechanism in place to manually force reload an application from a URL variable?"
What I do now is have my URL variable call ApplicationStop(). That way I can just use CF's underlying locking mechanism to fire the site back up just the way it would when the server first started up. Far less issues and code duplication.
From everything else, it doesn't actually sound like you need locks at all.
To match you point to point.
1) Setting application or session scoped variables shouldn't need locking unless there are specific race conditions you are trying to avoid.
2) If you are setting variables in onApplicationStart() it's locked to a single thread anyway.
3) I can't see a reason why you would be doing this - only 1 thread can create an instance of an object at a given point and time. (Unless you are doing, application.foo = createObject("Foo"); application.foo.init(); - in which case, just make it a one line call of createObject("Foo").init() )
4) Okay
5) Readonly locks are very useful. I'd suggest looking closely at how they work - but really only in circumstances where you have legitimate race conditions, for which it doesn't sound like you do (at least from this overview).
Hope that helps!
#9 by Brad Wood on 7/12/12 - 1:47 AM
Locking every access of a shared variable is not necessary. A lot of old habits came from the old, old C+ days of CF back when memory could actually get corrupted if multiple threads were writing to the same variable. Those days are gone, but you should use locks if you have multiple operations on a shared scope that either render that scope untouchable by outside processes (removing a variable from memory and repopulating it from an expensive query) or need repeatable reads of consistent data (adding up totals of a shopping cart object which might be changing if another thread was also adding additional items). Locking strategies for application resources aren't too different from database resources.
Locking a single line of code is typically pointless.
If any of your code uses named locks with randomly generated names (which would render them useless), please don't admit to it in public. Just fix them and pretend it never happened.
onApplicationStart is only thread safe when called automatically by CF. If you are calling it manually, then you do need to provide thread safety if any of your code in that method meets the requirements for needing locks.
I would shy away from ever using scoped locks since they are typically unnecessarily heavy-handed. I would use a name that represents the resource that needs locking.
The fact that you're not using any read only locks is sort of a red flag for me, but I'm not familiar with your code base and it's fairly likely that most of your locks are actually unnecessary, but benign.
As far as locking the setting of variables-scoped things in a CFC's init(), I'd say something is wrong there. If the CFC's are transients then the locks are pointless because only one thread ever handles the object. If the CFCs are singletons, then you need to be locking their creation so only one thread creates them and all other threads wait to use them. (Exclusive lock for creating with an existence check inside and out, and a read only lock for any use.)
Here is a typical locking strategy for a shared resource that is created automatically if it doesn't exist:
if shared resource doesn't exist
obtain exclusive lock named "myResource"
check again to see if shared resource still doesn't exist
create shared resource
/if
release exclusive lock named "myResource"
/if
--------------------------
obtain read only lock named "myResource"
access shared resource
release read only lock named "myResource"
Tariq is more gracious than I. I'd tell your hardware guys that they don't get paid to write code, you do. Their job is to supply you with the hardware your application needs and stop complaining. I'm sure you guys have wasted enough salaried time to buy more cores and RAM galore. :)
Good luck!
#10 by Rob Barthle on 7/25/12 - 10:01 PM
But I have learned a lot about locking practices in this, and I am thankful for all the insight!