System Administration - Quartz Utility – System Guide

CMiC has designed a utility approach for Quartz Schedulers to perform scheduled jobs in the background thus alleviating the resource consumption for Weblogic, additionally any interruptions that can cause performance to user experience, especially if the job is resource heavy processing.

Requirements

As of Patch V12-014-1, Quartz features to support OCR capabilities was added and requires Visual C++ redist 2019 for Windows. This is also required for the OCR feature within applications. Please see Implementation of Tesseract in Patch V12-014-1 for more details.

Available Scheduler Types

Below is the list of schedulers and its jobs which are supported to be started by the utility. More details can be found in the Scheduler Properties section.

  1. CmicScheduler – General, small tasks

    • Thumbnails, etc.

  2. DrawingMgmt – Drawing Management specific tasks

    • Upload, linking sheets, download drawing set

NOTE: This list will be growing in the future.

Runtime

The details below are related to the runtime files being used for Quartz utility.

NOTE: Once they exist there is no further maintenance from future patch installations, unless under special circumstances which would not affect any customizations.

Related Runtime Files

  1. Startup

    1. Location: {cm_home}\ias\j2ee\{env_name}\bin

      1. runLocalQuartz_{env_name}.cmd

        1. Startup script for the JVM container for all the schedulers

  2. Quartz runtime properties

    1. Location: {cm_home}\ias\j2ee\{env_name}\quartz

      1. cmicjobscheduler.standalone.log4j.properties

        1. Logging configuration

      2. cmicstandalone_{env_name}.quartz.properties

        1. Scheduler to be started

        2. Define the datasource

        3. Scheduler custom override properties

  3. Libraries

    1. Location: {cm_home}\V12\{env_name}\uig\quartz\lib

      1. All libraries required to run the different types of jobs

  4. Templates

    1. Location: {cm_home}\V12\{env_name}\uig\quartz

      1. cmicstandalone.quartz-template.properties

      2. cmicjobscheduler.standalone.log4j-template.properties

      NOTE: Please do not modify the templates. These will be overwritten by future updates.

How to Set Memory Allocation

In the startup file runLocalQuartz_{env_name}.cmd, look for the variable MEM_ARGS. By default, the JVM memory settings are defined as -Xms128m -Xmx2048m.

NOTE: The memory argument should be adjusted appropriately based on how many schedulers are expected to be started and the number of jobs each could run simultaneously.

Example:

If you choose to have CmicScheduler & DrawingMgmt started which by default both only run one job -Xmx2048m should be enough, while if choice is to run two jobs for DrawingMgmt it may require adjustment to -Xmx3072m.

This is not a baseline requirement there are many factors to the decision if you are trying to limit(minimize) the amount of memory allocated for the utility.

How to Set Which Scheduler Will Start/Run

In the properties file cmicstandalone_{env_name}.quartz.properties the property controlling this is cmic.run.schedulers.list. It is a comma-delimited list of scheduler names (case-sensitive), see available list.

Example:

cmic.run.schedulers.list=CmicScheduler,DrawingMgmt

Runtime Configuration

Startup for Quartz utility is through a task scheduler defined to execute runLocalQuartz_<env_name>.cmd. Please see the Setup Thumbnail Job article for information on how to create the task for Windows/Linux.

First Time Running

The first time runLocalQuartz_{env_name}.cmd is execute it will generate runtime files from the templates files we have released (see related runtime files) and place environment specific files to {cm_home}\ias\j2ee\{env_name}\quartz.

Any subsequent execution of the .cmd will not change the files to avoid overwriting any customization on top of what has been released, but if any required file is missing it will be recreated from the template.

NOTE: Please do not modify the templates, they will be overwritten by future updates.

Warning

If there is a choice to have multiple servers running the utility, please make sure to keep all updates in sync to avoid job processing failures due to missing updates as each scheduler runs as a cluster on the same queue so there is no guarantee which instance might run the job.

Logging

Logging is supported and the configuration settings are found in the custom log4j properties file.

File: {cm_home}\ias\j2ee\{env_name}\quartz\cmicjobscheduler.standalone.log4j.properties

The default settings are:

  1. Max size of each log file is 20MB

  2. Keep only the last 10 log files

  3. Log file name: cmicquartz.standalone.log

  4. Location: {cm_home}\ias\j2ee\{env_name}\bin\cmicjobschedulerlogs

    1. Prior to patch 13-2: {cm_home}\ias\j2ee\{env_name}\bin\logs

  5. Logging levels

    1. Package org.quartz = info

    2. Package cmic.job.scheduler = warn

      1. Changing warn to info is useful, especially on the initial state of testing that is not heavily logging. It's beneficial if there is an issue reported to CMiC.

Scheduler Properties

Provided is a template file which is being maintained on patch releases for new schedulers available and other default settings.

Location: {cm_home}\v12\{env_name}\uig\quartz

Template file: cmicstandalone.quartz-template.properties

This file is used to build the runtime properties when one does not exist, otherwise we leave the runtime as is to avoid overwriting customization.

The key property to make note of is cmic.run.schedulers.list which defines the schedulers that will be started.

Customize Scheduler Properties

Custom properties can be defined for a specific scheduler, there is no support for global customization.

We suggest using the specific section for a scheduler to place the customized properties.

To define any custom properties must be prefixed with the scheduler name and separated from the property name by an underscore. Further breakdown of properties that can be customized for all schedulers or for a specific scheduler are as follows:

Ex. {scheduler_name}_{property_name}

Replace {scheduler_name} with the right scheduler you want to target.

Replace {property_name} with the property to customize.

Place an underscore _ between them.

Global Properties

Below are the properties which can be customized for all the schedulers on initialization:

  1. cmic.scheduler.force.shutdown

    • During runtime, this will be read by the monitoring thread to shutdown scheduler (true/false). The default value is false.

    • Setting this property to true may take up to 15 minutes. This is the default time for our scheduler manager to review updates.

  2. org.quartz.threadPool.threadCount

    • Number of jobs the scheduler can run simultaneously which has a default value of 1.

CMiC Scheduler Properties

Below are properties which are additional available for customization to CmicScheduler only:

  1. cmic.thumbnail.maxruntime=60000

    • timeout in milliseconds for thumbnail process, default value is 60000

  2. cmic.thumbnail.process.external=true

    • set to true to run thumbnail jobs on external (JVM) process, set to false (default) to run on the current JVM

  3. cmic.thumbnail.process.external.maxheapsize=1536

    • max heap size in megabytes

Troubleshooting

It is important when reporting a problem to supply the current log files (see Logging) for analysis as it might not be clear the reason for the failure.

It may also be requested that the log level is tuned to get further details of the problem on the next attempt, and it may be suggested to log level for a specific job (CMiC would supply the details to change).

No Jobs Are Executing (Log is Not Being Updated)

If no jobs are executing, the first check would be to verify if the task is running, then check if the log is being updating or reporting an issue.

If the log is not being updated, the problem may be hidden and the next step would be to run the runLocalQuartz_{env_name}.cmd from the command line for additional diagnoses.

Java.lang.outofmemoryerror: Java Heap Space

The main cause for this is that the utility memory arguments may be too restrictive and the step is to review the current -Xmx value and if that can be increased.

If that has been done and it did not resolve the error, either a larger value should be set or the amount of jobs that can run simultaneously may need to be revised.

If requesting assistance from CMiC, please supply some details related to the memory arguments set and the run time properties (cmicstandalone_{env_name}.quartz.properties) details:

  • cmic.run.schedulers.list

  • any custom properties set

Java.lang.outofmemoryerror: GC Overhead Limit Exceeded

Please see the Java.lang.OutOfMemoryError: Java Heap Space section above.

File Permissions (Access Denied)

This could be caused by the task being run by a user who does not have the right permissions, please check the task configuration.

Since the utility is not run under a specific user and may even be run on a separate app server, it is important that the user who is running it has access to specific directories such as:

  1. Temporary folder (reference used java.io.tmpdir)

  2. Attachments folder (where all attachments exist)

Check the task to find out who is running it.

  • For Windows: Task Scheduler -> Task Scheduler Library -> find the task (e.g. QuartzOnTest) -> Security options -> user account that it runs under.

  • For Windows using Service instead of a Task Scheduler, check who the service is logged in as: Services -> find the service (e.g. cmicquartz_test12c) -> look at Log On As column (or from the service properties, check the Log On tab).

Java.lang.UnsatisfiedLinkError: The specified module could not be found

The likely cause is that the .dll for tesseract cannot be loaded and is missing the installation of MS Visual C++ redistribution 2019.

As of Patch V12-014-1, the requirements for OCR capabilities and further details can be found in the Implementation of Tesseract in Patch V12-014-1 article.

Invalid Memory Access

This is most likely due to missing installation files for the OCR process, where tessdata.jar and win32-x86-64.jar have not been deployed.

The resolution is to deploy tessdata.jar and win32-x86-64.jar which can be found here:

Copy 
{cm_home}\v12\{env}\jsp

The following shows results in an error that are reported in the error logs:

Copy 
Error opening data file D:\cm\ias\j2ee\prod\classes\tessdata/eng.traineddata
Please make sure the TESSDATA_PREFIX environment variable is set to your "tessdata" directory.
Failed loading language 'eng'
Tesseract couldn't load any languages!
Copy 
java.lang.Error: Invalid memory access
                at com.sun.jna.Native.invokePointer(Native Method)
                at com.sun.jna.Function.invokePointer(Function.java:470)
                at com.sun.jna.Function.invoke(Function.java:404)
                at com.sun.jna.Function.invoke(Function.java:315)
                at com.sun.jna.Library$Handler.invoke(Library.java:212)
                at com.sun.proxy.$Proxy4.TessBaseAPIGetUTF8Text(Unknown Source)
…