Managing PeopleSoft Master Scheduler Servers

This chapter provides an overview of PeopleSoft Master Scheduler functions and describes:

Circumstances where a Master Scheduler is required.
How to use multiple Master Schedulers.
Master Scheduler request prioritization.
How to Manage workload across servers.

Understanding PeopleSoft Master Scheduler Functions

A Master Scheduler enables load balancing of workload by automatically routing requests to available Process Scheduler servers, which ensures that you maintain optimal processing at all times. This feature also offers fault tolerance in your batch environment. In an event of a server failure, a Master Scheduler can redistribute queued requests among the remaining active Process Scheduler servers. In addition, an active Master Scheduler manages and controls all Process Scheduler server domains that are on the same PeopleSoft database. It enforces all of the rules specified in either the process or job definitions, and monitors the execution of all processes. It becomes the centralized control as it interrogates the Process Request table looking for any queued requests to run, and then dispatches them to an appropriate available Process Scheduler server.

A Master Scheduler can be activated in any of the Process Scheduler servers in Windows 2000 and UNIX. This option is enabled by default when configuring a new Process Scheduler server in NT and UNIX. However, this option is not available in IBM’s UNIX System Services (USS). For DB2/OS390 customers who intend to start a Process Scheduler in USS and want to take advantage of this feature, then a Process Scheduler server domain must be set up in either a Windows 2000 or supported UNIX operating system other than USS.

Disadvantages of Using Multiple Process Schedulers With No Master Scheduler

When a Master Scheduler is not used, each Process Scheduler server brought up is responsible for managing its own workload by querying the Process Request table. This can be problematic when multiple Process Scheduler servers are booted for the same database. Each server attempts to schedule requests specified to run either on this specific server or any server. If any request is set to run on any server, it’s possible for more than one server to attempt to schedule the same request. To work around this, specify a specific server through the Process Request page. However, this becomes disadvantageous if the specified server goes down because the request remains queued until the Process Scheduler server is brought up again.

One other disadvantage of bringing up multiple Process Scheduler servers without using a Master Scheduler, is the uneven balance of workload across all servers. PeopleSoft Process Scheduler is constrained to have new requests scheduled with no server name, to be picked up only by servers running in the operating system specified as the Primary Operating System in the System Settings page. This is illustrated in the diagram.

Example Master Scheduler setup using the Primary Operating System option

In this specific setup, there are multiple servers brought up in Windows 2000 (PSNT1 and PSNT2), UNIX(PSUNX) and OS390 (PSOS390), where Windows is the designated primary operating system. Assuming all new requests were scheduled with a blank server name, then only PSNT1 and PSNT2 are qualified to pick up these requests. The PSUNX or PSOS390 will only be utilized when requests are scheduled with the intended Process Scheduler server’s name. Also, it’s possible to see a scenario where PSNT1 will pick up most of the requests leaving PSNT2 under utilized.

The Master Scheduler resolves this problem by becoming the central point for querying the Process Request table. When a Master Scheduler is available, all active PeopleSoft Process Scheduler Servers switch into a remote server mode. Master Scheduler registers and monitors any active remote servers. Once the active Master Scheduler prioritizes all new queued requests, it will perform a round robin on all available servers to decide which remote server is the most appropriate for running a particular request at run time. It attempts to evenly load balance workload across all available servers, enabling the most effective use of overall computing resources.

Describing Circumstances Where a Master Scheduler is Required

The Master Scheduler is an optional server that enables you to distribute workload across multiple Process Schedulers. However, this table shows specific circumstances that mandate having an active Master Scheduler available:

Condition	Reason
Schedule PSJobs containing processes that need to run in different operating systems.	This instance is particular to database platforms that allow having Process Scheduler servers booted in multiple operating systems. When the Primary Operating System is set to UNIX or OS390, Process Scheduler will attempt to assign all processes within a PSJob to Process Schedulers with this operating system. However there are certain processes that can run exclusively from Windows 2000. For example, includes any Crystal or PS/nVision processes. Master Scheduler is required to redirect the PSJob item to PeopleSoft Process Scheduler on Windows 2000.
Active Schedule JobSets are defined.	Only a Master Scheduler can schedule any active Schedule JobSets. The Master Scheduler is also responsible for scheduling any recurring Schedule JobSets.
Impose system constraints, as defined in process or job definitions.	A process or job can now be defined with either Mutually Exclusive Processes or Max Concurrent values. These system constraints will be imposed only if a Master Scheduler is active.
The System Load Balancing Option is set to Assign To Server In Any O/S.	When a machine goes down, Master Scheduler can transfer queued requests assigned to the PeopleSoft Process Scheduler Server on a downed machine to a PeopleSoft Process Scheduler Server started on another machine.

Using Multiple Master Schedulers

Each Process Scheduler domain on Windows 2000 or UNIX, (except for USS), can be set up to have a Master Scheduler started. However, only one Master Scheduler is active to control the workload at any time. The other Master Schedulers remain in a state of idle. If the active Master Scheduler goes down, then one of the idle Master Scheduler servers take control. If a Master Scheduler is not available, then the PSPRCSRV servers, currently in remove server mode, switch back to standalone mode and query the Process Request table to find work.

The Process Monitor component identifies the Process Scheduler server where the Master Scheduler is active. From the Server List tab, where the list of active Process Scheduler servers are displayed, the Master column indicates if a Master Scheduler is activate in any of the servers.

Describing Master Scheduler Request Prioritization

When Master Scheduler tallies all new queued requests, it will attempt to prioritize all incoming requests before performing a round robin on all registered active servers to find the appropriate server. A set of rules were implemented to aid the Master Scheduler prioritize the accumulated queued requests found in the Process Request table. All requests are sorted based on these conditions:

Restartable or recovery process.

Any process set in the process definition as restartable with a non zero retry count that failed in previous attempts and currently has a run status of Restart, will be given a higher priority and will be on top of the priority list. Similarly, a process automatically scheduled by PeopleSoft Process Scheduler as a recovery process for a failed request will also be placed on top of the priority list.
Processes contained in active PSJobs.

The Master Scheduler monitors all active PSJobs that have processes currently initiated in one or more Process Scheduler (remote) servers. When the Master Scheduler detects that available slots are available to assign requests to a remote server and prepares to evaluate all queued requests, it will initiate processes within these active PSJobs prior to querying the Process Request table for new queued requests.
Accumulated priority value.

Each request is given an overall priority value based on the different priority options available in the system. Master Scheduler calculates the overall priority based on the order of importance of this priority, and will rank the request accordingly. However, each server definition may be customized with different priority options, and therefore may result in the same request ranking high in one server while it is positioned at the bottom of the list in another.

For example, a SQR report has a process category of Financials. This category has a priority of High in the PSNT server definition, while the same category has a priority of Low in the PSNT2 server definition. In this situation it is likely that the SQR report will be assigned to the PSNT server.

The overall priority will be calculated based on this order of importance of all these priority options that can be assigned to a request:
1. Process Category Priority: The system will assign the priority value of the process category specified in the server definition to the request. If a process belongs in a PSJob, then all processes in this PSJob will be assigned the process category of the PSJob. In the case of a complex PSJob where a PSJob is embedded within another PSJob, then all the processes will be assigned the process category of the main PSJob.
2. Process/Job Priority: This is the priority as defined in either the process or job definition. Similar to the process category, all processes within a PSJob will have the priority of the main PSJob.
3. Process Type Priority: This is the priority specified in the server definition for each process type it can process. In the case of PSJob, the process within a PSJob will have the priority based on its own process type.
Run date and time.

In the event of two or more requests having the same calculated priority based on all the criteria noted above, the request with an earlier run date and time will be scheduled first by the Master Scheduler.

Managing Workload Across Available Servers

This section discusses the parameters used to control how workload is managed across available servers.

Parameter	Modified In	Options
Primary Operating System	System Settings	Windows 2000 (default). UNIX. OS390.
Load Balancing Option	System Settings	Assign to Primary O/S Only (default). Assign to Server in Any O/S.
Server Load Balancing Option	Server Definition	Use for Load Balancing (default). Do Not Use for Load Balancing.
Redistribute Option	Server Definition	Redistribute to any O/S (default). Redistribute with same O/S. Do Not Redistribute.
Max API Aware	Server Definition	Numeric value (default = 5).
Process Category Max Concurrent	Server Definition	Numeric value not exceeding the Max API Aware.
Process Type Max Concurrent	Server Definition	Numeric value not exceeding the Max API Aware.
Server Status	NA	NA

Primary Operating System Option

The operating system specified in the Primary Operating System field in the System Settings component, is the default operating system assigned to all new queued requests with a blank server name specified. If the system detects that the process in the request cannot be run in the default operating based on the process type definition, then the system will assign the request with the operating system found in the process type definition.

Load Balancing Options

The Load Balancing Option affects how Master Scheduler performs the round-robin assignment for all available remote servers in attempt to load balance the workload. When the option Assign to Primary O/S Only is selected, Master Scheduler will only perform a round robin to all remote servers booted in the primary O/S.

The pattern on how Master Scheduler assigns requests to available servers with this option is illustrated in this diagram.

Example Master Scheduler setup using the Load Balancing - Assign to Primary O/S Only option

In this case, the Primary Operating System is Windows 2000. This is the operating system where both PSNT1 and PSNT2 are initiated. When Master Scheduler finds new queued requests with blank server name, the workload is evenly distributed between the two NT Process Scheduler servers only. Although PSUNIX1 and PSOS390 are available, no requests are assigned to these servers. The remote servers PSUNIX1 and PSOS390 will only be assigned with new requests scheduled with this specific server name.

If the option is set to Assign To Server In Any O/S, Master Scheduler will attempt to load balance workload to all active servers. At first, it tries to distribute work to servers residing in the primary operating systems. Once it has reached the server definition limitations it attempts to route work to the remaining active servers. For example, Master Scheduler will round robin the prioritized lists to both PSNT1 and PSNT2, as these servers are booted in the primary operating system. Assuming the Max API Aware for both PSNT1 and PSNT2 is three, then the first six process requests will be distributed between PSNT1 and PSNT2, and the reaming requests will be distributed to PSUNIX1 and PSOS390.

Example Master Scheduler setup using the Load Balancing - Assign To Server In Any O/S option

Server Load Balancing Options

The server definition's Server Load Balancing Options field indicates whether the server can be used for routing new requests with blank server names. Choose the Use For Load Balancing option if this server can be assigned requests with no specific server name specified. If this server is intended to be used only if new process requests are scheduled with this server’s name, set this server with the Do Not Use For Load Balancing option.

Redistribute Option

The server definition’s Redistribute Option field directs the Master Scheduler to a course of action in the event a server is shutdown or encounters a software/hardware failure. If Master Scheduler finds new queued process requests with the server’s name identifier and detects the server is currently unable to process any requests, one of the following three options can be selected:

Redistribute to any O/S: Master Scheduler attempts to redistribute requests with the server’s name into any available active Process Scheduler server.
Redistribute with same O/S: Master Scheduler attempts to reroute requests only to another Process Scheduler booted in the same operating system as this server.
Do Not Redistribute: Master scheduler does not attempt to reroute any requests with the server’s name identifier. In this case, requests remain queued until the server is booted up.

Max API Aware and Max Concurrent Options

Master Scheduler periodically monitors the current workload of all active Process Scheduler servers. It ensures that when performing a round robin assignment, to not exceed any of the following limitations specified in the server definition:

Max API Aware: Indicates the total number of tasks a Process Scheduler can initiate concurrently.
Process Category Max Concurrent: Indicates the upper limitation of how many processes with the same process category can be initiated concurrently

Note. The number assigned to this field cannot exceed the value specified in the Max API Aware field.
Process Type Max Concurrent: Similar to the Process Category Max Concurrent field, this indicates the limit based on the request’s process type.

Server Status

Master Scheduler will only route work to Process Scheduler servers with a server status of Running. If a server has a status of Suspended, Overload or Down, Master Scheduler will defer routing work to the server until the status is changed back to Running. Master Scheduler evaluates the appropriate action for process requests assigned to the server based on the Redistribute Option setting.