Prefix<\/p>\n<\/td>\n
Description<\/p>\n<\/td>\n
Explanation\/Example<\/p>\n<\/td>\n<\/tr>\n
First 3 character<\/p>\n<\/td>\n
Denotes your System<\/p>\n<\/td>\n
TRD \u2013 Trading<\/p>\n
REP \u2013 Reporting<\/p>\n
REC \u2013 Reconciliation<\/p>\n<\/td>\n<\/tr>\n
4th<\/sup> character<\/p>\n<\/td>\n Denotes schedule of your job<\/p>\n<\/td>\n 1 \u2013 Daily<\/p>\n 2 \u2013 Biweekly<\/p>\n 3 \u2013 Once in 3 weeks<\/p>\n 4- Monthly<\/p>\n 5 \u2013 Yearly<\/p>\n 9 \u2013 On demand\/Adhoc<\/p>\n<\/td>\n<\/tr>\n 5th<\/sup> & 6th<\/sup> character<\/p>\n<\/td>\n Denotes your interface\/internal process<\/p>\n<\/td>\n your own way of sequence number for your interface<\/p>\n Examples:<\/p>\n 01-Ratings data<\/p>\n 02-custodian data<\/p>\n 03-NAIC data<\/p>\n 04-Factors<\/p>\n 05-Coupons<\/p>\n 06-Foreign exchange<\/p>\n 09-Amortization calculation<\/p>\n<\/td>\n<\/tr>\n 7th<\/sup> & 8th<\/sup> character<\/p>\n<\/td>\n Denotes your batch purpose<\/p>\n<\/td>\n your own way of sequence number for your job<\/p>\n 01 \u2013 File watcher<\/p>\n 02 \u2013 validation job<\/p>\n 03 \u2013 ETL job<\/p>\n 04 \u2013 FTP job<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Example: TRD10101<\/p>\n In this example,<\/p>\n First 3-character TRD denotes Trading system Overall TRD10101 denotes, that this is daily job belongs to Trading system and it\u2019s a file watch job for rating file.<\/p>\n If your organization doesn\u2019t have restriction on the file name, then you can follow this naming convention. Job name contains system name followed by schedule, then interface and process name<\/p>\n Example: Invest-Dly-Rating-FW<\/p>\n This example denotes that job is Ratings File watcher job which runs daily for investment system<\/p>\n Now we have defined the naming convention for individual job, now we have to group them appropriately<\/p>\n Group individual jobs based on process like Inbound, Critical, Outbound, FTP<\/p>\n Fig 1. Grouping of jobs based on process flow\/events<\/strong><\/p>\n Inbound \u2013 Include all jobs that feeds data to your application. For example, in investment management application batch this group will contain the batch job that process security master, trades files, ratings, foreign exchange, factors, coupons, etc.<\/p>\n Critical \u2013 Group all your jobs that does core processing for your system. For example, in investment management system, the core jobs include nightly processing, accounting updates, amortization calculation, market value calculation etc.<\/p>\n Outbound \u2013 Group all your jobs that send files to downstream system or reporting extracts.<\/p>\n Group the batch job based on the timeframe. If your batch cycle for your system starts at 6pm, then you can group your jobs like 6pm-9pm, 9pm-2am,2am-6am.<\/p>\n Fig 2. Grouping of job based on timelines<\/p>\n Create late shout if the job does not run before certain time<\/li>\n During SDLC phase, developer should design a process that can be either scheduled to rerun the job or should be aborted which requires manual intervention during failure without causing any data loss or duplicate data. Let\u2019s say if your job does ETL process, if one of the records failed, the developer should have the system designed to either skip the record, continue the batch process and trigger a notification to support\/business team on the skipped record or fail the job. This should be identified as part of system design.<\/p>\n Most of the scheduler has feature to rerun a job automatically when it fails first time with time lag. Sometime batch might fail due to network connectivity, enabling this feature can resolve the batch failure automatically.<\/p>\n Though our objective of batch framework is to run the jobs without any manual intervention but at same time we wanted to capture the error which are generated during the batch run. Scheduler application like BMC control-m has feature to fail the job based on key word in the output log returned to the scheduler.<\/p>\n Let\u2019s say you have file watcher job and a processing job as a dependency. If the file received from upstream is empty, processing job will complete without any issue. If you have a log as 0 records processed. You can pro-actively fail the job.<\/p>\n This depend on the business rules that are pre-defined.<\/p>\n Run book is bible for application support team. So, runbook should capture detailed instruction for each job. If job is file watcher, then document the point of contact for the job including phone number, email of the support team responsible for transmitting the file. For processing job, capture the instruction to login procedure to the server, navigating to the application folder, steps to fix the problem such as running a script to exclude a bad record which is causing the batch to fail, mandatory files required to rerun the job, communication procedure to downstream or to the business users on the failure. The run book should also contain escalating procedure. Application support run book is a live document and it cannot be perfect initially and support team should keep updating this document as team gains more knowledge.<\/p>\n Take backup of database at beginning of batch start, at middle of the batch and at end of batch. If your system has non recoverable process, it is better to take a backup before and after the process. For example, investment management application has a process to roll forward accounting system date to next date. This process is very critical and any system failure during this critical process at times can make the system non recoverable. So, it is recommended to take database backup before and after the critical process.<\/p>\n Another advantage of taking backup during the batch is troubleshooting. As application data changes over the nightly batch, finding the root cause of production issue can be really challenging. With this additional backup, application support team can restore test database with backup taken during the batch and proceed with the investigation.<\/p>\n Application support is reactive but with below additional steps in batch can turn application support from reactive to pro-active<\/p>\n For example, accounting application usually runs longer during month end and first business day. Pro-actively batch support can be informed about this trend and downstream system can be informed about possible delay of file delivery.<\/p>\n Now we have seen different component of batch framework, let\u2019s see how we can incorporate these features in software development lifecycle process (both waterfall and agile).<\/p>\n Waterfall methodology is traditional process and it consists of requirement gathering, analysis, design, coding, testing and production release.<\/p>\n As part of requirement gathering, it is essential to identify below questions<\/p>\n Agile methodology follows an iterative development approach because of this planning, development, prototyping and other software development phases may appear more than once. So, implementing batch framework should be part of overall process. If possible, as part sprint planning there should be separate task for batch testing.<\/p>\n Often team thinks that batch is just calling a process in scheduling tool, but bad design and inconsistent approach can make application support complicated. Implementation of batch framework will create consistency across application<\/p>\n","protected":false},"excerpt":{"rendered":" In SDLC (software development life cycle) process, creating a batch job is the last step. As the application maintenance grows over period of time and new component added to the overall process can make over batch support complicated. This article provides guidelines to create batch jobs that can be well supported in future. Though this……<\/p>\n","protected":false},"author":324217,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[2],"tags":[],"coauthors":[105880],"class_list":["post-87553","post","type-post","status-publish","format-standard","hentry","category-blogs"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/87553","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/users\/324217"}],"replies":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/comments?post=87553"}],"version-history":[{"count":11,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/87553\/revisions"}],"predecessor-version":[{"id":104597,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/87553\/revisions\/104597"}],"wp:attachment":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media?parent=87553"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/categories?post=87553"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/tags?post=87553"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/coauthors?post=87553"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n \n \n \n \n \n \n \n \n \n
\n4th<\/sup> character 1 denotes Daily job
\n5th<\/sup> & 6th<\/sup> character 01 denotes Ratings data
\n7th<\/sup> & 8th<\/sup> character 01 denotes the job is file watcher<\/p>\nSystem-Sch-App-Process<\/h3>\n
Structuring of batch job<\/h2>\n
Grouping of jobs based on Process flow\/Events<\/h3>\n
![](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2020\/07\/word-image-35.png\")
<\/p>\nGrouping of job based on Timelines<\/h3>\n
![](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2020\/07\/word-image-36.png\")
<\/p>\nDesigning of individual batch job<\/h2>\n
3-part design<\/h3>\n
\n
\nMake sure that job gets triggered by any one of these events<\/p>\n\n
\nNext part is actual job to run. This is the core design of the system.<\/li>\n
\nLast part is housekeeping. Rename the file produced by the process with date time stamp.<\/li>\n<\/ul>\nAbility to rerun the job during failure<\/h3>\n
Fail a job with right attitude<\/h3>\n
Let\u2019s prepare for BAD Day<\/h2>\n
Run book for individual job<\/h3>\n
Recovering batch to a critical point<\/h3>\n
Housekeeping<\/h2>\n
\n
\nAfter daily batch, archive the files that are received and processed. Zip them and name with date.
\nFor example \u2013 InboundArch_12082015
\nAt end of month, move the individual zipped files to corresponding monthly folders. Similarly, at end of year, archive all monthly folders to the year folder.<\/li>\n
\nIt is recommended to align batch jobs to enterprise scheduling calendar. Create custom calendar for scheduling criteria that does not align with enterprise scheduling calendar.<\/li>\n
\nAs part of maintenance, you will need to stop running a job. This needs to be done by documenting, validating all the dependencies and make sure there are no upstream and downstream application waiting for this job. Once validated and signed off by all the team first step is to disable the job. Let it run for a month. Finally retire the job.<\/li>\n<\/ul>\nReactive vs Pro-active<\/h2>\n
\n
\n
\nFor example, we don\u2019t want to have a bad foreign exchange data in the system. Support team can have a batch job to identify bad foreign exchange before the start of critical process.<\/li>\n
\nAccounting\/Finance related application have special process or report to be processed during month end, quarter end and year end or special holiday. Support team can pro-actively create checklist to make sure to check if batch jobs are scheduled appropriately.<\/li>\n<\/ul>\nBatch framework in (SDLC) Software development lifecycle perspective<\/h2>\n
Requirement gathering:<\/h2>\n
\n
Analysis & Design:<\/h2>\n
\n
Testing:<\/h3>\n
\n
\n
\n
\n
SDLC \u2013 Agile methodology<\/h3>\n
Conclusion:<\/h2>\n