Extended cron
small-shell can provide Job automation & data Exchagne function by extending cron function (e-cron). you can define job relationship between several hosts. you can also exchange DATA between several hosts.
sudo /usr/local/small-shell/adm/gen -job
Type of job (1.job automation | 2.file exchange) : 1
--Define schedule--
Month [ any | 1-12 ]:
Date [ any | 1-31 ]:
Hour [ any | 0-23 ]:
Min [ any | 0-59 ]:
Week [ any | mon - sun ]:
Exec command or batch script: XXXX
Required
It's required to complete OS Env setup and installation of small-shell, see detail at Quickstart.
Features
e-cron has following features.
- Add job entry to cron through dialog
- Provide CLI status check tool
- Prevent double start of same job
- Logging job result and command dump
- Make relationship between jobs or hosts using messaing function
- Provide dataExchange platform between small-shell hosts
Basic concept
Basically e-cron extends cron function by providing utility scripts. and wrapper script will work as job management framework. following ideas are basic functions of e-cron.
Dialog
gen dialog -> generate cron entry
Diagram
cron -> wrapper.sh -> script #exec
CLI <-> wrapper.sh <- script #result
dir
Please upload script on following directory.
/usr/local/small-shell/util/scripts
Add Job through e-cron
This is an example of adding job through e-cron. also you can learn how to confirm reulst of the Job and dumped output of the Job or command.
sample job
sudo -u small-shell vi /usr/local/small-shell/util/scripts/job1.sh
--code--
#!/bin/bash
echo "`date` Hello" >> /usr/local/small-shell/util/scripts/log/hello.log
exit 0
--------
sudo -u small-shell chmod 755 /usr/local/small-shell/util/scripts/job1.sh
Automate Stand alone Job
You can generate job by gen command.
sudo /usr/local/small-shell/adm/gen -job
# -> dialog will be started
Following is an example of making job that will execuete script. it's just an examplem, you can execute any unix commands or any scripts.
Dialog
Job Name: job1
Type of job (1.job automation | 2.file exchange) : 1
--Define schedule--
Month [ any | 1-12 ]: any
Date [ any | 1-31 ]: any
Hour [ any | 0-23 ]: any
Min [ any | 0-59 ]: 1
Week [ any | mon - sun ]: any
Exec command or batch script: job1.sh
enabel/disable Job
sudo -u small-shell /usr/local/small-shell/bin/e-cron enable.$job
sudo -u small-shell /usr/local/small-shell/bin/e-cron disable.$job
(e.g.)
sudo -u small-shell /usr/local/small-shell/bin/e-cron enable.job1.sh
confirm job definition
sudo -u small-shell /usr/local/small-shell/bin/e-cron cat.job1.sh
> JOB: job1
def:/usr/local/small-shell/util/e-cron/def/job1.def
-------------SCHEDULE----------------
min: 1
hour: any
date: any
month: any
week: any
-------------DEFINITION----------------
exec_command="/home/ubuntu/job1.sh"
input_message=""
output_message=""
hubapi=""
authkey=""
confirm job list
sudo -u small-shell /usr/local/small-shell/bin/e-cron ls
--------------------------------------------------------------
job definition: /usr/local/small-shell/util/e-cron/def
--------------------------------------------------------------
del_session.enabled
del_util_log.enabled
job1.enabled
Exec job manually
sudo -u small-shell /usr/local/small-shell/bin/e-cron exec.job1
confirm job result
sudo -u small-shell /usr/local/small-shell/bin/e-cron stats
--------------------------------------------------------
LATEST STATUS OF JOB
--------------------------------------------------------
2021-04-12 12:15:01 ls_tmp successfully completed
cat /usr/local/small-shell/util/scripts/log/hello.log
Mon Jan 3 04:10:23 UTC 2022 Hello
confirm job log
There are 2 types of the log. job_log is execution history, job_command_dump record output from script or command.
ls /usr/local/small-shell/util/e-cron/log/joblog/*job1*
/usr/local/small-shell/util/e-cron/log/joblog/job1_command_dump_20220103
/usr/local/small-shell/util/e-cron/log/joblog/job1_log_20220103
change Job setting
sudo /usr/local/small-shell/adm/gen -job
> then input existing job name in the dialog
Job relation ship in same host
If you want to define job relationship in same same host, please modify job definition directly. This is an exmaple of job relationship. messae wil be exchanged in the host.
job1.success@host1 --> message (done.1) --> job2@host1 #if message is raedy, kicked
! Once message is grabed, message will be cleared
sample job
job1.sh
sudo -u small-shell vi /usr/local/small-shell/util/scripts/job1.sh
--code--
#!/bin/bash
echo "`date` Hello" >> /usr/local/small-shell/util/scripts/log/hello.log
exit 0
--------
sudo -u small-shell chmod 755 /usr/local/small-shell/util/scripts/job1.sh
job2.sh
sudo -u small-shell vi /usr/local/small-shell/util/scripts/job2.sh
--code--
#!/bin/bash
echo "`date` World" >> /usr/local/small-shell/util/scripts/log/world.log
exit 0
--------
sudo -u small-shell chmod 755 /usr/local/small-shell/util/scripts/job2.sh
Add job to small-shell and set message
sudo /usr/local/small-shell/adm/gen -job
#-> please create job1 on small-shell
sudo -u small-shell vi /usr/local/small-shell/util/e-cron/def/job1.def
-----------------------------------------------------------
SCHEDULE:1 * * * *
exec_command="/usr/local/small-shell/util/scripts/job1.sh"
input_message=""
output_message="done.1"
hubapi=""
authkey=""
-----------------------------------------------------------
sudo -u small-shell /usr/local/small-shell/bin/e-cron enable.job1
sudo /usr/local/small-shell/adm/gen -job
#-> please create job2 on small-shell
sudo -u small-shell vi /usr/local/small-shell/util/e-cron/def/job2.def
-----------------------------------------------------------
SCHEDULE:2 * * * *
exec_command="/usr/local/small-shell/util/scripts/job2.sh"
input_message="done.1"
output_message=""
hubapi=""
authkey=""
-----------------------------------------------------------
sudo -u small-shell /usr/local/small-shell/bin/e-cron enable.job2
Exec job
Then let's try to execute jobs. if you execute job2 first, job will not be started because message is not set by job1.
# execute manually
sudo -u small-shell /usr/local/small-shell/bin/e-cron exec.job2
#-> job will not be started. please exit Ctrl + C
sudo -u small-shell /usr/local/small-shell/bin/e-cron stats.job2
--------------------------------------------------------
LATEST STATUS OF JOB
filter: job2
--------------------------------------------------------
2022-01-03 04:34:53 job2 INFO input_message_done.1_was_not_set
# re-execute manually
sudo -u small-shell /usr/local/small-shell/bin/e-cron exec.job1
sudo -u small-shell /usr/local/small-shell/bin/e-cron exec.job2
# confirm result
sudo -u small-shell /usr/local/small-shell/bin/e-cron stats.job
--------------------------------------------------------
LATEST STATUS OF JOB
filter: job
--------------------------------------------------------
2022-01-03 04:38:38 job1 INFO ouput_message_done.1_pushed
2022-01-03 04:38:38 job1 successfully completed
2022-01-03 04:38:47 job2 successfully completed
cat /usr/local/small-shell/util/scripts/log/hello.log
Mon Jan 3 04:10:23 UTC 2022 Hello
Mon Jan 3 04:38:38 UTC 2022 Hello
cat /usr/local/small-shell/util/scripts/log/world.log
Mon Jan 3 04:38:47 UTC 2022 World
Job relationship between hosts
If you will define relation ship between serveral hosts, you need to generate Base APP first on HUB server. #see APP shell tour for further details for Base APP. Once API HUB is ready, you can define job relationship using message.
-----------------------------------------------
| API Server #MessageExchange |
-----------------------------------------------
| GET/PUSH message #http/https protocol
-----------------------------------------------
| Edge servers {hostsA,B,C,D} automated jobs | # jobs will be kicked once get message
-----------------------------------------------
confirm base information at API HUB server
@hub
sudo cat /usr/local/small-shell/web/base | grep hubapi
hubapi="http://ec2-**-**-**-**.com/cgi-bin/e-cron"
sudo cat /usr/local/small-shell/web/base | grep auth
api_authkey="YXBpOmE2ODcyODI2MGZlOGRmOTEzZjNlMmUyYjM2YzlmZD****"
Update job definition at each hosts
In this exmaple, hostB job2 will be kicked, once got message from hostA job1 through API HUB.
job1.success@hostA ---> message (done.1) --> job2@hostB #if message is raedy, kicked
sample job
job1.sh@hostA
sudo -u small-shell vi /usr/local/small-shell/util/scripts/job1.sh
--code--
#!/bin/bash
echo "`date` Hello" >> /usr/local/small-shell/util/scripts/log/hello.log
exit 0
--------
sudo -u small-shell chmod 755 /usr/local/small-shell/util/scripts/job1.sh
job2.sh@hostB
sudo -u small-shell vi /usr/local/small-shell/util/scripts/job2.sh
--code--
#!/bin/bash
echo "`date` World" >> /usr/local/small-shell/util/scripts/log/world.log
exit 0
--------
sudo -u small-shell chmod 755 /usr/local/small-shell/util/scripts/job2.sh
Add job to small-shell and set message
@hostA
sudo /usr/local/small-shell/adm/gen -job
#-> please create job2
sudo vi /usr/local/small-shell/util/e-cron/def/job1.def
-----------------------------------------------------------
SCHEDULE:1 * * * *
exec_command="/usr/local/small-shell/util/scripts/job1.sh"
input_message=""
output_message="done.1" #modify
hubapi="https://ec2-**-**-**-**.com/cgi-bin/e-cron" #modify
api_authkey="YXBpOmQ1NmZmYzEwZDI4NDVlN2EyZGFjZjcwYTRhOGRjM2QzMWM1YWE1NDQK" #modify
-----------------------------------------------------------
sudo -u small-shell /usr/local/small-shell/bin/e-cron enable.job1
#-> please enable ls_tmp job acordingly
@hostB
sudo /usr/local/small-shell/adm/gen -job
#-> please create job2
sudo vi /usr/local/small-shell/util/e-cron/def/job2.def
-----------------------------------------------------------
CHEDULE:2 * * * *
exec_command="/usr/local/small-shell/util/scripts/job2.sh"
input_message="done.1"
output_message=""
hubapi="https://ec2-**-**-**-**.com/cgi-bin/e-cron" #modify
authkey="YXBpOmQ1NmZmYzEwZDI4NDVlN2EyZGFjZjcwYTRhOGRjM2QzMWM1YWE1NDQK" #modify
-----------------------------------------------------------
sudo -u small-shell /usr/local/small-shell/bin/e-cron enable.job2
#-> please enable ls_tmp2 job acordingly
Execute manually and confirm result
Then let's try to execute jobs. if you execute job2@hostB first, job will not be started because message is not set by job1.
@hostB
sudo -u small-shell /usr/local/small-shell/bin/e-cron exec.job2
#-> job will not be started more than 1 minutes. then please exit Ctrl + C
sudo -u small-shell /usr/local/small-shell/bin/e-cron stat.job2
--------------------------------------------------------
LATEST STATUS OF JOB
filter: job2
2022-01-07 14:18:12 job2 INFO input_message_done.1_was_not_set
@hostA
sudo -u small-shell /usr/local/small-shell/bin/e-cron exec.job1
@hostB
sudo -u small-shell /usr/local/small-shell/bin/e-cron exec.job2
sudo -u small-shell /usr/local/small-shell/bin/e-cron stat
--------------------------------------------------------
LATEST STATUS OF JOB
filter: job2
--------------------------------------------------------
2022-01-07 14:19:12 job2 successfully completed
DATA (file) exchange
e-cron can be dataExchange platform between several hosets . dataExchange also needs Base APP, please genenerate it on HUB server beforehand. #see APP shell tour for further details. Once API HUB is ready, you can push or get the data through the HUB.
---------------------------------
| API Server #DataExchange | HUB dir: /usr/local/small-shell/util/e-cron/que/file
---------------------------------
|
| (http/https)
|
Edge servers {hostsA,B,C,D} # push or get files through HUB
confirm base information at API HUB server
@hub
sudo cat /usr/local/small-shell/web/base | grep hubapi
hubapi="https://ec2-**-**-**-**.com/cgi-bin/e-cron"
sudo cat /usr/local/small-shell/web/base | grep auth
api_authkey="YXBpOmE2ODcyODI2MGZlOGRmOTEzZjNlMmUyYjM2YzlmZD****"
DataExchange Job
This is exmpale of exchange dummy file from host1 to host2 through HUB API.
# data_push@hostA ---> dummy file ---> data_get@hostB
! file will not be cleared even if it's synced to other hosts
Job for pushing file
This is exmaple of generating job that will push dummy.log to HUB API server.
sudo /usr/local/small-shell/adm/gen -job
# -> diarlog will be started
generate file exchange job
Note
Please use "*" to file name if you want to upload or download several files at one.
e.g.) access.log*
@hostA
# generate job
sudo /usr/local/small-shell/adm/gen -job
Job Name: data_push
Type of job (1.job automation | 2.file exchange) : 2
--Define schedule--
Month [ any | 1-12 ]: any
Date [ any | 1-31 ]: 1
Hour [ any | 0-23 ]: 1
Min [ any | 0-59 ]: 0
Week [ any | mon - sun ]: any
Type of file exchange (push | get): push
local directory: /tmp
file_name: dummy*
HUB API URL: https://ec2-**-**-**-**.com/cgi-bin/e-cron
API authkey: YXBpOmE2ODcyODI2MGZlOGRmOTEzZjNlMmUyYjM2YzlmZD****
# enable job
sudo -u small-shell /usr/local/small-shell/bin/e-cron enable.data_push
@hostB
# generate job
sudo /usr/local/small-shell/adm/gen -job
Job Name: data_get
Type of job (1.job automation | 2.file exchange) : 2
--Define schedule--
Month [ any | 1-12 ]: any
Date [ any | 1-31 ]: 1
Hour [ any | 0-23 ]: 0
Min [ any | 0-59 ]: 2
Week [ any | mon - sun ]: any
Type of file exchange (push | get): get
local directory: /tmp
file_name: dummy*
HUB API URL: https://ec2-**-**-**-**.com/cgi-bin/e-cron
API authkey: YXBpOmE2ODcyODI2MGZlOGRmOTEzZjNlMmUyYjM2YzlmZD****
# enable job
sudo -u small-shell /usr/local/small-shell/bin/e-cron enable.data_get
Execute manually
Create dummy file
@hostA
# create dummy file
echo "dummy" > /tmp/dummy1
echo "dummy" > /tmp/dummy2
echo "dummy" > /tmp/dummy3
Push data to HUB
# push manually
sudo -u small-shell /usr/local/small-shell/bin/e-cron exec.data_push
dummy #data is synced to
Confirm file at HUB server
@hub
ls /usr/local/small-shell/util/e-cron/que/file/
dummy1 dummy2 dummy3
Get data from HUB
# get manually
@hostB
sudo -u small-shell /usr/local/small-shell/bin/e-cron exec.data_get
ls /tmp/dummy*
dummy1 dummy2 dummy3 #data is synced to local dir
Note
Once it's downloaded, file will be gone.
General queing
You can pass parameters using general que to the script that will be executed by e-cron. This function is useful for such like sending mails.
queing dir
dir: /usr/local/small-shell/util/que/common
function
Script can load parameters that is in the que file. For examaple mailque contain subjects, mailto, mailfrom, contents.
push -> que: mail -> mail[ subject="XXX", mailto="XXX", mailfrom="XXX", contetns="XXX" ].randomID
script <- wrapper {que: mail} <- mail[ subject="XXX", mailto="XXX", mailfrom="XXX", contetns="XXX" ].randomID.load
Push mail que
# send mail que to e-cron batch
session="`date +%s`.$RANDOM"
mkdir /tmp/$session
echo "subject=\"test1\"" > /tmp/$session/mail
echo "mailto=\"to@***.com\"" >> /tmp/$session/mail
echo "mailfrom=\"from@***.com\"" >> /tmp/$session/mail
echo "contents=\"test1\"" >> /tmp/$session/mail
echo "subject=\"test2\"" > /tmp/$session/mail
echo "mailto=\"to@***.com\"" >> /tmp/$session/mail
echo "mailfrom=\"from@***.com\"" >> /tmp/$session/mail
echo "contents=\"test2\"" >> /tmp/$session/mail
sudo -u small-shell /usr/local/small-shell/bin/e-cron push.que:/tmp/$session/mail
#->>pushed successfully
rm -rf /tmp/$session
Note
Plese setup MTA and mail command beforehand by yourself.
Create mail batch
sudo vi /usr/local/small-shell/util/scripts/send_mail
--code--
#!/bin/bash
# load params from que
mailto="$1"
mailfrom="$2"
subject="$3"
contents="$4"
# send mail
echo "$contents" | mail -s "$subject" -r $mailfrom $mailto
-------
Add batch to e-cron automation
chowm small-shell:small-shell /usr/local/small-shell/util/scripts/send_mail
chmod 755 /usr/local/small-shell/util/scripts/send_mail
# ADD AUTOMATION JOB
sudo /usr/local/small-shell/adm/gen -job
#-> diaglog start
job name: mailq
Type of job (1.job automation | 2.file exchange) : 1
--Define schedule--
Month [ any | 1-12 ]: any
Date [ any | 1-31 ]: any
Hour [ any | 0-23 ]: any
Min [ any | 0-59 ]: any
Week [ any | mon - sun ]: any
Exec command or batch script: send_mail "$mailto" "$mailfrom" "$subject" "$contents"
#-> dialog end
# update def
sudo vi /usr/local/small-shell/util/e-cron/def/mailq.def
--code--
SCHEDULE:* * * * *
exec_command="/usr/local/small-shell/util/scripts/send_mail \"$mailto\" \"$mailfrom\" \"$subject\" \"$contents\""
input_message=""
output_message=""
hubapi=""
authkey=""
# add manually
que="mail"
--------
enable job
sudo -u small-shell /usr/local/small-shell/bin/e-cron enable.mailq
Check log
tail -f /usr/local/small-shell/util/e-cron/log/joblog/mailq_command_dump_20210918
2021-09-18 01:28:01 /usr/local/small-shell/util/scripts/send_mail "to@***.com" "from@***.com" "test1" "test1" executed
2021-09-18 01:28:01 /usr/local/small-shell/util/scripts/send_mail "to@***.com" "from@***.com" "test2" "test2" executed
bin/e-cron
e-cron must be executed as small-shell permission
sudo -u small-shell /usr/local/small-shell/bin/e-cron
| param | description | usage |
|---|---|---|
| cat.$job | show job definition | e-cron cat.jobA |
| stat | show job status | e-cron stat |
| stat.$filter | show filtered job status | e-cron stat.ERROR |
| ls | show job list | e-cron ls |
| enable.$job | enable job | e-cron enable.jobA |
| disable.$job | disable job | e-cron disable.jobA |
| exec.$job | execute job manually | e-cron exec.jobA |
| clear.$job | clear job status | e-cron clear.jobA |
Util scripts for e-cron
Util scripts are provided on following directory.
ls /usr/local/small-shell/util/scripts
Usage is written in the shell header. please check more command.
bat_import.sh # import datasets as batch job
del_datasets.sh # delete datasets as batch job
del_log.sh # delete log of e-cron
del_job.sh # delete job of e-cron
del_session.sh # delete session of DATA shell
del_statistics.sh # delete statistics of pyshell
countup.sh # please see detail at pyshell tour
sumup.sh # please see detail at pyshell tour
Verified environment
API server
OS: Utuntu 2X (recommnded), Debian 1X, macOS Monterey, CentOS 7, RHEL 8
bash: 4.X
curl: 7.X
IoT Edge
OS: raspberrypi 4.X