Publi

Notifica, logea y enriquece tu experiencia de trabajo en Bash con este script

En nuestro trabajo diario peleando con sesiones de terminal hay ocasiones en las que, teniendo una sesión de terminal abierta, no sabemos a qué hora se ejecutó un comando determinado. O acabamos de iniciar una tarea que tiene pinta de ser muy larga y nos gustaría que el ordenador nos avisara cuando termine para no estar mirando cada poco tiempo. Además, seguro que a ti también te ha pasado, te acuerdas de que necesitas el aviso cuando la tarea está iniciada y no puedes pararla.

Pequeña introducción

No pretendo crear un sistema muy complejo para este propósito, para eso tenemos auditd, del que hablaré en próximos posts. Este es un pequeño sistema que consume pocos recursos y se dedica a:

  • Escribir en el log de sistema los comandos que se van ejecutando cuando concluyen.
  • Informar en la ventana de nuestra terminal de la hora que es, de lo que ha tardado en ejecutar un cierto comando y la carga del sistema en ese momento. Podremos configurarlo y mostrar más cosas.
  • Notificar con un programa externo cuando una orden ha finalizado. Ya sea por medio de notificación de escritorio, ventana emergente, destacando la ventana de terminal, o incluso enviando un webhook, ya depende de nosotros.

Podemos ver, tras la finalización de un comando que ha tardado más de 2 segundos (por ejemplo, comando_largo) el siguiente mensaje, notificación:

Además, como ha tardado más de 10 segundos (los tiempo podremos configurarlos), veremos lo siguiente en el escritorio:

Por supuesto, podemos elegir desactivar/activar las notificaciones, o cómo se va a notificar desde otra ventana mientras la tarea está en ejecución.

El script

Pongo aquí una primera versión del script. Ya que se me ocurren muchas mejoras, y pequeños cambios que podemos hacer para enriquecer la experiencia aún más.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
#!/bin/bash

readonly BASHISTANT_DIR=$HOME/.bashistant
readonly BASHISTANT_LOCK=$BASHISTANT_DIR/lock
readonly BASHISTANT_LOCKFD=99

BASHISTANT_COLOR_ENABLE=1
BASHISTANT_INFO_ENABLE=1
BASHISTANT_NOTIFY_ENABLE=1
BASHISTANT_LOG_ENABLE=1
BASHISTANT_TIMER_COLOR='34'
BASHISTANT_NOTIFY_TIME=10               # If command lasts more than 10 seconds, notify
BASHISTANT_NOTIFY_COMMAND="@default"
BASHISTANT_SHOW_TIME=2
BASHISTANT_NOW_FORMAT="%d/%m/%Y %H:%M:%S"
BASHISTANT_NOW_COLOR='35'
BASHISTANT_INFO_ALIGN="right"
BASHISTANT_INFO_PADDING="  "
BASHISTANT_ELAPSED_COLOR='36'
BASHISTANT_LOAD_COLOR='38'

BASHISTANT_INFO_FORMAT="[ %NOW | %ELAPSED | %CPULOAD ]"

_BASHISTANT_START=

readonly MYPID=$$

MYPIDS=()

function onexit() {
        flock -u BASHISTANT_LOCKFD
        [ ! -r "$BASHISTANT_LOCK" ] || rm -f "$BASHISTANT_LOCK"
        echo "Ocurrió un problema y el programa se cerró inesperadamente" >2
        logger "Bashistant: There was a problem here"
}

function __bashistant_init() {
        [ -d "$BASHISTANT_DIR" ] || mkdir "$BASHISTANT_DIR"
        eval "exec $BASHISTANT_LOCKFD>"$BASHISTANT_LOCKFD"";
        readonly WINDOWPID=$(ps -o ppid,pid| grep $$ | awk '{print $1;exit}')
        if xset q &>/dev/null && hash xdotool; then
                readonly WINDOWID=$(xdotool search --pid $WINDOWPID | tail -1)
        fi
        COMMANDS=()
}

__bashistant_init

function __bashistant_get_timestamp() {
        date +%s
}

function __bashistant_print_info() {
        local INFO="${BASHISTANT_INFO_PADDING}$1"

        local INFO_NOCOLOR="$(echo -e "$INFO" | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g")"

        if [ "
$BASHISTANT_INFO_ALIGN" = "right" ]; then
                echo -ne "
\033[${COLUMNS}C"
                echo -ne "
\033[${#INFO_NOCOLOR}D"
        fi
        if [ -n "
$BASHISTANT_COLOR_ENABLE" ] && [ $BASHISTANT_COLOR_ENABLE -eq 1 ]; then
                echo -e "
${INFO}"
        else
                echo -e "
${INFO_NOCOLOR}"
        fi
}

function __bashistant_set_color() {
        echo "
\033[${1}m"
}

function __bashistant_show_info() {
        local ELAPSED="
$1"

        if [ $ELAPSED -ge $BASHISTANT_SHOW_TIME ]; then
                local SHOWTIME="
"
                for elem in $BASHISTANT_INFO_FORMAT; do
                        SHOWTIME+="
"
                        case $elem in
                                "
%NOW")
                                        local NOW="
$(date +"${BASHISTANT_NOW_FORMAT}")"
                                        SHOWTIME+="
$(__bashistant_set_color $BASHISTANT_NOW_COLOR)${NOW}\033[0m"
                                        ;;
                                "
%ELAPSED")
                                        local ELTIME
                                        if [ $ELAPSED -eq 0 ]; then
                                                ELTIME="
0s"
                                        else
                                                ELTIME="
$((ELAPSED/86400))d $(date -ud@"$ELAPSED" "+%Hh %Mm %Ss")"
                                                ELTIME="
$(echo " $ELTIME" | sed -e 's/[[:space:]]00\?[dhms]//g' -e 's/^[[:space:]]*//')"
                                        fi
                                        SHOWTIME+="
$(__bashistant_set_color $BASHISTANT_ELAPSED_COLOR)$ELTIME\033[0m"
                                        ;;
                                "
%CPULOAD")
                                        local LOAD="
$(cat /proc/loadavg | awk '{ print $1 }')"
                                        SHOWTIME+="
$(__bashistant_set_color $BASHISTANT_LOAD_COLOR)$LOAD\033[0m"
                                        ;;
                                *)
                                        SHOWTIME+=$elem
                        esac
                done

                __bashistant_print_info "
$SHOWTIME"
        fi

}

function __bashistant_log_info() {
        local ELAPSED=$1
        local COMMAND="
$2"

        logger -t "
Bashistant" -i --id=$$ "($(id -un)) Time: ${ELAPSED}sec Command: $COMMAND"
}

function __bashistant_desktop_notification() {
        local COMAND="
$2"
        local ELAPSED="
$1"
        local MSG="
$3"
        if [ -z "
$MSG" ]; then
                MSG="
Comando finalizado: "$COMMAND" en $ELAPSED segundos"
        fi
        notify-send "
$MSG"
}

function __bashistant_zenity_notification() {
        local COMAND="
$2"
        local ELAPSED="
$1"
        local MSG="
$3"

        if [ -z "
$MSG" ]; then
                MSG="
Comando finalizado: "$COMMAND" en $ELAPSED segundos"
        fi
        echo zenity --info --width=300 --title="
Tarea finalizada" --text="$MSG"
}


function __bashistant_bringtofront_notification() {
        if [ -n "
$WINDOWID" ]; then
                xdotool windowactivate $WINDOWID
        fi
}


function __bashistant_notify_info() {
        local ELAPSED=$1
        local COMMAND="
$2"

        if [ $ELAPSED -ge $BASHISTANT_NOTIFY_TIME ]; then
                flock -x $BASHISTANT_LOCKFD
                NOTIFY="
$(cat $BASHISTANT_DIR/notify 2>/dev/null)"
                flock -u $BASHISTANT_LOCKFD
                rm -f "
$BASHISTANT_LOCK"
                NOTIFY="
${NOTIFY//%ELAPSED%/$ELAPSED}"
                NOTIFY="
${NOTIFY//%COMMAND%/$COMMAND}"
                NOTIFY="
${NOTIFY//%USER%/$(id -un)}"
                NOTIFY="
${NOTIFY//%HOSTNAME%/$(hostname)}"

                while read notifycommand; do
                        if [ -n "
$notifycommand" ]; then
                                declare -a "
ncommand=($notifycommand)"
                                case ${ncommand[0]} in
                                        "
@notify")
                                                __bashistant_desktop_notification "
$ELAPSED" "$COMMAND" "${ncommand[@]:1}"
                                                ;;
                                        "
@zenity")
                                                __bashistant_zenity_notification "
$ELAPSED" "$COMMAND" "${ncommamd[@]:1}"
                                                ;;
                                        "
@bringtofront")
                                                __bashistant_bringtofront_notification
                                                ;;
                                        *)
                                                "
${ncommand[@]}"
                                esac
                                unset ncommand
                        fi
                done <<< $NOTIFY

        fi
}

function notify() {
        local ARGUMENT="
$1"

        if [ -z "
$ARGUMENT" ]; then
                cat $BASHISTANT_DIR/notify 2>/dev/null
        else
                echo "
$ARGUMENT" > $BASHISTANT_DIR/notify
                echo "
Notificación definida con éxito"
        fi
}

function postcmd() {
    if [ "
${#COMMANDS[@]}" -gt 1 ]; then
                HISTORY=$(history 1)
                COMMAND="
${HISTORY:7}"
                if [ -z "
$_BASHISTANT_START" ]; then
                        # No start info
                        return
                fi
                local END=$(__bashistant_get_timestamp)
                local ELAPSED=$(($END - $_BASHISTANT_START))
                if [ -n "
$BASHISTANT_INFO_ENABLE" ] && [ $BASHISTANT_INFO_ENABLE -eq 1 ]; then
                        __bashistant_show_info "
$ELAPSED"
                fi

                if [ -n "
$BASHISTANT_LOG_ENABLE" ] && [ $BASHISTANT_INFO_ENABLE -eq 1 ]; then
                        __bashistant_log_info "
$ELAPSED" "$COMMAND"
                fi

            if [ -n "
$BASHISTANT_NOTIFY_ENABLE" ] && [ $BASHISTANT_NOTIFY_ENABLE -eq 1 ]; then
                        __bashistant_notify_info "
$ELAPSED" "$COMMAND"
                fi
fi;
    COMMANDS=();
    trap 'precmd' debug

}

function precmd() {
        if [ ${#COMMANDS[@]} -eq 0 ]; then
                _BASHISTANT_START=$(__bashistant_get_timestamp)
                #echo "
INICIA EJECUCIÓN: "$BASH_COMMAND
        fi
        COMMANDS+=("
$BASH_COMMAND");
}

readonly PROMPT_COMMAND="
postcmd"
trap 'precmd' debug

En principio el archivo lo llamé bashistant.sh (que viene de Bash Assistant, todo mezclado). Y quiero modificarlo un poco para integrarlo en los gscripts.

Activación

Para poder utilizar este script automáticamente en nuestras sesiones de terminal, podemos editar nuestro archivo ~/.bashrc y añadir la siguiente línea (cambiando la ruta del archivo por la adecuada en tu sistema):

1
source $HOME/gscripts/bashistant.sh

También podemos utilizar los archivo $HOME/.profile o /etc/profile. El último para instalar a nivel de sistema para todos los usuarios.
En principio se creará el directorio .bashistant en tu $HOME para almacenar información sobre las notificaciones, aunque en el futuro se utilizará para más cosas. La inicialización no es muy pesada. Aparte de crear el directorio mencionado anteriormente, obtenemos el ID del proceso emulador de terminal (konsole, xfce4-terminal, gnome-terminal…), y si estamos en un entorno gráfico, obtiene el ID de la ventana que lo gobierna, para resaltar la ventana cuando no nos encontramos visualizándola.

Log de comandos

Esta es la parte menos currada por el momento, se limita a llamar a logger con el comando una vez finalizado. Podemos ver un fragmento de /var/log/syslog aquí:

Dec 18 14:18:04 gaspy-ReDDraG0N Bashistant[25301]: (gaspy) Time: 0sec Command: cat pkey.pem
Dec 18 14:18:50 gaspy-ReDDraG0N Bashistant[25301]: message repeated 2 times: [ (malvado) Time: 4sec Command: rm -rf archivos_confidenciales]
Dec 18 14:43:48 gaspy-ReDDraG0N Bashistant[25301]: (gaspy) Time: 578sec Command: find -name ‘confidencial’
Dec 18 16:24:34 gaspy-ReDDraG0N Bashistant[10252]: (gaspy) Time: 0sec Command: pgrep -f apache

Y esto podría delatar al usuario malvado, que normalmente no debe tener permiso para editar logs ni nada parecido.

Este log podemos desactivarlo haciendo:

BASHISTANT_LOG_ENABLE=0

Si queremos pillar a alguien que ha ejecutado comandos malignos (que no es el cometido de este script), podríamos desactivar esta característica en el código.

Información tras la ejecución

En realidad esto lo encontré en el GitHub de Chuan Ji mientras buscaba información y me gustó la visualización que hacía tras cada comando. No le copié el código, como vemos, su proyecto tiene más precisión midiendo el tiempo. A mí, para Bash, no me interesaba tener demasiada precisión en ello. Pero además, quise completarlo y hacerlo algo más personalizable.

Para modificar el comportamiento de esta característica tenemos una serie de variables que podremos modificar desde nuestro terminal:

  • BASHISTANT_INFO_ENABLE=[0,1] : Activa o desactiva esta característica
  • BASHISTANT_SHOW_TIME=[n] : Número de segundos que debe tardar la tarea para mostrar esta línea. Podemos hacer que si un comando tarda demasiado poco no se muestre nada, o 0 si queremos que se muestre siempre.
  • BASHISTANT_NOW_FORMAT=»%d/%m/%Y %H:%M:%S»: Formato de fecha y hora (se usa el comando date.
  • BASHISTANT_NOW_COLOR=[color]: Código ANSI del color para mostrar la fecha y hora actuales.
  • BASHISTANT_INFO_ALIGN=[left|right] : Alineación del texto de información (izquierda o derecha).
  • BASHISTANT_INFO_PADDING=» «: Texto separador para que el recuadro no esté pegado al borde de la pantalla.
  • BASHISTANT_ELAPSED_COLOR=[color]: Código de color para el tiempo transcurrido en la ejecución.
  • BASHISTANT_LOAD_COLOR=[color]: Código de color para la carga del sistema.
  • BASHISTANT_INFO_FORMAT=»[ %NOW | %ELAPSED | %CPULOAD ]»: Formato por el que se muestra la información.

Después de unos días de uso se ha convertido en una buena herramienta sobre todo para determinar de un vistazo cuánto tiempo llevas trabajando en un servidor o necesitas saber cuánto ha tardado una tarea que has olvidado cronometrar. Sí, en ocasiones, lanzamos tareas como mysqldump, restauraciones de base de datos, instalaciones, orquestaciones de un sistema, etc. En definitiva, tareas que pueden llegar a tardar incluso varias horas y que, muchas veces te interesa saber cuánto tiempo han necesitado esas tareas, por ejemplo para hacer documentación. Pero cuando te acuerdas de que deberías cronometrarlo (bastaría con ejecutar un comando poniendo time delante), es cuando el proceso ha terminado, puede llevar ya una hora y no vas a cancelar el proceso a estas alturas… espero no ser el único al que le pasa esto 🙂

Configurar notificaciones

El archivo $HOME/.bashistant/notify lo podremos modificar con un editor de textos para introducir el comando que queremos que se ejecute para notificar la finalización de la orden de Bash. Igual que en el punto anterior, muchas veces puedo lanzar un comando que necesitará mucho tiempo para finalizar y me pongo a hacer otra cosa. Es en esos casos en los que olvido esa primera tarea que dejé haciéndose y puede pasar mucho tiempo hasta que me acuerdo de ella. Una solución rápida, sería ejecutar el comando así:

time comando_largo ; zenity --info --text=»El comando ha terminado»

Pero, como siempre, se me olvida hacer esto cuando voy a ejecutar la orden. Otra opción sería:

comando largo
^Z
fg ; zenity --info --text=»El comando ha terminado»

Es decir, una vez hemos lanzazdo el comando, pulsamos Control+Z para pausarlo y luego utilizamos fg para reanudarlo, haciendo que una vez reanudado se ejecute zenity para notificar su finalización. Aunque muchas veces no se pueden pausar los comandos sin cargarnos algo.

Así que este script, cuando termina la ejecución de la orden, mirará el archivo ~/.bashistant/notify para ver qué comando tiene que ejecutar. Lo que significa que, aunque la tarea esté en ejecución puedo modificar el contenido de ese archivo, que cuando el comando termine se leerá y se ejecutará lo que ahí diga. Para agilizar un poco más la definición de notificaciones podemos utilizar en el mismo comando de notificación las siguientes palabras clave:

  • %ELAPSED%: Para mostrar el tiempo empleado en segundos.
  • %COMMAND%: Para mostrar el comando que acaba de finalizar.
  • %USER%: Para mostrar el usuario que ha ejecutado el comando
  • %HOSTNAME%: Para mostrar el hostname del equipo

Por lo que el archivo ~/.bashistant/notify quedaría así:

1
zenity --info --text="El comando %COMMAND% ejecutado por %USER%@%HOSTNAME% ha finalizado en %ELAPSED% segundos."

Además, disponemos de algunos comandos rápidos como:

  • @notify : Para ejecutar notify-send
  • @zenity : Para generar un diálogo ded zenity
  • @bringtofront : Para traer al primer plano el terminal

Que permiten que ~/.bashistant/notify contenga:

1
@notify

Para realizar la función.

También podemos utilizar el comando notify para definir el comando de notificación. Por lo que, podemos ejecutar un comando muy largo en un terminal, y luego desde otro sitio (con el mismo usuario), hacer:

notify @bringtofront

Así, cuando termine el primer comando (el que nos lleva mucho tiempo), se traerá a primer plano la ventana de terminal desde la que se disparó.

Por otro lado, como sería muy pesado estar todo el rato viendo notificaciones de comandos terminados, ya que un simple cd o ls dispararía la notificación tenemos las variables:

  • BASHISTANT_NOTIFY_ENABLE=[0|1]: Que podemos usar para activar o desactivar la característica.
  • BASHISTANT_NOTIFY_TIME=[n]: Con la que podemos decir el tiempo mínimo para que una tarea se notifique. Por defecto vale 10, quiere decir que si una tarea lleva menos de 10 segundos, no disparará la notificación.

Más posibilidades

En el comando de notificaciones podríamos, por ejemplo, reproducir un sonido, o una voz, o generar una línea con cURL que dispare un webhook de slack, gitlab o cualquier otra aplicación. Podemos programar el envío de un e-mail. O incluso podemos ejecutar varios comandos, uno por línea.

Si miráis el código podéis ver que hay ciertas cosas que no se usan, por ahora, como la captura de los comandos justo antes de que se ejecuten, o la variable MYPIDS reservada para una futura sorpresa en la siguiente versión. Eso sí, estoy abierto a sugerencias y, por supuesto, os invito a contribuir con este pequeño script.

Foto principal: unsplash-logoMathew Schwartz

También podría interesarte....

There are 96 comments left Ir a comentario

  1. Jose Albert (Linux Post Install) /
    Usando Google Chrome Google Chrome 71.0.3578.99 en Android Android 6.0.1

    Excelente, te felicito.

  2. https://shinyessays.com/blog /
    Usando Google Chrome Google Chrome 71.0.3578.98 en Windows Windows 7

    This is a small system that consumes few resources and is designed for a very large range of purposes.

  3. essays editors /
    Usando Google Chrome Google Chrome 72.0.3626.121 en Windows Windows NT

    So, we can easily and simply execute a very long command in the terminal, and then from another site that we need.

  4. Jimmy Olano /
    Usando Mozilla Firefox Mozilla Firefox 74.0 en Linux Linux

    Una interesante herramienta para los administradores de sistema. Ya usamos Powerline para diferenciar máquinas locales de máquinas remotas; le veo buenas posibilidades a este proyecto bash.

    1. Gaspar Fernández / Post Author
      Usando Google Chrome Google Chrome 79.0.3945.79 en Ubuntu Linux Ubuntu Linux

      Muchas gracias Jimmy. Pues fíjate que llevo ya más de un año (cerca de dos) utilizando este script a diario y me resulta tremendamente útil. No es tanto para auditoría (porque te podrías saltar su funcionamiento sin problemas) como para controlar tú mismo tu trabajo, ver lo que tardan algunos procesos. De esas veces que dejas importando una base de datos y cuando vuelves ya está importada, y en ese momento te preguntas, ¿cuánto ha tardado?

      Muchas gracias por el comentario.

  5. minecraft classic /
    Usando Google Chrome Google Chrome 81.0.4044.113 en Windows Windows NT

    I feel what you put in the article and understand your thoughts. The information you give me is very good, thank you very much for sharing it.

  6. connect 4 /
    Usando Google Chrome Google Chrome 116.0.0.0 en Windows Windows NT

    y pequeños cambios que podemos hacer para enriquecer la experiencia aún más.

  7. Johnathon Dave /
    Usando Google Chrome Google Chrome 85.0.4183.83 en Windows Windows NT

    Thanks for sharing this helpful Info! It’s great to have easy access to valuable resources like this.»I really appreciate it.

  8. personal training services in Brisbane /
    Usando Google Chrome Google Chrome 116.0.0.0 en Mac OS X Mac OS X 10.15.7

    It’s great to hear that you’re looking for solutions to streamline your workflow

  9. LOL BEANS /
    Usando Google Chrome Google Chrome 116.0.0.0 en Windows Windows NT

    Thanks for a very interesting blog. What other kind of information could I get written in such a perfect approach lolbeans? I have a project that I’m currently working on, and I’ve been on the lookout for such information.

  10. prds /
    Usando Google Chrome Google Chrome 116.0.0.0 en Windows Windows NT

    It’s actually a nice and helpful piece of info. I’m happy that you shared this helpful information with us. Please stay us up to date like this.Thank’s for sharing.

    Here is My Homepage: OKBet manila

  11. Math Homework Help /
    Usando Google Chrome Google Chrome 100.0.4896.60 en Linux Linux

    I have read your blog. Your blog is very nice and useful !!
    If you are struggling with your math homework then do not worry,

  12. lily jane /
    Usando Google Chrome Google Chrome 116.0.0.0 en Windows Windows NT

    I am overjoyed to have found out about this helpful website. It teaches me a lot of intriguing information about everything that’s going on, especially the subject matter of the article that was just before this one. globle game

  13. jsimit seo /
    Usando Google Chrome Google Chrome 116.0.0.0 en Windows Windows NT

    Enough, the article is to a great degree the best point on this registry related issue. I fit in with your decisions and will vivaciously imagine your next updates. เกมสล็อตโรม่า

  14. jsimit seo /
    Usando Google Chrome Google Chrome 116.0.0.0 en Windows Windows NT

    Can pleasantly compose on comparative themes! Welcome to here you’ll discover how it should look. สล็อตเปิดใหม่

  15. 먹튀검증 /
    Usando Google Chrome Google Chrome 116.0.0.0 en Windows Windows NT

    This article provides a bright light that we can use to observe reality. This is a very good one and provides in-depth information.먹튀검증

  16. L. Martin /
    Usando Google Chrome Google Chrome 116.0.0.0 en Windows Windows NT

    Outstanding beat! I’d like to learn while you update your website.
    credit repair colorado springs

  17. 메이저사이트 /
    Usando Google Chrome Google Chrome 116.0.0.0 en Windows Windows NT

    Thank you very much for your post. I like your post. All the content you share with us is the latest and rich. Great post, great website. Thank you for your information!메이저사이트

  18. Usando Google Chrome Google Chrome 109.0.0.0 en Windows Windows NT

    I just found this blog and have high hopes for it to continue. Keep up the great work, its hard to find good ones. I have added to my favorites. Thank You 레모나주소

  19. 먹튀신고 /
    Usando Google Chrome Google Chrome 116.0.0.0 en Windows Windows NT

    It’s great to publish such a beneficial website. Your online logs are not only useful, but also very creative. Not many people are able to write such a simple and artistic post. Continue to write well.먹튀신고

  20. Fauxjacket /
    Usando Google Chrome Google Chrome 116.0.0.0 en Windows Windows NT

    The best faux leather jackets are the ones that actually make you happy. And our effort is to provide you with faux leather jacket yellow. So visit our site now.

  21. 멕시코 야구 중계 /
    Usando Google Chrome Google Chrome 109.0.0.0 en Windows Windows NT

    wonderful points altogether, you just received a new reader. 멕시코 야구 중계

  22. drywall /
    Usando Google Chrome Google Chrome 116.0.0.0 en Windows Windows NT

    Looking forward to trying it out! Thanks for sharing.

  23. 토토사이트 /
    Usando Google Chrome Google Chrome 116.0.0.0 en Windows Windows NT

    An exciting and rich article. You always share beautiful things with us. Thank you. Continue writing articles like this. Thank you for providing the information, which is really helpful.토토사이트

  24. James Broderick /
    Usando Google Chrome Google Chrome 116.0.0.0 en Windows Windows NT

    I am really glad I got to read this. I want to look at this more when I am done with my job at Tree Removal West Palm Springs Florida

  25. Taylor Swift Chiefs Jacket /
    Usando Google Chrome Google Chrome 116.0.0.0 en Windows Windows NT

    «Your dedication to this cause is inspiring. Your work is making a real difference in the world and I am grateful to be a part of it.» Taylor Swift Chiefs Jacket

  26. SimonWhitehead /
    Usando Google Chrome Google Chrome 117.0.0.0 en Windows Windows NT

    First You got a great blog .I will be interested in more similar topics. i see you got really very useful topics, i will be always checking your blog thanks. Dominican Republic travel

  27. 메이저사이트 /
    Usando Google Chrome Google Chrome 116.0.0.0 en Windows Windows NT

    How amazing this article is. In my opinion, every paragraph is well explained so I could easily understand. I can also share my opinion here. 메이저사이트

  28. Loki S02 Tom Hiddleston Brown Peacoat /
    Usando Google Chrome Google Chrome 116.0.0.0 en Windows Windows NT

    Keep this going, please. Great job!

  29. boL16 /
    Usando Google Chrome Google Chrome 116.0.0.0 en Windows Windows NT

    «Thanks for sharing this interesting blog here
    Tampa Fencing«

  30. Jamie Applebee /
    Usando Google Chrome Google Chrome 117.0.0.0 en Windows Windows NT

    I was able to use this great affect. thank you for this. Jamie of Stump removal Toledo Ohio

  31. Novak Djokovic 24 Jacket /
    Usando Google Chrome Google Chrome 117.0.0.0 en Windows Windows NT

    I appreciate that you address both the pros and cons of different approaches. It shows your commitment to providing a balanced perspective and helping your readers make informed decisions. Novak Djokovic 24 Jacket

  32. SimonWhitehead /
    Usando Google Chrome Google Chrome 117.0.0.0 en Windows Windows NT

    When you use a genuine service, you will be able to provide instructions, share materials and choose the formatting style. Puerto Rico all inclusive resort

  33. 메이저놀이터 /
    Usando Google Chrome Google Chrome 117.0.0.0 en Windows Windows NT

    I really like your writing style and incredible data. Thank you for posting메이저놀이터

  34. Anya /
    Usando Google Chrome Google Chrome 117.0.0.0 en Windows Windows NT

    Witty! I’m bookmarking you site for future use.
    web design tampa

  35. emblinds /
    Usando Google Chrome Google Chrome 117.0.0.0 en Windows Windows NT

    This is a great post! I will save this for future reference.

  36. 메이저사이트 /
    Usando Google Chrome Google Chrome 117.0.0.0 en Windows Windows NT

    Undoubtedly, the speed is uncomfortable, and if you support us, it’s like the usual situation indoors.메이저사이트

  37. Local citations /
    Usando Google Chrome Google Chrome 117.0.0.0 en Windows Windows NT

    It helps a lot in my work. From the introduction to the end, it was explained well. The first version of the scripts you elaborate well.

  38. 메이저사이트 /
    Usando Google Chrome Google Chrome 117.0.0.0 en Windows Windows NT

    It’s easy to say that this article is actually the best topic for this registry related issue. I agree with your conclusion and eagerly look forward to your next update메이저사이트

  39. 토토시큐리티 /
    Usando Google Chrome Google Chrome 117.0.0.0 en Windows Windows NT

    I started a homepage. I am also sharing good articles like 토토시큐리티

  40. spackle /
    Usando Google Chrome Google Chrome 117.0.0.0 en Windows Windows NT

    Hello there! I completely understand your concern about the complexity of the task at hand. However, I want to assure you that I have a solution that might interest you. In my upcoming posts, I’ll be discussing auditd, spackle and a straightforward system that won’t consume too many resources. It’s precisely what we need to get the job done.

  41. 토토사이트 /
    Usando Google Chrome Google Chrome 117.0.0.0 en Windows Windows NT

    You are actually a perfect website administrator. The website stacking speed is astonishing. Feeling like you’re making any obvious traps. Moreover, its essence is a perfect artwork.토토사이트

  42. One Technology Services /
    Usando Google Chrome Google Chrome 117.0.0.0 en Windows Windows NT

    Welcome to our nearshore software development company, is a business that provides software development services to clients in a nearby or geographically proximate location, typically within the same region or a neighboring country. This approach to outsourcing software development offers several advantages, such as reduced time zone differences, cultural similarities, and easier communication, while still often providing cost savings compared to onshore development.

  43. jsimit seo /
    Usando Google Chrome Google Chrome 118.0.0.0 en Windows Windows NT

    Endeavoring to offer significant thanks won’t simply be adequate, for the fantasti c clarity in your formed work. I will quickly get your rss channel to stay instructed of any updates. เกมสล็อตโรม่า

  44. jsimit seo /
    Usando Google Chrome Google Chrome 118.0.0.0 en Windows Windows NT

    Enough, the article is to a great degree the best point on this registry related issue. I fit in with your decisions and will vivaciously imagine your next updates. 168bet

  45. Ayan Akram /
    Usando Google Chrome Google Chrome 118.0.0.0 en Windows Windows NT

    Environmental and Industrial Solutions Co. offers a wide range of eco-friendly products to meet various industry needs. From sustainable solutions to R11 refrigerant for sale, they prioritize environmental responsibility while delivering top-notch products. Trust in their expertise for your industrial requirements.

  46. Landy /
    Usando Google Chrome Google Chrome 118.0.0.0 en Windows Windows NT

    Really love this blog! car recovery leeds

  47. ok bet /
    Usando Google Chrome Google Chrome 118.0.0.0 en Windows Windows NT

    Pretty nice post. I just stumbled upon your weblog and wanted to say that I have really enjoyed browsing your blog posts. After all I’ll be subscribing to your feed and I hope you write again soon!

    My website: online baccarat philippines

  48. Alenxander Tim /
    Usando Google Chrome Google Chrome 118.0.0.0 en Windows Windows NT

    When looking for a Real Leather Jacket, it is important to take into consideration the kind of leather, the fit, the design, and the quality of the workmanship to ensure that you choose a jacket that is tailored to your specific requirements and tastes. To further verify that you are purchasing an item of sufficient quality, you can look into businesses who are renowned for their skill in the production of leather clothing.

  49. 토토사이트 /
    Usando Google Chrome Google Chrome 118.0.0.0 en Windows Windows NT

    You make so many noteworthy points here that I have studied your article more than once. Your views coincide with my own to the greatest extent possible. This is extraordinary content for your readers.토토사이트

  50. Casanova /
    Usando Google Chrome Google Chrome 118.0.0.0 en Windows Windows NT

    Great stuff here! Thanks for this! skip hire

  51. builder16 /
    Usando Google Chrome Google Chrome 118.0.0.0 en Windows Windows NT

    Thanks for making this content so informative!
    Plastic Surgery Sacramento CA

  52. One Technology Services /
    Usando Google Chrome Google Chrome 118.0.0.0 en Windows Windows NT

    Elevate your business with our exceptional Custom Application Development Services. As industry leaders, we specialize in crafting solutions that are tailor-made to suit your unique needs. With our custom application development services, you’re not just getting software – you’re gaining a strategic tool that enhances efficiency, scalability, and user satisfaction. Embrace the benefits of solutions designed precisely to align with your vision. From optimized operations to elevated customer experiences, our custom application development services pave the way for your success. Experience the transformative power of technology in the form of solutions that are truly yours. Discover the possibilities with our Custom Application Development Services today.

  53. 먹튀신고 /
    Usando Google Chrome Google Chrome 118.0.0.0 en Windows Windows NT

    I have read your article, which is so interesting and charming. Thank you very much for sharing it먹튀신고

    1. Anna /
      Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

      Same here and I agree! sandblaster

  54. jackbeen /
    Usando Google Chrome Google Chrome 84.0.4147.89 en Windows Windows NT

    Official PCSO LOTTO RESULTS for 2D SWERTRES 6/42 6/45 6/49 6/55 6/58, Philippine PCSO lotto result history jackpot and results analysis pcso lotto shedule.

  55. giovannirocky /
    Usando Google Chrome Google Chrome 118.0.0.0 en Windows Windows NT

    I can totally relate to the challenges of working with terminal sessions on a daily basis. It’s frustrating when you have a terminal session open, and you can’t remember the exact time a particular command was executed. Or when you start a task that seems like it’s going to take forever, and you wish your computer could just give you a heads-up when it’s done, so you don’t have to keep checking it every few minutes. And don’t even get me started on those moments when you realize you needed that notification right when you started the task and can’t stop it now. It’s a real time-saver to have a solution for these situations!

    Households play a crucial role in our daily lives and in our communities. They are the foundation of our society, providing us with a sense of belonging, security, and comfort. It’s amazing to see how households come in all shapes and sizes, each unique in its own way. Check this out https://www.rockymountainoils.com/collections/household

  56. 토토사이트 /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    There is noticeably a lot to realize about this. I think you made it, so there are good features. It is an impressive and good writing. 토토사이트

  57. fnaf game /
    Usando Google Chrome Google Chrome 118.0.0.0 en Windows Windows NT

    The difficulties are laid out in great detail, and everything is extremely transparent. Without a doubt, it was educational. I find it quite handy to utilize your website. We appreciate your candor. Go ahead and check out my online portfolio. tunnel rush

  58. Kara /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    Glad to be here! Thanks for this! breakdown recovery luton

  59. Joji /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    Love this! car recovery plymouth

  60. One Technology Services /
    Usando Google Chrome Google Chrome 118.0.0.0 en Windows Windows NT

    Junior Python Developer, is an entry-level programmer who specializes in Python, a high-level, easy-to-read programming language. They work on various tasks, including coding, debugging, and maintaining software applications, websites, and databases.

  61. One Technology Services /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    Staff augmentation services, are a flexible and cost-effective way for businesses to access the specialized skills and expertise they need to meet their project goals and objectives. By partnering with a staff augmentation provider, businesses can quickly and easily add qualified professionals to their teams on a temporary or permanent basis, without the hassle and expense of traditional hiring and onboarding processes.

  62. Adam Kris /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    Hay muchos servicios de seguimiento de tareas en la nube disponibles. Estos servicios le permiten realizar un seguimiento de sus tareas desde cualquier lugar. Suika game

  63. Goody /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    Great stuff. breakdown recovery slough

  64. SgtShadow /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    It’s a lightweight solution for those who want to stay informed without the complexity of a larger system.

  65. 메이저놀이터 /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    You have put forward some good viewpoints here. I also searched for this issue on the internet메이저놀이터

  66. One Technology Services /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    Welcome to our Salesforce developer, is a software engineer who specializes in building and customizing Salesforce applications. They use their programming and declarative skills to extend and customize applications on the Lightning platform. Salesforce developers are in high demand, as Salesforce is the world’s leading customer relationship management (CRM) platform.

  67. Rachel /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    Great blog. Super love this! paver

  68. 먹튀검증 /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    Please continue to post good comments. l come every day. Please. 먹튀검증

  69. aslibaris /
    Usando Google Chrome Google Chrome 85.0.4183.83 en Windows Windows NT

    This blog is an amazing one, the contents here are always very educative and useful, the reason I make it my regular blog of visit so as to widen my knowledge every day, thanks for always sharing useful and interesting information, you can checkout this I’ve always wondered about this! Loved your advice!! Thanks so much for sharing your expertise 🙂
    Regards :
    Best PTE Test Centers, Fees, and Academic Excellence in UAE

  70. Sopranos Tracksuit /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    Thank you so much for sharing this amazing article.»Sopranos Tracksuits«

  71. Albert john /
    Usando Google Chrome Google Chrome 118.0.0.0 en Windows Windows NT

    Thank you very much for this article. It explains everything I need to know! This article is so peaceful and easy to understand.

  72. sofia noah /
    Usando Google Chrome Google Chrome 118.0.0.0 en Windows Windows NT

    It is really a nice and helpful piece of information. I’m satisfied that you simply shared this helpful info with us.

  73. Bebe /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    Nice nice blog. vehicle recovery sheffield

  74. Rae /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    Love this! sandblasting

  75. One Technology Services /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    Welcome to our In-house developers, have a number of advantages over outsourced developers. First, they have a better understanding of the company’s culture and needs. This means that they can develop software that is more tailored to the company’s specific requirements. Second, in-house developers are more likely to be committed to the company’s long-term success. This is because they are invested in the company’s growth and development.

  76. riyadh erotic massage /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    best riyadh

  77. 부천오피 /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    아주 잘 쓰여진 이야기. 저뿐만 아니라 그것을 활용하는 모든 사람에게 유용 할 것입니다. 좋은 일을 계속하십시오 – 나는 확실히 더 많은 포스트를 읽을 것입니다 인천오피

  78. Tori /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    Love this site. car recovery liverpool

  79. ufabet168 /
    Usando Google Chrome Google Chrome 107.0.0.0 en Windows Windows NT

    It’s really a great and helpful piece of info. I’m glad that you shared this helpful info with us. Please keep us up to date like this. Thanks for sharing. ufabet168

  80. adityaweighing /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    wireless weigh pads in a variety of platform sizes and capacities, to cater for every type of vehicle. There is also a choice of weight indicator to meet your requirement, our portable indicator will display the wheel, axle and total vehicle weights along with a simple print out. The portable indicator is completely customizable, you can store your customer database, product database and vehicle database. This indicator is very popular with vehicle modification engineers due to the centre of gravity feature.
    visit us
    https://www.adityaweighing.com/product-Wheel-and-Axle-Weighing-Systems.html

  81. Kaabir Singh /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    Indira Nagar escort is all set to satisfy these gusty feelings. Call Girls in Indira Nagar You would never get a reason to reject our young college girl escorts because they can fascinate anybody by offering seductive services.

  82. 토토사이트추천 /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    Great article. I personally enjoy this article. An interesting article. Thank you for sharing. Hi, I’m just checking if you mind posting a comment.토토사이트추천

  83. ok bet /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    It’s actually a nice and helpful piece of info. I’m happy that you shared this helpful information with us. Please stay us up to date like this.Thank’s for sharing.

    Here is My Homepage: how to play bingo online

  84. Jason kelce letterman jacket /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    «I really enjoyed the way you approached this topic. Your writing is always a joy to read, and your lighthearted tone really made this post stand out.» Jason kelce letterman jacket

  85. builder16 /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    Great blog! Thanks for the share.
    Landscaping Edmonton

  86. 메이저놀이터 /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    This is such a beautiful story, although it is rugged, the result is kind and beautiful, and now it has become a tradition passed down by every generation. I like every performance you do for it.메이저놀이터

  87. 영화 다시보기 /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    The article looks magnificent, but it would be beneficial if you can share more about the suchlike subjects in the future. Keep posting. 영화 다시보기

  88. sheetrock insulation /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    Looking forward to learning more about this system and its implementation in future posts! Happy 1-year anniversary! 🎉

  89. horse racing betting system /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    It’s actually a nice and helpful piece of info. I’m happy that you shared this helpful information with us. Please stay us up to date like this.Thank’s for sharing.

    Here is My Homepage: horse racing sports betting

  90. kards /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    It’s actually a nice and helpful piece of info. I’m happy that you shared this helpful information with us. Please stay us up to date like this.Thank’s for sharing.

    Here is My Homepage: baccarat online real money

  91. 메이저사이트 /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    Thanks for giving me grateful information. I think this is very important to me. Your post is quite different. That’s why I regularly visit your site. 메이저사이트

  92. betflikvip /
    Usando Google Chrome Google Chrome 107.0.0.0 en Windows Windows NT

    I recently came across your blog and have been reading along. I thought I would leave my first comment. I don’t know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often. betflikvip

  93. Call Girl Virar /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    We just not chose anyone to be a part of our Virar Call Girls Agency. As customer satisfaction is our motto the standard we have set for our working conditions are pretty high and not just anybody can achieve them. So the girls that are chosen are very special and unique. You will not find an experience like this anywhere in Virar.

  94. Callie Omalley /
    Usando Google Chrome Google Chrome 119.0.0.0 en Windows Windows NT

    Kudos from our Medicare FMO

Leave a Reply