How to keep files and directories synchronized across different devices using syncthing on Linux

Syncthing is defined as a continuous file synchronization program: it can be used to keep files and directories synchronized across different devices or “nodes”. The application uses TLS as encryption method, and it is, together with its protocol, free and open source software. When using Syncthing, our data remains on our device, and is transferred directly to the destination without relaying on a central server (peer to peer). In this tutorial we will see how to install, configure and use Syncthing on Linux.

In this tutorial you will learn:

  • How to install Syncthing on the most used Linux distributions
  • How to setup the firewall for Syncthing to work correctly
  • How to share and keep a directory synchronized across two devices
  • How to austostart the Syncthing daemon automatically on user login
syncthing-logo

Software requirements

CategoryRequirements, Conventions or Software Version Used
SystemDistribution independent
SoftwareSyncthing
OtherRoot permissions
Conventions# – requires given linux-commands to be executed with root privileges either directly as a root user or by use of sudo command
$ – requires given linux-commands to be executed as a regular non-privileged user

Installation



There are basically two methods we can use to install Syncthing in our favorite Linux distribution: we can use the version packaged in the native repositories, or we can download a tarball from the official Syncthing website. Additionally, only if using Debian or Ubuntu, we can add the official Syncthing repository as an additional software source and install the package from it.

Installing the native version

Syncthing is available in the native repositories of the most used Linux distributions, such as Fedora, Debian/Ubuntu, and Archlinux, therefore we can easily install it with our favorite package manager. This is always the recommended way to obtain software; the only problem is that, especially on distributions like Debian “stable”, the repository can contain outdated versions of the packages. To install Syncthing on Fedora, we can use the dnf package manager, and issue the following command:

$ sudo dnf install syncthing

On Debian and its many derivatives, instead, we can run:

$ sudo apt install syncthing

On Archlinux we can use pacman to install the syncthing package, which is part of the “Community” repository. Obtaining the software it’s just a matter of launching our terminal emulator and run:

$ sudo pacman -Sy syncthing

Downloading a tarball



The second method of installing Syncthing consists into downloading the dedicated Linux tarball from the downloads section of the official site. We should download the archive containing the version compatible with the architecture of our system. If we are using Debian or Ubuntu, we can also add the official repository to our software sources, and then install it using apt.

Setting up the firewall

Before we run Syncthing, for it to work correctly, we must setup some firewall rules in order to allow incoming traffic through certain ports. Recent versions of the Firewalld and Ufw firewall managers, already come with a pre-configured service for Syncthing (here with the word “service” we mean a defined set of rules). If using Firewalld, to activate the service, and thus to enable traffic through the required ports), we should run:

$ sudo firewall-cmd --permanent --add-service=syncthing && sudo firewall-cmd --reload

Notice that with the command above, since no zone is specified, the service will be added to the “default” one. If we want to add the service to a specific zone, we should specify it with the --zone option, passing the zone name as argument. To gather information about the service, and see what ports are included in it, we can run:

$ sudo firewall-cmd --info-service=custom

The output of the command above is the following:

syncthing
  ports: 22000/tcp 21027/udp
  protocols:
  source-ports:
  modules:
  destination:
  includes:
  helpers:



As we can see, the ports “involved” in the service are 22000/tcp and 21027/udp. If we are using Ufw (Uncomplicated Firewall), in order to activate the service, we should run:

$ sudo ufw allow syncthing

Using Syncthing

To start the Syncthing daemon, all we have to do is to invoke the Syncthing binary from our terminal:

$ syncthing

Syncthing comes with a web interface which can be used to manage the application, add shared directories and remote devices. Once we run the command above, the Syncthing daemon is launched, and a web browser tab is automatically opened to the 127.0.0.1:8384 address, which is where the interface can be reached.

The default Syncthing shared directory is ~/Sync; it and all the other ones which should be synchronized, are displayed on the left column of the interface. On the right column, instead, we can see information about the device Syncthing is running on, and a list of associated remote devices. In our case the list is still empty:The Syncthing web interface




To add a remote machine and share a directory with it, what we need to do is to exchange devices IDs. To check a device ID, we can click on the “Actions” button on the top right side of the web interface, and click on “Show ID”. The current device ID will be displayed together with the correspondent QRCode:

syncthing-gtk-id

For the application to work correctly the two (or more devices) should know their respective IDs. In the next section we will see how to add a remote device.

Adding a device

To start synchronizing a directory, we need to add a remote device to our configuration. For the sake of this tutorial, I will use the Linux machine where we installed Syncthing as the synchronization “source”, while the remote system will be an Android device (Syncthing can be easily installed from the Google Play Store); the two devices must be connected to the same LAN.

We click on the “Add Remote Device” button on the bottom right of the web interface. On the fist tab of the popup window that will be opened we need to enter the companion device ID, and an optional human-friendly name:



syncthing-add-device

As you can see, if auto-discover works as it should, the other Syncthing device in the LAN should be automatically discovered and reported under the “nearby devices” list. In such cases all we need to do is to click on the correspondent link, and the ID input field will be filled automatically.

On the second tab, “Sharing”, we can select what directories we want to share with the companion device. In this case we selected the default one:

syncthing-add-device

One convenient thing to do, is to check the “Auto Accept” checkbox: this way the system will not prompt us to confirm if we want to accept the synchronization of the directory offered by the remote system, but it will automatically create it.



The third tab contains “advanced settings”. Here we can choose to enter a static address for the remote device, if it has any and dynamic discovery doesn’t work. We could also choose incoming and outgoing rate limit for the device, and what data should be compressed (all data vs metadata only – the default).

syncthing-remote-status

Once done, we can click on the “Save” button. The remote device should than appear in the list on the main page. On the other device we perform the same operation (notice that on the Android app, by clicking on the “hamburger” menu, we have the option to use the web interface) and add our “source” device ID. Once the two devices are paired correctly their status should begin to sync. Once the synchronization process is finished, they will be reported as “Up to Date”:

syncthing-remote-status

At this point our (bi-directional) synchronization should be ready, and the directory will be kept in sync on the two devices.

Autostart the Syncthing daemon




To make the Syncthing daemon start automatically, we can use Systemd, which is the Linux init manager. In this case we will start it as a per-user service, so we won’t need administrative privileges. When a service is configured this way, it will be started only once the user logs in.

The first thing we want to do is to copy the systemd service file into the appropriate directory. If we installed Syncthing from the native repositories of our Linux distribution, such file should be: /usr/lib/systemd/user/syncthing.service; if we downloaded the tarball from the official Syncthing site, the file should be found inside the etc/linux-systemd/user directory.

We create the ~/.config/systemd/user directory if it doesn’t already exist, and copy the file inside it:

$ mkdir -p ~/.config/systemd/user && cp /usr/lib/systemd/user/syncthing.service ~/.config/systemd/user

Once the file is in place, we run the following command to enable the service:

$ systemctl --user enable syncthing.service

Conclusions

On this tutorial we learned how to install and configure Syncthing, a very handful application which let us keep one or more directories synchronized between multiple devices, without the need to use a “central” server. We saw how to install the application, how to setup the firewall to allow traffic through the required ports, how to share a directory between two devices and how to autostart the Syncthing daemon automatically when a user logs in.

Source: https://linuxconfig.org/how-to-keep-files-and-directories-synchronized-across-different-devices-using-syncthing-on-linux

grepWin is an open source tool that searches for files and inside documents using regex

What’s your preferred desktop search engine program? The answer is likely Everything, by Void Tools. I agree with you, it is a fantastic application.

grepWin is an open source tool that searches for files and inside documents using regex

GrepWin is an open source program that specializes in finding text in documents; it supports advanced regular expression filters, and you may want to give it a try for that purpose.

The program’s interface is not the most user-friendly, but that’s probably an impression caused by the various options on the screen.

grepWin example

Select the folder that you want to search by clicking the three-dot button at the top. Enter your query in the box that’s labeled “Search For”. For a more advanced approach, you may opt for the Regex search mode. GrepWin even has a test regex option for checking if your regular expression works or not. Hit F1 to view the help file, it has a list of all supported regex syntax.

grepWin regex commands

Hit the enter key or click on the search button to make the query and grepWin will list the results on the pane at the bottom. The information is split into various columns, such as file name, size, path, extension, encoding, and date modified. Double-click on a file to open it in its default app, e.g. TXT files in Notepad, audio files in your music player, and so on. Right-click on a result to access the Explorer shell menu.

GrepWin could list a ton of files, and if you have trouble finding the one you wanted, I recommend playing around with the Limit Search settings to restrict the process by file size, date, hidden, system or binary files, disabling recursive search (subfolders). This also shortens the time taken for the search to complete by a considerable duration.

grepWin regex search

You can blacklist specific folders from searches using the exclude dirs box, the syntax is ^(FOLDERNAME)$. To include file types, use wildcards like *.TXT, and to exclude types, add a – before it. You may add | to separate multiple items. Speaking of regular expressions, you can add the ones you use to the presets, which will help you add them quickly the next time. Don’t feel intimidated by these options, you don’t need to know regex commands to use the program for simple searches, though by doing so you’ll be missing out on some of its strongest filters.

Switch between the Files search mode and the Content finder by toggling the option in the bottom right corner of the window. This changes the columns in the search results pane, to display the relevant information. The Files mode lists the name, path of the documents which contain the search term.

GrepWin’s content mode can search inside documents and list every instance the phrase was found in each document along with the line name and a preview of the text. The application supports case-sensitive search, which can be handy if there are a lot of matches, and you want to filter them based on the case. The application can be used to replace content in documents directly, to use this option enter the words in the replace with box, and click on the Replace button. You may want to enable the create backup file option, before using the replace function.

The Search button in grepWin has a few additional options including an inverted lookup, i.e. find files that don’t match the entered query. You can also use it to run a search within the found results.

GrepWin is available for 32-bit and 64-bit computers, and comes in portable versions. If you’re using the installer version, you can access the program from the Windows Explorer context menu.

DnGrep is a similar software, in fact it is nearly identical to grepWin. I’m not sure if one of them is a fork of the other.

Source: https://www.ghacks.net/2021/04/09/grepwin-is-an-open-source-tool-that-searches-for-files-and-inside-documents-using-regex/

Automating Processes Using Power Automate Desktop and Cloud

The pandemic has facilitated the acceleration of services like Microsoft’s Power Platform, Amazon’s Honeycode, and Google’s AppSheet that enable workers to automate business processes. In this article, we take a closer look at two Microsoft platforms that create and manage these automated workflows.

When looked at as an ecosystem of tools that help with automating processes across businesses, Microsoft’s Power Automate is the key service for IT pros who want to start optimizing employee workflow. Power Automate consists of two main entry points – one via the cloud and via a desktop app.  In this article, to distinguish between them, they will be referred to as Power Automate Cloud and Power Automate Desktop.

Each brings a unique spectrum of tools and services across multiple options to start automating processes in any business and at any skill level. Consider Power Automate as the cloud-based enterprise technology that allows users to automate business processes that access cloud services, desktop applications, and legacy systems. Power Automate Desktop broadens the existing robotic process automation (RPA) capabilities in Power Automate by giving end users prebuilt drag-and-drop actions to help them automate workflows or by letting them record their own desktop workflows to run later.

Stephen Siciliano, product lead for Microsoft Power Automate, says that Microsoft sees these two services as a continuum that allows for users to enter at a point that accommodates their current skill level and the task at hand. As those skills advance or the end goal changes, the user can move into other areas of the two services to expand their work and automation. This is possible because, although they are two separate services, the work performed in one is available in the other and vice versa to support that continuum concept. Work done in either Power Automate platform can also be integrated into the company’s Power Apps service to take automation and data even further.

Ryan Cunningham, who is the product lead for Microsoft Power Apps, concurs. “Microsoft is really all-in on Power Platform as a way to tie together many of the assets that we have in the cloud today.” Cunningham said.

He went on to say that low-code/no-code elements in Power Platform and other related services have been percolating for years at Microsoft; these include Microsoft Excel, which was released almost 35 years ago and is the basis for Microsoft Power Fx, the low-code language for expressing logic across the Microsoft Power Platform.

In other words, while the pandemic has given the concept of low-code/no-code a new stage, the low-code/no-code technology is not necessarily new; it’s based on things that have been at the core of many of the company’s services for some time.

Now that we have talked about the platform, let’s break down some key elements for both Power Automate and Power Automate Desktop, starting with a feature comparison.https://app.everviz.com/embed/7JBoMwnrO/ 

Power Automate

Microsoftpower-automate-cloud-main-page.png

This completely web-based service is the best place to begin exploring automating processes in a company. New automation processes can begin from scratch, but one of the features in Power Automate is the massive collection of preprogrammed workflow templates, which may provide either a drop-in solution or the base for a company looking to automate specific workflow processes. Find that template, provide company-specific details for any services and connectors required for the workflow, then enable the new flow for testing, publication, and sharing. There are also plenty of customization options across the entire process to make the automation even more specific to a workflow.

There are more than 450 services available in the Power Automate cloud version, which includes connectors from MicrosoftGoogleAmazon, Salesforce, Dropbox, Adobe, plus most social media platforms and other third-party services.

Examples of automations that can be accomplished with Power Automate include:

  • Saving email attachments to cloud storage
  • Monitoring and saving social media entries about a keyword such as a product or service
  • Creating tasks based on calendar events
  • Posting messages in collaboration service such as Teams or Slack when team tasks are completed
  • Using RSS feeds to monitor website content and send alerts when certain keywords are matched

The absolute best way to learn how to use Power Automate is to just log in and begin tinkering with the various templates and services to see what can be created. Start simple to learn the UI and development process then dive into automating processes within the company.

Power Automate Desktop

Microsoftpower-automate-desktop-sign-in-dialog.png

The newer service is licensed for any Windows 10 installation and is accessible through a desktop software download, provided IT administrations approve the installation on managed devices. There is no additional cost to access or use Power Automate Desktop.

While the Power Automate cloud version provides numerous templates to get citizen developers started on automating processes, Power Automate Desktop takes the process up a notch. By providing both a desktop and web recorder to help users automate workflows across desktop software such as Microsoft’s Office productivity suite or sites accessed through a compatible desktop web browser. Automating complex tasks are basically point and click.

Think of the desktop and web recorders as a camera that looks at a user’s Windows 10 desktop and monitors the actions that a user takes to complete various tasks. When a recording with either the desktop or web recorder is started, the mouse movement and mouse clicks taken to accomplish a task are monitored and documented. Once the task process is completed and the recorder is stopped, all of those steps are inserted into the software to create a step-by-step workflow for that particular task.

Of course, each recorded automation can also be customized. Existing flows can also be re-used as a base for creating new desktop flows.

Although Power Automate Desktop does not contain templates like the base Power Automate offering, it does contain a collection of prebuilt actions that can be used as flows are built to trigger various actions based on elements such as variables, loops, files, folders, system, mouse, keyboard, and other capabilities on the desktop.

Some scenarios for using Power Automate Desktop include:

  • Capturing data from PDF’s, emails, websites, apps, and spreadsheets
  • Automating email to surface important information
  • Generating reports by collecting data and inputting it into other software
  • Validating forms for accuracy
  • Managing files from email, local and cloud storage

For Power Automate Desktop, the best starting point is using the web or desktop recorders to capture desktop workflows. Once the recorder is started, just perform those repetitive tasks to help in automating processes across the desktop environment and ultimately save valuable time for other business areas that need more focused attention.

Just like with Power Automate, experiment to learn how things flow together to get a better understanding of the Power Automate Desktop service on a safe/non-volatile process that does not involve the deletion of data. Once the basics of automating processes are understood, then start applying those lessons to workflows.

Source: https://www.itprotoday.com/agile/automating-processes-using-power-automate-desktop-and-cloud

How to Export and Import VirtualBox Virtual Machines

Exporting and Importing feature is available in all the virtualization software you use. This feature will be very useful when you want to copy and launch a configured VM from one machine to another machine.

This feature is very useful for me, as I have a lot of virtual machines running on VirtualBox for testing and writing articles, and for VM level protection I can take snapshots and protect my VM.

But if my Host machine crashes then I have to configure all the VM from scratch. It is a tedious task for me. So once I fully configure any VM I export it and save a copy in my external disk so when I lose the image I can import the image into VirtualBox.

Exporting and Importing Virtual Machines in Virtualbox

Virtualbox uses OVA file format which includes Descriptor file (OVF), Manifest File (MF), and other related files. This is a standard format used in many popular virtualization software too like Vmware. Exporting and Importing VM images can be done in few clicks.

Exporting Virtual Machines in Virtualbox

Go to File → Export Appliance or press “CTRL+E”.

Export Virtual Appliance
Export Virtual Appliance

You will get the list of Virtual Machines that can be exported. Press Expert Mode and configure all the options in a single window. You have to choose the virtualization format version, where your exported .ova file is going to be saved in the local file system.

VirtualBox Virtual Machines
VirtualBox Virtual Machines
Export Virtual Machines
Export Virtual Machines

Once you press Export, exporting process will be started.

Exporting Virtual Machine
Exporting Virtual Machine

You will see .ova file is created as a part of the export process.

$ ls -l *.ova

-rw------- 1 linuxshelltips linuxshelltips 988830720 May  5 08:57 Fedora_34.ova

Importing Virtual Machines in Virtualbox

Now you can reprovision the image by importing the VM. Go to File → Import Appliance or press CTRL + I.

Import Virtual Appliance
Import Virtual Appliance

You have to choose the exported image and press Import.

Import Virtual Machine
Import Virtual Machine
Import Virtual Machine Settings
Import Virtual Machine Settings

You will see the disk image getting imported. It will take some time and that depends on your disk image size.

Importing Virtual Machine
Importing Virtual Machine

Now I have successfully imported the image and can now start running the VM straight away without any changes.

Imported Virtual Machine
Imported Virtual Machine

In the next article, we will see how to install PHP Virtualbox and manage Virtual Machines in PHP VirtualBox – a web-based user interface.

Source: https://www.linuxshelltips.com/export-and-import-virtualbox-vm-images/

Ultimate guide to backup Ubuntu systems using Timeshift

In this tutorial, we will learn how to perform Ubuntu backups using Timeshift. Timeshift is a fabulous tool that is used for the backup & restoration of the Linux operating system, it takes incremental backup after the first initial complete backup. TImeshit creates filesystem snapshots using Rsync or BTRFS. It has a nice GUI as well as support for CLI.

Timeshifts also have support for scheduled snapshots, multiple backup levels & also exclude filters. Snapshots can be easily restored even if the system is running from a Live CD or USB.

So let’s now start with the installation of Timeshift.


Install Timeshift in Ubuntu 20.04 or up

If you are running Ubuntu 20.04 or up, then you just need to run a single command to install Timeshift on Ubuntu as the timeshift packages are available in default repositories,

$ sudo apt install timeshift

Install Timeshift in Ubuntu 19.10 or older

If you are running Ubuntu 19.10 or older versions (which you should not be using), then you are required to add the repository for timeshift first & then install timeshift. Use the following commands to perform the actions,

$ sudo add-apt-repository -y ppa:teejee2008/ppa

$ sudo apt update

$ sudo apt install timeshift

Performing Ubuntu Backup using Timeshift

Now that we have the timeshift installed on our Ubuntu systems, we can start creating the snapshots for our Ubuntu installation. Browse or search the menu for timeshift & open it.

Once timeshift is opened, you will be asked to select the preferred snapshot type. You will have the options for selecting,

1- RSYNC – Rsync takes backup with the rsync command and also used hard links. Rsync backups are browsable using file managers.

2- BTRFS – with this mode, timeshift uses in-built features of BTRFS & then to restore snapshots, BTRFS should be supported with the Ubuntu-type subvolume layout.

You can select either but we will be using rsync,

Once on the next screen, you are required to select the destination drive to save your snapshots. Select a suitable drive (which is not the source drive) & press ‘Next’. On the next screen, you will be asked to select screenshot level. It can be either ‘Monthly’, ‘Weekly’, ‘Daily’, ‘Hourly’ or on ‘Boot’,

ubuntu backup

Now click on ‘Finish’ to complete the setup. Now your initial setup is complete & you can start creating your backup snapshots. To create the backup, click on ‘Create’ on the main menu,

The backup process will now start and once it’s complete, click on ‘Finish’.

Restore the backup

Similar to how we created the backup, restoration is also pretty simple. From the main menu, click on the option ‘Restore’, & then select the snapshot from the list. Next, you need to provide the destination to where this snapshot needs to be restored.

Before clicking on restore, you can also perform ‘Dry Run’ to verify all files and folders. Once you are ok, you can click next & the restoration process will start. It will take some time to complete.

Now that we have discussed how to perform Ubuntu backup using the GUI, we will take a look at CLI commands as well.


Create Ubuntu backup using CLI

Timeshift also supports the command line to create and restore the backup. to create a backup, simply run the following command,

$ sudo timeshift –create –comments “Backup with CLI” –tags D

In tags, you can any of the following options,

–tags D stands for Daily Backup

–tags W stands for Weekly Backup

–tags M stands for Monthly Backup

–tags O stands for On-demand Backup

Restore a backup using  CLI

To restore a backup using the timeshift in CLI, use the following command,

$ sudo timeshift –restore

You will then be asked to select a snapshot from the list by pressing the number for that snapshot. Then press enter key to continue.

For the next step, it will ask to Re-install the GRUB2 bootloader, press ‘Y’, and then again press ‘Y’ & enter to start the restoration process.

With this, we end our tutorial on how to backup Ubuntu using Timeshift. If you have any questions or queries, please do send them in using the comment box below.

Source: https://linuxtechlab.com/backup-ubuntu-systems-using-timeshift/