I thought this would be easy but it's turning out to be relatively
complicated - at least in terms of finding a 'standard solution'.

What I want to do is to set up a filesystem hierarchy on Linux with
symlinks to 'interesting' directories as in

basedir
symlink to A
symlink to B
symlink to C

etc, where A, B and C are directories, and replicate the hierarchy so
that Windows ends up with

basedir
A
B
C

and the contents of those directories. The files (and link targets) can
be copied verbatim; I don't need any changes to line ends, for example.

* I thought of rsync but Windows support for it appears to be poor -
e.g. requiring Cygwin which I don't want to add or wsl which apparently
requires some tricks to avoid having to join the Windows Insiders Program.

* I thought of Syncthing but it's bidirectional and its documentation
says it will not follow the symlinks to directories as required.

* I thought of mapping a drive but that would require a copy job on the
Windows host and I really don't want to get into automated Windows
tasks. If any automation is done I'd rather control it from Linux which
I am more familiar with.

So that's the issue. I would think there must be an easier way. Hence
this query.

Is there a good way to sync a directory hierarchy unidirectionally from
Linux to Windows? There must be one, surely...!?

--
James Harris

On Mon, 17 May 2021 11:52:00 +0100, James Harris
<james.harris.1@gmail.com> wrote:

>I thought this would be easy but it's turning out to be relatively
>complicated - at least in terms of finding a 'standard solution'.
>
>What I want to do is to set up a filesystem hierarchy on Linux with
>symlinks to 'interesting' directories as in
>
> basedir
> symlink to A
> symlink to B
> symlink to C
>
>etc, where A, B and C are directories, and replicate the hierarchy so
>that Windows ends up with
>
> basedir
> A
> B
> C
>
>and the contents of those directories. The files (and link targets) can
>be copied verbatim; I don't need any changes to line ends, for example.
>
>
>* I thought of rsync but Windows support for it appears to be poor -
>e.g. requiring Cygwin which I don't want to add or wsl which apparently
>requires some tricks to avoid having to join the Windows Insiders Program.
>
>* I thought of Syncthing but it's bidirectional and its documentation
>says it will not follow the symlinks to directories as required.
>
>* I thought of mapping a drive but that would require a copy job on the
>Windows host and I really don't want to get into automated Windows
>tasks. If any automation is done I'd rather control it from Linux which
>I am more familiar with.
>
>
>So that's the issue. I would think there must be an easier way. Hence
>this query.
>
>Is there a good way to sync a directory hierarchy unidirectionally from
>Linux to Windows? There must be one, surely...!?

Sounds like a bunch of custom backup apps I've written.

For Winders, you will have to write your own "rsync". It's not
TOO hard, depending on how many options you actually
NEED. Actual rsync is a super-capable program, but for
what YOU seem to be wanting you don't need 90% of
said capabilities. The annoying part is transferring all
the permissions/dates.

To translate source dir-trees to dest dir-trees you will need
a special function. You will need the root source dir, the
current source dir (what your app is looking to reproduce
at the moment, a subdir of the root), and the dest dir. It's
only a few lines and some math. Worked it out one day
while riding a motorcycle down a boring interstate .... weird.
If you want it in 'C', Python or Pascal I'll post it for you.

On 18/05/2021 05:44, MSB wrote:
> On Mon, 17 May 2021 11:52:00 +0100, James Harris
> <james.harris.1@gmail.com> wrote:

....

>> Is there a good way to sync a directory hierarchy unidirectionally from
>> Linux to Windows? There must be one, surely...!?
>
> Sounds like a bunch of custom backup apps I've written.
>
> For Winders, you will have to write your own "rsync". It's not
> TOO hard, depending on how many options you actually
> NEED. Actual rsync is a super-capable program, but for
> what YOU seem to be wanting you don't need 90% of
> said capabilities. The annoying part is transferring all
> the permissions/dates.

rsync is pretty good in only copying changes. I'd like to keep that bit
of it. But, yes, rsync is rather awash with options. I wonder if it
couldn't be split into parts but that's another matter.

>
> To translate source dir-trees to dest dir-trees you will need
> a special function. You will need the root source dir, the
> current source dir (what your app is looking to reproduce
> at the moment, a subdir of the root), and the dest dir. It's
> only a few lines and some math. Worked it out one day
> while riding a motorcycle down a boring interstate .... weird.
> If you want it in 'C', Python or Pascal I'll post it for you.

Please do. I take it you don't mean you wrote an interface to rsync
(which would be even cooler) but designed your own protocol involving
communication of path names and file contents. It could certainly be
useful to see how that works. Python is probably the shortest and most
portable between Linux and Windows but C would do.

--
James Harris

On 17/05/2021 12.52, James Harris wrote:
> I thought this would be easy but it's turning out to be relatively
> complicated - at least in terms of finding a 'standard solution'.
>
> What I want to do is to set up a filesystem hierarchy on Linux with
> symlinks to 'interesting' directories as in
>
>  basedir
>     symlink to A
>     symlink to B
>     symlink to C
>
> etc, where A, B and C are directories, and replicate the hierarchy so
> that Windows ends up with
>
>  basedir
>     A
>     B
>     C

How is Windows going to see the Linux filesystem? Some ext2/3/4 driver?
Samba? What?

--
Cheers,
Carlos E.R.

On Tue, 18 May 2021 10:36:16 +0100, James Harris
<james.harris.1@gmail.com> wrote:

>On 18/05/2021 05:44, MSB wrote:
>> On Mon, 17 May 2021 11:52:00 +0100, James Harris
>> <james.harris.1@gmail.com> wrote:
>
>...
>
>>> Is there a good way to sync a directory hierarchy unidirectionally from
>>> Linux to Windows? There must be one, surely...!?
>>
>> Sounds like a bunch of custom backup apps I've written.
>>
>> For Winders, you will have to write your own "rsync". It's not
>> TOO hard, depending on how many options you actually
>> NEED. Actual rsync is a super-capable program, but for
>> what YOU seem to be wanting you don't need 90% of
>> said capabilities. The annoying part is transferring all
>> the permissions/dates.
>
>rsync is pretty good in only copying changes. I'd like to keep that bit
>of it. But, yes, rsync is rather awash with options. I wonder if it
>couldn't be split into parts but that's another matter.

If you are only interested in moving "changed" files, mostly
one-way, it's EASY to emulate that aspect of rsync. While
rsync offers a ton of ways to decide what is "changed"
those are rarely needed. Filename/date usually covers it.

THIS aspect can be writ in maybe 50 lines of Python, or
a few more of 'C' (mostly because of the crappy string
handling).

A potentially useful feature of rsync is its "--delete" options.
If a file ceases to exist on the source, it is then deleted from
the dest as well. It prevents "dead" files from buiding up.
For THIS you need to sort-of run things backwards - start
with the dest files and LOOK for matching files in the
source. If not there, delete said dest file. Filename alone
is almost always "good enough" here.

Another advantage of rsync is that it is multi-threaded/
multi-core. This speeds things up (though not AS much
as you'd think due to I/O latency with disks). This aspect
is a bit easier to do in 'C'.

I've recently done MOST of this for a Winders app (not
so much by choice). However other projects have intervened
and I've not quite finished/polished everything.

>> To translate source dir-trees to dest dir-trees you will need
>> a special function. You will need the root source dir, the
>> current source dir (what your app is looking to reproduce
>> at the moment, a subdir of the root), and the dest dir. It's
>> only a few lines and some math. Worked it out one day
>> while riding a motorcycle down a boring interstate .... weird.
>> If you want it in 'C', Python or Pascal I'll post it for you.
>
>Please do. I take it you don't mean you wrote an interface to rsync
>(which would be even cooler) but designed your own protocol involving
>communication of path names and file contents. It could certainly be
>useful to see how that works. Python is probably the shortest and most
>portable between Linux and Windows but C would do.

For BACKUPS, you want to create an echo of the
source subdir tree on the destination - starting at SOME
point. It CAN be done, clearly, in about five or six
lines of Python or Pascal - about a dozen of 'C' (again
because of the string-handling limitations).

I'll post some version tomorrow. 'C' ? Python ? Pascal ?

And you CAN run it in reverse to emulate rsync's "--delete"
capabilities.

I recently had to do a special version - from where the backups
were zipped or encrypted and no longer had the same names
or sizes of the source files. A tad trickier. Rsync "--delete"
could NOT help under such circumstances, HAD to roll
my own ...

Oh yea ... an oft-neglected feature of rsync is that you can
have it make a file of all "changed" files it encountered, and
thus altered on the destination. You can even partially format
said file so it doesn't contain too much junk. If you need to
do anything special with "changed" files, you can use this
record to see WHAT, exactly, you need to pay attention to.

In a time of ransomware, backups - LOTS of backups - are
THE thing to do. In an age of rip-offs - even from "reputable"
cloud providers, encrypting cloud copies are also of prime
importance. Space is relatively cheap these days, your
data is NOT.

James Harris <james.harris.1@gmail.com> writes:

> * I thought of rsync but Windows support for it appears to be poor -
> e.g. requiring Cygwin which I don't want to add or wsl which
> apparently requires some tricks to avoid having to join the Windows
> Insiders Program.

Just for the record, I find this a little odd. For example, here's how
to install Windows native rsync:
https://tlundberg.com/blog/2020-06-15/installing-rsync-on-windows/

Cygwin certainly also provides rsync as well as WSL. I don't know why
you think you need to join Windows Insiders Program for WSL? I have it
installed on one computer and put Debian in there from Microsoft's
store. No "insider program" needed. WSL starts way faster than a Linux
distribution in VirtualBox for example so it's convenient. And wsltty
provides a decent terminal emulator for it, even including sixel
graphics.

Oh yes ...

These were writ many years ago. I think they CAN be
"improved" ... though long-term comprehensibility is
also a feature of "improvement" . So, in my very
classic K&R style (mind the short line-lengths in
the usenet post) :

First in 'C' .....

// Translate source -> dest directory trees. It is assumed
// you have a function that is reading files from dirs/subdirs
// and spitting the paths HERE for translation. There may be a
// better way, but this works and will be comprehensible years
// from now.
//
// in two different pathnames - in 'c' this time.
// If the source path was "/A/B/C/D/filename" and the dest
// is "/W/X/Y/C/D/filename" how do we calc that ?
// We can't just concat for "/W/X/Y/A/B/C/D/" because that
// is ridiculous. We really want "/W/X/Y/C/D".
// We start with the dest path, "/W/X/Y/". But how do we
// know to chop off just "/A/B" ? We need a 'root path'
// supplied for reference, that would be "/A/B". Then we
// can remove the "/A/B" and splice what's left on to
// the "/W/X/Y".
//
// 'C' version of our Python MakeDPath() ...
// we can't really return a string neatly in 'c'
// so we'll use 'outS' to send the result back.
// By NOT adding end slashes here the math works
// out in a few less steps.
//
// Let's say we're copying everything from /home/fred/mySource to
/home/bill/mySource ...
//
// So : sfPth= path you're copying file FROM
(/home/fred/mySource/subfolder)
// oPth = the original source path, the one you sent this pgm
(/home/fred/mySource)
// because subdir iterations from lister functions
disguise that.
// dPth = root dest folder you sent this pgm, where file will go
TO
// that we blend with the source path
(/home/bill/mySource) to
// realize (/home/bill/mySource/subfolder)
// outS = send a null string, at end this will have the
calculated save path
//
int MakeDPath(char *sfPth,char *oPth,char *dPth,char *outS)
{ int nn=0;
char sfPthT[PATH_MAX]={0},dPthT[PATH_MAX]={0};
strcpy(sfPthT,sfPth); strcpy(dPthT,dPth); // make local copies
to mutilate
nn=(strlen(sfPthT)-strlen(oPth)) ; // subtract length of
oPth
if(nn<=0)
{
strcpy(outS,dPth); // nothing to add ?
rare but happens. outS=dPth
}
else
{
strncpyR(sfPthT,nn); // "reverse strncpy"
gets latter part of sfPthT
strcat(dPthT,sfPthT); strcpy(outS,dPthT); // combine sDfT and
fpthT and copy result to outS
}
return(strlen(outS)); // return length of
outS
}

//---------------------------

// "strncpy" but counting from REAR of string.
// note this goes with MakeDPath(). You can do
// it with pointer math, but indexes are more
// traditional and comprehensible.
//
int strncpyR(char *src, int nn)
{ char tmp[PATH_MAX]={0};
int x=0,slen=strlen(src); // init slen to len of src
if(nn>slen) { return(slen); } // beyond beginning of string
- so just quit now
strcpy(tmp,src); // make working copy, using
src as src and dest screws up
src[0]=0; // positively null src
strcpy(src,(char*)&tmp[slen-nn]); // copy x chars back into src,
starting at tmp[strlen(tmp)-nn]
return(strlen(src)); // return new length
}

And in Python3 ............

# params in :
# fpth = "from_path" of source_ROOT+subdirs, ie "/home/jx/red/etc/"
# sSf = base root path to SOURCE root folder, ie "/home/jx/"
# sDf = base root path to dropbox DEST folder, ie
"/home/Dropbox/abode/colors/"
# output :
# return = "/home/Dropbox/abode/colors/red/etc/'"
# # Remember, this just makes a PATH, not a full path+filename.
# You'll attach the filename elsewhere in your filename-processing
code
# def MakeDPath(fpth,sSf,sDf):
fpth=AddDelims(fpth) ; sSf=AddDelims(sSf) # fix up names - ensure
start+end slashes exist
nn= (len(fpth)-len(sSf)) * -1 # a little math
if nn>=0 : fpth="" # if sourcedir completely
replaced by destdir
return sDf+fpth[nn:] # return result

And yes, I always comment the crap out of things :-)