Post Reply 
HOWTO: make dxpc compressed X11 work gracefully over ssh (including Centos 5 hacks)
06-26-2011, 07:52 AM
Post: #1
HOWTO: make dxpc compressed X11 work gracefully over ssh (including Centos 5 hacks)
Hello, I recently had a need for my Gentoo x86_64 workstation to serve as an X11 server for some gui programs on a CentOS 5.6 i386 VPS with very high latency. I decided it was time to take my own advice and set up dxpc.

In my previous summary I had mentioned the lbx protocol, but it turns out that lbx (which is an XFree86 built-in feature) is deprecated and no longer works with bleeding edge xorg builds (which I have on my Gentoo box).

So dxpc it would be. At first I just installed dxpc in the regular Gentoo way on my workstation, and used the package from the dag rpmforge add-on repo on my Centos VPS. This failed miserably, however it didn't take much Googling to figure out a solution. Specifically, the slightly out-of-date version of dxpc on the dag repo has some kind of data-representation bug (endian-swapping iirc) which occurs when you try to get an i386 dxpc and an x86_64 dxpc to talk to each other.

However, I could not find any rpm packages of the latest dxpc version for Centos. To solve this, I downloaded the src.rpm package from the dag repo, installed it as an unprivileged user set up for rpmbuild usage, and modified the SPECS/dxpc.spec file as follows:

Code:
# $Id: dxpc.spec 7982 2009-11-03 03:41:01Z dag $
# Authority: dag
# Upstream: Kevin Vigor <kevin$vigor,nu>


%{?fc4:%define _without_modxorg 1}
%{?el4:%define _without_modxorg 1}
%{?fc3:%define _without_modxorg 1}
%{?fc2:%define _without_modxorg 1}
%{?fc1:%define _without_modxorg 1}
%{?el3:%define _without_modxorg 1}
%{?rh9:%define _without_modxorg 1}
%{?rh7:%define _without_modxorg 1}
%{?el2:%define _without_modxorg 1}
%{?yd3:%define _without_modxorg 1}

Summary: Differential X protocol compressor
Name: dxpc
Version: 3.9.2
Release: 0%{?dist}
License: BSD
Group: User Interface/X
URL: http://www.vigor.nu/dxpc/

Packager: Coinonymous <coinonymous@googlemail.com>
Vendor: Dag Apt Repository, http://dag.wieers.com/apt/

Source: http://www.vigor.nu/dxpc/dxpc-%{version}.tgz
BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root

BuildRequires: lzo-devel, gcc-c++
%{!?_without_modxorg:BuildRequires: libX11-devel}
%{?_without_modxorg:BuildRequires: XFree86-devel}

%description
dxpc is an X protocol compressor designed to improve the
speed of X11 applications run over low-bandwidth links
(such as dialup PPP connections).

%prep
%setup

%build
%configure
%{__make} %{?_smp_mflags}

%install
%{__rm} -rf %{buildroot}
%makeinstall \
    man1dir="%{buildroot}%{_mandir}/man1/"

%clean
%{__rm} -rf %{buildroot}

%files
%defattr(-, root, root, 0755)
%doc CHANGES README* TODO
%doc %{_mandir}/man1/dxpc.1*
%{_bindir}/dxpc

%changelog
* Thu Jun 23 2011 Coinonymous <coinonymous@googlemail.com> 3.9.2-0 - 7982/coin
- Initial package. (using DAR)

I guess I did a pretty half-assed job on the metadata so I won't bother trying to upstream this, but it is good enough for rock 'n roll. With this spec I am able to do:

Code:
SPECS# cd ../SOURCES
# note: the following line is just a lazy way to get your package downloaded -- wget would have worked just as well
Code:
SOURCES# spectool -g ../SPECS/dxpc.spec
SOURCES# cd ../SPECS
SPECS# rpmbuild -ba dxpc.spec
SPECS# cd ../RPMS/i386
i386# yum localinstall --nogpgcheck dxpc-3.9.2-0.i386.rpm

So now that the latest dxpc are installed on my workstation and my VPS, a quick test showed that everything was mostly working.

After some chin scratching I finally arrived at the following set of scripts to make it easy for me to maintain a dxpc connection to my host. Note that dxpc has an annoying problem (maybe there is some command-line option to make this better but I'm too lazy to learn it?) that it tends to shut down after a client-server connection is established, used, and then left idle for a few seconds. I tend to leave shells idle habitually and this is almost never what I want, so these scripts try to make that easier to manage:

On my workstation (the X11 server) I did the following:

First, I made a script to spawn a dxpc server process if none is running:
Code:
#!/bin/bash

# /home/coinonymous/bin/ensure_dxpc_server

# stfu: run a program, but silence stderr/stdout (even if it forks)
stfu() {
  ( "$@" 2>&1 ) 2>&1 > /dev/null
}

stfu pgrep -f 'dxpc -w' || stfu dxpc -w &

The documentation I found online said that dxpc wants a reverse TCP tunnel on port 4000 for the protocol to work correctly so I created a script to run ensure_dxpc_server and ssh to my VPS with this reverse tunnel:

Code:
#!/bin/bash

# /home/coinonymous/bin/dxssh

$(HOME)/bin/ensure_dxpc_server

ssh -R 4000:127.0.0.1:4000 -X -Y "$@"

This can be used just like ssh i.e.:

Code:
dxssh my.vps.net -l coinonymous

The difference is that it will run the dxpc server, and establish the X11 Forwarding and reverse tcp proxy on port 4000 required for dxpc to work.

On the vps, things are a less complicated to set up. I had some instructions that worked, but the annoything thing that kept happening was that dxpc would die on the server end (either due to timeouts of flakeouts) and I would be stuck with a broken DISPLAY variable and no way to remember what the correct variable is (in practice it's almost always the same but still, I felt uncomfortable making cavalier assumptions).

Eventually I settled on adding the following to my $(HOME)/.bashrc on the VPS:

Code:
dodxpc() {
  ( dxpc -w 127.0.0.1 2>&1 ) 2>&1 >/dev/null &
  DISPLAY=":8.0" bash -
}

So this gets the entire process of setting up a dxpc session down to a typing just a single extra command:

Code:
workstation$ dxssh myvps.net
vps$ dodxpc

That's pretty easy and makes dxpc a lot less confusing to use.

Also, it sets things up so that if dxpc ever went down (usually it dies simultaneously on the server and client sides) I can just do:

(on my workstation)
Code:
$ ensure_dxpc_server

and

(on the VPS)
Code:
$ exit # once we close the shell the old DISPLAY variable is back
$ dodxpc

with no need to tear down my ssh session.

Since this was a bit of a PITA to set up and makes dxpc super easy to use, I thought I'd share. HTH,

-?
User Tools
Quote this message in a reply


06-27-2011, 03:12 AM
Post: #2
RE: dxpc revisited
Nice tutorial! Am I correct in assuming that this basically compressed X11 forwarding?

I believe NX is based on dxpc and is supposed to be faster, but it shows a full X display.

Anyway, I'd urge you to submit your package to rpmforge, it may help someone out in the future. Here's how to do it if you do decide to http://repoforge.org/package/ It does seem like a lot of work to get something committed to their repo however.
User Tools
Quote this message in a reply
06-27-2011, 07:54 PM
Post: #3
RE: dxpc revisited
(06-27-2011 03:12 AM)f8ll Wrote:  Nice tutorial! Am I correct in assuming that this basically compressed X11 forwarding?

Yes, aside from making things more likely to crash (sad but true) it doesn't change anything except to make X11-over-tcp faster and more accommodating of latency.

(06-27-2011 03:12 AM)f8ll Wrote:  I believe NX is based on dxpc and is supposed to be faster, but it shows a full X display.

I have read this too. I presume you mean faster than native X11 -- I am not sure if they claim to be faster than dxpc so much as equivalent. I've never tried NX but hopefully it still supports the X11 protocol. It would seem pretty crazy to me to base your protocol on X and then only support vnc-style "remote desktop."

(06-27-2011 03:12 AM)f8ll Wrote:  Anyway, I'd urge you to submit your package to rpmforge, it may help someone out in the future. Here's how to do it if you do decide to http://repoforge.org/package/ It does seem like a lot of work to get something committed to their repo however.

Yeah I am just too lazy to fix up all my metadata and make the .spec correct. For starters I accidentally removed the old changelog entries in the .spec, so it almost looks like I'm trying to take credit for someone else's work... I do try to be good about upstreaming things I fix/improve in FOSS projects but right now I'm too busy... feel free to tidy this up for me and submit it Angel ...

In my experience most projects set the bar pretty high regarding what kind of submissions they will take. High enough that it could take several hours of research to figure out how to give them something that will get upstreamed.

I completely understand why this is so -- they don't want to have to accept a lot of crappy patches that work for one guy but totally violate their coding standards or create arch forks or don't jive with their projects intellectual property policies... etc. When possible I try to do give back but often I'm just not willing or able to figure out how to make something upstream-able (i.e.: I make a quick hack to make something compile, but it will clearly break for certain architectures or various ways that autoconf could have been invoked).

This is partly why I post it here. Once it's published on the Internet, cached in google/waybackmachine/etc., a sufficiently determined seeker can find it, so I feel like it's the next best thing to upstreaming my work. The changes were quite trivial in this case, so I feel especially non-guilty about it -- pretty much just needed a version bump.

-?
User Tools
Quote this message in a reply
06-27-2011, 10:36 PM
Post: #4
RE: HOWTO: make dxpc compressed X11 work gracefully over ssh (including Centos 5 hacks)
Thanks! it was very helpful Smile
User Tools
Quote this message in a reply
06-28-2011, 04:33 AM
Post: #5
RE: dxpc revisited
(06-27-2011 07:54 PM)coinonymous Wrote:  I have read this too. I presume you mean faster than native X11 -- I am not sure if they claim to be faster than dxpc so much as equivalent. I've never tried NX but hopefully it still supports the X11 protocol. It would seem pretty crazy to me to base your protocol on X and then only support vnc-style "remote desktop."

Well they claim to be faster than their predecessors because they do more than just compress their data. You have to use their client and server protocol however, no X11. I believe the idea is to be more like rdp than vnc (which is pretty dated tbh).

(06-27-2011 07:54 PM)coinonymous Wrote:  In my experience most projects set the bar pretty high regarding what kind of submissions they will take. High enough that it could take several hours of research to figure out how to give them something that will get upstreamed.

Not all do, and a lot will try to help you get the quality high enough to get committed initially. Of course you get the problems you mentioned and eventually end up with the mess that the linux kernel was in a few years ago.

Some projects I admit do have ridiculously high standards that make it a huge pain for anyone to contribute to it. Their software may be of high quality but it takes them forever to develop things.

Anyway, I maintain enough packages for another distro, so hopefully someone will eventually update this one.
User Tools
Quote this message in a reply
06-28-2011, 07:12 AM
Post: #6
RE: HOWTO: make dxpc compressed X11 work gracefully over ssh (including Centos 5 hacks)
Thanks! it was very helpful
User Tools
Quote this message in a reply
07-24-2011, 06:51 PM
Post: #7
RE: HOWTO: make dxpc compressed X11 work gracefully over ssh (including Centos 5 hacks)
I made some changes to the scripts in my original post -- but was reluctant to edit my original post because my changes obfuscate how the scripts work. By design, my original scripts only support one "dxssh" conversation at a time on a given remote host.

For example: Suppose Bob and Alice both want to dxssh from their laptops to the same host at the same time. If Bob logs in first, he will listen for connections to tcp port 4000 on the host.

If Alice now tries to log in, the ssh server will inform her that port 4000 is not available (there is only room for one listener on a tcp port). Although still be able to log in and use X11, she won't be able to use dxpc. If she ignores the ssh warning, she will get a confusing "authentication denied" message from the X11 client libraries when she unknowingly attempts to connect to Bob's X server (or, worse, if Bob doesn't use xauth, Alice's program will appear on Bob's screen!)

To get out of this mess we can change our scripts to be a bit more flexible, as follows:

ensure_dxpc_server is extended to take an optional argument via the "dxdelta" environment variable. This delta will be added (or subtracted if dxdelta contains a negative number) to the dxpc default port number, 4000. This change is not absolutely essential but provides for maximal flexibility if for some reason (i.e.: to talk to multiple X11 servers) we want to run multiple dxpc servers on the same host.

Code:
#!/bin/bash

# /home/coinonymous/bin/ensure_dxpc_server

# stfu: run a program, but silence stderr/stdout (even if it forks)
stfu() {
  ( "$@" 2>&1 ) 2>&1 > /dev/null
}

if [[ -v dxdelta ]] ; then
        cmd="dxpc -p $(( $dxdelta + 4000 )) -w"
else
        cmd="dxpc -w"
fi

stfu pgrep -f "${cmd}" || stfu ${cmd} &

With this in place it's easy to modify dxssh to respect the same dxdelta environment variable.

Code:
#!/bin/bash

# /home/coinonymous/bin/dxssh

$(HOME)/bin/ensure_dxpc_server

if [[ -v dxdelta ]] ; then
        dxport=$(( $dxdelta + 4000 ))
else
        dxport=4000
fi

ssh -R ${dxport}:127.0.0.1:${dxport} -X -Y "$@"

So now we can just type:

Code:
localpc$ dxdelta=3 dxssh foo.bar.net

And instead of using port 4000 everything will be using port 4003. Nothing changes operationally but if port 4000 is already taken we now have a work-around.

It would be really sexy if dxssh somehow magically projected this dxdeta environment variable onto the server side, where it could then be picked up by the dodxpc function. However, although openssh contains a feature allowing clients to request the creation of server-side environment variables prior to login, it is disabled, by default for security reasons.

I briefly considered enabling this feature but after thinking about the potential security implications I decided this feature was too dangerous for my tastes. A lot of crazy things can be made to happen by setting environment variables, after all!

So I added an optional argument to the dodxpc function code in .bashrc as follows:

Code:
dodxpc() {
  dxdelta=${1:-0}
  p=$(( $dxdelta + 4000 ))
  d=$(( $dxdelta + 8 ))
  ( dxpc -w 127.0.0.1 -p $p -d $d 2>&1 ) 2>&1 >/dev/null &
  DISPLAY=":${d}.0" bash -
}

So proceeding with the port 4003 example above, now we can just use:

Code:
remote-host$ dodxpc 3

-?
User Tools
Quote this message in a reply
Post Reply 


Forum Jump:



User(s) browsing this thread: