VS2017 64Bit Build

[wnjj]

I figured out why the HD200 was coming up on the server. I needed to include "RemoteClients.properties.defaults" in the Sage directory. That's where it gets most of its settings, including one that forces the rendering onto the remote device.

Now the extender connects and works! The fonts are still funny looking because of the missing FreetypeFontJNI.dll. I'll work on that next.

[jusjoken]

Originally Posted by wnjj

I PM'd him. I was kind of hoping he'd drop by but I'm sure he's a busy guy.

That would be great if you can help with the installer. I know nothing about that but have learned quite a bit about what's needed to run in 64-bit mode, the first of which is the 64-bit DX9 runtimes. I'm not even sure how external dependencies are handled. Does the installer include the 32-bit versions of those binaries today? Just a few DLL's or a whole DX9 install-able package? When I downloaded the DX9 runtimes, I ended up with a .zip file full of .cab files (remember those?). Then the DX9 installer program extracted the DLL's needed (and likely many more not needed ones).

The installer was a large task as like your endeavor there was not much info to go on other than research and trial and lots of error. Glad you have gotten so far.

For DirectX I learned which versions of DX were installed by the old installer and then determined which of the cabs were needed.

DSETUP.dll
dsetup32.dll
dxdllreg_x86.cab
DXSETUP.exe
dxupdate.cab
Jun2010_D3DCompiler_43_x86.cab
Jun2010_d3dx9_43_x86.cab

The WIX based installer then runs the needed setup as part of the install process if it finds this specific DX version is not already installed.

The other redistributables the installer handles as here...

ac3filter_2_6_0b_lite.exe
VC_redist.x86.exe
vcredist_x86.exe

You may want to try a clean install of sagetv from the latest installer on a clean VM/computer. Then before running sagetv for the first time, swap out the 32bit with 64bit files. then run sage and it will create the .properties files with all the paths correct and defaults needed. That may have resolved the extender issue sooner and may resolve other issues you have not found yet.

k

[tmiranda]

Originally Posted by wnjj

If this ffmpeg stuff becomes too difficult, shouldn't you still be able to run a headless server in 64-bit and then connect a 32-bit client to it? In other words is there enough reason to even get all of the video/audio playback filters working in 64-bit?

For that matter, isn't the Sage demux optional? I have to admit I don't fully understand how all of the filter graphs get built and what you don't really need.

When I started looking at this Narflex sent me an email detailing what the various bits and pieces did and which were necessary and which were optional if running a headless server. When I get home I'll dig up that email and post the important pieces here.

[Narflex]

I'll pay attention to this thread now...so if you have more questions let me know but it seems like you sorted most of it out.

[wnjj]

I got FreetypeFontJNI.dll built a couple days ago so the fonts are happy now.

After much thrashing and iteration, I managed to compile a 64-bit pushreader.dll, which means I should be able to compile the last of the VS source (the Sage demux StrmDeMux). Since the Pushreader source code had yet to be compiled as 32-bit open source it took some doing. I had to add some 'configure' options and even tweak a couple spots inside the ffmpeg code. I'm guessing something must have been setup slightly different when the 32-bit DLL was originally built. One difference may be the MinGW environment I'm running versus a years-old one.

There was one thing I couldn't make any sense of: In pushreader.c, there's a function prototype:

void *av_memcpy(void *dest, const void *src, unsigned int size);

The problem is there's no function defined anywhere. I grepped inside all of the Pushreader and ffmpeg source to no avail. It seems odd that this one function was prototyped here in the first place anyway. To get the compile going I just #defined it to plain memcpy.

Jeff, do you know why this function is there versus using a standard memcpy? Was it optimized for performance?


The next thing I need to figure out are the external .exe's that get called from within Java using Runtime.getRuntime().exec (e.g. SagetvTranscoder, jhead, ipconfig, etc.). It looks like the majority of them are for Linux so that helps. When called from a 64-bit JVM, they try to run in a 64-bit cmd.exe shell. I think the easiest solution is to run the 32-bit exe's via the 32-bit cmd.exe shell by either coding it into Java (with OS detection if/else) or wrapper batch files that detect and run the 32-bit cmd.exe. The Java option means tweaking the core code. The batch files are a big hack. Any thoughts on this?


The next major unknown for me is tuners. I assume those need 64-bit drivers from the manufacturer? Can OpenDCT bridge the gap? If nothing else, an encoder server could be run in 32-bit mode, right?


After that I'll have to see what else may need work and then it'll likely be ready for beta testers. You should be able to convert a working 32-bit system over to a 64-bit one by simply replacing binary files and registering the new filters. I'm sure there are plenty of video formats, options and modes that could have issues and even crashes. Anyone wanting to try this should keep a backup system ready to roll. The wiz.bin is completely interchangeable and sage.properties only has path differences in it so you could even copy it back over to a 32-bit system and edit it to get running.

[wnjj]

Ken,

If you want to start in on the installer, I can always bundle up the binaries I have and put them on Dropbox for you. Getting a clean build environment pull request into Github can take a bit longer. I looked through the installer files and most look like path differences and of course a different binary support file directory. Does it make sense to replicate the installer into a separate one or somehow build options into the existing one or at least share some of the common, bit-independent setup files between them?

There was one 64-bit change I had to make in Java. DirecTVSerialControl.java was passing a handle by and forth as an int instead of jlong. The exec calls may end up getting changed in Java too.

[JustFred]

Great work!!!!!

Originally Posted by wnjj

The next major unknown for me is tuners. I assume those need 64-bit drivers from the manufacturer? Can OpenDCT bridge the gap?

Tuner drivers shouldn't pose any issue. If the tuners are physically installed and running on a 64-bit Windows box, the correct 64-bit drivers are already installed, as they must be, regardless of whether Sage is 32 bit or 64 bit.

I have a SageTv test bed, running Win7-x64, with assorted Hauppauge PCIe tuners as well as several Hauppauge USB capture devices (and their 64-bit USB drivers). This is the system I used when I first ported all the Sage Native code from it's original VS2005 to VS2015. It's also the box used when I added support for the most recent Hauppauge devices in Sage v9. Let me know when you're ready for a smoke test.

[SHS]

Originally Posted by wnjj

void *av_memcpy(void *dest, const void *src, unsigned int size);

Maybe this bit of info wmjj will help
https://patchwork.ffmpeg.org/patch/2659/

I agree with JustFred the Hauppauge Tuner drivers shouldn't pose any issue other then older PVR 250/350 model which is a hardware issues it self, and being that main diff from 32 vs 64 bit has to do with how it handles large amounts of random access memory.

[wnjj]

Originally Posted by SHS

Maybe this bit of info wmjj will help
https://patchwork.ffmpeg.org/patch/2659/

Good catch. For some reason I didn't find that when searching or glossed over it. It looks like a "safe" memcpy may have been the goal. The interesting thing is this discussion is from 2017 yet the Sage source code had something in it years ago. Maybe the devs ran into the bug back then? Still strange how it's not part of the code.

[wnjj]

Originally Posted by JustFred

I have a SageTv test bed, running Win7-x64, with assorted Hauppauge PCIe tuners as well as several Hauppauge USB capture devices (and their 64-bit USB drivers). This is the system I used when I first ported all the Sage Native code from it's original VS2005 to VS2015. It's also the box used when I added support for the most recent Hauppauge devices in Sage v9. Let me know when you're ready for a smoke test.

That would be great if you want to run some tests. All I have is my main Win7/HDHR setup and an extra Win7 PC to play on so my testing is very limited. I also only ever deal with HDHR MPEG2, MP4 from video cameras and ripped DVD's.

I'm sure there will be some 64-bit -vs- pointer issues to resolve. I plan to recompile everything and go through all of the warnings. I've fixed a few already. For example, there are a lot of them where the difference between pointers is compared as an int (i.e. string compares) and practically speaking those should never overflow a 32-bit int.

[jusjoken]

Originally Posted by wnjj

Ken,

If you want to start in on the installer, I can always bundle up the binaries I have and put them on Dropbox for you. Getting a clean build environment pull request into Github can take a bit longer. I looked through the installer files and most look like path differences and of course a different binary support file directory. Does it make sense to replicate the installer into a separate one or somehow build options into the existing one or at least share some of the common, bit-independent setup files between them?

There was one 64-bit change I had to make in Java. DirecTVSerialControl.java was passing a handle by and forth as an int instead of jlong. The exec calls may end up getting changed in Java too.

Having the binaries will be a good start. PM me the location and as time permits I can start down this road.

The final solution for me needs to begin with source code and all the scripts for compiling them in 32 or 64 bit. I have the complete build from source scripted so I run 1 powershell script and it builds from source, creates the various installers and then uploads them to Bintray/Github. But if I have the binaries I can just start later in the process to work out all the steps.

I will also need to change my development environment over to VS2017 even though I don't use the GUI for the build as I use MSBuild.exe through the script.

As I research the WIX installer I see I will need to create a new separate MSI for 64 bit. I could combine 32/64 into a single bootstrapped exe (wrapper that I use that prompts the user) but I don't see the value in that and the exe size would just get large. So I am thinking 2 installers would be available for the server - 1 for 32 bit and 1 for 64. The 2 installers would certainly share any files that are the same between the 2 bitness.

Also, the default install folder for a 64 bit app on windows will be a different location if I follow standard windows process....which I believe I will. Which means an "upgrade" from 32bit to 64bit would require special handling. Likely the user could just point the installer at the old 32bit location (or the installer could search for it) but that will need to be tested to ensure that is a valid case. Likely an upgrade would be an install to the new 64bit location, run once to get the new properties established and then shutdown and copy in the wiz.bin... anyway. this will need input and testing.

k

[wnjj]

Originally Posted by jusjoken

Having the binaries will be a good start. PM me the location and as time permits I can start down this road.

The final solution for me needs to begin with source code and all the scripts for compiling them in 32 or 64 bit. I have the complete build from source scripted so I run 1 powershell script and it builds from source, creates the various installers and then uploads them to Bintray/Github. But if I have the binaries I can just start later in the process to work out all the steps.

I will also need to change my development environment over to VS2017 even though I don't use the GUI for the build as I use MSBuild.exe through the script.

As I research the WIX installer I see I will need to create a new separate MSI for 64 bit. I could combine 32/64 into a single bootstrapped exe (wrapper that I use that prompts the user) but I don't see the value in that and the exe size would just get large. So I am thinking 2 installers would be available for the server - 1 for 32 bit and 1 for 64. The 2 installers would certainly share any files that are the same between the 2 bitness.

Also, the default install folder for a 64 bit app on windows will be a different location if I follow standard windows process....which I believe I will. Which means an "upgrade" from 32bit to 64bit would require special handling. Likely the user could just point the installer at the old 32bit location (or the installer could search for it) but that will need to be tested to ensure that is a valid case. Likely an upgrade would be an install to the new 64bit location, run once to get the new properties established and then shutdown and copy in the wiz.bin... anyway. this will need input and testing.

k

First of all, I hijacked this thread and probably should have started a new one. I'm building all of this in VS2015 so no conversion needed there yet. I also used Windows SDK 7.1a since the link for 6.1 doesn't seem to work anymore. The only missing things in 7.1 were qedit.h and another include with a bunch of clsid's that I just found online. You can probably find and compare the equivalent one from the 6.1 SDK. I think putting those into the code base and using 7.1a makes sense.

I figured there's some value in you having the binaries early because of the logistics of install directories, registry, etc. I'm a bit uncertain about how to manage the compiles so your input is needed here:

I changed all of the VC project files to use $(Platform)/$(Configuration) for intermediate and output file locations. Everything ends up (locally and output) into Win32/Release or x64/Release. This allows building both 32 and 64 without having to move the output files out between builds.

When I got to the MinGW stuff, I realized none of those projects would do that as easily though it could be added (i.e. place output binaries into bit-specific folders). Is that worth doing or would you rather have your main build script build everything for 32-bit, copy the output somewhere, clean and then rebuild 64-bit in the same directories? If that's the case, I may put all of the VS projects back to just 'Release' (and 'Debug') no matter the bitness and just send the final output files into specific directories. The advantage to keeping them split is build speed since any unchanged object files will still be there, per-bitness. Your thoughts on this?

Also, I did get StrmDeMux.ax compiled last night but like so many other things, testing is another story.

[jusjoken]

Originally Posted by wnjj

I changed all of the VC project files to use $(Platform)/$(Configuration) for intermediate and output file locations. Everything ends up (locally and output) into Win32/Release or x64/Release. This allows building both 32 and 64 without having to move the output files out between builds.

When I got to the MinGW stuff, I realized none of those projects would do that as easily though it could be added (i.e. place output binaries into bit-specific folders). Is that worth doing or would you rather have your main build script build everything for 32-bit, copy the output somewhere, clean and then rebuild 64-bit in the same directories? If that's the case, I may put all of the VS projects back to just 'Release' (and 'Debug') no matter the bitness and just send the final output files into specific directories. The advantage to keeping them split is build speed since any unchanged object files will still be there, per-bitness. Your thoughts on this?

Also, I did get StrmDeMux.ax compiled last night but like so many other things, testing is another story.

I prefer that we keep them split as you have already done. There are enough shared files that the process will be easier to manage and quicker. We will just do the same for the MinGW stuff. Do you have all those building in 32bit as well? Many of the binaries previously had no build for them so we have just been copying the original that we had a copy of from the original installer. It would be nice to build as many as possible.

k

[wnjj]

Originally Posted by jusjoken

I prefer that we keep them split as you have already done. There are enough shared files that the process will be easier to manage and quicker. We will just do the same for the MinGW stuff. Do you have all those building in 32bit as well? Many of the binaries previously had no build for them so we have just been copying the original that we had a copy of from the original installer. It would be nice to build as many as possible.

k

Ok will leave them split. Splitting the MInGW stuff make take some work since they were all built with Automake but now just build with configure. I’ll have to see where the output paths are defined. I have to believe it’s a parameter somewhere.

I haven’t built the 32-bit MingGW stuff (other than Imageloader which I did previously) but it should be exactly the same scripts just running from a 32-bit shell. If you haven’t already installed it, Msys2 is the newest incarnation of MinGW and it lets you run either shell right from simple exe programs/shortcuts. It also has a complete package manager “pacman” to install what’s needed and I’ll give you a list of packages. Once in a shell, your drive is mapped already as /c/path/... You can probably launch the shells and start the compile from a batch file so it’s fully automated.

[wnjj]

Originally Posted by SHS

Maybe this bit of info wmjj will help
https://patchwork.ffmpeg.org/patch/2659/

I found this commit where Narflex replaced av_memcpy with memcpy:

https://github.com/google/sagetv/com...b79561ab18c1b9

[Narflex]

My guess on why av_memcpy is like that is because it probably exists in the libavutil binary that we were linking against on Windows...but then we updated the FFMPEG code later and that likely got removed from it but we never recompiled the libavutil binary. It would just be some kind of safe/potentially optimized in some way I can't think of variant of memcpy. (I think we did that for the HD100 which had a different memory setup, I'm not remembering some optimizations we did on that platform for memcpy)

Just defining it as memcpy with pointer validation should be sufficient.

[wnjj]

Just to update everyone:

I haven't stopped working on this but got a little busy recently so not much new to report. I've been cleaning up my code including all of the unnecessary dead ends and working to make the builds automated. Hopefully in the near future there will be at least a beta Windows installer for some of you to try. You can take your wiz.bin and (modified) Sage.properties with you when you switch over and/or take them back to 32-bit Sage if needed.

Ultimate, I plan to document most of the things I ran into for future benefit. I've kept messy notes along the way.

One thing I'm stuck on right now is getting some of the ASM code to compile in FFMPEG. I get an error like this:

C:\msys64\tmp\ccjdCxgt.s: Assembler messages:
C:\msys64\tmp\ccjdCxgt.s:15689: Error: operand type mismatch for `cmp'
make: *** [common.mak:43: libswscale/swscale.o] Error 1

I found exactly one comment on the internet with the exact same error in swscale.o with a dead link to what was wrong and a comment saying someone hadn't fixed it yet for FFMPEG 0.6. This may be a MinGW-specific option I need to tweak but it's not obvious.

I assume FFMPEG compiles fine on Linux 64-bit since it's part of the build scripts? When I got Pushreader compiled before, I had disabled asm in FFMPEG but from what I'm reading online that will cause a large performance hit so I need to get it working.

Jeff, I also have a few questions/comments I'll be sending in a PM.

[stuckless]

I can't help fix the problem, but I can say that ffmpeg does compile under linux (although, I think I had to use a specfic version of gcc (the newer ones don't work)).

It would be nice, if ffmpeg could simply be upgraded to use the latest, where we could pull the latest code, and then just patch it with our changes. I know I've tried that, but, it was too much for me.

[routerunner]

Originally Posted by stuckless

I can't help fix the problem, but I can say that ffmpeg does compile under linux (although, I think I had to use a specfic version of gcc (the newer ones don't work)).

Exactly, and that means that the assembly code doesn't need to change, but the Microsoft assembler (MASM) need to be instructed properly. Please look for conditional compilation part in the assembly source file and flags in the MASM part of your MSVC2017 project. Best hint is to look for existing conditionals and flags for the 32bit compiled version for Windows as starting point.

Huge and massive thanks to wnjj for doing this, it might help me to find time to complete the new generation of the TS decrypter + commercial skipper for the 64bit architectures (Linux/Windows) I've been working on

Cheers
Eddy

[wnjj]

Originally Posted by stuckless

I can't help fix the problem, but I can say that ffmpeg does compile under linux (although, I think I had to use a specfic version of gcc (the newer ones don't work)).

It would be nice, if ffmpeg could simply be upgraded to use the latest, where we could pull the latest code, and then just patch it with our changes. I know I've tried that, but, it was too much for me.

Knowing that Linux compiles helps. I’m trying to build it under MinGW which uses gcc but the configure scripts are platform-aware with sections for MinGW, etc. I got xvid working by fixing an actual bug in the Makefile that was MinGW x64-specific.

Using the latest or at leas something from this decade would be great. Maybe that will be a future challenge for me. . Jeff has suggested that not everything is commented but even using diff’s is tough because undoubtedly there are bug fixes done by the Sage team that have since been applied or aren’t needed in the newer code.

SageWisdom made an attempt here but I haven’t seen any posts from him for awhile. https://forums.sagetv.com/forums/showthread.php?t=62935